mm/swap: Fix release_pages() when releasing devmap pages
PATCH v3 1/1 mm/swap: Fix release_pages() when releasing devmap pages
From: Ira Weiny <[email protected]>
release_pages() is an optimized version of a loop around put_page(). Unfortunately for devmap pages the logic is not entirely correct in release_pages(). This is because device pages can be more than type MEMORY_DEVICE_PUBLIC. There are in fact 4 types, private, public, FS DAX, and PCI P2PDMA. Some of these have specific needs to "put" the page while others do not.
This logic to handle any special needs is contained in put_devmap_managed_page(). Therefore all devmap pages should be processed by this function where we can contain the correct logic for a page put.
Handle all device type pages within release_pages() by calling put_devmap_managed_page() on all devmap pages. If put_devmap_managed_page() returns true the page has been put and we continue with the next page. A false return of put_devmap_managed_page() means the page did not require special processing and should fall to "normal" processing.
This was found via code inspection while determining if release_pages() and the new put_user_pages() could be interchangeable.[1]
[1] https://lore.kernel.org/lkml/[email protected]/
---
Changes from V2:
Update changelog for more clarity as requested by Michal Update comment WRT "failing" of put_devmap_managed_page()
Changes from V1:
Add comment clarifying that put_devmap_managed_page() can still fail.
Add Reviewed-by tags.
mm/swap.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
mm/swap.c +9 -4
@@ -740,15 +740,20 @@ void release_pages(struct page **pages, int nr)if (is_huge_zero_page (page))continue ; -/* Device public page can not be huge page */ -if (is_device_public_page (page)) { +if (is_zone_device_page (page)) {if (locked_pgdat) {spin_unlock_irqrestore (&locked_pgdat->lru_lock, flags); locked_pgdat =NULL ; } -put_devmap_managed_page (page); -continue ; +/* +* Not all zone-device-pages require special +* processing. Those pages return 'false' from +* put_devmap_managed_page() expecting a call to +* put_page_testzero() +*/ +if (put_devmap_managed_page (page)) +continue ; } page =compound_head (page);
Signature · 1 line
2.20.1
Quoted 59 lines · On 6/4/19 9:48 AM, [email protected] wrote:
From: Ira Weiny <[email protected]>
release_pages()is an optimized version of a loop aroundput_page(). Unfortunately for devmap pages the logic is not entirely correct inrelease_pages(). This is because device pages can be more than typeMEMORY_DEVICE_PUBLIC. There are in fact 4 types, private, public, FS DAX, and PCI P2PDMA. Some of these have specific needs to "put" the page while others do not.This logic to handle any special needs is contained in
put_devmap_managed_page(). Therefore all devmap pages should be processed by this function where we can contain the correct logic for a page put.Handle all device type pages within
release_pages()by callingput_devmap_managed_page()on all devmap pages. Ifput_devmap_managed_page()returns true the page has been put and we continue with the next page. A false return ofput_devmap_managed_page()means the page did not require special processing and should fall to "normal" processing.This was found via code inspection while determining if
release_pages()and the newput_user_pages()could be interchangeable.[1][1] https://lore.kernel.org/lkml/[email protected]/
Cc Jérôme Glisse <[email protected]>Cc Michal Hocko <[email protected]>Reviewed-by Dan Williams <[email protected]>Reviewed-by John Hubbard <[email protected]>Signed-off-by Ira Weiny <[email protected]>---
Changes from V2:
Update changelog for more clarity as requested by Michal Update comment WRT "failing" ofput_devmap_managed_page()Changes from V1:
Add comment clarifying thatput_devmap_managed_page()can still fail.
Add Reviewed-by tags.mm/swap.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)mm/swap.c +1
-2@@ -740,15 +740,20 @@ void release_pages(struct page **pages, int nr)if (is_huge_zero_page (page))continue ; -/* Device public page can not be huge page */ -if (is_device_public_page (page)) { +if (is_zone_device_page (page)) {if (locked_pgdat) {spin_unlock_irqrestore (&locked_pgdat->lru_lock, flags);
mm/swap.c void release_pages(struct page **pages, int nr) · line 754
Quoted 10 lines
mm/swap.c +6
-2locked_pgdat =NULL ; } -put_devmap_managed_page (page); -continue ; +/* +* Not all zone-device-pages require special +* processing. Those pages return 'false' from +* put_devmap_managed_page() expecting a call to +* put_page_testzero() +*/
Just a documentation tweak: how about:
/*
* ZONE_DEVICE pages that return 'false' from
* put_devmap_managed_page() do not require special
* processing, and instead, expect a call to
* put_page_testzero().
*/
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Quoted 6 lines
mm/swap.c +2
-0+if (put_devmap_managed_page (page)) +continue ; } page =compound_head (page);
Quoted 74 lines · John Hubbard
Quoted 59 lines · On 6/4/19 9:48 AM, [email protected] wrote:
From: Ira Weiny <[email protected]>
release_pages()is an optimized version of a loop aroundput_page(). Unfortunately for devmap pages the logic is not entirely correct inrelease_pages(). This is because device pages can be more than typeMEMORY_DEVICE_PUBLIC. There are in fact 4 types, private, public, FS DAX, and PCI P2PDMA. Some of these have specific needs to "put" the page while others do not.This logic to handle any special needs is contained in
put_devmap_managed_page(). Therefore all devmap pages should be processed by this function where we can contain the correct logic for a page put.Handle all device type pages within
release_pages()by callingput_devmap_managed_page()on all devmap pages. Ifput_devmap_managed_page()returns true the page has been put and we continue with the next page. A false return ofput_devmap_managed_page()means the page did not require special processing and should fall to "normal" processing.This was found via code inspection while determining if
release_pages()and the newput_user_pages()could be interchangeable.[1][1] https://lore.kernel.org/lkml/[email protected]/
Cc Jérôme Glisse <[email protected]>Cc Michal Hocko <[email protected]>Reviewed-by Dan Williams <[email protected]>Reviewed-by John Hubbard <[email protected]>Signed-off-by Ira Weiny <[email protected]>---
Changes from V2:
Update changelog for more clarity as requested by Michal Update comment WRT "failing" ofput_devmap_managed_page()Changes from V1:
Add comment clarifying thatput_devmap_managed_page()can still fail.
Add Reviewed-by tags.mm/swap.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)mm/swap.c +1
-2@@ -740,15 +740,20 @@ void release_pages(struct page **pages, int nr)if (is_huge_zero_page (page))continue ; -/* Device public page can not be huge page */ -if (is_device_public_page (page)) { +if (is_zone_device_page (page)) {if (locked_pgdat) {spin_unlock_irqrestore (&locked_pgdat->lru_lock, flags);mm/swap.c void release_pages(struct page **pages, int nr) · line 754
Quoted 10 lines
mm/swap.c +6
-2locked_pgdat =NULL ; } -put_devmap_managed_page (page); -continue ; +/* +* Not all zone-device-pages require special +* processing. Those pages return 'false' from +* put_devmap_managed_page() expecting a call to +* put_page_testzero() +*/ Just a documentation tweak: how about:
Quoted 6 lines
/*
*ZONE_DEVICEpages that return 'false' from
*put_devmap_managed_page()do not require special
* processing, and instead, expect a call to
*put_page_testzero().
*/
Looks better to me, but maybe just go ahead and list those expectations explicitly. Something like:
/*
* put_devmap_managed_page() only handles
* ZONE_DEVICE (struct dev_pagemap managed)
* pages when the hosting dev_pagemap has the
* ->free() or ->fault() callback handlers
* implemented as indicated by
* dev_pagemap.type. Otherwise the expectation
* is to fall back to a plain decrement /
* put_page_testzero().
*/
Quoted 5 lines · Dan Williams
Quoted 4 lines · John Hubbard
Quoted 2 lines · On 6/4/19 9:48 AM, [email protected] wrote:
From: Ira Weiny <[email protected]>
...
Quoted 37 lines
Quoted 27 lines
Quoted 14 lines
mm/swap.c +1
-2@@ -740,15 +740,20 @@ void release_pages(struct page **pages, int nr)if (is_huge_zero_page (page))continue ; -/* Device public page can not be huge page */ -if (is_device_public_page (page)) { +if (is_zone_device_page (page)) {if (locked_pgdat) {spin_unlock_irqrestore (&locked_pgdat->lru_lock, flags);mm/swap.c void release_pages(struct page **pages, int nr) · line 754
Quoted 10 lines
mm/swap.c +6
-2locked_pgdat =NULL ; } -put_devmap_managed_page (page); -continue ; +/* +* Not all zone-device-pages require special +* processing. Those pages return 'false' from +* put_devmap_managed_page() expecting a call to +* put_page_testzero() +*/ Just a documentation tweak: how about:
Quoted 6 lines
/*
*ZONE_DEVICEpages that return 'false' from
*put_devmap_managed_page()do not require special
* processing, and instead, expect a call to
*put_page_testzero().
*/Looks better to me, but maybe just go ahead and list those expectations explicitly. Something like:
Quoted 10 lines
/*
*put_devmap_managed_page()only handles
*ZONE_DEVICE(structdev_pagemapmanaged)
* pages when the hostingdev_pagemaphas the
* ->free() or ->fault() callback handlers
* implemented as indicated by
* dev_pagemap.type. Otherwise the expectation
* is to fall back to a plain decrement /
*put_page_testzero().
*/
I like it--but not here, because it's too much internal detail in a call site that doesn't use that level of detail. The call site looks at the return value, only.
Let's instead put that blurb above (or in) the put_devmap_managed_page() routine itself. And leave the blurb that I wrote where it is. And then I think everything will have an appropriate level of detail in the right places.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Quoted 55 lines · John Hubbard
Quoted 5 lines · Dan Williams
Quoted 4 lines · John Hubbard
Quoted 2 lines · On 6/4/19 9:48 AM, [email protected] wrote:
From: Ira Weiny <[email protected]>
...
Quoted 37 lines
Quoted 27 lines
Quoted 14 lines
mm/swap.c +1
-2@@ -740,15 +740,20 @@ void release_pages(struct page **pages, int nr)if (is_huge_zero_page (page))continue ; -/* Device public page can not be huge page */ -if (is_device_public_page (page)) { +if (is_zone_device_page (page)) {if (locked_pgdat) {spin_unlock_irqrestore (&locked_pgdat->lru_lock, flags);mm/swap.c void release_pages(struct page **pages, int nr) · line 754
Quoted 10 lines
mm/swap.c +6
-2locked_pgdat =NULL ; } -put_devmap_managed_page (page); -continue ; +/* +* Not all zone-device-pages require special +* processing. Those pages return 'false' from +* put_devmap_managed_page() expecting a call to +* put_page_testzero() +*/ Just a documentation tweak: how about:
Quoted 6 lines
/*
*ZONE_DEVICEpages that return 'false' from
*put_devmap_managed_page()do not require special
* processing, and instead, expect a call to
*put_page_testzero().
*/Looks better to me, but maybe just go ahead and list those expectations explicitly. Something like:
Quoted 10 lines
/*
*put_devmap_managed_page()only handles
*ZONE_DEVICE(structdev_pagemapmanaged)
* pages when the hostingdev_pagemaphas the
* ->free() or ->fault() callback handlers
* implemented as indicated by
* dev_pagemap.type. Otherwise the expectation
* is to fall back to a plain decrement /
*put_page_testzero().
*/
Quoted 7 lines
I like it--but not here, because it's too much internal detail in a call site that doesn't use that level of detail. The call site looks at the return value, only.
Let's instead put that blurb above (or in) the
put_devmap_managed_page()routine itself. And leave the blurb that I wrote where it is. And then I think everything will have an appropriate level of detail in the right places.
I agree. This leaves it open that this handles any special processing which is required.
FWIW the same call is made in put_page() and has no comment so perhaps we are getting wrapped around the axle for no reason?
Frankly I questioned myself when I mentioned put_page_testzero() as well. But I'm ok with Johns suggestion. My wording was a bit "rushed". Sorry about that. I wanted to remove the word 'fail' from the comment because I think it is what caught Michal's eye.
Ira
Quoted 7 lines
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Quoted 56 lines · John Hubbard
Quoted 5 lines · Dan Williams
Quoted 4 lines · John Hubbard
Quoted 2 lines · On 6/4/19 9:48 AM, [email protected] wrote:
From: Ira Weiny <[email protected]>
...
Quoted 37 lines
Quoted 27 lines
Quoted 14 lines
mm/swap.c +1
-2@@ -740,15 +740,20 @@ void release_pages(struct page **pages, int nr)if (is_huge_zero_page (page))continue ; -/* Device public page can not be huge page */ -if (is_device_public_page (page)) { +if (is_zone_device_page (page)) {if (locked_pgdat) {spin_unlock_irqrestore (&locked_pgdat->lru_lock, flags);mm/swap.c void release_pages(struct page **pages, int nr) · line 754
Quoted 10 lines
mm/swap.c +6
-2locked_pgdat =NULL ; } -put_devmap_managed_page (page); -continue ; +/* +* Not all zone-device-pages require special +* processing. Those pages return 'false' from +* put_devmap_managed_page() expecting a call to +* put_page_testzero() +*/ Just a documentation tweak: how about:
Quoted 6 lines
/*
*ZONE_DEVICEpages that return 'false' from
*put_devmap_managed_page()do not require special
* processing, and instead, expect a call to
*put_page_testzero().
*/Looks better to me, but maybe just go ahead and list those expectations explicitly. Something like:
Quoted 10 lines
/*
*put_devmap_managed_page()only handles
*ZONE_DEVICE(structdev_pagemapmanaged)
* pages when the hostingdev_pagemaphas the
* ->free() or ->fault() callback handlers
* implemented as indicated by
* dev_pagemap.type. Otherwise the expectation
* is to fall back to a plain decrement /
*put_page_testzero().
*/
Quoted 7 lines
I like it--but not here, because it's too much internal detail in a call site that doesn't use that level of detail. The call site looks at the return value, only.
Let's instead put that blurb above (or in) the
put_devmap_managed_page()routine itself. And leave the blurb that I wrote where it is. And then I think everything will have an appropriate level of detail in the right places.
Ok. Ideally there wouldn't be any commentary needed at the call site and the put_page() could be handled internal to put_devmap_managed_page(), but I did not see a way to do that without breaking the compile out / static branch optimization when there are no active ZONE_DEVICE users.