mm/gup: track dma-pinned pages: FOLL_PIN
Subsystems (28)
John Hubbard
reviewerv6 cover mm/gup: track dma-pinned pages: FOLL_PIN
Hi,
Christoph Hellwig has a preference to do things a little differently, for the devmap cleanup in patch 5 ("mm: devmap: refactor 1-based refcounting for ZONE_DEVICE pages"). That came up in a different review thread, because the patch is out for review in two locations. Here's that review thread:
https://lore.kernel.org/r/[email protected]
...and I'm hoping that we can defer that request, because otherwise it derails this series, which is starting to otherwise look like it could be ready for 5.5.
There is a git repo and branch, for convenience:
[email protected]:johnhubbard/linux.git pin_user_pages_tracking_v6
Changes since v5:
* Fixed the refcounting for huge pages: in most cases, it was only taking one GUP_PIN_COUNTING_BIAS's worth of refs, when it should have been taking one GUP_PIN_COUNTING_BIAS for each subpage.
(Much thanks to Jan Kara for spotting that one!)
* Renamed user_page_ref_inc() to try_pin_page(), and added a new try_pin_compound_head(). This definitely improves readability.
* Factored out some more duplication in the FOLL_PIN and FOLL_GET cases, in gup.c.
* Fixed up some straggling "get_" --> "pin_" references in the comments.
* Added reviewed-by tags.
Changes since v4:
* Renamed put_user_page*() --> unpin_user_page().
* Removed all pin_longterm_pages*() calls. We will use FOLL_LONGTERM at the call sites. (FOLL_PIN, however, remains an internal gup flag).
This is very nice: many patches just change three characters now:
get_user_pages --> pin_user_pages. I think we've found the right balance of wrapper calls and gup flags, for the call sites.
* Updated a lot of documentation and commit logs to match the above two large changes.
* Changed gup_benchmark tests and run_vmtests, to adapt to one less use case: there is no pin_longterm_pages() call anymore.
* This includes a new devmap cleanup patch from Dan Williams, along with a rebased follow-up: patches 4 and 5, already mentioned above.
* Fixed patch 10 ("mm/gup: introduce pin_user_pages*() and FOLL_PIN"), so as to make pin_user_pages*() calls act as placeholders for the corresponding get_user_pages*() calls, until a later patch fully implements the DMA-pinning functionality.
Thanks to Jan Kara for noticing that.
* Fixed the implementation of pin_user_pages_remote().
* Further tweaked patch 2 ("mm/gup: factor out duplicate code from four routines"), in response to Jan Kara's feedback.
* Dropped a few reviewed-by tags due to changes that invalidated them.
Changes since v3:
* VFIO fix (patch 8): applied further cleanup: removed a pre-existing, unnecessary release and reacquire of mmap_sem. Moved the DAX vma checks from the vfio call site, to gup internals, and added comments (and commit log) to clarify.
* Due to the above, made a corresponding fix to the pin_longterm_pages_remote(), which was actually calling the wrong gup internal function.
* Changed put_user_page() comments, to refer to pin*() APIs, rather than get_user_pages*() APIs.
* Reverted an accidental whitespace-only change in the IB ODP code.
* Added a few more reviewed-by tags.
Changes since v2:
* Added a patch to convert IB/umem from normal gup, to gup_fast(). This is also posted separately, in order to hopefully get some runtime testing.
* Changed the page devmap code to be a little clearer, thanks to Jerome for that.
* Split out the page devmap changes into a separate patch (and moved Ira's Signed-off-by to that patch).
* Fixed my bug in IB: ODP code does not require pin_user_pages() semantics. Therefore, revert the put_user_page() calls to put_page(), and leave the get_user_pages() call as-is.
* As part of the revert, I am proposing here a change directly from put_user_pages(), to release_pages(). I'd feel better if someone agrees that this is the best way. It uses the more efficient release_pages(), instead of put_page() in a loop, and keep the change to just a few character on one line, but OTOH it is not a pure revert.
* Loosened the FOLL_LONGTERM restrictions in the __get_user_pages_locked() implementation, and used that in order to fix up a VFIO bug. Thanks to Jason for that idea.
* Note the use of release_pages() in IB: is that OK?
* Added a few more WARN's and clarifying comments nearby.
* Many documentation improvements in various comments.
* Moved the new pin_user_pages.rst from Documentation/vm/ to Documentation/core-api/ .
* Commit descriptions: added clarifying notes to the three patches (drm/via, fs/io_uring, net/xdp) that already had put_user_page() calls in place.
* Collected all pending Reviewed-by and Acked-by tags, from v1 and v2 email threads.
* Lot of churn from v2 --> v3, so it's possible that new bugs sneaked in.
NOT DONE: separate patchset is required:
* __get_user_pages_locked(): stop compensating for buggy callers who failed to set FOLL_GET. Instead, assert that FOLL_GET is set (and fail if it's not).
======================================================================
Original cover letter (edited to fix up the patch description numbers)
This applies cleanly to linux-next and mmotm, and also to linux.git if linux-next's commit 20cac10710c9 ("mm/gup_benchmark: fix MAP_HUGETLB case") is first applied there.
This provides tracking of dma-pinned pages. This is a prerequisite to solving the larger problem of proper interactions between file-backed pages, and [R]DMA activities, as discussed in [1], [2], [3], and in a remarkable number of email threads since about 2017. :)
A new internal gup flag, FOLL_PIN is introduced, and thoroughly documented in the last patch's Documentation/vm/pin_user_pages.rst.
I believe that this will provide a good starting point for doing the layout lease work that Ira Weiny has been working on. That's because these new wrapper functions provide a clean, constrained, systematically named set of functionality that, again, is required in order to even know if a page is "dma-pinned".
In contrast to earlier approaches, the page tracking can be incrementally applied to the kernel call sites that, until now, have been simply calling get_user_pages() ("gup"). In other words, opt-in by changing from this:
get_user_pages() (sets FOLL_GET)
put_page()
to this:
pin_user_pages() (sets FOLL_PIN)
put_user_page()
Because there are interdependencies with FOLL_LONGTERM, a similar conversion as for FOLL_PIN, was applied. The change was from this:
get_user_pages(FOLL_LONGTERM) (also sets FOLL_GET) put_page()
to this:
pin_longterm_pages() (sets FOLL_PIN | FOLL_LONGTERM) put_user_page()
============================================================
Patch summary:
* Patches 1-9: refactoring and preparatory cleanup, independent fixes
* Patch 10: introduce pin_user_pages(), FOLL_PIN, but no functional changes yet
* Patches 11-16: Convert existing put_user_page() callers, to use the
new pin*()
* Patch 17: Activate tracking of FOLL_PIN pages.
* Patches 18-20: convert various callers
* Patches: 21-23: gup_benchmark and run_vmtests support
* Patch 24: rename put_user_page*() --> unpin_user_page*()
============================================================
Testing:
* I've done some overall kernel testing (LTP, and a few other goodies), and some directed testing to exercise some of the changes. And as you can see, gup_benchmark is enhanced to exercise this. Basically, I've been able to runtime test the core get_user_pages() and pin_user_pages() and related routines, but not so much on several of the call sites--but those are generally just a couple of lines changed, each.
Not much of the kernel is actually using this, which on one hand reduces risk quite a lot. But on the other hand, testing coverage is low. So I'd love it if, in particular, the Infiniband and PowerPC folks could do a smoke test of this series for me.
Also, my runtime testing for the call sites so far is very weak:
* io_uring: Some directed tests from liburing exercise this, and they pass.
* process_vm_access.c: A small directed test passes.
* gup_benchmark: the enhanced version hits the new gup.c code, and passes.
* infiniband (still only have crude "IB pingpong" working, on a good day: it's not exercising my conversions at runtime...)
* VFIO: compiles (I'm vowing to set up a run time test soon, but it's not ready just yet)
* powerpc: it compiles...
* drm/via: compiles...
* goldfish: compiles...
* net/xdp: compiles...
* media/v4l2: compiles...
============================================================
Next:
* Get the block/bio_vec sites converted to use pin_user_pages().
* Work with Ira and Dave Chinner to weave this together with the layout lease stuff.
============================================================
[1] Some slow progress on get_user_pages() (Apr 2, 2019): https://lwn.net/Articles/784574/
[2] DMA and get_user_pages() (LPC: Dec 12, 2018): https://lwn.net/Articles/774411/
[3] The trouble with get_user_pages() (Apr 30, 2018): https://lwn.net/Articles/753027/
Dan Williams (1):
mm: Cleanup __put_devmap_managed_page() vs ->page_free()
John Hubbard (23):
mm/gup: pass flags arg to __gup_device_* functions mm/gup: factor out duplicate code from four routines mm/gup: move try_get_compound_head() to top, fix minor issues
mm: devmap: refactor 1-based refcounting for ZONE_DEVICE pages
goldish_pipe: rename local pin_user_pages() routine IB/umem: use get_user_pages_fast() to pin DMA pages media/v4l2-core: set pages dirty upon releasing DMA buffers vfio, mm: fix get_user_pages_remote() and FOLL_LONGTERM mm/gup: introduce pin_user_pages*() and FOLL_PIN
goldish_pipe: convert to pin_user_pages() and put_user_page() IB/{core,hw,umem}: set FOLL_PIN via pin_user_pages*(), fix up ODP mm/process_vm_access: set FOLL_PIN via pin_user_pages_remote() drm/via: set FOLL_PIN via pin_user_pages_fast() fs/io_uring: set FOLL_PIN via pin_user_pages()
net/xdp: set FOLL_PIN via pin_user_pages()
mm/gup: track FOLL_PIN pages
media/v4l2-core: pin_user_pages (FOLL_PIN) and put_user_page()
conversion
vfio, mm: pin_user_pages (FOLL_PIN) and put_user_page() conversion
powerpc: book3s64: convert to pin_user_pages() and put_user_page() mm/gup_benchmark: use proper FOLL_WRITE flags instead of hard-coding
"1"
mm/gup_benchmark: support pin_user_pages() and related calls selftests/vm: run_vmtests: invoke gup_benchmark with basic FOLL_PIN
coverage
mm, tree-wide: rename put_user_page*() to unpin_user_page*()
Documentation/core-api/index.rst | 1 + Documentation/core-api/pin_user_pages.rst | 233 +++++++++ arch/powerpc/mm/book3s64/iommu_api.c | 12 +- drivers/gpu/drm/via/via_dmablit.c | 6 +- drivers/infiniband/core/umem.c | 19 +- drivers/infiniband/core/umem_odp.c | 13 +- drivers/infiniband/hw/hfi1/user_pages.c | 4 +- drivers/infiniband/hw/mthca/mthca_memfree.c | 8 +- drivers/infiniband/hw/qib/qib_user_pages.c | 4 +- drivers/infiniband/hw/qib/qib_user_sdma.c | 8 +- drivers/infiniband/hw/usnic/usnic_uiom.c | 4 +- drivers/infiniband/sw/siw/siw_mem.c | 4 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 8 +- drivers/nvdimm/pmem.c | 6 - drivers/platform/goldfish/goldfish_pipe.c | 35 +- drivers/vfio/vfio_iommu_type1.c | 35 +- fs/io_uring.c | 6 +- include/linux/mm.h | 168 +++++- include/linux/mmzone.h | 2 + include/linux/page_ref.h | 10 + mm/gup.c | 548 +++++++++++++++----- mm/gup_benchmark.c | 74 ++- mm/huge_memory.c | 54 +- mm/hugetlb.c | 39 +- mm/memremap.c | 76 ++- mm/process_vm_access.c | 28 +- mm/vmstat.c | 2 + net/xdp/xdp_umem.c | 4 +- tools/testing/selftests/vm/gup_benchmark.c | 21 +- tools/testing/selftests/vm/run_vmtests | 22 + 30 files changed, 1104 insertions(+), 350 deletions(-) create mode 100644 Documentation/core-api/pin_user_pages.rst
Signature · 6 lines
2.24.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
John Hubbard
reviewerPATCH v6 1/24 mm/gup: pass flags arg to __gup_device_* functions
A subsequent patch requires access to gup flags, so pass the flags argument through to the __gup_device_* functions.
Also placate checkpatch.pl by shortening a nearby line.
--- mm/gup.c | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-)
mm/gup.c +18 -10
@@ -1890,7 +1890,8 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,#if defined(CONFIG_ARCH_HAS_PTE_DEVMAP ) && defined(CONFIG_TRANSPARENT_HUGEPAGE )static int __gup_device_huge (unsigned long pfn,unsigned long addr, -unsigned long end,struct page **pages,int *nr) +unsigned long end,unsigned int flags, +struct page **pages,int *nr) {int nr_start = *nr;struct dev_pagemap *pgmap =NULL ;
@@ -1916,13 +1917,14 @@ static int __gup_device_huge(unsigned long pfn, unsigned long addr, }static int __gup_device_huge_pmd (pmd_t orig,pmd_t *pmdp,unsigned long addr, -unsigned long end,struct page **pages,int *nr) +unsigned long end,unsigned int flags, +struct page **pages,int *nr) {unsigned long fault_pfn;int nr_start = *nr; fault_pfn =pmd_pfn (orig) + ((addr & ~PMD_MASK ) >>PAGE_SHIFT ); -if (!__gup_device_huge (fault_pfn, addr, end, pages, nr)) +if (!__gup_device_huge (fault_pfn, addr, end, flags, pages, nr))return 0 ;if (unlikely (pmd_val (orig) !=pmd_val (*pmdp))) {
@@ -1933,13 +1935,14 @@ static int __gup_device_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, }static int __gup_device_huge_pud (pud_t orig,pud_t *pudp,unsigned long addr, -unsigned long end,struct page **pages,int *nr) +unsigned long end,unsigned int flags, +struct page **pages,int *nr) {unsigned long fault_pfn;int nr_start = *nr; fault_pfn =pud_pfn (orig) + ((addr & ~PUD_MASK ) >>PAGE_SHIFT ); -if (!__gup_device_huge (fault_pfn, addr, end, pages, nr)) +if (!__gup_device_huge (fault_pfn, addr, end, flags, pages, nr))return 0 ;if (unlikely (pud_val (orig) !=pud_val (*pudp))) {
@@ -1950,14 +1953,16 @@ static int __gup_device_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, }#else static int __gup_device_huge_pmd (pmd_t orig,pmd_t *pmdp,unsigned long addr, -unsigned long end,struct page **pages,int *nr) +unsigned long end,unsigned int flags, +struct page **pages,int *nr) {BUILD_BUG ();return 0 ; }static int __gup_device_huge_pud (pud_t pud,pud_t *pudp,unsigned long addr, -unsigned long end,struct page **pages,int *nr) +unsigned long end,unsigned int flags, +struct page **pages,int *nr) {BUILD_BUG ();return 0 ;
@@ -2062,7 +2067,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr,if (pmd_devmap (orig)) {if (unlikely (flags &FOLL_LONGTERM ))return 0 ; -return __gup_device_huge_pmd (orig, pmdp, addr, end, pages, nr); +return __gup_device_huge_pmd (orig, pmdp, addr, end, flags, + pages, nr); } refs =0 ;
@@ -2092,7 +2098,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, }static int gup_huge_pud (pud_t orig,pud_t *pudp,unsigned long addr, -unsigned long end,unsigned int flags,struct page **pages,int *nr) +unsigned long end,unsigned int flags, +struct page **pages,int *nr) {struct page *head, *page;int refs;
@@ -2103,7 +2110,8 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr,if (pud_devmap (orig)) {if (unlikely (flags &FOLL_LONGTERM ))return 0 ; -return __gup_device_huge_pud (orig, pudp, addr, end, pages, nr); +return __gup_device_huge_pud (orig, pudp, addr, end, flags, + pages, nr); } refs =0 ;
Signature · 6 lines
2.24.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
John Hubbard
reviewerPATCH v6 2/24 mm/gup: factor out duplicate code from four routines
In reply to John HubbardThere are four locations in gup.c that have a fair amount of code duplication. This means that changing one requires making the same changes in four places, not to mention reading the same code four times, and wondering if there are subtle differences.
Factor out the common code into static functions, thus reducing the overall line count and the code's complexity.
Also, take the opportunity to slightly improve the efficiency of the error cases, by doing a mass subtraction of the refcount, surrounded by get_page()/put_page().
Also, further simplify (slightly), by waiting until the the successful end of each routine, to increment *nr.
--- mm/gup.c | 91 ++++++++++++++++++++++---------------------------------- 1 file changed, 36 insertions(+), 55 deletions(-)
mm/gup.c +36 -55
@@ -1969,6 +1969,25 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, }#endif +static int __record_subpages (struct page *page,unsigned long addr, +unsigned long end,struct page **pages) +{ +int nr; + +for (nr =0 ; addr != end; addr +=PAGE_SIZE ) + pages[nr++] = page++; + +return nr; +} + +static void put_compound_head (struct page *page,int refs) +{ +/* Do a get_page() first, in case refs == page->_refcount */ +get_page (page); +page_ref_sub (page, refs); +put_page (page); +} +#ifdef CONFIG_ARCH_HAS_HUGEPD static unsigned long hugepte_addr_end (unsigned long addr,unsigned long end,unsigned long sz)
@@ -1998,32 +2017,20 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr,/* hugepages are never "special" */ VM_BUG_ON (!pfn_valid (pte_pfn (pte))); - refs =0 ; head =pte_page (pte); - page = head + ((addr & (sz-1 )) >>PAGE_SHIFT ); -do { -VM_BUG_ON (compound_head (page) != head); - pages[*nr] = page; - (*nr)++; - page++; - refs++; - }while (addr +=PAGE_SIZE , addr != end); + refs =__record_subpages (page, addr, end, pages + *nr); head =try_get_compound_head (head, refs); -if (!head) { - *nr -= refs; +if (!head)return 0 ; - }if (unlikely (pte_val (pte) !=pte_val (*ptep))) { -/* Could be optimized better */ - *nr -= refs; -while (refs--) -put_page (head); +put_compound_head (head, refs);return 0 ; } + *nr += refs;SetPageReferenced (head);return 1 ; }
@@ -2071,28 +2078,19 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, pages, nr); } - refs =0 ; page =pmd_page (orig) + ((addr & ~PMD_MASK ) >>PAGE_SHIFT ); -do { - pages[*nr] = page; - (*nr)++; - page++; - refs++; - }while (addr +=PAGE_SIZE , addr != end); + refs =__record_subpages (page, addr, end, pages + *nr); head =try_get_compound_head (pmd_page (orig), refs); -if (!head) { - *nr -= refs; +if (!head)return 0 ; - }if (unlikely (pmd_val (orig) !=pmd_val (*pmdp))) { - *nr -= refs; -while (refs--) -put_page (head); +put_compound_head (head, refs);return 0 ; } + *nr += refs;SetPageReferenced (head);return 1 ; }
@@ -2114,28 +2112,19 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, pages, nr); } - refs =0 ; page =pud_page (orig) + ((addr & ~PUD_MASK ) >>PAGE_SHIFT ); -do { - pages[*nr] = page; - (*nr)++; - page++; - refs++; - }while (addr +=PAGE_SIZE , addr != end); + refs =__record_subpages (page, addr, end, pages + *nr); head =try_get_compound_head (pud_page (orig), refs); -if (!head) { - *nr -= refs; +if (!head)return 0 ; - }if (unlikely (pud_val (orig) !=pud_val (*pudp))) { - *nr -= refs; -while (refs--) -put_page (head); +put_compound_head (head, refs);return 0 ; } + *nr += refs;SetPageReferenced (head);return 1 ; }
@@ -2151,28 +2140,20 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr,return 0 ;BUILD_BUG_ON (pgd_devmap (orig)); - refs =0 ; + page =pgd_page (orig) + ((addr & ~PGDIR_MASK ) >>PAGE_SHIFT ); -do { - pages[*nr] = page; - (*nr)++; - page++; - refs++; - }while (addr +=PAGE_SIZE , addr != end); + refs =__record_subpages (page, addr, end, pages + *nr); head =try_get_compound_head (pgd_page (orig), refs); -if (!head) { - *nr -= refs; +if (!head)return 0 ; - }if (unlikely (pgd_val (orig) !=pgd_val (*pgdp))) { - *nr -= refs; -while (refs--) -put_page (head); +put_compound_head (head, refs);return 0 ; } + *nr += refs;SetPageReferenced (head);return 1 ; }
Signature · 6 lines
2.24.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH v6 02/24] mm/gup: factor out duplicate code from four routines
Quoted 15 lines · John Hubbard
There are four locations in
gup.cthat have a fair amount of code duplication. This means that changing one requires making the same changes in four places, not to mention reading the same code four times, and wondering if there are subtle differences.Factor out the common code into static functions, thus reducing the overall line count and the code's complexity.
Also, take the opportunity to slightly improve the efficiency of the error cases, by doing a mass subtraction of the refcount, surrounded by get_page()/put_page().
Also, further simplify (slightly), by waiting until the the successful end of each routine, to increment *nr.
Quoted 6 lines
Reviewed-by Jérôme Glisse <[email protected]>Cc Jan Kara <[email protected]>Cc Ira Weiny <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Aneesh Kumar K.V <[email protected]>Signed-off-by John Hubbard <[email protected]>
Looks good to me now! You can add:
Honza
Quoted 174 lines
--- mm/gup.c | 91 ++++++++++++++++++++++---------------------------------- 1 file changed, 36 insertions(+), 55 deletions(-)mm/gup.c +36
-55@@ -1969,6 +1969,25 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, }#endif +static int __record_subpages (struct page *page,unsigned long addr, +unsigned long end,struct page **pages) +{ +int nr; + +for (nr =0 ; addr != end; addr +=PAGE_SIZE ) + pages[nr++] = page++; + +return nr; +} + +static void put_compound_head (struct page *page,int refs) +{ +/* Do a get_page() first, in case refs == page->_refcount */ +get_page (page); +page_ref_sub (page, refs); +put_page (page); +} +#ifdef CONFIG_ARCH_HAS_HUGEPD static unsigned long hugepte_addr_end (unsigned long addr,unsigned long end,unsigned long sz)@@ -1998,32 +2017,20 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr,/* hugepages are never "special" */ VM_BUG_ON (!pfn_valid (pte_pfn (pte))); - refs =0 ; head =pte_page (pte); - page = head + ((addr & (sz-1 )) >>PAGE_SHIFT ); -do { -VM_BUG_ON (compound_head (page) != head); - pages[*nr] = page; - (*nr)++; - page++; - refs++; - }while (addr +=PAGE_SIZE , addr != end); + refs =__record_subpages (page, addr, end, pages + *nr); head =try_get_compound_head (head, refs); -if (!head) { - *nr -= refs; +if (!head)return 0 ; - }if (unlikely (pte_val (pte) !=pte_val (*ptep))) { -/* Could be optimized better */ - *nr -= refs; -while (refs--) -put_page (head); +put_compound_head (head, refs);return 0 ; } + *nr += refs;SetPageReferenced (head);return 1 ; }@@ -2071,28 +2078,19 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, pages, nr); } - refs =0 ; page =pmd_page (orig) + ((addr & ~PMD_MASK ) >>PAGE_SHIFT ); -do { - pages[*nr] = page; - (*nr)++; - page++; - refs++; - }while (addr +=PAGE_SIZE , addr != end); + refs =__record_subpages (page, addr, end, pages + *nr); head =try_get_compound_head (pmd_page (orig), refs); -if (!head) { - *nr -= refs; +if (!head)return 0 ; - }if (unlikely (pmd_val (orig) !=pmd_val (*pmdp))) { - *nr -= refs; -while (refs--) -put_page (head); +put_compound_head (head, refs);return 0 ; } + *nr += refs;SetPageReferenced (head);return 1 ; }@@ -2114,28 +2112,19 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, pages, nr); } - refs =0 ; page =pud_page (orig) + ((addr & ~PUD_MASK ) >>PAGE_SHIFT ); -do { - pages[*nr] = page; - (*nr)++; - page++; - refs++; - }while (addr +=PAGE_SIZE , addr != end); + refs =__record_subpages (page, addr, end, pages + *nr); head =try_get_compound_head (pud_page (orig), refs); -if (!head) { - *nr -= refs; +if (!head)return 0 ; - }if (unlikely (pud_val (orig) !=pud_val (*pudp))) { - *nr -= refs; -while (refs--) -put_page (head); +put_compound_head (head, refs);return 0 ; } + *nr += refs;SetPageReferenced (head);return 1 ; }@@ -2151,28 +2140,20 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr,return 0 ;BUILD_BUG_ON (pgd_devmap (orig)); - refs =0 ; + page =pgd_page (orig) + ((addr & ~PGDIR_MASK ) >>PAGE_SHIFT ); -do { - pages[*nr] = page; - (*nr)++; - page++; - refs++; - }while (addr +=PAGE_SIZE , addr != end); + refs =__record_subpages (page, addr, end, pages + *nr); head =try_get_compound_head (pgd_page (orig), refs); -if (!head) { - *nr -= refs; +if (!head)return 0 ; - }if (unlikely (pgd_val (orig) !=pgd_val (*pgdp))) { - *nr -= refs; -while (refs--) -put_page (head); +put_compound_head (head, refs);return 0 ; } + *nr += refs;SetPageReferenced (head);return 1 ; }Signature · 1 line
2.24.0
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
John Hubbard
reviewerPATCH v6 3/24 mm/gup: move try_get_compound_head() to top, fix minor issues
In reply to John HubbardAn upcoming patch uses try_get_compound_head() more widely, so move it to the top of gup.c.
Also fix a tiny spelling error and a checkpatch.pl warning.
--- mm/gup.c | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-)
mm/gup.c +15 -14
@@ -29,6 +29,21 @@ struct follow_page_context {unsigned int page_mask; }; +/* +* Return the compound head page with ref appropriately incremented, +* or NULL if that failed. +*/ +static inline struct page *try_get_compound_head (struct page *page,int refs) +{ +struct page *head =compound_head (page); + +if (WARN_ON_ONCE (page_ref_count (head) <0 )) +return NULL ; +if (unlikely (!page_cache_add_speculative (head, refs))) +return NULL ; +return head; +} +/** * put_user_pages_dirty_lock() - release and optionally dirty gup-pinned pages * @pages: array of pages to be maybe marked dirty, and definitely released.
@@ -1793,20 +1808,6 @@ static void __maybe_unused undo_dev_pagemap(int *nr, int nr_start, } } -/* -* Return the compund head page with ref appropriately incremented, -* or NULL if that failed. -*/ -static inline struct page *try_get_compound_head (struct page *page,int refs) -{ -struct page *head =compound_head (page); -if (WARN_ON_ONCE (page_ref_count (head) <0 )) -return NULL ; -if (unlikely (!page_cache_add_speculative (head, refs))) -return NULL ; -return head; -} -#ifdef CONFIG_ARCH_HAS_PTE_SPECIAL static int gup_pte_range (pmd_t pmd,unsigned long addr,unsigned long end,unsigned int flags,struct page **pages,int *nr)
Signature · 6 lines
2.24.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
John Hubbard
reviewerPATCH v6 4/24 mm: Cleanup __put_devmap_managed_page() vs ->page_free()
In reply to John HubbardFrom: Dan Williams <[email protected]>
After the removal of the device-public infrastructure there are only 2 ->page_free() call backs in the kernel. One of those is a device-private callback in the nouveau driver, the other is a generic wakeup needed in the DAX case. In the hopes that all ->page_free() callbacks can be migrated to common core kernel functionality, move the device-private specific actions in __put_devmap_managed_page() under the is_device_private_page() conditional, including the ->page_free() callback. For the other page types just open-code the generic wakeup.
Yes, the wakeup is only needed in the MEMORY_DEVICE_FSDAX case, but it does no harm in the MEMORY_DEVICE_DEVDAX and MEMORY_DEVICE_PCI_P2PDMA case.
--- drivers/nvdimm/pmem.c | 6 ---- mm/memremap.c | 80 ++++++++++++++++++++++++------------------- 2 files changed, 44 insertions(+), 42 deletions(-)
drivers/nvdimm/pmem.c +0 -6
@@ -338,13 +338,7 @@ static void pmem_release_disk(void *__pmem)put_disk (pmem->disk); } -static void pmem_pagemap_page_free (struct page *page) -{ -wake_up_var (&page->_refcount); -} -static const struct dev_pagemap_ops fsdax_pagemap_ops = { - .page_free = pmem_pagemap_page_free, .kill = pmem_pagemap_kill, .cleanup = pmem_pagemap_cleanup, };
mm/memremap.c +44 -36
95 lines not on this page · raw message
Signature · 6 lines
2.24.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
John Hubbard
reviewerPATCH v6 5/24 mm: devmap: refactor 1-based refcounting for ZONE_DEVICE pages
In reply to John HubbardAn upcoming patch changes and complicates the refcounting and especially the "put page" aspects of it. In order to keep everything clean, refactor the devmap page release routines:
* Rename put_devmap_managed_page() to page_is_devmap_managed(), and limit the functionality to "read only": return a bool, with no side effects.
* Add a new routine, put_devmap_managed_page(), to handle checking what kind of page it is, and what kind of refcount handling it requires.
* Rename __put_devmap_managed_page() to free_devmap_managed_page(), and limit the functionality to unconditionally freeing a devmap page.
This is originally based on a separate patch by Ira Weiny, which applied to an early version of the put_user_page() experiments. Since then, Jérôme Glisse suggested the refactoring described above.
--- include/linux/mm.h | 27 ++++++++++++++++++++++++--- mm/memremap.c | 16 ++-------------- 2 files changed, 26 insertions(+), 17 deletions(-)
include/linux/mm.h +24 -3
@@ -967,9 +967,10 @@ static inline bool is_zone_device_page(const struct page *page)#endif #ifdef CONFIG_DEV_PAGEMAP_OPS -void __put_devmap_managed_page (struct page *page); +void free_devmap_managed_page (struct page *page);DECLARE_STATIC_KEY_FALSE (devmap_managed_key); -static inline bool put_devmap_managed_page (struct page *page) + +static inline bool page_is_devmap_managed (struct page *page) {if (!static_branch_unlikely (&devmap_managed_key))return false;
@@ -978,7 +979,6 @@ static inline bool put_devmap_managed_page(struct page *page)switch (page->pgmap->type) {case MEMORY_DEVICE_PRIVATE :case MEMORY_DEVICE_FS_DAX : -__put_devmap_managed_page (page);return true;default :break ;
@@ -986,6 +986,27 @@ static inline bool put_devmap_managed_page(struct page *page) return false; } +static inline bool put_devmap_managed_page (struct page *page) +{ +bool is_devmap =page_is_devmap_managed (page); + +if (is_devmap) { +int count =page_ref_dec_return (page); + +/* +* devmap page refcounts are 1-based, rather than 0-based: if +* refcount is 1, then the page is free and the refcount is +* stable because nobody holds a reference on the page. +*/ +if (count ==1 ) +free_devmap_managed_page (page); +else if (!count) +__put_page (page); + } + +return is_devmap; +} +#else /* CONFIG_DEV_PAGEMAP_OPS */ static inline bool put_devmap_managed_page (struct page *page) {
mm/memremap.c +2 -14
@@ -411,20 +411,8 @@ struct dev_pagemap *get_dev_pagemap(unsigned long pfn,EXPORT_SYMBOL_GPL (get_dev_pagemap);#ifdef CONFIG_DEV_PAGEMAP_OPS -void __put_devmap_managed_page (struct page *page) +void free_devmap_managed_page (struct page *page) { -int count =page_ref_dec_return (page); - -/* still busy */ -if (count >1 ) -return ; - -/* only triggered by the dev_pagemap shutdown path */ -if (count ==0 ) { -__put_page (page); -return ; - } -/* notify page idle for dax */ if (!is_device_private_page (page)) {wake_up_var (&page->_refcount);
@@ -461,5 +449,5 @@ void __put_devmap_managed_page(struct page *page) page->mapping =NULL ; page->pgmap->ops->page_free(page); } -EXPORT_SYMBOL (__put_devmap_managed_page); +EXPORT_SYMBOL (free_devmap_managed_page);#endif /* CONFIG_DEV_PAGEMAP_OPS */
Signature · 1 line
2.24.0
John Hubbard
reviewerPATCH v6 6/24 goldish_pipe: rename local pin_user_pages() routine
In reply to John Hubbard1. Avoid naming conflicts: rename local static function from "pin_user_pages()" to "pin_goldfish_pages()".
An upcoming patch will introduce a global pin_user_pages() function.
--- drivers/platform/goldfish/goldfish_pipe.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
drivers/platform/goldfish/goldfish_pipe.c +9 -9
@@ -257,12 +257,12 @@ static int goldfish_pipe_error_convert(int status) } } -static int pin_user_pages (unsigned long first_page, -unsigned long last_page, -unsigned int last_page_size, -int is_write, -struct page *pages[MAX_BUFFERS_PER_COMMAND ], -unsigned int *iter_last_page_size) +static int pin_goldfish_pages (unsigned long first_page, +unsigned long last_page, +unsigned int last_page_size, +int is_write, +struct page *pages[MAX_BUFFERS_PER_COMMAND ], +unsigned int *iter_last_page_size) {int ret;int requested_pages = ((last_page - first_page) >>PAGE_SHIFT ) +1 ;
@@ -354,9 +354,9 @@ static int transfer_max_buffers(struct goldfish_pipe *pipe,if (mutex_lock_interruptible (&pipe->lock))return -ERESTARTSYS ; - pages_count =pin_user_pages (first_page, last_page, - last_page_size, is_write, - pipe->pages, &iter_last_page_size); + pages_count =pin_goldfish_pages (first_page, last_page, + last_page_size, is_write, + pipe->pages, &iter_last_page_size);if (pages_count <0 ) {mutex_unlock (&pipe->lock);return pages_count;
Signature · 6 lines
2.24.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
John Hubbard
reviewerPATCH v6 7/24 IB/umem: use get_user_pages_fast() to pin DMA pages
In reply to John HubbardAnd get rid of the mmap_sem calls, as part of that. Note that get_user_pages_fast() will, if necessary, fall back to __gup_longterm_unlocked(), which takes the mmap_sem as needed.
--- drivers/infiniband/core/umem.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
drivers/infiniband/core/umem.c +6 -11
@@ -271,16 +271,13 @@ struct ib_umem *ib_umem_get(struct ib_udata *udata, unsigned long addr, sg = umem->sg_head.sgl;while (npages) { -down_read (&mm->mmap_sem); - ret =get_user_pages (cur_base, - min_t(unsigned long, npages, -PAGE_SIZE /sizeof (struct page *)), - gup_flags |FOLL_LONGTERM , - page_list,NULL ); -if (ret <0 ) { -up_read (&mm->mmap_sem); + ret =get_user_pages_fast (cur_base, + min_t(unsigned long, npages, +PAGE_SIZE / +sizeof (struct page *)), + gup_flags |FOLL_LONGTERM , page_list); +if (ret <0 ) goto umem_release; - } cur_base += ret *PAGE_SIZE ; npages -= ret;
@@ -288,8 +285,6 @@ struct ib_umem *ib_umem_get(struct ib_udata *udata, unsigned long addr, sg =ib_umem_add_sg_table (sg, page_list, ret,dma_get_max_seg_size (context->device->dma_device), &umem->sg_nents); - -up_read (&mm->mmap_sem); }sg_mark_end (sg);
Signature · 6 lines
2.24.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
John Hubbard
reviewerPATCH v6 8/24 media/v4l2-core: set pages dirty upon releasing DMA buffers
In reply to John HubbardAfter DMA is complete, and the device and CPU caches are synchronized, it's still required to mark the CPU pages as dirty, if the data was coming from the device. However, this driver was just issuing a bare put_page() call, without any set_page_dirty*() call.
Fix the problem, by calling set_page_dirty_lock() if the CPU pages were potentially receiving data from the device.
--- drivers/media/v4l2-core/videobuf-dma-sg.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/media/v4l2-core/videobuf-dma-sg.c +4 -1
@@ -349,8 +349,11 @@ int videobuf_dma_free(struct videobuf_dmabuf *dma)BUG_ON (dma->sglen);if (dma->pages) { -for (i =0 ; i < dma->nr_pages; i++) +for (i =0 ; i < dma->nr_pages; i++) { +if (dma->direction ==DMA_FROM_DEVICE ) +set_page_dirty_lock (dma->pages[i]);put_page (dma->pages[i]); + }kfree (dma->pages); dma->pages =NULL ; }
Signature · 6 lines
2.24.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
John Hubbard
reviewerPATCH v6 9/24 vfio, mm: fix get_user_pages_remote() and FOLL_LONGTERM
In reply to John HubbardAs it says in the updated comment in gup.c: current FOLL_LONGTERM behavior is incompatible with FAULT_FLAG_ALLOW_RETRY because of the FS DAX check requirement on vmas.
However, the corresponding restriction in get_user_pages_remote() was slightly stricter than is actually required: it forbade all FOLL_LONGTERM callers, but we can actually allow FOLL_LONGTERM callers that do not set the "locked" arg.
Update the code and comments accordingly, and update the VFIO caller to take advantage of this, fixing a bug as a result: the VFIO caller is logically a FOLL_LONGTERM user.
Also, remove an unnessary pair of calls that were releasing and reacquiring the mmap_sem. There is no need to avoid holding mmap_sem just in order to call page_to_pfn().
Also, move the DAX check ("if a VMA is DAX, don't allow long term pinning") from the VFIO call site, all the way into the internals of get_user_pages_remote() and __gup_longterm_locked(). That is:get_user_pages_remote() calls __gup_longterm_locked(), which in turn calls check_dax_vmas(). It's lightly explained in the comments as well.
Thanks to Jason Gunthorpe for pointing out a clean way to fix this, and to Dan Williams for helping clarify the DAX refactoring.
--- drivers/vfio/vfio_iommu_type1.c | 30 +++++------------------------- mm/gup.c | 27 ++++++++++++++++++++++----- 2 files changed, 27 insertions(+), 30 deletions(-)
drivers/vfio/vfio_iommu_type1.c +5 -25
@@ -340,7 +340,6 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr, {struct page *page[1 ];struct vm_area_struct *vma; -struct vm_area_struct *vmas[1 ];unsigned int flags =0 ;int ret;
@@ -348,33 +347,14 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr, flags |=FOLL_WRITE ;down_read (&mm->mmap_sem); -if (mm == current->mm) { - ret =get_user_pages (vaddr,1 , flags |FOLL_LONGTERM , page, - vmas); - }else { - ret =get_user_pages_remote (NULL , mm, vaddr,1 , flags, page, - vmas,NULL ); -/* -* The lifetime of a vaddr_get_pfn() page pin is -* userspace-controlled. In the fs-dax case this could -* lead to indefinite stalls in filesystem operations. -* Disallow attempts to pin fs-dax pages via this -* interface. -*/ -if (ret >0 &&vma_is_fsdax (vmas[0 ])) { - ret = -EOPNOTSUPP ; -put_page (page[0 ]); - } - } -up_read (&mm->mmap_sem); - + ret =get_user_pages_remote (NULL , mm, vaddr,1 , flags |FOLL_LONGTERM , + page,NULL ,NULL );if (ret ==1 ) { *pfn =page_to_pfn (page[0 ]); -return 0 ; + ret =0 ; + goto done; } -down_read (&mm->mmap_sem); - vaddr =untagged_addr (vaddr); vma =find_vma_intersection (mm, vaddr, vaddr +1 );
@@ -384,7 +364,7 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr,if (is_invalid_reserved_pfn (*pfn)) ret =0 ; } - +done:up_read (&mm->mmap_sem);return ret; }
mm/gup.c +22 -5
@@ -29,6 +29,13 @@ struct follow_page_context {unsigned int page_mask; }; +static __always_inline long __gup_longterm_locked (struct task_struct *tsk, +struct mm_struct *mm, +unsigned long start, +unsigned long nr_pages, +struct page **pages, +struct vm_area_struct **vmas, +unsigned int flags);/* * Return the compound head page with ref appropriately incremented, * or NULL if that failed.
@@ -1167,13 +1174,23 @@ long get_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm, struct vm_area_struct **vmas, int *locked) {/* -* FIXME: Current FOLL_LONGTERM behavior is incompatible with +* Parts of FOLL_LONGTERM behavior are incompatible with * FAULT_FLAG_ALLOW_RETRY because of the FS DAX check requirement on -* vmas. As there are no users of this flag in this call we simply -* disallow this option for now. +* vmas. However, this only comes up if locked is set, and there are +* callers that do request FOLL_LONGTERM, but do not set locked. So, +* allow what we can. */ -if (WARN_ON_ONCE (gup_flags &FOLL_LONGTERM )) -return -EINVAL ; +if (gup_flags &FOLL_LONGTERM ) { +if (WARN_ON_ONCE (locked)) +return -EINVAL ; +/* +* This will check the vmas (even if our vmas arg is NULL) +* and return -ENOTSUPP if DAX isn't allowed in this case: +*/ +return __gup_longterm_locked (tsk, mm, start, nr_pages, pages, + vmas, gup_flags |FOLL_TOUCH | +FOLL_REMOTE ); + } return __get_user_pages_locked(tsk, mm, start, nr_pages, pages, vmas, locked,
Signature · 1 line
2.24.0
John Hubbard
reviewerPATCH v6 10/24 mm/gup: introduce pin_user_pages*() and FOLL_PIN
In reply to John HubbardIntroduce pin_user_pages*() variations of get_user_pages*() calls, and also pin_longterm_pages*() variations.
For now, these are placeholder calls, until the various call sites are converted to use the correct get_user_pages*() or pin_user_pages*() API.
These variants will eventually all set FOLL_PIN, which is also introduced, and thoroughly documented.
pin_user_pages()
pin_user_pages_remote()
pin_user_pages_fast()
All pages that are pinned via the above calls, must be unpinned via put_user_page().
The underlying rules are:
* FOLL_PIN is a gup-internal flag, so the call sites should not directly set it. That behavior is enforced with assertions.
* Call sites that want to indicate that they are going to do DirectIO ("DIO") or something with similar characteristics, should call a get_user_pages()-like wrapper call that sets FOLL_PIN. These wrappers will:
* Start with "pin_user_pages" instead of "get_user_pages". That makes it easy to find and audit the call sites.
* Set FOLL_PIN
* For pages that are received via FOLL_PIN, those pages must be returned via put_user_page().
Thanks to Jan Kara and Vlastimil Babka for explaining the 4 cases in this documentation. (I've reworded it and expanded upon it.)
--- Documentation/core-api/index.rst | 1 + Documentation/core-api/pin_user_pages.rst | 233 ++++++++++++++++++++++ include/linux/mm.h | 63 ++++-- mm/gup.c | 153 ++++++++++++-- 4 files changed, 416 insertions(+), 34 deletions(-) create mode 100644 Documentation/core-api/pin_user_pages.rst
Documentation/core-api/index.rst +1 -0
@@ -31,6 +31,7 @@ Core utilities generic-radix-tree memory-allocation mm-api + pin_user_pages gfp_mask-from-fs-io timekeeping boot-time-mm
Documentation/core-api/pin_user_pages.rst new file +233 -0
233 lines not on this page · raw message
include/linux/mm.h +50 -13
105 lines not on this page · raw message
mm/gup.c +132 -21
197 lines not on this page · raw message
Signature · 6 lines
2.24.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
John Hubbard
reviewerPATCH v6 11/24 goldish_pipe: convert to pin_user_pages() and put_user_page()
In reply to John Hubbard1. Call the new global pin_user_pages_fast(), from pin_goldfish_pages().
2. As required by pin_user_pages(), release these pages via put_user_page(). In this case, do so via put_user_pages_dirty_lock().
That has the side effect of calling set_page_dirty_lock(), instead of set_page_dirty(). This is probably more accurate.
As Christoph Hellwig put it, "set_page_dirty() is only safe if we are dealing with a file backed page where we have reference on the inode it hangs off." [1]
Another side effect is that the release code is simplified because the page[] loop is now in gup.c instead of here, so just delete the local release_user_pages() entirely, and call put_user_pages_dirty_lock() directly, instead.
[1] https://lore.kernel.org/r/[email protected]
--- drivers/platform/goldfish/goldfish_pipe.c | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-)
drivers/platform/goldfish/goldfish_pipe.c +3 -14
@@ -274,7 +274,7 @@ static int pin_goldfish_pages(unsigned long first_page, *iter_last_page_size = last_page_size; } - ret =get_user_pages_fast (first_page, requested_pages, + ret =pin_user_pages_fast (first_page, requested_pages, !is_write ?FOLL_WRITE :0 , pages);if (ret <=0 )
@@ -285,18 +285,6 @@ static int pin_goldfish_pages(unsigned long first_page,return ret; } -static void release_user_pages (struct page **pages,int pages_count, -int is_write,s32 consumed_size) -{ -int i; - -for (i =0 ; i < pages_count; i++) { -if (!is_write && consumed_size >0 ) -set_page_dirty (pages[i]); -put_page (pages[i]); - } -} -/* Populate the call parameters, merging adjacent pages together */ static void populate_rw_params(struct page **pages, int pages_count,
@@ -372,7 +360,8 @@ static int transfer_max_buffers(struct goldfish_pipe *pipe, *consumed_size = pipe->command_buffer->rw_params.consumed_size; -release_user_pages (pipe->pages, pages_count, is_write, *consumed_size); +put_user_pages_dirty_lock (pipe->pages, pages_count, + !is_write && *consumed_size >0 );mutex_unlock (&pipe->lock);return 0 ;
Signature · 1 line
2.24.0
John Hubbard
reviewerPATCH v6 12/24 IB/{core,hw,umem}: set FOLL_PIN via pin_user_pages*(), fix up ODP
In reply to John HubbardConvert infiniband to use the new pin_user_pages*() calls.
Also, revert earlier changes to Infiniband ODP that had it using put_user_page(). ODP is "Case 3" in Documentation/core-api/pin_user_pages.rst, which is to say, normal get_user_pages() and put_page() is the API to use there.
The new pin_user_pages*() calls replace corresponding get_user_pages*() calls, and set the FOLL_PIN flag. The FOLL_PIN flag requires that the caller must return the pages via put_user_page*() calls, but infiniband was already doing that as part of an earlier commit.
--- drivers/infiniband/core/umem.c | 2 +- drivers/infiniband/core/umem_odp.c | 13 ++++++------- drivers/infiniband/hw/hfi1/user_pages.c | 2 +- drivers/infiniband/hw/mthca/mthca_memfree.c | 2 +- drivers/infiniband/hw/qib/qib_user_pages.c | 2 +- drivers/infiniband/hw/qib/qib_user_sdma.c | 2 +- drivers/infiniband/hw/usnic/usnic_uiom.c | 2 +- drivers/infiniband/sw/siw/siw_mem.c | 2 +- 8 files changed, 13 insertions(+), 14 deletions(-)
drivers/infiniband/core/umem.c +1 -1
@@ -271,7 +271,7 @@ struct ib_umem *ib_umem_get(struct ib_udata *udata, unsigned long addr, sg = umem->sg_head.sgl;while (npages) { - ret = get_user_pages_fast(cur_base, + ret = pin_user_pages_fast(cur_base, min_t(unsigned long, npages, PAGE_SIZE / sizeof(struct page *)),
drivers/infiniband/core/umem_odp.c +6 -7
@@ -495,9 +495,8 @@ EXPORT_SYMBOL(ib_umem_odp_release);* The function returns -EFAULT if the DMA mapping operation fails. It returns * -EAGAIN if a concurrent invalidation prevents us from updating the page. * -* The page is released via put_user_page even if the operation failed. For -* on-demand pinning, the page is released whenever it isn't stored in the -* umem. +* The page is released via put_page even if the operation failed. For on-demand +* pinning, the page is released whenever it isn't stored in the umem. */ static int ib_umem_odp_map_dma_single_page( struct ib_umem_odp *umem_odp,
@@ -542,7 +541,7 @@ static int ib_umem_odp_map_dma_single_page( } out: -put_user_page (page); +put_page (page);if (remove_existing_mapping) {ib_umem_notifier_start_account (umem_odp);
@@ -665,7 +664,7 @@ int ib_umem_odp_map_dma_pages(struct ib_umem_odp *umem_odp, u64 user_virt, ret = -EFAULT ;break ; } -put_user_page (local_page_list[j]); +put_page (local_page_list[j]);continue ; }
@@ -692,8 +691,8 @@ int ib_umem_odp_map_dma_pages(struct ib_umem_odp *umem_odp, u64 user_virt,* ib_umem_odp_map_dma_single_page(). */ if (npages - (j +1 ) >0 ) -put_user_pages (&local_page_list[j+1 ], - npages - (j +1 )); +release_pages (&local_page_list[j+1 ], + npages - (j +1 ));break ; } }
drivers/infiniband/hw/hfi1/user_pages.c +1 -1
@@ -106,7 +106,7 @@ int hfi1_acquire_user_pages(struct mm_struct *mm, unsigned long vaddr, size_t npint ret;unsigned int gup_flags =FOLL_LONGTERM | (writable ?FOLL_WRITE :0 ); - ret =get_user_pages_fast (vaddr, npages, gup_flags, pages); + ret =pin_user_pages_fast (vaddr, npages, gup_flags, pages);if (ret <0 )return ret;
drivers/infiniband/hw/mthca/mthca_memfree.c +1 -1
@@ -472,7 +472,7 @@ int mthca_map_user_db(struct mthca_dev *dev, struct mthca_uar *uar, goto out; } - ret =get_user_pages_fast (uaddr &PAGE_MASK ,1 , + ret =pin_user_pages_fast (uaddr &PAGE_MASK ,1 ,FOLL_WRITE |FOLL_LONGTERM , pages);if (ret <0 ) goto out;
drivers/infiniband/hw/qib/qib_user_pages.c +1 -1
@@ -108,7 +108,7 @@ int qib_get_user_pages(unsigned long start_page, size_t num_pages,down_read (¤t->mm->mmap_sem);for (got =0 ; got < num_pages; got += ret) { - ret =get_user_pages (start_page + got *PAGE_SIZE , + ret =pin_user_pages (start_page + got *PAGE_SIZE , num_pages - got,FOLL_LONGTERM |FOLL_WRITE |FOLL_FORCE , p + got,NULL );
drivers/infiniband/hw/qib/qib_user_sdma.c +1 -1
@@ -670,7 +670,7 @@ static int qib_user_sdma_pin_pages(const struct qib_devdata *dd,else j = npages; - ret =get_user_pages_fast (addr, j,FOLL_LONGTERM , pages); + ret =pin_user_pages_fast (addr, j,FOLL_LONGTERM , pages);if (ret != j) { i =0 ; j = ret;
drivers/infiniband/hw/usnic/usnic_uiom.c +1 -1
@@ -141,7 +141,7 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable, ret =0 ;while (npages) { - ret = get_user_pages(cur_base, + ret = pin_user_pages(cur_base, min_t(unsigned long, npages, PAGE_SIZE / sizeof(struct page *)), gup_flags | FOLL_LONGTERM,
drivers/infiniband/sw/siw/siw_mem.c +1 -1
@@ -426,7 +426,7 @@ struct siw_umem *siw_umem_get(u64 start, u64 len, bool writable)while (nents) {struct page **plist = &umem->page_chunk[i].plist[got]; - rv =get_user_pages (first_page_va, nents, + rv =pin_user_pages (first_page_va, nents, foll_flags |FOLL_LONGTERM , plist,NULL );if (rv <0 )
Signature · 6 lines
2.24.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
John Hubbard
reviewerPATCH v6 13/24 mm/process_vm_access: set FOLL_PIN via pin_user_pages_remote()
In reply to John HubbardConvert process_vm_access to use the new pin_user_pages_remote() call, which sets FOLL_PIN. Setting FOLL_PIN is now required for code that requires tracking of pinned pages.
Also, release the pages via put_user_page*().
Also, rename "pages" to "pinned_pages", as this makes for easier reading of process_vm_rw_single_vec().
--- mm/process_vm_access.c | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-)
mm/process_vm_access.c +15 -13
@@ -42,12 +42,11 @@ static int process_vm_rw_pages(struct page **pages,if (copy > len) copy = len; -if (vm_write) { +if (vm_write) copied =copy_page_from_iter (page, offset, copy, iter); -set_page_dirty_lock (page); - }else { +else copied =copy_page_to_iter (page, offset, copy, iter); - } + len -= copied;if (copied < copy &&iov_iter_count (iter))return -EFAULT ;
@@ -96,7 +95,7 @@ static int process_vm_rw_single_vec(unsigned long addr, flags |=FOLL_WRITE ;while (!rc && nr_pages &&iov_iter_count (iter)) { -int pages =min (nr_pages, max_pages_per_loop); +int pinned_pages =min (nr_pages, max_pages_per_loop);int locked =1 ;size_t bytes;
@@ -106,14 +105,15 @@ static int process_vm_rw_single_vec(unsigned long addr,* current/current->mm */ down_read (&mm->mmap_sem); - pages =get_user_pages_remote (task, mm, pa, pages, flags, - process_pages,NULL , &locked); + pinned_pages =pin_user_pages_remote (task, mm, pa, pinned_pages, + flags, process_pages, +NULL , &locked);if (locked)up_read (&mm->mmap_sem); -if (pages <=0 ) +if (pinned_pages <=0 )return -EFAULT ; - bytes = pages *PAGE_SIZE - start_offset; + bytes = pinned_pages *PAGE_SIZE - start_offset;if (bytes > len) bytes = len;
@@ -122,10 +122,12 @@ static int process_vm_rw_single_vec(unsigned long addr, vm_write); len -= bytes; start_offset =0 ; - nr_pages -= pages; - pa += pages *PAGE_SIZE ; -while (pages) -put_page (process_pages[--pages]); + nr_pages -= pinned_pages; + pa += pinned_pages *PAGE_SIZE ; + +/* If vm_write is set, the pages need to be made dirty: */ +put_user_pages_dirty_lock (process_pages, pinned_pages, + vm_write); }return rc;
Signature · 6 lines
2.24.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
John Hubbard
reviewerPATCH v6 14/24 drm/via: set FOLL_PIN via pin_user_pages_fast()
In reply to John HubbardConvert drm/via to use the new pin_user_pages_fast() call, which sets FOLL_PIN. Setting FOLL_PIN is now required for code that requires tracking of pinned pages, and therefore for any code that calls put_user_page().
In partial anticipation of this work, the drm/via driver was already calling put_user_page() instead of put_page(). Therefore, in order to convert from the get_user_pages()/put_page() model, to the pin_user_pages()/put_user_page() model, the only change required is to change get_user_pages() to pin_user_pages().
--- drivers/gpu/drm/via/via_dmablit.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/via/via_dmablit.c +1 -1
@@ -239,7 +239,7 @@ via_lock_all_dma_pages(drm_via_sg_info_t *vsg, drm_via_dmablit_t *xfer) vsg->pages =vzalloc (array_size (sizeof (struct page *), vsg->num_pages));if (NULL == vsg->pages)return -ENOMEM ; - ret =get_user_pages_fast ((unsigned long )xfer->mem_addr, + ret =pin_user_pages_fast ((unsigned long )xfer->mem_addr, vsg->num_pages, vsg->direction ==DMA_FROM_DEVICE ?FOLL_WRITE :0 , vsg->pages);
Signature · 6 lines
2.24.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
John Hubbard
reviewerPATCH v6 15/24 fs/io_uring: set FOLL_PIN via pin_user_pages()
In reply to John HubbardConvert fs/io_uring to use the new pin_user_pages() call, which sets FOLL_PIN. Setting FOLL_PIN is now required for code that requires tracking of pinned pages, and therefore for any code that calls put_user_page().
In partial anticipation of this work, the io_uring code was already calling put_user_page() instead of put_page(). Therefore, in order to convert from the get_user_pages()/put_page() model, to the pin_user_pages()/put_user_page() model, the only change required here is to change get_user_pages() to pin_user_pages().
--- fs/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/io_uring.c +1 -1
@@ -3449,7 +3449,7 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, void __user *arg, ret =0 ;down_read (¤t->mm->mmap_sem); - pret =get_user_pages (ubuf, nr_pages, + pret =pin_user_pages (ubuf, nr_pages,FOLL_WRITE |FOLL_LONGTERM , pages, vmas);if (pret == nr_pages) {
Signature · 6 lines
2.24.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Jens Axboe
maintainerRe: [PATCH v6 15/24] fs/io_uring: set FOLL_PIN via pin_user_pages()
Quoted 5 lines · John Hubbard
Convert
fs/io_uringto use the newpin_user_pages()call, which setsFOLL_PIN. SettingFOLL_PINis now required for code that requires tracking of pinned pages, and therefore for any code that callsput_user_page().
Quoted 8 lines
In partial anticipation of this work, the
io_uringcode was already callingput_user_page()instead ofput_page(). Therefore, in order to convert from the get_user_pages()/put_page() model, to the pin_user_pages()/put_user_page() model, the only change required here is to changeget_user_pages()topin_user_pages().Reviewed-by Jan Kara <[email protected]>Signed-off-by John Hubbard <[email protected]>
You dropped my reviewed-by now... Given the file, you'd probably want to keep that.
Signature · 1 line
Jens Axboe
John Hubbard
reviewerRe: [PATCH v6 15/24] fs/io_uring: set FOLL_PIN via pin_user_pages()
Quoted 12 lines · Jens Axboe
Quoted 11 lines · John Hubbard
Convert
fs/io_uringto use the newpin_user_pages()call, which setsFOLL_PIN. SettingFOLL_PINis now required for code that requires tracking of pinned pages, and therefore for any code that callsput_user_page().In partial anticipation of this work, the
io_uringcode was already callingput_user_page()instead ofput_page(). Therefore, in order to convert from the get_user_pages()/put_page() model, to the pin_user_pages()/put_user_page() model, the only change required here is to changeget_user_pages()topin_user_pages().
Quoted 5 lines
Quoted 2 lines
Reviewed-by Jan Kara <[email protected]>Signed-off-by John Hubbard <[email protected]>You dropped my reviewed-by now... Given the file, you'd probably want to keep that.
Hi Jens,
Yes, I was being too conservative I guess. I changed the patch somewhat and dropped the reviewed-by because of those changes...I'm adding it back for v7 based on this, thanks!
thanks,
Signature · 2 lines
John Hubbard NVIDIA
John Hubbard
reviewerPATCH v6 16/24 net/xdp: set FOLL_PIN via pin_user_pages()
In reply to John HubbardConvert net/xdp to use the new pin_longterm_pages() call, which sets FOLL_PIN. Setting FOLL_PIN is now required for code that requires tracking of pinned pages.
In partial anticipation of this work, the net/xdp code was already calling put_user_page() instead of put_page(). Therefore, in order to convert from the get_user_pages()/put_page() model, to the pin_user_pages()/put_user_page() model, the only change required here is to change get_user_pages() to pin_user_pages().
--- net/xdp/xdp_umem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
net/xdp/xdp_umem.c +1 -1
@@ -291,7 +291,7 @@ static int xdp_umem_pin_pages(struct xdp_umem *umem)return -ENOMEM ;down_read (¤t->mm->mmap_sem); - npgs =get_user_pages (umem->address, umem->npgs, + npgs =pin_user_pages (umem->address, umem->npgs, gup_flags |FOLL_LONGTERM , &umem->pgs[0 ],NULL );up_read (¤t->mm->mmap_sem);
Signature · 1 line
2.24.0
John Hubbard
reviewerPATCH v6 17/24 mm/gup: track FOLL_PIN pages
In reply to John HubbardAdd tracking of pages that were pinned via FOLL_PIN.
As mentioned in the FOLL_PIN documentation, callers who effectively set FOLL_PIN are required to ultimately free such pages via put_user_page(). The effect is similar to FOLL_GET, and may be thought of as "FOLL_GET for DIO and/or RDMA use".
Pages that have been pinned via FOLL_PIN are identifiable via a new function call:
bool page_dma_pinned(struct page *page);
What to do in response to encountering such a page, is left to later patchsets. There is discussion about this in [1], [2], and [3].
This also changes a BUG_ON(), to a WARN_ON(), in follow_page_mask().
[1] Some slow progress on get_user_pages() (Apr 2, 2019):
https://lwn.net/Articles/784574/ [2] DMA and get_user_pages() (LPC: Dec 12, 2018): https://lwn.net/Articles/774411/ [3] The trouble with get_user_pages() (Apr 30, 2018): https://lwn.net/Articles/753027/
--- Documentation/core-api/pin_user_pages.rst | 2 +- include/linux/mm.h | 86 +++++-- include/linux/mmzone.h | 2 + include/linux/page_ref.h | 10 + mm/gup.c | 290 ++++++++++++++++------ mm/huge_memory.c | 54 +++- mm/hugetlb.c | 39 ++- mm/vmstat.c | 2 + 8 files changed, 390 insertions(+), 95 deletions(-)
Documentation/core-api/pin_user_pages.rst +1 -1
@@ -53,7 +53,7 @@ Which flags are set by each wrapper For these pin_user_pages*() functions, FOLL_PIN is OR'd in with whatever gup flags the caller provides. The caller is required to pass in a non-null struct pages* array, and the function then pin pages by incrementing each by a special -value. For now, that value is +1, just like get_user_pages*().:: +value: GUP_PIN_COUNTING_BIAS.:: Function --------
include/linux/mm.h +71 -15
106 lines not on this page · raw message
include/linux/mmzone.h +2 -0
@@ -244,6 +244,8 @@ enum node_stat_item {NR_DIRTIED ,/* page dirtyings since bootup */ NR_WRITTEN ,/* page writings since bootup */ NR_KERNEL_MISC_RECLAIMABLE ,/* reclaimable non-slab kernel pages */ +NR_FOLL_PIN_REQUESTED ,/* via: pin_user_page(), gup flag: FOLL_PIN */ +NR_FOLL_PIN_RETURNED ,/* pages returned via put_user_page() */ NR_VM_NODE_STAT_ITEMS };
include/linux/page_ref.h +10 -0
@@ -102,6 +102,16 @@ static inline void page_ref_sub(struct page *page, int nr)__page_ref_mod (page, -nr); } +static inline int page_ref_sub_return (struct page *page,int nr) +{ +int ret =atomic_sub_return (nr, &page->_refcount); + +if (page_ref_tracepoint_active (__tracepoint_page_ref_mod)) +__page_ref_mod (page, -nr); + +return ret; +} +static inline void page_ref_inc (struct page *page) {atomic_inc (&page->_refcount);
mm/gup.c +218 -72
438 lines not on this page · raw message
mm/huge_memory.c +50 -4
99 lines not on this page · raw message
mm/hugetlb.c +36 -3
@@ -4462,7 +4462,22 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, same_page:if (pages) { pages[i] =mem_map_offset (page, pfn_offset); -get_page (pages[i]); + +if (flags &FOLL_GET ) +get_page (pages[i]); +else if (flags &FOLL_PIN ) { +/* +* try_pin_page() is not actually expected to +* fail here because we hold the ptl. +*/ +if (unlikely (!try_pin_page (pages[i]))) { +spin_unlock (ptl); + remainder =0 ; + err = -ENOMEM ; +WARN_ON_ONCE (1 ); +break ; + } + } }if (vmas)
@@ -5022,6 +5037,12 @@ follow_huge_pmd(struct mm_struct *mm, unsigned long address,struct page *page =NULL ;spinlock_t *ptl;pte_t pte; + +/* FOLL_GET and FOLL_PIN are mutually exclusive. */ +if (WARN_ON_ONCE ((flags & (FOLL_PIN |FOLL_GET )) == + (FOLL_PIN |FOLL_GET ))) +return NULL ; + retry: ptl =pmd_lockptr (mm, pmd);spin_lock (ptl);
@@ -5034,8 +5055,20 @@ follow_huge_pmd(struct mm_struct *mm, unsigned long address, pte =huge_ptep_get ((pte_t *)pmd);if (pte_present (pte)) { page =pmd_page (*pmd) + ((address & ~PMD_MASK ) >>PAGE_SHIFT ); +if (flags &FOLL_GET )get_page (page); +else if (flags &FOLL_PIN ) { +/* +* try_pin_page() is not actually expected to fail +* here because we hold the ptl. +*/ +if (unlikely (!try_pin_page (page))) { +WARN_ON_ONCE (1 ); + page =NULL ; + goto out; + } + } }else {if (is_hugetlb_entry_migration (pte)) {spin_unlock (ptl);
@@ -5056,7 +5089,7 @@ struct page * __weak follow_huge_pud(struct mm_struct *mm, unsigned long address, pud_t *pud, int flags) { - if (flags & FOLL_GET) + if (flags & (FOLL_GET | FOLL_PIN)) return NULL;return pte_page (*(pte_t *)pud) + ((address & ~PUD_MASK ) >>PAGE_SHIFT );
@@ -5065,7 +5098,7 @@ follow_huge_pud(struct mm_struct *mm, unsigned long address,struct page *__weak follow_huge_pgd (struct mm_struct *mm,unsigned long address,pgd_t *pgd,int flags) { -if (flags &FOLL_GET ) +if (flags & (FOLL_GET |FOLL_PIN ))return NULL ;return pte_page (*(pte_t *)pgd) + ((address & ~PGDIR_MASK ) >>PAGE_SHIFT );
mm/vmstat.c +2 -0
@@ -1167,6 +1167,8 @@ const char * const vmstat_text[] = {"nr_dirtied" ,"nr_written" ,"nr_kernel_misc_reclaimable" , +"nr_foll_pin_requested" , +"nr_foll_pin_returned" ,/* enum writeback_stat_item counters */ "nr_dirty_threshold",
Signature · 1 line
2.24.0
Jan Kara
reviewerRe: [PATCH v6 17/24] mm/gup: track FOLL_PIN pages
mm/gup.c static int __record_subpages(struct page *page, unsigned long addr, · line 2153
Quoted 6 lines · John Hubbard
mm/gup.c +2
-0@@ -2025,6 +2149,20 @@ static int __record_subpages(struct page *page, unsigned long addr,return nr; } +static bool __pin_compound_head (struct page *head,int refs,unsigned int flags) +{
I don't quite like the proliferation of names starting with __. I don't think there's a good reason for that, particularly in this case. Also 'pin' here is somewhat misleading as we already use term "pin" for the particular way of pinning the page. We could have grab_compound_head() or maybe nail_compound_head() :), but you're native speaker so you may come up with better word.
Quoted 5 lines
mm/gup.c +5
-0+if (flags &FOLL_PIN ) { +if (unlikely (!try_pin_compound_head (head, refs))) +return false; + }else { + head =try_get_compound_head (head, refs);
mm/gup.c static int __record_subpages(struct page *page, unsigned long addr, · line 2168
Quoted 10 lines
mm/gup.c +7
-0+if (!head) +return false; + } + +return true; +} +static void put_compound_head (struct page *page,int refs) {/* Do a get_page() first, in case refs == page->_refcount */
put_compound_head() needs similar treatment as undo_dev_pagemap(), doesn't it?
Quoted 7 lines
mm/huge_memory.c +2
-1@@ -968,7 +973,18 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr,if (!*pgmap)return ERR_PTR (-EFAULT ); page =pfn_to_page (pfn); -get_page (page); + +if (flags &FOLL_GET )
mm/huge_memory.c struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, · line 987
Quoted 10 lines
mm/huge_memory.c +10
-0+get_page (page); + else if (flags & FOLL_PIN) { +/* +* try_pin_page() is not actually expected to fail here because +* we hold the pmd lock so no one can unmap the pmd and free the +* page that it points to. +*/ +if (unlikely (!try_pin_page (page))) + page =ERR_PTR (-EFAULT ); + }
This pattern is rather common. So maybe I'd add a helper grab_page(page, flags) doing
if (flags & FOLL_GET) get_page(page); else if (flags & FOLL_PIN) return try_pin_page(page); return true;
Otherwise the patch looks good to me now.
Honza
Signature · 6 lines
Jan Kara <[email protected]> SUSE Labs, CR _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
John Hubbard
reviewerRe: [PATCH v6 17/24] mm/gup: track FOLL_PIN pages
Quoted 5 lines · Jan Kara
Quoted 4 lines · John Hubbard
mm/gup.c +0
-0@@ -2025,6 +2149,20 @@ static int __record_subpages(struct page *page, unsigned long addr,return nr; }
Quoted 9 lines
mm/gup.c static int __record_subpages(struct page *page, unsigned long addr, · line 2153
Quoted 2 lines
mm/gup.c +2
-0+static bool __pin_compound_head (struct page *head,int refs,unsigned int flags) +{I don't quite like the proliferation of names starting with __. I don't think there's a good reason for that, particularly in this case. Also 'pin' here is somewhat misleading as we already use term "pin" for the particular way of pinning the page. We could have
grab_compound_head()or maybenail_compound_head():), but you're native speaker so you may come up with better word.
Yes, it is ugly naming, I'll change these as follows:
__pin_compound_head() --> grab_compound_head()
__record_subpages() --> record_subpages()I loved the "nail_compound_head()" suggestion, it just seems very vivid, but in the end, I figured I'd better keep it relatively drab and colorless. :)
Quoted 11 lines
Quoted 11 lines
mm/gup.c +10
-0+if (flags &FOLL_PIN ) { +if (unlikely (!try_pin_compound_head (head, refs))) +return false; + }else { + head =try_get_compound_head (head, refs); +if (!head) +return false; + } + +return true;
Quoted 9 lines
mm/gup.c static int __record_subpages(struct page *page, unsigned long addr, · line 2168
Quoted 5 lines
mm/gup.c +2
-0+} +static void put_compound_head (struct page *page,int refs) {/* Do a get_page() first, in case refs == page->_refcount */
put_compound_head()needs similar treatment asundo_dev_pagemap(), doesn't it?
Yes, will fix that up.
Quoted 21 lines
Quoted 7 lines
mm/huge_memory.c +2
-1@@ -968,7 +973,18 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr,if (!*pgmap)return ERR_PTR (-EFAULT ); page =pfn_to_page (pfn); -get_page (page); + +if (flags &FOLL_GET )mm/huge_memory.c struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, · line 987
Quoted 10 lines
mm/huge_memory.c +10
-0+get_page (page); + else if (flags & FOLL_PIN) { +/* +* try_pin_page() is not actually expected to fail here because +* we hold the pmd lock so no one can unmap the pmd and free the +* page that it points to. +*/ +if (unlikely (!try_pin_page (page))) + page =ERR_PTR (-EFAULT ); + }This pattern is rather common. So maybe I'd add a helper
grab_page(page, flags)doing
Quoted 6 lines
if (flags & FOLL_GET) get_page(page); else if (flags & FOLL_PIN) return try_pin_page(page); return true;
OK.
Quoted 3 lines
Otherwise the patch looks good to me now.
Honza
Great! I thought I'd have a v7 out today, but fate decided to have me repair my test machine instead. So, soon. ha. :)
thanks,
Signature · 6 lines
John Hubbard NVIDIA _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
John Hubbard
reviewerPATCH v6 18/24 media/v4l2-core: pin_user_pages (FOLL_PIN) and put_user_page() conversion
In reply to John Hubbard1. Change v4l2 from get_user_pages() to pin_user_pages().
2. Because all FOLL_PIN-acquired pages must be released via put_user_page(), also convert the put_page() call over to put_user_pages_dirty_lock().
--- drivers/media/v4l2-core/videobuf-dma-sg.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)
drivers/media/v4l2-core/videobuf-dma-sg.c +4 -7
@@ -183,12 +183,12 @@ static int videobuf_dma_init_user_locked(struct videobuf_dmabuf *dma,dprintk (1 ,"init user [0x%lx+0x%lx => %d pages]\n" , data, size, dma->nr_pages); - err =get_user_pages (data &PAGE_MASK , dma->nr_pages, + err =pin_user_pages (data &PAGE_MASK , dma->nr_pages, flags |FOLL_LONGTERM , dma->pages,NULL );if (err != dma->nr_pages) { dma->nr_pages = (err >=0 ) ? err :0 ; -dprintk (1 ,"get_user_pages: err=%d [%d]\n" , err, +dprintk (1 ,"pin_user_pages: err=%d [%d]\n" , err, dma->nr_pages);return err <0 ? err : -EINVAL ; }
@@ -349,11 +349,8 @@ int videobuf_dma_free(struct videobuf_dmabuf *dma)BUG_ON (dma->sglen);if (dma->pages) { -for (i =0 ; i < dma->nr_pages; i++) { -if (dma->direction ==DMA_FROM_DEVICE ) -set_page_dirty_lock (dma->pages[i]); -put_page (dma->pages[i]); - } +put_user_pages_dirty_lock (dma->pages, dma->nr_pages, + dma->direction ==DMA_FROM_DEVICE );kfree (dma->pages); dma->pages =NULL ; }
Signature · 1 line
2.24.0
John Hubbard
reviewerPATCH v6 19/24 vfio, mm: pin_user_pages (FOLL_PIN) and put_user_page() conversion
In reply to John Hubbard1. Change vfio from get_user_pages_remote(), topin_user_pages_remote().
2. Because all FOLL_PIN-acquired pages must be released via put_user_page(), also convert the put_page() call over to put_user_pages_dirty_lock().
Note that this effectively changes the code's behavior in vfio_iommu_type1.c: put_pfn(): it now ultimately calls set_page_dirty_lock(), instead of set_page_dirty(). This is probably more accurate.
As Christoph Hellwig put it, "set_page_dirty() is only safe if we are dealing with a file backed page where we have reference on the inode it hangs off." [1]
[1] https://lore.kernel.org/r/[email protected]
--- drivers/vfio/vfio_iommu_type1.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
drivers/vfio/vfio_iommu_type1.c +3 -4
@@ -327,9 +327,8 @@ static int put_pfn(unsigned long pfn, int prot) {if (!is_invalid_reserved_pfn (pfn)) {struct page *page =pfn_to_page (pfn); -if (prot &IOMMU_WRITE ) -SetPageDirty (page); -put_page (page); + +put_user_pages_dirty_lock (&page,1 , prot &IOMMU_WRITE );return 1 ; }return 0 ;
@@ -347,7 +346,7 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr, flags |=FOLL_WRITE ;down_read (&mm->mmap_sem); - ret =get_user_pages_remote (NULL , mm, vaddr,1 , flags |FOLL_LONGTERM , + ret =pin_user_pages_remote (NULL , mm, vaddr,1 , flags |FOLL_LONGTERM , page,NULL ,NULL );if (ret ==1 ) { *pfn =page_to_pfn (page[0 ]);
Signature · 1 line
2.24.0
John Hubbard
reviewerPATCH v6 20/24 powerpc: book3s64: convert to pin_user_pages() and put_user_page()
In reply to John Hubbard1. Convert from get_user_pages() to pin_user_pages().
2. As required by pin_user_pages(), release these pages via put_user_page(). In this case, do so via put_user_pages_dirty_lock().
That has the side effect of calling set_page_dirty_lock(), instead of set_page_dirty(). This is probably more accurate.
As Christoph Hellwig put it, "set_page_dirty() is only safe if we are dealing with a file backed page where we have reference on the inode it hangs off." [1]
3. Release each page in mem->hpages[] (instead of mem->hpas[]), because that is the array that pin_longterm_pages() filled in. This is more accurate and should be a little safer from a maintenance point of view.
[1] https://lore.kernel.org/r/[email protected]
--- arch/powerpc/mm/book3s64/iommu_api.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)
arch/powerpc/mm/book3s64/iommu_api.c +5 -7
@@ -103,7 +103,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,for (entry =0 ; entry < entries; entry += chunk) {unsigned long n =min (entries - entry, chunk); - ret =get_user_pages (ua + (entry <<PAGE_SHIFT ), n, + ret =pin_user_pages (ua + (entry <<PAGE_SHIFT ), n,FOLL_WRITE |FOLL_LONGTERM , mem->hpages + entry,NULL );if (ret == n) {
@@ -167,9 +167,8 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua,return 0 ; free_exit: -/* free the reference taken */ -for (i =0 ; i < pinned; i++) -put_page (mem->hpages[i]); +/* free the references taken */ +put_user_pages (mem->hpages, pinned);vfree (mem->hpas);kfree (mem);
@@ -212,10 +211,9 @@ static void mm_iommu_unpin(struct mm_iommu_table_group_mem_t *mem)if (!page)continue ; -if (mem->hpas[i] &MM_IOMMU_TABLE_GROUP_PAGE_DIRTY ) -SetPageDirty (page); +put_user_pages_dirty_lock (&mem->hpages[i],1 , +MM_IOMMU_TABLE_GROUP_PAGE_DIRTY ); -put_page (page); mem->hpas[i] =0 ; } }
Signature · 6 lines
2.24.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
John Hubbard
reviewerPATCH v6 21/24 mm/gup_benchmark: use proper FOLL_WRITE flags instead of hard-coding "1"
In reply to John HubbardFix the gup benchmark flags to use the symbolic FOLL_WRITE, instead of a hard-coded "1" value.
Also, clean up the filtering of gup flags a little, by just doing it once before issuing any of the get_user_pages*() calls. This makes it harder to overlook, instead of having little "gup_flags & 1" phrases in the function calls.
--- mm/gup_benchmark.c | 9 ++++++--- tools/testing/selftests/vm/gup_benchmark.c | 6 +++++- 2 files changed, 11 insertions(+), 4 deletions(-)
mm/gup_benchmark.c +6 -3
@@ -48,18 +48,21 @@ static int __gup_benchmark_ioctl(unsigned int cmd, nr = (next - addr) /PAGE_SIZE ; } +/* Filter out most gup flags: only allow a tiny subset here: */ + gup->flags &=FOLL_WRITE ; +switch (cmd) {case GUP_FAST_BENCHMARK : - nr =get_user_pages_fast (addr, nr, gup->flags &1 , + nr =get_user_pages_fast (addr, nr, gup->flags, pages + i);break ;case GUP_LONGTERM_BENCHMARK : nr =get_user_pages (addr, nr, - (gup->flags &1 ) |FOLL_LONGTERM , + gup->flags |FOLL_LONGTERM , pages + i,NULL );break ;case GUP_BENCHMARK : - nr =get_user_pages (addr, nr, gup->flags &1 , pages + i, + nr =get_user_pages (addr, nr, gup->flags, pages + i,NULL );break ;default :
tools/testing/selftests/vm/gup_benchmark.c +5 -1
@@ -18,6 +18,9 @@#define GUP_LONGTERM_BENCHMARK _IOWR('g', 2, struct gup_benchmark)#define GUP_BENCHMARK _IOWR('g', 3, struct gup_benchmark) +/* Just the flags we need, copied from mm.h: */ +#define FOLL_WRITE 0x01/* check pte is writable */ +struct gup_benchmark {__u64 get_delta_usec;__u64 put_delta_usec;
@@ -85,7 +88,8 @@ int main(int argc, char **argv) } gup.nr_pages_per_call = nr_pages; - gup.flags = write; +if (write) + gup.flags |=FOLL_WRITE ; fd =open ("/sys/kernel/debug/gup_benchmark" ,O_RDWR );if (fd ==-1 )
Signature · 6 lines
2.24.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
John Hubbard
reviewerPATCH v6 22/24 mm/gup_benchmark: support pin_user_pages() and related calls
In reply to John HubbardUp until now, gup_benchmark supported testing of the following kernel functions:
* get_user_pages(): via the '-U' command line option
* get_user_pages_longterm(): via the '-L' command line option
* get_user_pages_fast(): as the default (no options required)
Add test coverage for the new corresponding pin_*() functions:
* pin_user_pages(): via the '-c' command line option
* pin_user_pages_fast(): via the '-b' command line option
Also, add an option for clarity: '-u' for what is now (still) the default choice: get_user_pages_fast().
Also, for the commands that set FOLL_PIN, verify that the pages really are dma-pinned, via the new is_dma_pinned() routine. Those commands are:
PIN_FAST_BENCHMARK : calls pin_user_pages_fast()
PIN_BENCHMARK : calls pin_user_pages()
In between the calls to pin_*() and put_user_pages(), check each page: if page_dma_pinned() returns false, then WARN and return.
Do this outside of the benchmark timestamps, so that it doesn't affect reported times.
--- mm/gup_benchmark.c | 65 ++++++++++++++++++++-- tools/testing/selftests/vm/gup_benchmark.c | 15 ++++- 2 files changed, 74 insertions(+), 6 deletions(-)
mm/gup_benchmark.c +60 -5
@@ -8,6 +8,8 @@#define GUP_FAST_BENCHMARK _IOWR('g', 1, struct gup_benchmark)#define GUP_LONGTERM_BENCHMARK _IOWR('g', 2, struct gup_benchmark)#define GUP_BENCHMARK _IOWR('g', 3, struct gup_benchmark) +#define PIN_FAST_BENCHMARK _IOWR('g', 4, struct gup_benchmark) +#define PIN_BENCHMARK _IOWR('g', 5, struct gup_benchmark)struct gup_benchmark {__u64 get_delta_usec;
@@ -19,6 +21,42 @@ struct gup_benchmark {__u64 expansion[10 ];/* For future use */ }; +static void put_back_pages (int cmd,struct page **pages,unsigned long nr_pages) +{ +int i; + +switch (cmd) { +case GUP_FAST_BENCHMARK : +case GUP_LONGTERM_BENCHMARK : +case GUP_BENCHMARK : +for (i =0 ; i < nr_pages; i++) +put_page (pages[i]); +break ; + +case PIN_FAST_BENCHMARK : +case PIN_BENCHMARK : +put_user_pages (pages, nr_pages); +break ; + } +} + +static void verify_dma_pinned (int cmd,struct page **pages, +unsigned long nr_pages) +{ +int i; + +switch (cmd) { +case PIN_FAST_BENCHMARK : +case PIN_BENCHMARK : +for (i =0 ; i < nr_pages; i++) { +if (WARN (!page_dma_pinned (pages[i]), +"pages[%d] is NOT dma-pinned\n" , i)) +break ; + } +break ; + } +} +static int __gup_benchmark_ioctl (unsigned int cmd,struct gup_benchmark *gup) {
@@ -65,6 +103,14 @@ static int __gup_benchmark_ioctl(unsigned int cmd, nr =get_user_pages (addr, nr, gup->flags, pages + i,NULL );break ; +case PIN_FAST_BENCHMARK : + nr =pin_user_pages_fast (addr, nr, gup->flags, + pages + i); +break ; +case PIN_BENCHMARK : + nr =pin_user_pages (addr, nr, gup->flags, pages + i, +NULL ); +break ;default :return -1 ; }
@@ -75,15 +121,22 @@ static int __gup_benchmark_ioctl(unsigned int cmd, } end_time =ktime_get (); +/* Shifting the meaning of nr_pages: now it is actual number pinned: */ + nr_pages = i; + gup->get_delta_usec =ktime_us_delta (end_time, start_time); gup->size = addr - gup->addr; +/* +* Take an un-benchmark-timed moment to verify DMA pinned +* state: print a warning if any non-dma-pinned pages are found: +*/ +verify_dma_pinned (cmd, pages, nr_pages); + start_time =ktime_get (); -for (i =0 ; i < nr_pages; i++) { -if (!pages[i]) -break ; -put_page (pages[i]); - } + +put_back_pages (cmd, pages, nr_pages); + end_time =ktime_get (); gup->put_delta_usec =ktime_us_delta (end_time, start_time);
@@ -101,6 +154,8 @@ static long gup_benchmark_ioctl(struct file *filep, unsigned int cmd,case GUP_FAST_BENCHMARK :case GUP_LONGTERM_BENCHMARK :case GUP_BENCHMARK : +case PIN_FAST_BENCHMARK : +case PIN_BENCHMARK :break ;default :return -EINVAL ;
tools/testing/selftests/vm/gup_benchmark.c +14 -1
@@ -18,6 +18,10 @@#define GUP_LONGTERM_BENCHMARK _IOWR('g', 2, struct gup_benchmark)#define GUP_BENCHMARK _IOWR('g', 3, struct gup_benchmark) +/* Similar to above, but use FOLL_PIN instead of FOLL_GET. */ +#define PIN_FAST_BENCHMARK _IOWR('g', 4, struct gup_benchmark) +#define PIN_BENCHMARK _IOWR('g', 5, struct gup_benchmark) +/* Just the flags we need, copied from mm.h: */ #define FOLL_WRITE 0x01/* check pte is writable */
@@ -40,8 +44,14 @@ int main(int argc, char **argv)char *file ="/dev/zero" ;char *p; -while ((opt =getopt (argc, argv,"m:r:n:f:tTLUwSH" )) !=-1 ) { +while ((opt =getopt (argc, argv,"m:r:n:f:abtTLUuwSH" )) !=-1 ) {switch (opt) { +case 'a' : + cmd =PIN_FAST_BENCHMARK ; +break ; +case 'b' : + cmd =PIN_BENCHMARK ; +break ;case 'm' : size =atoi (optarg) *MB ;break ;
@@ -63,6 +73,9 @@ int main(int argc, char **argv)case 'U' : cmd =GUP_BENCHMARK ;break ; +case 'u' : + cmd =GUP_FAST_BENCHMARK ; +break ;case 'w' : write =1 ;break ;
Signature · 1 line
2.24.0
John Hubbard
reviewerPATCH v6 23/24 selftests/vm: run_vmtests: invoke gup_benchmark with basic FOLL_PIN coverage
In reply to John HubbardIt's good to have basic unit test coverage of the new FOLL_PIN behavior. Fortunately, the gup_benchmark unit test is extremely fast (a few milliseconds), so adding it the the run_vmtests suite is going to cause no noticeable change in running time.
So, add two new invocations to run_vmtests:
1) Run gup_benchmark with normal get_user_pages().
2) Run gup_benchmark with pin_user_pages(). This is much like the first call, except that it sets FOLL_PIN.
Running these two in quick succession also provide a visual comparison of the running times, which is convenient.
The new invocations are fairly early in the run_vmtests script, because with test suites, it's usually preferable to put the shorter, faster tests first, all other things being equal.
--- tools/testing/selftests/vm/run_vmtests | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
tools/testing/selftests/vm/run_vmtests +22 -0
@@ -104,6 +104,28 @@ echo "NOTE: The above hugetlb tests provide minimal coverage. Use" echo " https://github.com/libhugetlbfs/libhugetlbfs.git for" echo " hugetlb regression testing." +echo "--------------------------------------------" +echo "running 'gup_benchmark -U' (normal/slow gup)" +echo "--------------------------------------------" +./gup_benchmark -U +if [ $? -ne 0 ]; then + echo "[FAIL]" + exitcode=1 +else + echo "[PASS]" +fi + +echo "------------------------------------------" +echo "running gup_benchmark -b (pin_user_pages)" +echo "------------------------------------------" +./gup_benchmark -b +if [ $? -ne 0 ]; then + echo "[FAIL]" + exitcode=1 +else + echo "[PASS]" +fi + echo "-------------------" echo "running userfaultfd" echo "-------------------"
Signature · 1 line
2.24.0
John Hubbard
reviewerPATCH v6 24/24 mm, tree-wide: rename put_user_page*() to unpin_user_page*()
In reply to John HubbardIn order to provide a clearer, more symmetric API for pinning and unpinning DMA pages. This way, pin_user_pages*() calls match up with unpin_user_pages*() calls, and the API is a lot closer to being self-explanatory.
--- Documentation/core-api/pin_user_pages.rst | 2 +- arch/powerpc/mm/book3s64/iommu_api.c | 6 +-- drivers/gpu/drm/via/via_dmablit.c | 4 +- drivers/infiniband/core/umem.c | 2 +- drivers/infiniband/hw/hfi1/user_pages.c | 2 +- drivers/infiniband/hw/mthca/mthca_memfree.c | 6 +-- drivers/infiniband/hw/qib/qib_user_pages.c | 2 +- drivers/infiniband/hw/qib/qib_user_sdma.c | 6 +-- drivers/infiniband/hw/usnic/usnic_uiom.c | 2 +- drivers/infiniband/sw/siw/siw_mem.c | 2 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 4 +- drivers/platform/goldfish/goldfish_pipe.c | 4 +- drivers/vfio/vfio_iommu_type1.c | 2 +- fs/io_uring.c | 4 +- include/linux/mm.h | 30 +++++++------- include/linux/mmzone.h | 2 +- mm/gup.c | 46 ++++++++++----------- mm/gup_benchmark.c | 2 +- mm/process_vm_access.c | 4 +- net/xdp/xdp_umem.c | 2 +- 20 files changed, 67 insertions(+), 67 deletions(-)
Documentation/core-api/pin_user_pages.rst +1 -1
@@ -220,7 +220,7 @@ since the system was booted, via two new /proc/vmstat entries: :: /proc/vmstat/nr_foll_pin_requested Those are both going to show zero, unless CONFIG_DEBUG_VM is set. This is -because there is a noticeable performance drop in put_user_page(), when they +because there is a noticeable performance drop in unpin_user_page(), when they are activated. References
arch/powerpc/mm/book3s64/iommu_api.c +3 -3
@@ -168,7 +168,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua, free_exit:/* free the references taken */ -put_user_pages (mem->hpages, pinned); +unpin_user_pages (mem->hpages, pinned);vfree (mem->hpas);kfree (mem);
@@ -211,8 +211,8 @@ static void mm_iommu_unpin(struct mm_iommu_table_group_mem_t *mem)if (!page)continue ; -put_user_pages_dirty_lock (&mem->hpages[i],1 , -MM_IOMMU_TABLE_GROUP_PAGE_DIRTY ); +unpin_user_pages_dirty_lock (&mem->hpages[i],1 , +MM_IOMMU_TABLE_GROUP_PAGE_DIRTY ); mem->hpas[i] =0 ; }
drivers/gpu/drm/via/via_dmablit.c +2 -2
@@ -188,8 +188,8 @@ via_free_sg_info(struct pci_dev *pdev, drm_via_sg_info_t *vsg)kfree (vsg->desc_pages);/* fall through */ case dr_via_pages_locked: -put_user_pages_dirty_lock (vsg->pages, vsg->num_pages, - (vsg->direction ==DMA_FROM_DEVICE )); +unpin_user_pages_dirty_lock (vsg->pages, vsg->num_pages, + (vsg->direction ==DMA_FROM_DEVICE ));/* fall through */ case dr_via_pages_alloc:vfree (vsg->pages);
drivers/infiniband/core/umem.c +1 -1
@@ -54,7 +54,7 @@ static void __ib_umem_release(struct ib_device *dev, struct ib_umem *umem, int dfor_each_sg_page (umem->sg_head.sgl, &sg_iter, umem->sg_nents,0 ) { page =sg_page_iter_page (&sg_iter); -put_user_pages_dirty_lock (&page,1 , umem->writable && dirty); +unpin_user_pages_dirty_lock (&page,1 , umem->writable && dirty); }sg_free_table (&umem->sg_head);
drivers/infiniband/hw/hfi1/user_pages.c +1 -1
@@ -118,7 +118,7 @@ int hfi1_acquire_user_pages(struct mm_struct *mm, unsigned long vaddr, size_t npvoid hfi1_release_user_pages (struct mm_struct *mm,struct page **p,size_t npages,bool dirty) { -put_user_pages_dirty_lock (p, npages, dirty); +unpin_user_pages_dirty_lock (p, npages, dirty);if (mm) {/* during close after signal, mm can be NULL */ atomic64_sub (npages, &mm->pinned_vm);
drivers/infiniband/hw/mthca/mthca_memfree.c +3 -3
@@ -482,7 +482,7 @@ int mthca_map_user_db(struct mthca_dev *dev, struct mthca_uar *uar, ret =pci_map_sg (dev->pdev, &db_tab->page[i].mem,1 ,PCI_DMA_TODEVICE );if (ret <0 ) { -put_user_page (pages[0 ]); +unpin_user_page (pages[0 ]); goto out; }
@@ -490,7 +490,7 @@ int mthca_map_user_db(struct mthca_dev *dev, struct mthca_uar *uar, mthca_uarc_virt(dev, uar, i));if (ret) {pci_unmap_sg (dev->pdev, &db_tab->page[i].mem,1 ,PCI_DMA_TODEVICE ); -put_user_page (sg_page (&db_tab->page[i].mem)); +unpin_user_page (sg_page (&db_tab->page[i].mem)); goto out; }
@@ -556,7 +556,7 @@ void mthca_cleanup_user_db_tab(struct mthca_dev *dev, struct mthca_uar *uar,if (db_tab->page[i].uvirt) {mthca_UNMAP_ICM (dev,mthca_uarc_virt (dev, uar, i),1 );pci_unmap_sg (dev->pdev, &db_tab->page[i].mem,1 ,PCI_DMA_TODEVICE ); -put_user_page (sg_page (&db_tab->page[i].mem)); +unpin_user_page (sg_page (&db_tab->page[i].mem)); } }
drivers/infiniband/hw/qib/qib_user_pages.c +1 -1
@@ -40,7 +40,7 @@static void __qib_release_user_pages (struct page **p,size_t num_pages,int dirty) { -put_user_pages_dirty_lock (p, num_pages, dirty); +unpin_user_pages_dirty_lock (p, num_pages, dirty); }/**
drivers/infiniband/hw/qib/qib_user_sdma.c +3 -3
@@ -317,7 +317,7 @@ static int qib_user_sdma_page_to_frags(const struct qib_devdata *dd,* the caller can ignore this page. */ if (put) { -put_user_page (page); +unpin_user_page (page); }else {/* coalesce case */ kunmap (page);
@@ -631,7 +631,7 @@ static void qib_user_sdma_free_pkt_frag(struct device *dev,kunmap (pkt->addr[i].page);if (pkt->addr[i].put_page) -put_user_page (pkt->addr[i].page); +unpin_user_page (pkt->addr[i].page);else __free_page (pkt->addr[i].page); }else if (pkt->addr[i].kvaddr) {
@@ -706,7 +706,7 @@ static int qib_user_sdma_pin_pages(const struct qib_devdata *dd,/* if error, return all pages not managed by pkt */ free_pages:while (i < j) -put_user_page (pages[i++]); +unpin_user_page (pages[i++]); done:return ret;
drivers/infiniband/hw/usnic/usnic_uiom.c +1 -1
@@ -75,7 +75,7 @@ static void usnic_uiom_put_pages(struct list_head *chunk_list, int dirty)for_each_sg (chunk->page_list, sg, chunk->nents, i) { page =sg_page (sg); pa =sg_phys (sg); -put_user_pages_dirty_lock (&page,1 , dirty); +unpin_user_pages_dirty_lock (&page,1 , dirty);usnic_dbg ("pa: %pa\n" , &pa); }kfree (chunk);
drivers/infiniband/sw/siw/siw_mem.c +1 -1
@@ -63,7 +63,7 @@ struct siw_mem *siw_mem_id2obj(struct siw_device *sdev, int stag_index)static void siw_free_plist (struct siw_page_chunk *chunk,int num_pages,bool dirty) { -put_user_pages_dirty_lock (chunk->plist, num_pages, dirty); +unpin_user_pages_dirty_lock (chunk->plist, num_pages, dirty); }void siw_umem_release (struct siw_umem *umem,bool dirty)
drivers/media/v4l2-core/videobuf-dma-sg.c +2 -2
@@ -349,8 +349,8 @@ int videobuf_dma_free(struct videobuf_dmabuf *dma)BUG_ON (dma->sglen);if (dma->pages) { -put_user_pages_dirty_lock (dma->pages, dma->nr_pages, - dma->direction ==DMA_FROM_DEVICE ); +unpin_user_pages_dirty_lock (dma->pages, dma->nr_pages, + dma->direction ==DMA_FROM_DEVICE );kfree (dma->pages); dma->pages =NULL ; }
drivers/platform/goldfish/goldfish_pipe.c +2 -2
@@ -360,8 +360,8 @@ static int transfer_max_buffers(struct goldfish_pipe *pipe, *consumed_size = pipe->command_buffer->rw_params.consumed_size; -put_user_pages_dirty_lock (pipe->pages, pages_count, - !is_write && *consumed_size >0 ); +unpin_user_pages_dirty_lock (pipe->pages, pages_count, + !is_write && *consumed_size >0 );mutex_unlock (&pipe->lock);return 0 ;
drivers/vfio/vfio_iommu_type1.c +1 -1
@@ -328,7 +328,7 @@ static int put_pfn(unsigned long pfn, int prot)if (!is_invalid_reserved_pfn (pfn)) {struct page *page =pfn_to_page (pfn); -put_user_pages_dirty_lock (&page,1 , prot &IOMMU_WRITE ); +unpin_user_pages_dirty_lock (&page,1 , prot &IOMMU_WRITE );return 1 ; }return 0 ;
fs/io_uring.c +2 -2
@@ -3328,7 +3328,7 @@ static int io_sqe_buffer_unregister(struct io_ring_ctx *ctx)struct io_mapped_ubuf *imu = &ctx->user_bufs[i];for (j =0 ; j < imu->nr_bvecs; j++) -put_user_page (imu->bvec[j].bv_page); +unpin_user_page (imu->bvec[j].bv_page);if (ctx->account_mem)io_unaccount_mem (ctx->user, imu->nr_bvecs);
@@ -3473,7 +3473,7 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, void __user *arg,* release any pages we did get */ if (pret >0 ) -put_user_pages (pages, pret); +unpin_user_pages (pages, pret);if (ctx->account_mem)io_unaccount_mem (ctx->user, nr_pages);kvfree (imu->bvec);
include/linux/mm.h +15 -15
@@ -1093,18 +1093,18 @@ static inline void put_page(struct page *page)* made against the page. ("gup-pinned" is another term for the latter). * * With this scheme, pin_user_pages() becomes special: such pages are marked as -* distinct from normal pages. As such, the put_user_page() call (and its +* distinct from normal pages. As such, the unpin_user_page() call (and its * variants) must be used in order to release gup-pinned pages. * * Choice of value: * * By making GUP_PIN_COUNTING_BIAS a power of two, debugging of page reference -* counts with respect to pin_user_pages() and put_user_page() becomes simpler, -* due to the fact that adding an even power of two to the page refcount has the -* effect of using only the upper N bits, for the code that counts up using the -* bias value. This means that the lower bits are left for the exclusive use of -* the original code that increments and decrements by one (or at least, by much -* smaller values than the bias value). +* counts with respect to pin_user_pages() and unpin_user_page() becomes +* simpler, due to the fact that adding an even power of two to the page +* refcount has the effect of using only the upper N bits, for the code that +* counts up using the bias value. This means that the lower bits are left for +* the exclusive use of the original code that increments and decrements by one +* (or at least, by much smaller values than the bias value). * * Of course, once the lower bits overflow into the upper bits (and this is * OK, because subtraction recovers the original values), then visual inspection
@@ -1119,10 +1119,10 @@ static inline void put_page(struct page *page)*/ #define GUP_PIN_COUNTING_BIAS (1UL << 10) -void put_user_page (struct page *page); -void put_user_pages_dirty_lock (struct page **pages,unsigned long npages, -bool make_dirty); -void put_user_pages (struct page **pages,unsigned long npages); +void unpin_user_page (struct page *page); +void unpin_user_pages_dirty_lock (struct page **pages,unsigned long npages, +bool make_dirty); +void unpin_user_pages (struct page **pages,unsigned long npages);/** * page_dma_pinned() - report if a page is pinned for DMA.
@@ -2673,7 +2673,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,#define FOLL_ANON 0x8000/* don't do file mappings */ #define FOLL_LONGTERM 0x10000/* mapping lifetime is indefinite: see below */ #define FOLL_SPLIT_PMD 0x20000/* split huge pmd before returning */ -#define FOLL_PIN 0x40000/* pages must be released via put_user_page() */ +#define FOLL_PIN 0x40000/* pages must be released via unpin_user_page */ /* * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each
@@ -2708,7 +2708,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,* Direct IO). This lets the filesystem know that some non-file-system entity is * potentially changing the pages' data. In contrast to FOLL_GET (whose pages * are released via put_page()), FOLL_PIN pages must be released, ultimately, by -* a call to put_user_page(). +* a call to unpin_user_page(). * * FOLL_PIN is similar to FOLL_GET: both of these pin pages. They use different * and separate refcounting mechanisms, however, and that means that each has
@@ -2716,7 +2716,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,* * FOLL_GET: get_user_pages*() to acquire, and put_page() to release. * -* FOLL_PIN: pin_user_pages*() to acquire, and put_user_pages to release. +* FOLL_PIN: pin_user_pages*() to acquire, and unpin_user_pages to release. * * FOLL_PIN and FOLL_GET are mutually exclusive for a given function call. * (The underlying pages may experience both FOLL_GET-based and FOLL_PIN-based
@@ -2726,7 +2726,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,* FOLL_PIN should be set internally by the pin_user_pages*() APIs, never * directly by the caller. That's in order to help avoid mismatches when * releasing pages: get_user_pages*() pages must be released via put_page(), -* while pin_user_pages*() pages must be released via put_user_page(). +* while pin_user_pages*() pages must be released via unpin_user_page(). * * Please see Documentation/vm/pin_user_pages.rst for more information. */
include/linux/mmzone.h +1 -1
@@ -245,7 +245,7 @@ enum node_stat_item {NR_WRITTEN ,/* page writings since bootup */ NR_KERNEL_MISC_RECLAIMABLE ,/* reclaimable non-slab kernel pages */ NR_FOLL_PIN_REQUESTED ,/* via: pin_user_page(), gup flag: FOLL_PIN */ -NR_FOLL_PIN_RETURNED ,/* pages returned via put_user_page() */ +NR_FOLL_PIN_RETURNED ,/* pages returned via unpin_user_page() */ NR_VM_NODE_STAT_ITEMS };
mm/gup.c +23 -23
111 lines not on this page · raw message
mm/gup_benchmark.c +1 -1
@@ -35,7 +35,7 @@ static void put_back_pages(int cmd, struct page **pages, unsigned long nr_pages)case PIN_FAST_BENCHMARK :case PIN_BENCHMARK : -put_user_pages (pages, nr_pages); +unpin_user_pages (pages, nr_pages);break ; } }
mm/process_vm_access.c +2 -2
@@ -126,8 +126,8 @@ static int process_vm_rw_single_vec(unsigned long addr, pa += pinned_pages *PAGE_SIZE ;/* If vm_write is set, the pages need to be made dirty: */ -put_user_pages_dirty_lock (process_pages, pinned_pages, - vm_write); +unpin_user_pages_dirty_lock (process_pages, pinned_pages, + vm_write); }return rc;
net/xdp/xdp_umem.c +1 -1
@@ -212,7 +212,7 @@ static int xdp_umem_map_pages(struct xdp_umem *umem)static void xdp_umem_unpin_pages (struct xdp_umem *umem) { -put_user_pages_dirty_lock (umem->pgs, umem->npgs, true); +unpin_user_pages_dirty_lock (umem->pgs, umem->npgs, true);kfree (umem->pgs); umem->pgs =NULL ;
Signature · 1 line
2.24.0
Re: [PATCH v6 24/24] mm, tree-wide: rename put_user_page*() to unpin_user_page*()
Quoted 6 lines · John Hubbard
In order to provide a clearer, more symmetric API for pinning and unpinning DMA pages. This way, pin_user_pages*() calls match up with unpin_user_pages*() calls, and the API is a lot closer to being self-explanatory.
Signed-off-by John Hubbard <[email protected]>
Looks good to me. You can add:
Honza
Quoted 527 lines
--- Documentation/core-api/pin_user_pages.rst | 2 +- arch/powerpc/mm/book3s64/iommu_api.c | 6 +-- drivers/gpu/drm/via/via_dmablit.c | 4 +- drivers/infiniband/core/umem.c | 2 +- drivers/infiniband/hw/hfi1/user_pages.c | 2 +- drivers/infiniband/hw/mthca/mthca_memfree.c | 6 +-- drivers/infiniband/hw/qib/qib_user_pages.c | 2 +- drivers/infiniband/hw/qib/qib_user_sdma.c | 6 +-- drivers/infiniband/hw/usnic/usnic_uiom.c | 2 +- drivers/infiniband/sw/siw/siw_mem.c | 2 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 4 +- drivers/platform/goldfish/goldfish_pipe.c | 4 +- drivers/vfio/vfio_iommu_type1.c | 2 +- fs/io_uring.c | 4 +- include/linux/mm.h | 30 +++++++------- include/linux/mmzone.h | 2 +- mm/gup.c | 46 ++++++++++----------- mm/gup_benchmark.c | 2 +- mm/process_vm_access.c | 4 +- net/xdp/xdp_umem.c | 2 +- 20 files changed, 67 insertions(+), 67 deletions(-)Documentation/core-api/pin_user_pages.rst +1
-1@@ -220,7 +220,7 @@ since the system was booted, via two new /proc/vmstat entries: :: /proc/vmstat/nr_foll_pin_requested Those are both going to show zero, unless CONFIG_DEBUG_VM is set. This is -because there is a noticeable performance drop in put_user_page(), when they +because there is a noticeable performance drop in unpin_user_page(), when they are activated. Referencesarch/powerpc/mm/book3s64/iommu_api.c +3
-3@@ -168,7 +168,7 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua, free_exit:/* free the references taken */ -put_user_pages (mem->hpages, pinned); +unpin_user_pages (mem->hpages, pinned);vfree (mem->hpas);kfree (mem);@@ -211,8 +211,8 @@ static void mm_iommu_unpin(struct mm_iommu_table_group_mem_t *mem)if (!page)continue ; -put_user_pages_dirty_lock (&mem->hpages[i],1 , -MM_IOMMU_TABLE_GROUP_PAGE_DIRTY ); +unpin_user_pages_dirty_lock (&mem->hpages[i],1 , +MM_IOMMU_TABLE_GROUP_PAGE_DIRTY ); mem->hpas[i] =0 ; }drivers/gpu/drm/via/via_dmablit.c +2
-2@@ -188,8 +188,8 @@ via_free_sg_info(struct pci_dev *pdev, drm_via_sg_info_t *vsg)kfree (vsg->desc_pages);/* fall through */ case dr_via_pages_locked: -put_user_pages_dirty_lock (vsg->pages, vsg->num_pages, - (vsg->direction ==DMA_FROM_DEVICE )); +unpin_user_pages_dirty_lock (vsg->pages, vsg->num_pages, + (vsg->direction ==DMA_FROM_DEVICE ));/* fall through */ case dr_via_pages_alloc:vfree (vsg->pages);drivers/infiniband/core/umem.c +1
-1@@ -54,7 +54,7 @@ static void __ib_umem_release(struct ib_device *dev, struct ib_umem *umem, int dfor_each_sg_page (umem->sg_head.sgl, &sg_iter, umem->sg_nents,0 ) { page =sg_page_iter_page (&sg_iter); -put_user_pages_dirty_lock (&page,1 , umem->writable && dirty); +unpin_user_pages_dirty_lock (&page,1 , umem->writable && dirty); }sg_free_table (&umem->sg_head);drivers/infiniband/hw/hfi1/user_pages.c +1
-1@@ -118,7 +118,7 @@ int hfi1_acquire_user_pages(struct mm_struct *mm, unsigned long vaddr, size_t npvoid hfi1_release_user_pages (struct mm_struct *mm,struct page **p,size_t npages,bool dirty) { -put_user_pages_dirty_lock (p, npages, dirty); +unpin_user_pages_dirty_lock (p, npages, dirty);if (mm) {/* during close after signal, mm can be NULL */ atomic64_sub (npages, &mm->pinned_vm);drivers/infiniband/hw/mthca/mthca_memfree.c +3
-3@@ -482,7 +482,7 @@ int mthca_map_user_db(struct mthca_dev *dev, struct mthca_uar *uar, ret =pci_map_sg (dev->pdev, &db_tab->page[i].mem,1 ,PCI_DMA_TODEVICE );if (ret <0 ) { -put_user_page (pages[0 ]); +unpin_user_page (pages[0 ]); goto out; }@@ -490,7 +490,7 @@ int mthca_map_user_db(struct mthca_dev *dev, struct mthca_uar *uar, mthca_uarc_virt(dev, uar, i));if (ret) {pci_unmap_sg (dev->pdev, &db_tab->page[i].mem,1 ,PCI_DMA_TODEVICE ); -put_user_page (sg_page (&db_tab->page[i].mem)); +unpin_user_page (sg_page (&db_tab->page[i].mem)); goto out; }@@ -556,7 +556,7 @@ void mthca_cleanup_user_db_tab(struct mthca_dev *dev, struct mthca_uar *uar,if (db_tab->page[i].uvirt) {mthca_UNMAP_ICM (dev,mthca_uarc_virt (dev, uar, i),1 );pci_unmap_sg (dev->pdev, &db_tab->page[i].mem,1 ,PCI_DMA_TODEVICE ); -put_user_page (sg_page (&db_tab->page[i].mem)); +unpin_user_page (sg_page (&db_tab->page[i].mem)); } }drivers/infiniband/hw/qib/qib_user_pages.c +1
-1@@ -40,7 +40,7 @@static void __qib_release_user_pages (struct page **p,size_t num_pages,int dirty) { -put_user_pages_dirty_lock (p, num_pages, dirty); +unpin_user_pages_dirty_lock (p, num_pages, dirty); }/** drivers/infiniband/hw/qib/qib_user_sdma.c +3
-3@@ -317,7 +317,7 @@ static int qib_user_sdma_page_to_frags(const struct qib_devdata *dd,* the caller can ignore this page. */ if (put) { -put_user_page (page); +unpin_user_page (page); }else {/* coalesce case */ kunmap (page);@@ -631,7 +631,7 @@ static void qib_user_sdma_free_pkt_frag(struct device *dev,kunmap (pkt->addr[i].page);if (pkt->addr[i].put_page) -put_user_page (pkt->addr[i].page); +unpin_user_page (pkt->addr[i].page);else __free_page (pkt->addr[i].page); }else if (pkt->addr[i].kvaddr) {@@ -706,7 +706,7 @@ static int qib_user_sdma_pin_pages(const struct qib_devdata *dd,/* if error, return all pages not managed by pkt */ free_pages:while (i < j) -put_user_page (pages[i++]); +unpin_user_page (pages[i++]); done:return ret;drivers/infiniband/hw/usnic/usnic_uiom.c +1
-1@@ -75,7 +75,7 @@ static void usnic_uiom_put_pages(struct list_head *chunk_list, int dirty)for_each_sg (chunk->page_list, sg, chunk->nents, i) { page =sg_page (sg); pa =sg_phys (sg); -put_user_pages_dirty_lock (&page,1 , dirty); +unpin_user_pages_dirty_lock (&page,1 , dirty);usnic_dbg ("pa: %pa\n" , &pa); }kfree (chunk);drivers/infiniband/sw/siw/siw_mem.c +1
-1@@ -63,7 +63,7 @@ struct siw_mem *siw_mem_id2obj(struct siw_device *sdev, int stag_index)static void siw_free_plist (struct siw_page_chunk *chunk,int num_pages,bool dirty) { -put_user_pages_dirty_lock (chunk->plist, num_pages, dirty); +unpin_user_pages_dirty_lock (chunk->plist, num_pages, dirty); }void siw_umem_release (struct siw_umem *umem,bool dirty)drivers/media/v4l2-core/videobuf-dma-sg.c +2
-2@@ -349,8 +349,8 @@ int videobuf_dma_free(struct videobuf_dmabuf *dma)BUG_ON (dma->sglen);if (dma->pages) { -put_user_pages_dirty_lock (dma->pages, dma->nr_pages, - dma->direction ==DMA_FROM_DEVICE ); +unpin_user_pages_dirty_lock (dma->pages, dma->nr_pages, + dma->direction ==DMA_FROM_DEVICE );kfree (dma->pages); dma->pages =NULL ; }drivers/platform/goldfish/goldfish_pipe.c +2
-2@@ -360,8 +360,8 @@ static int transfer_max_buffers(struct goldfish_pipe *pipe, *consumed_size = pipe->command_buffer->rw_params.consumed_size; -put_user_pages_dirty_lock (pipe->pages, pages_count, - !is_write && *consumed_size >0 ); +unpin_user_pages_dirty_lock (pipe->pages, pages_count, + !is_write && *consumed_size >0 );mutex_unlock (&pipe->lock);return 0 ;drivers/vfio/vfio_iommu_type1.c +1
-1@@ -328,7 +328,7 @@ static int put_pfn(unsigned long pfn, int prot)if (!is_invalid_reserved_pfn (pfn)) {struct page *page =pfn_to_page (pfn); -put_user_pages_dirty_lock (&page,1 , prot &IOMMU_WRITE ); +unpin_user_pages_dirty_lock (&page,1 , prot &IOMMU_WRITE );return 1 ; }return 0 ;fs/io_uring.c +2
-2@@ -3328,7 +3328,7 @@ static int io_sqe_buffer_unregister(struct io_ring_ctx *ctx)struct io_mapped_ubuf *imu = &ctx->user_bufs[i];for (j =0 ; j < imu->nr_bvecs; j++) -put_user_page (imu->bvec[j].bv_page); +unpin_user_page (imu->bvec[j].bv_page);if (ctx->account_mem)io_unaccount_mem (ctx->user, imu->nr_bvecs);@@ -3473,7 +3473,7 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, void __user *arg,* release any pages we did get */ if (pret >0 ) -put_user_pages (pages, pret); +unpin_user_pages (pages, pret);if (ctx->account_mem)io_unaccount_mem (ctx->user, nr_pages);kvfree (imu->bvec);include/linux/mm.h +15
-15@@ -1093,18 +1093,18 @@ static inline void put_page(struct page *page)* made against the page. ("gup-pinned" is another term for the latter). * * With this scheme, pin_user_pages() becomes special: such pages are marked as -* distinct from normal pages. As such, the put_user_page() call (and its +* distinct from normal pages. As such, the unpin_user_page() call (and its * variants) must be used in order to release gup-pinned pages. * * Choice of value: * * By making GUP_PIN_COUNTING_BIAS a power of two, debugging of page reference -* counts with respect to pin_user_pages() and put_user_page() becomes simpler, -* due to the fact that adding an even power of two to the page refcount has the -* effect of using only the upper N bits, for the code that counts up using the -* bias value. This means that the lower bits are left for the exclusive use of -* the original code that increments and decrements by one (or at least, by much -* smaller values than the bias value). +* counts with respect to pin_user_pages() and unpin_user_page() becomes +* simpler, due to the fact that adding an even power of two to the page +* refcount has the effect of using only the upper N bits, for the code that +* counts up using the bias value. This means that the lower bits are left for +* the exclusive use of the original code that increments and decrements by one +* (or at least, by much smaller values than the bias value). * * Of course, once the lower bits overflow into the upper bits (and this is * OK, because subtraction recovers the original values), then visual inspection @@ -1119,10 +1119,10 @@ static inline void put_page(struct page *page)*/ #define GUP_PIN_COUNTING_BIAS (1UL << 10) -void put_user_page (struct page *page); -void put_user_pages_dirty_lock (struct page **pages,unsigned long npages, -bool make_dirty); -void put_user_pages (struct page **pages,unsigned long npages); +void unpin_user_page (struct page *page); +void unpin_user_pages_dirty_lock (struct page **pages,unsigned long npages, +bool make_dirty); +void unpin_user_pages (struct page **pages,unsigned long npages);/** * page_dma_pinned() - report if a page is pinned for DMA. @@ -2673,7 +2673,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,#define FOLL_ANON 0x8000/* don't do file mappings */ #define FOLL_LONGTERM 0x10000/* mapping lifetime is indefinite: see below */ #define FOLL_SPLIT_PMD 0x20000/* split huge pmd before returning */ -#define FOLL_PIN 0x40000/* pages must be released via put_user_page() */ +#define FOLL_PIN 0x40000/* pages must be released via unpin_user_page */ /* * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each @@ -2708,7 +2708,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,* Direct IO). This lets the filesystem know that some non-file-system entity is * potentially changing the pages' data. In contrast to FOLL_GET (whose pages * are released via put_page()), FOLL_PIN pages must be released, ultimately, by -* a call to put_user_page(). +* a call to unpin_user_page(). * * FOLL_PIN is similar to FOLL_GET: both of these pin pages. They use different * and separate refcounting mechanisms, however, and that means that each has @@ -2716,7 +2716,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,* * FOLL_GET: get_user_pages*() to acquire, and put_page() to release. * -* FOLL_PIN: pin_user_pages*() to acquire, and put_user_pages to release. +* FOLL_PIN: pin_user_pages*() to acquire, and unpin_user_pages to release. * * FOLL_PIN and FOLL_GET are mutually exclusive for a given function call. * (The underlying pages may experience both FOLL_GET-based and FOLL_PIN-based @@ -2726,7 +2726,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,* FOLL_PIN should be set internally by the pin_user_pages*() APIs, never * directly by the caller. That's in order to help avoid mismatches when * releasing pages: get_user_pages*() pages must be released via put_page(), -* while pin_user_pages*() pages must be released via put_user_page(). +* while pin_user_pages*() pages must be released via unpin_user_page(). * * Please see Documentation/vm/pin_user_pages.rst for more information. */ include/linux/mmzone.h +1
-1@@ -245,7 +245,7 @@ enum node_stat_item {NR_WRITTEN ,/* page writings since bootup */ NR_KERNEL_MISC_RECLAIMABLE ,/* reclaimable non-slab kernel pages */ NR_FOLL_PIN_REQUESTED ,/* via: pin_user_page(), gup flag: FOLL_PIN */ -NR_FOLL_PIN_RETURNED ,/* pages returned via put_user_page() */ +NR_FOLL_PIN_RETURNED ,/* pages returned via unpin_user_page() */ NR_VM_NODE_STAT_ITEMS };mm/gup.c +23
-23@@ -113,15 +113,15 @@ static bool __put_devmap_managed_user_page(struct page *page)#endif /* CONFIG_DEV_PAGEMAP_OPS */ /** -* put_user_page() - release a dma-pinned page +* unpin_user_page() - release a dma-pinned page * @page: pointer to page to be released * * Pages that were pinned via pin_user_pages*() must be released via either -* put_user_page(), or one of the put_user_pages*() routines. This is so that -* such pages can be separately tracked and uniquely handled. In particular, -* interactions with RDMA and filesystems need special handling. +* unpin_user_page(), or one of the unpin_user_pages*() routines. This is so +* that such pages can be separately tracked and uniquely handled. In +* particular, interactions with RDMA and filesystems need special handling. */ -void put_user_page (struct page *page) +void unpin_user_page (struct page *page) { page =compound_head (page);@@ -139,10 +139,10 @@ void put_user_page(struct page *page)__update_proc_vmstat (page,NR_FOLL_PIN_RETURNED ,1 ); } -EXPORT_SYMBOL (put_user_page); +EXPORT_SYMBOL (unpin_user_page);/** -* put_user_pages_dirty_lock() - release and optionally dirty gup-pinned pages +* unpin_user_pages_dirty_lock() - release and optionally dirty gup-pinned pages * @pages: array of pages to be maybe marked dirty, and definitely released. * @npages: number of pages in the @pages array. * @make_dirty: whether to mark the pages dirty @@ -152,19 +152,19 @@ EXPORT_SYMBOL(put_user_page);* * For each page in the @pages array, make that page (or its head page, if a * compound page) dirty, if @make_dirty is true, and if the page was previously -* listed as clean. In any case, releases all pages using put_user_page(), -* possibly via put_user_pages(), for the non-dirty case. +* listed as clean. In any case, releases all pages using unpin_user_page(), +* possibly via unpin_user_pages(), for the non-dirty case. * -* Please see the put_user_page() documentation for details. +* Please see the unpin_user_page() documentation for details. * * set_page_dirty_lock() is used internally. If instead, set_page_dirty() is * required, then the caller should a) verify that this is really correct, * because _lock() is usually required, and b) hand code it: -* set_page_dirty_lock(), put_user_page(). +* set_page_dirty_lock(), unpin_user_page(). * */ -void put_user_pages_dirty_lock (struct page **pages,unsigned long npages, -bool make_dirty) +void unpin_user_pages_dirty_lock (struct page **pages,unsigned long npages, +bool make_dirty) {unsigned long index;@@ -175,7 +175,7 @@ void put_user_pages_dirty_lock(struct page **pages, unsigned long npages,*/ if (!make_dirty) { -put_user_pages (pages, npages); +unpin_user_pages (pages, npages);return ; }@@ -203,21 +203,21 @@ void put_user_pages_dirty_lock(struct page **pages, unsigned long npages,*/ if (!PageDirty (page))set_page_dirty_lock (page); -put_user_page (page); +unpin_user_page (page); } } -EXPORT_SYMBOL (put_user_pages_dirty_lock); +EXPORT_SYMBOL (unpin_user_pages_dirty_lock);/** -* put_user_pages() - release an array of gup-pinned pages. +* unpin_user_pages() - release an array of gup-pinned pages. * @pages: array of pages to be marked dirty and released. * @npages: number of pages in the @pages array. * -* For each page in the @pages array, release the page using put_user_page(). +* For each page in the @pages array, release the page using unpin_user_page(). * -* Please see the put_user_page() documentation for details. +* Please see the unpin_user_page() documentation for details. */ -void put_user_pages (struct page **pages,unsigned long npages) +void unpin_user_pages (struct page **pages,unsigned long npages) {unsigned long index;@@ -227,9 +227,9 @@ void put_user_pages(struct page **pages, unsigned long npages)* single operation to the head page should suffice. */ for (index =0 ; index < npages; index++) -put_user_page (pages[index]); +unpin_user_page (pages[index]); } -EXPORT_SYMBOL (put_user_pages); +EXPORT_SYMBOL (unpin_user_pages);#ifdef CONFIG_MMU static struct page *no_page_table(struct vm_area_struct *vma,@@ -1956,7 +1956,7 @@ static void __maybe_unused undo_dev_pagemap(int *nr, int nr_start,ClearPageReferenced (page);if (flags &FOLL_PIN ) -put_user_page (page); +unpin_user_page (page);else put_page (page); }mm/gup_benchmark.c +1
-1@@ -35,7 +35,7 @@ static void put_back_pages(int cmd, struct page **pages, unsigned long nr_pages)case PIN_FAST_BENCHMARK :case PIN_BENCHMARK : -put_user_pages (pages, nr_pages); +unpin_user_pages (pages, nr_pages);break ; } }mm/process_vm_access.c +2
-2@@ -126,8 +126,8 @@ static int process_vm_rw_single_vec(unsigned long addr, pa += pinned_pages *PAGE_SIZE ;/* If vm_write is set, the pages need to be made dirty: */ -put_user_pages_dirty_lock (process_pages, pinned_pages, - vm_write); +unpin_user_pages_dirty_lock (process_pages, pinned_pages, + vm_write); }return rc;net/xdp/xdp_umem.c +1
-1@@ -212,7 +212,7 @@ static int xdp_umem_map_pages(struct xdp_umem *umem)static void xdp_umem_unpin_pages (struct xdp_umem *umem) { -put_user_pages_dirty_lock (umem->pgs, umem->npgs, true); +unpin_user_pages_dirty_lock (umem->pgs, umem->npgs, true);kfree (umem->pgs); umem->pgs =NULL ;Signature · 1 line
2.24.0
Signature · 6 lines
Jan Kara <[email protected]> SUSE Labs, CR _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel