Add FOLL_LONGTERM to GUP fast and use it
Subsystems (43)
v3 cover Add FOLL_LONGTERM to GUP fast and use it
From: Ira Weiny <[email protected]>
Following discussion and review[1] here are the cleanups requested.
The biggest change for V3 was the disabling of the ability to use FOLL_LONGTERM in get_user_pages[unlocked|locked|remote]
Comments were also enhanced throughout to show potential users what FOLL_LONGTERM is all about and limitations it has.
Minor review comments were fixed
Original cover letter:
HFI1, qib, and mthca, use get_user_pages_fast() due to it performance advantages. These pages can be held for a significant time. But get_user_pages_fast() does not protect against mapping FS DAX pages.
Introduce FOLL_LONGTERM and use this flag in get_user_pages_fast() which retains the performance while also adding the FS DAX checks. XDP has also shown interest in using this functionality.[1]
In addition we change get_user_pages() to use the new FOLL_LONGTERM flag and remove the specialized get_user_pages_longterm call.
[1] https://lkml.org/lkml/2019/3/19/939
Ira Weiny (7):
mm/gup: Replace get_user_pages_longterm() with FOLL_LONGTERM mm/gup: Change write parameter to flags in fast walk mm/gup: Change GUP fast to use flags rather than a write 'bool' mm/gup: Add FOLL_LONGTERM capability to GUP fast IB/hfi1: Use the new FOLL_LONGTERM flag to get_user_pages_fast() IB/qib: Use the new FOLL_LONGTERM flag to get_user_pages_fast() IB/mthca: Use the new FOLL_LONGTERM flag to get_user_pages_fast()
arch/mips/mm/gup.c | 11 +- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 +- arch/powerpc/kvm/e500_mmu.c | 2 +- arch/powerpc/mm/mmu_context_iommu.c | 3 +- arch/s390/kvm/interrupt.c | 2 +- arch/s390/mm/gup.c | 12 +- arch/sh/mm/gup.c | 11 +- arch/sparc/mm/gup.c | 9 +- arch/x86/kvm/paging_tmpl.h | 2 +- arch/x86/kvm/svm.c | 2 +- drivers/fpga/dfl-afu-dma-region.c | 2 +- drivers/gpu/drm/via/via_dmablit.c | 3 +- drivers/infiniband/core/umem.c | 5 +- drivers/infiniband/hw/hfi1/user_pages.c | 3 +- drivers/infiniband/hw/mthca/mthca_memfree.c | 3 +- drivers/infiniband/hw/qib/qib_user_pages.c | 8 +- drivers/infiniband/hw/qib/qib_user_sdma.c | 2 +- drivers/infiniband/hw/usnic/usnic_uiom.c | 9 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 6 +- drivers/misc/genwqe/card_utils.c | 2 +- drivers/misc/vmw_vmci/vmci_host.c | 2 +- drivers/misc/vmw_vmci/vmci_queue_pair.c | 6 +- drivers/platform/goldfish/goldfish_pipe.c | 3 +- drivers/rapidio/devices/rio_mport_cdev.c | 4 +- drivers/sbus/char/oradax.c | 2 +- drivers/scsi/st.c | 3 +- drivers/staging/gasket/gasket_page_table.c | 4 +- drivers/tee/tee_shm.c | 2 +- drivers/vfio/vfio_iommu_spapr_tce.c | 3 +- drivers/vfio/vfio_iommu_type1.c | 3 +- drivers/vhost/vhost.c | 2 +- drivers/video/fbdev/pvr2fb.c | 2 +- drivers/virt/fsl_hypervisor.c | 2 +- drivers/xen/gntdev.c | 2 +- fs/io_uring.c | 5 +- fs/orangefs/orangefs-bufmap.c | 2 +- include/linux/mm.h | 45 ++- kernel/futex.c | 2 +- lib/iov_iter.c | 7 +- mm/gup.c | 288 +++++++++++++------- mm/gup_benchmark.c | 5 +- mm/util.c | 8 +- net/ceph/pagevec.c | 2 +- net/rds/info.c | 2 +- net/rds/rdma.c | 3 +- net/xdp/xdp_umem.c | 4 +- 46 files changed, 314 insertions(+), 200 deletions(-)
Signature · 1 line
2.20.1
PATCH v3 1/7 mm/gup: Replace get_user_pages_longterm() with FOLL_LONGTERM
From: Ira Weiny <[email protected]>
This patch 1 of 7, starts a series which aims to support FOLL_LONGTERM in get_user_pages_fast(). Some callers who would like to do a longterm (user controlled pin) of pages with the fast variant of GUP for performance purposes.
Rather than have a separate get_user_pages_longterm() call, introduce FOLL_LONGTERM and change the longterm callers to use it.
This patch does not change any functionality. In the short term "longterm" or user controlled pins are unsafe for Filesystems and FS DAX in particular has been blocked. However, callers of get_user_pages_fast() were not "protected".
FOLL_LONGTERM can _only_ be supported with get_user_pages[_fast]() as it requires vmas to determine if DAX is in use.
NOTE: In merging with the CMA changes we opt to change the get_user_pages() call in check_and_migrate_cma_pages() to a call of __get_user_pages_locked() on the newly migrated pages. This makes the code read better in that we are calling __get_user_pages_locked() on the pages before and after a potential migration.
As a side affect some of the interfaces are cleaned up but this is not the primary purpose of the series.
In review[1] it was asked:
<quote>
Quoted 4 lines
This I don't get - if you do lock down long term mappings performance of the actual
get_user_pagescall shouldn't matter to start with.What do I miss?
A couple of points.
First "longterm" is a relative thing and at this point is probably a misnomer. This is really flagging a pin which is going to be given to hardware and can't move. I've thought of a couple of alternative names but I think we have to settle on if we are going to use FL_LAYOUT or something else to solve the "longterm" problem. Then I think we can change the flag to a better name.
Second, It depends on how often you are registering memory. I have spoken with some RDMA users who consider MR in the performance path... For the overall application performance. I don't have the numbers as the tests for HFI1 were done a long time ago. But there was a significant advantage. Some of which is probably due to the fact that you don't have to hold mmap_sem.
Finally, architecturally I think it would be good for everyone to use *_fast. There are patches submitted to the RDMA list which would allow the use of *_fast (they reworking the use of mmap_sem) and as soon as they are accepted I'll submit a patch to convert the RDMA core as well. Also to this point others are looking to use *_fast.
As an asside, Jasons pointed out in my previous submission that *_fast and *_unlocked look very much the same. I agree and I think further cleanup will be coming. But I'm focused on getting the final solution for DAX at the moment.
</quote>
[1] https://lore.kernel.org/lkml/[email protected]/T/#md6abad2569f3bf6c1f03686c8097ab6563e94965
---
Changes from V2:
Enhance commit message
Add discussion from Christoph Lameter
Fix comments
fix typo 's/uer/user'
Move FOLL_LONGTERM documentation next to FOLL_LONGTERM definition Enhance FOLL_LONGTERM documentation
remove __always_inline from __gup_longterm_locked()
Disallow FOLL_LONGTERM in calls which are incompatible
Changes from V1:
Rebased on 5.1 merge
Adjusted for changes introduced by CONFIG_CMA This included changing check_and_migrate_cma_pages to do the right thing
Convert a new users of GUP longterm
io_uring.c
xdp_umem.c
arch/powerpc/mm/mmu_context_iommu.c | 3 +- drivers/infiniband/core/umem.c | 5 +- drivers/infiniband/hw/qib/qib_user_pages.c | 8 +- drivers/infiniband/hw/usnic/usnic_uiom.c | 9 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 6 +- drivers/vfio/vfio_iommu_type1.c | 3 +- fs/io_uring.c | 5 +- include/linux/mm.h | 41 +++-- mm/gup.c | 190 +++++++++++++-------- mm/gup_benchmark.c | 5 +- net/xdp/xdp_umem.c | 4 +- 11 files changed, 172 insertions(+), 107 deletions(-)
arch/powerpc/mm/mmu_context_iommu.c +2 -1
@@ -148,7 +148,8 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua, }down_read (&mm->mmap_sem); - ret =get_user_pages_longterm (ua, entries,FOLL_WRITE , mem->hpages,NULL ); + ret =get_user_pages (ua, entries,FOLL_WRITE |FOLL_LONGTERM , + mem->hpages,NULL );up_read (&mm->mmap_sem);if (ret != entries) {/* free the reference taken */
drivers/infiniband/core/umem.c +3 -2
@@ -189,10 +189,11 @@ struct ib_umem *ib_umem_get(struct ib_udata *udata, unsigned long addr,while (npages) {down_read (&mm->mmap_sem); - ret =get_user_pages_longterm (cur_base, + ret =get_user_pages (cur_base, min_t(unsigned long, npages,PAGE_SIZE /sizeof (struct page *)), - gup_flags, page_list, vma_list); + gup_flags |FOLL_LONGTERM , + page_list, vma_list);if (ret <0 ) {up_read (&mm->mmap_sem); goto umem_release;
drivers/infiniband/hw/qib/qib_user_pages.c +4 -4
@@ -114,10 +114,10 @@ int qib_get_user_pages(unsigned long start_page, size_t num_pages,down_read (¤t->mm->mmap_sem);for (got =0 ; got < num_pages; got += ret) { - ret =get_user_pages_longterm (start_page + got *PAGE_SIZE , - num_pages - got, -FOLL_WRITE |FOLL_FORCE , - p + got,NULL ); + ret =get_user_pages (start_page + got *PAGE_SIZE , + num_pages - got, +FOLL_LONGTERM |FOLL_WRITE |FOLL_FORCE , + p + got,NULL );if (ret <0 ) {up_read (¤t->mm->mmap_sem); goto bail_release;
drivers/infiniband/hw/usnic/usnic_uiom.c +5 -4
@@ -143,10 +143,11 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable, ret =0 ;while (npages) { - ret =get_user_pages_longterm (cur_base, - min_t(unsigned long, npages, -PAGE_SIZE /sizeof (struct page *)), - gup_flags, page_list,NULL ); + ret =get_user_pages (cur_base, + min_t(unsigned long, npages, +PAGE_SIZE /sizeof (struct page *)), + gup_flags |FOLL_LONGTERM , + page_list,NULL );if (ret <0 ) goto out;
drivers/media/v4l2-core/videobuf-dma-sg.c +3 -3
@@ -186,12 +186,12 @@ static int videobuf_dma_init_user_locked(struct videobuf_dmabuf *dma,dprintk (1 ,"init user [0x%lx+0x%lx => %d pages]\n" , data, size, dma->nr_pages); - err =get_user_pages_longterm (data &PAGE_MASK , dma->nr_pages, - flags, dma->pages,NULL ); + err =get_user_pages (data &PAGE_MASK , dma->nr_pages, + flags |FOLL_LONGTERM , dma->pages,NULL );if (err != dma->nr_pages) { dma->nr_pages = (err >=0 ) ? err :0 ; -dprintk (1 ,"get_user_pages_longterm: err=%d [%d]\n" , err, +dprintk (1 ,"get_user_pages: err=%d [%d]\n" , err, dma->nr_pages);return err <0 ? err : -EINVAL ; }
drivers/vfio/vfio_iommu_type1.c +2 -1
@@ -351,7 +351,8 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr,down_read (&mm->mmap_sem);if (mm == current->mm) { - ret =get_user_pages_longterm (vaddr,1 , flags, page, vmas); + ret =get_user_pages (vaddr,1 , flags |FOLL_LONGTERM , page, + vmas); }else { ret =get_user_pages_remote (NULL , mm, vaddr,1 , flags, page, vmas,NULL );
fs/io_uring.c +3 -2
@@ -2450,8 +2450,9 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, void __user *arg, ret =0 ;down_read (¤t->mm->mmap_sem); - pret =get_user_pages_longterm (ubuf, nr_pages,FOLL_WRITE , - pages, vmas); + pret =get_user_pages (ubuf, nr_pages, +FOLL_WRITE |FOLL_LONGTERM , + pages, vmas);if (pret == nr_pages) {/* don't support file backed memory */ for (j =0 ; j < nr_pages; j++) {
include/linux/mm.h +28 -13
53 lines not on this page · raw message
mm/gup.c +117 -73
259 lines not on this page · raw message
mm/gup_benchmark.c +3 -2
@@ -54,8 +54,9 @@ static int __gup_benchmark_ioctl(unsigned int cmd, pages + i);break ;case GUP_LONGTERM_BENCHMARK : - nr =get_user_pages_longterm (addr, nr, gup->flags &1 , - pages + i,NULL ); + nr =get_user_pages (addr, nr, + (gup->flags &1 ) |FOLL_LONGTERM , + pages + i,NULL );break ;case GUP_BENCHMARK : nr = get_user_pages(addr, nr, gup->flags & 1, pages + i,
net/xdp/xdp_umem.c +2 -2
@@ -267,8 +267,8 @@ static int xdp_umem_pin_pages(struct xdp_umem *umem)return -ENOMEM ;down_read (¤t->mm->mmap_sem); - npgs =get_user_pages_longterm (umem->address, umem->npgs, - gup_flags, &umem->pgs[0 ],NULL ); + npgs =get_user_pages (umem->address, umem->npgs, + gup_flags |FOLL_LONGTERM , &umem->pgs[0 ],NULL );up_read (¤t->mm->mmap_sem);if (npgs != umem->npgs) {
Signature · 1 line
2.20.1
PATCH v3 2/7 mm/gup: Change write parameter to flags in fast walk
In reply to [email protected]From: Ira Weiny <[email protected]>
In order to support more options in the GUP fast walk, change the write parameter to flags throughout the call stack.
This patch does not change functionality and passes FOLL_WRITE where write was previously used.
---
Changes from V2
Add review by
mm/gup.c | 52 ++++++++++++++++++++++++++-------------------------- 1 file changed, 26 insertions(+), 26 deletions(-)
mm/gup.c +26 -26
@@ -1591,7 +1591,7 @@ static void undo_dev_pagemap(int *nr, int nr_start, struct page **pages)#ifdef CONFIG_ARCH_HAS_PTE_SPECIAL static int gup_pte_range (pmd_t pmd,unsigned long addr,unsigned long end, -int write,struct page **pages,int *nr) +unsigned int flags,struct page **pages,int *nr) {struct dev_pagemap *pgmap =NULL ;int nr_start = *nr, ret =0 ;
@@ -1609,7 +1609,7 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,if (pte_protnone (pte)) goto pte_unmap; -if (!pte_access_permitted (pte, write)) +if (!pte_access_permitted (pte, flags &FOLL_WRITE )) goto pte_unmap;if (pte_devmap (pte)) {
@@ -1661,7 +1661,7 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,* useful to have gup_huge_pmd even if we can't operate on ptes. */ static int gup_pte_range (pmd_t pmd,unsigned long addr,unsigned long end, -int write,struct page **pages,int *nr) +unsigned int flags,struct page **pages,int *nr) {return 0 ; }
@@ -1744,12 +1744,12 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr,#endif static int gup_huge_pmd (pmd_t orig,pmd_t *pmdp,unsigned long addr, -unsigned long end,int write,struct page **pages,int *nr) +unsigned long end,unsigned int flags,struct page **pages,int *nr) {struct page *head, *page;int refs; -if (!pmd_access_permitted (orig, write)) +if (!pmd_access_permitted (orig, flags &FOLL_WRITE ))return 0 ;if (pmd_devmap (orig))
@@ -1782,12 +1782,12 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, }static int gup_huge_pud (pud_t orig,pud_t *pudp,unsigned long addr, -unsigned long end,int write,struct page **pages,int *nr) +unsigned long end,unsigned int flags,struct page **pages,int *nr) {struct page *head, *page;int refs; -if (!pud_access_permitted (orig, write)) +if (!pud_access_permitted (orig, flags &FOLL_WRITE ))return 0 ;if (pud_devmap (orig))
@@ -1820,13 +1820,13 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, }static int gup_huge_pgd (pgd_t orig,pgd_t *pgdp,unsigned long addr, -unsigned long end,int write, +unsigned long end,unsigned int flags,struct page **pages,int *nr) {int refs;struct page *head, *page; -if (!pgd_access_permitted (orig, write)) +if (!pgd_access_permitted (orig, flags &FOLL_WRITE ))return 0 ;BUILD_BUG_ON (pgd_devmap (orig));
@@ -1857,7 +1857,7 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, }static int gup_pmd_range (pud_t pud,unsigned long addr,unsigned long end, -int write,struct page **pages,int *nr) +unsigned int flags,struct page **pages,int *nr) {unsigned long next;pmd_t *pmdp;
@@ -1880,7 +1880,7 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,if (pmd_protnone (pmd))return 0 ; -if (!gup_huge_pmd (pmd, pmdp, addr, next, write, +if (!gup_huge_pmd (pmd, pmdp, addr, next, flags, pages, nr))return 0 ;
@@ -1890,9 +1890,9 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,* pmd format and THP pmd format */ if (!gup_huge_pd (__hugepd (pmd_val (pmd)), addr, -PMD_SHIFT , next, write, pages, nr)) +PMD_SHIFT , next, flags, pages, nr))return 0 ; - }else if (!gup_pte_range (pmd, addr, next, write, pages, nr)) + }else if (!gup_pte_range (pmd, addr, next, flags, pages, nr))return 0 ; }while (pmdp++, addr = next, addr != end);
@@ -1900,7 +1900,7 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, }static int gup_pud_range (p4d_t p4d,unsigned long addr,unsigned long end, -int write,struct page **pages,int *nr) +unsigned int flags,struct page **pages,int *nr) {unsigned long next;pud_t *pudp;
@@ -1913,14 +1913,14 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end,if (pud_none (pud))return 0 ;if (unlikely (pud_huge (pud))) { -if (!gup_huge_pud (pud, pudp, addr, next, write, +if (!gup_huge_pud (pud, pudp, addr, next, flags, pages, nr))return 0 ; }else if (unlikely (is_hugepd (__hugepd (pud_val (pud))))) {if (!gup_huge_pd (__hugepd (pud_val (pud)), addr, -PUD_SHIFT , next, write, pages, nr)) +PUD_SHIFT , next, flags, pages, nr))return 0 ; - }else if (!gup_pmd_range (pud, addr, next, write, pages, nr)) + }else if (!gup_pmd_range (pud, addr, next, flags, pages, nr))return 0 ; }while (pudp++, addr = next, addr != end);
@@ -1928,7 +1928,7 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, }static int gup_p4d_range (pgd_t pgd,unsigned long addr,unsigned long end, -int write,struct page **pages,int *nr) +unsigned int flags,struct page **pages,int *nr) {unsigned long next;p4d_t *p4dp;
@@ -1943,9 +1943,9 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end,BUILD_BUG_ON (p4d_huge (p4d));if (unlikely (is_hugepd (__hugepd (p4d_val (p4d))))) {if (!gup_huge_pd (__hugepd (p4d_val (p4d)), addr, -P4D_SHIFT , next, write, pages, nr)) +P4D_SHIFT , next, flags, pages, nr))return 0 ; - }else if (!gup_pud_range (p4d, addr, next, write, pages, nr)) + }else if (!gup_pud_range (p4d, addr, next, flags, pages, nr))return 0 ; }while (p4dp++, addr = next, addr != end);
@@ -1953,7 +1953,7 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, }static void gup_pgd_range (unsigned long addr,unsigned long end, -int write,struct page **pages,int *nr) +unsigned int flags,struct page **pages,int *nr) {unsigned long next;pgd_t *pgdp;
@@ -1966,14 +1966,14 @@ static void gup_pgd_range(unsigned long addr, unsigned long end,if (pgd_none (pgd))return ;if (unlikely (pgd_huge (pgd))) { -if (!gup_huge_pgd (pgd, pgdp, addr, next, write, +if (!gup_huge_pgd (pgd, pgdp, addr, next, flags, pages, nr))return ; }else if (unlikely (is_hugepd (__hugepd (pgd_val (pgd))))) {if (!gup_huge_pd (__hugepd (pgd_val (pgd)), addr, -PGDIR_SHIFT , next, write, pages, nr)) +PGDIR_SHIFT , next, flags, pages, nr))return ; - }else if (!gup_p4d_range (pgd, addr, next, write, pages, nr)) + }else if (!gup_p4d_range (pgd, addr, next, flags, pages, nr))return ; }while (pgdp++, addr = next, addr != end); }
@@ -2027,7 +2027,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,if (gup_fast_permitted (start, nr_pages)) {local_irq_save (flags); -gup_pgd_range (start, end, write, pages, &nr); +gup_pgd_range (start, end, write ?FOLL_WRITE :0 , pages, &nr);local_irq_restore (flags); }
@@ -2069,7 +2069,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write,if (gup_fast_permitted (start, nr_pages)) {local_irq_disable (); -gup_pgd_range (addr, end, write, pages, &nr); +gup_pgd_range (addr, end, write ?FOLL_WRITE :0 , pages, &nr);local_irq_enable (); ret = nr; }
Signature · 1 line
2.20.1
PATCH v3 3/7 mm/gup: Change GUP fast to use flags rather than a write 'bool'
In reply to [email protected]From: Ira Weiny <[email protected]>
To facilitate additional options to get_user_pages_fast() change the singular write parameter to be gup_flags.
This patch does not change any functionality. New functionality will follow in subsequent patches.
Some of the get_user_pages_fast() call sites were unchanged because they already passed FOLL_WRITE or 0 for the write parameter.
NOTE: It was suggested to change the ordering of the get_user_pages_fast() arguments to ensure that callers were converted. This breaks the current GUP call site convention of having the returned pages be the final parameter. So the suggestion was rejected.
---
Changes from V2:
Added Review tag
Changes from V1:
Rebase to current merge tree arch/powerpc/mm/mmu_context_iommu.c no longer calls gup_fast
The gup_longterm it used was converted in a previous patch
arch/mips/mm/gup.c | 11 ++++++----- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 ++-- arch/powerpc/kvm/e500_mmu.c | 2 +- arch/s390/kvm/interrupt.c | 2 +- arch/s390/mm/gup.c | 12 ++++++------ arch/sh/mm/gup.c | 11 ++++++----- arch/sparc/mm/gup.c | 9 +++++---- arch/x86/kvm/paging_tmpl.h | 2 +- arch/x86/kvm/svm.c | 2 +- drivers/fpga/dfl-afu-dma-region.c | 2 +- drivers/gpu/drm/via/via_dmablit.c | 3 ++- drivers/infiniband/hw/hfi1/user_pages.c | 3 ++- drivers/misc/genwqe/card_utils.c | 2 +- drivers/misc/vmw_vmci/vmci_host.c | 2 +- drivers/misc/vmw_vmci/vmci_queue_pair.c | 6 ++++-- drivers/platform/goldfish/goldfish_pipe.c | 3 ++- drivers/rapidio/devices/rio_mport_cdev.c | 4 +++- drivers/sbus/char/oradax.c | 2 +- drivers/scsi/st.c | 3 ++- drivers/staging/gasket/gasket_page_table.c | 4 ++-- drivers/tee/tee_shm.c | 2 +- drivers/vfio/vfio_iommu_spapr_tce.c | 3 ++- drivers/vhost/vhost.c | 2 +- drivers/video/fbdev/pvr2fb.c | 2 +- drivers/virt/fsl_hypervisor.c | 2 +- drivers/xen/gntdev.c | 2 +- fs/orangefs/orangefs-bufmap.c | 2 +- include/linux/mm.h | 4 ++-- kernel/futex.c | 2 +- lib/iov_iter.c | 7 +++++-- mm/gup.c | 10 +++++----- mm/util.c | 8 ++++---- net/ceph/pagevec.c | 2 +- net/rds/info.c | 2 +- net/rds/rdma.c | 3 ++- 35 files changed, 79 insertions(+), 63 deletions(-)
arch/mips/mm/gup.c +6 -5
@@ -235,7 +235,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* get_user_pages_fast() - pin user pages in memory * @start: starting user address * @nr_pages: number of pages from start to pin -* @write: whether pages will be written to +* @gup_flags: flags modifying pin behaviour * @pages: array that receives pointers to the pages pinned. * Should be at least nr_pages long. *
@@ -247,8 +247,8 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* requested. If nr_pages is 0 or negative, returns 0. If no pages * were pinned, returns -errno. */ -int get_user_pages_fast (unsigned long start,int nr_pages,int write, -struct page **pages) +int get_user_pages_fast (unsigned long start,int nr_pages, +unsigned int gup_flags,struct page **pages) {struct mm_struct *mm = current->mm;unsigned long addr, len, end;
@@ -273,7 +273,8 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write, next =pgd_addr_end (addr, end);if (pgd_none (pgd)) goto slow; -if (!gup_pud_range (pgd, addr, next, write, pages, &nr)) +if (!gup_pud_range (pgd, addr, next, gup_flags &FOLL_WRITE , + pages, &nr)) goto slow; }while (pgdp++, addr = next, addr != end);local_irq_enable ();
@@ -289,7 +290,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write, pages += nr; ret =get_user_pages_unlocked (start, (end - start) >>PAGE_SHIFT , - pages, write ?FOLL_WRITE :0 ); + pages, gup_flags);/* Have to be a bit careful with return values */ if (nr >0 ) {
arch/powerpc/kvm/book3s_64_mmu_hv.c +2 -2
@@ -600,7 +600,7 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu,/* If writing != 0, then the HPTE must allow writing, if we get here */ write_ok = writing; hva =gfn_to_hva_memslot (memslot, gfn); - npages =get_user_pages_fast (hva,1 , writing, pages); + npages =get_user_pages_fast (hva,1 , writing ?FOLL_WRITE :0 , pages);if (npages <1 ) {/* Check if it's an I/O mapping */ down_read (¤t->mm->mmap_sem);
@@ -1193,7 +1193,7 @@ void *kvmppc_pin_guest_page(struct kvm *kvm, unsigned long gpa,if (!memslot || (memslot->flags &KVM_MEMSLOT_INVALID )) goto err; hva =gfn_to_hva_memslot (memslot, gfn); - npages =get_user_pages_fast (hva,1 ,1 , pages); + npages =get_user_pages_fast (hva,1 ,FOLL_WRITE , pages);if (npages <1 ) goto err; page = pages[0 ];
arch/powerpc/kvm/e500_mmu.c +1 -1
@@ -783,7 +783,7 @@ int kvm_vcpu_ioctl_config_tlb(struct kvm_vcpu *vcpu,if (!pages)return -ENOMEM ; - ret =get_user_pages_fast (cfg->array, num_pages,1 , pages); + ret =get_user_pages_fast (cfg->array, num_pages,FOLL_WRITE , pages);if (ret <0 ) goto free_pages;
arch/s390/kvm/interrupt.c +1 -1
@@ -2376,7 +2376,7 @@ static int kvm_s390_adapter_map(struct kvm *kvm, unsigned int id, __u64 addr) ret = -EFAULT ; goto out; } - ret =get_user_pages_fast (map->addr,1 ,1 , &map->page); + ret =get_user_pages_fast (map->addr,1 ,FOLL_WRITE , &map->page);if (ret <0 ) goto out;BUG_ON (ret !=1 );
arch/s390/mm/gup.c +6 -6
@@ -265,7 +265,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* get_user_pages_fast() - pin user pages in memory * @start: starting user address * @nr_pages: number of pages from start to pin -* @write: whether pages will be written to +* @gup_flags: flags modifying pin behaviour * @pages: array that receives pointers to the pages pinned. * Should be at least nr_pages long. *
@@ -277,22 +277,22 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* requested. If nr_pages is 0 or negative, returns 0. If no pages * were pinned, returns -errno. */ -int get_user_pages_fast (unsigned long start,int nr_pages,int write, -struct page **pages) +int get_user_pages_fast (unsigned long start,int nr_pages, +unsigned int gup_flags,struct page **pages) {int nr, ret;might_sleep (); start &=PAGE_MASK ; - nr =__get_user_pages_fast (start, nr_pages, write, pages); + nr =__get_user_pages_fast (start, nr_pages, gup_flags &FOLL_WRITE , + pages);if (nr == nr_pages)return nr;/* Try to get the remaining pages with get_user_pages */ start += nr <<PAGE_SHIFT ; pages += nr; - ret =get_user_pages_unlocked (start, nr_pages - nr, pages, - write ?FOLL_WRITE :0 ); + ret =get_user_pages_unlocked (start, nr_pages - nr, pages, gup_flags);/* Have to be a bit careful with return values */ if (nr >0 ) ret = (ret <0 ) ? nr : ret + nr;
arch/sh/mm/gup.c +6 -5
@@ -204,7 +204,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* get_user_pages_fast() - pin user pages in memory * @start: starting user address * @nr_pages: number of pages from start to pin -* @write: whether pages will be written to +* @gup_flags: flags modifying pin behaviour * @pages: array that receives pointers to the pages pinned. * Should be at least nr_pages long. *
@@ -216,8 +216,8 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* requested. If nr_pages is 0 or negative, returns 0. If no pages * were pinned, returns -errno. */ -int get_user_pages_fast (unsigned long start,int nr_pages,int write, -struct page **pages) +int get_user_pages_fast (unsigned long start,int nr_pages, +unsigned int gup_flags,struct page **pages) {struct mm_struct *mm = current->mm;unsigned long addr, len, end;
@@ -241,7 +241,8 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write, next =pgd_addr_end (addr, end);if (pgd_none (pgd)) goto slow; -if (!gup_pud_range (pgd, addr, next, write, pages, &nr)) +if (!gup_pud_range (pgd, addr, next, gup_flags &FOLL_WRITE , + pages, &nr)) goto slow; }while (pgdp++, addr = next, addr != end);local_irq_enable ();
@@ -261,7 +262,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write, ret =get_user_pages_unlocked (start, (end - start) >>PAGE_SHIFT , pages, - write ?FOLL_WRITE :0 ); + gup_flags);/* Have to be a bit careful with return values */ if (nr >0 ) {
arch/sparc/mm/gup.c +5 -4
@@ -245,8 +245,8 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,return nr; } -int get_user_pages_fast (unsigned long start,int nr_pages,int write, -struct page **pages) +int get_user_pages_fast (unsigned long start,int nr_pages, +unsigned int gup_flags,struct page **pages) {struct mm_struct *mm = current->mm;unsigned long addr, len, end;
@@ -303,7 +303,8 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write, next =pgd_addr_end (addr, end);if (pgd_none (pgd)) goto slow; -if (!gup_pud_range (pgd, addr, next, write, pages, &nr)) +if (!gup_pud_range (pgd, addr, next, gup_flags &FOLL_WRITE , + pages, &nr)) goto slow; }while (pgdp++, addr = next, addr != end);
@@ -324,7 +325,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write, ret =get_user_pages_unlocked (start, (end - start) >>PAGE_SHIFT , pages, - write ?FOLL_WRITE :0 ); + gup_flags);/* Have to be a bit careful with return values */ if (nr >0 ) {
arch/x86/kvm/paging_tmpl.h +1 -1
@@ -140,7 +140,7 @@ static int FNAME(cmpxchg_gpte)(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,pt_element_t *table;struct page *page; - npages =get_user_pages_fast ((unsigned long )ptep_user,1 ,1 , &page); + npages =get_user_pages_fast ((unsigned long )ptep_user,1 ,FOLL_WRITE , &page);/* Check if the user is doing something meaningless. */ if (unlikely (npages !=1 ))return -EFAULT ;
arch/x86/kvm/svm.c +1 -1
@@ -1804,7 +1804,7 @@ static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr,return NULL ;/* Pin the user virtual address. */ - npinned =get_user_pages_fast (uaddr, npages, write ?FOLL_WRITE :0 , pages); + npinned =get_user_pages_fast (uaddr, npages,FOLL_WRITE , pages);if (npinned != npages) {pr_err ("SEV: Failure locking %lu pages.\n" , npages); goto err;
drivers/fpga/dfl-afu-dma-region.c +1 -1
@@ -102,7 +102,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata, goto unlock_vm; } - pinned =get_user_pages_fast (region->user_addr, npages,1 , + pinned =get_user_pages_fast (region->user_addr, npages,FOLL_WRITE , region->pages);if (pinned <0 ) { ret = pinned;
drivers/gpu/drm/via/via_dmablit.c +2 -1
@@ -243,7 +243,8 @@ via_lock_all_dma_pages(drm_via_sg_info_t *vsg, drm_via_dmablit_t *xfer)if (NULL == vsg->pages)return -ENOMEM ; ret =get_user_pages_fast ((unsigned long )xfer->mem_addr, - vsg->num_pages, vsg->direction ==DMA_FROM_DEVICE , + vsg->num_pages, + vsg->direction ==DMA_FROM_DEVICE ?FOLL_WRITE :0 , vsg->pages);if (ret != vsg->num_pages) {if (ret <0 )
drivers/infiniband/hw/hfi1/user_pages.c +2 -1
@@ -105,7 +105,8 @@ int hfi1_acquire_user_pages(struct mm_struct *mm, unsigned long vaddr, size_t np {int ret; - ret =get_user_pages_fast (vaddr, npages, writable, pages); + ret =get_user_pages_fast (vaddr, npages, writable ?FOLL_WRITE :0 , + pages);if (ret <0 )return ret;
drivers/misc/genwqe/card_utils.c +1 -1
@@ -603,7 +603,7 @@ int genwqe_user_vmap(struct genwqe_dev *cd, struct dma_mapping *m, void *uaddr,/* pin user pages in memory */ rc =get_user_pages_fast (data &PAGE_MASK ,/* page aligned addr */ m->nr_pages, - m->write,/* readable/writable */ + m->write ?FOLL_WRITE :0 ,/* readable/writable */ m->page_list);/* ptrs to pages */ if (rc <0 ) goto fail_get_user_pages;
drivers/misc/vmw_vmci/vmci_host.c +1 -1
@@ -242,7 +242,7 @@ static int vmci_host_setup_notify(struct vmci_ctx *context,/* * Lock physical page backing a given user VA. */ - retval =get_user_pages_fast (uva,1 ,1 , &context->notify_page); + retval =get_user_pages_fast (uva,1 ,FOLL_WRITE , &context->notify_page);if (retval !=1 ) { context->notify_page =NULL ;return VMCI_ERROR_GENERIC ;
drivers/misc/vmw_vmci/vmci_queue_pair.c +4 -2
@@ -659,7 +659,8 @@ static int qp_host_get_user_memory(u64 produce_uva,int err =VMCI_SUCCESS ; retval =get_user_pages_fast ((uintptr_t ) produce_uva, - produce_q->kernel_if->num_pages,1 , + produce_q->kernel_if->num_pages, +FOLL_WRITE , produce_q->kernel_if->u.h.header_page);if (retval < (int )produce_q->kernel_if->num_pages) { pr_debug("get_user_pages_fast(produce) failed (retval=%d)",
@@ -671,7 +672,8 @@ static int qp_host_get_user_memory(u64 produce_uva, } retval =get_user_pages_fast ((uintptr_t ) consume_uva, - consume_q->kernel_if->num_pages,1 , + consume_q->kernel_if->num_pages, +FOLL_WRITE , consume_q->kernel_if->u.h.header_page);if (retval < (int )consume_q->kernel_if->num_pages) { pr_debug("get_user_pages_fast(consume) failed (retval=%d)",
drivers/platform/goldfish/goldfish_pipe.c +2 -1
@@ -274,7 +274,8 @@ static int pin_user_pages(unsigned long first_page, *iter_last_page_size = last_page_size; } - ret =get_user_pages_fast (first_page, requested_pages, !is_write, + ret =get_user_pages_fast (first_page, requested_pages, + !is_write ?FOLL_WRITE :0 , pages);if (ret <=0 )return -EFAULT ;
drivers/rapidio/devices/rio_mport_cdev.c +3 -1
@@ -868,7 +868,9 @@ rio_dma_transfer(struct file *filp, u32 transfer_mode, pinned =get_user_pages_fast ( (unsigned long )xfer->loc_addr &PAGE_MASK , - nr_pages, dir ==DMA_FROM_DEVICE , page_list); + nr_pages, + dir ==DMA_FROM_DEVICE ?FOLL_WRITE :0 , + page_list);if (pinned != nr_pages) {if (pinned <0 ) {
drivers/sbus/char/oradax.c +1 -1
@@ -437,7 +437,7 @@ static int dax_lock_page(void *va, struct page **p)dax_dbg ("uva %p" , va); - ret =get_user_pages_fast ((unsigned long )va,1 ,1 , p); + ret =get_user_pages_fast ((unsigned long )va,1 ,FOLL_WRITE , p);if (ret ==1 ) {dax_dbg ("locked page %p, for VA %p" , *p, va);return 0 ;
drivers/scsi/st.c +2 -1
@@ -4922,7 +4922,8 @@ static int sgl_map_user_pages(struct st_buffer *STbp,/* Try to fault in all of the necessary pages */ /* rw==READ means read from drive, write into memory area */ - res =get_user_pages_fast (uaddr, nr_pages, rw ==READ , pages); + res =get_user_pages_fast (uaddr, nr_pages, rw ==READ ?FOLL_WRITE :0 , + pages);/* Errors and no page mapped should return here */ if (res < nr_pages)
drivers/staging/gasket/gasket_page_table.c +2 -2
@@ -486,8 +486,8 @@ static int gasket_perform_mapping(struct gasket_page_table *pg_tbl, ptes[i].dma_addr = pg_tbl->coherent_pages[0 ].paddr + off + i *PAGE_SIZE ; }else { - ret =get_user_pages_fast (page_addr - offset,1 ,1 , - &page); + ret =get_user_pages_fast (page_addr - offset,1 , +FOLL_WRITE , &page);if (ret <=0 ) { dev_err(pg_tbl->device,
drivers/tee/tee_shm.c +1 -1
@@ -273,7 +273,7 @@ struct tee_shm *tee_shm_register(struct tee_context *ctx, unsigned long addr, goto err; } - rc =get_user_pages_fast (start, num_pages,1 , shm->pages); + rc =get_user_pages_fast (start, num_pages,FOLL_WRITE , shm->pages);if (rc >0 ) shm->num_pages = rc;if (rc != num_pages) {
drivers/vfio/vfio_iommu_spapr_tce.c +2 -1
@@ -532,7 +532,8 @@ static int tce_iommu_use_page(unsigned long tce, unsigned long *hpa)enum dma_data_direction direction =iommu_tce_direction (tce);if (get_user_pages_fast (tce &PAGE_MASK ,1 , - direction !=DMA_TO_DEVICE , &page) !=1 ) + direction !=DMA_TO_DEVICE ?FOLL_WRITE :0 , + &page) !=1 )return -EFAULT ; *hpa =__pa ((unsigned long )page_address (page));
drivers/vhost/vhost.c +1 -1
@@ -1700,7 +1700,7 @@ static int set_bit_to_user(int nr, void __user *addr)int bit = nr + (log %PAGE_SIZE ) *8 ;int r; - r =get_user_pages_fast (log,1 ,1 , &page); + r =get_user_pages_fast (log,1 ,FOLL_WRITE , &page);if (r <0 )return r;BUG_ON (r !=1 );
drivers/video/fbdev/pvr2fb.c +1 -1
@@ -686,7 +686,7 @@ static ssize_t pvr2fb_write(struct fb_info *info, const char *buf,if (!pages)return -ENOMEM ; - ret =get_user_pages_fast ((unsigned long )buf, nr_pages, true, pages); + ret =get_user_pages_fast ((unsigned long )buf, nr_pages,FOLL_WRITE , pages);if (ret < nr_pages) { nr_pages = ret; ret = -EINVAL ;
drivers/virt/fsl_hypervisor.c +1 -1
@@ -244,7 +244,7 @@ static long ioctl_memcpy(struct fsl_hv_ioctl_memcpy __user *p)/* Get the physical addresses of the source buffer */ num_pinned =get_user_pages_fast (param.local_vaddr - lb_offset, - num_pages, param.source !=-1 , pages); + num_pages, param.source !=-1 ?FOLL_WRITE :0 , pages);if (num_pinned != num_pages) {/* get_user_pages() failed */
drivers/xen/gntdev.c +1 -1
@@ -852,7 +852,7 @@ static int gntdev_get_page(struct gntdev_copy_batch *batch, void __user *virt,unsigned long xen_pfn;int ret; - ret =get_user_pages_fast (addr,1 , writeable, &page); + ret =get_user_pages_fast (addr,1 , writeable ?FOLL_WRITE :0 , &page);if (ret <0 )return ret;
fs/orangefs/orangefs-bufmap.c +1 -1
@@ -269,7 +269,7 @@ orangefs_bufmap_map(struct orangefs_bufmap *bufmap,/* map the pages */ ret =get_user_pages_fast ((unsigned long )user_desc->ptr, - bufmap->page_count,1 , bufmap->page_array); + bufmap->page_count,FOLL_WRITE , bufmap->page_array);if (ret <0 )return ret;
include/linux/mm.h +2 -2
@@ -1531,8 +1531,8 @@ long get_user_pages_locked(unsigned long start, unsigned long nr_pages,long get_user_pages_unlocked (unsigned long start,unsigned long nr_pages,struct page **pages,unsigned int gup_flags); -int get_user_pages_fast (unsigned long start,int nr_pages,int write, -struct page **pages); +int get_user_pages_fast (unsigned long start,int nr_pages, +unsigned int gup_flags,struct page **pages);/* Container for pinned pfns / pages */ struct frame_vector {
kernel/futex.c +1 -1
@@ -543,7 +543,7 @@ get_futex_key(u32 __user *uaddr, int fshared, union futex_key *key, enum futex_aif (unlikely (should_fail_futex (fshared)))return -EFAULT ; - err =get_user_pages_fast (address,1 ,1 , &page); + err =get_user_pages_fast (address,1 ,FOLL_WRITE , &page);/* * If write access is not required (eg. FUTEX_WAIT), try * and get read-only access.
lib/iov_iter.c +5 -2
@@ -1293,7 +1293,9 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, len = maxpages *PAGE_SIZE ; addr &= ~(PAGE_SIZE -1 ); n =DIV_ROUND_UP (len,PAGE_SIZE ); - res =get_user_pages_fast (addr, n,iov_iter_rw (i) !=WRITE , pages); + res =get_user_pages_fast (addr, n, +iov_iter_rw (i) !=WRITE ?FOLL_WRITE :0 , + pages);if (unlikely (res <0 ))return res;return (res == n ? len : res *PAGE_SIZE ) - *start;
@@ -1374,7 +1376,8 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, p =get_pages_array (n);if (!p)return -ENOMEM ; - res =get_user_pages_fast (addr, n,iov_iter_rw (i) !=WRITE , p); + res =get_user_pages_fast (addr, n, +iov_iter_rw (i) !=WRITE ?FOLL_WRITE :0 , p);if (unlikely (res <0 )) {kvfree (p);return res;
mm/gup.c +5 -5
@@ -2038,7 +2038,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* get_user_pages_fast() - pin user pages in memory * @start: starting user address * @nr_pages: number of pages from start to pin -* @write: whether pages will be written to +* @gup_flags: flags modifying pin behaviour * @pages: array that receives pointers to the pages pinned. * Should be at least nr_pages long. *
@@ -2050,8 +2050,8 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* requested. If nr_pages is 0 or negative, returns 0. If no pages * were pinned, returns -errno. */ -int get_user_pages_fast (unsigned long start,int nr_pages,int write, -struct page **pages) +int get_user_pages_fast (unsigned long start,int nr_pages, +unsigned int gup_flags,struct page **pages) {unsigned long addr, len, end;int nr =0 , ret =0 ;
@@ -2069,7 +2069,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write,if (gup_fast_permitted (start, nr_pages)) {local_irq_disable (); -gup_pgd_range (addr, end, write ?FOLL_WRITE :0 , pages, &nr); +gup_pgd_range (addr, end, gup_flags, pages, &nr);local_irq_enable (); ret = nr; }
@@ -2080,7 +2080,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write, pages += nr; ret =get_user_pages_unlocked (start, nr_pages - nr, pages, - write ?FOLL_WRITE :0 ); + gup_flags);/* Have to be a bit careful with return values */ if (nr >0 ) {
mm/util.c +4 -4
@@ -318,7 +318,7 @@ EXPORT_SYMBOL_GPL(__get_user_pages_fast);* get_user_pages_fast() - pin user pages in memory * @start: starting user address * @nr_pages: number of pages from start to pin -* @write: whether pages will be written to +* @gup_flags: flags modifying pin behaviour * @pages: array that receives pointers to the pages pinned. * Should be at least nr_pages long. *
@@ -339,10 +339,10 @@ EXPORT_SYMBOL_GPL(__get_user_pages_fast);* were pinned, returns -errno. */ int __weak get_user_pages_fast (unsigned long start, -int nr_pages,int write,struct page **pages) +int nr_pages,unsigned int gup_flags, +struct page **pages) { -return get_user_pages_unlocked (start, nr_pages, pages, - write ?FOLL_WRITE :0 ); +return get_user_pages_unlocked (start, nr_pages, pages, gup_flags); }EXPORT_SYMBOL_GPL (get_user_pages_fast);
net/ceph/pagevec.c +1 -1
@@ -27,7 +27,7 @@ struct page **ceph_get_direct_page_vector(const void __user *data,while (got < num_pages) { rc =get_user_pages_fast ( (unsigned long )data + ((unsigned long )got *PAGE_SIZE ), - num_pages - got, write_page, pages + got); + num_pages - got, write_page ?FOLL_WRITE :0 , pages + got);if (rc <0 )break ;BUG_ON (rc ==0 );
net/rds/info.c +1 -1
@@ -193,7 +193,7 @@ int rds_info_getsockopt(struct socket *sock, int optname, char __user *optval, ret = -ENOMEM ; goto out; } - ret =get_user_pages_fast (start, nr_pages,1 , pages); + ret =get_user_pages_fast (start, nr_pages,FOLL_WRITE , pages);if (ret != nr_pages) {if (ret >0 ) nr_pages = ret;
net/rds/rdma.c +2 -1
@@ -158,7 +158,8 @@ static int rds_pin_pages(unsigned long user_addr, unsigned int nr_pages, {int ret; - ret =get_user_pages_fast (user_addr, nr_pages, write, pages); + ret =get_user_pages_fast (user_addr, nr_pages, write ?FOLL_WRITE :0 , + pages);if (ret >=0 && ret < nr_pages) {while (ret--)
Signature · 1 line
2.20.1
PATCH v3 4/7 mm/gup: Add FOLL_LONGTERM capability to GUP fast
In reply to [email protected]From: Ira Weiny <[email protected]>
DAX pages were previously unprotected from longterm pins when users called get_user_pages_fast().
Use the new FOLL_LONGTERM flag to check for DEVMAP pages and fall back to regular GUP processing if a DEVMAP page is encountered.
---
Changes from V2:
Add comment on special use case of FOLL_LONGTERM
mm/gup.c | 40 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 36 insertions(+), 4 deletions(-)
mm/gup.c +36 -4
@@ -1613,6 +1613,9 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, goto pte_unmap;if (pte_devmap (pte)) { +if (unlikely (flags &FOLL_LONGTERM )) + goto pte_unmap; + pgmap =get_dev_pagemap (pte_pfn (pte), pgmap);if (unlikely (!pgmap)) {undo_dev_pagemap (nr, nr_start, pages);
@@ -1752,8 +1755,11 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr,if (!pmd_access_permitted (orig, flags &FOLL_WRITE ))return 0 ; -if (pmd_devmap (orig)) +if (pmd_devmap (orig)) { +if (unlikely (flags &FOLL_LONGTERM )) +return 0 ;return __gup_device_huge_pmd (orig, pmdp, addr, end, pages, nr); + } refs =0 ; page =pmd_page (orig) + ((addr & ~PMD_MASK ) >>PAGE_SHIFT );
@@ -1790,8 +1796,11 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr,if (!pud_access_permitted (orig, flags &FOLL_WRITE ))return 0 ; -if (pud_devmap (orig)) +if (pud_devmap (orig)) { +if (unlikely (flags &FOLL_LONGTERM )) +return 0 ;return __gup_device_huge_pud (orig, pudp, addr, end, pages, nr); + } refs =0 ; page =pud_page (orig) + ((addr & ~PUD_MASK ) >>PAGE_SHIFT );
@@ -2034,6 +2043,29 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,return nr; } +static int __gup_longterm_unlocked (unsigned long start,int nr_pages, +unsigned int gup_flags,struct page **pages) +{ +int ret; + +/* +* FIXME: FOLL_LONGTERM does not work with +* get_user_pages_unlocked() (see comments in that function) +*/ +if (gup_flags &FOLL_LONGTERM ) { +down_read (¤t->mm->mmap_sem); + ret =__gup_longterm_locked (current, current->mm, + start, nr_pages, + pages,NULL , gup_flags); +up_read (¤t->mm->mmap_sem); + }else { + ret =get_user_pages_unlocked (start, nr_pages, + pages, gup_flags); + } + +return ret; +} +/** * get_user_pages_fast() - pin user pages in memory * @start: starting user address
@@ -2079,8 +2111,8 @@ int get_user_pages_fast(unsigned long start, int nr_pages, start += nr <<PAGE_SHIFT ; pages += nr; - ret =get_user_pages_unlocked (start, nr_pages - nr, pages, - gup_flags); + ret =__gup_longterm_unlocked (start, nr_pages - nr, + gup_flags, pages);/* Have to be a bit careful with return values */ if (nr >0 ) {
Signature · 1 line
2.20.1
PATCH v3 5/7 IB/hfi1: Use the new FOLL_LONGTERM flag to get_user_pages_fast()
In reply to [email protected]From: Ira Weiny <[email protected]>
Use the new FOLL_LONGTERM to get_user_pages_fast() to protect against FS DAX pages being mapped.
---
Changes from V2
Per Dan Williams: set FOLL_LONGTERM on declaration line
drivers/infiniband/hw/hfi1/user_pages.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/hw/hfi1/user_pages.c +2 -2
@@ -104,9 +104,9 @@ int hfi1_acquire_user_pages(struct mm_struct *mm, unsigned long vaddr, size_t np bool writable, struct page **pages) { int ret; +unsigned int gup_flags =FOLL_LONGTERM | (writable ?FOLL_WRITE :0 ); - ret =get_user_pages_fast (vaddr, npages, writable ?FOLL_WRITE :0 , - pages); + ret =get_user_pages_fast (vaddr, npages, gup_flags, pages);if (ret <0 )return ret;
Signature · 1 line
2.20.1
PATCH v3 6/7 IB/qib: Use the new FOLL_LONGTERM flag to get_user_pages_fast()
In reply to [email protected]From: Ira Weiny <[email protected]>
Use the new FOLL_LONGTERM to get_user_pages_fast() to protect against FS DAX pages being mapped.
---
Changes from V2:
added reviewed-by
drivers/infiniband/hw/qib/qib_user_sdma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/hw/qib/qib_user_sdma.c +1 -1
@@ -673,7 +673,7 @@ static int qib_user_sdma_pin_pages(const struct qib_devdata *dd,else j = npages; - ret =get_user_pages_fast (addr, j,0 , pages); + ret =get_user_pages_fast (addr, j,FOLL_LONGTERM , pages);if (ret != j) { i =0 ; j = ret;
Signature · 1 line
2.20.1
PATCH v3 7/7 IB/mthca: Use the new FOLL_LONGTERM flag to get_user_pages_fast()
In reply to [email protected]From: Ira Weiny <[email protected]>
Use the new FOLL_LONGTERM to get_user_pages_fast() to protect against FS DAX pages being mapped.
---
Changes from V2:
<none>
drivers/infiniband/hw/mthca/mthca_memfree.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/hw/mthca/mthca_memfree.c +2 -1
@@ -472,7 +472,8 @@ int mthca_map_user_db(struct mthca_dev *dev, struct mthca_uar *uar, goto out; } - ret =get_user_pages_fast (uaddr &PAGE_MASK ,1 ,FOLL_WRITE , pages); + ret =get_user_pages_fast (uaddr &PAGE_MASK ,1 , +FOLL_WRITE |FOLL_LONGTERM , pages);if (ret <0 ) goto out;
Signature · 1 line
2.20.1
Quoted 9 lines · Ira Weiny
From: Ira Weiny <[email protected]>
Following discussion and review[1] here are the cleanups requested.
The biggest change for V3 was the disabling of the ability to use
FOLL_LONGTERMin get_user_pages[unlocked|locked|remote]Comments were also enhanced throughout to show potential users what
FOLL_LONGTERMis all about and limitations it has.
Does anyone have any problems with these changes?
I would like to get official Reviewed-by tags if possible.
Thanks,
Ira
Quoted 80 lines
Minor review comments were fixed
Original cover letter:
HFI1, qib, and mthca, use
get_user_pages_fast()due to it performance advantages. These pages can be held for a significant time. Butget_user_pages_fast()does not protect against mapping FS DAX pages.Introduce
FOLL_LONGTERMand use this flag inget_user_pages_fast()which retains the performance while also adding the FS DAX checks. XDP has also shown interest in using this functionality.[1]In addition we change
get_user_pages()to use the newFOLL_LONGTERMflag and remove the specializedget_user_pages_longtermcall.[1] https://lkml.org/lkml/2019/3/19/939
Ira Weiny (7):
mm/gup: Replaceget_user_pages_longterm()withFOLL_LONGTERMmm/gup: Change write parameter to flags in fast walk mm/gup: Change GUP fast to use flags rather than a write 'bool' mm/gup: AddFOLL_LONGTERMcapability to GUP fast IB/hfi1: Use the newFOLL_LONGTERMflag toget_user_pages_fast()IB/qib: Use the newFOLL_LONGTERMflag toget_user_pages_fast()IB/mthca: Use the newFOLL_LONGTERMflag toget_user_pages_fast()arch/mips/mm/gup.c | 11 +- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 +- arch/powerpc/kvm/e500_mmu.c | 2 +- arch/powerpc/mm/mmu_context_iommu.c | 3 +- arch/s390/kvm/interrupt.c | 2 +- arch/s390/mm/gup.c | 12 +- arch/sh/mm/gup.c | 11 +- arch/sparc/mm/gup.c | 9 +- arch/x86/kvm/paging_tmpl.h | 2 +- arch/x86/kvm/svm.c | 2 +- drivers/fpga/dfl-afu-dma-region.c | 2 +- drivers/gpu/drm/via/via_dmablit.c | 3 +- drivers/infiniband/core/umem.c | 5 +- drivers/infiniband/hw/hfi1/user_pages.c | 3 +- drivers/infiniband/hw/mthca/mthca_memfree.c | 3 +- drivers/infiniband/hw/qib/qib_user_pages.c | 8 +- drivers/infiniband/hw/qib/qib_user_sdma.c | 2 +- drivers/infiniband/hw/usnic/usnic_uiom.c | 9 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 6 +- drivers/misc/genwqe/card_utils.c | 2 +- drivers/misc/vmw_vmci/vmci_host.c | 2 +- drivers/misc/vmw_vmci/vmci_queue_pair.c | 6 +- drivers/platform/goldfish/goldfish_pipe.c | 3 +- drivers/rapidio/devices/rio_mport_cdev.c | 4 +- drivers/sbus/char/oradax.c | 2 +- drivers/scsi/st.c | 3 +- drivers/staging/gasket/gasket_page_table.c | 4 +- drivers/tee/tee_shm.c | 2 +- drivers/vfio/vfio_iommu_spapr_tce.c | 3 +- drivers/vfio/vfio_iommu_type1.c | 3 +- drivers/vhost/vhost.c | 2 +- drivers/video/fbdev/pvr2fb.c | 2 +- drivers/virt/fsl_hypervisor.c | 2 +- drivers/xen/gntdev.c | 2 +- fs/io_uring.c | 5 +- fs/orangefs/orangefs-bufmap.c | 2 +- include/linux/mm.h | 45 ++- kernel/futex.c | 2 +- lib/iov_iter.c | 7 +- mm/gup.c | 288 +++++++++++++------- mm/gup_benchmark.c | 5 +- mm/util.c | 8 +- net/ceph/pagevec.c | 2 +- net/rds/info.c | 2 +- net/rds/rdma.c | 3 +- net/xdp/xdp_umem.c | 4 +- 46 files changed, 314 insertions(+), 200 deletions(-)Signature · 1 line
2.20.1