Add FOLL_LONGTERM to GUP fast and use it
Subsystems (43)
cover Add FOLL_LONGTERM to GUP fast and use it
From: Ira Weiny <[email protected]>
Resending after rebasing to the latest mm tree.
HFI1, qib, and mthca, use get_user_pages_fast() due to it performance advantages. These pages can be held for a significant time. But get_user_pages_fast() does not protect against mapping FS DAX pages.
Introduce FOLL_LONGTERM and use this flag in get_user_pages_fast() which retains the performance while also adding the FS DAX checks. XDP has also shown interest in using this functionality.[1]
In addition we change get_user_pages() to use the new FOLL_LONGTERM flag and remove the specialized get_user_pages_longterm call.
[1] https://lkml.org/lkml/2019/2/11/1789
Ira Weiny (7):
mm/gup: Replace get_user_pages_longterm() with FOLL_LONGTERM mm/gup: Change write parameter to flags in fast walk mm/gup: Change GUP fast to use flags rather than a write 'bool' mm/gup: Add FOLL_LONGTERM capability to GUP fast IB/hfi1: Use the new FOLL_LONGTERM flag to get_user_pages_fast() IB/qib: Use the new FOLL_LONGTERM flag to get_user_pages_fast() IB/mthca: Use the new FOLL_LONGTERM flag to get_user_pages_fast()
arch/mips/mm/gup.c | 11 +- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 +- arch/powerpc/kvm/e500_mmu.c | 2 +- arch/powerpc/mm/mmu_context_iommu.c | 3 +- arch/s390/kvm/interrupt.c | 2 +- arch/s390/mm/gup.c | 12 +- arch/sh/mm/gup.c | 11 +- arch/sparc/mm/gup.c | 9 +- arch/x86/kvm/paging_tmpl.h | 2 +- arch/x86/kvm/svm.c | 2 +- drivers/fpga/dfl-afu-dma-region.c | 2 +- drivers/gpu/drm/via/via_dmablit.c | 3 +- drivers/infiniband/core/umem.c | 5 +- drivers/infiniband/hw/hfi1/user_pages.c | 5 +- drivers/infiniband/hw/mthca/mthca_memfree.c | 3 +- drivers/infiniband/hw/qib/qib_user_pages.c | 8 +- drivers/infiniband/hw/qib/qib_user_sdma.c | 2 +- drivers/infiniband/hw/usnic/usnic_uiom.c | 9 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 6 +- drivers/misc/genwqe/card_utils.c | 2 +- drivers/misc/vmw_vmci/vmci_host.c | 2 +- drivers/misc/vmw_vmci/vmci_queue_pair.c | 6 +- drivers/platform/goldfish/goldfish_pipe.c | 3 +- drivers/rapidio/devices/rio_mport_cdev.c | 4 +- drivers/sbus/char/oradax.c | 2 +- drivers/scsi/st.c | 3 +- drivers/staging/gasket/gasket_page_table.c | 4 +- drivers/tee/tee_shm.c | 2 +- drivers/vfio/vfio_iommu_spapr_tce.c | 3 +- drivers/vfio/vfio_iommu_type1.c | 3 +- drivers/vhost/vhost.c | 2 +- drivers/video/fbdev/pvr2fb.c | 2 +- drivers/virt/fsl_hypervisor.c | 2 +- drivers/xen/gntdev.c | 2 +- fs/io_uring.c | 5 +- fs/orangefs/orangefs-bufmap.c | 2 +- include/linux/mm.h | 18 +- kernel/futex.c | 2 +- lib/iov_iter.c | 7 +- mm/gup.c | 258 ++++++++++++-------- mm/gup_benchmark.c | 5 +- mm/util.c | 8 +- net/ceph/pagevec.c | 2 +- net/rds/info.c | 2 +- net/rds/rdma.c | 3 +- net/xdp/xdp_umem.c | 4 +- 46 files changed, 262 insertions(+), 197 deletions(-)
Signature · 1 line
2.20.1
PATCH RESEND 1/7 mm/gup: Replace get_user_pages_longterm() with FOLL_LONGTERM
From: Ira Weiny <[email protected]>
Rather than have a separate get_user_pages_longterm() call, introduce FOLL_LONGTERM and change the longterm callers to use it.
This patch does not change any functionality.
FOLL_LONGTERM can only be supported with get_user_pages() as it requires vmas to determine if DAX is in use.
---
Changes from V1:
Rebased on 5.1 merge
Adjusted for changes introduced by CONFIG_CMA Convert new users of GUP longterm
io_uring.c
xdp_umem.c
arch/powerpc/mm/mmu_context_iommu.c | 3 +- drivers/infiniband/core/umem.c | 5 +- drivers/infiniband/hw/qib/qib_user_pages.c | 8 +- drivers/infiniband/hw/usnic/usnic_uiom.c | 9 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 6 +- drivers/vfio/vfio_iommu_type1.c | 3 +- fs/io_uring.c | 5 +- include/linux/mm.h | 14 +- mm/gup.c | 171 ++++++++++++--------- mm/gup_benchmark.c | 5 +- net/xdp/xdp_umem.c | 4 +- 11 files changed, 129 insertions(+), 104 deletions(-)
arch/powerpc/mm/mmu_context_iommu.c +2 -1
@@ -148,7 +148,8 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua, }down_read (&mm->mmap_sem); - ret =get_user_pages_longterm (ua, entries,FOLL_WRITE , mem->hpages,NULL ); + ret =get_user_pages (ua, entries,FOLL_WRITE |FOLL_LONGTERM , + mem->hpages,NULL );up_read (&mm->mmap_sem);if (ret != entries) {/* free the reference taken */
drivers/infiniband/core/umem.c +3 -2
@@ -189,10 +189,11 @@ struct ib_umem *ib_umem_get(struct ib_udata *udata, unsigned long addr,while (npages) {down_read (&mm->mmap_sem); - ret =get_user_pages_longterm (cur_base, + ret =get_user_pages (cur_base, min_t(unsigned long, npages,PAGE_SIZE /sizeof (struct page *)), - gup_flags, page_list, vma_list); + gup_flags |FOLL_LONGTERM , + page_list, vma_list);if (ret <0 ) {up_read (&mm->mmap_sem); goto umem_release;
drivers/infiniband/hw/qib/qib_user_pages.c +4 -4
@@ -114,10 +114,10 @@ int qib_get_user_pages(unsigned long start_page, size_t num_pages,down_read (¤t->mm->mmap_sem);for (got =0 ; got < num_pages; got += ret) { - ret =get_user_pages_longterm (start_page + got *PAGE_SIZE , - num_pages - got, -FOLL_WRITE |FOLL_FORCE , - p + got,NULL ); + ret =get_user_pages (start_page + got *PAGE_SIZE , + num_pages - got, +FOLL_LONGTERM |FOLL_WRITE |FOLL_FORCE , + p + got,NULL );if (ret <0 ) {up_read (¤t->mm->mmap_sem); goto bail_release;
drivers/infiniband/hw/usnic/usnic_uiom.c +5 -4
@@ -143,10 +143,11 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable, ret =0 ;while (npages) { - ret =get_user_pages_longterm (cur_base, - min_t(unsigned long, npages, -PAGE_SIZE /sizeof (struct page *)), - gup_flags, page_list,NULL ); + ret =get_user_pages (cur_base, + min_t(unsigned long, npages, +PAGE_SIZE /sizeof (struct page *)), + gup_flags |FOLL_LONGTERM , + page_list,NULL );if (ret <0 ) goto out;
drivers/media/v4l2-core/videobuf-dma-sg.c +3 -3
@@ -186,12 +186,12 @@ static int videobuf_dma_init_user_locked(struct videobuf_dmabuf *dma,dprintk (1 ,"init user [0x%lx+0x%lx => %d pages]\n" , data, size, dma->nr_pages); - err =get_user_pages_longterm (data &PAGE_MASK , dma->nr_pages, - flags, dma->pages,NULL ); + err =get_user_pages (data &PAGE_MASK , dma->nr_pages, + flags |FOLL_LONGTERM , dma->pages,NULL );if (err != dma->nr_pages) { dma->nr_pages = (err >=0 ) ? err :0 ; -dprintk (1 ,"get_user_pages_longterm: err=%d [%d]\n" , err, +dprintk (1 ,"get_user_pages: err=%d [%d]\n" , err, dma->nr_pages);return err <0 ? err : -EINVAL ; }
drivers/vfio/vfio_iommu_type1.c +2 -1
@@ -351,7 +351,8 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr,down_read (&mm->mmap_sem);if (mm == current->mm) { - ret =get_user_pages_longterm (vaddr,1 , flags, page, vmas); + ret =get_user_pages (vaddr,1 , flags |FOLL_LONGTERM , page, + vmas); }else { ret =get_user_pages_remote (NULL , mm, vaddr,1 , flags, page, vmas,NULL );
fs/io_uring.c +3 -2
@@ -2450,8 +2450,9 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, void __user *arg, ret =0 ;down_read (¤t->mm->mmap_sem); - pret =get_user_pages_longterm (ubuf, nr_pages,FOLL_WRITE , - pages, vmas); + pret =get_user_pages (ubuf, nr_pages, +FOLL_WRITE |FOLL_LONGTERM , + pages, vmas);if (pret == nr_pages) {/* don't support file backed memory */ for (j =0 ; j < nr_pages; j++) {
include/linux/mm.h +1 -13
@@ -1531,19 +1531,6 @@ long get_user_pages_locked(unsigned long start, unsigned long nr_pages,long get_user_pages_unlocked (unsigned long start,unsigned long nr_pages,struct page **pages,unsigned int gup_flags); -#if defined(CONFIG_FS_DAX ) || defined(CONFIG_CMA ) -long get_user_pages_longterm (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages, -struct vm_area_struct **vmas); -#else -static inline long get_user_pages_longterm (unsigned long start, -unsigned long nr_pages,unsigned int gup_flags, -struct page **pages,struct vm_area_struct **vmas) -{ -return get_user_pages (start, nr_pages, gup_flags, pages, vmas); -} -#endif /* CONFIG_FS_DAX */ -int get_user_pages_fast (unsigned long start,int nr_pages,int write,struct page **pages);
@@ -2609,6 +2596,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,#define FOLL_REMOTE 0x2000/* we are working on non-current tsk/mm */ #define FOLL_COW 0x4000/* internal GUP flag */ #define FOLL_ANON 0x8000/* don't do file mappings */ +#define FOLL_LONGTERM 0x10000/* mapping is intended for a long term pin */ static inline int vm_fault_to_errno (vm_fault_t vm_fault,int foll_flags) {
mm/gup.c +101 -70
225 lines not on this page · raw message
mm/gup_benchmark.c +3 -2
@@ -54,8 +54,9 @@ static int __gup_benchmark_ioctl(unsigned int cmd, pages + i);break ;case GUP_LONGTERM_BENCHMARK : - nr =get_user_pages_longterm (addr, nr, gup->flags &1 , - pages + i,NULL ); + nr =get_user_pages (addr, nr, + (gup->flags &1 ) |FOLL_LONGTERM , + pages + i,NULL );break ;case GUP_BENCHMARK : nr = get_user_pages(addr, nr, gup->flags & 1, pages + i,
net/xdp/xdp_umem.c +2 -2
@@ -267,8 +267,8 @@ static int xdp_umem_pin_pages(struct xdp_umem *umem)return -ENOMEM ;down_read (¤t->mm->mmap_sem); - npgs =get_user_pages_longterm (umem->address, umem->npgs, - gup_flags, &umem->pgs[0 ],NULL ); + npgs =get_user_pages (umem->address, umem->npgs, + gup_flags |FOLL_LONGTERM , &umem->pgs[0 ],NULL );up_read (¤t->mm->mmap_sem);if (npgs != umem->npgs) {
Signature · 1 line
2.20.1
Re: [RESEND 1/7] mm/gup: Replace get_user_pages_longterm() with FOLL_LONGTERM
Quoted 16 lines · On Sun, Mar 17, 2019 at 7:36 PM <[email protected]> wrote:
From: Ira Weiny <[email protected]>
Rather than have a separate
get_user_pages_longterm()call, introduceFOLL_LONGTERMand change the longterm callers to use it.This patch does not change any functionality.
FOLL_LONGTERMcan only be supported withget_user_pages()as it requires vmas to determine if DAX is in use.CC Aneesh Kumar K.V <[email protected]>CC Andrew Morton <[email protected]>CC Michal Hocko <[email protected]>Signed-off-by Ira Weiny <[email protected]>
[..]
Quoted 4 lines
include/linux/mm.h +0
-0
[..]
include/linux/mm.h struct page *follow_page(struct vm_area_struct *vma, unsigned long address, · line 2599
Quoted 5 lines
include/linux/mm.h +1
-0@@ -2609,6 +2596,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,#define FOLL_REMOTE 0x2000/* we are working on non-current tsk/mm */ #define FOLL_COW 0x4000/* internal GUP flag */ #define FOLL_ANON 0x8000/* don't do file mappings */ +#define FOLL_LONGTERM 0x10000/* mapping is intended for a long term pin */
Let's change this comment to say something like /* mapping lifetime is indefinite / at the discretion of userspace */, since "longterm is not well defined.
I think it should also include a /* FIXME: */ to say something about the havoc a long term pin might wreak on fs and mm code paths.
Quoted 68 lines
include/linux/mm.h +0
-0static inline int vm_fault_to_errno (vm_fault_t vm_fault,int foll_flags) {mm/gup.c +6
-29@@ -1112,26 +1112,7 @@ long get_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm, }EXPORT_SYMBOL (get_user_pages_remote); -/* -* This is the same as get_user_pages_remote(), just with a -* less-flexible calling convention where we assume that the task -* and mm being operated on are the current task's and don't allow -* passing of a locked parameter. We also obviously don't pass -* FOLL_REMOTE in here. -*/ -long get_user_pages (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages, -struct vm_area_struct **vmas) -{ -return __get_user_pages_locked (current, current->mm, start, nr_pages, - pages, vmas,NULL , - gup_flags |FOLL_TOUCH ); -} -EXPORT_SYMBOL (get_user_pages); -#if defined(CONFIG_FS_DAX ) || defined (CONFIG_CMA ) - -#ifdef CONFIG_FS_DAXstatic bool check_dax_vmas (struct vm_area_struct **vmas,long nr_pages) {long i;@@ -1150,12 +1131,6 @@ static bool check_dax_vmas(struct vm_area_struct **vmas, long nr_pages) }return false; } -#else -static inline bool check_dax_vmas (struct vm_area_struct **vmas,long nr_pages) -{ -return false; -} -#endif #ifdef CONFIG_CMA static struct page *new_non_cma_page (struct page *page,unsigned long private)@@ -1209,10 +1184,13 @@ static struct page *new_non_cma_page(struct page *page, unsigned long private)return __alloc_pages_node (nid, gfp_mask,0 ); } -static long check_and_migrate_cma_pages (unsigned long start,long nr_pages, -unsigned int gup_flags, +static long check_and_migrate_cma_pages (struct task_struct *tsk, +struct mm_struct *mm, +unsigned long start, +unsigned long nr_pages,struct page **pages, -struct vm_area_struct **vmas) +struct vm_area_struct **vmas, +unsigned int gup_flags) {long i;bool drain_allow = true;@@ -1268,10 +1246,14 @@ static long check_and_migrate_cma_pages(unsigned long start, long nr_pages,putback_movable_pages (&cma_page_list); }/* -* We did migrate all the pages, Try to get the page references again
mm/gup.c static long check_and_migrate_cma_pages(unsigned long start, long nr_pages, · line 1256
Quoted 10 lines
mm/gup.c +7
-2-* migrating any new CMA pages which we failed to isolate earlier. +* We did migrate all the pages, Try to get the page references +* again migrating any new CMA pages which we failed to isolate +* earlier. */ - nr_pages =get_user_pages (start, nr_pages, gup_flags, pages, vmas); + nr_pages =__get_user_pages_locked (tsk, mm, start, nr_pages, + pages, vmas,NULL , + gup_flags); +
Why did this need to change to __get_user_pages_locked?
Quoted 22 lines
mm/gup.c +7
-4if ((nr_pages >0 ) && migrate_allow) { drain_allow = true; goto check_again;@@ -1281,66 +1263,115 @@ static long check_and_migrate_cma_pages(unsigned long start, long nr_pages, return nr_pages; }#else -static inline long check_and_migrate_cma_pages (unsigned long start,long nr_pages, -unsigned int gup_flags, -struct page **pages, -struct vm_area_struct **vmas) +static long check_and_migrate_cma_pages (struct task_struct *tsk, +struct mm_struct *mm, +unsigned long start, +unsigned long nr_pages, +struct page **pages, +struct vm_area_struct **vmas, +unsigned int gup_flags) {return nr_pages; }#endif
mm/gup.c static long check_and_migrate_cma_pages(unsigned long start, long nr_pages, · line 1279
Quoted 10 lines
mm/gup.c +1
-7/* -* This is the same as get_user_pages() in that it assumes we are -* operating on the current task's mm, but it goes further to validate -* that the vmas associated with the address range are suitable for -* longterm elevated page reference counts. For example, filesystem-dax -* mappings are subject to the lifetime enforced by the filesystem and -* we need guarantees that longterm users like RDMA and V4L2 only -* establish mappings that have a kernel enforced revocation mechanism. +* __gup_longterm_locked() is a wrapper for __get_uer_pages_locked which
s/uer/user/
Quoted 5 lines
mm/gup.c +5
-0+* allows us to process the FOLL_LONGTERM flag if present. +* +* FOLL_LONGTERM Checks for either DAX VMAs or PPC CMA regions and either fails +* the pin or attempts to migrate the page as appropriate. +*
mm/gup.c static long check_and_migrate_cma_pages(unsigned long start, long nr_pages, · line 1294
Quoted 10 lines
mm/gup.c +7
-0+* In the filesystem-dax case mappings are subject to the lifetime enforced by +* the filesystem and we need guarantees that longterm users like RDMA and V4L2 +* only establish mappings that have a kernel enforced revocation mechanism. +* +* In the CMA case pages can't be pinned in a CMA region as this would +* unnecessarily fragment that region. So CMA attempts to migrate the page +* before pinning. * * "longterm" == userspace controlled elevated page count lifetime. * Contrast this to iov_iter_get_pages() usages which are transient.
Ah, here's the longterm documentation, but if I was a developer considering whether to use FOLL_LONGTERM or not I would expect to find the documentation at the flag definition site.
I think it has become more clear since get_user_pages_longterm() was initially merged that we need to warn people not to use it, or at least seriously reconsider whether they want an interface to support indefinite pins.
mm/gup.c static long check_and_migrate_cma_pages(unsigned long start, long nr_pages, · line 1296
Quoted 5 lines
mm/gup.c +1
-3*/ -long get_user_pages_longterm (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages, -struct vm_area_struct **vmas_arg) +static __always_inline long __gup_longterm_locked(struct task_struct *tsk,
...why the __always_inline?
Re: [RESEND 1/7] mm/gup: Replace get_user_pages_longterm() with FOLL_LONGTERM
Quoted 33 lines · Dan Williams
Quoted 16 lines · On Sun, Mar 17, 2019 at 7:36 PM <[email protected]> wrote:
From: Ira Weiny <[email protected]>
Rather than have a separate
get_user_pages_longterm()call, introduceFOLL_LONGTERMand change the longterm callers to use it.This patch does not change any functionality.
FOLL_LONGTERMcan only be supported withget_user_pages()as it requires vmas to determine if DAX is in use.CC Aneesh Kumar K.V <[email protected]>CC Andrew Morton <[email protected]>CC Michal Hocko <[email protected]>Signed-off-by Ira Weiny <[email protected]>[..]
Quoted 4 lines
include/linux/mm.h +0
-0[..]
include/linux/mm.h struct page *follow_page(struct vm_area_struct *vma, unsigned long address, · line 2599
Quoted 5 lines
include/linux/mm.h +1
-0@@ -2609,6 +2596,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,#define FOLL_REMOTE 0x2000/* we are working on non-current tsk/mm */ #define FOLL_COW 0x4000/* internal GUP flag */ #define FOLL_ANON 0x8000/* don't do file mappings */ +#define FOLL_LONGTERM 0x10000/* mapping is intended for a long term pin */ Let's change this comment to say something like /* mapping lifetime is indefinite / at the discretion of userspace */, since "longterm is not well defined.
Quoted 2 lines
I think it should also include a /* FIXME: */ to say something about the havoc a long term pin might wreak on fs and mm code paths.
Will do.
Quoted 73 lines
Quoted 73 lines
include/linux/mm.h +0
-0static inline int vm_fault_to_errno (vm_fault_t vm_fault,int foll_flags) {mm/gup.c +9
-30@@ -1112,26 +1112,7 @@ long get_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm, }EXPORT_SYMBOL (get_user_pages_remote); -/* -* This is the same as get_user_pages_remote(), just with a -* less-flexible calling convention where we assume that the task -* and mm being operated on are the current task's and don't allow -* passing of a locked parameter. We also obviously don't pass -* FOLL_REMOTE in here. -*/ -long get_user_pages (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages, -struct vm_area_struct **vmas) -{ -return __get_user_pages_locked (current, current->mm, start, nr_pages, - pages, vmas,NULL , - gup_flags |FOLL_TOUCH ); -} -EXPORT_SYMBOL (get_user_pages); -#if defined(CONFIG_FS_DAX ) || defined (CONFIG_CMA ) - -#ifdef CONFIG_FS_DAXstatic bool check_dax_vmas (struct vm_area_struct **vmas,long nr_pages) {long i;@@ -1150,12 +1131,6 @@ static bool check_dax_vmas(struct vm_area_struct **vmas, long nr_pages) }return false; } -#else -static inline bool check_dax_vmas (struct vm_area_struct **vmas,long nr_pages) -{ -return false; -} -#endif #ifdef CONFIG_CMA static struct page *new_non_cma_page (struct page *page,unsigned long private)@@ -1209,10 +1184,13 @@ static struct page *new_non_cma_page(struct page *page, unsigned long private)return __alloc_pages_node (nid, gfp_mask,0 ); } -static long check_and_migrate_cma_pages (unsigned long start,long nr_pages, -unsigned int gup_flags, +static long check_and_migrate_cma_pages (struct task_struct *tsk, +struct mm_struct *mm, +unsigned long start, +unsigned long nr_pages,struct page **pages, -struct vm_area_struct **vmas) +struct vm_area_struct **vmas, +unsigned int gup_flags) {long i;bool drain_allow = true;@@ -1268,10 +1246,14 @@ static long check_and_migrate_cma_pages(unsigned long start, long nr_pages,putback_movable_pages (&cma_page_list); }/* -* We did migrate all the pages, Try to get the page references again -* migrating any new CMA pages which we failed to isolate earlier. +* We did migrate all the pages, Try to get the page references +* again migrating any new CMA pages which we failed to isolate +* earlier.
Quoted 8 lines
mm/gup.c static long check_and_migrate_cma_pages(unsigned long start, long nr_pages, · line 1256
Quoted 6 lines
mm/gup.c +4
-1*/ - nr_pages =get_user_pages (start, nr_pages, gup_flags, pages, vmas); + nr_pages =__get_user_pages_locked (tsk, mm, start, nr_pages, + pages, vmas,NULL , + gup_flags); +Why did this need to change to
__get_user_pages_locked?
__get_uer_pages_locked() is now the "internal call" for get_user_pages.
Technically it did not _have_ to change but there is no need to call get_user_pages() again because the FOLL_TOUCH flags is already set. Also this call then matches the __get_user_pages_locked() which was called on the pages we migrated from. Mostly this keeps the code "symmetrical" in that we called __get_user_pages_locked() on the pages we are migrating from and the same call on the pages we migrated to.
While the change here looks funny I think the final code is better.
Quoted 24 lines
Quoted 24 lines
mm/gup.c +7
-4if ((nr_pages >0 ) && migrate_allow) { drain_allow = true; goto check_again;@@ -1281,66 +1263,115 @@ static long check_and_migrate_cma_pages(unsigned long start, long nr_pages, return nr_pages; }#else -static inline long check_and_migrate_cma_pages (unsigned long start,long nr_pages, -unsigned int gup_flags, -struct page **pages, -struct vm_area_struct **vmas) +static long check_and_migrate_cma_pages (struct task_struct *tsk, +struct mm_struct *mm, +unsigned long start, +unsigned long nr_pages, +struct page **pages, +struct vm_area_struct **vmas, +unsigned int gup_flags) {return nr_pages; }#endif
Quoted 11 lines
mm/gup.c static long check_and_migrate_cma_pages(unsigned long start, long nr_pages, · line 1279
Quoted 9 lines
mm/gup.c +1
-7/* -* This is the same as get_user_pages() in that it assumes we are -* operating on the current task's mm, but it goes further to validate -* that the vmas associated with the address range are suitable for -* longterm elevated page reference counts. For example, filesystem-dax -* mappings are subject to the lifetime enforced by the filesystem and -* we need guarantees that longterm users like RDMA and V4L2 only -* establish mappings that have a kernel enforced revocation mechanism. +* __gup_longterm_locked() is a wrapper for __get_uer_pages_locked which s/uer/user/
Check
Quoted 17 lines
Quoted 6 lines
mm/gup.c +5
-0+* allows us to process the FOLL_LONGTERM flag if present. +* +* FOLL_LONGTERM Checks for either DAX VMAs or PPC CMA regions and either fails +* the pin or attempts to migrate the page as appropriate. +* mm/gup.c static long check_and_migrate_cma_pages(unsigned long start, long nr_pages, · line 1294
Quoted 10 lines
mm/gup.c +7
-0+* In the filesystem-dax case mappings are subject to the lifetime enforced by +* the filesystem and we need guarantees that longterm users like RDMA and V4L2 +* only establish mappings that have a kernel enforced revocation mechanism. +* +* In the CMA case pages can't be pinned in a CMA region as this would +* unnecessarily fragment that region. So CMA attempts to migrate the page +* before pinning. * * "longterm" == userspace controlled elevated page count lifetime. * Contrast this to iov_iter_get_pages() usages which are transient.
Quoted 8 lines
Ah, here's the longterm documentation, but if I was a developer considering whether to use
FOLL_LONGTERMor not I would expect to find the documentation at the flag definition site.I think it has become more clear since
get_user_pages_longterm()was initially merged that we need to warn people not to use it, or at least seriously reconsider whether they want an interface to support indefinite pins.
Good point will move
Quoted 8 lines
mm/gup.c static long check_and_migrate_cma_pages(unsigned long start, long nr_pages, · line 1296
Quoted 6 lines
mm/gup.c +1
-3*/ -long get_user_pages_longterm (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages, -struct vm_area_struct **vmas_arg) +static __always_inline long __gup_longterm_locked(struct task_struct *tsk,...why the
__always_inline?
This was because it was only called from get_user_pages() in this patch. But later on I use it elsewhere so __always_inline is wrong.
Ira
Re: [RESEND 1/7] mm/gup: Replace get_user_pages_longterm() with FOLL_LONGTERM
On Mon, Mar 25, 2019 at 7:21 AM Ira Weiny <[email protected]> wrote:
[..]
Quoted 20 lines
Quoted 9 lines
Quoted 9 lines
mm/gup.c +3
-2@@ -1268,10 +1246,14 @@ static long check_and_migrate_cma_pages(unsigned long start, long nr_pages,putback_movable_pages (&cma_page_list); }/* -* We did migrate all the pages, Try to get the page references again -* migrating any new CMA pages which we failed to isolate earlier. +* We did migrate all the pages, Try to get the page references +* again migrating any new CMA pages which we failed to isolate +* earlier. Quoted 8 lines
mm/gup.c static long check_and_migrate_cma_pages(unsigned long start, long nr_pages, · line 1256
Quoted 6 lines
mm/gup.c +4
-1*/ - nr_pages =get_user_pages (start, nr_pages, gup_flags, pages, vmas); + nr_pages =__get_user_pages_locked (tsk, mm, start, nr_pages, + pages, vmas,NULL , + gup_flags); +Why did this need to change to
__get_user_pages_locked?
__get_uer_pages_locked()is now the "internal call" forget_user_pages.
Quoted 8 lines
Technically it did not _have_ to change but there is no need to call
get_user_pages()again because theFOLL_TOUCHflags is already set. Also this call then matches the__get_user_pages_locked()which was called on the pages we migrated from. Mostly this keeps the code "symmetrical" in that we called__get_user_pages_locked()on the pages we are migrating from and the same call on the pages we migrated to.While the change here looks funny I think the final code is better.
Agree, but I think that either needs to be noted in the changelog so it's not a surprise, or moved to a follow-on cleanup patch.
Re: [RESEND 1/7] mm/gup: Replace get_user_pages_longterm() with FOLL_LONGTERM
Quoted 29 lines · Dan Williams
On Mon, Mar 25, 2019 at 7:21 AM Ira Weiny <[email protected]> wrote:
[..]Quoted 27 lines
Quoted 9 lines
Quoted 9 lines
mm/gup.c +3
-2@@ -1268,10 +1246,14 @@ static long check_and_migrate_cma_pages(unsigned long start, long nr_pages,putback_movable_pages (&cma_page_list); }/* -* We did migrate all the pages, Try to get the page references again -* migrating any new CMA pages which we failed to isolate earlier. +* We did migrate all the pages, Try to get the page references +* again migrating any new CMA pages which we failed to isolate +* earlier. Quoted 8 lines
mm/gup.c static long check_and_migrate_cma_pages(unsigned long start, long nr_pages, · line 1256
Quoted 6 lines
mm/gup.c +4
-1*/ - nr_pages =get_user_pages (start, nr_pages, gup_flags, pages, vmas); + nr_pages =__get_user_pages_locked (tsk, mm, start, nr_pages, + pages, vmas,NULL , + gup_flags); +Why did this need to change to
__get_user_pages_locked?
__get_uer_pages_locked()is now the "internal call" forget_user_pages.Technically it did not _have_ to change but there is no need to call
get_user_pages()again because theFOLL_TOUCHflags is already set. Also this call then matches the__get_user_pages_locked()which was called on the pages we migrated from. Mostly this keeps the code "symmetrical" in that we called__get_user_pages_locked()on the pages we are migrating from and the same call on the pages we migrated to.
Quoted 5 lines
Quoted 1 line
While the change here looks funny I think the final code is better.
Agree, but I think that either needs to be noted in the changelog so it's not a surprise, or moved to a follow-on cleanup patch.
Can do...
Thanks!
Ira
Re: [RESEND 1/7] mm/gup: Replace get_user_pages_longterm() with FOLL_LONGTERM
In reply to Dan WilliamsQuoted 1 line · Dan Williams
On Sun, Mar 17, 2019 at 7:36 PM <[email protected]> wrote:
[snip]
Quoted 20 lines
Quoted 1 line
+ *
__gup_longterm_locked()is a wrapper for__get_uer_pages_lockedwhichs/uer/user/
Quoted 7 lines
+ * allows us to process the
FOLL_LONGTERMflag if present.
+ *
+ *FOLL_LONGTERMChecks for either DAX VMAs or PPC CMA regions and either fails
+ * the pin or attempts to migrate the page as appropriate.
+ *
+ * In the filesystem-dax case mappings are subject to the lifetime enforced by
+ * the filesystem and we need guarantees that longterm users like RDMA and V4L2Quoted 8 lines
+ * only establish mappings that have a kernel enforced revocation mechanism.
+ *
+ * In the CMA case pages can't be pinned in a CMA region as this would
+ * unnecessarily fragment that region. So CMA attempts to migrate the page
+ * before pinning.
*
* "longterm" == userspace controlled elevated page count lifetime.
* Contrast this toiov_iter_get_pages()usages which are transient.
Quoted 8 lines
Ah, here's the longterm documentation, but if I was a developer considering whether to use
FOLL_LONGTERMor not I would expect to find the documentation at the flag definition site.I think it has become more clear since
get_user_pages_longterm()was initially merged that we need to warn people not to use it, or at least seriously reconsider whether they want an interface to support indefinite pins.
I will move the comment to the flag definition but...
In reviewing this comment it occurs to me that the addition of special casing CMA regions via FOLL_LONGTERM has made it less experimental/temporary and now simply implies intent to the GUP code as to the use of the pages.
As I'm not super familiar with the CMA use case I can't say for certain but it seems that it is not a temporary solution.
So I'm not going to refrain from a FIXME WRT removing the flag.
New suggested text below.
include/linux/mm.h +22 -1
@@ -2596,7 +2596,28 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address,#define FOLL_REMOTE 0x2000/* we are working on non-current tsk/mm */ #define FOLL_COW 0x4000/* internal GUP flag */ #define FOLL_ANON 0x8000/* don't do file mappings */ -#define FOLL_LONGTERM 0x10000/* mapping is intended for a long term pin */ +#define FOLL_LONGTERM 0x10000/* mapping lifetime is indefinite: see below */ + +/* +* NOTE on FOLL_LONGTERM: +* +* FOLL_LONGTERM indicates that the page will be held for an indefinite time +* period _often_ under userspace control. This is contrasted with +* iov_iter_get_pages() where usages which are transient. +* +* FIXME: For pages which are part of a filesystem, mappings are subject to the +* lifetime enforced by the filesystem and we need guarantees that longterm +* users like RDMA and V4L2 only establish mappings which coordinate usage with +* the filesystem. Ideas for this coordination include revoking the longterm +* pin, delaying writeback, bounce buffer page writeback, etc. As FS DAX was +* added after the problem with filesystems was found FS DAX VMAs are +* specifically failed. Filesystem pages are still subject to bugs and use of +* FOLL_LONGTERM should be avoided on those pages. +* +* In the CMA case: longterm pins in a CMA region would unnecessarily fragment +* that region. And so CMA attempts to migrate the page before pinning when +* FOLL_LONGTERM is specified. +*/ static inline int vm_fault_to_errno (vm_fault_t vm_fault,int foll_flags) {
PATCH RESEND 2/7 mm/gup: Change write parameter to flags in fast walk
In reply to [email protected]From: Ira Weiny <[email protected]>
In order to support more options in the GUP fast walk, change the write parameter to flags throughout the call stack.
This patch does not change functionality and passes FOLL_WRITE where write was previously used.
--- mm/gup.c | 52 ++++++++++++++++++++++++++-------------------------- 1 file changed, 26 insertions(+), 26 deletions(-)
mm/gup.c +26 -26
@@ -1578,7 +1578,7 @@ static void undo_dev_pagemap(int *nr, int nr_start, struct page **pages)#ifdef CONFIG_ARCH_HAS_PTE_SPECIAL static int gup_pte_range (pmd_t pmd,unsigned long addr,unsigned long end, -int write,struct page **pages,int *nr) +unsigned int flags,struct page **pages,int *nr) {struct dev_pagemap *pgmap =NULL ;int nr_start = *nr, ret =0 ;
@@ -1596,7 +1596,7 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,if (pte_protnone (pte)) goto pte_unmap; -if (!pte_access_permitted (pte, write)) +if (!pte_access_permitted (pte, flags &FOLL_WRITE )) goto pte_unmap;if (pte_devmap (pte)) {
@@ -1648,7 +1648,7 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,* useful to have gup_huge_pmd even if we can't operate on ptes. */ static int gup_pte_range (pmd_t pmd,unsigned long addr,unsigned long end, -int write,struct page **pages,int *nr) +unsigned int flags,struct page **pages,int *nr) {return 0 ; }
@@ -1731,12 +1731,12 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr,#endif static int gup_huge_pmd (pmd_t orig,pmd_t *pmdp,unsigned long addr, -unsigned long end,int write,struct page **pages,int *nr) +unsigned long end,unsigned int flags,struct page **pages,int *nr) {struct page *head, *page;int refs; -if (!pmd_access_permitted (orig, write)) +if (!pmd_access_permitted (orig, flags &FOLL_WRITE ))return 0 ;if (pmd_devmap (orig))
@@ -1769,12 +1769,12 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, }static int gup_huge_pud (pud_t orig,pud_t *pudp,unsigned long addr, -unsigned long end,int write,struct page **pages,int *nr) +unsigned long end,unsigned int flags,struct page **pages,int *nr) {struct page *head, *page;int refs; -if (!pud_access_permitted (orig, write)) +if (!pud_access_permitted (orig, flags &FOLL_WRITE ))return 0 ;if (pud_devmap (orig))
@@ -1807,13 +1807,13 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, }static int gup_huge_pgd (pgd_t orig,pgd_t *pgdp,unsigned long addr, -unsigned long end,int write, +unsigned long end,unsigned int flags,struct page **pages,int *nr) {int refs;struct page *head, *page; -if (!pgd_access_permitted (orig, write)) +if (!pgd_access_permitted (orig, flags &FOLL_WRITE ))return 0 ;BUILD_BUG_ON (pgd_devmap (orig));
@@ -1844,7 +1844,7 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, }static int gup_pmd_range (pud_t pud,unsigned long addr,unsigned long end, -int write,struct page **pages,int *nr) +unsigned int flags,struct page **pages,int *nr) {unsigned long next;pmd_t *pmdp;
@@ -1867,7 +1867,7 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,if (pmd_protnone (pmd))return 0 ; -if (!gup_huge_pmd (pmd, pmdp, addr, next, write, +if (!gup_huge_pmd (pmd, pmdp, addr, next, flags, pages, nr))return 0 ;
@@ -1877,9 +1877,9 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,* pmd format and THP pmd format */ if (!gup_huge_pd (__hugepd (pmd_val (pmd)), addr, -PMD_SHIFT , next, write, pages, nr)) +PMD_SHIFT , next, flags, pages, nr))return 0 ; - }else if (!gup_pte_range (pmd, addr, next, write, pages, nr)) + }else if (!gup_pte_range (pmd, addr, next, flags, pages, nr))return 0 ; }while (pmdp++, addr = next, addr != end);
@@ -1887,7 +1887,7 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, }static int gup_pud_range (p4d_t p4d,unsigned long addr,unsigned long end, -int write,struct page **pages,int *nr) +unsigned int flags,struct page **pages,int *nr) {unsigned long next;pud_t *pudp;
@@ -1900,14 +1900,14 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end,if (pud_none (pud))return 0 ;if (unlikely (pud_huge (pud))) { -if (!gup_huge_pud (pud, pudp, addr, next, write, +if (!gup_huge_pud (pud, pudp, addr, next, flags, pages, nr))return 0 ; }else if (unlikely (is_hugepd (__hugepd (pud_val (pud))))) {if (!gup_huge_pd (__hugepd (pud_val (pud)), addr, -PUD_SHIFT , next, write, pages, nr)) +PUD_SHIFT , next, flags, pages, nr))return 0 ; - }else if (!gup_pmd_range (pud, addr, next, write, pages, nr)) + }else if (!gup_pmd_range (pud, addr, next, flags, pages, nr))return 0 ; }while (pudp++, addr = next, addr != end);
@@ -1915,7 +1915,7 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, }static int gup_p4d_range (pgd_t pgd,unsigned long addr,unsigned long end, -int write,struct page **pages,int *nr) +unsigned int flags,struct page **pages,int *nr) {unsigned long next;p4d_t *p4dp;
@@ -1930,9 +1930,9 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end,BUILD_BUG_ON (p4d_huge (p4d));if (unlikely (is_hugepd (__hugepd (p4d_val (p4d))))) {if (!gup_huge_pd (__hugepd (p4d_val (p4d)), addr, -P4D_SHIFT , next, write, pages, nr)) +P4D_SHIFT , next, flags, pages, nr))return 0 ; - }else if (!gup_pud_range (p4d, addr, next, write, pages, nr)) + }else if (!gup_pud_range (p4d, addr, next, flags, pages, nr))return 0 ; }while (p4dp++, addr = next, addr != end);
@@ -1940,7 +1940,7 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, }static void gup_pgd_range (unsigned long addr,unsigned long end, -int write,struct page **pages,int *nr) +unsigned int flags,struct page **pages,int *nr) {unsigned long next;pgd_t *pgdp;
@@ -1953,14 +1953,14 @@ static void gup_pgd_range(unsigned long addr, unsigned long end,if (pgd_none (pgd))return ;if (unlikely (pgd_huge (pgd))) { -if (!gup_huge_pgd (pgd, pgdp, addr, next, write, +if (!gup_huge_pgd (pgd, pgdp, addr, next, flags, pages, nr))return ; }else if (unlikely (is_hugepd (__hugepd (pgd_val (pgd))))) {if (!gup_huge_pd (__hugepd (pgd_val (pgd)), addr, -PGDIR_SHIFT , next, write, pages, nr)) +PGDIR_SHIFT , next, flags, pages, nr))return ; - }else if (!gup_p4d_range (pgd, addr, next, write, pages, nr)) + }else if (!gup_p4d_range (pgd, addr, next, flags, pages, nr))return ; }while (pgdp++, addr = next, addr != end); }
@@ -2014,7 +2014,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,if (gup_fast_permitted (start, nr_pages)) {local_irq_save (flags); -gup_pgd_range (start, end, write, pages, &nr); +gup_pgd_range (start, end, write ?FOLL_WRITE :0 , pages, &nr);local_irq_restore (flags); }
@@ -2056,7 +2056,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write,if (gup_fast_permitted (start, nr_pages)) {local_irq_disable (); -gup_pgd_range (addr, end, write, pages, &nr); +gup_pgd_range (addr, end, write ?FOLL_WRITE :0 , pages, &nr);local_irq_enable (); ret = nr; }
Signature · 1 line
2.20.1
Re: [RESEND 2/7] mm/gup: Change write parameter to flags in fast walk
Quoted 10 lines · On Sun, Mar 17, 2019 at 7:36 PM <[email protected]> wrote:
From: Ira Weiny <[email protected]>
In order to support more options in the GUP fast walk, change the write parameter to flags throughout the call stack.
This patch does not change functionality and passes
FOLL_WRITEwhere write was previously used.Signed-off-by Ira Weiny <[email protected]>
Looks good,
PATCH RESEND 3/7 mm/gup: Change GUP fast to use flags rather than a write 'bool'
In reply to [email protected]From: Ira Weiny <[email protected]>
To facilitate additional options to get_user_pages_fast() change the singular write parameter to be gup_flags.
This patch does not change any functionality. New functionality will follow in subsequent patches.
Some of the get_user_pages_fast() call sites were unchanged because they already passed FOLL_WRITE or 0 for the write parameter.
---
Changes from V1:
Rebase to current merge tree arch/powerpc/mm/mmu_context_iommu.c no longer calls gup_fast
The gup_longterm was converted in patch 1
arch/mips/mm/gup.c | 11 ++++++----- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 ++-- arch/powerpc/kvm/e500_mmu.c | 2 +- arch/s390/kvm/interrupt.c | 2 +- arch/s390/mm/gup.c | 12 ++++++------ arch/sh/mm/gup.c | 11 ++++++----- arch/sparc/mm/gup.c | 9 +++++---- arch/x86/kvm/paging_tmpl.h | 2 +- arch/x86/kvm/svm.c | 2 +- drivers/fpga/dfl-afu-dma-region.c | 2 +- drivers/gpu/drm/via/via_dmablit.c | 3 ++- drivers/infiniband/hw/hfi1/user_pages.c | 3 ++- drivers/misc/genwqe/card_utils.c | 2 +- drivers/misc/vmw_vmci/vmci_host.c | 2 +- drivers/misc/vmw_vmci/vmci_queue_pair.c | 6 ++++-- drivers/platform/goldfish/goldfish_pipe.c | 3 ++- drivers/rapidio/devices/rio_mport_cdev.c | 4 +++- drivers/sbus/char/oradax.c | 2 +- drivers/scsi/st.c | 3 ++- drivers/staging/gasket/gasket_page_table.c | 4 ++-- drivers/tee/tee_shm.c | 2 +- drivers/vfio/vfio_iommu_spapr_tce.c | 3 ++- drivers/vhost/vhost.c | 2 +- drivers/video/fbdev/pvr2fb.c | 2 +- drivers/virt/fsl_hypervisor.c | 2 +- drivers/xen/gntdev.c | 2 +- fs/orangefs/orangefs-bufmap.c | 2 +- include/linux/mm.h | 4 ++-- kernel/futex.c | 2 +- lib/iov_iter.c | 7 +++++-- mm/gup.c | 10 +++++----- mm/util.c | 8 ++++---- net/ceph/pagevec.c | 2 +- net/rds/info.c | 2 +- net/rds/rdma.c | 3 ++- 35 files changed, 79 insertions(+), 63 deletions(-)
arch/mips/mm/gup.c +6 -5
@@ -235,7 +235,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* get_user_pages_fast() - pin user pages in memory * @start: starting user address * @nr_pages: number of pages from start to pin -* @write: whether pages will be written to +* @gup_flags: flags modifying pin behaviour * @pages: array that receives pointers to the pages pinned. * Should be at least nr_pages long. *
@@ -247,8 +247,8 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* requested. If nr_pages is 0 or negative, returns 0. If no pages * were pinned, returns -errno. */ -int get_user_pages_fast (unsigned long start,int nr_pages,int write, -struct page **pages) +int get_user_pages_fast (unsigned long start,int nr_pages, +unsigned int gup_flags,struct page **pages) {struct mm_struct *mm = current->mm;unsigned long addr, len, end;
@@ -273,7 +273,8 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write, next =pgd_addr_end (addr, end);if (pgd_none (pgd)) goto slow; -if (!gup_pud_range (pgd, addr, next, write, pages, &nr)) +if (!gup_pud_range (pgd, addr, next, gup_flags &FOLL_WRITE , + pages, &nr)) goto slow; }while (pgdp++, addr = next, addr != end);local_irq_enable ();
@@ -289,7 +290,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write, pages += nr; ret =get_user_pages_unlocked (start, (end - start) >>PAGE_SHIFT , - pages, write ?FOLL_WRITE :0 ); + pages, gup_flags);/* Have to be a bit careful with return values */ if (nr >0 ) {
arch/powerpc/kvm/book3s_64_mmu_hv.c +2 -2
@@ -600,7 +600,7 @@ int kvmppc_book3s_hv_page_fault(struct kvm_run *run, struct kvm_vcpu *vcpu,/* If writing != 0, then the HPTE must allow writing, if we get here */ write_ok = writing; hva =gfn_to_hva_memslot (memslot, gfn); - npages =get_user_pages_fast (hva,1 , writing, pages); + npages =get_user_pages_fast (hva,1 , writing ?FOLL_WRITE :0 , pages);if (npages <1 ) {/* Check if it's an I/O mapping */ down_read (¤t->mm->mmap_sem);
@@ -1193,7 +1193,7 @@ void *kvmppc_pin_guest_page(struct kvm *kvm, unsigned long gpa,if (!memslot || (memslot->flags &KVM_MEMSLOT_INVALID )) goto err; hva =gfn_to_hva_memslot (memslot, gfn); - npages =get_user_pages_fast (hva,1 ,1 , pages); + npages =get_user_pages_fast (hva,1 ,FOLL_WRITE , pages);if (npages <1 ) goto err; page = pages[0 ];
arch/powerpc/kvm/e500_mmu.c +1 -1
@@ -783,7 +783,7 @@ int kvm_vcpu_ioctl_config_tlb(struct kvm_vcpu *vcpu,if (!pages)return -ENOMEM ; - ret =get_user_pages_fast (cfg->array, num_pages,1 , pages); + ret =get_user_pages_fast (cfg->array, num_pages,FOLL_WRITE , pages);if (ret <0 ) goto free_pages;
arch/s390/kvm/interrupt.c +1 -1
@@ -2376,7 +2376,7 @@ static int kvm_s390_adapter_map(struct kvm *kvm, unsigned int id, __u64 addr) ret = -EFAULT ; goto out; } - ret =get_user_pages_fast (map->addr,1 ,1 , &map->page); + ret =get_user_pages_fast (map->addr,1 ,FOLL_WRITE , &map->page);if (ret <0 ) goto out;BUG_ON (ret !=1 );
arch/s390/mm/gup.c +6 -6
@@ -265,7 +265,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* get_user_pages_fast() - pin user pages in memory * @start: starting user address * @nr_pages: number of pages from start to pin -* @write: whether pages will be written to +* @gup_flags: flags modifying pin behaviour * @pages: array that receives pointers to the pages pinned. * Should be at least nr_pages long. *
@@ -277,22 +277,22 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* requested. If nr_pages is 0 or negative, returns 0. If no pages * were pinned, returns -errno. */ -int get_user_pages_fast (unsigned long start,int nr_pages,int write, -struct page **pages) +int get_user_pages_fast (unsigned long start,int nr_pages, +unsigned int gup_flags,struct page **pages) {int nr, ret;might_sleep (); start &=PAGE_MASK ; - nr =__get_user_pages_fast (start, nr_pages, write, pages); + nr =__get_user_pages_fast (start, nr_pages, gup_flags &FOLL_WRITE , + pages);if (nr == nr_pages)return nr;/* Try to get the remaining pages with get_user_pages */ start += nr <<PAGE_SHIFT ; pages += nr; - ret =get_user_pages_unlocked (start, nr_pages - nr, pages, - write ?FOLL_WRITE :0 ); + ret =get_user_pages_unlocked (start, nr_pages - nr, pages, gup_flags);/* Have to be a bit careful with return values */ if (nr >0 ) ret = (ret <0 ) ? nr : ret + nr;
arch/sh/mm/gup.c +6 -5
@@ -204,7 +204,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* get_user_pages_fast() - pin user pages in memory * @start: starting user address * @nr_pages: number of pages from start to pin -* @write: whether pages will be written to +* @gup_flags: flags modifying pin behaviour * @pages: array that receives pointers to the pages pinned. * Should be at least nr_pages long. *
@@ -216,8 +216,8 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* requested. If nr_pages is 0 or negative, returns 0. If no pages * were pinned, returns -errno. */ -int get_user_pages_fast (unsigned long start,int nr_pages,int write, -struct page **pages) +int get_user_pages_fast (unsigned long start,int nr_pages, +unsigned int gup_flags,struct page **pages) {struct mm_struct *mm = current->mm;unsigned long addr, len, end;
@@ -241,7 +241,8 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write, next =pgd_addr_end (addr, end);if (pgd_none (pgd)) goto slow; -if (!gup_pud_range (pgd, addr, next, write, pages, &nr)) +if (!gup_pud_range (pgd, addr, next, gup_flags &FOLL_WRITE , + pages, &nr)) goto slow; }while (pgdp++, addr = next, addr != end);local_irq_enable ();
@@ -261,7 +262,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write, ret =get_user_pages_unlocked (start, (end - start) >>PAGE_SHIFT , pages, - write ?FOLL_WRITE :0 ); + gup_flags);/* Have to be a bit careful with return values */ if (nr >0 ) {
arch/sparc/mm/gup.c +5 -4
@@ -245,8 +245,8 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,return nr; } -int get_user_pages_fast (unsigned long start,int nr_pages,int write, -struct page **pages) +int get_user_pages_fast (unsigned long start,int nr_pages, +unsigned int gup_flags,struct page **pages) {struct mm_struct *mm = current->mm;unsigned long addr, len, end;
@@ -303,7 +303,8 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write, next =pgd_addr_end (addr, end);if (pgd_none (pgd)) goto slow; -if (!gup_pud_range (pgd, addr, next, write, pages, &nr)) +if (!gup_pud_range (pgd, addr, next, gup_flags &FOLL_WRITE , + pages, &nr)) goto slow; }while (pgdp++, addr = next, addr != end);
@@ -324,7 +325,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write, ret =get_user_pages_unlocked (start, (end - start) >>PAGE_SHIFT , pages, - write ?FOLL_WRITE :0 ); + gup_flags);/* Have to be a bit careful with return values */ if (nr >0 ) {
arch/x86/kvm/paging_tmpl.h +1 -1
@@ -140,7 +140,7 @@ static int FNAME(cmpxchg_gpte)(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,pt_element_t *table;struct page *page; - npages =get_user_pages_fast ((unsigned long )ptep_user,1 ,1 , &page); + npages =get_user_pages_fast ((unsigned long )ptep_user,1 ,FOLL_WRITE , &page);/* Check if the user is doing something meaningless. */ if (unlikely (npages !=1 ))return -EFAULT ;
arch/x86/kvm/svm.c +1 -1
@@ -1804,7 +1804,7 @@ static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr,return NULL ;/* Pin the user virtual address. */ - npinned =get_user_pages_fast (uaddr, npages, write ?FOLL_WRITE :0 , pages); + npinned =get_user_pages_fast (uaddr, npages,FOLL_WRITE , pages);if (npinned != npages) {pr_err ("SEV: Failure locking %lu pages.\n" , npages); goto err;
drivers/fpga/dfl-afu-dma-region.c +1 -1
@@ -102,7 +102,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata, goto unlock_vm; } - pinned =get_user_pages_fast (region->user_addr, npages,1 , + pinned =get_user_pages_fast (region->user_addr, npages,FOLL_WRITE , region->pages);if (pinned <0 ) { ret = pinned;
drivers/gpu/drm/via/via_dmablit.c +2 -1
@@ -243,7 +243,8 @@ via_lock_all_dma_pages(drm_via_sg_info_t *vsg, drm_via_dmablit_t *xfer)if (NULL == vsg->pages)return -ENOMEM ; ret =get_user_pages_fast ((unsigned long )xfer->mem_addr, - vsg->num_pages, vsg->direction ==DMA_FROM_DEVICE , + vsg->num_pages, + vsg->direction ==DMA_FROM_DEVICE ?FOLL_WRITE :0 , vsg->pages);if (ret != vsg->num_pages) {if (ret <0 )
drivers/infiniband/hw/hfi1/user_pages.c +2 -1
@@ -105,7 +105,8 @@ int hfi1_acquire_user_pages(struct mm_struct *mm, unsigned long vaddr, size_t np {int ret; - ret =get_user_pages_fast (vaddr, npages, writable, pages); + ret =get_user_pages_fast (vaddr, npages, writable ?FOLL_WRITE :0 , + pages);if (ret <0 )return ret;
drivers/misc/genwqe/card_utils.c +1 -1
@@ -603,7 +603,7 @@ int genwqe_user_vmap(struct genwqe_dev *cd, struct dma_mapping *m, void *uaddr,/* pin user pages in memory */ rc =get_user_pages_fast (data &PAGE_MASK ,/* page aligned addr */ m->nr_pages, - m->write,/* readable/writable */ + m->write ?FOLL_WRITE :0 ,/* readable/writable */ m->page_list);/* ptrs to pages */ if (rc <0 ) goto fail_get_user_pages;
drivers/misc/vmw_vmci/vmci_host.c +1 -1
@@ -242,7 +242,7 @@ static int vmci_host_setup_notify(struct vmci_ctx *context,/* * Lock physical page backing a given user VA. */ - retval =get_user_pages_fast (uva,1 ,1 , &context->notify_page); + retval =get_user_pages_fast (uva,1 ,FOLL_WRITE , &context->notify_page);if (retval !=1 ) { context->notify_page =NULL ;return VMCI_ERROR_GENERIC ;
drivers/misc/vmw_vmci/vmci_queue_pair.c +4 -2
@@ -659,7 +659,8 @@ static int qp_host_get_user_memory(u64 produce_uva,int err =VMCI_SUCCESS ; retval =get_user_pages_fast ((uintptr_t ) produce_uva, - produce_q->kernel_if->num_pages,1 , + produce_q->kernel_if->num_pages, +FOLL_WRITE , produce_q->kernel_if->u.h.header_page);if (retval < (int )produce_q->kernel_if->num_pages) { pr_debug("get_user_pages_fast(produce) failed (retval=%d)",
@@ -671,7 +672,8 @@ static int qp_host_get_user_memory(u64 produce_uva, } retval =get_user_pages_fast ((uintptr_t ) consume_uva, - consume_q->kernel_if->num_pages,1 , + consume_q->kernel_if->num_pages, +FOLL_WRITE , consume_q->kernel_if->u.h.header_page);if (retval < (int )consume_q->kernel_if->num_pages) { pr_debug("get_user_pages_fast(consume) failed (retval=%d)",
drivers/platform/goldfish/goldfish_pipe.c +2 -1
@@ -274,7 +274,8 @@ static int pin_user_pages(unsigned long first_page, *iter_last_page_size = last_page_size; } - ret =get_user_pages_fast (first_page, requested_pages, !is_write, + ret =get_user_pages_fast (first_page, requested_pages, + !is_write ?FOLL_WRITE :0 , pages);if (ret <=0 )return -EFAULT ;
drivers/rapidio/devices/rio_mport_cdev.c +3 -1
@@ -868,7 +868,9 @@ rio_dma_transfer(struct file *filp, u32 transfer_mode, pinned =get_user_pages_fast ( (unsigned long )xfer->loc_addr &PAGE_MASK , - nr_pages, dir ==DMA_FROM_DEVICE , page_list); + nr_pages, + dir ==DMA_FROM_DEVICE ?FOLL_WRITE :0 , + page_list);if (pinned != nr_pages) {if (pinned <0 ) {
drivers/sbus/char/oradax.c +1 -1
@@ -437,7 +437,7 @@ static int dax_lock_page(void *va, struct page **p)dax_dbg ("uva %p" , va); - ret =get_user_pages_fast ((unsigned long )va,1 ,1 , p); + ret =get_user_pages_fast ((unsigned long )va,1 ,FOLL_WRITE , p);if (ret ==1 ) {dax_dbg ("locked page %p, for VA %p" , *p, va);return 0 ;
drivers/scsi/st.c +2 -1
@@ -4922,7 +4922,8 @@ static int sgl_map_user_pages(struct st_buffer *STbp,/* Try to fault in all of the necessary pages */ /* rw==READ means read from drive, write into memory area */ - res =get_user_pages_fast (uaddr, nr_pages, rw ==READ , pages); + res =get_user_pages_fast (uaddr, nr_pages, rw ==READ ?FOLL_WRITE :0 , + pages);/* Errors and no page mapped should return here */ if (res < nr_pages)
drivers/staging/gasket/gasket_page_table.c +2 -2
@@ -486,8 +486,8 @@ static int gasket_perform_mapping(struct gasket_page_table *pg_tbl, ptes[i].dma_addr = pg_tbl->coherent_pages[0 ].paddr + off + i *PAGE_SIZE ; }else { - ret =get_user_pages_fast (page_addr - offset,1 ,1 , - &page); + ret =get_user_pages_fast (page_addr - offset,1 , +FOLL_WRITE , &page);if (ret <=0 ) { dev_err(pg_tbl->device,
drivers/tee/tee_shm.c +1 -1
@@ -273,7 +273,7 @@ struct tee_shm *tee_shm_register(struct tee_context *ctx, unsigned long addr, goto err; } - rc =get_user_pages_fast (start, num_pages,1 , shm->pages); + rc =get_user_pages_fast (start, num_pages,FOLL_WRITE , shm->pages);if (rc >0 ) shm->num_pages = rc;if (rc != num_pages) {
drivers/vfio/vfio_iommu_spapr_tce.c +2 -1
@@ -532,7 +532,8 @@ static int tce_iommu_use_page(unsigned long tce, unsigned long *hpa)enum dma_data_direction direction =iommu_tce_direction (tce);if (get_user_pages_fast (tce &PAGE_MASK ,1 , - direction !=DMA_TO_DEVICE , &page) !=1 ) + direction !=DMA_TO_DEVICE ?FOLL_WRITE :0 , + &page) !=1 )return -EFAULT ; *hpa =__pa ((unsigned long )page_address (page));
drivers/vhost/vhost.c +1 -1
@@ -1700,7 +1700,7 @@ static int set_bit_to_user(int nr, void __user *addr)int bit = nr + (log %PAGE_SIZE ) *8 ;int r; - r =get_user_pages_fast (log,1 ,1 , &page); + r =get_user_pages_fast (log,1 ,FOLL_WRITE , &page);if (r <0 )return r;BUG_ON (r !=1 );
drivers/video/fbdev/pvr2fb.c +1 -1
@@ -686,7 +686,7 @@ static ssize_t pvr2fb_write(struct fb_info *info, const char *buf,if (!pages)return -ENOMEM ; - ret =get_user_pages_fast ((unsigned long )buf, nr_pages, true, pages); + ret =get_user_pages_fast ((unsigned long )buf, nr_pages,FOLL_WRITE , pages);if (ret < nr_pages) { nr_pages = ret; ret = -EINVAL ;
drivers/virt/fsl_hypervisor.c +1 -1
@@ -244,7 +244,7 @@ static long ioctl_memcpy(struct fsl_hv_ioctl_memcpy __user *p)/* Get the physical addresses of the source buffer */ num_pinned =get_user_pages_fast (param.local_vaddr - lb_offset, - num_pages, param.source !=-1 , pages); + num_pages, param.source !=-1 ?FOLL_WRITE :0 , pages);if (num_pinned != num_pages) {/* get_user_pages() failed */
drivers/xen/gntdev.c +1 -1
@@ -852,7 +852,7 @@ static int gntdev_get_page(struct gntdev_copy_batch *batch, void __user *virt,unsigned long xen_pfn;int ret; - ret =get_user_pages_fast (addr,1 , writeable, &page); + ret =get_user_pages_fast (addr,1 , writeable ?FOLL_WRITE :0 , &page);if (ret <0 )return ret;
fs/orangefs/orangefs-bufmap.c +1 -1
@@ -269,7 +269,7 @@ orangefs_bufmap_map(struct orangefs_bufmap *bufmap,/* map the pages */ ret =get_user_pages_fast ((unsigned long )user_desc->ptr, - bufmap->page_count,1 , bufmap->page_array); + bufmap->page_count,FOLL_WRITE , bufmap->page_array);if (ret <0 )return ret;
include/linux/mm.h +2 -2
@@ -1531,8 +1531,8 @@ long get_user_pages_locked(unsigned long start, unsigned long nr_pages,long get_user_pages_unlocked (unsigned long start,unsigned long nr_pages,struct page **pages,unsigned int gup_flags); -int get_user_pages_fast (unsigned long start,int nr_pages,int write, -struct page **pages); +int get_user_pages_fast (unsigned long start,int nr_pages, +unsigned int gup_flags,struct page **pages);/* Container for pinned pfns / pages */ struct frame_vector {
kernel/futex.c +1 -1
@@ -543,7 +543,7 @@ get_futex_key(u32 __user *uaddr, int fshared, union futex_key *key, enum futex_aif (unlikely (should_fail_futex (fshared)))return -EFAULT ; - err =get_user_pages_fast (address,1 ,1 , &page); + err =get_user_pages_fast (address,1 ,FOLL_WRITE , &page);/* * If write access is not required (eg. FUTEX_WAIT), try * and get read-only access.
lib/iov_iter.c +5 -2
@@ -1293,7 +1293,9 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, len = maxpages *PAGE_SIZE ; addr &= ~(PAGE_SIZE -1 ); n =DIV_ROUND_UP (len,PAGE_SIZE ); - res =get_user_pages_fast (addr, n,iov_iter_rw (i) !=WRITE , pages); + res =get_user_pages_fast (addr, n, +iov_iter_rw (i) !=WRITE ?FOLL_WRITE :0 , + pages);if (unlikely (res <0 ))return res;return (res == n ? len : res *PAGE_SIZE ) - *start;
@@ -1374,7 +1376,8 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, p =get_pages_array (n);if (!p)return -ENOMEM ; - res =get_user_pages_fast (addr, n,iov_iter_rw (i) !=WRITE , p); + res =get_user_pages_fast (addr, n, +iov_iter_rw (i) !=WRITE ?FOLL_WRITE :0 , p);if (unlikely (res <0 )) {kvfree (p);return res;
mm/gup.c +5 -5
@@ -2025,7 +2025,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* get_user_pages_fast() - pin user pages in memory * @start: starting user address * @nr_pages: number of pages from start to pin -* @write: whether pages will be written to +* @gup_flags: flags modifying pin behaviour * @pages: array that receives pointers to the pages pinned. * Should be at least nr_pages long. *
@@ -2037,8 +2037,8 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* requested. If nr_pages is 0 or negative, returns 0. If no pages * were pinned, returns -errno. */ -int get_user_pages_fast (unsigned long start,int nr_pages,int write, -struct page **pages) +int get_user_pages_fast (unsigned long start,int nr_pages, +unsigned int gup_flags,struct page **pages) {unsigned long addr, len, end;int nr =0 , ret =0 ;
@@ -2056,7 +2056,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write,if (gup_fast_permitted (start, nr_pages)) {local_irq_disable (); -gup_pgd_range (addr, end, write ?FOLL_WRITE :0 , pages, &nr); +gup_pgd_range (addr, end, gup_flags, pages, &nr);local_irq_enable (); ret = nr; }
@@ -2067,7 +2067,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write, pages += nr; ret =get_user_pages_unlocked (start, nr_pages - nr, pages, - write ?FOLL_WRITE :0 ); + gup_flags);/* Have to be a bit careful with return values */ if (nr >0 ) {
mm/util.c +4 -4
@@ -318,7 +318,7 @@ EXPORT_SYMBOL_GPL(__get_user_pages_fast);* get_user_pages_fast() - pin user pages in memory * @start: starting user address * @nr_pages: number of pages from start to pin -* @write: whether pages will be written to +* @gup_flags: flags modifying pin behaviour * @pages: array that receives pointers to the pages pinned. * Should be at least nr_pages long. *
@@ -339,10 +339,10 @@ EXPORT_SYMBOL_GPL(__get_user_pages_fast);* were pinned, returns -errno. */ int __weak get_user_pages_fast (unsigned long start, -int nr_pages,int write,struct page **pages) +int nr_pages,unsigned int gup_flags, +struct page **pages) { -return get_user_pages_unlocked (start, nr_pages, pages, - write ?FOLL_WRITE :0 ); +return get_user_pages_unlocked (start, nr_pages, pages, gup_flags); }EXPORT_SYMBOL_GPL (get_user_pages_fast);
net/ceph/pagevec.c +1 -1
@@ -27,7 +27,7 @@ struct page **ceph_get_direct_page_vector(const void __user *data,while (got < num_pages) { rc =get_user_pages_fast ( (unsigned long )data + ((unsigned long )got *PAGE_SIZE ), - num_pages - got, write_page, pages + got); + num_pages - got, write_page ?FOLL_WRITE :0 , pages + got);if (rc <0 )break ;BUG_ON (rc ==0 );
net/rds/info.c +1 -1
@@ -193,7 +193,7 @@ int rds_info_getsockopt(struct socket *sock, int optname, char __user *optval, ret = -ENOMEM ; goto out; } - ret =get_user_pages_fast (start, nr_pages,1 , pages); + ret =get_user_pages_fast (start, nr_pages,FOLL_WRITE , pages);if (ret != nr_pages) {if (ret >0 ) nr_pages = ret;
net/rds/rdma.c +2 -1
@@ -158,7 +158,8 @@ static int rds_pin_pages(unsigned long user_addr, unsigned int nr_pages, {int ret; - ret =get_user_pages_fast (user_addr, nr_pages, write, pages); + ret =get_user_pages_fast (user_addr, nr_pages, write ?FOLL_WRITE :0 , + pages);if (ret >=0 && ret < nr_pages) {while (ret--)
Signature · 1 line
2.20.1
Re: [RESEND 3/7] mm/gup: Change GUP fast to use flags rather than a write 'bool'
Quoted 73 lines · On Sun, Mar 17, 2019 at 7:36 PM <[email protected]> wrote:
From: Ira Weiny <[email protected]>
To facilitate additional options to
get_user_pages_fast()change the singular write parameter to begup_flags.This patch does not change any functionality. New functionality will follow in subsequent patches.
Some of the
get_user_pages_fast()call sites were unchanged because they already passedFOLL_WRITEor 0 for the write parameter.Signed-off-by Ira Weiny <[email protected]>---
Changes from V1:
Rebase to current merge treearch/powerpc/mm/mmu_context_iommu.cno longer callsgup_fast
Thegup_longtermwas converted in patch 1arch/mips/mm/gup.c | 11 ++++++----- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 ++-- arch/powerpc/kvm/e500_mmu.c | 2 +- arch/s390/kvm/interrupt.c | 2 +- arch/s390/mm/gup.c | 12 ++++++------ arch/sh/mm/gup.c | 11 ++++++----- arch/sparc/mm/gup.c | 9 +++++---- arch/x86/kvm/paging_tmpl.h | 2 +- arch/x86/kvm/svm.c | 2 +- drivers/fpga/dfl-afu-dma-region.c | 2 +- drivers/gpu/drm/via/via_dmablit.c | 3 ++- drivers/infiniband/hw/hfi1/user_pages.c | 3 ++- drivers/misc/genwqe/card_utils.c | 2 +- drivers/misc/vmw_vmci/vmci_host.c | 2 +- drivers/misc/vmw_vmci/vmci_queue_pair.c | 6 ++++-- drivers/platform/goldfish/goldfish_pipe.c | 3 ++- drivers/rapidio/devices/rio_mport_cdev.c | 4 +++- drivers/sbus/char/oradax.c | 2 +- drivers/scsi/st.c | 3 ++- drivers/staging/gasket/gasket_page_table.c | 4 ++-- drivers/tee/tee_shm.c | 2 +- drivers/vfio/vfio_iommu_spapr_tce.c | 3 ++- drivers/vhost/vhost.c | 2 +- drivers/video/fbdev/pvr2fb.c | 2 +- drivers/virt/fsl_hypervisor.c | 2 +- drivers/xen/gntdev.c | 2 +- fs/orangefs/orangefs-bufmap.c | 2 +- include/linux/mm.h | 4 ++-- kernel/futex.c | 2 +- lib/iov_iter.c | 7 +++++-- mm/gup.c | 10 +++++----- mm/util.c | 8 ++++---- net/ceph/pagevec.c | 2 +- net/rds/info.c | 2 +- net/rds/rdma.c | 3 ++- 35 files changed, 79 insertions(+), 63 deletions(-)arch/mips/mm/gup.c +1
-1@@ -235,7 +235,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* get_user_pages_fast() - pin user pages in memory * @start: starting user address * @nr_pages: number of pages from start to pin -* @write: whether pages will be written to +* @gup_flags: flags modifying pin behaviour * @pages: array that receives pointers to the pages pinned. * Should be at least nr_pages long. *
arch/mips/mm/gup.c int __get_user_pages_fast(unsigned long start, int nr_pages, int write, · line 251
Quoted 7 lines
arch/mips/mm/gup.c +2
-2@@ -247,8 +247,8 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* requested. If nr_pages is 0 or negative, returns 0. If no pages * were pinned, returns -errno. */ -int get_user_pages_fast (unsigned long start,int nr_pages,int write, -struct page **pages) +int get_user_pages_fast (unsigned long start,int nr_pages, +unsigned int gup_flags,struct page **pages)
This looks a tad scary given all related thrash especially when it's only 1 user that wants to do get_user_page_fast_longterm, right? Maybe something like the following. Note I explicitly moved the flags to the end so that someone half paying attention that calls __get_user_pages_fast will get a compile error if they specify the args in the same order.
include/linux/mm.h +0 -0
@@ -1505,8 +1505,15 @@ static inline long
get_user_pages_longterm(unsigned long start,
}
#endif /* CONFIG_FS_DAX */
-int get_user_pages_fast(unsigned long start, int nr_pages, int write,
- struct page **pages);
+
+int __get_user_pages_fast(unsigned long start, int nr_pages,
+ struct page **pages, unsigned int gup_flags);
+
+static inline int get_user_pages_fast(unsigned long start, int nr_pages, int write,
+ struct page **pages)
+{
+ return __get_user_pages_fast(start, nr_pages, pages, write ?FOLL_WRITE);
+}
/* Container for pinned pfns / pages */
struct frame_vector {
Re: [RESEND 3/7] mm/gup: Change GUP fast to use flags rather than a write 'bool'
Quoted 76 lines · Dan Williams
Quoted 75 lines · On Sun, Mar 17, 2019 at 7:36 PM <[email protected]> wrote:
From: Ira Weiny <[email protected]>
To facilitate additional options to
get_user_pages_fast()change the singular write parameter to begup_flags.This patch does not change any functionality. New functionality will follow in subsequent patches.
Some of the
get_user_pages_fast()call sites were unchanged because they already passedFOLL_WRITEor 0 for the write parameter.Signed-off-by Ira Weiny <[email protected]>---
Changes from V1:
Rebase to current merge treearch/powerpc/mm/mmu_context_iommu.cno longer callsgup_fast
Thegup_longtermwas converted in patch 1arch/mips/mm/gup.c | 11 ++++++----- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 ++-- arch/powerpc/kvm/e500_mmu.c | 2 +- arch/s390/kvm/interrupt.c | 2 +- arch/s390/mm/gup.c | 12 ++++++------ arch/sh/mm/gup.c | 11 ++++++----- arch/sparc/mm/gup.c | 9 +++++---- arch/x86/kvm/paging_tmpl.h | 2 +- arch/x86/kvm/svm.c | 2 +- drivers/fpga/dfl-afu-dma-region.c | 2 +- drivers/gpu/drm/via/via_dmablit.c | 3 ++- drivers/infiniband/hw/hfi1/user_pages.c | 3 ++- drivers/misc/genwqe/card_utils.c | 2 +- drivers/misc/vmw_vmci/vmci_host.c | 2 +- drivers/misc/vmw_vmci/vmci_queue_pair.c | 6 ++++-- drivers/platform/goldfish/goldfish_pipe.c | 3 ++- drivers/rapidio/devices/rio_mport_cdev.c | 4 +++- drivers/sbus/char/oradax.c | 2 +- drivers/scsi/st.c | 3 ++- drivers/staging/gasket/gasket_page_table.c | 4 ++-- drivers/tee/tee_shm.c | 2 +- drivers/vfio/vfio_iommu_spapr_tce.c | 3 ++- drivers/vhost/vhost.c | 2 +- drivers/video/fbdev/pvr2fb.c | 2 +- drivers/virt/fsl_hypervisor.c | 2 +- drivers/xen/gntdev.c | 2 +- fs/orangefs/orangefs-bufmap.c | 2 +- include/linux/mm.h | 4 ++-- kernel/futex.c | 2 +- lib/iov_iter.c | 7 +++++-- mm/gup.c | 10 +++++----- mm/util.c | 8 ++++---- net/ceph/pagevec.c | 2 +- net/rds/info.c | 2 +- net/rds/rdma.c | 3 ++- 35 files changed, 79 insertions(+), 63 deletions(-)arch/mips/mm/gup.c +1
-1@@ -235,7 +235,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* get_user_pages_fast() - pin user pages in memory * @start: starting user address * @nr_pages: number of pages from start to pin -* @write: whether pages will be written to +* @gup_flags: flags modifying pin behaviour * @pages: array that receives pointers to the pages pinned. * Should be at least nr_pages long. * @@ -247,8 +247,8 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* requested. If nr_pages is 0 or negative, returns 0. If no pages * were pinned, returns -errno.
Quoted 8 lines
arch/mips/mm/gup.c int __get_user_pages_fast(unsigned long start, int nr_pages, int write, · line 251
Quoted 5 lines
arch/mips/mm/gup.c +2
-2*/ -int get_user_pages_fast (unsigned long start,int nr_pages,int write, -struct page **pages) +int get_user_pages_fast (unsigned long start,int nr_pages, +unsigned int gup_flags,struct page **pages)This looks a tad scary given all related thrash especially when it's only 1 user that wants to do
get_user_page_fast_longterm, right?
Agreed but the discussion back in Feb agreed that it would be better to make get_user_pages_fast() use flags rather than add another *_longterm call, and I agree.
Quoted 5 lines
Maybe
something like the following. Note I explicitly moved the flags to the end so that someone half paying attention that calls__get_user_pages_fastwill get a compile error if they specify the args in the same order.
I did this to remain consistent with the other public calls. They put the "return" pages parameter at the end. For example get_user_pages() is defined this way with pages and vmas at the end.
long get_user_pages(unsigned long start, unsigned long nr_pages,
unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas)
I'm pretty sure I got all the callers. Is this worth making the signature "non-standard" WRT the other calls?
Ira
Quoted 27 lines
include/linux/mm.h +9
-2@@ -1505,8 +1505,15 @@ static inline long get_user_pages_longterm(unsigned long start, }#endif /* CONFIG_FS_DAX */ -int get_user_pages_fast (unsigned long start,int nr_pages,int write, -struct page **pages); + +int __get_user_pages_fast (unsigned long start,int nr_pages, +struct page **pages,unsigned int gup_flags); + +static inline int get_user_pages_fast (unsigned long start,int nr_pages,int write, +struct page **pages) +{ + return __get_user_pages_fast(start, nr_pages, pages, write ? FOLL_WRITE); +}/* Container for pinned pfns / pages */
structframe_vector{
PATCH RESEND 4/7 mm/gup: Add FOLL_LONGTERM capability to GUP fast
In reply to [email protected]From: Ira Weiny <[email protected]>
DAX pages were previously unprotected from longterm pins when users called get_user_pages_fast().
Use the new FOLL_LONGTERM flag to check for DEVMAP pages and fall back to regular GUP processing if a DEVMAP page is encountered.
--- mm/gup.c | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-)
mm/gup.c +25 -4
@@ -1600,6 +1600,9 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, goto pte_unmap;if (pte_devmap (pte)) { +if (unlikely (flags &FOLL_LONGTERM )) + goto pte_unmap; + pgmap =get_dev_pagemap (pte_pfn (pte), pgmap);if (unlikely (!pgmap)) {undo_dev_pagemap (nr, nr_start, pages);
@@ -1739,8 +1742,11 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr,if (!pmd_access_permitted (orig, flags &FOLL_WRITE ))return 0 ; -if (pmd_devmap (orig)) +if (pmd_devmap (orig)) { +if (unlikely (flags &FOLL_LONGTERM )) +return 0 ;return __gup_device_huge_pmd (orig, pmdp, addr, end, pages, nr); + } refs =0 ; page =pmd_page (orig) + ((addr & ~PMD_MASK ) >>PAGE_SHIFT );
@@ -1777,8 +1783,11 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr,if (!pud_access_permitted (orig, flags &FOLL_WRITE ))return 0 ; -if (pud_devmap (orig)) +if (pud_devmap (orig)) { +if (unlikely (flags &FOLL_LONGTERM )) +return 0 ;return __gup_device_huge_pud (orig, pudp, addr, end, pages, nr); + } refs =0 ; page =pud_page (orig) + ((addr & ~PUD_MASK ) >>PAGE_SHIFT );
@@ -2066,8 +2075,20 @@ int get_user_pages_fast(unsigned long start, int nr_pages, start += nr <<PAGE_SHIFT ; pages += nr; - ret =get_user_pages_unlocked (start, nr_pages - nr, pages, - gup_flags); +if (gup_flags &FOLL_LONGTERM ) { +down_read (¤t->mm->mmap_sem); + ret =__gup_longterm_locked (current, current->mm, + start, nr_pages - nr, + pages,NULL , gup_flags); +up_read (¤t->mm->mmap_sem); + }else { +/* +* retain FAULT_FOLL_ALLOW_RETRY optimization if +* possible +*/ + ret =get_user_pages_unlocked (start, nr_pages - nr, + pages, gup_flags); + }/* Have to be a bit careful with return values */ if (nr >0 ) {
Signature · 1 line
2.20.1
Re: [RESEND 4/7] mm/gup: Add FOLL_LONGTERM capability to GUP fast
Quoted 63 lines · On Sun, Mar 17, 2019 at 7:36 PM <[email protected]> wrote:
From: Ira Weiny <[email protected]>
DAX pages were previously unprotected from longterm pins when users called
get_user_pages_fast().Use the new
FOLL_LONGTERMflag to check for DEVMAP pages and fall back to regular GUP processing if a DEVMAP page is encountered.Signed-off-by Ira Weiny <[email protected]>--- mm/gup.c | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-)mm/gup.c +14
-4@@ -1600,6 +1600,9 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, goto pte_unmap;if (pte_devmap (pte)) { +if (unlikely (flags &FOLL_LONGTERM )) + goto pte_unmap; + pgmap =get_dev_pagemap (pte_pfn (pte), pgmap);if (unlikely (!pgmap)) {undo_dev_pagemap (nr, nr_start, pages);@@ -1739,8 +1742,11 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr,if (!pmd_access_permitted (orig, flags &FOLL_WRITE ))return 0 ; -if (pmd_devmap (orig)) +if (pmd_devmap (orig)) { +if (unlikely (flags &FOLL_LONGTERM )) +return 0 ;return __gup_device_huge_pmd (orig, pmdp, addr, end, pages, nr); + } refs =0 ; page =pmd_page (orig) + ((addr & ~PMD_MASK ) >>PAGE_SHIFT );@@ -1777,8 +1783,11 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr,if (!pud_access_permitted (orig, flags &FOLL_WRITE ))return 0 ; -if (pud_devmap (orig)) +if (pud_devmap (orig)) { +if (unlikely (flags &FOLL_LONGTERM )) +return 0 ;return __gup_device_huge_pud (orig, pudp, addr, end, pages, nr); + } refs =0 ; page =pud_page (orig) + ((addr & ~PUD_MASK ) >>PAGE_SHIFT );@@ -2066,8 +2075,20 @@ int get_user_pages_fast(unsigned long start, int nr_pages, start += nr <<PAGE_SHIFT ; pages += nr; - ret =get_user_pages_unlocked (start, nr_pages - nr, pages, - gup_flags); +if (gup_flags &FOLL_LONGTERM ) { +down_read (¤t->mm->mmap_sem); + ret = __gup_longterm_locked(current, current->mm,
mm/gup.c int get_user_pages_fast(unsigned long start, int nr_pages, · line 2090
Quoted 10 lines
mm/gup.c +10
-0+ start, nr_pages - nr, + pages, NULL, gup_flags); +up_read (¤t->mm->mmap_sem); + }else { +/* +* retain FAULT_FOLL_ALLOW_RETRY optimization if +* possible +*/ + ret =get_user_pages_unlocked (start, nr_pages - nr, + pages, gup_flags);
I couldn't immediately grok why this path needs to branch on FOLL_LONGTERM? Won't get_user_pages_unlocked(..., FOLL_LONGTERM) do the right thing?
Re: [RESEND 4/7] mm/gup: Add FOLL_LONGTERM capability to GUP fast
Quoted 70 lines · Dan Williams
Quoted 69 lines · On Sun, Mar 17, 2019 at 7:36 PM <[email protected]> wrote:
From: Ira Weiny <[email protected]>
DAX pages were previously unprotected from longterm pins when users called
get_user_pages_fast().Use the new
FOLL_LONGTERMflag to check for DEVMAP pages and fall back to regular GUP processing if a DEVMAP page is encountered.Signed-off-by Ira Weiny <[email protected]>--- mm/gup.c | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-)mm/gup.c +20
-4@@ -1600,6 +1600,9 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, goto pte_unmap;if (pte_devmap (pte)) { +if (unlikely (flags &FOLL_LONGTERM )) + goto pte_unmap; + pgmap =get_dev_pagemap (pte_pfn (pte), pgmap);if (unlikely (!pgmap)) {undo_dev_pagemap (nr, nr_start, pages);@@ -1739,8 +1742,11 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr,if (!pmd_access_permitted (orig, flags &FOLL_WRITE ))return 0 ; -if (pmd_devmap (orig)) +if (pmd_devmap (orig)) { +if (unlikely (flags &FOLL_LONGTERM )) +return 0 ;return __gup_device_huge_pmd (orig, pmdp, addr, end, pages, nr); + } refs =0 ; page =pmd_page (orig) + ((addr & ~PMD_MASK ) >>PAGE_SHIFT );@@ -1777,8 +1783,11 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr,if (!pud_access_permitted (orig, flags &FOLL_WRITE ))return 0 ; -if (pud_devmap (orig)) +if (pud_devmap (orig)) { +if (unlikely (flags &FOLL_LONGTERM )) +return 0 ;return __gup_device_huge_pud (orig, pudp, addr, end, pages, nr); + } refs =0 ; page =pud_page (orig) + ((addr & ~PUD_MASK ) >>PAGE_SHIFT );@@ -2066,8 +2075,20 @@ int get_user_pages_fast(unsigned long start, int nr_pages, start += nr <<PAGE_SHIFT ; pages += nr; - ret =get_user_pages_unlocked (start, nr_pages - nr, pages, - gup_flags); +if (gup_flags &FOLL_LONGTERM ) { +down_read (¤t->mm->mmap_sem); + ret =__gup_longterm_locked (current, current->mm, + start, nr_pages - nr, + pages,NULL , gup_flags); +up_read (¤t->mm->mmap_sem); + }else { +/* +* retain FAULT_FOLL_ALLOW_RETRY optimization if
Quoted 8 lines
mm/gup.c int get_user_pages_fast(unsigned long start, int nr_pages, · line 2090
Quoted 4 lines
mm/gup.c +4
-0+* possible +*/ + ret =get_user_pages_unlocked (start, nr_pages - nr, + pages, gup_flags);I couldn't immediately grok why this path needs to branch on
FOLL_LONGTERM? Won'tget_user_pages_unlocked(..., FOLL_LONGTERM)do the right thing?
Unfortunately holding the lock is required to support FOLL_LONGTERM (to check the VMAs) but we don't want to hold the lock to be optimal (specifically allow FAULT_FOLL_ALLOW_RETRY). So I'm maintaining the optimization for *_fast users who do not specify FOLL_LONGTERM.
Another way to do this would have been to define __gup_longterm_unlocked with the above logic, but that seemed overkill at this point.
Ira
Re: [RESEND 4/7] mm/gup: Add FOLL_LONGTERM capability to GUP fast
Quoted 78 lines · Ira Weiny
Quoted 68 lines · Dan Williams
Quoted 67 lines · On Sun, Mar 17, 2019 at 7:36 PM <[email protected]> wrote:
From: Ira Weiny <[email protected]>
DAX pages were previously unprotected from longterm pins when users called
get_user_pages_fast().Use the new
FOLL_LONGTERMflag to check for DEVMAP pages and fall back to regular GUP processing if a DEVMAP page is encountered.Signed-off-by Ira Weiny <[email protected]>mm/gup.c | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-)mm/gup.c +21
-4+++ b/mm/gup.c@@ -1600,6 +1600,9 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, goto pte_unmap;if (pte_devmap (pte)) { +if (unlikely (flags &FOLL_LONGTERM )) + goto pte_unmap; + pgmap =get_dev_pagemap (pte_pfn (pte), pgmap);if (unlikely (!pgmap)) {undo_dev_pagemap (nr, nr_start, pages);@@ -1739,8 +1742,11 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr,if (!pmd_access_permitted (orig, flags &FOLL_WRITE ))return 0 ; -if (pmd_devmap (orig)) +if (pmd_devmap (orig)) { +if (unlikely (flags &FOLL_LONGTERM )) +return 0 ;return __gup_device_huge_pmd (orig, pmdp, addr, end, pages, nr); + } refs =0 ; page =pmd_page (orig) + ((addr & ~PMD_MASK ) >>PAGE_SHIFT );@@ -1777,8 +1783,11 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr,if (!pud_access_permitted (orig, flags &FOLL_WRITE ))return 0 ; -if (pud_devmap (orig)) +if (pud_devmap (orig)) { +if (unlikely (flags &FOLL_LONGTERM )) +return 0 ;return __gup_device_huge_pud (orig, pudp, addr, end, pages, nr); + } refs =0 ; page =pud_page (orig) + ((addr & ~PUD_MASK ) >>PAGE_SHIFT );@@ -2066,8 +2075,20 @@ int get_user_pages_fast(unsigned long start, int nr_pages, start += nr <<PAGE_SHIFT ; pages += nr; - ret =get_user_pages_unlocked (start, nr_pages - nr, pages, - gup_flags); +if (gup_flags &FOLL_LONGTERM ) { +down_read (¤t->mm->mmap_sem); + ret =__gup_longterm_locked (current, current->mm, + start, nr_pages - nr, + pages,NULL , gup_flags); +up_read (¤t->mm->mmap_sem); + }else { +/* +* retain FAULT_FOLL_ALLOW_RETRY optimization if Quoted 8 lines
mm/gup.c int get_user_pages_fast(unsigned long start, int nr_pages, · line 2090
Quoted 4 lines
mm/gup.c +4
-0+* possible +*/ + ret =get_user_pages_unlocked (start, nr_pages - nr, + pages, gup_flags);I couldn't immediately grok why this path needs to branch on
FOLL_LONGTERM? Won'tget_user_pages_unlocked(..., FOLL_LONGTERM)do the right thing?
Quoted 7 lines
Unfortunately holding the lock is required to support
FOLL_LONGTERM(to check the VMAs) but we don't want to hold the lock to be optimal (specifically allowFAULT_FOLL_ALLOW_RETRY). So I'm maintaining the optimization for*_fastusers who do not specifyFOLL_LONGTERM.Another way to do this would have been to define
__gup_longterm_unlockedwith the above logic, but that seemed overkill at this point.
get_user_pages_unlocked() is an exported symbol, shouldn't it work with the FOLL_LONGTERM flag?
I think it should even though we have no user..
Otherwise the GUP API just gets more confusing.
Jason
Re: [RESEND 4/7] mm/gup: Add FOLL_LONGTERM capability to GUP fast
Quoted 87 lines · Jason Gunthorpe
Quoted 78 lines · Ira Weiny
Quoted 68 lines · Dan Williams
Quoted 67 lines · On Sun, Mar 17, 2019 at 7:36 PM <[email protected]> wrote:
From: Ira Weiny <[email protected]>
DAX pages were previously unprotected from longterm pins when users called
get_user_pages_fast().Use the new
FOLL_LONGTERMflag to check for DEVMAP pages and fall back to regular GUP processing if a DEVMAP page is encountered.Signed-off-by Ira Weiny <[email protected]>mm/gup.c | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-)mm/gup.c +21
-4+++ b/mm/gup.c@@ -1600,6 +1600,9 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, goto pte_unmap;if (pte_devmap (pte)) { +if (unlikely (flags &FOLL_LONGTERM )) + goto pte_unmap; + pgmap =get_dev_pagemap (pte_pfn (pte), pgmap);if (unlikely (!pgmap)) {undo_dev_pagemap (nr, nr_start, pages);@@ -1739,8 +1742,11 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr,if (!pmd_access_permitted (orig, flags &FOLL_WRITE ))return 0 ; -if (pmd_devmap (orig)) +if (pmd_devmap (orig)) { +if (unlikely (flags &FOLL_LONGTERM )) +return 0 ;return __gup_device_huge_pmd (orig, pmdp, addr, end, pages, nr); + } refs =0 ; page =pmd_page (orig) + ((addr & ~PMD_MASK ) >>PAGE_SHIFT );@@ -1777,8 +1783,11 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr,if (!pud_access_permitted (orig, flags &FOLL_WRITE ))return 0 ; -if (pud_devmap (orig)) +if (pud_devmap (orig)) { +if (unlikely (flags &FOLL_LONGTERM )) +return 0 ;return __gup_device_huge_pud (orig, pudp, addr, end, pages, nr); + } refs =0 ; page =pud_page (orig) + ((addr & ~PUD_MASK ) >>PAGE_SHIFT );@@ -2066,8 +2075,20 @@ int get_user_pages_fast(unsigned long start, int nr_pages, start += nr <<PAGE_SHIFT ; pages += nr; - ret =get_user_pages_unlocked (start, nr_pages - nr, pages, - gup_flags); +if (gup_flags &FOLL_LONGTERM ) { +down_read (¤t->mm->mmap_sem); + ret =__gup_longterm_locked (current, current->mm, + start, nr_pages - nr, + pages,NULL , gup_flags); +up_read (¤t->mm->mmap_sem); + }else { +/* +* retain FAULT_FOLL_ALLOW_RETRY optimization if Quoted 8 lines
mm/gup.c int get_user_pages_fast(unsigned long start, int nr_pages, · line 2090
Quoted 4 lines
mm/gup.c +4
-0+* possible +*/ + ret =get_user_pages_unlocked (start, nr_pages - nr, + pages, gup_flags);I couldn't immediately grok why this path needs to branch on
FOLL_LONGTERM? Won'tget_user_pages_unlocked(..., FOLL_LONGTERM)do the right thing?Quoted 7 lines
Unfortunately holding the lock is required to support
FOLL_LONGTERM(to check the VMAs) but we don't want to hold the lock to be optimal (specifically allowFAULT_FOLL_ALLOW_RETRY). So I'm maintaining the optimization for*_fastusers who do not specifyFOLL_LONGTERM.Another way to do this would have been to define
__gup_longterm_unlockedwith the above logic, but that seemed overkill at this point.
Quoted 6 lines
get_user_pages_unlocked()is an exported symbol, shouldn't it work with theFOLL_LONGTERMflag?I think it should even though we have no user..
Otherwise the GUP API just gets more confusing.
I agree WRT to the API. But I think callers of get_user_pages_unlocked() are not going to get the behavior they want if they specify FOLL_LONGTERM.
What I could do is BUG_ON (or just WARN_ON) if unlocked is called with FOLL_LONGTERM similar to the code in get_user_pages_locked() which does not allow locked and vmas to be passed together:
if (locked) {
/* if VM_FAULT_RETRY can be returned, vmas become invalid */
BUG_ON(vmas);
/* check caller initialized locked */
BUG_ON(*locked != 1);
}Combining Dan's comment and yours; I could do something like below? (not even compile tested) Coded with WARN_ON because technically I _think_ the FOLL_LONGTERM would "work" but not be optimal. I'm just not 100% sure. A BUG_ON would be most protective IMO.
mm/gup.c +26 -23
@@ -1014,6 +1014,29 @@ long get_user_pages_locked(unsigned long start, unsigned long nr_pages, }EXPORT_SYMBOL (get_user_pages_locked); +long __gup_longterm_unlocked (unsigned long start,unsigned long nr_pages, +struct page **pages,unsigned int gup_flags) +{ +struct mm_struct *mm = current->mm; +int locked =1 ; +long ret; + +down_read (&mm->mmap_sem); +if (gup_flags &FOLL_LONGTERM ) { + ret =__gup_longterm_locked (current, mm, + start, nr_pages - nr, + pages,NULL , gup_flags); + }else { + ret =__get_user_pages_locked (current, mm, start, nr_pages, + pages,NULL , &locked, + gup_flags |FOLL_TOUCH ); + } +if (locked) +up_read (&mm->mmap_sem); + +return ret; +} +/* * get_user_pages_unlocked() is suitable to replace the form: *
@@ -1032,16 +1055,9 @@ EXPORT_SYMBOL(get_user_pages_locked);long get_user_pages_unlocked (unsigned long start,unsigned long nr_pages,struct page **pages,unsigned int gup_flags) { -struct mm_struct *mm = current->mm; -int locked =1 ; -long ret; +WARN_ON (gup_flags &FOLL_LONGTERM ); -down_read (&mm->mmap_sem); - ret =__get_user_pages_locked (current, mm, start, nr_pages, pages,NULL , - &locked, gup_flags |FOLL_TOUCH ); -if (locked) -up_read (&mm->mmap_sem); -return ret; +__gup_longterm_unlocked (start, nr_pages, pages, gup_flags); }EXPORT_SYMBOL (get_user_pages_unlocked);
@@ -2075,20 +2091,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages, start += nr <<PAGE_SHIFT ; pages += nr; -if (gup_flags &FOLL_LONGTERM ) { -down_read (¤t->mm->mmap_sem); - ret =__gup_longterm_locked (current, current->mm, - start, nr_pages - nr, - pages,NULL , gup_flags); -up_read (¤t->mm->mmap_sem); - }else { -/* -* retain FAULT_FOLL_ALLOW_RETRY optimization if -* possible -*/ - ret =get_user_pages_unlocked (start, nr_pages - nr, - pages, gup_flags); - } +__gup_longterm_unlocked ();/* Have to be a bit careful with return values */ if (nr >0 ) {
Re: [RESEND 4/7] mm/gup: Add FOLL_LONGTERM capability to GUP fast
Quoted 11 lines · Ira Weiny
Quoted 11 lines
Quoted 7 lines
Unfortunately holding the lock is required to support
FOLL_LONGTERM(to check the VMAs) but we don't want to hold the lock to be optimal (specifically allowFAULT_FOLL_ALLOW_RETRY). So I'm maintaining the optimization for*_fastusers who do not specifyFOLL_LONGTERM.Another way to do this would have been to define
__gup_longterm_unlockedwith the above logic, but that seemed overkill at this point.
get_user_pages_unlocked()is an exported symbol, shouldn't it work with theFOLL_LONGTERMflag?
Quoted 6 lines
Quoted 3 lines
I think it should even though we have no user..
Otherwise the GUP API just gets more confusing.
I agree WRT to the API. But I think callers of
get_user_pages_unlocked()are not going to get the behavior they want if they specifyFOLL_LONGTERM.
Oh? Isn't the only thing FOLL_LONGTERM does is block the call on DAX? Why does the locking mode matter to this test?
Quoted 3 lines
What I could do is
BUG_ON(or justWARN_ON) if unlocked is called withFOLL_LONGTERMsimilar to the code inget_user_pages_locked()which does not allow locked and vmas to be passed together:
The GUP call should fail if you are doing something like this. But I'd rather not see confusing specialc cases in code without a clear comment explaining why it has to be there.
Jason
Re: [RESEND 4/7] mm/gup: Add FOLL_LONGTERM capability to GUP fast
Quoted 16 lines · Jason Gunthorpe
Quoted 15 lines · Ira Weiny
Quoted 14 lines
Quoted 7 lines
Unfortunately holding the lock is required to support
FOLL_LONGTERM(to check the VMAs) but we don't want to hold the lock to be optimal (specifically allowFAULT_FOLL_ALLOW_RETRY). So I'm maintaining the optimization for*_fastusers who do not specifyFOLL_LONGTERM.Another way to do this would have been to define
__gup_longterm_unlockedwith the above logic, but that seemed overkill at this point.
get_user_pages_unlocked()is an exported symbol, shouldn't it work with theFOLL_LONGTERMflag?I think it should even though we have no user..
Otherwise the GUP API just gets more confusing.
Quoted 8 lines
Quoted 2 lines
I agree WRT to the API. But I think callers of
get_user_pages_unlocked()are not going to get the behavior they want if they specifyFOLL_LONGTERM.Oh? Isn't the only thing
FOLL_LONGTERMdoes is block the call on DAX?From an API yes.
Why does the locking mode matter to this test?
DAX checks for VMA's being Filesystem DAX. Therefore, it requires collection of VMA's as the GUP code executes. The unlocked version can drop the lock and therefore the VMAs may become invalid. Therefore, the 2 code paths are incompatible.
Users of GUP unlocked are going to want the benefit of FAULT_FOLL_ALLOW_RETRY. So I don't anticipate anyone using FOLL_LONGTERM with get_user_pages_unlocked().
FWIW this thread is making me think my original patch which simply implemented get_user_pages_fast_longterm() would be more clear. There is some evidence that the GUP API was trending that way (see get_user_pages_remote). That seems wrong but I don't know how to ensure users don't specify the wrong flag.
Quoted 8 lines
Quoted 4 lines
What I could do is
BUG_ON(or justWARN_ON) if unlocked is called withFOLL_LONGTERMsimilar to the code inget_user_pages_locked()which does not allow locked and vmas to be passed together:The GUP call should fail if you are doing something like this. But I'd rather not see confusing specialc cases in code without a clear comment explaining why it has to be there.
Code comment would be necessary, sure. Was just throwing ideas out there.
Ira
Re: [RESEND 4/7] mm/gup: Add FOLL_LONGTERM capability to GUP fast
On Mon, Mar 25, 2019 at 3:22 PM Ira Weiny <[email protected]> wrote:
[..]
Quoted 4 lines
FWIW this thread is making me think my original patch which simply implemented
get_user_pages_fast_longterm()would be more clear. There is some evidence that the GUP API was trending that way (seeget_user_pages_remote). That seems wrong but I don't know how to ensure users don't specify the wrong flag.
What about just making the existing get_user_pages_longterm() have a fast path option?
Re: [RESEND 4/7] mm/gup: Add FOLL_LONGTERM capability to GUP fast
Quoted 9 lines · Dan Williams
On Mon, Mar 25, 2019 at 3:22 PM Ira Weiny <[email protected]> wrote:
[..]Quoted 4 lines
FWIW this thread is making me think my original patch which simply implemented
get_user_pages_fast_longterm()would be more clear. There is some evidence that the GUP API was trending that way (seeget_user_pages_remote). That seems wrong but I don't know how to ensure users don't specify the wrong flag.What about just making the existing
get_user_pages_longterm()have a fast path option?
That would work but was not the direction we agreed upon before.[1]
At this point I would rather see this patch set applied, focus on fixing the filesystem issues, and once that is done determine if FOLL_LONGTERM is needed in any GUP calls.
Ira
PATCH RESEND 5/7 IB/hfi1: Use the new FOLL_LONGTERM flag to get_user_pages_fast()
In reply to [email protected]From: Ira Weiny <[email protected]>
Use the new FOLL_LONGTERM to get_user_pages_fast() to protect against FS DAX pages being mapped.
--- drivers/infiniband/hw/hfi1/user_pages.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/infiniband/hw/hfi1/user_pages.c +4 -2
@@ -104,9 +104,11 @@ int hfi1_acquire_user_pages(struct mm_struct *mm, unsigned long vaddr, size_t np bool writable, struct page **pages) { int ret; +unsigned int gup_flags = writable ?FOLL_WRITE :0 ; - ret =get_user_pages_fast (vaddr, npages, writable ?FOLL_WRITE :0 , - pages); + gup_flags |=FOLL_LONGTERM ; + + ret =get_user_pages_fast (vaddr, npages, gup_flags, pages);if (ret <0 )return ret;
Signature · 1 line
2.20.1
Re: [RESEND 5/7] IB/hfi1: Use the new FOLL_LONGTERM flag to get_user_pages_fast()
Quoted 16 lines · On Sun, Mar 17, 2019 at 7:36 PM <[email protected]> wrote:
From: Ira Weiny <[email protected]>
Use the new
FOLL_LONGTERMtoget_user_pages_fast()to protect against FS DAX pages being mapped.Signed-off-by Ira Weiny <[email protected]>--- drivers/infiniband/hw/hfi1/user_pages.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)drivers/infiniband/hw/hfi1/user_pages.c +0
-0
drivers/infiniband/hw/hfi1/user_pages.c int hfi1_acquire_user_pages(struct mm_struct *mm, unsigned long vaddr, size_t np · line 107
Quoted 4 lines
drivers/infiniband/hw/hfi1/user_pages.c +1
-0@@ -104,9 +104,11 @@ int hfi1_acquire_user_pages(struct mm_struct *mm, unsigned long vaddr, size_t np bool writable, struct page **pages) { int ret; +unsigned int gup_flags = writable ?FOLL_WRITE :0 ;
Maybe:
unsigned int gup_flags = FOLL_LONGTERM | (writable ? FOLL_WRITE : 0);
?
Quoted 12 lines
drivers/infiniband/hw/hfi1/user_pages.c +3
-3- ret =get_user_pages_fast (vaddr, npages, writable ?FOLL_WRITE :0 , - pages); + gup_flags |=FOLL_LONGTERM ; + + ret =get_user_pages_fast (vaddr, npages, gup_flags, pages);if (ret <0 )return ret; --2.20.1
Re: [RESEND 5/7] IB/hfi1: Use the new FOLL_LONGTERM flag to get_user_pages_fast()
Quoted 24 lines · Dan Williams
Quoted 16 lines · On Sun, Mar 17, 2019 at 7:36 PM <[email protected]> wrote:
From: Ira Weiny <[email protected]>
Use the new
FOLL_LONGTERMtoget_user_pages_fast()to protect against FS DAX pages being mapped.Signed-off-by Ira Weiny <[email protected]>--- drivers/infiniband/hw/hfi1/user_pages.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)drivers/infiniband/hw/hfi1/user_pages.c +0
-0drivers/infiniband/hw/hfi1/user_pages.c int hfi1_acquire_user_pages(struct mm_struct *mm, unsigned long vaddr, size_t np · line 107
Quoted 4 lines
drivers/infiniband/hw/hfi1/user_pages.c +1
-0@@ -104,9 +104,11 @@ int hfi1_acquire_user_pages(struct mm_struct *mm, unsigned long vaddr, size_t np bool writable, struct page **pages) { int ret; +unsigned int gup_flags = writable ?FOLL_WRITE :0 ;Maybe:
Quoted 3 lines
unsigned int gup_flags = FOLL_LONGTERM | (writable ? FOLL_WRITE : 0);?
Sure looks good.
Ira
Quoted 14 lines
Quoted 13 lines
drivers/infiniband/hw/hfi1/user_pages.c +3
-3- ret =get_user_pages_fast (vaddr, npages, writable ?FOLL_WRITE :0 , - pages); + gup_flags |=FOLL_LONGTERM ; + + ret =get_user_pages_fast (vaddr, npages, gup_flags, pages);if (ret <0 )return ret; --2.20.1
PATCH RESEND 6/7 IB/qib: Use the new FOLL_LONGTERM flag to get_user_pages_fast()
In reply to [email protected]From: Ira Weiny <[email protected]>
Use the new FOLL_LONGTERM to get_user_pages_fast() to protect against FS DAX pages being mapped.
--- drivers/infiniband/hw/qib/qib_user_sdma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/hw/qib/qib_user_sdma.c +1 -1
@@ -673,7 +673,7 @@ static int qib_user_sdma_pin_pages(const struct qib_devdata *dd,else j = npages; - ret =get_user_pages_fast (addr, j,0 , pages); + ret =get_user_pages_fast (addr, j,FOLL_LONGTERM , pages);if (ret != j) { i =0 ; j = ret;
Signature · 1 line
2.20.1
Re: [RESEND 6/7] IB/qib: Use the new FOLL_LONGTERM flag to get_user_pages_fast()
Quoted 7 lines · On Sun, Mar 17, 2019 at 7:36 PM <[email protected]> wrote:
From: Ira Weiny <[email protected]>
Use the new
FOLL_LONGTERMtoget_user_pages_fast()to protect against FS DAX pages being mapped.Signed-off-by Ira Weiny <[email protected]>
Looks good modulo potential __get_user_pages_fast() suggestion.
PATCH RESEND 7/7 IB/mthca: Use the new FOLL_LONGTERM flag to get_user_pages_fast()
In reply to [email protected]From: Ira Weiny <[email protected]>
Use the new FOLL_LONGTERM to get_user_pages_fast() to protect against FS DAX pages being mapped.
--- drivers/infiniband/hw/mthca/mthca_memfree.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/hw/mthca/mthca_memfree.c +2 -1
@@ -472,7 +472,8 @@ int mthca_map_user_db(struct mthca_dev *dev, struct mthca_uar *uar, goto out; } - ret =get_user_pages_fast (uaddr &PAGE_MASK ,1 ,FOLL_WRITE , pages); + ret =get_user_pages_fast (uaddr &PAGE_MASK ,1 , +FOLL_WRITE |FOLL_LONGTERM , pages);if (ret <0 ) goto out;
Signature · 1 line
2.20.1
Andrew Morton
maintainerQuoted 6 lines · On Sun, 17 Mar 2019 11:34:31 -0700 [email protected] wrote:
Resending after rebasing to the latest mm tree.
HFI1, qib, and mthca, use
get_user_pages_fast()due to it performance advantages. These pages can be held for a significant time. Butget_user_pages_fast()does not protect against mapping FS DAX pages.
Quoted 6 lines
Introduce
FOLL_LONGTERMand use this flag inget_user_pages_fast()which retains the performance while also adding the FS DAX checks. XDP has also shown interest in using this functionality.[1]In addition we change
get_user_pages()to use the newFOLL_LONGTERMflag and remove the specializedget_user_pages_longtermcall.
It would be helpful to include your response to Christoph's question
(http://lkml.kernel.org/r/[email protected])
in the changelog. Because if one person was wondering about this, others will likely do so.
We have no record of acks or reviewed-by's. At least one was missed
(http://lkml.kernel.org/r/CAOg9mSTTcD-9bCSDfC0WRYqfVrNB4TwOzL0c4+6QXi-N_Y43Vw@mail.gmail.com),
but that is very very partial.
This patchset is fairly DAX-centered, but Dan wasn't cc'ed!
So ho hum. I'll scoop them up and shall make the above changes to the [1/n] changelog, but we still have some work to do.
Quoted 15 lines · Andrew Morton
Quoted 6 lines · On Sun, 17 Mar 2019 11:34:31 -0700 [email protected] wrote:
Resending after rebasing to the latest mm tree.
HFI1, qib, and mthca, use
get_user_pages_fast()due to it performance advantages. These pages can be held for a significant time. Butget_user_pages_fast()does not protect against mapping FS DAX pages.Quoted 6 lines
Introduce
FOLL_LONGTERMand use this flag inget_user_pages_fast()which retains the performance while also adding the FS DAX checks. XDP has also shown interest in using this functionality.[1]In addition we change
get_user_pages()to use the newFOLL_LONGTERMflag and remove the specializedget_user_pages_longtermcall.
Quoted 8 lines
It would be helpful to include your response to Christoph's question
(http://lkml.kernel.org/r/[email protected])
in the changelog. Because if one person was wondering about this, others will likely do so.We have no record of acks or reviewed-by's. At least one was missed
(http://lkml.kernel.org/r/CAOg9mSTTcD-9bCSDfC0WRYqfVrNB4TwOzL0c4+6QXi-N_Y43Vw@mail.gmail.com),
but that is very very partial.
That is my bad. Sorry to Mike. And I have added him.
Quoted 2 lines
This patchset is fairly DAX-centered, but Dan wasn't cc'ed!
Agreed, I'm new to changing things which affect this many sub-systems and I struggled with who should be CC'ed (get_maintainer.pl returned a very large list :-(.
I fear I may have cc'ed too many people, and the wrong people apparently, so that may be affecting the review...
So again my apologies. I don't know if Dan is going to get a chance to put a reviewed-by on them this week but I thought I would send this note to let you know I'm not ignoring your feedback. Just waiting a bit before resending to hopefully get some more acks/reviewed bys.
Thanks,
Ira
Quoted 4 lines
So ho hum. I'll scoop them up and shall make the above changes to the [1/n] changelog, but we still have some work to do.