IB/umem: use get_user_pages_fast() to pin DMA pages
Subsystems (3)
John Hubbard
reviewerv2 cover IB/umem: use get_user_pages_fast() to pin DMA pages
Hi,
The reason I'm posting this is to request a review for patch 1. Once everything seems OK, these two patches are going to be part of a larger set (again, destined for linux-mm) that introduces FOLL_PIN and renames some get_user_pages() cases to pin_user_pages(). I'm doing this a little before -rc1, because it's a small and easy thing to get out of the way early.
These are ultimately destined to go in via linux-mm (as opposed to linux-rdma), in order to change some names in just one kernel release cycle.
The first version of this patchset only had the IB/umem changes (patch 2), and I also lacked a runtime Infiniband test, so Leon Romanovsky reported a failure [1]. Patch 1 fixes that failure.
Since v1, I have (finally!) set up a basic two-node Infiniband system, and as a result, I've reproduced the failure that Leon saw, via a trivial run of "ib_write_bw", and confirmed that it's fixed in this new patchset. Sorry it took me so long to do that; I am going to vaguely blame "OFED" for the delay. :)
Entertaining IB side note: Jason: sure enough, as you mentioned, the OFED installation did in fact hopelessly mangle my system. Once I went back to a clean distro installation without all those OFED goodies, everything Just Worked on the first try. Much simpler to understand, too. ha.
This applies to today's linux.git: commit aedc0650f913 ("Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm").
[1] https://lore.kernel.org/r/20191124100724.GH136476@unreal
John Hubbard (2):
mm/gup: allow FOLL_FORCE for get_user_pages_fast() IB/umem: use get_user_pages_fast() to pin DMA pages
drivers/infiniband/core/umem.c | 17 ++++++----------- mm/gup.c | 3 ++- 2 files changed, 8 insertions(+), 12 deletions(-)
Signature · 1 line
2.24.0
John Hubbard
reviewerPATCH v2 1/2 mm/gup: allow FOLL_FORCE for get_user_pages_fast()
Commit 817be129e6f2 ("mm: validate get_user_pages_fast flags") allowed only FOLL_WRITE and FOLL_LONGTERM to be passed to get_user_pages_fast(). This, combined with the fact that get_user_pages_fast() falls back to "slow gup", which *does* accept FOLL_FORCE, leads to an odd situation:
if you need FOLL_FORCE, you cannot call get_user_pages_fast().
There does not appear to be any reason for filtering out FOLL_FORCE. There is nothing in the _fast() implementation that requires that we avoid writing to the pages. So it appears to have been an oversight.
Fix by allowing FOLL_FORCE to be set for get_user_pages_fast().
--- mm/gup.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/gup.c +2 -1
@@ -2415,7 +2415,8 @@ int get_user_pages_fast(unsigned long start, int nr_pages,unsigned long addr, len, end;int nr =0 , ret =0 ; -if (WARN_ON_ONCE (gup_flags & ~(FOLL_WRITE |FOLL_LONGTERM ))) +if (WARN_ON_ONCE (gup_flags & ~(FOLL_WRITE |FOLL_LONGTERM | +FOLL_FORCE )))return -EINVAL ; start =untagged_addr (start) &PAGE_MASK ;
Signature · 1 line
2.24.0
Re: [PATCH v2 1/2] mm/gup: allow FOLL_FORCE for get_user_pages_fast()
Quoted 10 lines · John Hubbard
Commit
817be129e6f2("mm: validateget_user_pages_fastflags") allowed onlyFOLL_WRITEandFOLL_LONGTERMto be passed toget_user_pages_fast(). This, combined with the fact thatget_user_pages_fast()falls back to "slow gup", which *does* acceptFOLL_FORCE, leads to an odd situation:
if you needFOLL_FORCE, you cannot callget_user_pages_fast().There does not appear to be any reason for filtering out
FOLL_FORCE. There is nothing in the_fast()implementation that requires that we avoid writing to the pages. So it appears to have been an oversight.
Quoted 9 lines
Fix by allowing
FOLL_FORCEto be set forget_user_pages_fast().Fixes 817be129e6f2 ("mm: validate get_user_pages_fast flags")Cc Christoph Hellwig <[email protected]>Signed-off-by John Hubbard <[email protected]>--- mm/gup.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Thanks,
John Hubbard
reviewerRe: [PATCH v2 1/2] mm/gup: allow FOLL_FORCE for get_user_pages_fast()
Quoted 21 lines · Leon Romanovsky
Quoted 10 lines · John Hubbard
Commit
817be129e6f2("mm: validateget_user_pages_fastflags") allowed onlyFOLL_WRITEandFOLL_LONGTERMto be passed toget_user_pages_fast(). This, combined with the fact thatget_user_pages_fast()falls back to "slow gup", which *does* acceptFOLL_FORCE, leads to an odd situation:
if you needFOLL_FORCE, you cannot callget_user_pages_fast().There does not appear to be any reason for filtering out
FOLL_FORCE. There is nothing in the_fast()implementation that requires that we avoid writing to the pages. So it appears to have been an oversight.Quoted 9 lines
Fix by allowing
FOLL_FORCEto be set forget_user_pages_fast().Fixes 817be129e6f2 ("mm: validate get_user_pages_fast flags")Cc Christoph Hellwig <[email protected]>Signed-off-by John Hubbard <[email protected]>--- mm/gup.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Quoted 3 lines
Thanks,
Reviewed-by Leon Romanovsky <[email protected]>
Hi Leon, thanks for the reviews, great timing! I'll add the tags to the commits, which I'm just about to post as part of the larger "pin user pages" patchset.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
John Hubbard
reviewerPATCH v2 2/2 IB/umem: use get_user_pages_fast() to pin DMA pages
In reply to John HubbardAnd get rid of the mmap_sem calls, as part of that. Note that get_user_pages_fast() will, if necessary, fall back to __gup_longterm_unlocked(), which takes the mmap_sem as needed.
--- drivers/infiniband/core/umem.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
drivers/infiniband/core/umem.c +6 -11
@@ -266,16 +266,13 @@ struct ib_umem *ib_umem_get(struct ib_udata *udata, unsigned long addr, sg = umem->sg_head.sgl;while (npages) { -down_read (&mm->mmap_sem); - ret =get_user_pages (cur_base, - min_t(unsigned long, npages, -PAGE_SIZE /sizeof (struct page *)), - gup_flags |FOLL_LONGTERM , - page_list,NULL ); -if (ret <0 ) { -up_read (&mm->mmap_sem); + ret =get_user_pages_fast (cur_base, + min_t(unsigned long, npages, +PAGE_SIZE / +sizeof (struct page *)), + gup_flags |FOLL_LONGTERM , page_list); +if (ret <0 ) goto umem_release; - } cur_base += ret *PAGE_SIZE ; npages -= ret;
@@ -283,8 +280,6 @@ struct ib_umem *ib_umem_get(struct ib_udata *udata, unsigned long addr, sg =ib_umem_add_sg_table (sg, page_list, ret,dma_get_max_seg_size (context->device->dma_device), &umem->sg_nents); - -up_read (&mm->mmap_sem); }sg_mark_end (sg);
Signature · 1 line
2.24.0
Quoted 4 lines · John Hubbard
And get rid of the
mmap_semcalls, as part of that. Note thatget_user_pages_fast()will, if necessary, fall back to__gup_longterm_unlocked(), which takes themmap_semas needed.
Quoted 9 lines
Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by Jan Kara <[email protected]>Reviewed-by Jason Gunthorpe <[email protected]>Reviewed-by Ira Weiny <[email protected]>Signed-off-by John Hubbard <[email protected]>--- drivers/infiniband/core/umem.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
Thanks,