switch the remaining architectures to use generic GUP v3
Subsystems (15)
Hi Linus and maintainers,
below is a series to switch mips, sh and sparc64 to use the generic GUP code so that we only have one codebase to touch for further improvements to this code. I don't have hardware for any of these architectures, and generally no clue about their page table management, so handle with care.
Changes since v2:
- rebase to mainline to pick up the untagged_addr definition
- fix the gup range check to be start <= end to catch the 0 length case
- use pfn based version for the missing pud_page/pgd_page definitions
- fix a wrong check in the sparc64 version of pte_access_permitted
Changes since v1:
- fix various issues found by the build bot
- cherry pick and use the untagged_addr helper form Andrey
- add various refactoring patches to share more code over architectures
- move the powerpc hugepd code to mm/gup.c and sync it with the generic hup semantics
PATCH 1/16 mm: use untagged_addr() for get_user_pages_fast addresses
This will allow sparc64 to override its ADI tags for get_user_pages and get_user_pages_fast.
--- mm/gup.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/gup.c +2 -2
@@ -2146,7 +2146,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,unsigned long flags;int nr =0 ; - start &=PAGE_MASK ; + start =untagged_addr (start) &PAGE_MASK ; len = (unsigned long ) nr_pages <<PAGE_SHIFT ; end = start + len;
@@ -2219,7 +2219,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages,unsigned long addr, len, end;int nr =0 , ret =0 ; - start &=PAGE_MASK ; + start =untagged_addr (start) &PAGE_MASK ; addr = start; len = (unsigned long ) nr_pages <<PAGE_SHIFT ; end = start + len;
Signature · 1 line
2.20.1
Re: [PATCH 01/16] mm: use untagged_addr() for get_user_pages_fast addresses
Quoted 5 lines · Christoph Hellwig
This will allow sparc64 to override its ADI tags for
get_user_pagesandget_user_pages_fast.Signed-off-by Christoph Hellwig <[email protected]>---
Commit message is sparc64 specific but the goal here is to allow any architecture with memory tagging to use this. So I would suggest rewording the commit log. Other than that:
Quoted 26 lines
mm/gup.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)mm/gup.c +2
-2@@ -2146,7 +2146,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,unsigned long flags;int nr =0 ; - start &=PAGE_MASK ; + start =untagged_addr (start) &PAGE_MASK ; len = (unsigned long ) nr_pages <<PAGE_SHIFT ; end = start + len;@@ -2219,7 +2219,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages,unsigned long addr, len, end;int nr =0 , ret =0 ; - start &=PAGE_MASK ; + start =untagged_addr (start) &PAGE_MASK ; addr = start; len = (unsigned long ) nr_pages <<PAGE_SHIFT ; end = start + len;
Re: [PATCH 01/16] mm: use untagged_addr() for get_user_pages_fast addresses
In reply to Christoph HellwigQuoted 7 lines · Christoph Hellwig
This will allow sparc64 to override its ADI tags for
get_user_pagesandget_user_pages_fast.Signed-off-by Christoph Hellwig <[email protected]>--- mm/gup.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Re: [PATCH 01/16] mm: use untagged_addr() for get_user_pages_fast addresses
In reply to Christoph HellwigQuoted 11 lines · Christoph Hellwig
This will allow sparc64 to override its ADI tags for
get_user_pagesandget_user_pages_fast.Signed-off-by Christoph Hellwig <[email protected]>mm/gup.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)mm/gup.c +1
-0+++ b/mm/gup.c
mm/gup.c int __get_user_pages_fast(unsigned long start, int nr_pages, int write, · line 2151
Quoted 7 lines
mm/gup.c +1
-1@@ -2146,7 +2146,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,unsigned long flags;int nr =0 ; - start &=PAGE_MASK ; + start =untagged_addr (start) &PAGE_MASK ; len = (unsigned long ) nr_pages <<PAGE_SHIFT ; end = start + len;
Hmm, this function, and the other, goes on to do:
if (unlikely(!access_ok((void __user *)start, len)))
return 0;and I thought that access_ok takes in the tagged pointer?
How about re-order it a bit?
mm/gup.c +3 -2
@@ -2148,11 +2148,12 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write, start &=PAGE_MASK ; len = (unsigned long ) nr_pages <<PAGE_SHIFT ; - end = start + len; -if (unlikely (!access_ok ((void __user *)start, len)))return 0 ; + start =untagged_ptr (start); + end = start + len; +/* * Disable interrupts. We use the nested form as we can already have * interrupts disabled by get_futex_key.
Re: [PATCH 01/16] mm: use untagged_addr() for get_user_pages_fast addresses
Quoted 22 lines · Jason Gunthorpe
Quoted 11 lines · Christoph Hellwig
This will allow sparc64 to override its ADI tags for
get_user_pagesandget_user_pages_fast.Signed-off-by Christoph Hellwig <[email protected]>mm/gup.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)mm/gup.c +1
-0+++ b/mm/gup.cmm/gup.c int __get_user_pages_fast(unsigned long start, int nr_pages, int write, · line 2151
Quoted 7 lines
mm/gup.c +1
-1@@ -2146,7 +2146,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,unsigned long flags;int nr =0 ; - start &=PAGE_MASK ; + start =untagged_addr (start) &PAGE_MASK ; len = (unsigned long ) nr_pages <<PAGE_SHIFT ; end = start + len;Hmm, this function, and the other, goes on to do:
Quoted 6 lines
if (unlikely(!access_ok((void __user *)start, len))) return 0;and I thought that
access_oktakes in the tagged pointer?How about re-order it a bit?
access_ok() can handle tagged or untagged pointers. It just strips the tag bits from the top bits. Current order doesn't really matter from functionality point of view. There might be minor gain in delaying untagging in __get_user_pages_fast() but I could go either way.
--
Khalid
Quoted 21 lines
mm/gup.c +3
-2@@ -2148,11 +2148,12 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write, start &=PAGE_MASK ; len = (unsigned long ) nr_pages <<PAGE_SHIFT ; - end = start + len; -if (unlikely (!access_ok ((void __user *)start, len)))return 0 ; + start =untagged_ptr (start); + end = start + len; +/* * Disable interrupts. We use the nested form as we can already have * interrupts disabled by get_futex_key.
Re: [PATCH 01/16] mm: use untagged_addr() for get_user_pages_fast addresses
Quoted 26 lines · Khalid Aziz
Quoted 25 lines · Jason Gunthorpe
Quoted 11 lines · Christoph Hellwig
This will allow sparc64 to override its ADI tags for
get_user_pagesandget_user_pages_fast.Signed-off-by Christoph Hellwig <[email protected]>mm/gup.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)mm/gup.c +1
-0+++ b/mm/gup.cmm/gup.c int __get_user_pages_fast(unsigned long start, int nr_pages, int write, · line 2151
Quoted 7 lines
mm/gup.c +1
-1@@ -2146,7 +2146,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,unsigned long flags;int nr =0 ; - start &=PAGE_MASK ; + start =untagged_addr (start) &PAGE_MASK ; len = (unsigned long ) nr_pages <<PAGE_SHIFT ; end = start + len;Hmm, this function, and the other, goes on to do:
if (unlikely(!access_ok((void __user *)start, len))) return 0;
Quoted 8 lines
Quoted 3 lines
and I thought that
access_oktakes in the tagged pointer?How about re-order it a bit?
access_ok()can handle tagged or untagged pointers. It just strips the tag bits from the top bits. Current order doesn't really matter from functionality point of view. There might be minor gain in delaying untagging in__get_user_pages_fast()but I could go either way.
I understand the current ARM and SPARC implementations don't do much with the tags, but it feels like a really big assumption for the core code that all future uses of tags will be fine to have them stripped out of 'void __user *' pointers. IMHO that is something we should not be doing in the core kernel..
Jason
Re: [PATCH 01/16] mm: use untagged_addr() for get_user_pages_fast addresses
In reply to Jason GunthorpeQuoted 9 lines · Jason Gunthorpe
Hmm, this function, and the other, goes on to do:
if (unlikely(!access_ok((void __user *)start, len))) return 0;and I thought that
access_oktakes in the tagged pointer?How about re-order it a bit?
The patch doesn't really work as-as an misses the main get_user_pages_fast fast path, but I'll add something equivalent.
Re: [PATCH 01/16] mm: use untagged_addr() for get_user_pages_fast addresses
In reply to Jason GunthorpeQuoted 8 lines · Jason Gunthorpe
Hmm, this function, and the other, goes on to do:
if (unlikely(!access_ok((void __user *)start, len))) return 0;and I thought that
access_oktakes in the tagged pointer?How about re-order it a bit?
Actually.. I we reorder this we'd need to to duplicate a few things due to the zero/negative length checking. Given the feedback from Khalid I'd thus rather skip the reorder for now. If we have a good reason we could add it back, but it would be a bit involved.
PATCH 2/16 mm: simplify gup_fast_permitted
In reply to Christoph HellwigPass in the already calculated end value instead of recomputing it, and leave the end > start check in the callers instead of duplicating them in the arch code.
--- arch/s390/include/asm/pgtable.h | 8 +------- arch/x86/include/asm/pgtable_64.h | 8 +------- mm/gup.c | 17 +++++++---------- 3 files changed, 9 insertions(+), 24 deletions(-)
arch/s390/include/asm/pgtable.h +1 -7
@@ -1270,14 +1270,8 @@ static inline pte_t *pte_offset(pmd_t *pmd, unsigned long address)#define pte_offset_map (pmd, address) pte_offset_kernel(pmd, address)#define pte_unmap (pte) do { } while (0) -static inline bool gup_fast_permitted (unsigned long start,int nr_pages) +static inline bool gup_fast_permitted (unsigned long start,unsigned long end) { -unsigned long len, end; - - len = (unsigned long ) nr_pages <<PAGE_SHIFT ; - end = start + len; -if (end < start) -return false;return end <= current->mm->context.asce_limit; }#define gup_fast_permitted gup_fast_permitted
arch/x86/include/asm/pgtable_64.h +1 -7
@@ -259,14 +259,8 @@ extern void init_extra_mapping_uc(unsigned long phys, unsigned long size);extern void init_extra_mapping_wb (unsigned long phys,unsigned long size);#define gup_fast_permitted gup_fast_permitted -static inline bool gup_fast_permitted (unsigned long start,int nr_pages) +static inline bool gup_fast_permitted (unsigned long start,unsigned long end) { -unsigned long len, end; - - len = (unsigned long )nr_pages <<PAGE_SHIFT ; - end = start + len; -if (end < start) -return false;if (end >> __VIRTUAL_MASK_SHIFT)return false;return true;
mm/gup.c +7 -10
@@ -2123,13 +2123,9 @@ static void gup_pgd_range(unsigned long addr, unsigned long end,* Check if it's allowed to use __get_user_pages_fast() for the range, or * we need to fall back to the slow version: */ -bool gup_fast_permitted (unsigned long start,int nr_pages) +static bool gup_fast_permitted (unsigned long start,unsigned long end) { -unsigned long len, end; - - len = (unsigned long ) nr_pages <<PAGE_SHIFT ; - end = start + len; -return end >= start; +return true; }#endif
@@ -2150,6 +2146,8 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write, len = (unsigned long ) nr_pages <<PAGE_SHIFT ; end = start + len; +if (end <= start) +return 0 ;if (unlikely (!access_ok ((void __user *)start, len)))return 0 ;
@@ -2165,7 +2163,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* block IPIs that come from THPs splitting. */ -if (gup_fast_permitted (start, nr_pages)) { +if (gup_fast_permitted (start, end)) {local_irq_save (flags);gup_pgd_range (start, end, write ?FOLL_WRITE :0 , pages, &nr);local_irq_restore (flags);
@@ -2224,13 +2222,12 @@ int get_user_pages_fast(unsigned long start, int nr_pages, len = (unsigned long ) nr_pages <<PAGE_SHIFT ; end = start + len; -if (nr_pages <=0 ) +if (end <= start)return 0 ; -if (unlikely (!access_ok ((void __user *)start, len)))return -EFAULT ; -if (gup_fast_permitted (start, nr_pages)) { +if (gup_fast_permitted (start, end)) {local_irq_disable ();gup_pgd_range (addr, end, gup_flags, pages, &nr);local_irq_enable ();
Signature · 1 line
2.20.1
Re: [PATCH 02/16] mm: simplify gup_fast_permitted
Quoted 10 lines · Christoph Hellwig
Pass in the already calculated end value instead of recomputing it, and leave the end > start check in the callers instead of duplicating them in the arch code.
Signed-off-by Christoph Hellwig <[email protected]>--- arch/s390/include/asm/pgtable.h | 8 +------- arch/x86/include/asm/pgtable_64.h | 8 +------- mm/gup.c | 17 +++++++---------- 3 files changed, 9 insertions(+), 24 deletions(-)
Much cleaner
Jason
PATCH 3/16 mm: lift the x86_32 PAE version of gup_get_pte to common code
In reply to Christoph HellwigThe split low/high access is the only non-READ_ONCE version of gup_get_pte that did show up in the various arch implemenations. Lift it to common code and drop the ifdef based arch override.
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable-3level.h | 47 ------------------------ arch/x86/kvm/mmu.c | 2 +- mm/Kconfig | 3 ++ mm/gup.c | 51 ++++++++++++++++++++++++--- 5 files changed, 52 insertions(+), 52 deletions(-)
arch/x86/Kconfig +1 -0
@@ -121,6 +121,7 @@ config X86 select GENERIC_STRNCPY_FROM_USER select GENERIC_STRNLEN_USER select GENERIC_TIME_VSYSCALL + select GUP_GET_PTE_LOW_HIGH if X86_PAE select HARDLOCKUP_CHECK_TIMESTAMP if X86_64 select HAVE_ACPI_APEI if ACPI select HAVE_ACPI_APEI_NMI if ACPI
arch/x86/include/asm/pgtable-3level.h +0 -47
53 lines not on this page · raw message
arch/x86/kvm/mmu.c +1 -1
@@ -653,7 +653,7 @@ static u64 __update_clear_spte_slow(u64 *sptep, u64 spte)/* * The idea using the light way get the spte on x86_32 guest is from -* gup_get_pte(arch/x86/mm/gup.c). +* gup_get_pte (mm/gup.c). * * An spte tlb flush may be pending, because kvm_set_pte_rmapp * coalesces them and we are running out of the MMU lock. Therefore
mm/Kconfig +3 -0
@@ -762,6 +762,9 @@ config GUP_BENCHMARK See tools/testing/selftests/vm/gup_benchmark.c +config GUP_GET_PTE_LOW_HIGH + bool + config ARCH_HAS_PTE_SPECIAL bool
mm/gup.c +47 -4
64 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH 03/16] mm: lift the x86_32 PAE version of gup_get_pte to common code
Quoted 4 lines · Christoph Hellwig
The split low/high access is the only non-READ_ONCE version of
gup_get_ptethat did show up in the various arch implemenations. Lift it to common code and drop the ifdef based arch override.
Quoted 8 lines
Signed-off-by Christoph Hellwig <[email protected]>--- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable-3level.h | 47 ------------------------ arch/x86/kvm/mmu.c | 2 +- mm/Kconfig | 3 ++ mm/gup.c | 51 ++++++++++++++++++++++++--- 5 files changed, 52 insertions(+), 52 deletions(-)
Yep, the sh and mips conversions look right too.
mm/Kconfig config GUP_BENCHMARK · line 767
Quoted 11 lines
mm/Kconfig +3
-0@@ -762,6 +762,9 @@ config GUP_BENCHMARK See tools/testing/selftests/vm/gup_benchmark.c +config GUP_GET_PTE_LOW_HIGH + bool +
The config name seems a bit out of place though, should it be prefixed with GENERIC_ or ARCH_?
Jason
PATCH 4/16 MIPS: use the generic get_user_pages_fast code
In reply to Christoph HellwigThe mips code is mostly equivalent to the generic one, minus various bugfixes and an arch override for gup_fast_permitted.
Note that this defines ARCH_HAS_PTE_SPECIAL for mips as mips has pte_special and pte_mkspecial implemented and used in the existing gup code. They are no-op stubs, though which makes me a little unsure if this is really right thing to do.
--- arch/mips/Kconfig | 3 + arch/mips/include/asm/pgtable.h | 3 + arch/mips/mm/Makefile | 1 - arch/mips/mm/gup.c | 303 -------------------------------- 4 files changed, 6 insertions(+), 304 deletions(-) delete mode 100644 arch/mips/mm/gup.c
arch/mips/Kconfig +3 -0
@@ -6,6 +6,7 @@ config MIPS select ARCH_BINFMT_ELF_STATE if MIPS_FP_SUPPORT select ARCH_CLOCKSOURCE_DATA select ARCH_HAS_ELF_RANDOMIZE + select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST select ARCH_HAS_UBSAN_SANITIZE_ALL select ARCH_SUPPORTS_UPROBES
@@ -34,6 +35,7 @@ config MIPS select GENERIC_SCHED_CLOCK if !CAVIUM_OCTEON_SOC select GENERIC_SMP_IDLE_THREAD select GENERIC_TIME_VSYSCALL + select GUP_GET_PTE_LOW_HIGH if CPU_MIPS32 && PHYS_ADDR_T_64BIT select HANDLE_DOMAIN_IRQ select HAVE_ARCH_COMPILER_H select HAVE_ARCH_JUMP_LABEL
@@ -55,6 +57,7 @@ config MIPS select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER + select HAVE_GENERIC_GUP select HAVE_IDE select HAVE_IOREMAP_PROT select HAVE_IRQ_EXIT_ON_IRQ_STACK
arch/mips/include/asm/pgtable.h +3 -0
@@ -20,6 +20,7 @@#include <asm/cmpxchg.h> #include <asm/io.h> #include <asm/pgtable-bits.h> +#include <asm/cpu-features.h> struct mm_struct ;struct vm_area_struct ;
@@ -626,6 +627,8 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#define gup_fast_permitted (start, end) (!cpu_has_dc_aliases) +#include <asm-generic/pgtable.h> /*
arch/mips/mm/Makefile +0 -1
@@ -7,7 +7,6 @@ obj-y += cache.o obj-y += context.o obj-y += extable.o obj-y += fault.o -obj-y += gup.o obj-y += init.o obj-y += mmap.o obj-y += page.o
arch/mips/mm/gup.c deleted +0 -303
303 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH 04/16] MIPS: use the generic get_user_pages_fast code
Quoted 12 lines · Christoph Hellwig
arch/mips/include/asm/pgtable.h +2
-0+++ b/arch/mips/include/asm/pgtable.h@@ -20,6 +20,7 @@#include <asm/cmpxchg.h> #include <asm/io.h> #include <asm/pgtable-bits.h> +#include <asm/cpu-features.h> struct mm_struct ;struct vm_area_struct ;
arch/mips/include/asm/pgtable.h static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, · line 631
Quoted 5 lines
arch/mips/include/asm/pgtable.h +2
-0@@ -626,6 +627,8 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#define gup_fast_permitted (start, end) (!cpu_has_dc_aliases) +
Today this check is only being done on the get_user_pages_fast() - after this patch it is also done for __get_user_pages_fast().
Which means __get_user_pages_fast is now non-functional on a range of MIPS CPUs, but that seems OK as far as I can tell, so:
However, looks to me like this patch is also a bug fix for this:
commit 5b167c123b3c3582f62cf1896465019bc40fe526
Author: Kamal Dasu <[email protected]> Date: Fri Jun 14 17:10:03 2013 +0000
MIPS: Fix get_user_page_fast() for mips with cache alias
get_user_pages_fast() is missing cache flushes for MIPS platforms with cache aliases. Filesystem failures observed with DirectIO operations due to missing flush_anon_page() that use page coloring logic to work with cache aliases. This fix falls through to take slow_irqon path that calls get_user_pages() that has required logic for platforms where cpu_has_dc_aliases is true.
arch/mips/include/asm/pgtable.h static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,
Quoted 9 lines
arch/mips/include/asm/pgtable.h +0
-9- pgdp =pgd_offset (mm, addr); - do { - pgd_t pgd = *pgdp; - - next = pgd_addr_end(addr, end); - if (pgd_none(pgd)) - goto slow; - if (!gup_pud_range(pgd, addr, next, gup_flags & FOLL_WRITE, - pages, &nr))
This is different too, the core code has a p4d layer, but I see that whole thing gets NOP'd by the compiler as mips uses pgtable-nop4d.h - right?
Jason
Re: [PATCH 04/16] MIPS: use the generic get_user_pages_fast code
Quoted 7 lines · Jason Gunthorpe
Today this check is only being done on the
get_user_pages_fast()- after this patch it is also done for__get_user_pages_fast().Which means
__get_user_pages_fastis now non-functional on a range of MIPS CPUs, but that seems OK as far as I can tell, so:However, looks to me like this patch is also a bug fix for this:
Yes.
Quoted 5 lines
Quoted 5 lines
arch/mips/mm/gup.c deleted +0
-5- pgdp =pgd_offset (mm, addr); - do { - pgd_t pgd = *pgdp; - - next = pgd_addr_end(addr, end);
Quoted 8 lines
arch/mips/mm/gup.c line 277
Quoted 4 lines
arch/mips/mm/gup.c deleted +0
-4-if (pgd_none (pgd)) - goto slow; -if (!gup_pud_range (pgd, addr, next, gup_flags &FOLL_WRITE , - pages, &nr))This is different too, the core code has a p4d layer, but I see that whole thing gets NOP'd by the compiler as mips uses
pgtable-nop4d.h- right?
Exactly.
PATCH 5/16 sh: add the missing pud_page definition
In reply to Christoph Hellwigsh only had pud_page_vaddr, but not pud_page.
--- arch/sh/include/asm/pgtable-3level.h | 1 + 1 file changed, 1 insertion(+)
arch/sh/include/asm/pgtable-3level.h +1 -0
@@ -37,6 +37,7 @@ static inline unsigned long pud_page_vaddr(pud_t pud) {return pud_val (pud); } +#define pud_page (pud) pfn_to_page(pud_pfn(pud))#define pmd_index (address) (((address) >> PMD_SHIFT) & (PTRS_PER_PMD-1))static inline pmd_t *pmd_offset (pud_t *pud,unsigned long address)
Signature · 1 line
2.20.1
PATCH 6/16 sh: use the generic get_user_pages_fast code
In reply to Christoph HellwigThe sh code is mostly equivalent to the generic one, minus various bugfixes and two arch overrides that this patch adds to pgtable.h.
--- arch/sh/Kconfig | 2 + arch/sh/include/asm/pgtable.h | 37 +++++ arch/sh/mm/Makefile | 2 +- arch/sh/mm/gup.c | 277 ---------------------------------- 4 files changed, 40 insertions(+), 278 deletions(-) delete mode 100644 arch/sh/mm/gup.c
arch/sh/Kconfig +2 -0
@@ -14,6 +14,7 @@ config SUPERH select HAVE_ARCH_TRACEHOOK select HAVE_PERF_EVENTS select HAVE_DEBUG_BUGVERBOSE + select HAVE_GENERIC_GUP select ARCH_HAVE_CUSTOM_GPIO_H select ARCH_HAVE_NMI_SAFE_CMPXCHG if (GUSA_RB || CPU_SH4A) select ARCH_HAS_GCOV_PROFILE_ALL
@@ -63,6 +64,7 @@ config SUPERH config SUPERH32 def_bool "$(ARCH)" = "sh" select ARCH_32BIT_OFF_T + select GUP_GET_PTE_LOW_HIGH if X2TLB select HAVE_KPROBES select HAVE_KRETPROBES select HAVE_IOREMAP_PROT if MMU && !X2TLB
arch/sh/include/asm/pgtable.h +37 -0
@@ -149,6 +149,43 @@ extern void paging_init(void);extern void page_table_range_init (unsigned long start,unsigned long end,pgd_t *pgd); +static inline bool __pte_access_permitted (pte_t pte,u64 prot) +{ +return (pte_val (pte) & (prot | _PAGE_SPECIAL)) = prot; +} + +#ifdef CONFIG_X2TLB +static inline bool pte_access_permitted (pte_t pte,bool write) +{ +u64 prot = _PAGE_PRESENT; + + prot |=_PAGE_EXT (_PAGE_EXT_KERN_READ | _PAGE_EXT_USER_READ); +if (write) + prot |=_PAGE_EXT (_PAGE_EXT_KERN_WRITE | _PAGE_EXT_USER_WRITE); +return __pte_access_permitted (pte, prot); +} +#elif defined(CONFIG_SUPERH64 ) +static inline bool pte_access_permitted (pte_t pte,bool write) +{ +u64 prot = _PAGE_PRESENT | _PAGE_USER | _PAGE_READ; + +if (write) + prot |= _PAGE_WRITE; +return __pte_access_permitted (pte, prot); +} +#else +static inline bool pte_access_permitted (pte_t pte,bool write) +{ +u64 prot = _PAGE_PRESENT | _PAGE_USER; + +if (write) + prot |= _PAGE_RW; +return __pte_access_permitted (pte, prot); +} +#endif + +#define pte_access_permitted pte_access_permitted +/* arch/sh/mm/mmap.c */ #define HAVE_ARCH_UNMAPPED_AREA #define HAVE_ARCH_UNMAPPED_AREA_TOPDOWN
arch/sh/mm/Makefile +1 -1
@@ -17,7 +17,7 @@ cacheops-$(CONFIG_CPU_SHX3) += cache-shx3.o obj-y += $(cacheops-y) mmu-y := nommu.o extable_32.o -mmu-$(CONFIG_MMU) := extable_$(BITS).o fault.o gup.o ioremap.o kmap.o \ +mmu-$(CONFIG_MMU) := extable_$(BITS).o fault.o ioremap.o kmap.o \ pgtable.o tlbex_$(BITS).o tlbflush_$(BITS).o obj-y += $(mmu-y)
arch/sh/mm/gup.c deleted +0 -277
277 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH 7/16 sparc64: add the missing pgd_page definition
In reply to Christoph Hellwigsparc64 only had pgd_page_vaddr, but not pgd_page.
--- arch/sparc/include/asm/pgtable_64.h | 1 + 1 file changed, 1 insertion(+)
arch/sparc/include/asm/pgtable_64.h +1 -0
@@ -861,6 +861,7 @@ static inline unsigned long pud_page_vaddr(pud_t pud)#define pud_clear (pudp) (pud_val(*(pudp)) = 0UL)#define pgd_page_vaddr (pgd) \ ((unsigned long) __va(pgd_val(pgd))) +#define pgd_page (pgd) pfn_to_page(pgd_pfn(pgd))#define pgd_present (pgd) (pgd_val(pgd) != 0U)#define pgd_clear (pgdp) (pgd_val(*(pgdp)) = 0UL)
Signature · 1 line
2.20.1
PATCH 8/16 sparc64: define untagged_addr()
In reply to Christoph HellwigAdd a helper to untag a user pointer. This is needed for ADI support in get_user_pages_fast.
--- arch/sparc/include/asm/pgtable_64.h | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
arch/sparc/include/asm/pgtable_64.h +22 -0
@@ -1076,6 +1076,28 @@ static inline int io_remap_pfn_range(struct vm_area_struct *vma, }#define io_remap_pfn_range io_remap_pfn_range +static inline unsigned long untagged_addr (unsigned long start) +{ +if (adi_capable ()) { +long addr = start; + +/* If userspace has passed a versioned address, kernel +* will not find it in the VMAs since it does not store +* the version tags in the list of VMAs. Storing version +* tags in list of VMAs is impractical since they can be +* changed any time from userspace without dropping into +* kernel. Any address search in VMAs will be done with +* non-versioned addresses. Ensure the ADI version bits +* are dropped here by sign extending the last bit before +* ADI bits. IOMMU does not implement version tags. +*/ +return (addr << (long )adi_nbits ()) >> (long )adi_nbits (); + } + +return start; +} +#define untagged_addr untagged_addr +#include <asm/tlbflush.h> #include <asm-generic/pgtable.h>
Signature · 1 line
2.20.1
Re: [PATCH 08/16] sparc64: define untagged_addr()
Quoted 7 lines · Christoph Hellwig
Add a helper to untag a user pointer. This is needed for ADI support in
get_user_pages_fast.Signed-off-by Christoph Hellwig <[email protected]>--- arch/sparc/include/asm/pgtable_64.h | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
Looks good to me.
Quoted 35 lines
arch/sparc/include/asm/pgtable_64.h +22
-0@@ -1076,6 +1076,28 @@ static inline int io_remap_pfn_range(struct vm_area_struct *vma, }#define io_remap_pfn_range io_remap_pfn_range +static inline unsigned long untagged_addr (unsigned long start) +{ +if (adi_capable ()) { +long addr = start; + +/* If userspace has passed a versioned address, kernel +* will not find it in the VMAs since it does not store +* the version tags in the list of VMAs. Storing version +* tags in list of VMAs is impractical since they can be +* changed any time from userspace without dropping into +* kernel. Any address search in VMAs will be done with +* non-versioned addresses. Ensure the ADI version bits +* are dropped here by sign extending the last bit before +* ADI bits. IOMMU does not implement version tags. +*/ +return (addr << (long )adi_nbits ()) >> (long )adi_nbits (); + } + +return start; +} +#define untagged_addr untagged_addr +#include <asm/tlbflush.h> #include <asm-generic/pgtable.h>
PATCH 9/16 sparc64: use the generic get_user_pages_fast code
In reply to Christoph HellwigThe sparc64 code is mostly equivalent to the generic one, minus various bugfixes and two arch overrides that this patch adds to pgtable.h.
--- arch/sparc/Kconfig | 1 + arch/sparc/include/asm/pgtable_64.h | 18 ++ arch/sparc/mm/Makefile | 2 +- arch/sparc/mm/gup.c | 340 ---------------------------- 4 files changed, 20 insertions(+), 341 deletions(-) delete mode 100644 arch/sparc/mm/gup.c
arch/sparc/Kconfig +1 -0
@@ -28,6 +28,7 @@ config SPARC select RTC_DRV_M48T59 select RTC_SYSTOHC select HAVE_ARCH_JUMP_LABEL if SPARC64 + select HAVE_GENERIC_GUP if SPARC64 select GENERIC_IRQ_SHOW select ARCH_WANT_IPC_PARSE_VERSION select GENERIC_PCI_IOMAP
arch/sparc/include/asm/pgtable_64.h +18 -0
@@ -1098,6 +1098,24 @@ static inline unsigned long untagged_addr(unsigned long start) }#define untagged_addr untagged_addr +static inline bool pte_access_permitted (pte_t pte,bool write) +{ +u64 prot; + +if (tlb_type = hypervisor) { + prot = _PAGE_PRESENT_4V | _PAGE_P_4V; +if (write) + prot |= _PAGE_WRITE_4V; + }else { + prot = _PAGE_PRESENT_4U | _PAGE_P_4U; +if (write) + prot |= _PAGE_WRITE_4U; + } + +return (pte_val (pte) & (prot | _PAGE_SPECIAL)) = prot; +} +#define pte_access_permitted pte_access_permitted +#include <asm/tlbflush.h> #include <asm-generic/pgtable.h>
arch/sparc/mm/Makefile +1 -1
@@ -5,7 +5,7 @@ asflags-y := -ansi ccflags-y := -Werror -obj-$(CONFIG_SPARC64) += ultra.o tlb.o tsb.o gup.o +obj-$(CONFIG_SPARC64) += ultra.o tlb.o tsb.o obj-y += fault_$(BITS).o obj-y += init_$(BITS).o obj-$(CONFIG_SPARC32) += extable.o srmmu.o iommu.o io-unit.o
arch/sparc/mm/gup.c deleted +0 -340
340 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH 09/16] sparc64: use the generic get_user_pages_fast code
Quoted 12 lines · Christoph Hellwig
The sparc64 code is mostly equivalent to the generic one, minus various bugfixes and two arch overrides that this patch adds to
pgtable.h.Signed-off-by Christoph Hellwig <[email protected]>--- arch/sparc/Kconfig | 1 + arch/sparc/include/asm/pgtable_64.h | 18 ++ arch/sparc/mm/Makefile | 2 +- arch/sparc/mm/gup.c | 340 ---------------------------- 4 files changed, 20 insertions(+), 341 deletions(-) delete mode 100644 arch/sparc/mm/gup.c
PATCH 10/16 mm: rename CONFIG_HAVE_GENERIC_GUP to CONFIG_HAVE_FAST_GUP
In reply to Christoph HellwigWe only support the generic GUP now, so rename the config option to be more clear, and always use the mm/Kconfig definition of the symbol and select it from the arch Kconfigs.
--- arch/arm/Kconfig | 5 +---- arch/arm64/Kconfig | 4 +--- arch/mips/Kconfig | 2 +- arch/powerpc/Kconfig | 2 +- arch/s390/Kconfig | 2 +- arch/sh/Kconfig | 2 +- arch/sparc/Kconfig | 2 +- arch/x86/Kconfig | 4 +--- mm/Kconfig | 2 +- mm/gup.c | 4 ++-- 10 files changed, 11 insertions(+), 18 deletions(-)
arch/arm/Kconfig +1 -4
@@ -73,6 +73,7 @@ config ARM select HAVE_DYNAMIC_FTRACE_WITH_REGS if HAVE_DYNAMIC_FTRACE select HAVE_EFFICIENT_UNALIGNED_ACCESS if (CPU_V6 || CPU_V6K || CPU_V7) && MMU select HAVE_EXIT_THREAD + select HAVE_FAST_GUP if ARM_LPAE select HAVE_FTRACE_MCOUNT_RECORD if !XIP_KERNEL select HAVE_FUNCTION_GRAPH_TRACER if !THUMB2_KERNEL && !CC_IS_CLANG select HAVE_FUNCTION_TRACER if !XIP_KERNEL
@@ -1596,10 +1597,6 @@ config ARCH_SELECT_MEMORY_MODEL config HAVE_ARCH_PFN_VALID def_bool ARCH_HAS_HOLES_MEMORYMODEL || !SPARSEMEM -config HAVE_GENERIC_GUP - def_bool y - depends on ARM_LPAE - config HIGHMEM bool "High Memory Support" depends on MMU
arch/arm64/Kconfig +1 -3
@@ -140,6 +140,7 @@ config ARM64 select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_EFFICIENT_UNALIGNED_ACCESS + select HAVE_FAST_GUP select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_TRACER select HAVE_FUNCTION_GRAPH_TRACER
@@ -262,9 +263,6 @@ config GENERIC_CALIBRATE_DELAY config ZONE_DMA32 def_bool y -config HAVE_GENERIC_GUP - def_bool y - config ARCH_ENABLE_MEMORY_HOTPLUG def_bool y
arch/mips/Kconfig +1 -1
@@ -54,10 +54,10 @@ config MIPS select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_EXIT_THREAD + select HAVE_FAST_GUP select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER - select HAVE_GENERIC_GUP select HAVE_IDE select HAVE_IOREMAP_PROT select HAVE_IRQ_EXIT_ON_IRQ_STACK
arch/powerpc/Kconfig +1 -1
@@ -185,12 +185,12 @@ config PPC select HAVE_DYNAMIC_FTRACE_WITH_REGS if MPROFILE_KERNEL select HAVE_EBPF_JIT if PPC64 select HAVE_EFFICIENT_UNALIGNED_ACCESS if !(CPU_LITTLE_ENDIAN && POWER7_CPU) + select HAVE_FAST_GUP select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_ERROR_INJECTION select HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER select HAVE_GCC_PLUGINS if GCC_VERSION >= 50200 # plugin support on gcc <= 5.1 is buggy on PPC - select HAVE_GENERIC_GUP select HAVE_HW_BREAKPOINT if PERF_EVENTS && (PPC_BOOK3S || PPC_8xx) select HAVE_IDE select HAVE_IOREMAP_PROT
arch/s390/Kconfig +1 -1
@@ -137,6 +137,7 @@ config S390 select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_REGS + select HAVE_FAST_GUP select HAVE_EFFICIENT_UNALIGNED_ACCESS select HAVE_FENTRY select HAVE_FTRACE_MCOUNT_RECORD
@@ -144,7 +145,6 @@ config S390 select HAVE_FUNCTION_TRACER select HAVE_FUTEX_CMPXCHG if FUTEX select HAVE_GCC_PLUGINS - select HAVE_GENERIC_GUP select HAVE_KERNEL_BZIP2 select HAVE_KERNEL_GZIP select HAVE_KERNEL_LZ4
arch/sh/Kconfig +1 -1
@@ -14,7 +14,7 @@ config SUPERH select HAVE_ARCH_TRACEHOOK select HAVE_PERF_EVENTS select HAVE_DEBUG_BUGVERBOSE - select HAVE_GENERIC_GUP + select HAVE_FAST_GUP select ARCH_HAVE_CUSTOM_GPIO_H select ARCH_HAVE_NMI_SAFE_CMPXCHG if (GUSA_RB || CPU_SH4A) select ARCH_HAS_GCOV_PROFILE_ALL
arch/sparc/Kconfig +1 -1
@@ -28,7 +28,7 @@ config SPARC select RTC_DRV_M48T59 select RTC_SYSTOHC select HAVE_ARCH_JUMP_LABEL if SPARC64 - select HAVE_GENERIC_GUP if SPARC64 + select HAVE_FAST_GUP if SPARC64 select GENERIC_IRQ_SHOW select ARCH_WANT_IPC_PARSE_VERSION select GENERIC_PCI_IOMAP
arch/x86/Kconfig +1 -3
@@ -157,6 +157,7 @@ config X86 select HAVE_EFFICIENT_UNALIGNED_ACCESS select HAVE_EISA select HAVE_EXIT_THREAD + select HAVE_FAST_GUP select HAVE_FENTRY if X86_64 || DYNAMIC_FTRACE select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_GRAPH_TRACER
@@ -2874,9 +2875,6 @@ config HAVE_ATOMIC_IOMAP config X86_DEV_DMA_OPS bool -config HAVE_GENERIC_GUP - def_bool y - source "drivers/firmware/Kconfig" source "arch/x86/kvm/Kconfig"
mm/Kconfig +1 -1
@@ -132,7 +132,7 @@ config HAVE_MEMBLOCK_NODE_MAP config HAVE_MEMBLOCK_PHYS_MAP bool -config HAVE_GENERIC_GUP +config HAVE_FAST_GUP bool config ARCH_KEEP_MEMBLOCK
mm/gup.c +2 -2
@@ -1651,7 +1651,7 @@ struct page *get_dump_page(unsigned long addr)#endif /* CONFIG_ELF_CORE */ /* -* Generic Fast GUP +* Fast GUP * * get_user_pages_fast attempts to pin user pages by walking the page * tables directly and avoids taking locks. Thus the walker needs to be
@@ -1683,7 +1683,7 @@ struct page *get_dump_page(unsigned long addr)* * This code is based heavily on the PowerPC implementation by Nick Piggin. */ -#ifdef CONFIG_HAVE_GENERIC_GUP +#ifdef CONFIG_HAVE_FAST_GUP #ifdef CONFIG_GUP_GET_PTE_LOW_HIGH /* * WARNING: only to be used in the get_user_pages_fast() implementation.
Signature · 1 line
2.20.1
Re: [PATCH 10/16] mm: rename CONFIG_HAVE_GENERIC_GUP to CONFIG_HAVE_FAST_GUP
Quoted 5 lines · Christoph Hellwig
We only support the generic GUP now, so rename the config option to be more clear, and always use the mm/Kconfig definition of the symbol and select it from the arch Kconfigs.
Signed-off-by Christoph Hellwig <[email protected]>
Quoted 13 lines
--- arch/arm/Kconfig | 5 +---- arch/arm64/Kconfig | 4 +--- arch/mips/Kconfig | 2 +- arch/powerpc/Kconfig | 2 +- arch/s390/Kconfig | 2 +- arch/sh/Kconfig | 2 +- arch/sparc/Kconfig | 2 +- arch/x86/Kconfig | 4 +--- mm/Kconfig | 2 +- mm/gup.c | 4 ++-- 10 files changed, 11 insertions(+), 18 deletions(-)
Looks good.
Re: [PATCH 10/16] mm: rename CONFIG_HAVE_GENERIC_GUP to CONFIG_HAVE_FAST_GUP
In reply to Christoph HellwigQuoted 3 lines · Christoph Hellwig
We only support the generic GUP now, so rename the config option to be more clear, and always use the mm/Kconfig definition of the symbol and select it from the arch Kconfigs.
Looks OK to me
But could you also roll something like this in to the series? There is no longer any reason for the special __weak stuff that I can see - just follow the normal pattern for stubbing config controlled functions through the header file.
include/linux/mm.h +18 -0
@@ -1561,8 +1561,17 @@ long get_user_pages_locked(unsigned long start, unsigned long nr_pages,long get_user_pages_unlocked (unsigned long start,unsigned long nr_pages,struct page **pages,unsigned int gup_flags); +#ifdef CONFIG_HAVE_FAST_GUP int get_user_pages_fast (unsigned long start,int nr_pages,unsigned int gup_flags,struct page **pages); +#else +static inline int get_user_pages_fast (unsigned long start,int nr_pages, +unsigned int gup_flags, +struct page **pages) +{ +return get_user_pages_unlocked (start, nr_pages, pages, gup_flags); +} +#endif /* Container for pinned pfns / pages */ struct frame_vector {
@@ -1668,8 +1677,17 @@ extern int mprotect_fixup(struct vm_area_struct *vma,/* * doesn't attempt to fault and will return short. */ +#ifdef CONFIG_HAVE_FAST_GUP int __get_user_pages_fast (unsigned long start,int nr_pages,int write,struct page **pages); +#else +static inline int __get_user_pages_fast (unsigned long start,int nr_pages, +int write,struct page **pages) +{ +return 0 ; +} +#endif +/* * per-process(per-mm_struct) statistics. */
mm/util.c +0 -47
53 lines not on this page · raw message
Re: [PATCH 10/16] mm: rename CONFIG_HAVE_GENERIC_GUP to CONFIG_HAVE_FAST_GUP
Quoted 5 lines · Jason Gunthorpe
Quoted 3 lines · Christoph Hellwig
We only support the generic GUP now, so rename the config option to be more clear, and always use the mm/Kconfig definition of the symbol and select it from the arch Kconfigs.
Quoted 8 lines
Looks OK to me
Reviewed-by Jason Gunthorpe <[email protected]>But could you also roll something like this in to the series? There is no longer any reason for the special
__weakstuff that I can see - just follow the normal pattern for stubbing config controlled functions through the header file.
Something pretty similar is done later in this series.
PATCH 11/16 mm: consolidate the get_user_pages* implementations
In reply to Christoph HellwigAlways build mm/gup.c, and move the nommu versions and replace the separate stubs for various functions by the default ones, with the _fast version always falling back to the slow path because gup_fast_permitted always returns false now if HAVE_FAST_GUP is not set, and we use the nommu version of __get_user_pages while keeping all the wrappers common.
This also ensures the new put_user_pages* helpers are available for nommu, as those are currently missing, which would create a problem as soon as we actually grew users for it.
--- mm/Kconfig | 1 + mm/Makefile | 4 +- mm/gup.c | 476 +++++++++++++++++++++++++++++----------------------- mm/nommu.c | 88 ---------- mm/util.c | 47 ------ 5 files changed, 269 insertions(+), 347 deletions(-)
mm/Kconfig +1 -0
@@ -133,6 +133,7 @@ config HAVE_MEMBLOCK_PHYS_MAP bool config HAVE_FAST_GUP + depends on MMU bool config ARCH_KEEP_MEMBLOCK
mm/Makefile +2 -2
@@ -22,7 +22,7 @@ KCOV_INSTRUMENT_mmzone.o := n KCOV_INSTRUMENT_vmstat.o := n mmu-y := nommu.o -mmu-$(CONFIG_MMU) := gup.o highmem.o memory.o mincore.o \ +mmu-$(CONFIG_MMU) := highmem.o memory.o mincore.o \ mlock.o mmap.o mmu_gather.o mprotect.o mremap.o \ msync.o page_vma_mapped.o pagewalk.o \ pgtable-generic.o rmap.o vmalloc.o
@@ -39,7 +39,7 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ mm_init.o mmu_context.o percpu.o slab_common.o \ compaction.o vmacache.o \ interval_tree.o list_lru.o workingset.o \ - debug.o $(mmu-y) + debug.o gup.o $(mmu-y) # Give 'page_alloc' its own module-parameter namespace page-alloc-y := page_alloc.o
mm/gup.c +266 -210
560 lines not on this page · raw message
mm/nommu.c +0 -88
94 lines not on this page · raw message
mm/util.c +0 -47
53 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH 11/16] mm: consolidate the get_user_pages* implementations
mm/gup.c static void gup_pgd_range(unsigned long addr, unsigned long end, · line 2224
Quoted 6 lines · Christoph Hellwig
mm/gup.c +1
-1@@ -2168,7 +2221,7 @@ static void gup_pgd_range(unsigned long addr, unsigned long end,*/ static bool gup_fast_permitted (unsigned long start,unsigned long end) { -return true; +return IS_ENABLED (CONFIG_HAVE_FAST_GUP ) ? true : false;
The ?: is needed with IS_ENABLED?
Review static void gup_pgd_range(unsigned long addr, unsigned long end, · line 2226
Quoted 2 lines
(quoted diff) +0
-0} #endif
Oh, you fixed the util.c this way instead of the headerfile #ifdef..
I'd suggest to revise this block a tiny bit:
-#ifndef gup_fast_permitted
+#if !IS_ENABLED(CONFIG_HAVE_FAST_GUP) || !defined(gup_fast_permitted)
/*
* Check if it's allowed to use __get_user_pages_fast() for the range, or
* we need to fall back to the slow version:
*/
-bool gup_fast_permitted(unsigned long start, int nr_pages) +static bool gup_fast_permitted(unsigned long start, int nr_pages)
{
Just in case some future arch code mismatches the header and kconfig..
Regards,
Jason
Re: [PATCH 11/16] mm: consolidate the get_user_pages* implementations
Quoted 6 lines · Jason Gunthorpe
mm/gup.c static void gup_pgd_range(unsigned long addr, unsigned long end, · line 2224
Quoted 4 lines
static bool
gup_fast_permitted(unsigned long start, unsigned long end){mm/gup.c +1
-1-return true; +return IS_ENABLED (CONFIG_HAVE_FAST_GUP ) ? true : false;The ?: is needed with
IS_ENABLED?
It shouldn't, I'll fix it up.
Quoted 12 lines
I'd suggest to revise this block a tiny bit:
(quoted diff) +2
-2-#ifndef gup_fast_permitted +#if !IS_ENABLED(CONFIG_HAVE_FAST_GUP) || !defined(gup_fast_permitted) /* * Check if it's allowed to use __get_user_pages_fast() for the range, or * we need to fall back to the slow version: */ -bool gup_fast_permitted(unsigned long start, int nr_pages) +static bool gup_fast_permitted(unsigned long start, int nr_pages) {
Quoted 1 line
Just in case some future arch code mismatches the header and kconfig..
IS_ENABLED outside a function doesn't really make sense. But I'll just life the IS_ENABLED(CONFIG_HAVE_FAST_GUP) checks into the two callers.
Re: [PATCH 11/16] mm: consolidate the get_user_pages* implementations
Quoted 22 lines · Christoph Hellwig
Quoted 6 lines · Jason Gunthorpe
mm/gup.c static void gup_pgd_range(unsigned long addr, unsigned long end, · line 2224
Quoted 4 lines
static bool
gup_fast_permitted(unsigned long start, unsigned long end){mm/gup.c +1
-1-return true; +return IS_ENABLED (CONFIG_HAVE_FAST_GUP ) ? true : false;The ?: is needed with
IS_ENABLED?It shouldn't, I'll fix it up.
Quoted 12 lines
I'd suggest to revise this block a tiny bit:
(quoted diff) +2
-2-#ifndef gup_fast_permitted +#if !IS_ENABLED(CONFIG_HAVE_FAST_GUP) || !defined(gup_fast_permitted) /* * Check if it's allowed to use __get_user_pages_fast() for the range, or * we need to fall back to the slow version: */ -bool gup_fast_permitted(unsigned long start, int nr_pages) +static bool gup_fast_permitted(unsigned long start, int nr_pages) {
Quoted 5 lines
Quoted 1 line
Just in case some future arch code mismatches the header and kconfig..
IS_ENABLEDoutside a function doesn't really make sense. But I'll just life theIS_ENABLED(CONFIG_HAVE_FAST_GUP)checks into the two callers.
I often see '#if IS_ENABLED(CONFIG_X)', IIRC last I looked at that, it was needed because the usual #ifdef CONFIG_X didn't work if the value was =m?
Would be interested to know if that is not the right way to use kconfig
Jason
PATCH 12/16 mm: validate get_user_pages_fast flags
In reply to Christoph HellwigWe can only deal with FOLL_WRITE and/or FOLL_LONGTERM in get_user_pages_fast, so reject all other flags.
--- mm/gup.c | 3 +++ 1 file changed, 3 insertions(+)
mm/gup.c +3 -0
@@ -2317,6 +2317,9 @@ int get_user_pages_fast(unsigned long start, int nr_pages,unsigned long addr, len, end;int nr =0 , ret =0 ; +if (WARN_ON_ONCE (gup_flags & ~(FOLL_WRITE |FOLL_LONGTERM ))) +return -EINVAL ; + start =untagged_addr (start) &PAGE_MASK ; addr = start; len = (unsigned long ) nr_pages <<PAGE_SHIFT ;
Signature · 1 line
2.20.1
PATCH 13/16 mm: move the powerpc hugepd code to mm/gup.c
In reply to Christoph HellwigWhile only powerpc supports the hugepd case, the code is pretty generic and I'd like to keep all GUP internals in one place.
--- arch/powerpc/Kconfig | 1 + arch/powerpc/mm/hugetlbpage.c | 72 ------------------------------ include/linux/hugetlb.h | 18 -------- mm/Kconfig | 10 +++++ mm/gup.c | 82 +++++++++++++++++++++++++++++++++++ 5 files changed, 93 insertions(+), 90 deletions(-)
arch/powerpc/Kconfig +1 -0
@@ -125,6 +125,7 @@ config PPC select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_GCOV_PROFILE_ALL select ARCH_HAS_KCOV + select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_HAS_MMIOWB if PPC64 select ARCH_HAS_PHYS_TO_DMA select ARCH_HAS_PMEM_API if PPC64
arch/powerpc/mm/hugetlbpage.c +0 -72
81 lines not on this page · raw message
include/linux/hugetlb.h +0 -18
@@ -16,29 +16,11 @@ struct user_struct;struct mmu_gather ;#ifndef is_hugepd -/* -* Some architectures requires a hugepage directory format that is -* required to support multiple hugepage sizes. For example -* a4fe3ce76 "powerpc/mm: Allow more flexible layouts for hugepage pagetables" -* introduced the same on powerpc. This allows for a more flexible hugepage -* pagetable layout. -*/ typedef struct {unsigned long pd; }hugepd_t ;#define is_hugepd (hugepd) (0)#define __hugepd (x) ((hugepd_t) { (x) }) -static inline int gup_huge_pd (hugepd_t hugepd,unsigned long addr, -unsigned pdshift,unsigned long end, -int write,struct page **pages,int *nr) -{ -return 0 ; -} -#else -extern int gup_huge_pd (hugepd_t hugepd,unsigned long addr, -unsigned pdshift,unsigned long end, -int write,struct page **pages,int *nr);#endif -#ifdef CONFIG_HUGETLB_PAGE #include <linux/mempolicy.h>
mm/Kconfig +10 -0
@@ -769,4 +769,14 @@ config GUP_GET_PTE_LOW_HIGH config ARCH_HAS_PTE_SPECIAL bool +# +# Some architectures require a special hugepage directory format that is +# required to support multiple hugepage sizes. For example a4fe3ce76 +# "powerpc/mm: Allow more flexible layouts for hugepage pagetables" +# introduced it on powerpc. This allows for a more flexible hugepage +# pagetable layouts. +# +config ARCH_HAS_HUGEPD + bool + endmenu
mm/gup.c +82 -0
88 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH 14/16 mm: switch gup_hugepte to use try_get_compound_head
In reply to Christoph HellwigThis applies the overflow fixes from 8fde12ca79aff ("mm: prevent get_user_pages() from overflowing page refcount") to the powerpc hugepd code and brings it back in sync with the other GUP cases.
--- mm/gup.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/gup.c +2 -1
@@ -2007,7 +2007,8 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, refs++; }while (addr +=PAGE_SIZE , addr != end); -if (!page_cache_add_speculative (head, refs)) { + head =try_get_compound_head (head, refs); +if (!head) { *nr -= refs;return 0 ; }
Signature · 1 line
2.20.1
PATCH 15/16 mm: mark the page referenced in gup_hugepte
In reply to Christoph HellwigAll other get_user_page_fast cases mark the page referenced, so do this here as well.
--- mm/gup.c | 1 + 1 file changed, 1 insertion(+)
mm/gup.c +1 -0
@@ -2021,6 +2021,7 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr,return 0 ; } +SetPageReferenced (head);return 1 ; }
Signature · 1 line
2.20.1
PATCH 16/16 mm: pass get_user_pages_fast iterator arguments in a structure
In reply to Christoph HellwigInstead of passing a set of always repeated arguments down the get_user_pages_fast iterators, create a struct gup_args to hold them and pass that by reference. This leads to an over 100 byte .text size reduction for x86-64.
--- mm/gup.c | 338 ++++++++++++++++++++++++++----------------------------- 1 file changed, 158 insertions(+), 180 deletions(-)
mm/gup.c +158 -180
@@ -24,6 +24,13 @@#include "internal.h" +struct gup_args { +unsigned long addr; +unsigned int flags; +struct page **pages; +unsigned int nr; +}; +struct follow_page_context {struct dev_pagemap *pgmap;unsigned int page_mask;
@@ -1786,10 +1793,10 @@ static inline pte_t gup_get_pte(pte_t *ptep) }#endif /* CONFIG_GUP_GET_PTE_LOW_HIGH */ -static void undo_dev_pagemap (int *nr,int nr_start,struct page **pages) +static void undo_dev_pagemap (struct gup_args *args,int nr_start) { -while ((*nr) - nr_start) { -struct page *page = pages[--(*nr)]; +while (args->nr - nr_start) { +struct page *page = args->pages[--args->nr];ClearPageReferenced (page);put_page (page);
@@ -1811,14 +1818,13 @@ static inline struct page *try_get_compound_head(struct page *page, int refs) }#ifdef CONFIG_ARCH_HAS_PTE_SPECIAL -static int gup_pte_range (pmd_t pmd,unsigned long addr,unsigned long end, -unsigned int flags,struct page **pages,int *nr) +static int gup_pte_range (struct gup_args *args,pmd_t pmd,unsigned long end) {struct dev_pagemap *pgmap =NULL ; -int nr_start = *nr, ret =0 ; +int nr_start = args->nr, ret =0 ;pte_t *ptep, *ptem; - ptem = ptep =pte_offset_map (&pmd, addr); + ptem = ptep =pte_offset_map (&pmd, args->addr); do { pte_t pte = gup_get_pte(ptep); struct page *head, *page;
@@ -1830,16 +1836,16 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,if (pte_protnone (pte)) goto pte_unmap; -if (!pte_access_permitted (pte, flags &FOLL_WRITE )) +if (!pte_access_permitted (pte, args->flags &FOLL_WRITE )) goto pte_unmap;if (pte_devmap (pte)) { -if (unlikely (flags &FOLL_LONGTERM )) +if (unlikely (args->flags &FOLL_LONGTERM )) goto pte_unmap; pgmap =get_dev_pagemap (pte_pfn (pte), pgmap);if (unlikely (!pgmap)) { -undo_dev_pagemap (nr, nr_start, pages); +undo_dev_pagemap (args, nr_start); goto pte_unmap; } }else if (pte_special (pte))
@@ -1860,10 +1866,8 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,VM_BUG_ON_PAGE (compound_head (page) != head, page);SetPageReferenced (page); - pages[*nr] = page; - (*nr)++; - - }while (ptep++, addr +=PAGE_SIZE , addr != end); + args->pages[args->nr++] = page; + }while (ptep++, args->addr +=PAGE_SIZE , args->addr != end); ret =1 ;
@@ -1884,18 +1888,17 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,* __get_user_pages_fast implementation that can pin pages. Thus it's still * useful to have gup_huge_pmd even if we can't operate on ptes. */ -static int gup_pte_range (pmd_t pmd,unsigned long addr,unsigned long end, -unsigned int flags,struct page **pages,int *nr) +static int gup_pte_range (struct gup_args *args,pmd_t pmd,unsigned long end) {return 0 ; }#endif /* CONFIG_ARCH_HAS_PTE_SPECIAL */ #if defined(__HAVE_ARCH_PTE_DEVMAP) && defined(CONFIG_TRANSPARENT_HUGEPAGE ) -static int __gup_device_huge (unsigned long pfn,unsigned long addr, -unsigned long end,struct page **pages,int *nr) +static int __gup_device_huge (struct gup_args *args,unsigned long pfn, +unsigned long end) { -int nr_start = *nr; +int nr_start = args->nr;struct dev_pagemap *pgmap =NULL ; do {
@@ -1903,64 +1906,63 @@ static int __gup_device_huge(unsigned long pfn, unsigned long addr, pgmap =get_dev_pagemap (pfn, pgmap);if (unlikely (!pgmap)) { -undo_dev_pagemap (nr, nr_start, pages); +undo_dev_pagemap (args, nr_start);return 0 ; }SetPageReferenced (page); - pages[*nr] = page; + args->pages[args->nr++] = page;get_page (page); - (*nr)++; pfn++; - }while (addr +=PAGE_SIZE , addr != end); + }while (args->addr +=PAGE_SIZE , args->addr != end);if (pgmap)put_dev_pagemap (pgmap);return 1 ; } -static int __gup_device_huge_pmd (pmd_t orig,pmd_t *pmdp,unsigned long addr, -unsigned long end,struct page **pages,int *nr) +static int __gup_device_huge_pmd (struct gup_args *args,pmd_t orig,pmd_t *pmdp, +unsigned long end) {unsigned long fault_pfn; -int nr_start = *nr; +int nr_start = args->nr; - fault_pfn =pmd_pfn (orig) + ((addr & ~PMD_MASK ) >>PAGE_SHIFT ); -if (!__gup_device_huge (fault_pfn, addr, end, pages, nr)) + fault_pfn =pmd_pfn (orig) + ((args->addr & ~PMD_MASK ) >>PAGE_SHIFT ); +if (!__gup_device_huge (args, fault_pfn, end))return 0 ;if (unlikely (pmd_val (orig) !=pmd_val (*pmdp))) { -undo_dev_pagemap (nr, nr_start, pages); +undo_dev_pagemap (args, nr_start);return 0 ; }return 1 ; } -static int __gup_device_huge_pud (pud_t orig,pud_t *pudp,unsigned long addr, -unsigned long end,struct page **pages,int *nr) +static int __gup_device_huge_pud (struct gup_args *args,pud_t orig,pud_t *pudp, +unsigned long end) {unsigned long fault_pfn; -int nr_start = *nr; +int nr_start = args->nr; - fault_pfn =pud_pfn (orig) + ((addr & ~PUD_MASK ) >>PAGE_SHIFT ); -if (!__gup_device_huge (fault_pfn, addr, end, pages, nr)) + fault_pfn =pud_pfn (orig) + ((args->addr & ~PUD_MASK ) >>PAGE_SHIFT ); +if (!__gup_device_huge (args, fault_pfn, end))return 0 ;if (unlikely (pud_val (orig) !=pud_val (*pudp))) { -undo_dev_pagemap (nr, nr_start, pages); +undo_dev_pagemap (args, nr_start);return 0 ; }return 1 ; }#else -static int __gup_device_huge_pmd (pmd_t orig,pmd_t *pmdp,unsigned long addr, -unsigned long end,struct page **pages,int *nr) +static int __gup_device_huge_pmd (struct gup_args *args,pmd_t orig,pmd_t *pmdp, +unsigned long end) {BUILD_BUG ();return 0 ; } -static int __gup_device_huge_pud (pud_t pud,pud_t *pudp,unsigned long addr, -unsigned long end,struct page **pages,int *nr) +static int __gup_device_huge_pud (struct gup_args *args,pud_t pud,pud_t *pudp, +unsigned long end) {BUILD_BUG ();return 0 ;
@@ -1975,21 +1977,21 @@ static unsigned long hugepte_addr_end(unsigned long addr, unsigned long end,return (__boundary -1 < end -1 ) ? __boundary : end; } -static int gup_hugepte (pte_t *ptep,unsigned long sz,unsigned long addr, -unsigned long end,int write,struct page **pages,int *nr) +static int gup_hugepte (struct gup_args *args,pte_t *ptep,unsigned long sz, +unsigned long end) {unsigned long pte_end;struct page *head, *page;pte_t pte;int refs; - pte_end = (addr + sz) & ~(sz-1 ); + pte_end = (args->addr + sz) & ~(sz -1 );if (pte_end < end) end = pte_end; pte =READ_ONCE (*ptep); -if (!pte_access_permitted (pte, write)) +if (!pte_access_permitted (pte, args->flags &FOLL_WRITE ))return 0 ;/* hugepages are never "special" */
@@ -1998,24 +2000,23 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, refs =0 ; head =pte_page (pte); - page = head + ((addr & (sz-1 )) >>PAGE_SHIFT ); + page = head + ((args->addr & (sz -1 )) >>PAGE_SHIFT );do {VM_BUG_ON (compound_head (page) != head); - pages[*nr] = page; - (*nr)++; + args->pages[args->nr++] = page; page++; refs++; - }while (addr +=PAGE_SIZE , addr != end); + }while (args->addr +=PAGE_SIZE , args->addr != end); head =try_get_compound_head (head, refs);if (!head) { - *nr -= refs; + args->nr -= refs;return 0 ; }if (unlikely (pte_val (pte) !=pte_val (*ptep))) {/* Could be optimized better */ - *nr -= refs; + args->nr -= refs;while (refs--)put_page (head);return 0 ;
@@ -2025,64 +2026,61 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr,return 1 ; } -static int gup_huge_pd (hugepd_t hugepd,unsigned long addr, -unsigned int pdshift,unsigned long end,int write, -struct page **pages,int *nr) +static int gup_huge_pd (struct gup_args *args,hugepd_t hugepd,unsigned pdshift, +unsigned long end) {pte_t *ptep;unsigned long sz =1UL <<hugepd_shift (hugepd);unsigned long next; - ptep =hugepte_offset (hugepd, addr, pdshift); + ptep =hugepte_offset (hugepd, args->addr, pdshift);do { - next =hugepte_addr_end (addr, end, sz); -if (!gup_hugepte (ptep, sz, addr, end, write, pages, nr)) + next =hugepte_addr_end (args->addr, end, sz); +if (!gup_hugepte (args, ptep, sz, next))return 0 ; - }while (ptep++, addr = next, addr != end); + }while (ptep++, args->addr != end);return 1 ; }#else -static inline int gup_huge_pd (hugepd_t hugepd,unsigned long addr, -unsigned pdshift,unsigned long end,int write, -struct page **pages,int *nr) +static inline int gup_huge_pd (struct gup_args *args,hugepd_t hugepd, +unsigned pdshift,unsigned long end) {return 0 ; }#endif /* CONFIG_ARCH_HAS_HUGEPD */ -static int gup_huge_pmd (pmd_t orig,pmd_t *pmdp,unsigned long addr, -unsigned long end,unsigned int flags,struct page **pages,int *nr) +static int gup_huge_pmd (struct gup_args *args,pmd_t orig,pmd_t *pmdp, +unsigned long end) {struct page *head, *page;int refs; -if (!pmd_access_permitted (orig, flags &FOLL_WRITE )) +if (!pmd_access_permitted (orig, args->flags &FOLL_WRITE ))return 0 ;if (pmd_devmap (orig)) { -if (unlikely (flags &FOLL_LONGTERM )) +if (unlikely (args->flags &FOLL_LONGTERM ))return 0 ; -return __gup_device_huge_pmd (orig, pmdp, addr, end, pages, nr); +return __gup_device_huge_pmd (args, orig, pmdp, end); } refs =0 ; - page =pmd_page (orig) + ((addr & ~PMD_MASK ) >>PAGE_SHIFT ); + page =pmd_page (orig) + ((args->addr & ~PMD_MASK ) >>PAGE_SHIFT );do { - pages[*nr] = page; - (*nr)++; + args->pages[args->nr++] = page; page++; refs++; - }while (addr +=PAGE_SIZE , addr != end); + }while (args->addr +=PAGE_SIZE , args->addr != end); head =try_get_compound_head (pmd_page (orig), refs);if (!head) { - *nr -= refs; + args->nr -= refs;return 0 ; }if (unlikely (pmd_val (orig) !=pmd_val (*pmdp))) { - *nr -= refs; + args->nr -= refs;while (refs--)put_page (head);return 0 ;
@@ -2092,38 +2090,37 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr,return 1 ; } -static int gup_huge_pud (pud_t orig,pud_t *pudp,unsigned long addr, -unsigned long end,unsigned int flags,struct page **pages,int *nr) +static int gup_huge_pud (struct gup_args *args,pud_t orig,pud_t *pudp, +unsigned long end) {struct page *head, *page;int refs; -if (!pud_access_permitted (orig, flags &FOLL_WRITE )) +if (!pud_access_permitted (orig, args->flags &FOLL_WRITE ))return 0 ;if (pud_devmap (orig)) { -if (unlikely (flags &FOLL_LONGTERM )) +if (unlikely (args->flags &FOLL_LONGTERM ))return 0 ; -return __gup_device_huge_pud (orig, pudp, addr, end, pages, nr); +return __gup_device_huge_pud (args, orig, pudp, end); } refs =0 ; - page =pud_page (orig) + ((addr & ~PUD_MASK ) >>PAGE_SHIFT ); + page =pud_page (orig) + ((args->addr & ~PUD_MASK ) >>PAGE_SHIFT );do { - pages[*nr] = page; - (*nr)++; + args->pages[args->nr++] = page; page++; refs++; - }while (addr +=PAGE_SIZE , addr != end); + }while (args->addr +=PAGE_SIZE , args->addr != end); head =try_get_compound_head (pud_page (orig), refs);if (!head) { - *nr -= refs; + args->nr -= refs;return 0 ; }if (unlikely (pud_val (orig) !=pud_val (*pudp))) { - *nr -= refs; + args->nr -= refs;while (refs--)put_page (head);return 0 ;
@@ -2133,34 +2130,32 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr,return 1 ; } -static int gup_huge_pgd (pgd_t orig,pgd_t *pgdp,unsigned long addr, -unsigned long end,unsigned int flags, -struct page **pages,int *nr) +static int gup_huge_pgd (struct gup_args *args,pgd_t orig,pgd_t *pgdp, +unsigned long end) { -int refs;struct page *head, *page; +int refs; -if (!pgd_access_permitted (orig, flags &FOLL_WRITE )) +if (!pgd_access_permitted (orig, args->flags &FOLL_WRITE ))return 0 ;BUILD_BUG_ON (pgd_devmap (orig)); refs =0 ; - page =pgd_page (orig) + ((addr & ~PGDIR_MASK ) >>PAGE_SHIFT ); + page =pgd_page (orig) + ((args->addr & ~PGDIR_MASK ) >>PAGE_SHIFT );do { - pages[*nr] = page; - (*nr)++; + args->pages[args->nr++] = page; page++; refs++; - }while (addr +=PAGE_SIZE , addr != end); + }while (args->addr +=PAGE_SIZE , args->addr != end); head =try_get_compound_head (pgd_page (orig), refs);if (!head) { - *nr -= refs; + args->nr -= refs;return 0 ; }if (unlikely (pgd_val (orig) !=pgd_val (*pgdp))) { - *nr -= refs; + args->nr -= refs;while (refs--)put_page (head);return 0 ;
@@ -2170,17 +2165,16 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr,return 1 ; } -static int gup_pmd_range (pud_t pud,unsigned long addr,unsigned long end, -unsigned int flags,struct page **pages,int *nr) +static int gup_pmd_range (struct gup_args *args,pud_t pud,unsigned long end) {unsigned long next;pmd_t *pmdp; - pmdp =pmd_offset (&pud, addr); + pmdp =pmd_offset (&pud, args->addr); do { pmd_t pmd = READ_ONCE(*pmdp); - next = pmd_addr_end(addr, end); + next = pmd_addr_end(args->addr, end); if (!pmd_present(pmd)) return 0;
@@ -2194,8 +2188,7 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,if (pmd_protnone (pmd))return 0 ; -if (!gup_huge_pmd (pmd, pmdp, addr, next, flags, - pages, nr)) +if (!gup_huge_pmd (args, pmd, pmdp, next))return 0 ; }else if (unlikely (is_hugepd (__hugepd (pmd_val (pmd))))) {
@@ -2203,93 +2196,88 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,* architecture have different format for hugetlbfs * pmd format and THP pmd format */ -if (!gup_huge_pd (__hugepd (pmd_val (pmd)), addr, -PMD_SHIFT , next, flags, pages, nr)) +if (!gup_huge_pd (args,__hugepd (pmd_val (pmd)), +PMD_SHIFT , next)) return 0; - }else if (!gup_pte_range (pmd, addr, next, flags, pages, nr)) + }else if (!gup_pte_range (args, pmd, next))return 0 ; - }while (pmdp++, addr = next, addr != end); + }while (pmdp++, args->addr != end);return 1 ; } -static int gup_pud_range (p4d_t p4d,unsigned long addr,unsigned long end, -unsigned int flags,struct page **pages,int *nr) +static int gup_pud_range (struct gup_args *args,p4d_t p4d,unsigned long end) {unsigned long next;pud_t *pudp; - pudp =pud_offset (&p4d, addr); + pudp =pud_offset (&p4d, args->addr);do {pud_t pud =READ_ONCE (*pudp); - next =pud_addr_end (addr, end); + next =pud_addr_end (args->addr, end);if (pud_none (pud))return 0 ;if (unlikely (pud_huge (pud))) { -if (!gup_huge_pud (pud, pudp, addr, next, flags, - pages, nr)) +if (!gup_huge_pud (args, pud, pudp, next))return 0 ; }else if (unlikely (is_hugepd (__hugepd (pud_val (pud))))) { -if (!gup_huge_pd (__hugepd (pud_val (pud)), addr, -PUD_SHIFT , next, flags, pages, nr)) +if (!gup_huge_pd (args,__hugepd (pud_val (pud)), +PUD_SHIFT , next))return 0 ; - }else if (!gup_pmd_range (pud, addr, next, flags, pages, nr)) + }else if (!gup_pmd_range (args, pud, next))return 0 ; - }while (pudp++, addr = next, addr != end); + }while (pudp++, args->addr != end);return 1 ; } -static int gup_p4d_range (pgd_t pgd,unsigned long addr,unsigned long end, -unsigned int flags,struct page **pages,int *nr) +static int gup_p4d_range (struct gup_args *args,pgd_t pgd,unsigned long end) {unsigned long next;p4d_t *p4dp; - p4dp =p4d_offset (&pgd, addr); + p4dp =p4d_offset (&pgd, args->addr);do {p4d_t p4d =READ_ONCE (*p4dp); - next =p4d_addr_end (addr, end); + next =p4d_addr_end (args->addr, end);if (p4d_none (p4d))return 0 ;BUILD_BUG_ON (p4d_huge (p4d));if (unlikely (is_hugepd (__hugepd (p4d_val (p4d))))) { -if (!gup_huge_pd (__hugepd (p4d_val (p4d)), addr, -P4D_SHIFT , next, flags, pages, nr)) +if (!gup_huge_pd (args,__hugepd (p4d_val (p4d)), +P4D_SHIFT , next))return 0 ; - }else if (!gup_pud_range (p4d, addr, next, flags, pages, nr)) + }else if (!gup_pud_range (args, p4d, next))return 0 ; - }while (p4dp++, addr = next, addr != end); + }while (p4dp++, args->addr != end);return 1 ; } -static void gup_pgd_range (unsigned long addr,unsigned long end, -unsigned int flags,struct page **pages,int *nr) +static void gup_pgd_range (struct gup_args *args,unsigned long end) {unsigned long next;pgd_t *pgdp; - pgdp =pgd_offset (current->mm, addr); + pgdp =pgd_offset (current->mm, args->addr);do {pgd_t pgd =READ_ONCE (*pgdp); - next =pgd_addr_end (addr, end); + next =pgd_addr_end (args->addr, end);if (pgd_none (pgd))return ;if (unlikely (pgd_huge (pgd))) { -if (!gup_huge_pgd (pgd, pgdp, addr, next, flags, - pages, nr)) +if (!gup_huge_pgd (args, pgd, pgdp, next))return ; }else if (unlikely (is_hugepd (__hugepd (pgd_val (pgd))))) { -if (!gup_huge_pd (__hugepd (pgd_val (pgd)), addr, -PGDIR_SHIFT , next, flags, pages, nr)) +if (!gup_huge_pd (args,__hugepd (pgd_val (pgd)), +PGDIR_SHIFT , next))return ; - }else if (!gup_p4d_range (pgd, addr, next, flags, pages, nr)) + }else if (!gup_p4d_range (args, pgd, next))return ; - }while (pgdp++, addr = next, addr != end); + }while (pgdp++, args->addr != end); }#else static inline void gup_pgd_range(unsigned long addr, unsigned long end,
@@ -2321,17 +2309,18 @@ static bool gup_fast_permitted(unsigned long start, unsigned long end)int __get_user_pages_fast (unsigned long start,int nr_pages,int write,struct page **pages) { -unsigned long len, end; +struct gup_args args = { + .addr =untagged_addr (start) &PAGE_MASK , + .flags = write ?FOLL_WRITE :0 , + .pages = pages, + }; +unsigned long len = (unsigned long )nr_pages <<PAGE_SHIFT ; +unsigned long end = args.addr + len;unsigned long flags; -int nr =0 ; - - start =untagged_addr (start) &PAGE_MASK ; - len = (unsigned long ) nr_pages <<PAGE_SHIFT ; - end = start + len; -if (end <= start) +if (end <= args.addr)return 0 ; -if (unlikely (!access_ok ((void __user *)start, len))) +if (unlikely (!access_ok ((void __user *)args.addr, len)))return 0 ;/*
@@ -2345,38 +2334,42 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* We do not adopt an rcu_read_lock(.) here as we also want to * block IPIs that come from THPs splitting. */ - -if (gup_fast_permitted (start, end)) { +if (gup_fast_permitted (args.addr, end)) {local_irq_save (flags); -gup_pgd_range (start, end, write ?FOLL_WRITE :0 , pages, &nr); +gup_pgd_range (&args, end);local_irq_restore (flags); } -return nr; +return args.nr; }EXPORT_SYMBOL_GPL (__get_user_pages_fast); -static int __gup_longterm_unlocked (unsigned long start,int nr_pages, -unsigned int gup_flags,struct page **pages) +static int get_user_pages_fallback (struct gup_args *args,int nr_pages) { +struct page **pages = args->pages + args->nr;int ret; + nr_pages -= args->nr; +/* * FIXME: FOLL_LONGTERM does not work with * get_user_pages_unlocked() (see comments in that function) */ -if (gup_flags &FOLL_LONGTERM ) { +if (args->flags &FOLL_LONGTERM ) {down_read (¤t->mm->mmap_sem); ret =__gup_longterm_locked (current, current->mm, - start, nr_pages, - pages,NULL , gup_flags); + args->addr, nr_pages, + pages,NULL , args->flags);up_read (¤t->mm->mmap_sem); }else { - ret =get_user_pages_unlocked (start, nr_pages, - pages, gup_flags); + ret =get_user_pages_unlocked (args->addr, nr_pages, pages, + args->flags); } -return ret; +/* Have to be a bit careful with return values */ +if (ret >0 ) + args->nr += ret; +return args->nr ? args->nr : ret; }/**
@@ -2398,46 +2391,31 @@ static int __gup_longterm_unlocked(unsigned long start, int nr_pages,int get_user_pages_fast (unsigned long start,int nr_pages,unsigned int gup_flags,struct page **pages) { -unsigned long addr, len, end; -int nr =0 , ret =0 ; +struct gup_args args = { + .addr =untagged_addr (start) &PAGE_MASK , + .flags = gup_flags, + .pages = pages, + }; +unsigned long len = (unsigned long )nr_pages <<PAGE_SHIFT ; +unsigned long end = args.addr + len;if (WARN_ON_ONCE (gup_flags & ~(FOLL_WRITE |FOLL_LONGTERM )))return -EINVAL ; - start =untagged_addr (start) &PAGE_MASK ; - addr = start; - len = (unsigned long ) nr_pages <<PAGE_SHIFT ; - end = start + len; - -if (end <= start) +if (end <= args.addr)return 0 ; -if (unlikely (!access_ok ((void __user *)start, len))) +if (unlikely (!access_ok ((void __user *)args.addr, len)))return -EFAULT ; -if (gup_fast_permitted (start, end)) { +if (gup_fast_permitted (args.addr, end)) {local_irq_disable (); -gup_pgd_range (addr, end, gup_flags, pages, &nr); +gup_pgd_range (&args, end);local_irq_enable (); - ret = nr; - } - -if (nr < nr_pages) { -/* Try to get the remaining pages with get_user_pages */ - start += nr <<PAGE_SHIFT ; - pages += nr; - - ret =__gup_longterm_unlocked (start, nr_pages - nr, - gup_flags, pages); - -/* Have to be a bit careful with return values */ -if (nr >0 ) { -if (ret <0 ) - ret = nr; -else - ret += nr; - } } -return ret; +/* Try to get the remaining pages with get_user_pages */ +if (args.nr < nr_pages) +return get_user_pages_fallback (&args, nr_pages); +return args.nr; }EXPORT_SYMBOL_GPL (get_user_pages_fast);
Signature · 1 line
2.20.1
Nicholas Piggin
reviewerRe: [PATCH 16/16] mm: pass get_user_pages_fast iterator arguments in a structure
Christoph Hellwig's on June 12, 2019 12:41 am:
Quoted 4 lines
Instead of passing a set of always repeated arguments down the
get_user_pages_fastiterators, create a structgup_argsto hold them and pass that by reference. This leads to an over 100 byte .text size reduction for x86-64.
What does this do for performance? I've found this pattern can be bad for store aliasing detection.
Thanks,
Nick
Re: [PATCH 16/16] mm: pass get_user_pages_fast iterator arguments in a structure
Quoted 3 lines · Nicholas Piggin
What does this do for performance? I've found this pattern can be bad for store aliasing detection.
I wouldn't expect it to be noticeable, and the lack of argument reloading etc should make up for it. Plus inlining makes it a non-issue when that happens.
But I guess we could also at least look at using "restrict", if that ends up helping. Unlike the completely bogus type-based aliasing rules (that we disable because I think the C people were on some bad bad drugs when they came up with them), restricted pointers are a real thing that makes sense.
That said, we haven't traditionally used it, and I don't know how much it helps gcc. Maybe gcc ignores it entirely? S
Linus
Nicholas Piggin
reviewerRe: [PATCH 16/16] mm: pass get_user_pages_fast iterator arguments in a structure
Linus Torvalds's on June 12, 2019 11:09 am:
Quoted 8 lines · Nicholas Piggin
Quoted 3 lines
What does this do for performance? I've found this pattern can be bad for store aliasing detection.
I wouldn't expect it to be noticeable, and the lack of argument reloading etc should make up for it. Plus inlining makes it a non-issue when that happens.
Maybe in isolation. Just seems like a strange pattern to sprinkle around randomly, I wouldn't like it to proliferate.
I understand in some cases where a big set of parameters or basically state gets sent around through a lot of interfaces. Within one file to make lines a bit shorter or save a few bytes isn't such a strong case.
Quoted 9 lines
But I guess we could also at least look at using "restrict", if that ends up helping. Unlike the completely bogus type-based aliasing rules (that we disable because I think the C people were on some bad bad drugs when they came up with them), restricted pointers are a real thing that makes sense.
That said, we haven't traditionally used it, and I don't know how much it helps gcc. Maybe gcc ignores it entirely? S
Ahh, it's not compiler store alias analysis I'm talking about, but processor (but you raise an interesting point about compiler too, would be nice if we could improve that in general).
The processor aliasing problem happens because the struct will be initialised with stores using one base register (e.g., stack register), and then same memory is loaded using a different register (e.g., parameter register). Processor's static heuristics for determining a load doesn't alias with an earlier store doesn't do so well in that case.
Just about everywhere I've seen those kind of misspeculation and flushes in the kernel has been this pattern, so I'm wary of it in performance critical code.
Thanks,
Nick
Re: [PATCH 16/16] mm: pass get_user_pages_fast iterator arguments in a structure
Quoted 5 lines · Nicholas Piggin
The processor aliasing problem happens because the struct will be initialised with stores using one base register (e.g., stack register), and then same memory is loaded using a different register (e.g., parameter register).
Hmm. Honestly, I've never seen anything like that in any kernel profiles.
Compared to the problems I _do_ see (which is usually the obvious cache misses, and locking), it must either be in the noise or it's some problem specific to whatever CPU you are doing performance work on?
I've occasionally seen pipeline hiccups in profiles, but it's usually been either some serious glass jaw of the core, or it's been something really stupid we did (or occasionally that the compiler did: one in particular I remember was how there was a time when gcc would narrow stores when it could, so if you set a bit in a word, it would do it with a byte store, and then when you read the whole word afterwards you'd get a major pipeline stall and it happened to show up in some really hot paths).
Linus
Re: [PATCH 16/16] mm: pass get_user_pages_fast iterator arguments in a structure
Quoted 7 lines · Linus Torvalds
Hmm. Honestly, I've never seen anything like that in any kernel profiles.
Compared to the problems I _do_ see (which is usually the obvious cache misses, and locking), it must either be in the noise or it's some problem specific to whatever CPU you are doing performance work on?
Quoted 8 lines
I've occasionally seen pipeline hiccups in profiles, but it's usually been either some serious glass jaw of the core, or it's been something really stupid we did (or occasionally that the compiler did: one in particular I remember was how there was a time when gcc would narrow stores when it could, so if you set a bit in a word, it would do it with a byte store, and then when you read the whole word afterwards you'd get a major pipeline stall and it happened to show up in some really hot paths).
I've not seen any difference in the GUP bench output here ar all.
But I'm fine with skipping this patch for now, I have a potential series I'm looking into that would benefit a lot from it, but we can discusss it in that context and make sure all the other works gets in in time.
Nicholas Piggin
reviewerRe: [PATCH 16/16] mm: pass get_user_pages_fast iterator arguments in a structure
Christoph Hellwig's on June 21, 2019 6:15 pm:
Quoted 17 lines · Linus Torvalds
Quoted 7 lines
Hmm. Honestly, I've never seen anything like that in any kernel profiles.
Compared to the problems I _do_ see (which is usually the obvious cache misses, and locking), it must either be in the noise or it's some problem specific to whatever CPU you are doing performance work on?
Quoted 8 lines
I've occasionally seen pipeline hiccups in profiles, but it's usually been either some serious glass jaw of the core, or it's been something really stupid we did (or occasionally that the compiler did: one in particular I remember was how there was a time when gcc would narrow stores when it could, so if you set a bit in a word, it would do it with a byte store, and then when you read the whole word afterwards you'd get a major pipeline stall and it happened to show up in some really hot paths).
Quoted 7 lines
I've not seen any difference in the GUP bench output here ar all.
But I'm fine with skipping this patch for now, I have a potential series I'm looking into that would benefit a lot from it, but we can discusss it in that context and make sure all the other works gets in in time.
If you can, that would be good. I don't like to object based on handwaving so I'll see if I can find any benchmarks that will give better confidence. Those old TPC-C tests were good, and there was some DB2 workload that was the reason I added gup fast in the first place. I'll do some digging.
Thanks,
Nick
Nicholas Piggin
reviewerRe: [PATCH 16/16] mm: pass get_user_pages_fast iterator arguments in a structure
In reply to Linus TorvaldsLinus Torvalds's on June 21, 2019 3:21 am:
Quoted 7 lines · Nicholas Piggin
Quoted 5 lines
The processor aliasing problem happens because the struct will be initialised with stores using one base register (e.g., stack register), and then same memory is loaded using a different register (e.g., parameter register).
Quoted 6 lines
Hmm. Honestly, I've never seen anything like that in any kernel profiles.
Compared to the problems I _do_ see (which is usually the obvious cache misses, and locking), it must either be in the noise or it's some problem specific to whatever CPU you are doing performance work on?
No you're right, the performance hit from these flushes is not a big hit that stands out in cycle counts. I just look at kernel code for various flushes. Branches not surprisingly are usually the main culprit, but they're normally not so interesting.
Static alias prediction seems to work well outside this case. It's interesting, you need both a store ; load sequence that does not predict well (e.g., using a different base register), and you also need that load to be executed ahead of the store.
The small stack structure for arguments is the perfect case. Bad pattern, and load executed right after store. Even then you also need a reason to delay the store (e.g., source not ready or store queue full), but those hazards do show up.
Now, even when all that goes wrong, there are dynamic heuristics that can take over. So if you run a repetitive microbenchmark you won't see it.
Some CPUs seem to be quite aggressive about giving up and turning off the alias prediction globally if you take misses (Intel x86 used to do that IIRC, not sure if they still do). So in that case you wouldn't even see it show up in one place, everything will just run slightly slower.
What I worry about is high rate direct IO workloads that see single flushes in these paths as significant. Or if this thing creeps in to the kernel too much and just slightly raises global misses enough, then it will cause disambiguation to be significantly shut down.
Thanks,
Nick
Re: [PATCH 16/16] mm: pass get_user_pages_fast iterator arguments in a structure
In reply to Nicholas PigginQuoted 10 lines · Nicholas Piggin
Christoph Hellwig's on June 12, 2019 12:41 am:
Quoted 4 lines
Instead of passing a set of always repeated arguments down the
get_user_pages_fastiterators, create a structgup_argsto hold them and pass that by reference. This leads to an over 100 byte .text size reduction for x86-64.What does this do for performance? I've found this pattern can be bad for store aliasing detection.
Note that sometimes such an optimization can also have adverse effect due to stack protector code that gcc emits when you use such structs.
Matthew Wilcox encountered such a case:
https://patchwork.kernel.org/patch/10702741/
I just noticed I didn't have Andrew explicitly on the receipents list, so adding him.
Is everyone happy enough to give this a spin in -mm and linux-next?
Quoted 7 lines · Christoph Hellwig
Hi Linus and maintainers,
below is a series to switch mips, sh and sparc64 to use the generic GUP code so that we only have one codebase to touch for further improvements to this code. I don't have hardware for any of these architectures, and generally no clue about their page table management, so handle with care.
I like this series, ther are alot of people talking about work for GUP and this will make any of that so much easier to do.
Jason