RFC: switch the remaining architectures to use generic GUP v2
Subsystems (15)
Hi Linus and maintainers,
below is a series to switch mips, sh and sparc64 to use the generic GUP code so that we only have one codebase to touch for further improvements to this code. I don't have hardware for any of these architectures, and generally no clue about their page table management, so handle with care.
Changes since v1:
- fix various issues found by the build bot
- cherry pick and use the untagged_addr helper form Andrey
- add various refactoring patches to share more code over architectures
- move the powerpc hugepd code to mm/gup.c and sync it with the generic hup semantics
PATCH 1/16 uaccess: add untagged_addr definition for other arches
From: Andrey Konovalov <[email protected]>
To allow arm64 syscalls to accept tagged pointers from userspace, we must untag them when they are passed to the kernel. Since untagging is done in generic parts of the kernel, the untagged_addr macro needs to be defined for all architectures.
Define it as a noop for architectures other than arm64.
--- include/linux/mm.h | 4 ++++ 1 file changed, 4 insertions(+)
include/linux/mm.h +4 -0
@@ -99,6 +99,10 @@ extern int mmap_rnd_compat_bits __read_mostly;#include <asm/pgtable.h> #include <asm/processor.h> +#ifndef untagged_addr +#define untagged_addr (addr) (addr) +#endif +#ifndef __pa_symbol#define __pa_symbol (x) __pa(RELOC_HIDE((unsigned long)(x), 0))#endif
Signature · 1 line
2.20.1
Re: [PATCH 01/16] uaccess: add untagged_addr definition for other arches
Quoted 8 lines · Christoph Hellwig
From: Andrey Konovalov <[email protected]>
To allow arm64 syscalls to accept tagged pointers from userspace, we must untag them when they are passed to the kernel. Since untagging is done in generic parts of the kernel, the
untagged_addrmacro needs to be defined for all architectures.Define it as a noop for architectures other than arm64.
Could you reword above sentence? We are already starting off with untagged_addr() not being no-op for arm64 and sparc64. It will expand further potentially. So something more along the lines of "Define it as noop for architectures that do not support memory tagging". The first paragraph in the log can also be rewritten to be not specific to arm64.
--
Khalid
Quoted 24 lines
Acked-by Catalin Marinas <[email protected]>Signed-off-by Andrey Konovalov <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>--- include/linux/mm.h | 4 ++++ 1 file changed, 4 insertions(+)include/linux/mm.h +4
-0@@ -99,6 +99,10 @@ extern int mmap_rnd_compat_bits __read_mostly;#include <asm/pgtable.h> #include <asm/processor.h> +#ifndef untagged_addr +#define untagged_addr (addr) (addr) +#endif +#ifndef __pa_symbol#define __pa_symbol (x) __pa(RELOC_HIDE((unsigned long)(x), 0))#endif
Re: [PATCH 01/16] uaccess: add untagged_addr definition for other arches
Quoted 5 lines · Khalid Aziz
Could you reword above sentence? We are already starting off with
untagged_addr()not being no-op for arm64 and sparc64. It will expand further potentially. So something more along the lines of "Define it as noop for architectures that do not support memory tagging". The first paragraph in the log can also be rewritten to be not specific to arm64.
Well, as of this patch this actually is a no-op for everyone.
Linus, what do you think of applying this patch (maybe with a slightly fixed up commit log) to 5.2-rc so that we remove a cross dependency between the series?
Re: [PATCH 01/16] uaccess: add untagged_addr definition for other arches
Quoted 8 lines · Christoph Hellwig
Quoted 5 lines · Khalid Aziz
Could you reword above sentence? We are already starting off with
untagged_addr()not being no-op for arm64 and sparc64. It will expand further potentially. So something more along the lines of "Define it as noop for architectures that do not support memory tagging". The first paragraph in the log can also be rewritten to be not specific to arm64.
Quoted 5 lines
Well, as of this patch this actually is a no-op for everyone.
Linus, what do you think of applying this patch (maybe with a slightly fixed up commit log) to 5.2-rc so that we remove a cross dependency between the series?
(I have adjusted the patch description and have just sent it out separately from the series).
PATCH 2/16 mm: use untagged_addr() for get_user_pages_fast addresses
In reply to Christoph HellwigThis will allow sparc64 to override its ADI tags for get_user_pages and get_user_pages_fast.
--- mm/gup.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/gup.c +2 -2
@@ -2145,7 +2145,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,unsigned long flags;int nr =0 ; - start &=PAGE_MASK ; + start =untagged_addr (start) &PAGE_MASK ; len = (unsigned long ) nr_pages <<PAGE_SHIFT ; end = start + len;
@@ -2218,7 +2218,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages,unsigned long addr, len, end;int nr =0 , ret =0 ; - start &=PAGE_MASK ; + start =untagged_addr (start) &PAGE_MASK ; addr = start; len = (unsigned long ) nr_pages <<PAGE_SHIFT ; end = start + len;
Signature · 1 line
2.20.1
PATCH 3/16 mm: simplify gup_fast_permitted
In reply to Christoph HellwigPass in the already calculated end value instead of recomputing it, and leave the end > start check in the callers instead of duplicating them in the arch code.
--- arch/s390/include/asm/pgtable.h | 8 +------- arch/x86/include/asm/pgtable_64.h | 8 +------- mm/gup.c | 17 +++++++---------- 3 files changed, 9 insertions(+), 24 deletions(-)
arch/s390/include/asm/pgtable.h +1 -7
@@ -1270,14 +1270,8 @@ static inline pte_t *pte_offset(pmd_t *pmd, unsigned long address)#define pte_offset_map (pmd, address) pte_offset_kernel(pmd, address)#define pte_unmap (pte) do { } while (0) -static inline bool gup_fast_permitted (unsigned long start,int nr_pages) +static inline bool gup_fast_permitted (unsigned long start,unsigned long end) { -unsigned long len, end; - - len = (unsigned long ) nr_pages <<PAGE_SHIFT ; - end = start + len; -if (end < start) -return false;return end <= current->mm->context.asce_limit; }#define gup_fast_permitted gup_fast_permitted
arch/x86/include/asm/pgtable_64.h +1 -7
@@ -259,14 +259,8 @@ extern void init_extra_mapping_uc(unsigned long phys, unsigned long size);extern void init_extra_mapping_wb (unsigned long phys,unsigned long size);#define gup_fast_permitted gup_fast_permitted -static inline bool gup_fast_permitted (unsigned long start,int nr_pages) +static inline bool gup_fast_permitted (unsigned long start,unsigned long end) { -unsigned long len, end; - - len = (unsigned long )nr_pages <<PAGE_SHIFT ; - end = start + len; -if (end < start) -return false;if (end >> __VIRTUAL_MASK_SHIFT)return false;return true;
mm/gup.c +7 -10
@@ -2122,13 +2122,9 @@ static void gup_pgd_range(unsigned long addr, unsigned long end,* Check if it's allowed to use __get_user_pages_fast() for the range, or * we need to fall back to the slow version: */ -bool gup_fast_permitted (unsigned long start,int nr_pages) +static bool gup_fast_permitted (unsigned long start,unsigned long end) { -unsigned long len, end; - - len = (unsigned long ) nr_pages <<PAGE_SHIFT ; - end = start + len; -return end >= start; +return true; }#endif
@@ -2149,6 +2145,8 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write, len = (unsigned long ) nr_pages <<PAGE_SHIFT ; end = start + len; +if (end < start) +return 0 ;if (unlikely (!access_ok ((void __user *)start, len)))return 0 ;
@@ -2164,7 +2162,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,* block IPIs that come from THPs splitting. */ -if (gup_fast_permitted (start, nr_pages)) { +if (gup_fast_permitted (start, end)) {local_irq_save (flags);gup_pgd_range (start, end, write ?FOLL_WRITE :0 , pages, &nr);local_irq_restore (flags);
@@ -2223,13 +2221,12 @@ int get_user_pages_fast(unsigned long start, int nr_pages, len = (unsigned long ) nr_pages <<PAGE_SHIFT ; end = start + len; -if (nr_pages <=0 ) +if (end < start)return 0 ; -if (unlikely (!access_ok ((void __user *)start, len)))return -EFAULT ; -if (gup_fast_permitted (start, nr_pages)) { +if (gup_fast_permitted (start, end)) {local_irq_disable ();gup_pgd_range (addr, end, gup_flags, pages, &nr);local_irq_enable ();
Signature · 1 line
2.20.1
Re: [PATCH 03/16] mm: simplify gup_fast_permitted
Quoted 4 lines · Christoph Hellwig
Pass in the already calculated end value instead of recomputing it, and leave the end > start check in the callers instead of duplicating them in the arch code.
Good cleanup, except it's wrong.
mm/gup.c int get_user_pages_fast(unsigned long start, int nr_pages, · line 2225
Quoted 3 lines
mm/gup.c +1
-1-if (nr_pages <=0 ) +if (end < start)return 0 ;
You moved the overflow test to generic code - good.
You removed the sign and zero test on nr_pages - bad.
The zero test in particular is _important_ - the GUP range operators know and depend on the fact that they are passed a non-empty range.
The sign test it less so, but is definitely appropriate. It might be even better to check that the "<< PAGE_SHIFT" doesn't overflow in "long", of course, but with callers being supposed to be trusted, the sign test at least checks for stupid underflow issues.
So at the very least that "(end < start)" needs to be "(end <start)", but honestly, I think the sign of the nr_pages should be continued to be checked.
Linus
Re: [PATCH 03/16] mm: simplify gup_fast_permitted
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Christoph Hellwig
Pass in the already calculated end value instead of recomputing it, and leave the end > start check in the callers instead of duplicating them in the arch code.
Good cleanup, except it's wrong.
Quoted 7 lines
mm/gup.c int get_user_pages_fast(unsigned long start, int nr_pages, · line 2225
Quoted 3 lines
mm/gup.c +1
-1-if (nr_pages <=0 ) +if (end < start)return 0 ;You moved the overflow test to generic code - good.
You removed the sign and zero test on
nr_pages- bad.
I only removed a duplicate of it. The full (old) code in get_user_pages_fast() looks like this:
if (nr_pages <= 0)
return 0;
if (unlikely(!access_ok((void __user *)start, len)))
return -EFAULT;
if (gup_fast_permitted(start, nr_pages)) {Re: [PATCH 03/16] mm: simplify gup_fast_permitted
Quoted 2 lines · Christoph Hellwig
I only removed a duplicate of it.
I don't see any remaining cases.
Quoted 9 lines
The full (old) code in
get_user_pages_fast()looks like this:if (nr_pages <= 0) return 0; if (unlikely(!access_ok((void __user *)start, len))) return -EFAULT; if (gup_fast_permitted(start, nr_pages)) {
Yes, and that code was correct.
The new code has no test at all for "nr_pages = 0", afaik.
Linus
Re: [PATCH 03/16] mm: simplify gup_fast_permitted
Quoted 2 lines · Linus Torvalds
The new code has no test at all for "
nr_pages= 0", afaik.
Note that it really is important to check for that, because right now we do
if (gup_fast_permitted(start, nr_pages)) {
local_irq_save(flags);
gup_pgd_range(start, end, write ? FOLL_WRITE : 0, pages, &nr);
local_irq_restore(flags);
}and that gup_pgd_range() function *depends* on the range being non-zero, and does
pgdp = pgd_offset(current->mm, addr);
do {
pgd_t pgd = READ_ONCE(*pgdp);
...
} while (pgdp++, addr = next, addr != end);Note how a zero range would turn into an infinite range here.
And the only check for 0 was that
if (nr_pages <= 0)
return 0;in get_user_pages_fast() that you removed.
(Admittedly, it would be much better to have that check in __get_user_pages_fast() itself, because we do have callers that call the double-underscore version)
Now, I sincerely hope that we don't have anybody that passes in a zero nr_pages (or a negative one), but we do actually have a comment saying it's ok.
Note that the check for "if (end < start)" not only does not check for 0, it also doesn't really check for negative. It checks for _overflow_. Admittedly most negative values would be expected to overflow, but it's still a very different issue.
Maybe you added the check for negative somewhere else (in another patch), but I don't see it.
Linus
Re: [PATCH 03/16] mm: simplify gup_fast_permitted
Quoted 6 lines · Linus Torvalds
Quoted 2 lines · Linus Torvalds
The new code has no test at all for "
nr_pages= 0", afaik.Note that it really is important to check for that, because right now we do
True. The 0 check got lost. I'll make sure we do the right thing for the next version.
PATCH 4/16 mm: lift the x86_32 PAE version of gup_get_pte to common code
In reply to Christoph HellwigThe split low/high access is the only non-READ_ONCE version of gup_get_pte that did show up in the various arch implemenations. Lift it to common code and drop the ifdef based arch override.
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable-3level.h | 47 ------------------------ arch/x86/kvm/mmu.c | 2 +- mm/Kconfig | 3 ++ mm/gup.c | 51 ++++++++++++++++++++++++--- 5 files changed, 52 insertions(+), 52 deletions(-)
arch/x86/Kconfig +1 -0
@@ -121,6 +121,7 @@ config X86 select GENERIC_STRNCPY_FROM_USER select GENERIC_STRNLEN_USER select GENERIC_TIME_VSYSCALL + select GUP_GET_PTE_LOW_HIGH if X86_PAE select HARDLOCKUP_CHECK_TIMESTAMP if X86_64 select HAVE_ACPI_APEI if ACPI select HAVE_ACPI_APEI_NMI if ACPI
arch/x86/include/asm/pgtable-3level.h +0 -47
53 lines not on this page · raw message
arch/x86/kvm/mmu.c +1 -1
@@ -653,7 +653,7 @@ static u64 __update_clear_spte_slow(u64 *sptep, u64 spte)/* * The idea using the light way get the spte on x86_32 guest is from -* gup_get_pte(arch/x86/mm/gup.c). +* gup_get_pte (mm/gup.c). * * An spte tlb flush may be pending, because kvm_set_pte_rmapp * coalesces them and we are running out of the MMU lock. Therefore
mm/Kconfig +3 -0
@@ -762,6 +762,9 @@ config GUP_BENCHMARK See tools/testing/selftests/vm/gup_benchmark.c +config GUP_GET_PTE_LOW_HIGH + bool + config ARCH_HAS_PTE_SPECIAL bool
mm/gup.c +47 -4
64 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH 5/16 MIPS: use the generic get_user_pages_fast code
In reply to Christoph HellwigThe mips code is mostly equivalent to the generic one, minus various bugfixes and an arch override for gup_fast_permitted.
Note that this defines ARCH_HAS_PTE_SPECIAL for mips as mips has pte_special and pte_mkspecial implemented and used in the existing gup code. They are no-op stubs, though which makes me a little unsure if this is really right thing to do.
--- arch/mips/Kconfig | 3 + arch/mips/include/asm/pgtable.h | 3 + arch/mips/mm/Makefile | 1 - arch/mips/mm/gup.c | 303 -------------------------------- 4 files changed, 6 insertions(+), 304 deletions(-) delete mode 100644 arch/mips/mm/gup.c
arch/mips/Kconfig +3 -0
@@ -6,6 +6,7 @@ config MIPS select ARCH_BINFMT_ELF_STATE if MIPS_FP_SUPPORT select ARCH_CLOCKSOURCE_DATA select ARCH_HAS_ELF_RANDOMIZE + select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST select ARCH_HAS_UBSAN_SANITIZE_ALL select ARCH_SUPPORTS_UPROBES
@@ -34,6 +35,7 @@ config MIPS select GENERIC_SCHED_CLOCK if !CAVIUM_OCTEON_SOC select GENERIC_SMP_IDLE_THREAD select GENERIC_TIME_VSYSCALL + select GUP_GET_PTE_LOW_HIGH if CPU_MIPS32 && PHYS_ADDR_T_64BIT select HANDLE_DOMAIN_IRQ select HAVE_ARCH_COMPILER_H select HAVE_ARCH_JUMP_LABEL
@@ -55,6 +57,7 @@ config MIPS select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER + select HAVE_GENERIC_GUP select HAVE_IDE select HAVE_IOREMAP_PROT select HAVE_IRQ_EXIT_ON_IRQ_STACK
arch/mips/include/asm/pgtable.h +3 -0
@@ -20,6 +20,7 @@#include <asm/cmpxchg.h> #include <asm/io.h> #include <asm/pgtable-bits.h> +#include <asm/cpu-features.h> struct mm_struct ;struct vm_area_struct ;
@@ -626,6 +627,8 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#define gup_fast_permitted (start, end) (!cpu_has_dc_aliases) +#include <asm-generic/pgtable.h> /*
arch/mips/mm/Makefile +0 -1
@@ -7,7 +7,6 @@ obj-y += cache.o obj-y += context.o obj-y += extable.o obj-y += fault.o -obj-y += gup.o obj-y += init.o obj-y += mmap.o obj-y += page.o
arch/mips/mm/gup.c deleted +0 -303
303 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH 6/16 sh: add the missing pud_page definition
In reply to Christoph Hellwigsh only had pud_page_vaddr, but not pud_page.
--- arch/sh/include/asm/pgtable-3level.h | 1 + 1 file changed, 1 insertion(+)
arch/sh/include/asm/pgtable-3level.h +1 -0
@@ -37,6 +37,7 @@ static inline unsigned long pud_page_vaddr(pud_t pud) {return pud_val (pud); } +#define pud_page (pud) virt_to_page((void *)pud_page_vaddr(pud))#define pmd_index (address) (((address) >> PMD_SHIFT) & (PTRS_PER_PMD-1))static inline pmd_t *pmd_offset (pud_t *pud,unsigned long address)
Signature · 1 line
2.20.1
PATCH 7/16 sh: use the generic get_user_pages_fast code
In reply to Christoph HellwigThe sh code is mostly equivalent to the generic one, minus various bugfixes and two arch overrides that this patch adds to pgtable.h.
--- arch/sh/Kconfig | 2 + arch/sh/include/asm/pgtable.h | 37 +++++ arch/sh/mm/Makefile | 2 +- arch/sh/mm/gup.c | 277 ---------------------------------- 4 files changed, 40 insertions(+), 278 deletions(-) delete mode 100644 arch/sh/mm/gup.c
arch/sh/Kconfig +2 -0
@@ -14,6 +14,7 @@ config SUPERH select HAVE_ARCH_TRACEHOOK select HAVE_PERF_EVENTS select HAVE_DEBUG_BUGVERBOSE + select HAVE_GENERIC_GUP select ARCH_HAVE_CUSTOM_GPIO_H select ARCH_HAVE_NMI_SAFE_CMPXCHG if (GUSA_RB || CPU_SH4A) select ARCH_HAS_GCOV_PROFILE_ALL
@@ -63,6 +64,7 @@ config SUPERH config SUPERH32 def_bool "$(ARCH)" = "sh" select ARCH_32BIT_OFF_T + select GUP_GET_PTE_LOW_HIGH if X2TLB select HAVE_KPROBES select HAVE_KRETPROBES select HAVE_IOREMAP_PROT if MMU && !X2TLB
arch/sh/include/asm/pgtable.h +37 -0
@@ -149,6 +149,43 @@ extern void paging_init(void);extern void page_table_range_init (unsigned long start,unsigned long end,pgd_t *pgd); +static inline bool __pte_access_permitted (pte_t pte,u64 prot) +{ +return (pte_val (pte) & (prot | _PAGE_SPECIAL)) = prot; +} + +#ifdef CONFIG_X2TLB +static inline bool pte_access_permitted (pte_t pte,bool write) +{ +u64 prot = _PAGE_PRESENT; + + prot |=_PAGE_EXT (_PAGE_EXT_KERN_READ | _PAGE_EXT_USER_READ); +if (write) + prot |=_PAGE_EXT (_PAGE_EXT_KERN_WRITE | _PAGE_EXT_USER_WRITE); +return __pte_access_permitted (pte, prot); +} +#elif defined(CONFIG_SUPERH64 ) +static inline bool pte_access_permitted (pte_t pte,bool write) +{ +u64 prot = _PAGE_PRESENT | _PAGE_USER | _PAGE_READ; + +if (write) + prot |= _PAGE_WRITE; +return __pte_access_permitted (pte, prot); +} +#else +static inline bool pte_access_permitted (pte_t pte,bool write) +{ +u64 prot = _PAGE_PRESENT | _PAGE_USER; + +if (write) + prot |= _PAGE_RW; +return __pte_access_permitted (pte, prot); +} +#endif + +#define pte_access_permitted pte_access_permitted +/* arch/sh/mm/mmap.c */ #define HAVE_ARCH_UNMAPPED_AREA #define HAVE_ARCH_UNMAPPED_AREA_TOPDOWN
arch/sh/mm/Makefile +1 -1
@@ -17,7 +17,7 @@ cacheops-$(CONFIG_CPU_SHX3) += cache-shx3.o obj-y += $(cacheops-y) mmu-y := nommu.o extable_32.o -mmu-$(CONFIG_MMU) := extable_$(BITS).o fault.o gup.o ioremap.o kmap.o \ +mmu-$(CONFIG_MMU) := extable_$(BITS).o fault.o ioremap.o kmap.o \ pgtable.o tlbex_$(BITS).o tlbflush_$(BITS).o obj-y += $(mmu-y)
arch/sh/mm/gup.c deleted +0 -277
277 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH 8/16 sparc64: add the missing pgd_page definition
In reply to Christoph Hellwigsparc64 only had pgd_page_vaddr, but not pgd_page.
--- arch/sparc/include/asm/pgtable_64.h | 1 + 1 file changed, 1 insertion(+)
arch/sparc/include/asm/pgtable_64.h +1 -0
@@ -861,6 +861,7 @@ static inline unsigned long pud_page_vaddr(pud_t pud)#define pud_clear (pudp) (pud_val(*(pudp)) = 0UL)#define pgd_page_vaddr (pgd) \ ((unsigned long) __va(pgd_val(pgd))) +#define pgd_page (pgd) virt_to_page(__va(pgd_val(pgd)))#define pgd_present (pgd) (pgd_val(pgd) != 0U)#define pgd_clear (pgdp) (pgd_val(*(pgdp)) = 0UL)
Signature · 1 line
2.20.1
Re: [PATCH 08/16] sparc64: add the missing pgd_page definition
Both sparc64 and sh had this pattern, but now that I look at it more closely, I think your version is wrong, or at least nonoptimal.
Quoted 2 lines · Christoph Hellwig
+#define
pgd_page(pgd)virt_to_page(__va(pgd_val(pgd)))
Going through the virtual address is potentially very inefficient, and might in some cases just be wrong (ie it's definitely wrong for HIGHMEM style setups).
It would likely be much better to go through the physical address and use "pfn_to_page()". I realize that we don't have a "pgd to physical", but neither do we really have a "pgd to virtual", and your "__va(pgd_val(x))" thing is not at allguaranteed to work. You're basically assuming that "pgd_val(x)" is the physical address, which is likely not entirely incorrect, but it should be checked by the architecture people.
The pgd value could easily have high bits with meaning, which would also potentially screw up the __va(x) model.
So I thgink this would be better done with
#define pgd_page(pgd) pfn_to_page(pgd_pfn(pgd))
where that "pgd_pfn()" would need to be a new (but likely very trivial) function. That's what we do for pte_pfn().
IOW, it would likely end up something like
#define pgd_to_pfn(pgd) (pgd_val(x) >> PFN_PGD_SHIFT)
David?
Linus
Re: [PATCH 08/16] sparc64: add the missing pgd_page definition
Quoted 2 lines · Linus Torvalds
Both sparc64 and sh had this pattern, but now that I look at it more closely, I think your version is wrong, or at least nonoptimal.
I bet it is. Then again these symbols are just required for the code to compile, as neither sparc64 nor sh actually use the particular variant of huge pages we need it for. Then again even actually dead code should better be not too buggy if it isn't just a stub.
Quoted 10 lines
So I thgink this would be better done with
#define pgd_page(pgd) pfn_to_page(pgd_pfn(pgd))where that "
pgd_pfn()" would need to be a new (but likely very trivial) function. That's what we do forpte_pfn().IOW, it would likely end up something like
#define pgd_to_pfn(pgd) (pgd_val(x) >> PFN_PGD_SHIFT)
True. I guess it would be best if we could get most if not all architectures to use common versions of these macros so that we have the issue settled once.
PATCH 9/16 sparc64: define untagged_addr()
In reply to Christoph HellwigAdd a helper to untag a user pointer. This is needed for ADI support in get_user_pages_fast.
--- arch/sparc/include/asm/pgtable_64.h | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
arch/sparc/include/asm/pgtable_64.h +22 -0
@@ -1076,6 +1076,28 @@ static inline int io_remap_pfn_range(struct vm_area_struct *vma, }#define io_remap_pfn_range io_remap_pfn_range +static inline unsigned long untagged_addr (unsigned long start) +{ +if (adi_capable ()) { +long addr = start; + +/* If userspace has passed a versioned address, kernel +* will not find it in the VMAs since it does not store +* the version tags in the list of VMAs. Storing version +* tags in list of VMAs is impractical since they can be +* changed any time from userspace without dropping into +* kernel. Any address search in VMAs will be done with +* non-versioned addresses. Ensure the ADI version bits +* are dropped here by sign extending the last bit before +* ADI bits. IOMMU does not implement version tags. +*/ +return (addr << (long )adi_nbits ()) >> (long )adi_nbits (); + } + +return start; +} +#define untagged_addr untagged_addr +#include <asm/tlbflush.h> #include <asm-generic/pgtable.h>
Signature · 1 line
2.20.1
PATCH 10/16 sparc64: use the generic get_user_pages_fast code
In reply to Christoph HellwigThe sparc64 code is mostly equivalent to the generic one, minus various bugfixes and two arch overrides that this patch adds to pgtable.h.
--- arch/sparc/Kconfig | 1 + arch/sparc/include/asm/pgtable_64.h | 18 ++ arch/sparc/mm/Makefile | 2 +- arch/sparc/mm/gup.c | 340 ---------------------------- 4 files changed, 20 insertions(+), 341 deletions(-) delete mode 100644 arch/sparc/mm/gup.c
arch/sparc/Kconfig +1 -0
@@ -28,6 +28,7 @@ config SPARC select RTC_DRV_M48T59 select RTC_SYSTOHC select HAVE_ARCH_JUMP_LABEL if SPARC64 + select HAVE_GENERIC_GUP if SPARC64 select GENERIC_IRQ_SHOW select ARCH_WANT_IPC_PARSE_VERSION select GENERIC_PCI_IOMAP
arch/sparc/include/asm/pgtable_64.h +18 -0
@@ -1098,6 +1098,24 @@ static inline unsigned long untagged_addr(unsigned long start) }#define untagged_addr untagged_addr +static inline bool pte_access_permitted (pte_t pte,bool write) +{ +u64 prot; + +if (tlb_type = hypervisor) { + prot = _PAGE_PRESENT_4V | _PAGE_P_4V; +if (prot) + prot |= _PAGE_WRITE_4V; + }else { + prot = _PAGE_PRESENT_4U | _PAGE_P_4U; +if (write) + prot |= _PAGE_WRITE_4U; + } + +return (pte_val (pte) & (prot | _PAGE_SPECIAL)) = prot; +} +#define pte_access_permitted pte_access_permitted +#include <asm/tlbflush.h> #include <asm-generic/pgtable.h>
arch/sparc/mm/Makefile +1 -1
@@ -5,7 +5,7 @@ asflags-y := -ansi ccflags-y := -Werror -obj-$(CONFIG_SPARC64) += ultra.o tlb.o tsb.o gup.o +obj-$(CONFIG_SPARC64) += ultra.o tlb.o tsb.o obj-y += fault_$(BITS).o obj-y += init_$(BITS).o obj-$(CONFIG_SPARC32) += extable.o srmmu.o iommu.o io-unit.o
arch/sparc/mm/gup.c deleted +0 -340
340 lines not on this page · raw message
Signature · 1 line
2.20.1
Re: [PATCH 10/16] sparc64: use the generic get_user_pages_fast code
Quoted 13 lines · Hillf Danton
Hi Christoph
Quoted 9 lines · On Sat, 1 Jun 2019 09:49:53 +0200 Christoph Hellwig wrote:
arch/sparc/include/asm/pgtable_64.h +0
-0@@ -1098,6 +1098,24 @@ static inline unsigned long untagged_addr(unsigned long start) }#define untagged_addr untagged_addr
Quoted 11 lines
arch/sparc/include/asm/pgtable_64.h static inline unsigned long untagged_addr(unsigned long start) · line 1107
Quoted 7 lines
arch/sparc/include/asm/pgtable_64.h +7
-0+static inline bool pte_access_permitted (pte_t pte,bool write) +{ +u64 prot; + +if (tlb_type = hypervisor) { + prot = _PAGE_PRESENT_4V | _PAGE_P_4V; +if (prot)Feel free to correct me if I misread or miss anything.
It looks like a typo: s/prot/write/, as checking_PAGE_PRESENT_4Vand_PAGE_P_4Vmakes prot always have_PAGE_WRITE_4Vset, regardless of write.
True, the if prot should be if write.
PATCH 11/16 mm: rename CONFIG_HAVE_GENERIC_GUP to CONFIG_HAVE_FAST_GUP
In reply to Christoph HellwigWe only support the generic GUP now, so rename the config option to be more clear, and always use the mm/Kconfig definition of the symbol and select it from the arch Kconfigs.
--- arch/arm/Kconfig | 5 +---- arch/arm64/Kconfig | 4 +--- arch/mips/Kconfig | 2 +- arch/powerpc/Kconfig | 2 +- arch/s390/Kconfig | 2 +- arch/sh/Kconfig | 2 +- arch/sparc/Kconfig | 2 +- arch/x86/Kconfig | 4 +--- mm/Kconfig | 2 +- mm/gup.c | 4 ++-- 10 files changed, 11 insertions(+), 18 deletions(-)
arch/arm/Kconfig +1 -4
@@ -73,6 +73,7 @@ config ARM select HAVE_DYNAMIC_FTRACE_WITH_REGS if HAVE_DYNAMIC_FTRACE select HAVE_EFFICIENT_UNALIGNED_ACCESS if (CPU_V6 || CPU_V6K || CPU_V7) && MMU select HAVE_EXIT_THREAD + select HAVE_FAST_GUP if ARM_LPAE select HAVE_FTRACE_MCOUNT_RECORD if !XIP_KERNEL select HAVE_FUNCTION_GRAPH_TRACER if !THUMB2_KERNEL && !CC_IS_CLANG select HAVE_FUNCTION_TRACER if !XIP_KERNEL
@@ -1596,10 +1597,6 @@ config ARCH_SELECT_MEMORY_MODEL config HAVE_ARCH_PFN_VALID def_bool ARCH_HAS_HOLES_MEMORYMODEL || !SPARSEMEM -config HAVE_GENERIC_GUP - def_bool y - depends on ARM_LPAE - config HIGHMEM bool "High Memory Support" depends on MMU
arch/arm64/Kconfig +1 -3
@@ -140,6 +140,7 @@ config ARM64 select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_EFFICIENT_UNALIGNED_ACCESS + select HAVE_FAST_GUP select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_TRACER select HAVE_FUNCTION_GRAPH_TRACER
@@ -262,9 +263,6 @@ config GENERIC_CALIBRATE_DELAY config ZONE_DMA32 def_bool y -config HAVE_GENERIC_GUP - def_bool y - config ARCH_ENABLE_MEMORY_HOTPLUG def_bool y
arch/mips/Kconfig +1 -1
@@ -54,10 +54,10 @@ config MIPS select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_EXIT_THREAD + select HAVE_FAST_GUP select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER - select HAVE_GENERIC_GUP select HAVE_IDE select HAVE_IOREMAP_PROT select HAVE_IRQ_EXIT_ON_IRQ_STACK
arch/powerpc/Kconfig +1 -1
@@ -185,12 +185,12 @@ config PPC select HAVE_DYNAMIC_FTRACE_WITH_REGS if MPROFILE_KERNEL select HAVE_EBPF_JIT if PPC64 select HAVE_EFFICIENT_UNALIGNED_ACCESS if !(CPU_LITTLE_ENDIAN && POWER7_CPU) + select HAVE_FAST_GUP select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_ERROR_INJECTION select HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER select HAVE_GCC_PLUGINS if GCC_VERSION >= 50200 # plugin support on gcc <= 5.1 is buggy on PPC - select HAVE_GENERIC_GUP select HAVE_HW_BREAKPOINT if PERF_EVENTS && (PPC_BOOK3S || PPC_8xx) select HAVE_IDE select HAVE_IOREMAP_PROT
arch/s390/Kconfig +1 -1
@@ -137,6 +137,7 @@ config S390 select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE select HAVE_DYNAMIC_FTRACE_WITH_REGS + select HAVE_FAST_GUP select HAVE_EFFICIENT_UNALIGNED_ACCESS select HAVE_FENTRY select HAVE_FTRACE_MCOUNT_RECORD
@@ -144,7 +145,6 @@ config S390 select HAVE_FUNCTION_TRACER select HAVE_FUTEX_CMPXCHG if FUTEX select HAVE_GCC_PLUGINS - select HAVE_GENERIC_GUP select HAVE_KERNEL_BZIP2 select HAVE_KERNEL_GZIP select HAVE_KERNEL_LZ4
arch/sh/Kconfig +1 -1
@@ -14,7 +14,7 @@ config SUPERH select HAVE_ARCH_TRACEHOOK select HAVE_PERF_EVENTS select HAVE_DEBUG_BUGVERBOSE - select HAVE_GENERIC_GUP + select HAVE_FAST_GUP select ARCH_HAVE_CUSTOM_GPIO_H select ARCH_HAVE_NMI_SAFE_CMPXCHG if (GUSA_RB || CPU_SH4A) select ARCH_HAS_GCOV_PROFILE_ALL
arch/sparc/Kconfig +1 -1
@@ -28,7 +28,7 @@ config SPARC select RTC_DRV_M48T59 select RTC_SYSTOHC select HAVE_ARCH_JUMP_LABEL if SPARC64 - select HAVE_GENERIC_GUP if SPARC64 + select HAVE_FAST_GUP if SPARC64 select GENERIC_IRQ_SHOW select ARCH_WANT_IPC_PARSE_VERSION select GENERIC_PCI_IOMAP
arch/x86/Kconfig +1 -3
@@ -157,6 +157,7 @@ config X86 select HAVE_EFFICIENT_UNALIGNED_ACCESS select HAVE_EISA select HAVE_EXIT_THREAD + select HAVE_FAST_GUP select HAVE_FENTRY if X86_64 || DYNAMIC_FTRACE select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_GRAPH_TRACER
@@ -2874,9 +2875,6 @@ config HAVE_ATOMIC_IOMAP config X86_DEV_DMA_OPS bool -config HAVE_GENERIC_GUP - def_bool y - source "drivers/firmware/Kconfig" source "arch/x86/kvm/Kconfig"
mm/Kconfig +1 -1
@@ -132,7 +132,7 @@ config HAVE_MEMBLOCK_NODE_MAP config HAVE_MEMBLOCK_PHYS_MAP bool -config HAVE_GENERIC_GUP +config HAVE_FAST_GUP bool config ARCH_KEEP_MEMBLOCK
mm/gup.c +2 -2
@@ -1650,7 +1650,7 @@ struct page *get_dump_page(unsigned long addr)#endif /* CONFIG_ELF_CORE */ /* -* Generic Fast GUP +* Fast GUP * * get_user_pages_fast attempts to pin user pages by walking the page * tables directly and avoids taking locks. Thus the walker needs to be
@@ -1682,7 +1682,7 @@ struct page *get_dump_page(unsigned long addr)* * This code is based heavily on the PowerPC implementation by Nick Piggin. */ -#ifdef CONFIG_HAVE_GENERIC_GUP +#ifdef CONFIG_HAVE_FAST_GUP #ifdef CONFIG_GUP_GET_PTE_LOW_HIGH /* * WARNING: only to be used in the get_user_pages_fast() implementation.
Signature · 1 line
2.20.1
PATCH 12/16 mm: consolidate the get_user_pages* implementations
In reply to Christoph HellwigAlways build mm/gup.c, and move the nommu versions and replace the separate stubs for various functions by the default ones, with the _fast version always falling back to the slow path because gup_fast_permitted always returns false now if HAVE_FAST_GUP is not set, and we use the nommu version of __get_user_pages while keeping all the wrappers common.
This also ensures the new put_user_pages* helpers are available for nommu, as those are currently missing, which would create a problem as soon as we actually grew users for it.
--- mm/Kconfig | 1 + mm/Makefile | 4 +- mm/gup.c | 476 +++++++++++++++++++++++++++++----------------------- mm/nommu.c | 88 ---------- mm/util.c | 47 ------ 5 files changed, 269 insertions(+), 347 deletions(-)
mm/Kconfig +1 -0
@@ -133,6 +133,7 @@ config HAVE_MEMBLOCK_PHYS_MAP bool config HAVE_FAST_GUP + depends on MMU bool config ARCH_KEEP_MEMBLOCK
mm/Makefile +2 -2
@@ -22,7 +22,7 @@ KCOV_INSTRUMENT_mmzone.o := n KCOV_INSTRUMENT_vmstat.o := n mmu-y := nommu.o -mmu-$(CONFIG_MMU) := gup.o highmem.o memory.o mincore.o \ +mmu-$(CONFIG_MMU) := highmem.o memory.o mincore.o \ mlock.o mmap.o mmu_gather.o mprotect.o mremap.o \ msync.o page_vma_mapped.o pagewalk.o \ pgtable-generic.o rmap.o vmalloc.o
@@ -39,7 +39,7 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ mm_init.o mmu_context.o percpu.o slab_common.o \ compaction.o vmacache.o \ interval_tree.o list_lru.o workingset.o \ - debug.o $(mmu-y) + debug.o gup.o $(mmu-y) # Give 'page_alloc' its own module-parameter namespace page-alloc-y := page_alloc.o
mm/gup.c +266 -210
560 lines not on this page · raw message
mm/nommu.c +0 -88
94 lines not on this page · raw message
mm/util.c +0 -47
53 lines not on this page · raw message
Signature · 1 line
2.20.1
John Hubbard
reviewerRe: [PATCH 12/16] mm: consolidate the get_user_pages* implementations
Quoted 10 lines · Christoph Hellwig
Always build
mm/gup.c, and move the nommu versions and replace the separate stubs for various functions by the default ones, with the_fastversion always falling back to the slow path becausegup_fast_permittedalways returns false now ifHAVE_FAST_GUPis not set, and we use the nommu version of__get_user_pageswhile keeping all the wrappers common.This also ensures the new put_user_pages* helpers are available for nommu, as those are currently missing, which would create a problem as soon as we actually grew users for it.
Hi Christoph,
Thanks for fixing up the nommu case. And the patchset overall is a huge relief to see, because I'd filed those arches under the "despair" category for the gup conversions. :)
I started reviewing this one patch, and it's kind of messy figuring out if the code motion preserves everything because of
all the consolidation from other places, plus having to move things in and out of the ifdef blocks. So I figured I'd check and see if this is going to make it past RFC status soon, and if it's going before or after Ira's recent RFC ("RDMA/FS DAX truncate proposal").
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Quoted 776 lines
Signed-off-by Christoph Hellwig <[email protected]>--- mm/Kconfig | 1 + mm/Makefile | 4 +- mm/gup.c | 476 +++++++++++++++++++++++++++++----------------------- mm/nommu.c | 88 ---------- mm/util.c | 47 ------ 5 files changed, 269 insertions(+), 347 deletions(-)mm/Kconfig +1
-0@@ -133,6 +133,7 @@ config HAVE_MEMBLOCK_PHYS_MAP bool config HAVE_FAST_GUP + depends on MMU bool config ARCH_KEEP_MEMBLOCKmm/Makefile +2
-2@@ -22,7 +22,7 @@ KCOV_INSTRUMENT_mmzone.o := n KCOV_INSTRUMENT_vmstat.o := n mmu-y := nommu.o -mmu-$(CONFIG_MMU) := gup.o highmem.o memory.o mincore.o \ +mmu-$(CONFIG_MMU) := highmem.o memory.o mincore.o \ mlock.o mmap.o mmu_gather.o mprotect.o mremap.o \ msync.o page_vma_mapped.o pagewalk.o \ pgtable-generic.o rmap.o vmalloc.o@@ -39,7 +39,7 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ mm_init.o mmu_context.o percpu.o slab_common.o \ compaction.o vmacache.o \ interval_tree.o list_lru.o workingset.o \ - debug.o $(mmu-y) + debug.o gup.o $(mmu-y) # Give 'page_alloc' its own module-parameter namespace page-alloc-y := page_alloc.omm/gup.c +266
-210@@ -134,6 +134,7 @@ void put_user_pages(struct page **pages, unsigned long npages) }EXPORT_SYMBOL (put_user_pages); +#ifdef CONFIG_MMU static struct page *no_page_table (struct vm_area_struct *vma,unsigned int flags) {@@ -1099,86 +1100,6 @@ static __always_inline long __get_user_pages_locked(struct task_struct *tsk,return pages_done; } -/* -* We can leverage the VM_FAULT_RETRY functionality in the page fault -* paths better by using either get_user_pages_locked() or -* get_user_pages_unlocked(). -* -* get_user_pages_locked() is suitable to replace the form: -* -* down_read(&mm->mmap_sem); -* do_something() -* get_user_pages(tsk, mm, ..., pages, NULL); -* up_read(&mm->mmap_sem); -* -* to: -* -* int locked = 1; -* down_read(&mm->mmap_sem); -* do_something() -* get_user_pages_locked(tsk, mm, ..., pages, &locked); -* if (locked) -* up_read(&mm->mmap_sem); -*/ -long get_user_pages_locked (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages, -int *locked) -{ -/* -* FIXME: Current FOLL_LONGTERM behavior is incompatible with -* FAULT_FLAG_ALLOW_RETRY because of the FS DAX check requirement on -* vmas. As there are no users of this flag in this call we simply -* disallow this option for now. -*/ -if (WARN_ON_ONCE (gup_flags &FOLL_LONGTERM )) -return -EINVAL ; - -return __get_user_pages_locked (current, current->mm, start, nr_pages, - pages,NULL , locked, - gup_flags |FOLL_TOUCH ); -} -EXPORT_SYMBOL (get_user_pages_locked); - -/* -* get_user_pages_unlocked() is suitable to replace the form: -* -* down_read(&mm->mmap_sem); -* get_user_pages(tsk, mm, ..., pages, NULL); -* up_read(&mm->mmap_sem); -* -* with: -* -* get_user_pages_unlocked(tsk, mm, ..., pages); -* -* It is functionally equivalent to get_user_pages_fast so -* get_user_pages_fast should be used instead if specific gup_flags -* (e.g. FOLL_FORCE) are not required. -*/ -long get_user_pages_unlocked (unsigned long start,unsigned long nr_pages, -struct page **pages,unsigned int gup_flags) -{ -struct mm_struct *mm = current->mm; -int locked =1 ; -long ret; - -/* -* FIXME: Current FOLL_LONGTERM behavior is incompatible with -* FAULT_FLAG_ALLOW_RETRY because of the FS DAX check requirement on -* vmas. As there are no users of this flag in this call we simply -* disallow this option for now. -*/ -if (WARN_ON_ONCE (gup_flags &FOLL_LONGTERM )) -return -EINVAL ; - -down_read (&mm->mmap_sem); - ret =__get_user_pages_locked (current, mm, start, nr_pages, pages,NULL , - &locked, gup_flags |FOLL_TOUCH ); -if (locked) -up_read (&mm->mmap_sem); -return ret; -} -EXPORT_SYMBOL (get_user_pages_unlocked); -/* * get_user_pages_remote() - pin user pages in memory * @tsk: the task_struct to use for page fault accounting, or @@ -1255,6 +1176,199 @@ long get_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm, }EXPORT_SYMBOL (get_user_pages_remote); +/** +* populate_vma_page_range() - populate a range of pages in the vma. +* @vma: target vma +* @start: start address +* @end: end address +* @nonblocking: +* +* This takes care of mlocking the pages too if VM_LOCKED is set. +* +* return 0 on success, negative error code on error. +* +* vma->vm_mm->mmap_sem must be held. +* +* If @nonblocking is NULL, it may be held for read or write and will +* be unperturbed. +* +* If @nonblocking is non-NULL, it must held for read only and may be +* released. If it's released, *@nonblocking will be set to 0. +*/ +long populate_vma_page_range (struct vm_area_struct *vma, +unsigned long start,unsigned long end,int *nonblocking) +{ +struct mm_struct *mm = vma->vm_mm; +unsigned long nr_pages = (end - start) /PAGE_SIZE ; +int gup_flags; + +VM_BUG_ON (start & ~PAGE_MASK ); +VM_BUG_ON (end & ~PAGE_MASK ); +VM_BUG_ON_VMA (start < vma->vm_start, vma); +VM_BUG_ON_VMA (end > vma->vm_end, vma); +VM_BUG_ON_MM (!rwsem_is_locked (&mm->mmap_sem), mm); + + gup_flags =FOLL_TOUCH |FOLL_POPULATE |FOLL_MLOCK ; +if (vma->vm_flags &VM_LOCKONFAULT ) + gup_flags &= ~FOLL_POPULATE ; +/* +* We want to touch writable mappings with a write fault in order +* to break COW, except for shared mappings because these don't COW +* and we would not want to dirty them for nothing. +*/ +if ((vma->vm_flags & (VM_WRITE |VM_SHARED )) =VM_WRITE ) + gup_flags |=FOLL_WRITE ; + +/* +* We want mlock to succeed for regions that have any permissions +* other than PROT_NONE. +*/ +if (vma->vm_flags & (VM_READ |VM_WRITE |VM_EXEC )) + gup_flags |=FOLL_FORCE ; + +/* +* We made sure addr is within a VMA, so the following will +* not result in a stack expansion that recurses back here. +*/ +return __get_user_pages (current, mm, start, nr_pages, gup_flags, +NULL ,NULL , nonblocking); +} + +/* +* __mm_populate - populate and/or mlock pages within a range of address space. +* +* This is used to implement mlock() and the MAP_POPULATE / MAP_LOCKED mmap +* flags. VMAs must be already marked with the desired vm_flags, and +* mmap_sem must not be held. +*/ +int __mm_populate (unsigned long start,unsigned long len,int ignore_errors) +{ +struct mm_struct *mm = current->mm; +unsigned long end, nstart, nend; +struct vm_area_struct *vma =NULL ; +int locked =0 ; +long ret =0 ; + + end = start + len; + +for (nstart = start; nstart < end; nstart = nend) { +/* +* We want to fault in pages for [nstart; end) address range. +* Find first corresponding VMA. +*/ +if (!locked) { + locked =1 ; +down_read (&mm->mmap_sem); + vma =find_vma (mm, nstart); + }else if (nstart >= vma->vm_end) + vma = vma->vm_next; +if (!vma || vma->vm_start >= end) +break ; +/* +* Set [nstart; nend) to intersection of desired address +* range with the first VMA. Also, skip undesirable VMA types. +*/ + nend =min (end, vma->vm_end); +if (vma->vm_flags & (VM_IO |VM_PFNMAP )) +continue ; +if (nstart < vma->vm_start) + nstart = vma->vm_start; +/* +* Now fault in a range of pages. populate_vma_page_range() +* double checks the vma flags, so that it won't mlock pages +* if the vma was already munlocked. +*/ + ret =populate_vma_page_range (vma, nstart, nend, &locked); +if (ret <0 ) { +if (ignore_errors) { + ret =0 ; +continue ;/* continue at next VMA */ + } +break ; + } + nend = nstart + ret *PAGE_SIZE ; + ret =0 ; + } +if (locked) +up_read (&mm->mmap_sem); +return ret;/* 0 or negative error code */ +} + +/** +* get_dump_page() - pin user page in memory while writing it to core dump +* @addr: user address +* +* Returns struct page pointer of user page pinned for dump, +* to be freed afterwards by put_page(). +* +* Returns NULL on any kind of failure - a hole must then be inserted into +* the corefile, to preserve alignment with its headers; and also returns +* NULL wherever the ZERO_PAGE, or an anonymous pte_none, has been found - +* allowing a hole to be left in the corefile to save diskspace. +* +* Called without mmap_sem, but after all other threads have been killed. +*/ +#ifdef CONFIG_ELF_CORE +struct page *get_dump_page (unsigned long addr) +{ +struct vm_area_struct *vma; +struct page *page; + +if (__get_user_pages (current, current->mm, addr,1 , +FOLL_FORCE |FOLL_DUMP |FOLL_GET , &page, &vma, +NULL ) <1 ) +return NULL ; +flush_cache_page (vma, addr,page_to_pfn (page)); +return page; +} +#endif /* CONFIG_ELF_CORE */ + +#else /* CONFIG_MMU */ +static long __get_user_pages_locked (struct task_struct *tsk, +struct mm_struct *mm,unsigned long start, +unsigned long nr_pages,struct page **pages, +struct vm_area_struct **vmas,int *locked, +unsigned int foll_flags) +{ +struct vm_area_struct *vma; +unsigned long vm_flags; +int i; + +/* calculate required read or write permissions. +* If FOLL_FORCE is set, we only require the "MAY" flags. +*/ + vm_flags = (foll_flags &FOLL_WRITE ) ? + (VM_WRITE |VM_MAYWRITE ) : (VM_READ |VM_MAYREAD ); + vm_flags &= (foll_flags &FOLL_FORCE ) ? + (VM_MAYREAD |VM_MAYWRITE ) : (VM_READ |VM_WRITE ); + +for (i =0 ; i < nr_pages; i++) { + vma =find_vma (mm, start); +if (!vma) + goto finish_or_fault; + +/* protect what we can, including chardevs */ +if ((vma->vm_flags & (VM_IO |VM_PFNMAP )) || + !(vm_flags & vma->vm_flags)) + goto finish_or_fault; + +if (pages) { + pages[i] =virt_to_page (start); +if (pages[i]) +get_page (pages[i]); + } +if (vmas) + vmas[i] = vma; + start = (start +PAGE_SIZE ) &PAGE_MASK ; + } + +return i; + +finish_or_fault: +return i ? : -EFAULT ; +} +#endif /* !CONFIG_MMU */ +#if defined(CONFIG_FS_DAX ) || defined (CONFIG_CMA )static bool check_dax_vmas (struct vm_area_struct **vmas,long nr_pages) {@@ -1416,7 +1530,7 @@ static long check_and_migrate_cma_pages(struct task_struct *tsk, {return nr_pages; } -#endif +#endif /* CONFIG_CMA */ /* * __gup_longterm_locked() is a wrapper for __get_user_pages_locked which @@ -1502,152 +1616,85 @@ long get_user_pages(unsigned long start, unsigned long nr_pages, }EXPORT_SYMBOL (get_user_pages); -/** -* populate_vma_page_range() - populate a range of pages in the vma. -* @vma: target vma -* @start: start address -* @end: end address -* @nonblocking: -* -* This takes care of mlocking the pages too if VM_LOCKED is set. +/* +* We can leverage the VM_FAULT_RETRY functionality in the page fault +* paths better by using either get_user_pages_locked() or +* get_user_pages_unlocked(). * -* return 0 on success, negative error code on error. +* get_user_pages_locked() is suitable to replace the form: * -* vma->vm_mm->mmap_sem must be held. +* down_read(&mm->mmap_sem); +* do_something() +* get_user_pages(tsk, mm, ..., pages, NULL); +* up_read(&mm->mmap_sem); * -* If @nonblocking is NULL, it may be held for read or write and will -* be unperturbed. +* to: * -* If @nonblocking is non-NULL, it must held for read only and may be -* released. If it's released, *@nonblocking will be set to 0. +* int locked = 1; +* down_read(&mm->mmap_sem); +* do_something() +* get_user_pages_locked(tsk, mm, ..., pages, &locked); +* if (locked) +* up_read(&mm->mmap_sem); */ -long populate_vma_page_range (struct vm_area_struct *vma, -unsigned long start,unsigned long end,int *nonblocking) +long get_user_pages_locked (unsigned long start,unsigned long nr_pages, +unsigned int gup_flags,struct page **pages, +int *locked) { -struct mm_struct *mm = vma->vm_mm; -unsigned long nr_pages = (end - start) /PAGE_SIZE ; -int gup_flags; - -VM_BUG_ON (start & ~PAGE_MASK ); -VM_BUG_ON (end & ~PAGE_MASK ); -VM_BUG_ON_VMA (start < vma->vm_start, vma); -VM_BUG_ON_VMA (end > vma->vm_end, vma); -VM_BUG_ON_MM (!rwsem_is_locked (&mm->mmap_sem), mm); - - gup_flags =FOLL_TOUCH |FOLL_POPULATE |FOLL_MLOCK ; -if (vma->vm_flags &VM_LOCKONFAULT ) - gup_flags &= ~FOLL_POPULATE ; -/* -* We want to touch writable mappings with a write fault in order -* to break COW, except for shared mappings because these don't COW -* and we would not want to dirty them for nothing. -*/ -if ((vma->vm_flags & (VM_WRITE |VM_SHARED )) =VM_WRITE ) - gup_flags |=FOLL_WRITE ; -/* -* We want mlock to succeed for regions that have any permissions -* other than PROT_NONE. +* FIXME: Current FOLL_LONGTERM behavior is incompatible with +* FAULT_FLAG_ALLOW_RETRY because of the FS DAX check requirement on +* vmas. As there are no users of this flag in this call we simply +* disallow this option for now. */ -if (vma->vm_flags & (VM_READ |VM_WRITE |VM_EXEC )) - gup_flags |=FOLL_FORCE ; +if (WARN_ON_ONCE (gup_flags &FOLL_LONGTERM )) +return -EINVAL ; -/* -* We made sure addr is within a VMA, so the following will -* not result in a stack expansion that recurses back here. -*/ -return __get_user_pages (current, mm, start, nr_pages, gup_flags, -NULL ,NULL , nonblocking); +return __get_user_pages_locked (current, current->mm, start, nr_pages, + pages,NULL , locked, + gup_flags |FOLL_TOUCH ); } +EXPORT_SYMBOL (get_user_pages_locked);/* -* __mm_populate - populate and/or mlock pages within a range of address space. +* get_user_pages_unlocked() is suitable to replace the form: * -* This is used to implement mlock() and the MAP_POPULATE / MAP_LOCKED mmap -* flags. VMAs must be already marked with the desired vm_flags, and -* mmap_sem must not be held. +* down_read(&mm->mmap_sem); +* get_user_pages(tsk, mm, ..., pages, NULL); +* up_read(&mm->mmap_sem); +* +* with: +* +* get_user_pages_unlocked(tsk, mm, ..., pages); +* +* It is functionally equivalent to get_user_pages_fast so +* get_user_pages_fast should be used instead if specific gup_flags +* (e.g. FOLL_FORCE) are not required. */ -int __mm_populate (unsigned long start,unsigned long len,int ignore_errors) +long get_user_pages_unlocked (unsigned long start,unsigned long nr_pages, +struct page **pages,unsigned int gup_flags) {struct mm_struct *mm = current->mm; -unsigned long end, nstart, nend; -struct vm_area_struct *vma =NULL ; -int locked =0 ; -long ret =0 ; +int locked =1 ; +long ret; - end = start + len; +/* +* FIXME: Current FOLL_LONGTERM behavior is incompatible with +* FAULT_FLAG_ALLOW_RETRY because of the FS DAX check requirement on +* vmas. As there are no users of this flag in this call we simply +* disallow this option for now. +*/ +if (WARN_ON_ONCE (gup_flags &FOLL_LONGTERM )) +return -EINVAL ; -for (nstart = start; nstart < end; nstart = nend) { -/* -* We want to fault in pages for [nstart; end) address range. -* Find first corresponding VMA. -*/ -if (!locked) { - locked =1 ; -down_read (&mm->mmap_sem); - vma =find_vma (mm, nstart); - }else if (nstart >= vma->vm_end) - vma = vma->vm_next; -if (!vma || vma->vm_start >= end) -break ; -/* -* Set [nstart; nend) to intersection of desired address -* range with the first VMA. Also, skip undesirable VMA types. -*/ - nend =min (end, vma->vm_end); -if (vma->vm_flags & (VM_IO |VM_PFNMAP )) -continue ; -if (nstart < vma->vm_start) - nstart = vma->vm_start; -/* -* Now fault in a range of pages. populate_vma_page_range() -* double checks the vma flags, so that it won't mlock pages -* if the vma was already munlocked. -*/ - ret =populate_vma_page_range (vma, nstart, nend, &locked); -if (ret <0 ) { -if (ignore_errors) { - ret =0 ; -continue ;/* continue at next VMA */ - } -break ; - } - nend = nstart + ret *PAGE_SIZE ; - ret =0 ; - } +down_read (&mm->mmap_sem); + ret =__get_user_pages_locked (current, mm, start, nr_pages, pages,NULL , + &locked, gup_flags |FOLL_TOUCH );if (locked)up_read (&mm->mmap_sem); -return ret;/* 0 or negative error code */ -} - -/** -* get_dump_page() - pin user page in memory while writing it to core dump -* @addr: user address -* -* Returns struct page pointer of user page pinned for dump, -* to be freed afterwards by put_page(). -* -* Returns NULL on any kind of failure - a hole must then be inserted into -* the corefile, to preserve alignment with its headers; and also returns -* NULL wherever the ZERO_PAGE, or an anonymous pte_none, has been found - -* allowing a hole to be left in the corefile to save diskspace. -* -* Called without mmap_sem, but after all other threads have been killed. -*/ -#ifdef CONFIG_ELF_CORE -struct page *get_dump_page (unsigned long addr) -{ -struct vm_area_struct *vma; -struct page *page; - -if (__get_user_pages (current, current->mm, addr,1 , -FOLL_FORCE |FOLL_DUMP |FOLL_GET , &page, &vma, -NULL ) <1 ) -return NULL ; -flush_cache_page (vma, addr,page_to_pfn (page)); -return page; +return ret; } -#endif /* CONFIG_ELF_CORE */ +EXPORT_SYMBOL (get_user_pages_unlocked);/* * Fast GUP @@ -1682,7 +1729,7 @@ struct page *get_dump_page(unsigned long addr)* * This code is based heavily on the PowerPC implementation by Nick Piggin. */ -#ifdef CONFIG_HAVE_FAST_GUP +#if defined(CONFIG_MMU ) && defined(CONFIG_HAVE_FAST_GUP )#ifdef CONFIG_GUP_GET_PTE_LOW_HIGH /* * WARNING: only to be used in the get_user_pages_fast() implementation. @@ -2159,6 +2206,12 @@ static void gup_pgd_range(unsigned long addr, unsigned long end, return; }while (pgdp++, addr = next, addr != end); } +#else +static inline void gup_pgd_range (unsigned long addr,unsigned long end, +unsigned int flags,struct page **pages,int *nr) +{ +} +#endif /* CONFIG_HAVE_FAST_GUP */ #ifndef gup_fast_permitted/* @@ -2167,7 +2220,7 @@ static void gup_pgd_range(unsigned long addr, unsigned long end,*/ static bool gup_fast_permitted (unsigned long start,unsigned long end) { -return true; +return IS_ENABLED (CONFIG_HAVE_FAST_GUP ) ? true : false; }#endif @@ -2176,6 +2229,9 @@ static bool gup_fast_permitted(unsigned long start, unsigned long end)* the regular GUP. * Note a difference with get_user_pages_fast: this always returns the * number of pages pinned, 0 if no pages were pinned. +* +* If the architecture does not support this function, simply return with no +* pages pinned. */ int __get_user_pages_fast (unsigned long start,int nr_pages,int write,struct page **pages)@@ -2213,6 +2269,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,return nr; } +EXPORT_SYMBOL_GPL (__get_user_pages_fast);static int __gup_longterm_unlocked (unsigned long start,int nr_pages,unsigned int gup_flags,struct page **pages)@@ -2295,5 +2352,4 @@ int get_user_pages_fast(unsigned long start, int nr_pages,return ret; } - -#endif /* CONFIG_HAVE_GENERIC_GUP */ +EXPORT_SYMBOL_GPL (get_user_pages_fast);mm/nommu.c +0
-88@@ -111,94 +111,6 @@ unsigned int kobjsize(const void *objp)return PAGE_SIZE <<compound_order (page); } -static long __get_user_pages (struct task_struct *tsk,struct mm_struct *mm, -unsigned long start,unsigned long nr_pages, -unsigned int foll_flags,struct page **pages, -struct vm_area_struct **vmas,int *nonblocking) -{ -struct vm_area_struct *vma; -unsigned long vm_flags; -int i; - -/* calculate required read or write permissions. -* If FOLL_FORCE is set, we only require the "MAY" flags. -*/ - vm_flags = (foll_flags &FOLL_WRITE ) ? - (VM_WRITE |VM_MAYWRITE ) : (VM_READ |VM_MAYREAD ); - vm_flags &= (foll_flags &FOLL_FORCE ) ? - (VM_MAYREAD |VM_MAYWRITE ) : (VM_READ |VM_WRITE ); - -for (i =0 ; i < nr_pages; i++) { - vma =find_vma (mm, start); -if (!vma) - goto finish_or_fault; - -/* protect what we can, including chardevs */ -if ((vma->vm_flags & (VM_IO |VM_PFNMAP )) || - !(vm_flags & vma->vm_flags)) - goto finish_or_fault; - -if (pages) { - pages[i] =virt_to_page (start); -if (pages[i]) -get_page (pages[i]); - } -if (vmas) - vmas[i] = vma; - start = (start +PAGE_SIZE ) &PAGE_MASK ; - } - -return i; - -finish_or_fault: -return i ? : -EFAULT ; -} - -/* -* get a list of pages in an address range belonging to the specified process -* and indicate the VMA that covers each page -* - this is potentially dodgy as we may end incrementing the page count of a -* slab page or a secondary page from a compound page -* - don't permit access to VMAs that don't support it, such as I/O mappings -*/ -long get_user_pages (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages, -struct vm_area_struct **vmas) -{ -return __get_user_pages (current, current->mm, start, nr_pages, - gup_flags, pages, vmas,NULL ); -} -EXPORT_SYMBOL (get_user_pages); - -long get_user_pages_locked (unsigned long start,unsigned long nr_pages, -unsigned int gup_flags,struct page **pages, -int *locked) -{ -return get_user_pages (start, nr_pages, gup_flags, pages,NULL ); -} -EXPORT_SYMBOL (get_user_pages_locked); - -static long __get_user_pages_unlocked (struct task_struct *tsk, -struct mm_struct *mm,unsigned long start, -unsigned long nr_pages,struct page **pages, -unsigned int gup_flags) -{ -long ret; -down_read (&mm->mmap_sem); - ret =__get_user_pages (tsk, mm, start, nr_pages, gup_flags, pages, -NULL ,NULL ); -up_read (&mm->mmap_sem); -return ret; -} - -long get_user_pages_unlocked (unsigned long start,unsigned long nr_pages, -struct page **pages,unsigned int gup_flags) -{ -return __get_user_pages_unlocked (current, current->mm, start, nr_pages, - pages, gup_flags); -} -EXPORT_SYMBOL (get_user_pages_unlocked); -/** * follow_pfn - look up PFN at a user virtual address * @vma: memory mapping mm/util.c +0
-47@@ -300,53 +300,6 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) }#endif -/* -* Like get_user_pages_fast() except its IRQ-safe in that it won't fall -* back to the regular GUP. -* Note a difference with get_user_pages_fast: this always returns the -* number of pages pinned, 0 if no pages were pinned. -* If the architecture does not support this function, simply return with no -* pages pinned. -*/ -int __weak __get_user_pages_fast (unsigned long start, -int nr_pages,int write,struct page **pages) -{ -return 0 ; -} -EXPORT_SYMBOL_GPL (__get_user_pages_fast); - -/** -* get_user_pages_fast() - pin user pages in memory -* @start: starting user address -* @nr_pages: number of pages from start to pin -* @gup_flags: flags modifying pin behaviour -* @pages: array that receives pointers to the pages pinned. -* Should be at least nr_pages long. -* -* get_user_pages_fast provides equivalent functionality to get_user_pages, -* operating on current and current->mm, with force=0 and vma=NULL. However -* unlike get_user_pages, it must be called without mmap_sem held. -* -* get_user_pages_fast may take mmap_sem and page table locks, so no -* assumptions can be made about lack of locking. get_user_pages_fast is to be -* implemented in a way that is advantageous (vs get_user_pages()) when the -* user memory area is already faulted in and present in ptes. However if the -* pages have to be faulted in, it may turn out to be slightly slower so -* callers need to carefully consider what to use. On many architectures, -* get_user_pages_fast simply falls back to get_user_pages. -* -* Return: number of pages pinned. This may be fewer than the number -* requested. If nr_pages is 0 or negative, returns 0. If no pages -* were pinned, returns -errno. -*/ -int __weak get_user_pages_fast (unsigned long start, -int nr_pages,unsigned int gup_flags, -struct page **pages) -{ -return get_user_pages_unlocked (start, nr_pages, pages, gup_flags); -} -EXPORT_SYMBOL_GPL (get_user_pages_fast); -unsigned long vm_mmap_pgoff (struct file *file,unsigned long addr,unsigned long len,unsigned long prot,unsigned long flag,unsigned long pgoff)
Re: [PATCH 12/16] mm: consolidate the get_user_pages* implementations
Quoted 6 lines · John Hubbard
I started reviewing this one patch, and it's kind of messy figuring out if the code motion preserves everything because of
all the consolidation from other places, plus having to move things in and out of the ifdef blocks. So I figured I'd check and see if this is going to make it past RFC status soon, and if it's going before or after Ira's recent RFC ("RDMA/FS DAX truncate proposal").
I don't like the huge moves either, but I can't really think of any better way to do it. Proposals welcome, though.
John Hubbard
reviewerRe: [PATCH 12/16] mm: consolidate the get_user_pages* implementations
Quoted 11 lines · Christoph Hellwig
Quoted 6 lines · John Hubbard
I started reviewing this one patch, and it's kind of messy figuring out if the code motion preserves everything because of
all the consolidation from other places, plus having to move things in and out of the ifdef blocks. So I figured I'd check and see if this is going to make it past RFC status soon, and if it's going before or after Ira's recent RFC ("RDMA/FS DAX truncate proposal").I don't like the huge moves either, but I can't really think of any better way to do it. Proposals welcome, though.
One way would be to do it in two patches:
1) Move the code into gup.c, maybe at the bottom. Surround each function or group of functions by whatever ifdefs they need.
2) Move code out of the bottom of gup.c, into the final location.
...but I'm not certain that will be that much better. In the spirit of not creating gratuitous work for others, I could try it out and send out something if it looks like it's noticeably easier to verify/review.
thanks,
Signature · 2 lines
John Hubbard NVIDIA
PATCH 13/16 mm: validate get_user_pages_fast flags
In reply to Christoph HellwigWe can only deal with FOLL_WRITE and/or FOLL_LONGTERM in get_user_pages_fast, so reject all other flags.
--- mm/gup.c | 3 +++ 1 file changed, 3 insertions(+)
mm/gup.c +3 -0
@@ -2316,6 +2316,9 @@ int get_user_pages_fast(unsigned long start, int nr_pages,unsigned long addr, len, end;int nr =0 , ret =0 ; +if (WARN_ON_ONCE (gup_flags & ~(FOLL_WRITE |FOLL_LONGTERM ))) +return -EINVAL ; + start =untagged_addr (start) &PAGE_MASK ; addr = start; len = (unsigned long ) nr_pages <<PAGE_SHIFT ;
Signature · 1 line
2.20.1
PATCH 14/16 mm: move the powerpc hugepd code to mm/gup.c
In reply to Christoph HellwigWhile only powerpc supports the hugepd case, the code is pretty generic and I'd like to keep all GUP internals in one place.
--- arch/powerpc/Kconfig | 1 + arch/powerpc/mm/hugetlbpage.c | 72 ------------------------------ include/linux/hugetlb.h | 18 -------- mm/Kconfig | 10 +++++ mm/gup.c | 82 +++++++++++++++++++++++++++++++++++ 5 files changed, 93 insertions(+), 90 deletions(-)
arch/powerpc/Kconfig +1 -0
@@ -125,6 +125,7 @@ config PPC select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_GCOV_PROFILE_ALL select ARCH_HAS_KCOV + select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_HAS_MMIOWB if PPC64 select ARCH_HAS_PHYS_TO_DMA select ARCH_HAS_PMEM_API if PPC64
arch/powerpc/mm/hugetlbpage.c +0 -72
81 lines not on this page · raw message
include/linux/hugetlb.h +0 -18
@@ -16,29 +16,11 @@ struct user_struct;struct mmu_gather ;#ifndef is_hugepd -/* -* Some architectures requires a hugepage directory format that is -* required to support multiple hugepage sizes. For example -* a4fe3ce76 "powerpc/mm: Allow more flexible layouts for hugepage pagetables" -* introduced the same on powerpc. This allows for a more flexible hugepage -* pagetable layout. -*/ typedef struct {unsigned long pd; }hugepd_t ;#define is_hugepd (hugepd) (0)#define __hugepd (x) ((hugepd_t) { (x) }) -static inline int gup_huge_pd (hugepd_t hugepd,unsigned long addr, -unsigned pdshift,unsigned long end, -int write,struct page **pages,int *nr) -{ -return 0 ; -} -#else -extern int gup_huge_pd (hugepd_t hugepd,unsigned long addr, -unsigned pdshift,unsigned long end, -int write,struct page **pages,int *nr);#endif -#ifdef CONFIG_HUGETLB_PAGE #include <linux/mempolicy.h>
mm/Kconfig +10 -0
@@ -769,4 +769,14 @@ config GUP_GET_PTE_LOW_HIGH config ARCH_HAS_PTE_SPECIAL bool +# +# Some architectures require a special hugepage directory format that is +# required to support multiple hugepage sizes. For example a4fe3ce76 +# "powerpc/mm: Allow more flexible layouts for hugepage pagetables" +# introduced it on powerpc. This allows for a more flexible hugepage +# pagetable layouts. +# +config ARCH_HAS_HUGEPD + bool + endmenu
mm/gup.c +82 -0
88 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH 15/16 mm: switch gup_hugepte to use try_get_compound_head
In reply to Christoph HellwigThis applies the overflow fixes from 8fde12ca79aff ("mm: prevent get_user_pages() from overflowing page refcount") to the powerpc hugepd code and brings it back in sync with the other GUP cases.
--- mm/gup.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/gup.c +2 -1
@@ -2006,7 +2006,8 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, refs++; }while (addr +=PAGE_SIZE , addr != end); -if (!page_cache_add_speculative (head, refs)) { + head =try_get_compound_head (head, refs); +if (!head) { *nr -= refs;return 0 ; }
Signature · 1 line
2.20.1
PATCH 16/16 mm: mark the page referenced in gup_hugepte
In reply to Christoph HellwigAll other get_user_page_fast cases mark the page referenced, so do this here as well.
--- mm/gup.c | 1 + 1 file changed, 1 insertion(+)
mm/gup.c +1 -0
@@ -2020,6 +2020,7 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr,return 0 ; } +SetPageReferenced (head);return 1 ; }
Signature · 1 line
2.20.1
David Miller
maintainerFrom: Christoph Hellwig <[email protected]>
Date: Sat, 1 Jun 2019 09:49:43 +0200
Quoted 6 lines
below is a series to switch mips, sh and sparc64 to use the generic GUP code so that we only have one codebase to touch for further improvements to this code. I don't have hardware for any of these architectures, and generally no clue about their page table management, so handle with care.
Quoted 6 lines
Changes since v1:
- fix various issues found by the build bot
- cherry pick and use theuntagged_addrhelper form Andrey
- add various refactoring patches to share more code over architectures
- move the powerpc hugepd code tomm/gup.cand sync it with the generic hup semantics
I will today look seriously at the sparc64 stuff wrt. tagged pointers.