Add support for fast mremap
Subsystems (29)
cover Add support for fast mremap
Hi,
Here is the latest "fast mremap" series. This just a repost with Kirill's Acked-bys added. I would like this to be considered for linux -next. I also dropped the CONFIG enablement patch for arm64 since I am yet to test it with the new TLB flushing code that is in very recent kernel releases. (None of my arm64 devices run mainline right now.) so I will post the arm64 enablement once I get to that. The performance numbers in the series are for x86.
List of patches in series:
(1) mm: select HAVE_MOVE_PMD in x86 for faster mremap
(2) mm: speed up mremap by 20x on large regions (v4) v1->v2: Added support for per-arch enablement (Kirill Shutemov) v2->v3: Updated commit message to state the optimization may also run for non-thp type of systems (Daniel Col).v3->v4: Remove useless pmd_lock check (Kirill Shutemov)
Rebased ontop of Linus's master, updated perf results based on x86 testing. Added Kirill's Acks.
(3) mm: treewide: remove unused address argument from pte_alloc functions (v2) v1->v2: fix arch/um/ prototype which was missed in v1 (Anton Ivanov) update changelog with manual fixups for m68k and microblaze.
not included - (4) mm: select HAVE_MOVE_PMD in arm64 for faster mremap
This patch is dropped since last posting pending further performance testing on arm64 with new TLB gather updates. See notes in patch titled "mm: speed up mremap by 500x on large regions" for more details.
Joel Fernandes (Google) (3):
mm: treewide: remove unused address argument from pte_alloc functions
(v2)
mm: speed up mremap by 20x on large regions (v4)
mm: select HAVE_MOVE_PMD in x86 for faster mremap
arch/Kconfig | 5 ++ arch/alpha/include/asm/pgalloc.h | 6 +- arch/arc/include/asm/pgalloc.h | 5 +- arch/arm/include/asm/pgalloc.h | 4 +- arch/arm64/include/asm/pgalloc.h | 4 +- arch/hexagon/include/asm/pgalloc.h | 6 +- arch/ia64/include/asm/pgalloc.h | 5 +- arch/m68k/include/asm/mcf_pgalloc.h | 8 +-- arch/m68k/include/asm/motorola_pgalloc.h | 4 +- arch/m68k/include/asm/sun3_pgalloc.h | 6 +- arch/microblaze/include/asm/pgalloc.h | 19 +----- arch/microblaze/mm/pgtable.c | 3 +- arch/mips/include/asm/pgalloc.h | 6 +- arch/nds32/include/asm/pgalloc.h | 5 +- arch/nios2/include/asm/pgalloc.h | 6 +- arch/openrisc/include/asm/pgalloc.h | 5 +- arch/openrisc/mm/ioremap.c | 3 +- arch/parisc/include/asm/pgalloc.h | 4 +- arch/powerpc/include/asm/book3s/32/pgalloc.h | 4 +- arch/powerpc/include/asm/book3s/64/pgalloc.h | 12 ++-- arch/powerpc/include/asm/nohash/32/pgalloc.h | 4 +- arch/powerpc/include/asm/nohash/64/pgalloc.h | 6 +- arch/powerpc/mm/pgtable-book3s64.c | 2 +- arch/powerpc/mm/pgtable_32.c | 4 +- arch/riscv/include/asm/pgalloc.h | 6 +- arch/s390/include/asm/pgalloc.h | 4 +- arch/sh/include/asm/pgalloc.h | 6 +- arch/sparc/include/asm/pgalloc_32.h | 5 +- arch/sparc/include/asm/pgalloc_64.h | 6 +- arch/sparc/mm/init_64.c | 6 +- arch/sparc/mm/srmmu.c | 4 +- arch/um/include/asm/pgalloc.h | 4 +- arch/um/kernel/mem.c | 4 +- arch/unicore32/include/asm/pgalloc.h | 4 +- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgalloc.h | 4 +- arch/x86/mm/pgtable.c | 4 +- arch/xtensa/include/asm/pgalloc.h | 8 +-- include/linux/mm.h | 13 ++-- mm/huge_memory.c | 8 +-- mm/kasan/kasan_init.c | 2 +- mm/memory.c | 17 +++--- mm/migrate.c | 2 +- mm/mremap.c | 62 +++++++++++++++++++- mm/userfaultfd.c | 2 +- virt/kvm/arm/mmu.c | 2 +- 46 files changed, 163 insertions(+), 147 deletions(-)
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
PATCH 1/3 mm: treewide: remove unused address argument from pte_alloc functions (v2)
From: "Joel Fernandes (Google)" <[email protected]>
This series speeds up mremap(2) syscall by copying page tables at the PMD level even for non-THP systems. There is concern that the extra 'address' argument that mremap passes to pte_alloc may do something subtle architecture related in the future that may make the scheme not work. Also we find that there is no point in passing the 'address' to pte_alloc since its unused. So this patch therefore removes this argument tree-wide resulting in a nice negative diff as well. Also ensuring along the way that the enabled architectures do not do anything funky with 'address' argument that goes unnoticed by the optimization.
Build and boot tested on x86-64. Build tested on arm64.
The changes were obtained by applying the following Coccinelle script. (thanks Julia for answering all Coccinelle questions!). Following fix ups were done manually:
* Removal of address argument from pte_fragment_alloc
* Removal of pte_alloc_one_fast definitions from m68k and microblaze.
// Options: --include-headers --no-includes
// Note: I split the 'identifier fn' line, so if you are manually // running it, please unsplit it so it runs for you.
virtual patch
@pte_alloc_func_def depends on patch exists@
identifier E2;
identifier fn =~ "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$"; type T2;
@@
fn(...
- , T2 E2
)
{ ... }
@pte_alloc_func_proto_noarg depends on patch exists@
type T1, T2, T3, T4;
identifier fn =~ "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$"; @@
(
- T3 fn(T1, T2);
+ T3 fn(T1);
|
- T3 fn(T1, T2, T4);
+ T3 fn(T1, T2);
)
@pte_alloc_func_proto depends on patch exists@
identifier E1, E2, E4;
type T1, T2, T3, T4;
identifier fn =~ "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$"; @@
(
- T3 fn(T1 E1, T2 E2);
+ T3 fn(T1 E1);
|
- T3 fn(T1 E1, T2 E2, T4 E4);
+ T3 fn(T1 E1, T2 E2);
)
@pte_alloc_func_call depends on patch exists@
expression E2;
identifier fn =~ "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$"; @@
fn(...
-, E2
)
@pte_alloc_macro depends on patch exists@
identifier fn =~ "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$"; identifier a, b, c;
expression e;
position p;
@@
(
- #define fn(a, b, c) e
+ #define fn(a, b) e
|
- #define fn(a, b) e
+ #define fn(a) e
)
--- arch/alpha/include/asm/pgalloc.h | 6 +++--- arch/arc/include/asm/pgalloc.h | 5 ++--- arch/arm/include/asm/pgalloc.h | 4 ++-- arch/arm64/include/asm/pgalloc.h | 4 ++-- arch/hexagon/include/asm/pgalloc.h | 6 ++---- arch/ia64/include/asm/pgalloc.h | 5 ++--- arch/m68k/include/asm/mcf_pgalloc.h | 8 ++------ arch/m68k/include/asm/motorola_pgalloc.h | 4 ++-- arch/m68k/include/asm/sun3_pgalloc.h | 6 ++---- arch/microblaze/include/asm/pgalloc.h | 19 ++----------------- arch/microblaze/mm/pgtable.c | 3 +-- arch/mips/include/asm/pgalloc.h | 6 ++---- arch/nds32/include/asm/pgalloc.h | 5 ++--- arch/nios2/include/asm/pgalloc.h | 6 ++---- arch/openrisc/include/asm/pgalloc.h | 5 ++--- arch/openrisc/mm/ioremap.c | 3 +-- arch/parisc/include/asm/pgalloc.h | 4 ++-- arch/powerpc/include/asm/book3s/32/pgalloc.h | 4 ++-- arch/powerpc/include/asm/book3s/64/pgalloc.h | 12 +++++------- arch/powerpc/include/asm/nohash/32/pgalloc.h | 4 ++-- arch/powerpc/include/asm/nohash/64/pgalloc.h | 6 ++---- arch/powerpc/mm/pgtable-book3s64.c | 2 +- arch/powerpc/mm/pgtable_32.c | 4 ++-- arch/riscv/include/asm/pgalloc.h | 6 ++---- arch/s390/include/asm/pgalloc.h | 4 ++-- arch/sh/include/asm/pgalloc.h | 6 ++---- arch/sparc/include/asm/pgalloc_32.h | 5 ++--- arch/sparc/include/asm/pgalloc_64.h | 6 ++---- arch/sparc/mm/init_64.c | 6 ++---- arch/sparc/mm/srmmu.c | 4 ++-- arch/um/include/asm/pgalloc.h | 4 ++-- arch/um/kernel/mem.c | 4 ++-- arch/unicore32/include/asm/pgalloc.h | 4 ++-- arch/x86/include/asm/pgalloc.h | 4 ++-- arch/x86/mm/pgtable.c | 4 ++-- arch/xtensa/include/asm/pgalloc.h | 8 +++----- include/linux/mm.h | 13 ++++++------- mm/huge_memory.c | 8 ++++---- mm/kasan/kasan_init.c | 2 +- mm/memory.c | 17 ++++++++--------- mm/migrate.c | 2 +- mm/mremap.c | 2 +- mm/userfaultfd.c | 2 +- virt/kvm/arm/mmu.c | 2 +- 44 files changed, 97 insertions(+), 147 deletions(-)
arch/alpha/include/asm/pgalloc.h +3 -3
@@ -52,7 +52,7 @@ pmd_free(struct mm_struct *mm, pmd_t *pmd) }static inline pte_t * -pte_alloc_one_kernel (struct mm_struct *mm,unsigned long address) +pte_alloc_one_kernel (struct mm_struct *mm) {pte_t *pte = (pte_t *)__get_free_page (GFP_KERNEL |__GFP_ZERO);return pte;
@@ -65,9 +65,9 @@ pte_free_kernel(struct mm_struct *mm, pte_t *pte) }static inline pgtable_t -pte_alloc_one (struct mm_struct *mm,unsigned long address) +pte_alloc_one (struct mm_struct *mm) { -pte_t *pte =pte_alloc_one_kernel (mm, address); +pte_t *pte =pte_alloc_one_kernel (mm);struct page *page;if (!pte)
arch/arc/include/asm/pgalloc.h +2 -3
@@ -90,8 +90,7 @@ static inline int __get_order_pte(void)return get_order (PTRS_PER_PTE *sizeof (pte_t)); } -static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm, -unsigned long address) +static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {pte_t *pte;
@@ -102,7 +101,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, }static inline pgtable_t -pte_alloc_one (struct mm_struct *mm,unsigned long address) +pte_alloc_one (struct mm_struct *mm) {pgtable_t pte_pg;struct page *page;
arch/arm/include/asm/pgalloc.h +2 -2
@@ -81,7 +81,7 @@ static inline void clean_pte_table(pte_t *pte)* +------------+ */ static inline pte_t * -pte_alloc_one_kernel (struct mm_struct *mm,unsigned long addr) +pte_alloc_one_kernel (struct mm_struct *mm) {pte_t *pte;
@@ -93,7 +93,7 @@ pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) }static inline pgtable_t -pte_alloc_one (struct mm_struct *mm,unsigned long addr) +pte_alloc_one (struct mm_struct *mm) {struct page *pte;
arch/arm64/include/asm/pgalloc.h +2 -2
@@ -91,13 +91,13 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm);extern void pgd_free (struct mm_struct *mm,pgd_t *pgdp);static inline pte_t * -pte_alloc_one_kernel (struct mm_struct *mm,unsigned long addr) +pte_alloc_one_kernel (struct mm_struct *mm) {return (pte_t *)__get_free_page (PGALLOC_GFP ); }static inline pgtable_t -pte_alloc_one (struct mm_struct *mm,unsigned long addr) +pte_alloc_one (struct mm_struct *mm) {struct page *pte;
arch/hexagon/include/asm/pgalloc.h +2 -4
@@ -59,8 +59,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)free_page ((unsigned long ) pgd); } -static inline struct page *pte_alloc_one (struct mm_struct *mm, -unsigned long address) +static inline struct page *pte_alloc_one (struct mm_struct *mm) {struct page *pte;
@@ -75,8 +74,7 @@ static inline struct page *pte_alloc_one(struct mm_struct *mm, }/* _kernel variant gets to use a different allocator */ -static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm, -unsigned long address) +static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {gfp_t flags =GFP_KERNEL | __GFP_ZERO;return (pte_t *)__get_free_page (flags);
arch/ia64/include/asm/pgalloc.h +2 -3
@@ -83,7 +83,7 @@ pmd_populate_kernel(struct mm_struct *mm, pmd_t * pmd_entry, pte_t * pte)pmd_val (*pmd_entry) =__pa (pte); } -static inline pgtable_t pte_alloc_one (struct mm_struct *mm,unsigned long addr) +static inline pgtable_t pte_alloc_one (struct mm_struct *mm) {struct page *page;void *pg;
@@ -99,8 +99,7 @@ static inline pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr)return page; } -static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm, -unsigned long addr) +static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {return quicklist_alloc (0 ,GFP_KERNEL ,NULL ); }
arch/m68k/include/asm/mcf_pgalloc.h +2 -6
@@ -12,8 +12,7 @@ extern inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)extern const char bad_pmd_string[]; -extern inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm, -unsigned long address) +extern inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {unsigned long page =__get_free_page (GFP_DMA );
@@ -32,8 +31,6 @@ extern inline pmd_t *pmd_alloc_kernel(pgd_t *pgd, unsigned long address)#define pmd_alloc_one_fast (mm, address) ({ BUG(); ((pmd_t *)1); })#define pmd_alloc_one (mm, address) ({ BUG(); ((pmd_t *)2); }) -#define pte_alloc_one_fast (mm, addr) pte_alloc_one(mm, addr) -#define pmd_populate (mm, pmd, page) (pmd_val(*pmd) = \ (unsigned long)(page_address(page)))
@@ -50,8 +47,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t page,#define __pmd_free_tlb (tlb, pmd, address) do { } while (0) -static inline struct page *pte_alloc_one (struct mm_struct *mm, -unsigned long address) +static inline struct page *pte_alloc_one (struct mm_struct *mm) {struct page *page =alloc_pages (GFP_DMA ,0 );pte_t *pte;
arch/m68k/include/asm/motorola_pgalloc.h +2 -2
@@ -8,7 +8,7 @@extern pmd_t *get_pointer_table (void );extern int free_pointer_table (pmd_t *); -static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm,unsigned long address) +static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {pte_t *pte;
@@ -28,7 +28,7 @@ static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)free_page ((unsigned long ) pte); } -static inline pgtable_t pte_alloc_one (struct mm_struct *mm,unsigned long address) +static inline pgtable_t pte_alloc_one (struct mm_struct *mm) {struct page *page;pte_t *pte;
arch/m68k/include/asm/sun3_pgalloc.h +2 -4
@@ -35,8 +35,7 @@ do { \tlb_remove_page ((tlb), pte); \ }while (0 ) -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, -unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) {unsigned long page =__get_free_page (GFP_KERNEL );
@@ -47,8 +46,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,return (pte_t *) (page); } -static inline pgtable_t pte_alloc_one (struct mm_struct *mm, -unsigned long address) +static inline pgtable_t pte_alloc_one (struct mm_struct *mm) {struct page *page =alloc_pages (GFP_KERNEL ,0 );
arch/microblaze/include/asm/pgalloc.h +2 -17
@@ -108,10 +108,9 @@ static inline void free_pgd_slow(pgd_t *pgd)#define pmd_alloc_one_fast (mm, address) ({ BUG(); ((pmd_t *)1); })#define pmd_alloc_one (mm, address) ({ BUG(); ((pmd_t *)2); }) -extern pte_t *pte_alloc_one_kernel (struct mm_struct *mm,unsigned long addr); +extern pte_t *pte_alloc_one_kernel (struct mm_struct *mm); -static inline struct page *pte_alloc_one (struct mm_struct *mm, -unsigned long address) +static inline struct page *pte_alloc_one (struct mm_struct *mm) {struct page *ptepage;
@@ -132,20 +131,6 @@ static inline struct page *pte_alloc_one(struct mm_struct *mm,return ptepage; } -static inline pte_t *pte_alloc_one_fast (struct mm_struct *mm, -unsigned long address) -{ -unsigned long *ret; - - ret = pte_quicklist; -if (ret !=NULL ) { - pte_quicklist = (unsigned long *)(*ret); - ret[0 ] =0 ; - pgtable_cache_size--; - } -return (pte_t *)ret; -} -static inline void pte_free_fast (pte_t *pte) { *(unsigned long **)pte = pte_quicklist;
arch/microblaze/mm/pgtable.c +1 -2
@@ -235,8 +235,7 @@ unsigned long iopa(unsigned long addr)return pa; } -__ref pte_t *pte_alloc_one_kernel (struct mm_struct *mm, -unsigned long address) +__ref pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {pte_t *pte;if (mem_init_done) {
arch/mips/include/asm/pgalloc.h +2 -4
@@ -50,14 +50,12 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)free_pages ((unsigned long )pgd,PGD_ORDER ); } -static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm, -unsigned long address) +static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {return (pte_t *)__get_free_pages (GFP_KERNEL | __GFP_ZERO,PTE_ORDER ); } -static inline struct page *pte_alloc_one (struct mm_struct *mm, -unsigned long address) +static inline struct page *pte_alloc_one (struct mm_struct *mm) {struct page *pte;
arch/nds32/include/asm/pgalloc.h +2 -3
@@ -22,8 +22,7 @@ extern void pgd_free(struct mm_struct *mm, pgd_t * pgd);#define check_pgt_cache () do { } while (0) -static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm, -unsigned long addr) +static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {pte_t *pte;
@@ -34,7 +33,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,return pte; } -static inline pgtable_t pte_alloc_one (struct mm_struct *mm,unsigned long addr) +static inline pgtable_t pte_alloc_one (struct mm_struct *mm) {pgtable_t pte;
arch/nios2/include/asm/pgalloc.h +2 -4
@@ -37,8 +37,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)free_pages ((unsigned long )pgd,PGD_ORDER ); } -static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm, -unsigned long address) +static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {pte_t *pte;
@@ -47,8 +46,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,return pte; } -static inline pgtable_t pte_alloc_one (struct mm_struct *mm, -unsigned long address) +static inline pgtable_t pte_alloc_one (struct mm_struct *mm) {struct page *pte;
arch/openrisc/include/asm/pgalloc.h +2 -3
@@ -70,10 +70,9 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)free_page ((unsigned long )pgd); } -extern pte_t *pte_alloc_one_kernel (struct mm_struct *mm,unsigned long address); +extern pte_t *pte_alloc_one_kernel (struct mm_struct *mm); -static inline struct page *pte_alloc_one (struct mm_struct *mm, -unsigned long address) +static inline struct page *pte_alloc_one (struct mm_struct *mm) {struct page *pte; pte =alloc_pages (GFP_KERNEL ,0 );
arch/openrisc/mm/ioremap.c +1 -2
@@ -118,8 +118,7 @@ EXPORT_SYMBOL(iounmap);* the memblock infrastructure. */ -pte_t __ref *pte_alloc_one_kernel (struct mm_struct *mm, -unsigned long address) +pte_t __ref *pte_alloc_one_kernel (struct mm_struct *mm) {pte_t *pte;
arch/parisc/include/asm/pgalloc.h +2 -2
@@ -122,7 +122,7 @@ pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte)#define pmd_pgtable (pmd) pmd_page(pmd)static inline pgtable_t -pte_alloc_one (struct mm_struct *mm,unsigned long address) +pte_alloc_one (struct mm_struct *mm) {struct page *page =alloc_page (GFP_KERNEL |__GFP_ZERO);if (!page)
@@ -135,7 +135,7 @@ pte_alloc_one(struct mm_struct *mm, unsigned long address) }static inline pte_t * -pte_alloc_one_kernel (struct mm_struct *mm,unsigned long addr) +pte_alloc_one_kernel (struct mm_struct *mm) {pte_t *pte = (pte_t *)__get_free_page (GFP_KERNEL |__GFP_ZERO);return pte;
arch/powerpc/include/asm/book3s/32/pgalloc.h +2 -2
@@ -82,8 +82,8 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmdp,#define pmd_pgtable (pmd) pmd_page(pmd)#endif -extern pte_t *pte_alloc_one_kernel (struct mm_struct *mm,unsigned long addr); -extern pgtable_t pte_alloc_one (struct mm_struct *mm,unsigned long addr); +extern pte_t *pte_alloc_one_kernel (struct mm_struct *mm); +extern pgtable_t pte_alloc_one (struct mm_struct *mm);static inline void pte_free_kernel (struct mm_struct *mm,pte_t *pte) {
arch/powerpc/include/asm/book3s/64/pgalloc.h +5 -7
@@ -42,7 +42,7 @@ extern struct kmem_cache *pgtable_cache[]; pgtable_cache[(shift) -1 ]; \ }) -extern pte_t *pte_fragment_alloc (struct mm_struct *,unsigned long ,int ); +extern pte_t *pte_fragment_alloc (struct mm_struct *,int );extern pmd_t *pmd_fragment_alloc (struct mm_struct *,unsigned long );extern void pte_fragment_free (unsigned long *,int );extern void pmd_fragment_free (unsigned long *);
@@ -192,16 +192,14 @@ static inline pgtable_t pmd_pgtable(pmd_t pmd)return (pgtable_t )pmd_page_vaddr (pmd); } -static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm, -unsigned long address) +static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm) { -return (pte_t *)pte_fragment_alloc (mm, address,1 ); +return (pte_t *)pte_fragment_alloc (mm,1 ); } -static inline pgtable_t pte_alloc_one (struct mm_struct *mm, -unsigned long address) +static inline pgtable_t pte_alloc_one (struct mm_struct *mm) { -return (pgtable_t )pte_fragment_alloc (mm, address,0 ); +return (pgtable_t )pte_fragment_alloc (mm,0 ); }static inline void pte_free_kernel (struct mm_struct *mm,pte_t *pte)
arch/powerpc/include/asm/nohash/32/pgalloc.h +2 -2
@@ -83,8 +83,8 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmdp,#define pmd_pgtable (pmd) pmd_page(pmd)#endif -extern pte_t *pte_alloc_one_kernel (struct mm_struct *mm,unsigned long addr); -extern pgtable_t pte_alloc_one (struct mm_struct *mm,unsigned long addr); +extern pte_t *pte_alloc_one_kernel (struct mm_struct *mm); +extern pgtable_t pte_alloc_one (struct mm_struct *mm);static inline void pte_free_kernel (struct mm_struct *mm,pte_t *pte) {
arch/powerpc/include/asm/nohash/64/pgalloc.h +2 -4
@@ -96,14 +96,12 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) } -static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm, -unsigned long address) +static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {return (pte_t *)__get_free_page (GFP_KERNEL | __GFP_ZERO); } -static inline pgtable_t pte_alloc_one (struct mm_struct *mm, -unsigned long address) +static inline pgtable_t pte_alloc_one (struct mm_struct *mm) {struct page *page;pte_t *pte;
arch/powerpc/mm/pgtable-book3s64.c +1 -1
@@ -384,7 +384,7 @@ static pte_t *__alloc_for_ptecache(struct mm_struct *mm, int kernel)return (pte_t *)ret; } -pte_t *pte_fragment_alloc (struct mm_struct *mm,unsigned long vmaddr,int kernel) +pte_t *pte_fragment_alloc (struct mm_struct *mm,int kernel) {pte_t *pte;
arch/powerpc/mm/pgtable_32.c +2 -2
@@ -43,7 +43,7 @@ EXPORT_SYMBOL(ioremap_bot); /* aka VMALLOC_END */extern char etext[], _stext[], _sinittext[], _einittext[]; -__ref pte_t *pte_alloc_one_kernel (struct mm_struct *mm,unsigned long address) +__ref pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {pte_t *pte;
@@ -57,7 +57,7 @@ __ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)return pte; } -pgtable_t pte_alloc_one (struct mm_struct *mm,unsigned long address) +pgtable_t pte_alloc_one (struct mm_struct *mm) {struct page *ptepage;
arch/riscv/include/asm/pgalloc.h +2 -4
@@ -82,15 +82,13 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)#endif /* __PAGETABLE_PMD_FOLDED */ -static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm, -unsigned long address) +static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {return (pte_t *)__get_free_page (GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_ZERO); } -static inline struct page *pte_alloc_one (struct mm_struct *mm, -unsigned long address) +static inline struct page *pte_alloc_one (struct mm_struct *mm) {struct page *pte;
arch/s390/include/asm/pgalloc.h +2 -2
@@ -139,8 +139,8 @@ static inline void pmd_populate(struct mm_struct *mm,/* * page table entry allocation/free routines. */ -#define pte_alloc_one_kernel (mm, vmaddr) ((pte_t *) page_table_alloc(mm)) -#define pte_alloc_one (mm, vmaddr) ((pte_t *) page_table_alloc(mm)) +#define pte_alloc_one_kernel (mm) ((pte_t *)page_table_alloc(mm)) +#define pte_alloc_one (mm) ((pte_t *)page_table_alloc(mm))#define pte_free_kernel (mm, pte) page_table_free(mm, (unsigned long *) pte)#define pte_free (mm, pte) page_table_free(mm, (unsigned long *) pte)
arch/sh/include/asm/pgalloc.h +2 -4
@@ -32,14 +32,12 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd,/* * Allocate and free page tables. */ -static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm, -unsigned long address) +static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {return quicklist_alloc (QUICK_PT ,GFP_KERNEL ,NULL ); } -static inline pgtable_t pte_alloc_one (struct mm_struct *mm, -unsigned long address) +static inline pgtable_t pte_alloc_one (struct mm_struct *mm) {struct page *page;void *pg;
arch/sparc/include/asm/pgalloc_32.h +2 -3
@@ -58,10 +58,9 @@ void pmd_populate(struct mm_struct *mm, pmd_t *pmdp, struct page *ptep);void pmd_set (pmd_t *pmdp,pte_t *ptep);#define pmd_populate_kernel (MM ,PMD ,PTE ) pmd_set(PMD, PTE) -pgtable_t pte_alloc_one (struct mm_struct *mm,unsigned long address); +pgtable_t pte_alloc_one (struct mm_struct *mm); -static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm, -unsigned long address) +static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {return srmmu_get_nocache (PTE_SIZE ,PTE_SIZE ); }
arch/sparc/include/asm/pgalloc_64.h +2 -4
@@ -60,10 +60,8 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)kmem_cache_free (pgtable_cache, pmd); } -pte_t *pte_alloc_one_kernel (struct mm_struct *mm, -unsigned long address); -pgtable_t pte_alloc_one (struct mm_struct *mm, -unsigned long address); +pte_t *pte_alloc_one_kernel (struct mm_struct *mm); +pgtable_t pte_alloc_one (struct mm_struct *mm);void pte_free_kernel (struct mm_struct *mm,pte_t *pte);void pte_free (struct mm_struct *mm,pgtable_t ptepage);
arch/sparc/mm/init_64.c +2 -4
@@ -2925,8 +2925,7 @@ void __flush_tlb_all(void) : : "r" (pstate)); } -pte_t *pte_alloc_one_kernel (struct mm_struct *mm, -unsigned long address) +pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {struct page *page =alloc_page (GFP_KERNEL | __GFP_ZERO);pte_t *pte =NULL ;
@@ -2937,8 +2936,7 @@ pte_t *pte_alloc_one_kernel(struct mm_struct *mm,return pte; } -pgtable_t pte_alloc_one (struct mm_struct *mm, -unsigned long address) +pgtable_t pte_alloc_one (struct mm_struct *mm) {struct page *page =alloc_page (GFP_KERNEL | __GFP_ZERO);if (!page)
arch/sparc/mm/srmmu.c +2 -2
@@ -364,12 +364,12 @@ pgd_t *get_pgd_fast(void)* Alignments up to the page size are the same for physical and virtual * addresses of the nocache area. */ -pgtable_t pte_alloc_one (struct mm_struct *mm,unsigned long address) +pgtable_t pte_alloc_one (struct mm_struct *mm) {unsigned long pte;struct page *page; -if ((pte = (unsigned long )pte_alloc_one_kernel (mm, address)) ==0 ) +if ((pte = (unsigned long )pte_alloc_one_kernel (mm)) ==0 )return NULL ; page =pfn_to_page (__nocache_pa (pte) >>PAGE_SHIFT );if (!pgtable_page_ctor (page)) {
arch/um/include/asm/pgalloc.h +2 -2
@@ -25,8 +25,8 @@extern pgd_t *pgd_alloc (struct mm_struct *);extern void pgd_free (struct mm_struct *mm,pgd_t *pgd); -extern pte_t *pte_alloc_one_kernel (struct mm_struct *,unsigned long ); -extern pgtable_t pte_alloc_one (struct mm_struct *,unsigned long ); +extern pte_t *pte_alloc_one_kernel (struct mm_struct *); +extern pgtable_t pte_alloc_one (struct mm_struct *);static inline void pte_free_kernel (struct mm_struct *mm,pte_t *pte) {
arch/um/kernel/mem.c +2 -2
@@ -199,7 +199,7 @@ void pgd_free(struct mm_struct *mm, pgd_t *pgd)free_page ((unsigned long ) pgd); } -pte_t *pte_alloc_one_kernel (struct mm_struct *mm,unsigned long address) +pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {pte_t *pte;
@@ -207,7 +207,7 @@ pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)return pte; } -pgtable_t pte_alloc_one (struct mm_struct *mm,unsigned long address) +pgtable_t pte_alloc_one (struct mm_struct *mm) {struct page *pte;
arch/unicore32/include/asm/pgalloc.h +2 -2
@@ -34,7 +34,7 @@ extern void free_pgd_slow(struct mm_struct *mm, pgd_t *pgd);* Allocate one PTE table. */ static inline pte_t * -pte_alloc_one_kernel (struct mm_struct *mm,unsigned long addr) +pte_alloc_one_kernel (struct mm_struct *mm) {pte_t *pte;
@@ -46,7 +46,7 @@ pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) }static inline pgtable_t -pte_alloc_one (struct mm_struct *mm,unsigned long addr) +pte_alloc_one (struct mm_struct *mm) {struct page *pte;
arch/x86/include/asm/pgalloc.h +2 -2
@@ -47,8 +47,8 @@ extern gfp_t __userpte_alloc_gfp;extern pgd_t *pgd_alloc (struct mm_struct *);extern void pgd_free (struct mm_struct *mm,pgd_t *pgd); -extern pte_t *pte_alloc_one_kernel (struct mm_struct *,unsigned long ); -extern pgtable_t pte_alloc_one (struct mm_struct *,unsigned long ); +extern pte_t *pte_alloc_one_kernel (struct mm_struct *); +extern pgtable_t pte_alloc_one (struct mm_struct *);/* Should really implement gc for free page table pages. This could be done with a reference count in struct page. */
arch/x86/mm/pgtable.c +2 -2
@@ -23,12 +23,12 @@ EXPORT_SYMBOL(physical_mask);gfp_t __userpte_alloc_gfp =PGALLOC_GFP |PGALLOC_USER_GFP ; -pte_t *pte_alloc_one_kernel (struct mm_struct *mm,unsigned long address) +pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {return (pte_t *)__get_free_page (PGALLOC_GFP & ~__GFP_ACCOUNT); } -pgtable_t pte_alloc_one (struct mm_struct *mm,unsigned long address) +pgtable_t pte_alloc_one (struct mm_struct *mm) {struct page *pte;
arch/xtensa/include/asm/pgalloc.h +3 -5
@@ -38,8 +38,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)free_page ((unsigned long )pgd); } -static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm, -unsigned long address) +static inline pte_t *pte_alloc_one_kernel (struct mm_struct *mm) {pte_t *ptep;int i;
@@ -52,13 +51,12 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,return ptep; } -static inline pgtable_t pte_alloc_one (struct mm_struct *mm, -unsigned long addr) +static inline pgtable_t pte_alloc_one (struct mm_struct *mm) {pte_t *pte;struct page *page; - pte =pte_alloc_one_kernel (mm, addr); + pte =pte_alloc_one_kernel (mm);if (!pte)return NULL ; page =virt_to_page (pte);
include/linux/mm.h +6 -7
@@ -1809,8 +1809,8 @@ static inline void mm_inc_nr_ptes(struct mm_struct *mm) {}static inline void mm_dec_nr_ptes (struct mm_struct *mm) {}#endif -int __pte_alloc (struct mm_struct *mm,pmd_t *pmd,unsigned long address); -int __pte_alloc_kernel (pmd_t *pmd,unsigned long address); +int __pte_alloc (struct mm_struct *mm,pmd_t *pmd); +int __pte_alloc_kernel (pmd_t *pmd);/* * The following ifdef needed to get the 4level-fixup.h header to work.
@@ -1948,18 +1948,17 @@ static inline void pgtable_page_dtor(struct page *page)pte_unmap (pte); \ } while (0) -#define pte_alloc (mm, pmd, address) \ - (unlikely(pmd_none(*(pmd))) && __pte_alloc(mm, pmd, address)) +#define pte_alloc (mm, pmd) (unlikely(pmd_none(*(pmd))) && __pte_alloc(mm, pmd))#define pte_alloc_map (mm, pmd, address) \ - (pte_alloc(mm, pmd, address) ? NULL : pte_offset_map(pmd, address)) + (pte_alloc(mm, pmd) ? NULL : pte_offset_map(pmd, address))#define pte_alloc_map_lock (mm, pmd, address, ptlp) \ - (pte_alloc(mm, pmd, address) ? \ + (pte_alloc(mm, pmd) ? \ NULL : pte_offset_map_lock(mm, pmd, address, ptlp))#define pte_alloc_kernel (pmd, address) \ - ((unlikely(pmd_none(*(pmd))) && __pte_alloc_kernel(pmd, address))? \ + ((unlikely(pmd_none(*(pmd))) && __pte_alloc_kernel(pmd))? \ NULL: pte_offset_kernel(pmd, address))#if USE_SPLIT_PMD_PTLOCKS
mm/huge_memory.c +4 -4
@@ -558,7 +558,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf,return VM_FAULT_FALLBACK ; } - pgtable =pte_alloc_one (vma->vm_mm, haddr); + pgtable =pte_alloc_one (vma->vm_mm);if (unlikely (!pgtable)) { ret =VM_FAULT_OOM ; goto release;
@@ -683,7 +683,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)struct page *zero_page;bool set;vm_fault_t ret; - pgtable =pte_alloc_one (vma->vm_mm, haddr); + pgtable =pte_alloc_one (vma->vm_mm);if (unlikely (!pgtable))return VM_FAULT_OOM ; zero_page =mm_get_huge_zero_page (vma->vm_mm);
@@ -772,7 +772,7 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr,return VM_FAULT_SIGBUS ;if (arch_needs_pgtable_deposit ()) { - pgtable =pte_alloc_one (vma->vm_mm, addr); + pgtable =pte_alloc_one (vma->vm_mm);if (!pgtable)return VM_FAULT_OOM ; }
@@ -908,7 +908,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,if (!vma_is_anonymous (vma))return 0 ; - pgtable =pte_alloc_one (dst_mm, addr); + pgtable =pte_alloc_one (dst_mm);if (unlikely (!pgtable)) goto out;
mm/kasan/kasan_init.c +1 -1
@@ -120,7 +120,7 @@ static int __ref zero_pmd_populate(pud_t *pud, unsigned long addr,pte_t *p;if (slab_is_available ()) - p =pte_alloc_one_kernel (&init_mm, addr); + p =pte_alloc_one_kernel (&init_mm);else p =early_alloc (PAGE_SIZE ,NUMA_NO_NODE );if (!p)
mm/memory.c +8 -9
@@ -400,10 +400,10 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, } } -int __pte_alloc (struct mm_struct *mm,pmd_t *pmd,unsigned long address) +int __pte_alloc (struct mm_struct *mm,pmd_t *pmd) {spinlock_t *ptl; -pgtable_t new =pte_alloc_one (mm, address); +pgtable_t new =pte_alloc_one (mm);if (!new)return -ENOMEM ;
@@ -434,9 +434,9 @@ int __pte_alloc(struct mm_struct *mm, pmd_t *pmd, unsigned long address)return 0 ; } -int __pte_alloc_kernel (pmd_t *pmd,unsigned long address) +int __pte_alloc_kernel (pmd_t *pmd) { -pte_t *new =pte_alloc_one_kernel (&init_mm, address); +pte_t *new =pte_alloc_one_kernel (&init_mm);if (!new)return -ENOMEM ;
@@ -2895,7 +2895,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)* * Here we only have down_read(mmap_sem). */ -if (pte_alloc (vma->vm_mm, vmf->pmd, vmf->address)) +if (pte_alloc (vma->vm_mm, vmf->pmd))return VM_FAULT_OOM ;/* See the comment in pte_alloc_one_map() */
@@ -3042,7 +3042,7 @@ static vm_fault_t pte_alloc_one_map(struct vm_fault *vmf)pmd_populate (vma->vm_mm, vmf->pmd, vmf->prealloc_pte);spin_unlock (vmf->ptl); vmf->prealloc_pte =NULL ; - }else if (unlikely (pte_alloc (vma->vm_mm, vmf->pmd, vmf->address))) { + }else if (unlikely (pte_alloc (vma->vm_mm, vmf->pmd))) {return VM_FAULT_OOM ; } map_pte:
@@ -3121,7 +3121,7 @@ static vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page)* related to pte entry. Use the preallocated table for that. */ if (arch_needs_pgtable_deposit () && !vmf->prealloc_pte) { - vmf->prealloc_pte =pte_alloc_one (vma->vm_mm, vmf->address); + vmf->prealloc_pte =pte_alloc_one (vma->vm_mm);if (!vmf->prealloc_pte)return VM_FAULT_OOM ;smp_wmb ();/* See comment in __pte_alloc() */
@@ -3359,8 +3359,7 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf) start_pgoff + nr_pages - 1);if (pmd_none (*vmf->pmd)) { - vmf->prealloc_pte =pte_alloc_one (vmf->vma->vm_mm, - vmf->address); + vmf->prealloc_pte =pte_alloc_one (vmf->vma->vm_mm);if (!vmf->prealloc_pte) goto out;smp_wmb ();/* See comment in __pte_alloc() */
mm/migrate.c +1 -1
@@ -2599,7 +2599,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,* * Here we only have down_read(mmap_sem). */ -if (pte_alloc (mm, pmdp, addr)) +if (pte_alloc (mm, pmdp)) goto abort;/* See the comment in pte_alloc_one_map() */
mm/mremap.c +1 -1
@@ -238,7 +238,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,if (pmd_trans_unstable (old_pmd))continue ; } -if (pte_alloc (new_vma->vm_mm, new_pmd, new_addr)) +if (pte_alloc (new_vma->vm_mm, new_pmd))break ; next = (new_addr +PMD_SIZE ) &PMD_MASK ;if (extent > next - new_addr)
mm/userfaultfd.c +1 -1
@@ -513,7 +513,7 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm,break ; }if (unlikely (pmd_none (dst_pmdval)) && -unlikely (__pte_alloc (dst_mm, dst_pmd, dst_addr))) { +unlikely (__pte_alloc (dst_mm, dst_pmd))) { err = -ENOMEM ;break ; }
virt/kvm/arm/mmu.c +1 -1
@@ -628,7 +628,7 @@ static int create_hyp_pmd_mappings(pud_t *pud, unsigned long start,BUG_ON (pmd_sect (*pmd));if (pmd_none (*pmd)) { - pte =pte_alloc_one_kernel (NULL , addr); + pte =pte_alloc_one_kernel (NULL );if (!pte) {kvm_err ("Cannot allocate Hyp pte\n" );return -ENOMEM ;
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Re: [PATCH -next v2 1/3] mm: treewide: remove unused address argument from pte_alloc functions
Quoted 1 line
… There is concern that …
Does this wording need a small adjustment?
Quoted 1 line
The changes were obtained by applying the following Coccinelle script.
I would find it nicer if previous patch review comments will trigger further useful effects here.
https://patchwork.kernel.org/patch/10637703/#22265203
https://lore.kernel.org/linuxppc-dev/[email protected]/
https://www.mail-archive.com/[email protected]/msg140009.html
If you have got difficulties with the usage of advanced regular expressions for SmPL constraints, I suggest to use desired function names in SmPL lists or disjunctions instead because of different run time characteristics for such a source code transformation approach.
Quoted 2 lines
// Note: I split the 'identifier fn' line, so if you are manually // running it, please unsplit it so it runs for you.
Please delete this questionable comment.
* The semantic patch language should handle the mentioned code formatting.
* You can use multi-line regular expressions (if it would be desired).
Quoted 3 lines
@pte_alloc_func_def depends on patch exists@
identifier E2;
identifier fn =~
How do you think about to avoid the repetition of a SmPL key word at such places?
Regards,
Markus
_______________________________________________
linux-snps-arc mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
PATCH 2/3 mm: speed up mremap by 20x on large regions (v4)
In reply to Joel FernandesFrom: "Joel Fernandes (Google)" <[email protected]>
Android needs to mremap large regions of memory during memory management related operations. The mremap system call can be really slow if THP is not enabled. The bottleneck is move_page_tables, which is copying each pte at a time, and can be really slow across a large map. Turning on THP may not be a viable option, and is not for us. This patch speeds up the performance for non-THP system by copying at the PMD level when possible.
The speed up is an order of magnitude on x86 (~20x). On a 1GB mremap, the mremap completion times drops from 3.4-3.6 milliseconds to 144-160 microseconds.
Before:
Total mremap time for 1GB data: 3521942 nanoseconds. Total mremap time for 1GB data: 3449229 nanoseconds. Total mremap time for 1GB data: 3488230 nanoseconds.
After:
Total mremap time for 1GB data: 150279 nanoseconds. Total mremap time for 1GB data: 144665 nanoseconds. Total mremap time for 1GB data: 158708 nanoseconds.
Incase THP is enabled, the optimization is mostly skipped except in certain situations.
---
Note that since the bug fix in [1], we now have to flush the TLB every PMD move. The above numbers were obtained on x86 with a flush done every move. For arm64, I previously encountered performance issues doing a flush everytime we move, however Will Deacon says [2] the performance should be better now with recent release. Until we can evaluate arm64, I am dropping the HAVE_MOVE_PMD config enable patch for ARM64 for now. It can be added back once we finish the performance evaluation. Also of note is that the speed up on arm64 with this patch but without the TLB flush every PMD move is around 500x.
[1] https://bugs.chromium.org/p/project-zero/issues/detail?id=1695
[2] https://www.mail-archive.com/[email protected]/msg140837.html
arch/Kconfig | 5 +++++ mm/mremap.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+)
arch/Kconfig +5 -0
@@ -535,6 +535,11 @@ config HAVE_IRQ_TIME_ACCOUNTING Archs need to ensure they use a high enough resolution clock to support irq time accounting and then call enable_sched_clock_irqtime(). +config HAVE_MOVE_PMD + bool + help + Archs that select this are able to move page tables at the PMD level. + config HAVE_ARCH_TRANSPARENT_HUGEPAGE bool
mm/mremap.c +60 -0
73 lines not on this page · raw message
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
kbuild test robot
build botRe: [PATCH -next 2/3] mm: speed up mremap by 20x on large regions (v4)
Thank you for the patch! Yet something to improve:
Hi Joel,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on next-20181102]
url: https://github.com/0day-ci/linux/commits/Joel-Fernandes/Add-support-for-fast-mremap/20181103-224908
config: xtensa-allmodconfig (attached as .config)
compiler: xtensa-linux-gcc (GCC) 8.1.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=8.1.0 make.cross ARCH=xtensa
All errors (new ones prefixed by >>):
mm/mremap.c: In function 'move_normal_pmd':
>> mm/mremap.c:229:2: error: implicit declaration of function 'set_pmd_at'; did you mean 'set_pte_at'? [-Werror=implicit-function-declaration]
set_pmd_at(mm, new_addr, new_pmd, pmd);
^~~~~~~~~~
set_pte_atcc1: some warnings being treated as errors
vim +229 mm/mremap.c
193
194 static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr,
195 unsigned long new_addr, unsigned long old_end,
196 pmd_t *old_pmd, pmd_t *new_pmd)
197 {
198 spinlock_t *old_ptl, *new_ptl;
199 struct mm_struct *mm = vma->vm_mm;
200 pmd_t pmd;
201
202 if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK)
203 || old_end - old_addr < PMD_SIZE)
204 return false;
205
206 /*
207 * The destination pmd shouldn't be established, free_pgtables()
208 * should have release it.
209 */
210 if (WARN_ON(!pmd_none(*new_pmd)))
211 return false;
212
213 /*
214 * We don't have to worry about the ordering of src and dst
215 * ptlocks because exclusive mmap_sem prevents deadlock.
216 */
217 old_ptl = pmd_lock(vma->vm_mm, old_pmd);
218 new_ptl = pmd_lockptr(mm, new_pmd);
219 if (new_ptl != old_ptl)
220 spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING);
221
222 /* Clear the pmd */
223 pmd = *old_pmd;
224 pmd_clear(old_pmd);
225
226 VM_BUG_ON(!pmd_none(*new_pmd));
227
228 /* Set the new pmd */
> 229 set_pmd_at(mm, new_addr, new_pmd, pmd);
230 flush_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
231 if (new_ptl != old_ptl)
232 spin_unlock(new_ptl);
233 spin_unlock(old_ptl);
234
235 return true;
236 }
237
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
_______________________________________________
linux-snps-arc mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
kbuild test robot
build botRe: [PATCH -next 2/3] mm: speed up mremap by 20x on large regions (v4)
In reply to Joel FernandesThank you for the patch! Yet something to improve:
Hi Joel,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on next-20181102]
url: https://github.com/0day-ci/linux/commits/Joel-Fernandes/Add-support-for-fast-mremap/20181103-224908
config: openrisc-or1ksim_defconfig (attached as .config)
compiler: or1k-linux-gcc (GCC) 6.0.0 20160327 (experimental)
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=openrisc
All errors (new ones prefixed by >>):
mm/mremap.c: In function 'move_normal_pmd':
>> mm/mremap.c:229:2: error: implicit declaration of function 'set_pmd_at' [-Werror=implicit-function-declaration]
set_pmd_at(mm, new_addr, new_pmd, pmd);
^~~~~~~~~~cc1: some warnings being treated as errors
vim +/set_pmd_at +229 mm/mremap.c
193
194 static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr,
195 unsigned long new_addr, unsigned long old_end,
196 pmd_t *old_pmd, pmd_t *new_pmd)
197 {
198 spinlock_t *old_ptl, *new_ptl;
199 struct mm_struct *mm = vma->vm_mm;
200 pmd_t pmd;
201
202 if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK)
203 || old_end - old_addr < PMD_SIZE)
204 return false;
205
206 /*
207 * The destination pmd shouldn't be established, free_pgtables()
208 * should have release it.
209 */
210 if (WARN_ON(!pmd_none(*new_pmd)))
211 return false;
212
213 /*
214 * We don't have to worry about the ordering of src and dst
215 * ptlocks because exclusive mmap_sem prevents deadlock.
216 */
217 old_ptl = pmd_lock(vma->vm_mm, old_pmd);
218 new_ptl = pmd_lockptr(mm, new_pmd);
219 if (new_ptl != old_ptl)
220 spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING);
221
222 /* Clear the pmd */
223 pmd = *old_pmd;
224 pmd_clear(old_pmd);
225
226 VM_BUG_ON(!pmd_none(*new_pmd));
227
228 /* Set the new pmd */
> 229 set_pmd_at(mm, new_addr, new_pmd, pmd);
230 flush_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
231 if (new_ptl != old_ptl)
232 spin_unlock(new_ptl);
233 spin_unlock(old_ptl);
234
235 return true;
236 }
237
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH 3/3 mm: select HAVE_MOVE_PMD in x86 for faster mremap
In reply to Joel FernandesFrom: "Joel Fernandes (Google)" <[email protected]>
Moving page-tables at the PMD-level on x86 is known to be safe. Enable this option so that we can do fast mremap when possible.
--- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+)
arch/x86/Kconfig +1 -0
@@ -173,6 +173,7 @@ config X86 select HAVE_MEMBLOCK_NODE_MAP select HAVE_MIXED_BREAKPOINTS_REGS select HAVE_MOD_ARCH_SPECIFIC + select HAVE_MOVE_PMD select HAVE_NMI select HAVE_OPROFILE select HAVE_OPTPROBES
Signature · 1 line
2.19.1.930.g4563a0d9d0-goog
Richard Weinberger
maintainerJoel,
Am Samstag, 3. November 2018, 05:00:38 CET schrieb Joel Fernandes:
Quoted 24 lines
Hi,
Here is the latest "fast mremap" series. This just a repost with Kirill's Acked-bys added. I would like this to be considered for linux -next. I also dropped the CONFIG enablement patch for arm64 since I am yet to test it with the new TLB flushing code that is in very recent kernel releases. (None of my arm64 devices run mainline right now.) so I will post the arm64 enablement once I get to that. The performance numbers in the series are for x86.List of patches in series:
(1) mm: select
HAVE_MOVE_PMDin x86 for faster mremap(2) mm: speed up mremap by 20x on large regions (v4)
v1->v2: Added support for per-arch enablement (Kirill Shutemov)v2->v3: Updated commit message to state the optimization may also run for non-thp type of systems (Daniel Col).v3->v4: Remove uselesspmd_lockcheck (Kirill Shutemov)
Rebased ontop of Linus's master, updated perf results based on x86 testing. Added Kirill's Acks.(3) mm: treewide: remove unused address argument from
pte_allocfunctions (v2)v1->v2: fixarch/um/prototype which was missed in v1 (Anton Ivanov) update changelog with manual fixups for m68k and microblaze.
Quoted 6 lines
not included - (4) mm: select
HAVE_MOVE_PMDin arm64 for faster mremap
This patch is dropped since last posting pending further performance testing on arm64 with new TLB gather updates. See notes in patch titled "mm: speed up mremap by 500x on large regions" for more details.
This breaks UML build:
CC mm/mremap.omm/mremap.c: In function ‘move_normal_pmd’:mm/mremap.c:229:2: error: implicit declaration of function ‘set_pmd_at’; did you mean ‘set_pte_at’? [-Werror=implicit-function-declaration]
set_pmd_at(mm, new_addr, new_pmd, pmd);
^~~~~~~~~~
set_pte_at
CC crypto/rng.o
CC fs/direct-io.o
cc1: some warnings being treated as errors
To test yourself, just run on a x86 box:
$ make defconfig ARCH=um $ make linux ARCH=um
Thanks,
//richard
_______________________________________________
linux-snps-arc mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
Quoted 42 lines · Richard Weinberger
Joel,
Am Samstag, 3. November 2018, 05:00:38 CET schrieb Joel Fernandes:
Quoted 24 lines
Hi,
Here is the latest "fast mremap" series. This just a repost with Kirill's Acked-bys added. I would like this to be considered for linux -next. I also dropped the CONFIG enablement patch for arm64 since I am yet to test it with the new TLB flushing code that is in very recent kernel releases. (None of my arm64 devices run mainline right now.) so I will post the arm64 enablement once I get to that. The performance numbers in the series are for x86.List of patches in series:
(1) mm: select
HAVE_MOVE_PMDin x86 for faster mremap(2) mm: speed up mremap by 20x on large regions (v4)
v1->v2: Added support for per-arch enablement (Kirill Shutemov)v2->v3: Updated commit message to state the optimization may also run for non-thp type of systems (Daniel Col).v3->v4: Remove uselesspmd_lockcheck (Kirill Shutemov)
Rebased ontop of Linus's master, updated perf results based
on x86 testing. Added Kirill's Acks.(3) mm: treewide: remove unused address argument from
pte_allocfunctions (v2)v1->v2: fixarch/um/prototype which was missed in v1 (Anton Ivanov) update changelog with manual fixups for m68k and microblaze.Quoted 6 lines
not included - (4) mm: select
HAVE_MOVE_PMDin arm64 for faster mremap
This patch is dropped since last posting pending further performance testing on arm64 with new TLB gather updates. See notes in patch titled "mm: speed up mremap by 500x on large regions" for more details.This breaks UML build:
CCmm/mremap.omm/mremap.c: In function ‘move_normal_pmd’:mm/mremap.c:229:2: error: implicit declaration of function ‘set_pmd_at’; did you mean ‘set_pte_at’? [-Werror=implicit-function-declaration]set_pmd_at(mm, new_addr, new_pmd, pmd); ^~~~~~~~~~ set_pte_at CC crypto/rng.o CC fs/direct-io.o
Quoted 11 lines
cc1: some warnings being treated as errors
To test yourself, just run on a x86 box:
$ make defconfig ARCH=um $ make linux ARCH=umThanks,
//richard
UM somehow managed to miss one of the 3-level functions, I sent a patch at some point to add to the mmremap series, but it looks like it did not get included in the final version.
You need these two incremental on top of Joel's patch. Richard - feel free to relocate the actual implementation of the set_pgd_at elsewhere - I put it at the end of tlb.c
arch/um/include/asm/pgtable.h +0 -0
@@ -359,4 +359,7 @@ do { \
__flush_tlb_one((vaddr)); \
} while (0)
+extern void set_pmd_at(struct mm_struct *mm, unsigned long addr, + pmd_t *pmdp, pmd_t pmd);
+
#endif
arch/um/kernel/tlb.c +0 -0
@@ -647,3 +647,9 @@ void force_flush_all(void)
vma = vma->vm_next;
}
}
+void set_pmd_at(struct mm_struct *mm, unsigned long addr, + pmd_t *pmdp, pmd_t pmd)
+{
+ *pmdp = pmd;
+}
+
_______________________________________________
linux-snps-arc mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
Quoted 81 lines · Anton Ivanov
Quoted 42 lines · Richard Weinberger
Joel,
Am Samstag, 3. November 2018, 05:00:38 CET schrieb Joel Fernandes:
Quoted 24 lines
Hi,
Here is the latest "fast mremap" series. This just a repost with Kirill's Acked-bys added. I would like this to be considered for linux -next. I also dropped the CONFIG enablement patch for arm64 since I am yet to test it with the new TLB flushing code that is in very recent kernel releases. (None of my arm64 devices run mainline right now.) so I will post the arm64 enablement once I get to that. The performance numbers in the series are for x86.List of patches in series:
(1) mm: select
HAVE_MOVE_PMDin x86 for faster mremap(2) mm: speed up mremap by 20x on large regions (v4)
v1->v2: Added support for per-arch enablement (Kirill Shutemov)v2->v3: Updated commit message to state the optimization may also run for non-thp type of systems (Daniel Col).v3->v4: Remove uselesspmd_lockcheck (Kirill Shutemov)
Rebased ontop of Linus's master, updated perf results based
on x86 testing. Added Kirill's Acks.(3) mm: treewide: remove unused address argument from
pte_allocfunctions (v2)v1->v2: fixarch/um/prototype which was missed in v1 (Anton Ivanov) update changelog with manual fixups for m68k and microblaze.Quoted 6 lines
not included - (4) mm: select
HAVE_MOVE_PMDin arm64 for faster mremap
This patch is dropped since last posting pending further performance testing on arm64 with new TLB gather updates. See notes in patch titled "mm: speed up mremap by 500x on large regions" for more details.This breaks UML build:
CCmm/mremap.omm/mremap.c: In function ‘move_normal_pmd’:mm/mremap.c:229:2: error: implicit declaration of function ‘set_pmd_at’; did you mean ‘set_pte_at’? [-Werror=implicit-function-declaration]set_pmd_at(mm, new_addr, new_pmd, pmd); ^~~~~~~~~~ set_pte_at CC crypto/rng.o CC fs/direct-io.oQuoted 11 lines
cc1: some warnings being treated as errors
To test yourself, just run on a x86 box:
$ make defconfig ARCH=um $ make linux ARCH=umThanks,
//richardUM somehow managed to miss one of the 3-level functions, I sent a patch at some point to add to the mmremap series, but it looks like it did not get included in the final version.
You need these two incremental on top of Joel's patch. Richard - feel free to relocate the actual implementation of the
set_pgd_atelsewhere - I put it at the end oftlb.carch/um/include/asm/pgtable.h +3
-0@@ -359,4 +359,7 @@ do { \ __flush_tlb_one((vaddr)); \ } while (0) +extern void set_pmd_at(struct mm_struct *mm, unsigned long addr, + pmd_t *pmdp, pmd_t pmd); + #endifarch/um/kernel/tlb.c +0
-0
arch/um/kernel/tlb.c void force_flush_all(void) · line 655
Quoted 9 lines
arch/um/kernel/tlb.c +6
-0@@ -647,3 +647,9 @@ void force_flush_all(void) vma = vma->vm_next; } } +void set_pmd_at (struct mm_struct *mm,unsigned long addr, + pmd_t *pmdp, pmd_t pmd) +{ + *pmdp = pmd; +} +
I see it now:
https://www.mail-archive.com/[email protected]/msg140005.html
Sorry about that.
Actually the reason the suggestion got missed is it did not belong in the patch removing the pte_alloc address argument. The pte_alloc parts of the patch you proposed are infact already included.
This set_pmd_at for UM should go into a separate patch and should not be rolled into any existing one. Could you send a proper patch adding this function with a commit message and everything? I can then include it as a separate patch of my series.
thanks!
- Joel
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 81 lines · Anton Ivanov
Quoted 42 lines · Richard Weinberger
Joel,
Am Samstag, 3. November 2018, 05:00:38 CET schrieb Joel Fernandes:
Quoted 24 lines
Hi,
Here is the latest "fast mremap" series. This just a repost with Kirill's Acked-bys added. I would like this to be considered for linux -next. I also dropped the CONFIG enablement patch for arm64 since I am yet to test it with the new TLB flushing code that is in very recent kernel releases. (None of my arm64 devices run mainline right now.) so I will post the arm64 enablement once I get to that. The performance numbers in the series are for x86.List of patches in series:
(1) mm: select
HAVE_MOVE_PMDin x86 for faster mremap(2) mm: speed up mremap by 20x on large regions (v4)
v1->v2: Added support for per-arch enablement (Kirill Shutemov)v2->v3: Updated commit message to state the optimization may also run for non-thp type of systems (Daniel Col).v3->v4: Remove uselesspmd_lockcheck (Kirill Shutemov)
Rebased ontop of Linus's master, updated perf results based
on x86 testing. Added Kirill's Acks.(3) mm: treewide: remove unused address argument from
pte_allocfunctions (v2)v1->v2: fixarch/um/prototype which was missed in v1 (Anton Ivanov) update changelog with manual fixups for m68k and microblaze.Quoted 6 lines
not included - (4) mm: select
HAVE_MOVE_PMDin arm64 for faster mremap
This patch is dropped since last posting pending further performance testing on arm64 with new TLB gather updates. See notes in patch titled "mm: speed up mremap by 500x on large regions" for more details.This breaks UML build:
CCmm/mremap.omm/mremap.c: In function ‘move_normal_pmd’:mm/mremap.c:229:2: error: implicit declaration of function ‘set_pmd_at’; did you mean ‘set_pte_at’? [-Werror=implicit-function-declaration]set_pmd_at(mm, new_addr, new_pmd, pmd); ^~~~~~~~~~ set_pte_at CC crypto/rng.o CC fs/direct-io.oQuoted 11 lines
cc1: some warnings being treated as errors
To test yourself, just run on a x86 box:
$ make defconfig ARCH=um $ make linux ARCH=umThanks,
//richardUM somehow managed to miss one of the 3-level functions, I sent a patch at some point to add to the mmremap series, but it looks like it did not get included in the final version.
You need these two incremental on top of Joel's patch. Richard - feel free to relocate the actual implementation of the
set_pgd_atelsewhere - I put it at the end oftlb.carch/um/include/asm/pgtable.h +3
-0@@ -359,4 +359,7 @@ do { \ __flush_tlb_one((vaddr)); \ } while (0) +extern void set_pmd_at(struct mm_struct *mm, unsigned long addr, + pmd_t *pmdp, pmd_t pmd); + #endifarch/um/kernel/tlb.c +0
-0
arch/um/kernel/tlb.c void force_flush_all(void) · line 655
Quoted 9 lines
arch/um/kernel/tlb.c +6
-0@@ -647,3 +647,9 @@ void force_flush_all(void) vma = vma->vm_next; } } +void set_pmd_at (struct mm_struct *mm,unsigned long addr, + pmd_t *pmdp, pmd_t pmd) +{ + *pmdp = pmd; +} +
Looks like more architectures don't define set_pmd_at. I am thinking the easiest way forward is to just do the following, instead of defining set_pmd_at for every architecture that doesn't care about it. Thoughts?
mm/mremap.c +3 -1
@@ -281,7 +281,8 @@ unsigned long move_page_tables(struct vm_area_struct *vma,split_huge_pmd (vma, old_pmd, old_addr);if (pmd_trans_unstable (old_pmd))continue ; - }else if (extent ==PMD_SIZE &&IS_ENABLED (CONFIG_HAVE_MOVE_PMD )) { + }else if (extent ==PMD_SIZE ) { +#ifdef CONFIG_HAVE_MOVE_PMD /* * If the extent is PMD-sized, try to speed the move by * moving at the PMD level if possible.
@@ -296,6 +297,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,drop_rmap_locks (vma);if (moved)continue ; +#endif }if (pte_alloc (new_vma->vm_mm, new_pmd))
_______________________________________________
linux-snps-arc mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
Quoted 24 lines · Joel Fernandes
Looks like more architectures don't define
set_pmd_at. I am thinking the easiest way forward is to just do the following, instead of definingset_pmd_atfor every architecture that doesn't care about it. Thoughts?mm/mremap.c +2
-1@@ -281,7 +281,8 @@ unsigned long move_page_tables(struct vm_area_struct *vma,split_huge_pmd (vma, old_pmd, old_addr);if (pmd_trans_unstable (old_pmd))continue ; - }else if (extent ==PMD_SIZE &&IS_ENABLED (CONFIG_HAVE_MOVE_PMD )) { + }else if (extent ==PMD_SIZE ) { +#ifdef CONFIG_HAVE_MOVE_PMD /* * If the extent is PMD-sized, try to speed the move by * moving at the PMD level if possible.
mm/mremap.c unsigned long move_page_tables(struct vm_area_struct *vma, · line 303
Quoted 7 lines
mm/mremap.c +1
-0@@ -296,6 +297,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,drop_rmap_locks (vma);if (moved)continue ; +#endif }if (pte_alloc (new_vma->vm_mm, new_pmd))
That seems reasonable as there are going to be a lot of architectures that never have mappings at the PMD level.
Have you thought about what might be needed to extend this paradigm to be able to perform remaps at the PUD level, given many architectures already support PUD-mapped pages?
William Kucharski
Quoted 29 lines · William Kucharski
Quoted 29 lines · Joel Fernandes
Looks like more architectures don't define
set_pmd_at. I am thinking the easiest way forward is to just do the following, instead of definingset_pmd_atfor every architecture that doesn't care about it. Thoughts?mm/mremap.c +3
-1@@ -281,7 +281,8 @@ unsigned long move_page_tables(struct vm_area_struct *vma,split_huge_pmd (vma, old_pmd, old_addr);if (pmd_trans_unstable (old_pmd))continue ; - }else if (extent ==PMD_SIZE &&IS_ENABLED (CONFIG_HAVE_MOVE_PMD )) { + }else if (extent ==PMD_SIZE ) { +#ifdef CONFIG_HAVE_MOVE_PMD /* * If the extent is PMD-sized, try to speed the move by * moving at the PMD level if possible. @@ -296,6 +297,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,drop_rmap_locks (vma);if (moved)continue ; +#endif }
Quoted 5 lines
mm/mremap.c unsigned long move_page_tables(struct vm_area_struct *vma, · line 303
Quoted 2 lines
mm/mremap.c +0
-0if (pte_alloc (new_vma->vm_mm, new_pmd))That seems reasonable as there are going to be a lot of architectures that never have mappings at the PMD level.
Ok, I will do it like this and resend.
Quoted 4 lines
Have you thought about what might be needed to extend this paradigm to be able to perform remaps at the PUD level, given many architectures already support PUD-mapped pages?
I have thought about this. I believe it is doable in the future. Off the top I don't see an issue doing it, and it will also reduce the number of flushes.
thanks,
- Joel