ARM: mm: HugeTLB + THP support.
cover ARM: mm: HugeTLB + THP support.
Hello,
The following patches bring both HugeTLB support and Transparent HugePage (THP) support to ARM.
These are not intended for 3.9.
Both short descriptors (non-LPAE) and long descriptors (LPAE) are supported.
The non-LPAE HugeTLB code is based on patches by Bill Carson [1], but instead of allocating extra memory to store "Linux PTEs", it re-purposes the domain bits of section descriptors and constructs huge Linux PTEs on demand.
As PMDs are walked directly by the kernel THP functions (there are no huge_pmd_offset style functions), any "linux PMD"/"hardware PMD" distinction would require some re-working of the ARM PMD/PTE code. Use of the domain bits allows for a more straightforward THP implementation.
Some general HugeTLB code relating to huge page migration on memory failure (CONFIG_MEMORY_FAILURE) de-references huge pte_t *s directly rather than use the huge_ptep_get and set_huge_pte_at functions. Thus this config option is incompatible with non-LPAE hugepages. At the moment I can only see x86 using CONFIG_MEMORY_FAILURE though.
Non-LPAE code was tested on an Arndale board (Exynos 5250) and a Tegra 2 TrimSlice.
The LPAE code manipulates the hardware page tables directly as the long descriptors are wide enough to contain all the Linux PTE information.
The LPAE code has been tested on an Arndale board (Exynos 5250).
This patch set is based on 3.8-rc6.
Major changes since the RFC:
* huge pmd sharing removed from the 3-level code as this was found to be very rarely, if ever?, used. This allowed for some code simplification.
* hardware pmd bits for 2-levels of paging are now taken from mmu.c. Also the mapping code now uses pte/pmd bit helper functions rather than the custom pre-processor logic.
Cheers,
Signature · 41 lines
Steve [1] - http://lists.infradead.org/pipermail/linux-arm-kernel/2012-February/084359.html Catalin Marinas (2): ARM: mm: HugeTLB support for LPAE systems. ARM: mm: Transparent huge page support for LPAE systems. Steve Capper (4): ARM: mm: correct pte_same behaviour for LPAE. ARM: mm: Add support for flushing HugeTLB pages. ARM: mm: HugeTLB support for non-LPAE systems. ARM: mm: Transparent huge page support for non-LPAE systems. arch/arm/Kconfig | 8 ++ arch/arm/include/asm/hugetlb-2level.h | 121 +++++++++++++++++++++ arch/arm/include/asm/hugetlb-3level.h | 61 +++++++++++ arch/arm/include/asm/hugetlb.h | 87 +++++++++++++++ arch/arm/include/asm/pgtable-2level.h | 151 +++++++++++++++++++++++++++ arch/arm/include/asm/pgtable-3level-hwdef.h | 2 + arch/arm/include/asm/pgtable-3level.h | 83 +++++++++++++++ arch/arm/include/asm/pgtable.h | 13 ++- arch/arm/include/asm/tlb.h | 16 ++- arch/arm/include/asm/tlbflush.h | 2 + arch/arm/kernel/head.S | 10 +- arch/arm/mm/Makefile | 1 + arch/arm/mm/dma-mapping.c | 2 +- arch/arm/mm/fault.c | 11 -- arch/arm/mm/flush.c | 26 +++-- arch/arm/mm/fsr-2level.c | 4 +- arch/arm/mm/fsr-3level.c | 4 +- arch/arm/mm/hugetlbpage.c | 100 ++++++++++++++++++ arch/arm/mm/mmu.c | 27 +++++ 19 files changed, 697 insertions(+), 32 deletions(-) create mode 100644 arch/arm/include/asm/hugetlb-2level.h create mode 100644 arch/arm/include/asm/hugetlb-3level.h create mode 100644 arch/arm/include/asm/hugetlb.h create mode 100644 arch/arm/mm/hugetlbpage.c -- 1.7.9.5
PATCH 1/6 ARM: mm: correct pte_same behaviour for LPAE.
For 3 levels of paging the PTE_EXT_NG bit will be set for user address ptes that are written to a page table but not for ptes created with mk_pte.
This can cause some comparison tests made by pte_same to fail spuriously and lead to other problems.
To correct this behaviour, we mask off PTE_EXT_NG for any pte that is present before running the comparison.
--- arch/arm/include/asm/pgtable-3level.h | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
arch/arm/include/asm/pgtable-3level.h +17 -0
@@ -148,6 +148,23 @@ static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr)clean_pmd_entry (pmdp); \ } while (0) +/* +* For 3 levels of paging the PTE_EXT_NG bit will be set for user address ptes +* that are written to a page table but not for ptes created with mk_pte. +* +* In hugetlb_no_page, a new huge pte (new_pte) is generated and passed to +* hugetlb_cow, where it is compared with an entry in a page table. +* This comparison test fails erroneously leading ultimately to a memory leak. +* +* To correct this behaviour, we mask off PTE_EXT_NG for any pte that is +* present before running the comparison. + */ +#define __HAVE_ARCH_PTE_SAME +#define pte_same (pte_a,pte_b) ((pte_present(pte_a) ? pte_val(pte_a) & ~PTE_EXT_NG \ + : pte_val(pte_a)) \ + == (pte_present(pte_b) ? pte_val(pte_b) & ~PTE_EXT_NG \ + : pte_val(pte_b))) +#define set_pte_ext (ptep,pte,ext) cpu_set_pte_ext(ptep,__pte(pte_val(pte)|(ext)))#endif /* __ASSEMBLY__ */
Signature · 1 line
1.7.9.5
PATCH 2/6 ARM: mm: Add support for flushing HugeTLB pages.
In reply to Steve CapperOn ARM we use the __flush_dcache_page function to flush the dcache of pages when needed; usually when the PG_dcache_clean bit is unset and we are setting a PTE.
A HugeTLB page is represented as a compound page consisting of an array of pages. Thus to flush the dcache of a HugeTLB page, one must flush more than a single page.
This patch modifies __flush_dcache_page such that all constituent pages of a HugeTLB page are flushed.
--- arch/arm/mm/flush.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-)
arch/arm/mm/flush.c +16 -10
@@ -17,6 +17,7 @@#include <asm/highmem.h> #include <asm/smp_plat.h> #include <asm/tlbflush.h> +#include <linux/hugetlb.h> #include "mm.h"
@@ -168,17 +169,22 @@ void __flush_dcache_page(struct address_space *mapping, struct page *page)* coherent with the kernels mapping. */ if (!PageHighMem (page)) { -__cpuc_flush_dcache_area (page_address (page),PAGE_SIZE ); +size_t page_size =PAGE_SIZE <<compound_order (page); +__cpuc_flush_dcache_area (page_address (page), page_size); }else { -void *addr =kmap_high_get (page); -if (addr) { -__cpuc_flush_dcache_area (addr,PAGE_SIZE ); -kunmap_high (page); - }else if (cache_is_vipt ()) { -/* unmapped pages might still be cached */ - addr =kmap_atomic (page); -__cpuc_flush_dcache_area (addr,PAGE_SIZE ); -kunmap_atomic (addr); +unsigned long i; +for (i =0 ; i < (1 <<compound_order (page)); i++) { +struct page *cpage = page + i; +void *addr =kmap_high_get (cpage); +if (addr) { +__cpuc_flush_dcache_area (addr,PAGE_SIZE ); +kunmap_high (cpage); + }else if (cache_is_vipt ()) { +/* unmapped pages might still be cached */ + addr =kmap_atomic (cpage); +__cpuc_flush_dcache_area (addr,PAGE_SIZE ); +kunmap_atomic (addr); + } } }
Signature · 1 line
1.7.9.5
PATCH 3/6 ARM: mm: HugeTLB support for LPAE systems.
In reply to Steve CapperFrom: Catalin Marinas <[email protected]>
This patch adds support for hugetlbfs based on the x86 implementation. It allows mapping of 2MB sections (see Documentation/vm/hugetlbpage.txt for usage). The 64K pages configuration is not supported (section size is 512MB in this case).
[steve.capper at arm.com: symbolic constants replace numbers in places. Split up into multiple files, to simplify future non-LPAE support, removed huge_pmd_share code, as this is very rarely executed].
--- arch/arm/Kconfig | 4 ++ arch/arm/include/asm/hugetlb-3level.h | 61 ++++++++++++++++++++ arch/arm/include/asm/hugetlb.h | 83 +++++++++++++++++++++++++++ arch/arm/include/asm/pgtable-3level.h | 13 +++++ arch/arm/mm/Makefile | 1 + arch/arm/mm/dma-mapping.c | 2 +- arch/arm/mm/fsr-3level.c | 2 +- arch/arm/mm/hugetlbpage.c | 100 +++++++++++++++++++++++++++++++++ 8 files changed, 264 insertions(+), 2 deletions(-) create mode 100644 arch/arm/include/asm/hugetlb-3level.h create mode 100644 arch/arm/include/asm/hugetlb.h create mode 100644 arch/arm/mm/hugetlbpage.c
arch/arm/Kconfig +4 -0
@@ -1778,6 +1778,10 @@ config HW_PERF_EVENTS Enable hardware performance counter support for perf events. If disabled, perf events will use software events only. +config SYS_SUPPORTS_HUGETLBFS + def_bool y + depends on ARM_LPAE + source "mm/Kconfig" config FORCE_MAX_ZONEORDER
arch/arm/include/asm/hugetlb-3level.h new file +61 -0
61 lines not on this page · raw message
arch/arm/include/asm/hugetlb.h new file +83 -0
83 lines not on this page · raw message
arch/arm/include/asm/pgtable-3level.h +13 -0
@@ -62,6 +62,14 @@#define USER_PTRS_PER_PGD (PAGE_OFFSET / PGDIR_SIZE)/* +* Hugetlb definitions. +*/ +#define HPAGE_SHIFT PMD_SHIFT +#define HPAGE_SIZE (_AC(1, UL) << HPAGE_SHIFT) +#define HPAGE_MASK (~(HPAGE_SIZE - 1)) +#define HUGETLB_PAGE_ORDER (HPAGE_SHIFT - PAGE_SHIFT) + +/* * "Linux" PTE definitions for LPAE. * * These bits overlap with the hardware bits but the naming is preserved for
@@ -167,6 +175,11 @@ static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr)#define set_pte_ext (ptep,pte,ext) cpu_set_pte_ext(ptep,__pte(pte_val(pte)|(ext))) +#define pte_huge (pte) ((pte_val(pte) & PMD_TYPE_MASK) == PMD_TYPE_SECT) + +#define pte_mkhuge (pte) (__pte((pte_val(pte) & ~PMD_TYPE_MASK) | PMD_TYPE_SECT)) + +#endif /* __ASSEMBLY__ */ #endif /* _ASM_PGTABLE_3LEVEL_H */
arch/arm/mm/Makefile +1 -0
@@ -16,6 +16,7 @@ obj-$(CONFIG_MODULES) += proc-syms.o obj-$(CONFIG_ALIGNMENT_TRAP) += alignment.o obj-$(CONFIG_HIGHMEM) += highmem.o +obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_CPU_ABRT_NOMMU) += abort-nommu.o obj-$(CONFIG_CPU_ABRT_EV4) += abort-ev4.o
arch/arm/mm/dma-mapping.c +1 -1
@@ -239,7 +239,7 @@ static void __dma_free_buffer(struct page *page, size_t size)#ifdef CONFIG_MMU #ifdef CONFIG_HUGETLB_PAGE -#error ARM Coherent DMA allocator does not (yet) support huge TLB +#warning ARM Coherent DMA allocator does not (yet) support huge TLB#endif static void *__alloc_from_contiguous(struct device *dev, size_t size,
arch/arm/mm/fsr-3level.c +1 -1
@@ -13,7 +13,7 @@ static struct fsr_info fsr_info[] = { { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 3 access flag fault" }, { do_bad, SIGBUS, 0, "reserved permission fault" }, { do_bad, SIGSEGV, SEGV_ACCERR, "level 1 permission fault" }, - { do_sect_fault, SIGSEGV, SEGV_ACCERR, "level 2 permission fault" }, + { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 2 permission fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 3 permission fault" }, { do_bad, SIGBUS, 0, "synchronous external abort" }, { do_bad, SIGBUS, 0, "asynchronous external abort" },
arch/arm/mm/hugetlbpage.c new file +100 -0
100 lines not on this page · raw message
Signature · 1 line
1.7.9.5
PATCH 4/6 ARM: mm: HugeTLB support for non-LPAE systems.
In reply to Steve CapperBased on Bill Carson's HugeTLB patch, with the big difference being in the way PTEs are passed back to the memory manager. Rather than store a "Linux Huge PTE" separately; we make one up on the fly in huge_ptep_get. Also rather than consider 16M supersections, we focus solely on 2x1M sections.
To construct a huge PTE on the fly we need additional information (such as the accessed flag and dirty bit) which we choose to store in the domain bits of the short section descriptor. In order to use these domain bits for storage, we need to make ourselves a client for all 16 domains and this is done in head.S.
Storing extra information in the domain bits also makes it a lot easier to implement Transparent Huge Pages, and some of the code in pgtable-2level.h is arranged to facilitate THP support in a later patch.
Non-LPAE HugeTLB pages are incompatible with the huge page migration code (enabled when CONFIG_MEMORY_FAILURE is selected) as that code dereferences PTEs directly, rather than calling huge_ptep_get and set_huge_pte_at.
--- arch/arm/Kconfig | 2 +- arch/arm/include/asm/hugetlb-2level.h | 121 +++++++++++++++++++++++++++++++++ arch/arm/include/asm/hugetlb.h | 4 ++ arch/arm/include/asm/pgtable-2level.h | 102 +++++++++++++++++++++++++++ arch/arm/include/asm/pgtable.h | 2 + arch/arm/include/asm/tlb.h | 10 ++- arch/arm/kernel/head.S | 10 ++- arch/arm/mm/fault.c | 11 --- arch/arm/mm/fsr-2level.c | 4 +- arch/arm/mm/mmu.c | 27 ++++++++ 10 files changed, 276 insertions(+), 17 deletions(-) create mode 100644 arch/arm/include/asm/hugetlb-2level.h
arch/arm/Kconfig +1 -1
@@ -1780,7 +1780,7 @@ config HW_PERF_EVENTS config SYS_SUPPORTS_HUGETLBFS def_bool y - depends on ARM_LPAE + depends on ARM_LPAE || (!CPU_USE_DOMAINS && !MEMORY_FAILURE) source "mm/Kconfig"
arch/arm/include/asm/hugetlb-2level.h new file +121 -0
121 lines not on this page · raw message
arch/arm/include/asm/hugetlb.h +4 -0
@@ -24,7 +24,11 @@#include <asm/page.h> +#ifdef CONFIG_ARM_LPAE #include <asm/hugetlb-3level.h> +#else +#include <asm/hugetlb-2level.h> +#endif static inline void hugetlb_free_pgd_range(struct mmu_gather *tlb, unsigned long addr, unsigned long end,
arch/arm/include/asm/pgtable-2level.h +102 -0
108 lines not on this page · raw message
arch/arm/include/asm/pgtable.h +2 -0
@@ -235,6 +235,8 @@ PTE_BIT_FUNC(mkclean, &= ~L_PTE_DIRTY); PTE_BIT_FUNC(mkdirty, |= L_PTE_DIRTY); PTE_BIT_FUNC(mkold, &= ~L_PTE_YOUNG); PTE_BIT_FUNC(mkyoung, |= L_PTE_YOUNG); +PTE_BIT_FUNC(mkexec, &= ~L_PTE_XN); +PTE_BIT_FUNC(mknexec, |= L_PTE_XN);static inline pte_t pte_mkspecial (pte_t pte) {return pte; }
arch/arm/include/asm/tlb.h +8 -2
@@ -92,10 +92,16 @@ static inline void tlb_flush(struct mmu_gather *tlb)static inline void tlb_add_flush (struct mmu_gather *tlb,unsigned long addr) {if (!tlb->fullmm) { +unsigned long size =PAGE_SIZE ; +if (addr < tlb->range_start) tlb->range_start = addr; -if (addr +PAGE_SIZE > tlb->range_end) - tlb->range_end = addr +PAGE_SIZE ; + +if (tlb->vma &&is_vm_hugetlb_page (tlb->vma)) + size =HPAGE_SIZE ; + +if (addr + size > tlb->range_end) + tlb->range_end = addr + size; } }
arch/arm/kernel/head.S +9 -1
@@ -413,13 +413,21 @@ __enable_mmu: mov r5, #0 mcrr p15, 0, r4, r5, c2 @ load TTBR0 #else +#ifndef CONFIG_SYS_SUPPORTS_HUGETLBFS mov r5, #(domain_val(DOMAIN_USER, DOMAIN_MANAGER) | \ domain_val(DOMAIN_KERNEL, DOMAIN_MANAGER) | \ domain_val(DOMAIN_TABLE, DOMAIN_MANAGER) | \ domain_val(DOMAIN_IO, DOMAIN_CLIENT)) +#else + @ set ourselves as the client in all domains + @ this allows us to then use the 4 domain bits in the + @ section descriptors in our transparent huge pages + ldr r5, =0x55555555 +#endif /* CONFIG_SYS_SUPPORTS_HUGETLBFS */ + mcr p15, 0, r5, c3, c0, 0 @ load domain access register mcr p15, 0, r4, c2, c0, 0 @ load page table pointer -#endif +#endif /* CONFIG_ARM_LPAE */ b __turn_mmu_on ENDPROC(__enable_mmu)
arch/arm/mm/fault.c +0 -11
@@ -488,17 +488,6 @@ do_translation_fault(unsigned long addr, unsigned int fsr,#endif /* CONFIG_MMU */ /* -* Some section permission faults need to be handled gracefully. -* They can happen due to a __{get,put}_user during an oops. -*/ -static int -do_sect_fault (unsigned long addr,unsigned int fsr,struct pt_regs *regs) -{ -do_bad_area (addr, fsr, regs); -return 0 ; -} - -/* * This abort handler always returns "fault". */ static int
arch/arm/mm/fsr-2level.c +2 -2
@@ -16,7 +16,7 @@ static struct fsr_info fsr_info[] = { { do_bad, SIGBUS, 0, "external abort on non-linefetch" }, { do_bad, SIGSEGV, SEGV_ACCERR, "page domain fault" }, { do_bad, SIGBUS, 0, "external abort on translation" }, - { do_sect_fault, SIGSEGV, SEGV_ACCERR, "section permission fault" }, + { do_page_fault, SIGSEGV, SEGV_ACCERR, "section permission fault" }, { do_bad, SIGBUS, 0, "external abort on translation" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "page permission fault" },/*
@@ -56,7 +56,7 @@ static struct fsr_info ifsr_info[] = { { do_bad, SIGBUS, 0, "unknown 10" }, { do_bad, SIGSEGV, SEGV_ACCERR, "page domain fault" }, { do_bad, SIGBUS, 0, "external abort on translation" }, - { do_sect_fault, SIGSEGV, SEGV_ACCERR, "section permission fault" }, + { do_page_fault, SIGSEGV, SEGV_ACCERR, "section permission fault" }, { do_bad, SIGBUS, 0, "external abort on translation" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "page permission fault" }, { do_bad, SIGBUS, 0, "unknown 16" },
arch/arm/mm/mmu.c +27 -0
@@ -303,6 +303,21 @@ const struct mem_type *get_mem_type(unsigned int type)EXPORT_SYMBOL (get_mem_type);/* +* If the system supports huge pages and we are running with short descriptors, +* then compute the pmd and linux pte prot values for a huge page. +* +* These values are used by both the HugeTLB and THP code. +*/ +#if defined(CONFIG_SYS_SUPPORTS_HUGETLBFS ) && !defined(CONFIG_ARM_LPAE ) +pmdval_t arm_hugepmdprotval; +EXPORT_SYMBOL (arm_hugepmdprotval); + +pteval_t arm_hugepteprotval; +EXPORT_SYMBOL (arm_hugepteprotval); +#endif + + +/* * Adjust the PMD section entries according to the CPU in use. */ static void __init build_mem_type_table (void )
@@ -526,6 +541,18 @@ static void __init build_mem_type_table(void)if (t->prot_sect) t->prot_sect |=PMD_DOMAIN (t->domain); } + +#if defined(CONFIG_SYS_SUPPORTS_HUGETLBFS ) && !defined(CONFIG_ARM_LPAE ) +/* +* we assume all huge pages are user pages and that hardware access +* flag updates are disabled (i.e. SCTLR.AFE == 0b). +*/ + arm_hugepteprotval = mem_types[MT_MEMORY ].prot_pte |L_PTE_USER |L_PTE_VALID ; + + arm_hugepmdprotval = mem_types[MT_MEMORY ].prot_sect |PMD_SECT_AP_READ + | PMD_SECT_nG; +#endif + }#ifdef CONFIG_ARM_DMA_MEM_BUFFERABLE
Signature · 1 line
1.7.9.5
PATCH 5/6 ARM: mm: Transparent huge page support for LPAE systems.
In reply to Steve CapperFrom: Catalin Marinas <[email protected]>
The patch adds support for THP (transparent huge pages) to LPAE systems. When this feature is enabled, the kernel tries to map anonymous pages as 2MB sections where possible.
[steve.capper at arm.com: symbolic constants used, value of PMD_SECT_SPLITTING adjusted, tlbflush.h included in pgtable.h]
--- arch/arm/Kconfig | 4 +++ arch/arm/include/asm/pgtable-2level.h | 2 ++ arch/arm/include/asm/pgtable-3level-hwdef.h | 2 ++ arch/arm/include/asm/pgtable-3level.h | 51 +++++++++++++++++++++++++++ arch/arm/include/asm/pgtable.h | 4 ++- arch/arm/include/asm/tlb.h | 6 ++++ arch/arm/include/asm/tlbflush.h | 2 ++ arch/arm/mm/fsr-3level.c | 2 +- 8 files changed, 71 insertions(+), 2 deletions(-)
arch/arm/Kconfig +4 -0
@@ -1782,6 +1782,10 @@ config SYS_SUPPORTS_HUGETLBFS def_bool y depends on ARM_LPAE || (!CPU_USE_DOMAINS && !MEMORY_FAILURE) +config HAVE_ARCH_TRANSPARENT_HUGEPAGE + def_bool y + depends on ARM_LPAE + source "mm/Kconfig" config FORCE_MAX_ZONEORDER
arch/arm/include/asm/pgtable-2level.h +2 -0
@@ -179,6 +179,8 @@ static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr)/* we don't need complex calculations here as the pmd is folded into the pgd */ #define pmd_addr_end (addr,end) (end) +#define pmd_present (pmd) ((pmd_val(pmd) & PMD_TYPE_MASK) != PMD_TYPE_FAULT) +#define set_pte_ext (ptep,pte,ext) cpu_set_pte_ext(ptep,pte,ext)
arch/arm/include/asm/pgtable-3level-hwdef.h +2 -0
@@ -38,6 +38,8 @@*/ #define PMD_SECT_BUFFERABLE (_AT(pmdval_t, 1) << 2)#define PMD_SECT_CACHEABLE (_AT(pmdval_t, 1) << 3) +#define PMD_SECT_USER (_AT(pmdval_t, 1) << 6)/* AP[1] */ +#define PMD_SECT_RDONLY (_AT(pmdval_t, 1) << 7)/* AP[2] */ #define PMD_SECT_S (_AT(pmdval_t, 3) << 8)#define PMD_SECT_AF (_AT(pmdval_t, 1) << 10)#define PMD_SECT_nG (_AT(pmdval_t, 1) << 11)
arch/arm/include/asm/pgtable-3level.h +51 -0
63 lines not on this page · raw message
arch/arm/include/asm/pgtable.h +3 -1
@@ -24,6 +24,9 @@#include <asm/memory.h> #include <asm/pgtable-hwdef.h> + +#include <asm/tlbflush.h> +#ifdef CONFIG_ARM_LPAE #include <asm/pgtable-3level.h> #else
@@ -163,7 +166,6 @@ extern pgd_t swapper_pg_dir[PTRS_PER_PGD];#define pgd_offset_k (addr) pgd_offset(&init_mm, addr)#define pmd_none (pmd) (!pmd_val(pmd)) -#define pmd_present (pmd) (pmd_val(pmd))static inline pte_t *pmd_page_vaddr (pmd_t pmd) {
arch/arm/include/asm/tlb.h +6 -0
@@ -229,6 +229,12 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp,#endif } +static inline void +tlb_remove_pmd_tlb_entry (struct mmu_gather *tlb,pmd_t *pmdp,unsigned long addr) +{ +tlb_add_flush (tlb, addr); +} +#define pte_free_tlb (tlb, ptep, addr) __pte_free_tlb(tlb, ptep, addr)#define pmd_free_tlb (tlb, pmdp, addr) __pmd_free_tlb(tlb, pmdp, addr)#define pud_free_tlb (tlb, pudp, addr) pud_free((tlb)->mm, pudp)
arch/arm/include/asm/tlbflush.h +2 -0
@@ -505,6 +505,8 @@ static inline void update_mmu_cache(struct vm_area_struct *vma, }#endif +#define update_mmu_cache_pmd (vma, address, pmd) do { } while (0) +#endif #endif /* CONFIG_MMU */
arch/arm/mm/fsr-3level.c +1 -1
@@ -9,7 +9,7 @@ static struct fsr_info fsr_info[] = { { do_page_fault, SIGSEGV, SEGV_MAPERR, "level 3 translation fault" }, { do_bad, SIGBUS, 0, "reserved access flag fault" }, { do_bad, SIGSEGV, SEGV_ACCERR, "level 1 access flag fault" }, - { do_bad, SIGSEGV, SEGV_ACCERR, "level 2 access flag fault" }, + { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 2 access flag fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 3 access flag fault" }, { do_bad, SIGBUS, 0, "reserved permission fault" }, { do_bad, SIGSEGV, SEGV_ACCERR, "level 1 permission fault" },
Signature · 1 line
1.7.9.5
PATCH 6/6 ARM: mm: Transparent huge page support for non-LPAE systems.
In reply to Steve CapperMuch of the required code for THP has been implemented in the earlier non-LPAE HugeTLB patch.
One more domain bit is used (to store whether or not the THP is splitting).
Some THP helper functions are defined; and we have to re-define pmd_page such that it distinguishes between page tables and sections.
--- arch/arm/Kconfig | 2 +- arch/arm/include/asm/pgtable-2level.h | 47 +++++++++++++++++++++++++++++++++ arch/arm/include/asm/pgtable-3level.h | 2 ++ arch/arm/include/asm/pgtable.h | 7 +++-- 4 files changed, 55 insertions(+), 3 deletions(-)
arch/arm/Kconfig +1 -1
@@ -1784,7 +1784,7 @@ config SYS_SUPPORTS_HUGETLBFS config HAVE_ARCH_TRANSPARENT_HUGEPAGE def_bool y - depends on ARM_LPAE + depends on SYS_SUPPORTS_HUGETLBFS source "mm/Kconfig"
arch/arm/include/asm/pgtable-2level.h +47 -0
59 lines not on this page · raw message
arch/arm/include/asm/pgtable-3level.h +2 -0
@@ -211,6 +211,8 @@ PMD_BIT_FUNC(mknotpresent, &= ~PMD_TYPE_MASK);#define pfn_pmd (pfn,prot) (__pmd(((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)))#define mk_pmd (page,prot) pfn_pmd(page_to_pfn(page),prot) +#define pmd_page (pmd) pfn_to_page(__phys_to_pfn(pmd_val(pmd) & PHYS_MASK)) +static inline pmd_t pmd_modify (pmd_t pmd,pgprot_t newprot) {const pmdval_t mask =PMD_SECT_USER |PMD_SECT_XN |PMD_SECT_RDONLY ;
arch/arm/include/asm/pgtable.h +5 -2
@@ -169,11 +169,14 @@ extern pgd_t swapper_pg_dir[PTRS_PER_PGD];static inline pte_t *pmd_page_vaddr (pmd_t pmd) { +#ifdef SYS_SUPPORTS_HUGETLBFS +if ((pmd_val (pmd) &PMD_TYPE_MASK ) ==PMD_TYPE_SECT ) +return __va (pmd_val (pmd) &HPAGE_MASK ); +#endif +return __va (pmd_val (pmd) &PHYS_MASK & (s32 )PAGE_MASK ); } -#define pmd_page (pmd) pfn_to_page(__phys_to_pfn(pmd_val(pmd) & PHYS_MASK)) -#ifndef CONFIG_HIGHPTE #define __pte_map (pmd) pmd_page_vaddr(*(pmd))#define __pte_unmap (pte) do { } while (0)
Signature · 1 line
1.7.9.5
Quoted 5 lines · Steve Capper
Hello,
The following patches bring both HugeTLB support and Transparent HugePage (THP) support to ARM.These are not intended for 3.9.
I've tested this on 3.2 with this series and dependencies (LPAE and some others) backported because I currently only have full board support on 3.2. This was done on OMAP3/CortexA8 pandora board running PSX emulator that renders graphics to 16bpp 1024x512 framebuffer (sometimes 2048x1024 in "enhanced" mode). Here are some average frames per second counter readings without huge pages and with them:
(not showing games as they are irrelevant here)
no hugetlb / hugetlb
game1: ~52 / 57
game2: ~67 / 77
game3: 21 / 26
game4: 22 / 25
As it can be seen here huge pages are very useful for this kind of load, so I'm really looking forward for this to get merged.
FWIW:
Signature · 1 line
Gra?vydas