HugeTLB and THP support for ARM64.
Subsystems (8)
cover HugeTLB and THP support for ARM64.
This series brings huge pages and transparent huge pages to ARM64. The functionality is very similar to x86, and a lot of code that can be used by both ARM64 and x86 is brought into mm to avoid the need for code duplication.
One notable difference from x86 is that ARM64 supports normal pages that are 64KB. When 64KB pages are enabled, huge page and transparent huge pages are 512MB only, otherwise the sizes match x86.
This series applies to 3.10-rc2.
I've tested this under the ARMv8 Fast model and the x86 code has been tested in a KVM guest. libhugetlbfs was used for testing under both architectures.
Changelog:
Patch:
* pud_large usage replaced with pud_huge for general hugetlb code imported into mm.
* comments tidied up for bit swap of PTE_FILE, PTE_PROT_NONE.
RFC v2:
* PROT_NONE support added for HugeTLB and THP.
* pmd_modify implementation fixed.
* Superfluous huge dcache flushing code removed.
* Simplified (and corrected) MAX_ORDER raise for THP && 64KB pages.
* The MAX_ORDER check in huge_mm.h has been corrected.
---
Steve Capper (11):
mm: hugetlb: Copy huge_pmd_share from x86 to mm.
x86: mm: Remove x86 version of huge_pmd_share.
mm: hugetlb: Copy general hugetlb code from x86 to mm.
x86: mm: Remove general hugetlb code from x86.
mm: thp: Correct the HPAGE_PMD_ORDER check.
ARM64: mm: Restore memblock limit when map_mem finished.
ARM64: mm: Make PAGE_NONE pages read only and no-execute.
ARM64: mm: Swap PTE_FILE and PTE_PROT_NONE bits.
ARM64: mm: HugeTLB support.
ARM64: mm: Raise MAX_ORDER for 64KB pages and THP.
ARM64: mm: THP support.
arch/arm64/Kconfig | 17 +++ arch/arm64/include/asm/hugetlb.h | 117 ++++++++++++++++++ arch/arm64/include/asm/pgtable-hwdef.h | 12 ++ arch/arm64/include/asm/pgtable.h | 84 +++++++++++-- arch/arm64/include/asm/tlb.h | 6 + arch/arm64/include/asm/tlbflush.h | 2 + arch/arm64/mm/Makefile | 1 + arch/arm64/mm/fault.c | 19 +-- arch/arm64/mm/hugetlbpage.c | 70 +++++++++++ arch/arm64/mm/mmu.c | 19 ++- arch/x86/Kconfig | 6 + arch/x86/mm/hugetlbpage.c | 187 ---------------------------- include/linux/huge_mm.h | 2 +- include/linux/hugetlb.h | 4 + mm/hugetlb.c | 219 +++++++++++++++++++++++++++++++-- 15 files changed, 537 insertions(+), 228 deletions(-) create mode 100644 arch/arm64/include/asm/hugetlb.h create mode 100644 arch/arm64/mm/hugetlbpage.c
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/11 mm: hugetlb: Copy huge_pmd_share from x86 to mm.
Under x86, multiple puds can be made to reference the same bank of huge pmds provided that they represent a full PUD_SIZE of shared huge memory that is aligned to a PUD_SIZE boundary.
The code to share pmds does not require any architecture specific knowledge other than the fact that pmds can be indexed, thus can be beneficial to some other architectures.
This patch copies the huge pmd sharing (and unsharing) logic from x86/ to mm/ and introduces a new config option to activate it:CONFIG_ARCH_WANTS_HUGE_PMD_SHARE
--- include/linux/hugetlb.h | 4 ++ mm/hugetlb.c | 122 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 126 insertions(+)
include/linux/hugetlb.h +4 -0
@@ -69,6 +69,10 @@ void hugetlb_unreserve_pages(struct inode *inode, long offset, long freed);int dequeue_hwpoisoned_huge_page (struct page *page);void copy_huge_page (struct page *dst,struct page *src); +#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE +pte_t *huge_pmd_share (struct mm_struct *mm,unsigned long addr,pud_t *pud); +#endif +extern unsigned long hugepages_treat_as_movable;extern const unsigned long hugetlb_zero, hugetlb_infinity;extern int sysctl_hugetlb_shm_group;
mm/hugetlb.c +122 -0
128 lines not on this page · raw message
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 01/11] mm: hugetlb: Copy huge_pmd_share from x86 to mm.
Quoted 8 lines · Steve Capper
Under x86, multiple puds can be made to reference the same bank of huge pmds provided that they represent a full
PUD_SIZEof shared huge memory that is aligned to aPUD_SIZEboundary.The code to share pmds does not require any architecture specific knowledge other than the fact that pmds can be indexed, thus can be beneficial to some other architectures.
Quoted 5 lines
This patch copies the huge pmd sharing (and unsharing) logic from x86/ to
mm/and introduces a new config option to activate it:CONFIG_ARCH_WANTS_HUGE_PMD_SHARESigned-off-by Steve Capper <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/11 x86: mm: Remove x86 version of huge_pmd_share.
In reply to Steve CapperThe huge_pmd_share code has been copied over to mm/hugetlb.c to make it accessible to other architectures.
Remove the x86 copy of the huge_pmd_share code and enable the ARCH_WANT_HUGE_PMD_SHARE config flag. That way we reference the general one.
--- arch/x86/Kconfig | 3 ++ arch/x86/mm/hugetlbpage.c | 120 ---------------------------------------------- 2 files changed, 3 insertions(+), 120 deletions(-)
arch/x86/Kconfig +3 -0
@@ -207,6 +207,9 @@ config ARCH_HIBERNATION_POSSIBLE config ARCH_SUSPEND_POSSIBLE def_bool y +config ARCH_WANT_HUGE_PMD_SHARE + def_bool y + config ZONE_DMA32 bool default X86_64
arch/x86/mm/hugetlbpage.c +0 -120
126 lines not on this page · raw message
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 02/11] x86: mm: Remove x86 version of huge_pmd_share.
Quoted 8 lines · Steve Capper
The
huge_pmd_sharecode has been copied over tomm/hugetlb.cto make it accessible to other architectures.Remove the x86 copy of the
huge_pmd_sharecode and enable theARCH_WANT_HUGE_PMD_SHAREconfig flag. That way we reference the general one.Signed-off-by Steve Capper <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 3/11 mm: hugetlb: Copy general hugetlb code from x86 to mm.
In reply to Steve CapperThe huge_pte_alloc, huge_pte_offset and follow_huge_p[mu]d functions in x86/mm/hugetlbpage.c do not rely on any architecture specific knowledge other than the fact that pmds and puds can be treated as huge ptes.
To allow other architectures to use this code (and reduce the need for code duplication), this patch copies these functions into mm, replaces the use of pud_large with pud_huge and provides a config flag to activate them:CONFIG_ARCH_WANT_GENERAL_HUGETLB
If CONFIG_ARCH_WANT_HUGE_PMD_SHARE is also active then the huge_pmd_share code will be called by huge_pte_alloc (othewise we call pmd_alloc and skip the sharing code).
--- mm/hugetlb.c | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 88 insertions(+), 9 deletions(-)
mm/hugetlb.c +88 -9
@@ -2931,15 +2931,6 @@ out_mutex:return ret; } -/* Can be overriden by architectures */ -__attribute__((weak))struct page * -follow_huge_pud (struct mm_struct *mm,unsigned long address, -pud_t *pud,int write) -{ -BUG (); -return NULL ; -} - long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, struct page **pages, struct vm_area_struct **vmas, unsigned long *position, unsigned long *nr_pages,
@@ -3289,8 +3280,96 @@ int huge_pmd_unshare(struct mm_struct *mm, unsigned long *addr, pte_t *ptep) *addr =ALIGN (*addr,HPAGE_SIZE *PTRS_PER_PTE ) -HPAGE_SIZE ;return 1 ; } +#define want_pmd_share () (1) +#else /* !CONFIG_ARCH_WANT_HUGE_PMD_SHARE */ +pte_t *huge_pmd_share (struct mm_struct *mm,unsigned long addr,pud_t *pud) +{ +return NULL ; +} +#define want_pmd_share () (0)#endif /* CONFIG_ARCH_WANT_HUGE_PMD_SHARE */ +#ifdef CONFIG_ARCH_WANT_GENERAL_HUGETLB +pte_t *huge_pte_alloc (struct mm_struct *mm, +unsigned long addr,unsigned long sz) +{ +pgd_t *pgd; +pud_t *pud; +pte_t *pte =NULL ; + + pgd =pgd_offset (mm, addr); + pud =pud_alloc (mm, pgd, addr); +if (pud) { +if (sz ==PUD_SIZE ) { + pte = (pte_t *)pud; + }else { +BUG_ON (sz !=PMD_SIZE ); +if (want_pmd_share () &&pud_none (*pud)) + pte =huge_pmd_share (mm, addr, pud); +else + pte = (pte_t *)pmd_alloc (mm, pud, addr); + } + } +BUG_ON (pte && !pte_none (*pte) && !pte_huge (*pte)); + +return pte; +} + +pte_t *huge_pte_offset (struct mm_struct *mm,unsigned long addr) +{ +pgd_t *pgd; +pud_t *pud; +pmd_t *pmd =NULL ; + + pgd =pgd_offset (mm, addr); +if (pgd_present (*pgd)) { + pud =pud_offset (pgd, addr); +if (pud_present (*pud)) { +if (pud_huge (*pud)) +return (pte_t *)pud; + pmd =pmd_offset (pud, addr); + } + } +return (pte_t *) pmd; +} + +struct page * +follow_huge_pmd (struct mm_struct *mm,unsigned long address, +pmd_t *pmd,int write) +{ +struct page *page; + + page =pte_page (*(pte_t *)pmd); +if (page) + page += ((address & ~PMD_MASK ) >>PAGE_SHIFT ); +return page; +} + +struct page * +follow_huge_pud (struct mm_struct *mm,unsigned long address, +pud_t *pud,int write) +{ +struct page *page; + + page =pte_page (*(pte_t *)pud); +if (page) + page += ((address & ~PUD_MASK ) >>PAGE_SHIFT ); +return page; +} + +#else /* !CONFIG_ARCH_WANT_GENERAL_HUGETLB */ + +/* Can be overriden by architectures */ +__attribute__((weak))struct page * +follow_huge_pud (struct mm_struct *mm,unsigned long address, +pud_t *pud,int write) +{ +BUG (); +return NULL ; +} + +#endif /* CONFIG_ARCH_WANT_GENERAL_HUGETLB */ +#ifdef CONFIG_MEMORY_FAILURE /* Should be called in hugetlb_lock */
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 03/11] mm: hugetlb: Copy general hugetlb code from x86 to mm.
Quoted 11 lines · Steve Capper
The
huge_pte_alloc,huge_pte_offsetand follow_huge_p[mu]d functions inx86/mm/hugetlbpage.cdo not rely on any architecture specific knowledge other than the fact that pmds and puds can be treated as huge ptes.To allow other architectures to use this code (and reduce the need for code duplication), this patch copies these functions into mm, replaces the use of
pud_largewithpud_hugeand provides a config flag to activate them:CONFIG_ARCH_WANT_GENERAL_HUGETLB
Quoted 5 lines
If
CONFIG_ARCH_WANT_HUGE_PMD_SHAREis also active then thehuge_pmd_sharecode will be called byhuge_pte_alloc(othewise we callpmd_allocand skip the sharing code).Signed-off-by Steve Capper <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Catalin Marinas
maintainerRe: [PATCH 03/11] mm: hugetlb: Copy general hugetlb code from x86 to mm.
In reply to Steve CapperQuoted 10 lines · Steve Capper
The
huge_pte_alloc,huge_pte_offsetand follow_huge_p[mu]d functions inx86/mm/hugetlbpage.cdo not rely on any architecture specific knowledge other than the fact that pmds and puds can be treated as huge ptes.To allow other architectures to use this code (and reduce the need for code duplication), this patch copies these functions into mm, replaces the use of
pud_largewithpud_hugeand provides a config flag to activate them:CONFIG_ARCH_WANT_GENERAL_HUGETLB
BTW, I think it's worth mentioning that pud_large is inline while pud_huge is not.
Signature · 7 lines
Catalin -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 4/11 x86: mm: Remove general hugetlb code from x86.
In reply to Steve Capperhuge_pte_alloc, huge_pte_offset and follow_huge_p[mu]d have already been copied over to mm.
This patch removes the x86 copies of these functions and activates the general ones by enabling:CONFIG_ARCH_WANT_GENERAL_HUGETLB
--- arch/x86/Kconfig | 3 +++ arch/x86/mm/hugetlbpage.c | 67 ----------------------------------------------- 2 files changed, 3 insertions(+), 67 deletions(-)
arch/x86/Kconfig +3 -0
@@ -210,6 +210,9 @@ config ARCH_SUSPEND_POSSIBLE config ARCH_WANT_HUGE_PMD_SHARE def_bool y +config ARCH_WANT_GENERAL_HUGETLB + def_bool y + config ZONE_DMA32 bool default X86_64
arch/x86/mm/hugetlbpage.c +0 -67
79 lines not on this page · raw message
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 04/11] x86: mm: Remove general hugetlb code from x86.
Quoted 8 lines · Steve Capper
huge_pte_alloc,huge_pte_offsetand follow_huge_p[mu]d have already been copied over to mm.This patch removes the x86 copies of these functions and activates the general ones by enabling:
CONFIG_ARCH_WANT_GENERAL_HUGETLBSigned-off-by Steve Capper <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 5/11 mm: thp: Correct the HPAGE_PMD_ORDER check.
In reply to Steve CapperAll Transparent Huge Pages are allocated by the buddy allocator.
A compile time check is in place that fails when the order of a transparent huge page is too large to be allocated by the buddy allocator. Unfortunately that compile time check passes when:HPAGE_PMD_ORDER == MAX_ORDER
( which is incorrect as the buddy allocator can only allocate memory of order strictly less than MAX_ORDER. )
This patch updates the compile time check to fail in the above case.
--- include/linux/huge_mm.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/huge_mm.h +1 -1
@@ -123,7 +123,7 @@ extern void __split_huge_page_pmd(struct vm_area_struct *vma, }while (0 ) extern void split_huge_page_pmd_mm(struct mm_struct *mm, unsigned long address,pmd_t *pmd); -#if HPAGE_PMD_ORDER >MAX_ORDER +#if HPAGE_PMD_ORDER >=MAX_ORDER #error "hugepages can't be allocated by the buddy allocator"#endif extern int hugepage_madvise(struct vm_area_struct *vma,
Signature · 7 lines
1.8.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 05/11] mm: thp: Correct the HPAGE_PMD_ORDER check.
Quoted 9 lines · Steve Capper
All Transparent Huge Pages are allocated by the buddy allocator.
A compile time check is in place that fails when the order of a transparent huge page is too large to be allocated by the buddy allocator. Unfortunately that compile time check passes when:
HPAGE_PMD_ORDER==MAX_ORDER
( which is incorrect as the buddy allocator can only allocate memory of order strictly less thanMAX_ORDER. )
Quoted 4 lines
This patch updates the compile time check to fail in the above case.
Signed-off-by Steve Capper <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 6/11 ARM64: mm: Restore memblock limit when map_mem finished.
In reply to Steve CapperIn paging_init the memblock limit is set to restrict any addresses returned by early_alloc to fit within the initial direct kernel mapping in swapper_pg_dir. This allows map_mem to allocate puds, pmds and ptes from the initial direct kernel mapping.
The limit stays low after paging_init() though, meaning any bootmem allocations will be from a restricted subset of memory. Gigabyte huge pages, for instance, are normally allocated from bootmem as their order (18) is too large for the default buddy allocator (MAX_ORDER = 11).
This patch restores the memblock limit when map_mem has finished, allowing gigabyte huge pages (and other objects) to be allocated from all of bootmem.
--- arch/arm64/mm/mmu.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)
arch/arm64/mm/mmu.c +13 -6
@@ -297,6 +297,16 @@ static void __init map_mem(void) {struct memblock_region *reg; +/* +* Temporarily limit the memblock range. We need to do this as +* create_mapping requires puds, pmds and ptes to be allocated from +* memory addressable from the initial direct kernel mapping. +* +* The initial direct kernel mapping, located at swapper_pg_dir, +* gives us PGDIR_SIZE memory starting from PHYS_OFFSET (aligned). +*/ +memblock_set_current_limit ((PHYS_OFFSET &PGDIR_MASK ) +PGDIR_SIZE ); +/* map all the memory banks */ for_each_memblock (memory, reg) {phys_addr_t start = reg->base;
@@ -307,6 +317,9 @@ static void __init map_mem(void)create_mapping (start,__phys_to_virt (start), end - start); } + +/* Limit no longer required. */ +memblock_set_current_limit (MEMBLOCK_ALLOC_ANYWHERE ); }/*
@@ -317,12 +330,6 @@ void __init paging_init(void) {void *zero_page; -/* -* Maximum PGDIR_SIZE addressable via the initial direct kernel -* mapping in swapper_pg_dir. -*/ -memblock_set_current_limit ((PHYS_OFFSET &PGDIR_MASK ) +PGDIR_SIZE ); -init_mem_pgprot ();map_mem ();
Signature · 1 line
1.8.1.4
PATCH 7/11 ARM64: mm: Make PAGE_NONE pages read only and no-execute.
In reply to Steve CapperIf we consider the following code sequence:
my_pte = pte_modify(entry, myprot); x = pte_write(my_pte); y = pte_exec(my_pte);
If myprot comes from a PROT_NONE page, then x and y will both be true which is undesireable behaviour.
This patch sets the no-execute and read-only bits for PAGE_NONE such that the code above will return false for both x and y.
--- arch/arm64/include/asm/pgtable.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/arm64/include/asm/pgtable.h +2 -2
@@ -66,7 +66,7 @@ extern pgprot_t pgprot_default;#define _MOD_PROT (p, b) __pgprot_modify(p, 0, b) -#define PAGE_NONE __pgprot_modify(pgprot_default, PTE_TYPE_MASK, PTE_PROT_NONE) +#define PAGE_NONE __pgprot_modify(pgprot_default, PTE_TYPE_MASK, PTE_PROT_NONE | PTE_RDONLY | PTE_PXN | PTE_UXN)#define PAGE_SHARED _MOD_PROT(pgprot_default, PTE_USER | PTE_NG | PTE_PXN | PTE_UXN)#define PAGE_SHARED_EXEC _MOD_PROT(pgprot_default, PTE_USER | PTE_NG | PTE_PXN)#define PAGE_COPY _MOD_PROT(pgprot_default, PTE_USER | PTE_NG | PTE_PXN | PTE_UXN | PTE_RDONLY)
@@ -76,7 +76,7 @@ extern pgprot_t pgprot_default;#define PAGE_KERNEL _MOD_PROT(pgprot_default, PTE_PXN | PTE_UXN | PTE_DIRTY)#define PAGE_KERNEL_EXEC _MOD_PROT(pgprot_default, PTE_UXN | PTE_DIRTY) -#define __PAGE_NONE __pgprot(((_PAGE_DEFAULT) & ~PTE_TYPE_MASK) | PTE_PROT_NONE) +#define __PAGE_NONE __pgprot(((_PAGE_DEFAULT) & ~PTE_TYPE_MASK) | PTE_PROT_NONE | PTE_RDONLY | PTE_PXN | PTE_UXN)#define __PAGE_SHARED __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN | PTE_UXN)#define __PAGE_SHARED_EXEC __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN)#define __PAGE_COPY __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN | PTE_UXN | PTE_RDONLY)
Signature · 1 line
1.8.1.4
Re: [PATCH 07/11] ARM64: mm: Make PAGE_NONE pages read only and no-execute.
Quoted 6 lines · Steve Capper
If we consider the following code sequence:
my_pte = pte_modify(entry, myprot); x = pte_write(my_pte); y = pte_exec(my_pte);
Quoted 7 lines
If myprot comes from a
PROT_NONEpage, then x and y will both be true which is undesireable behaviour.This patch sets the no-execute and read-only bits for
PAGE_NONEsuch that the code above will return false for both x and y.Signed-off-by Steve Capper <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 8/11 ARM64: mm: Swap PTE_FILE and PTE_PROT_NONE bits.
In reply to Steve CapperUnder ARM64, PTEs can be broadly categorised as follows:
- Present and valid: Bit #0 is set. The PTE is valid and memory access to the region may fault.
- Present and invalid: Bit #0 is clear and bit #1 is set. Represents present memory with PROT_NONE protection. The PTE is an invalid entry, and the user fault handler will raise a SIGSEGV.
- Not present (file): Bits #0 and #1 are clear, bit #2 is set. Memory represented has been paged out. The PTE is an invalid entry, and the fault handler will try and re-populate the memory where necessary.
Huge PTEs are block descriptors that have bit #1 clear. If we wish to represent PROT_NONE huge PTEs we then run into a problem as there is no way to distinguish between regular and huge PTEs if we set bit #1.
As huge PTEs are always present, the meaning of bits #1 and #2 can be swapped for invalid PTEs. This patch swaps the PTE_FILE and PTE_PROT_NONE constants, allowing us to represent PROT_NONE huge PTEs.
--- arch/arm64/include/asm/pgtable.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
arch/arm64/include/asm/pgtable.h +6 -6
@@ -25,8 +25,8 @@* Software defined PTE bits definition. */ #define PTE_VALID (_AT(pteval_t, 1) << 0) -#define PTE_PROT_NONE (_AT(pteval_t, 1) << 1)/* only when !PTE_VALID */ -#define PTE_FILE (_AT(pteval_t, 1) << 2)/* only when !pte_present() */ +#define PTE_FILE (_AT(pteval_t, 1) << 1)/* only when !pte_present() */ +#define PTE_PROT_NONE (_AT(pteval_t, 1) << 2)/* only when !PTE_VALID */ #define PTE_DIRTY (_AT(pteval_t, 1) << 55)#define PTE_SPECIAL (_AT(pteval_t, 1) << 56)
@@ -281,8 +281,8 @@ extern pgd_t idmap_pg_dir[PTRS_PER_PGD];/* * Encode and decode a swap entry: -* bits 0-1: present (must be zero) -* bit 2: PTE_FILE +* bits 0, 2: present (must both be zero) +* bit 1: PTE_FILE * bits 3-8: swap type * bits 9-63: swap offset */
@@ -306,8 +306,8 @@ extern pgd_t idmap_pg_dir[PTRS_PER_PGD];/* * Encode and decode a file entry: -* bits 0-1: present (must be zero) -* bit 2: PTE_FILE +* bits 0, 2: present (must both be zero) +* bit 1: PTE_FILE * bits 3-63: file offset / PAGE_SIZE */ #define pte_file (pte) (pte_val(pte) & PTE_FILE)
Signature · 1 line
1.8.1.4
PATCH 9/11 ARM64: mm: HugeTLB support.
In reply to Steve CapperAdd huge page support to ARM64, different huge page sizes are supported depending on the size of normal pages:
PAGE_SIZE is 4KB:
2MB - (pmds) these can be allocated at any time.
1024MB - (puds) usually allocated on bootup with the command line
with something like: hugepagesz=1G hugepages=6
PAGE_SIZE is 64KB:
512MB - (pmds) usually allocated on bootup via command line.
--- arch/arm64/Kconfig | 9 +++ arch/arm64/include/asm/hugetlb.h | 117 +++++++++++++++++++++++++++++++++ arch/arm64/include/asm/pgtable-hwdef.h | 8 +++ arch/arm64/include/asm/pgtable.h | 13 +++- arch/arm64/mm/Makefile | 1 + arch/arm64/mm/fault.c | 19 ++---- arch/arm64/mm/hugetlbpage.c | 70 ++++++++++++++++++++ 7 files changed, 220 insertions(+), 17 deletions(-) create mode 100644 arch/arm64/include/asm/hugetlb.h create mode 100644 arch/arm64/mm/hugetlbpage.c
arch/arm64/Kconfig +9 -0
@@ -180,6 +180,15 @@ config HW_PERF_EVENTS Enable hardware performance counter support for perf events. If disabled, perf events will use software events only. +config SYS_SUPPORTS_HUGETLBFS + def_bool y + +config ARCH_WANT_GENERAL_HUGETLB + def_bool y + +config ARCH_WANT_HUGE_PMD_SHARE + def_bool y if !ARM64_64K_PAGES + source "mm/Kconfig" endmenu
arch/arm64/include/asm/hugetlb.h new file +117 -0
117 lines not on this page · raw message
arch/arm64/include/asm/pgtable-hwdef.h +8 -0
@@ -25,12 +25,19 @@/* * Hardware page table definitions. * +* Level 1 descriptor (PUD). +*/ + +#define PUD_TABLE_BIT (_AT(pgdval_t, 1) << 1) + +/* * Level 2 descriptor (PMD). */ #define PMD_TYPE_MASK (_AT(pmdval_t, 3) << 0)#define PMD_TYPE_FAULT (_AT(pmdval_t, 0) << 0)#define PMD_TYPE_TABLE (_AT(pmdval_t, 3) << 0)#define PMD_TYPE_SECT (_AT(pmdval_t, 1) << 0) +#define PMD_TABLE_BIT (_AT(pmdval_t, 1) << 1)/* * Section
@@ -53,6 +60,7 @@#define PTE_TYPE_MASK (_AT(pteval_t, 3) << 0)#define PTE_TYPE_FAULT (_AT(pteval_t, 0) << 0)#define PTE_TYPE_PAGE (_AT(pteval_t, 3) << 0) +#define PTE_TABLE_BIT (_AT(pteval_t, 1) << 1)#define PTE_USER (_AT(pteval_t, 1) << 6)/* AP[1] */ #define PTE_RDONLY (_AT(pteval_t, 1) << 7)/* AP[2] */ #define PTE_SHARED (_AT(pteval_t, 3) << 8)/* SH[1:0], inner shareable */
arch/arm64/include/asm/pgtable.h +11 -2
@@ -173,8 +173,17 @@ static inline void set_pte_at(struct mm_struct *mm, unsigned long addr,/* * Huge pte definitions. */ -#define pte_huge (pte) ((pte_val(pte) & PTE_TYPE_MASK) == PTE_TYPE_HUGEPAGE) -#define pte_mkhuge (pte) (__pte((pte_val(pte) & ~PTE_TYPE_MASK) | PTE_TYPE_HUGEPAGE)) +#define pte_huge (pte) (!(pte_val(pte) & PTE_TABLE_BIT)) +#define pte_mkhuge (pte) (__pte(pte_val(pte) & ~PTE_TABLE_BIT)) + +/* +* Hugetlb definitions. +*/ +#define HUGE_MAX_HSTATE 2 +#define HPAGE_SHIFT PMD_SHIFT +#define HPAGE_SIZE (_AC(1, UL) << HPAGE_SHIFT) +#define HPAGE_MASK (~(HPAGE_SIZE - 1)) +#define HUGETLB_PAGE_ORDER (HPAGE_SHIFT - PAGE_SHIFT)#define __HAVE_ARCH_PTE_SPECIAL
arch/arm64/mm/Makefile +1 -0
@@ -2,3 +2,4 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ cache.o copypage.o flush.o \ ioremap.o mmap.o pgd.o mmu.o \ context.o tlb.o proc.o +obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o
arch/arm64/mm/fault.c +4 -15
@@ -364,17 +364,6 @@ static int __kprobes do_translation_fault(unsigned long addr, }/* -* Some section permission faults need to be handled gracefully. They can -* happen due to a __{get,put}_user during an oops. -*/ -static int do_sect_fault (unsigned long addr,unsigned int esr, -struct pt_regs *regs) -{ -do_bad_area (addr, esr, regs); -return 0 ; -} - -/* * This abort handler always returns "fault". */ static int do_bad (unsigned long addr,unsigned int esr,struct pt_regs *regs)
@@ -397,12 +386,12 @@ static struct fault_info { { do_translation_fault, SIGSEGV, SEGV_MAPERR, "level 2 translation fault" }, { do_page_fault, SIGSEGV, SEGV_MAPERR, "level 3 translation fault" }, { do_bad, SIGBUS, 0, "reserved access flag fault" }, - { do_bad, SIGSEGV, SEGV_ACCERR, "level 1 access flag fault" }, - { do_bad, SIGSEGV, SEGV_ACCERR, "level 2 access flag fault" }, + { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 1 access flag fault" }, + { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 2 access flag fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 3 access flag fault" }, { do_bad, SIGBUS, 0, "reserved permission fault" }, - { do_bad, SIGSEGV, SEGV_ACCERR, "level 1 permission fault" }, - { do_sect_fault, SIGSEGV, SEGV_ACCERR, "level 2 permission fault" }, + { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 1 permission fault" }, + { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 2 permission fault" }, { do_page_fault, SIGSEGV, SEGV_ACCERR, "level 3 permission fault" }, { do_bad, SIGBUS, 0, "synchronous external abort" }, { do_bad, SIGBUS, 0, "asynchronous external abort" },
arch/arm64/mm/hugetlbpage.c new file +70 -0
70 lines not on this page · raw message
Signature · 1 line
1.8.1.4
Re: [PATCH 09/11] ARM64: mm: HugeTLB support.
Quoted 8 lines · Steve Capper
Add huge page support to ARM64, different huge page sizes are supported depending on the size of normal pages:
PAGE_SIZEis 4KB:
2MB - (pmds) these can be allocated at any time.
1024MB - (puds) usually allocated on bootup with the command line
with something like: hugepagesz=1G hugepages=6
Quoted 4 lines
PAGE_SIZEis 64KB:
512MB - (pmds) usually allocated on bootup via command line.Signed-off-by Steve Capper <[email protected]>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 10/11 ARM64: mm: Raise MAX_ORDER for 64KB pages and THP.
In reply to Steve CapperThe buddy allocator has a default MAX_ORDER of 11, which is too low to allocate enough memory for 512MB Transparent HugePages if our base page size is 64KB.
This patch introduces MAX_ZONE_ORDER and sets it to 14 when 64KB pages are used in conjuction with THP, otherwise the default value of 11 is used.
--- arch/arm64/Kconfig | 5 +++++ 1 file changed, 5 insertions(+)
arch/arm64/Kconfig +5 -0
@@ -191,6 +191,11 @@ config ARCH_WANT_HUGE_PMD_SHARE source "mm/Kconfig" +config FORCE_MAX_ZONEORDER + int + default "14" if (ARM64_64K_PAGES && TRANSPARENT_HUGEPAGE) + default "11" + endmenu menu "Boot options"
Signature · 1 line
1.8.1.4
PATCH 11/11 ARM64: mm: THP support.
In reply to Steve CapperBring Transparent HugePage support to ARM. The size of a transparent huge page depends on the normal page size. A transparent huge page is always represented as a pmd.
If PAGE_SIZE is 4KB, THPs are 2MB.
If PAGE_SIZE is 64KB, THPs are 512MB.
--- arch/arm64/Kconfig | 3 ++ arch/arm64/include/asm/pgtable-hwdef.h | 4 +++ arch/arm64/include/asm/pgtable.h | 55 ++++++++++++++++++++++++++++++++++ arch/arm64/include/asm/tlb.h | 6 ++++ arch/arm64/include/asm/tlbflush.h | 2 ++ 5 files changed, 70 insertions(+)
arch/arm64/Kconfig +3 -0
@@ -189,6 +189,9 @@ config ARCH_WANT_GENERAL_HUGETLB config ARCH_WANT_HUGE_PMD_SHARE def_bool y if !ARM64_64K_PAGES +config HAVE_ARCH_TRANSPARENT_HUGEPAGE + def_bool y + source "mm/Kconfig" config FORCE_MAX_ZONEORDER
arch/arm64/include/asm/pgtable-hwdef.h +4 -0
@@ -42,6 +42,10 @@/* * Section */ +#define PMD_SECT_VALID (_AT(pmdval_t, 1) << 0) +#define PMD_SECT_PROT_NONE (_AT(pmdval_t, 1) << 2) +#define PMD_SECT_USER (_AT(pmdval_t, 1) << 6)/* AP[1] */ +#define PMD_SECT_RDONLY (_AT(pmdval_t, 1) << 7)/* AP[2] */ #define PMD_SECT_S (_AT(pmdval_t, 3) << 8)#define PMD_SECT_AF (_AT(pmdval_t, 1) << 10)#define PMD_SECT_NG (_AT(pmdval_t, 1) << 11)
arch/arm64/include/asm/pgtable.h +55 -0
61 lines not on this page · raw message
arch/arm64/include/asm/tlb.h +6 -0
@@ -187,4 +187,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp,#define tlb_migrate_finish (mm) do { } while (0) +static inline void +tlb_remove_pmd_tlb_entry (struct mmu_gather *tlb,pmd_t *pmdp,unsigned long addr) +{ +tlb_add_flush (tlb, addr); +} +#endif
arch/arm64/include/asm/tlbflush.h +2 -0
@@ -117,6 +117,8 @@ static inline void update_mmu_cache(struct vm_area_struct *vma,dsb (); } +#define update_mmu_cache_pmd (vma, address, pmd) do { } while (0) +#endif #endif
Signature · 1 line
1.8.1.4
Quoted 32 lines · Steve Capper
This series brings huge pages and transparent huge pages to ARM64. The functionality is very similar to x86, and a lot of code that can be used by both ARM64 and x86 is brought into mm to avoid the need for code duplication.
One notable difference from x86 is that ARM64 supports normal pages that are 64KB. When 64KB pages are enabled, huge page and transparent huge pages are 512MB only, otherwise the sizes match x86.
This series applies to 3.10-rc2.
I've tested this under the ARMv8 Fast model and the x86 code has been tested in a KVM guest. libhugetlbfs was used for testing under both architectures.
Changelog:
Patch:
*pud_largeusage replaced withpud_hugefor general hugetlb code imported into mm.
* comments tidied up for bit swap ofPTE_FILE,PTE_PROT_NONE.RFC v2:
*PROT_NONEsupport added for HugeTLB and THP.
*pmd_modifyimplementation fixed.
* Superfluous huge dcache flushing code removed.
* Simplified (and corrected)MAX_ORDERraise for THP && 64KB pages.
* TheMAX_ORDERcheck inhuge_mm.hhas been corrected.---
Quoted 12 lines
Steve Capper (11):
mm: hugetlb: Copyhuge_pmd_sharefrom x86 to mm.
x86: mm: Remove x86 version ofhuge_pmd_share.
mm: hugetlb: Copy general hugetlb code from x86 to mm.
x86: mm: Remove general hugetlb code from x86.
mm: thp: Correct theHPAGE_PMD_ORDERcheck.
ARM64: mm: Restore memblock limit whenmap_memfinished.
ARM64: mm: MakePAGE_NONEpages read only and no-execute.
ARM64: mm: SwapPTE_FILEandPTE_PROT_NONEbits.
ARM64: mm: HugeTLB support.
ARM64: mm: RaiseMAX_ORDERfor 64KB pages and THP.
ARM64: mm: THP support.
[ ... ]
Hello,
I was just wondering if there were any comments on the mm and x86 patches in this series, or should I send a pull request for them?
Catalin has acked the ARM64 ones but we need the x86->mm code move in place before the ARM64 code is merged. The idea behind the code move was to avoid code duplication between x86 and ARM64 (and ARM).
Thanks,
Signature · 7 lines
Steve -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 52 lines · Steve Capper
Quoted 32 lines · Steve Capper
This series brings huge pages and transparent huge pages to ARM64. The functionality is very similar to x86, and a lot of code that can be used by both ARM64 and x86 is brought into mm to avoid the need for code duplication.
One notable difference from x86 is that ARM64 supports normal pages that are 64KB. When 64KB pages are enabled, huge page and transparent huge pages are 512MB only, otherwise the sizes match x86.
This series applies to 3.10-rc2.
I've tested this under the ARMv8 Fast model and the x86 code has been tested in a KVM guest. libhugetlbfs was used for testing under both architectures.
Changelog:
Patch:
*pud_largeusage replaced withpud_hugefor general hugetlb code imported into mm.
* comments tidied up for bit swap ofPTE_FILE,PTE_PROT_NONE.RFC v2:
*PROT_NONEsupport added for HugeTLB and THP.
*pmd_modifyimplementation fixed.
* Superfluous huge dcache flushing code removed.
* Simplified (and corrected)MAX_ORDERraise for THP && 64KB pages.
* TheMAX_ORDERcheck inhuge_mm.hhas been corrected.---
Quoted 12 lines
Steve Capper (11):
mm: hugetlb: Copyhuge_pmd_sharefrom x86 to mm.
x86: mm: Remove x86 version ofhuge_pmd_share.
mm: hugetlb: Copy general hugetlb code from x86 to mm.
x86: mm: Remove general hugetlb code from x86.
mm: thp: Correct theHPAGE_PMD_ORDERcheck.
ARM64: mm: Restore memblock limit whenmap_memfinished.
ARM64: mm: MakePAGE_NONEpages read only and no-execute.
ARM64: mm: SwapPTE_FILEandPTE_PROT_NONEbits.
ARM64: mm: HugeTLB support.
ARM64: mm: RaiseMAX_ORDERfor 64KB pages and THP.
ARM64: mm: THP support.[ ... ]
Hello,
I was just wondering if there were any comments on the mm and x86 patches in this series, or should I send a pull request for them?
Quoted 7 lines
Catalin has acked the ARM64 ones but we need the
x86->mmcode move in place before the ARM64 code is merged. The idea behind the code move was to avoid code duplication between x86 and ARM64 (and ARM).Thanks,
Signature · 1 line
Steve
Hello,
Just a polite ping on the above. With the to: field expanded out (apologies if I was using the incorrect address before).
I was wondering if there were any comments on the x86/mm patches in this series?
For reference the series can be found archived at:
http://marc.info/?l=linux-mm&m=136932889118818&w=4
Thanks,
Signature · 7 lines
Steve -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andrew Morton
maintainerQuoted 7 lines · On Tue, 11 Jun 2013 10:07:15 +0100 Steve Capper <[email protected]> wrote:
Hello,
I was just wondering if there were any comments on the mm and x86 patches in this series, or should I send a pull request for them?Catalin has acked the ARM64 ones but we need the
x86->mmcode move in place before the ARM64 code is merged. The idea behind the code move was to avoid code duplication between x86 and ARM64 (and ARM).
Ack from me on patches 1, 2, 3, 4 and 5. Please get the whole series into linux-next asap and merge it up at the appropriate time.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 6 lines · Andrew Morton
Quoted 4 lines · On Tue, 11 Jun 2013 10:07:15 +0100 Steve Capper <[email protected]> wrote:
Hello,
I was just wondering if there were any comments on the mm and x86 patches in this series, or should I send a pull request for them?
Quoted 6 lines
Quoted 3 lines
Catalin has acked the ARM64 ones but we need the
x86->mmcode move in place before the ARM64 code is merged. The idea behind the code move was to avoid code duplication between x86 and ARM64 (and ARM).Ack from me on patches 1, 2, 3, 4 and 5. Please get the whole series into linux-next asap and merge it up at the appropriate time.
Thanks,
I've sent a request to the linux-next to pull the series.
Cheers,
Signature · 7 lines
Steve -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>