Skip to contentSkip to conversation

HugeTLB and THP support for ARM64.

PATCH In review Steve Capper 3 participants 24 messages May 23, 2013 – Jun 14, 2013

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

cover HugeTLB and THP support for ARM64.

This series brings huge pages and transparent huge pages to ARM64. The functionality is very similar to x86, and a lot of code that can be used by both ARM64 and x86 is brought into mm to avoid the need for code duplication.

One notable difference from x86 is that ARM64 supports normal pages that are 64KB. When 64KB pages are enabled, huge page and transparent huge pages are 512MB only, otherwise the sizes match x86.

This series applies to 3.10-rc2.

I've tested this under the ARMv8 Fast model and the x86 code has been tested in a KVM guest. libhugetlbfs was used for testing under both architectures.

Changelog:
Patch:
* pud_large usage replaced with pud_huge for general hugetlb code imported into mm.
* comments tidied up for bit swap of PTE_FILE, PTE_PROT_NONE.

RFC v2:
* PROT_NONE support added for HugeTLB and THP.
* pmd_modify implementation fixed.
* Superfluous huge dcache flushing code removed.
* Simplified (and corrected) MAX_ORDER raise for THP && 64KB pages.
* The MAX_ORDER check in huge_mm.h has been corrected.

---

Steve Capper (11):
mm: hugetlb: Copy huge_pmd_share from x86 to mm.
x86: mm: Remove x86 version of huge_pmd_share.
mm: hugetlb: Copy general hugetlb code from x86 to mm.
x86: mm: Remove general hugetlb code from x86.
mm: thp: Correct the HPAGE_PMD_ORDER check.
ARM64: mm: Restore memblock limit when map_mem finished.
ARM64: mm: Make PAGE_NONE pages read only and no-execute.
ARM64: mm: Swap PTE_FILE and PTE_PROT_NONE bits.
ARM64: mm: HugeTLB support.
ARM64: mm: Raise MAX_ORDER for 64KB pages and THP.
ARM64: mm: THP support.

 arch/arm64/Kconfig                     |  17 +++
 arch/arm64/include/asm/hugetlb.h       | 117 ++++++++++++++++++
 arch/arm64/include/asm/pgtable-hwdef.h |  12 ++
 arch/arm64/include/asm/pgtable.h       |  84 +++++++++++--
 arch/arm64/include/asm/tlb.h           |   6 +
 arch/arm64/include/asm/tlbflush.h      |   2 +
 arch/arm64/mm/Makefile                 |   1 +
 arch/arm64/mm/fault.c                  |  19 +--
 arch/arm64/mm/hugetlbpage.c            |  70 +++++++++++
 arch/arm64/mm/mmu.c                    |  19 ++-
 arch/x86/Kconfig                       |   6 +
 arch/x86/mm/hugetlbpage.c              | 187 ----------------------------
 include/linux/huge_mm.h                |   2 +-
 include/linux/hugetlb.h                |   4 +
 mm/hugetlb.c                           | 219 +++++++++++++++++++++++++++++++--
 15 files changed, 537 insertions(+), 228 deletions(-)
 create mode 100644 arch/arm64/include/asm/hugetlb.h
 create mode 100644 arch/arm64/mm/hugetlbpage.c
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#1 lore raw

PATCH 1/11 mm: hugetlb: Copy huge_pmd_share from x86 to mm.

Under x86, multiple puds can be made to reference the same bank of huge pmds provided that they represent a full PUD_SIZE of shared huge memory that is aligned to a PUD_SIZE boundary.

The code to share pmds does not require any architecture specific knowledge other than the fact that pmds can be indexed, thus can be beneficial to some other architectures.

This patch copies the huge pmd sharing (and unsharing) logic from x86/ to mm/ and introduces a new config option to activate it:
CONFIG_ARCH_WANTS_HUGE_PMD_SHARE

---
 include/linux/hugetlb.h |   4 ++
 mm/hugetlb.c            | 122 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 126 insertions(+)
include/linux/hugetlb.h +4 -0
index 6b4890f..981546a 100644
@@ -69,6 +69,10 @@ void hugetlb_unreserve_pages(struct inode *inode, long offset, long freed);
 int dequeue_hwpoisoned_huge_page(struct page *page);
 void copy_huge_page(struct page *dst, struct page *src);
 
+#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE
+pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud);
+#endif
+
 extern unsigned long hugepages_treat_as_movable;
 extern const unsigned long hugetlb_zero, hugetlb_infinity;
 extern int sysctl_hugetlb_shm_group;
mm/hugetlb.c +122 -0

128 lines not on this page · raw message

Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
maintainer Acked-by on v1 1/11
#2 lore raw

Re: [PATCH 01/11] mm: hugetlb: Copy huge_pmd_share from x86 to mm.

Quoted 8 lines · Steve Capper

Under x86, multiple puds can be made to reference the same bank of huge pmds provided that they represent a full PUD_SIZE of shared huge memory that is aligned to a PUD_SIZE boundary.

The code to share pmds does not require any architecture specific knowledge other than the fact that pmds can be indexed, thus can be beneficial to some other architectures.

Quoted 5 lines

This patch copies the huge pmd sharing (and unsharing) logic from x86/ to mm/ and introduces a new config option to activate it:
CONFIG_ARCH_WANTS_HUGE_PMD_SHARE

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw

PATCH 2/11 x86: mm: Remove x86 version of huge_pmd_share.

In reply to Steve Capper

The huge_pmd_share code has been copied over to mm/hugetlb.c to make it accessible to other architectures.

Remove the x86 copy of the huge_pmd_share code and enable the ARCH_WANT_HUGE_PMD_SHARE config flag. That way we reference the general one.

---
 arch/x86/Kconfig          |   3 ++
 arch/x86/mm/hugetlbpage.c | 120 ----------------------------------------------
 2 files changed, 3 insertions(+), 120 deletions(-)
arch/x86/Kconfig +3 -0
index 685692c..476f786 100644
@@ -207,6 +207,9 @@ config ARCH_HIBERNATION_POSSIBLE
 config ARCH_SUSPEND_POSSIBLE
 	def_bool y
 
+config ARCH_WANT_HUGE_PMD_SHARE
+	def_bool y
+
 config ZONE_DMA32
 	bool
 	default X86_64
arch/x86/mm/hugetlbpage.c +0 -120

126 lines not on this page · raw message

Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
maintainer Acked-by on v1 2/11
#4 lore raw

Re: [PATCH 02/11] x86: mm: Remove x86 version of huge_pmd_share.

Quoted 8 lines · Steve Capper

The huge_pmd_share code has been copied over to mm/hugetlb.c to make it accessible to other architectures.

Remove the x86 copy of the huge_pmd_share code and enable the ARCH_WANT_HUGE_PMD_SHARE config flag. That way we reference the general one.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#5 lore raw

PATCH 3/11 mm: hugetlb: Copy general hugetlb code from x86 to mm.

In reply to Steve Capper

The huge_pte_alloc, huge_pte_offset and follow_huge_p[mu]d functions in x86/mm/hugetlbpage.c do not rely on any architecture specific knowledge other than the fact that pmds and puds can be treated as huge ptes.

To allow other architectures to use this code (and reduce the need for code duplication), this patch copies these functions into mm, replaces the use of pud_large with pud_huge and provides a config flag to activate them:
CONFIG_ARCH_WANT_GENERAL_HUGETLB

If CONFIG_ARCH_WANT_HUGE_PMD_SHARE is also active then the huge_pmd_share code will be called by huge_pte_alloc (othewise we call pmd_alloc and skip the sharing code).

---
 mm/hugetlb.c | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 88 insertions(+), 9 deletions(-)
mm/hugetlb.c +88 -9
index b0bfb29..6321726 100644
@@ -2931,15 +2931,6 @@ out_mutex:
 	return ret;
 }
 
-/* Can be overriden by architectures */
-__attribute__((weak)) struct page *
-follow_huge_pud(struct mm_struct *mm, unsigned long address,
-	       pud_t *pud, int write)
-{
-	BUG();
-	return NULL;
-}
-
 long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
 			 struct page **pages, struct vm_area_struct **vmas,
 			 unsigned long *position, unsigned long *nr_pages,
@@ -3289,8 +3280,96 @@ int huge_pmd_unshare(struct mm_struct *mm, unsigned long *addr, pte_t *ptep)
 	*addr = ALIGN(*addr, HPAGE_SIZE * PTRS_PER_PTE) - HPAGE_SIZE;
 	return 1;
 }
+#define want_pmd_share()	(1)
+#else /* !CONFIG_ARCH_WANT_HUGE_PMD_SHARE */
+pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud)
+{
+	return NULL;
+}
+#define want_pmd_share()	(0)
 #endif /* CONFIG_ARCH_WANT_HUGE_PMD_SHARE */
 
+#ifdef CONFIG_ARCH_WANT_GENERAL_HUGETLB
+pte_t *huge_pte_alloc(struct mm_struct *mm,
+			unsigned long addr, unsigned long sz)
+{
+	pgd_t *pgd;
+	pud_t *pud;
+	pte_t *pte = NULL;
+
+	pgd = pgd_offset(mm, addr);
+	pud = pud_alloc(mm, pgd, addr);
+	if (pud) {
+		if (sz == PUD_SIZE) {
+			pte = (pte_t *)pud;
+		} else {
+			BUG_ON(sz != PMD_SIZE);
+			if (want_pmd_share() && pud_none(*pud))
+				pte = huge_pmd_share(mm, addr, pud);
+			else
+				pte = (pte_t *)pmd_alloc(mm, pud, addr);
+		}
+	}
+	BUG_ON(pte && !pte_none(*pte) && !pte_huge(*pte));
+
+	return pte;
+}
+
+pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr)
+{
+	pgd_t *pgd;
+	pud_t *pud;
+	pmd_t *pmd = NULL;
+
+	pgd = pgd_offset(mm, addr);
+	if (pgd_present(*pgd)) {
+		pud = pud_offset(pgd, addr);
+		if (pud_present(*pud)) {
+			if (pud_huge(*pud))
+				return (pte_t *)pud;
+			pmd = pmd_offset(pud, addr);
+		}
+	}
+	return (pte_t *) pmd;
+}
+
+struct page *
+follow_huge_pmd(struct mm_struct *mm, unsigned long address,
+		pmd_t *pmd, int write)
+{
+	struct page *page;
+
+	page = pte_page(*(pte_t *)pmd);
+	if (page)
+		page += ((address & ~PMD_MASK) >> PAGE_SHIFT);
+	return page;
+}
+
+struct page *
+follow_huge_pud(struct mm_struct *mm, unsigned long address,
+		pud_t *pud, int write)
+{
+	struct page *page;
+
+	page = pte_page(*(pte_t *)pud);
+	if (page)
+		page += ((address & ~PUD_MASK) >> PAGE_SHIFT);
+	return page;
+}
+
+#else /* !CONFIG_ARCH_WANT_GENERAL_HUGETLB */
+
+/* Can be overriden by architectures */
+__attribute__((weak)) struct page *
+follow_huge_pud(struct mm_struct *mm, unsigned long address,
+	       pud_t *pud, int write)
+{
+	BUG();
+	return NULL;
+}
+
+#endif /* CONFIG_ARCH_WANT_GENERAL_HUGETLB */
+
 #ifdef CONFIG_MEMORY_FAILURE
 
 /* Should be called in hugetlb_lock */
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
maintainer Acked-by on v1 3/11
#6 lore raw

Re: [PATCH 03/11] mm: hugetlb: Copy general hugetlb code from x86 to mm.

Quoted 11 lines · Steve Capper

The huge_pte_alloc, huge_pte_offset and follow_huge_p[mu]d functions in x86/mm/hugetlbpage.c do not rely on any architecture specific knowledge other than the fact that pmds and puds can be treated as huge ptes.

To allow other architectures to use this code (and reduce the need for code duplication), this patch copies these functions into mm, replaces the use of pud_large with pud_huge and provides a config flag to activate them:
CONFIG_ARCH_WANT_GENERAL_HUGETLB

Quoted 5 lines

If CONFIG_ARCH_WANT_HUGE_PMD_SHARE is also active then the huge_pmd_share code will be called by huge_pte_alloc (othewise we call pmd_alloc and skip the sharing code).

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

maintainer
#7 lore raw

Re: [PATCH 03/11] mm: hugetlb: Copy general hugetlb code from x86 to mm.

In reply to Steve Capper
Quoted 10 lines · Steve Capper

The huge_pte_alloc, huge_pte_offset and follow_huge_p[mu]d functions in x86/mm/hugetlbpage.c do not rely on any architecture specific knowledge other than the fact that pmds and puds can be treated as huge ptes.

To allow other architectures to use this code (and reduce the need for code duplication), this patch copies these functions into mm, replaces the use of pud_large with pud_huge and provides a config flag to activate them:
CONFIG_ARCH_WANT_GENERAL_HUGETLB

BTW, I think it's worth mentioning that pud_large is inline while pud_huge is not.

Signature · 7 lines
Catalin

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#8 lore raw

PATCH 4/11 x86: mm: Remove general hugetlb code from x86.

In reply to Steve Capper

huge_pte_alloc, huge_pte_offset and follow_huge_p[mu]d have already been copied over to mm.

This patch removes the x86 copies of these functions and activates the general ones by enabling:
CONFIG_ARCH_WANT_GENERAL_HUGETLB

---
 arch/x86/Kconfig          |  3 +++
 arch/x86/mm/hugetlbpage.c | 67 -----------------------------------------------
 2 files changed, 3 insertions(+), 67 deletions(-)
arch/x86/Kconfig +3 -0
index 476f786..191c4e3 100644
@@ -210,6 +210,9 @@ config ARCH_SUSPEND_POSSIBLE
 config ARCH_WANT_HUGE_PMD_SHARE
 	def_bool y
 
+config ARCH_WANT_GENERAL_HUGETLB
+	def_bool y
+
 config ZONE_DMA32
 	bool
 	default X86_64
arch/x86/mm/hugetlbpage.c +0 -67

79 lines not on this page · raw message

Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
maintainer Acked-by on v1 4/11
#9 lore raw

Re: [PATCH 04/11] x86: mm: Remove general hugetlb code from x86.

Quoted 8 lines · Steve Capper

huge_pte_alloc, huge_pte_offset and follow_huge_p[mu]d have already been copied over to mm.

This patch removes the x86 copies of these functions and activates the general ones by enabling:
CONFIG_ARCH_WANT_GENERAL_HUGETLB

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#10 lore raw

PATCH 5/11 mm: thp: Correct the HPAGE_PMD_ORDER check.

In reply to Steve Capper

All Transparent Huge Pages are allocated by the buddy allocator.

A compile time check is in place that fails when the order of a transparent huge page is too large to be allocated by the buddy allocator. Unfortunately that compile time check passes when:
HPAGE_PMD_ORDER == MAX_ORDER
( which is incorrect as the buddy allocator can only allocate memory of order strictly less than MAX_ORDER. )

This patch updates the compile time check to fail in the above case.

---
 include/linux/huge_mm.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/huge_mm.h +1 -1
index 528454c..26ee56c 100644
@@ -123,7 +123,7 @@ extern void __split_huge_page_pmd(struct vm_area_struct *vma,
 	} while (0)
 extern void split_huge_page_pmd_mm(struct mm_struct *mm, unsigned long address,
 		pmd_t *pmd);
-#if HPAGE_PMD_ORDER > MAX_ORDER
+#if HPAGE_PMD_ORDER >= MAX_ORDER
 #error "hugepages can't be allocated by the buddy allocator"
 #endif
 extern int hugepage_madvise(struct vm_area_struct *vma,
Signature · 7 lines
1.8.1.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
maintainer Acked-by on v1 5/11
#11 lore raw

Re: [PATCH 05/11] mm: thp: Correct the HPAGE_PMD_ORDER check.

Quoted 9 lines · Steve Capper

All Transparent Huge Pages are allocated by the buddy allocator.

A compile time check is in place that fails when the order of a transparent huge page is too large to be allocated by the buddy allocator. Unfortunately that compile time check passes when:
HPAGE_PMD_ORDER == MAX_ORDER
( which is incorrect as the buddy allocator can only allocate memory of order strictly less than MAX_ORDER. )

Quoted 4 lines

This patch updates the compile time check to fail in the above case.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#12 lore raw

PATCH 6/11 ARM64: mm: Restore memblock limit when map_mem finished.

In reply to Steve Capper

In paging_init the memblock limit is set to restrict any addresses returned by early_alloc to fit within the initial direct kernel mapping in swapper_pg_dir. This allows map_mem to allocate puds, pmds and ptes from the initial direct kernel mapping.

The limit stays low after paging_init() though, meaning any bootmem allocations will be from a restricted subset of memory. Gigabyte huge pages, for instance, are normally allocated from bootmem as their order (18) is too large for the default buddy allocator (MAX_ORDER = 11).

This patch restores the memblock limit when map_mem has finished, allowing gigabyte huge pages (and other objects) to be allocated from all of bootmem.

---
 arch/arm64/mm/mmu.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)
arch/arm64/mm/mmu.c +13 -6
index eeecc9c..9fa027b 100644
@@ -297,6 +297,16 @@ static void __init map_mem(void)
 {
 	struct memblock_region *reg;
 
+	/*
+	 * Temporarily limit the memblock range. We need to do this as
+	 * create_mapping requires puds, pmds and ptes to be allocated from
+	 * memory addressable from the initial direct kernel mapping.
+	 *
+	 * The initial direct kernel mapping, located at swapper_pg_dir,
+	 * gives us PGDIR_SIZE memory starting from PHYS_OFFSET (aligned).
+	 */
+	memblock_set_current_limit((PHYS_OFFSET & PGDIR_MASK) + PGDIR_SIZE);
+
 	/* map all the memory banks */
 	for_each_memblock(memory, reg) {
 		phys_addr_t start = reg->base;
@@ -307,6 +317,9 @@ static void __init map_mem(void)
 
 		create_mapping(start, __phys_to_virt(start), end - start);
 	}
+
+	/* Limit no longer required. */
+	memblock_set_current_limit(MEMBLOCK_ALLOC_ANYWHERE);
 }
 
 /*
@@ -317,12 +330,6 @@ void __init paging_init(void)
 {
 	void *zero_page;
 
-	/*
-	 * Maximum PGDIR_SIZE addressable via the initial direct kernel
-	 * mapping in swapper_pg_dir.
-	 */
-	memblock_set_current_limit((PHYS_OFFSET & PGDIR_MASK) + PGDIR_SIZE);
-
 	init_mem_pgprot();
 	map_mem();
 
Signature · 1 line
1.8.1.4
#13 lore raw

PATCH 7/11 ARM64: mm: Make PAGE_NONE pages read only and no-execute.

In reply to Steve Capper

If we consider the following code sequence:

	my_pte = pte_modify(entry, myprot);
	x = pte_write(my_pte);
	y = pte_exec(my_pte);

If myprot comes from a PROT_NONE page, then x and y will both be true which is undesireable behaviour.

This patch sets the no-execute and read-only bits for PAGE_NONE such that the code above will return false for both x and y.

---
 arch/arm64/include/asm/pgtable.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/arm64/include/asm/pgtable.h +2 -2
index e333a24..77b09d6 100644
@@ -66,7 +66,7 @@ extern pgprot_t pgprot_default;
 
 #define _MOD_PROT(p, b)		__pgprot_modify(p, 0, b)
 
-#define PAGE_NONE		__pgprot_modify(pgprot_default, PTE_TYPE_MASK, PTE_PROT_NONE)
+#define PAGE_NONE		__pgprot_modify(pgprot_default, PTE_TYPE_MASK, PTE_PROT_NONE | PTE_RDONLY | PTE_PXN | PTE_UXN)
 #define PAGE_SHARED		_MOD_PROT(pgprot_default, PTE_USER | PTE_NG | PTE_PXN | PTE_UXN)
 #define PAGE_SHARED_EXEC	_MOD_PROT(pgprot_default, PTE_USER | PTE_NG | PTE_PXN)
 #define PAGE_COPY		_MOD_PROT(pgprot_default, PTE_USER | PTE_NG | PTE_PXN | PTE_UXN | PTE_RDONLY)
@@ -76,7 +76,7 @@ extern pgprot_t pgprot_default;
 #define PAGE_KERNEL		_MOD_PROT(pgprot_default, PTE_PXN | PTE_UXN | PTE_DIRTY)
 #define PAGE_KERNEL_EXEC	_MOD_PROT(pgprot_default, PTE_UXN | PTE_DIRTY)
 
-#define __PAGE_NONE		__pgprot(((_PAGE_DEFAULT) & ~PTE_TYPE_MASK) | PTE_PROT_NONE)
+#define __PAGE_NONE		__pgprot(((_PAGE_DEFAULT) & ~PTE_TYPE_MASK) | PTE_PROT_NONE | PTE_RDONLY | PTE_PXN | PTE_UXN)
 #define __PAGE_SHARED		__pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN | PTE_UXN)
 #define __PAGE_SHARED_EXEC	__pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN)
 #define __PAGE_COPY		__pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN | PTE_UXN | PTE_RDONLY)
Signature · 1 line
1.8.1.4
maintainer Acked-by on v1 7/11
#14 lore raw

Re: [PATCH 07/11] ARM64: mm: Make PAGE_NONE pages read only and no-execute.

Quoted 6 lines · Steve Capper

If we consider the following code sequence:

	my_pte = pte_modify(entry, myprot);
	x = pte_write(my_pte);
	y = pte_exec(my_pte);
Quoted 7 lines

If myprot comes from a PROT_NONE page, then x and y will both be true which is undesireable behaviour.

This patch sets the no-execute and read-only bits for PAGE_NONE such that the code above will return false for both x and y.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#15 lore raw

PATCH 8/11 ARM64: mm: Swap PTE_FILE and PTE_PROT_NONE bits.

In reply to Steve Capper

Under ARM64, PTEs can be broadly categorised as follows:
- Present and valid: Bit #0 is set. The PTE is valid and memory access to the region may fault.

- Present and invalid: Bit #0 is clear and bit #1 is set. Represents present memory with PROT_NONE protection. The PTE is an invalid entry, and the user fault handler will raise a SIGSEGV.

- Not present (file): Bits #0 and #1 are clear, bit #2 is set. Memory represented has been paged out. The PTE is an invalid entry, and the fault handler will try and re-populate the memory where necessary.

Huge PTEs are block descriptors that have bit #1 clear. If we wish to represent PROT_NONE huge PTEs we then run into a problem as there is no way to distinguish between regular and huge PTEs if we set bit #1.

As huge PTEs are always present, the meaning of bits #1 and #2 can be swapped for invalid PTEs. This patch swaps the PTE_FILE and PTE_PROT_NONE constants, allowing us to represent PROT_NONE huge PTEs.

---
 arch/arm64/include/asm/pgtable.h | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
arch/arm64/include/asm/pgtable.h +6 -6
index 77b09d6..8867282 100644
@@ -25,8 +25,8 @@
  * Software defined PTE bits definition.
  */
 #define PTE_VALID		(_AT(pteval_t, 1) << 0)
-#define PTE_PROT_NONE		(_AT(pteval_t, 1) << 1)	/* only when !PTE_VALID */
-#define PTE_FILE		(_AT(pteval_t, 1) << 2)	/* only when !pte_present() */
+#define PTE_FILE		(_AT(pteval_t, 1) << 1)	/* only when !pte_present() */
+#define PTE_PROT_NONE		(_AT(pteval_t, 1) << 2)	/* only when !PTE_VALID */
 #define PTE_DIRTY		(_AT(pteval_t, 1) << 55)
 #define PTE_SPECIAL		(_AT(pteval_t, 1) << 56)
 
@@ -281,8 +281,8 @@ extern pgd_t idmap_pg_dir[PTRS_PER_PGD];
 
 /*
  * Encode and decode a swap entry:
- *	bits 0-1:	present (must be zero)
- *	bit  2:		PTE_FILE
+ *	bits 0, 2:	present (must both be zero)
+ *	bit  1:		PTE_FILE
  *	bits 3-8:	swap type
  *	bits 9-63:	swap offset
  */
@@ -306,8 +306,8 @@ extern pgd_t idmap_pg_dir[PTRS_PER_PGD];
 
 /*
  * Encode and decode a file entry:
- *	bits 0-1:	present (must be zero)
- *	bit  2:		PTE_FILE
+ *	bits 0, 2:	present (must both be zero)
+ *	bit  1:		PTE_FILE
  *	bits 3-63:	file offset / PAGE_SIZE
  */
 #define pte_file(pte)		(pte_val(pte) & PTE_FILE)
Signature · 1 line
1.8.1.4
#16 lore raw

PATCH 9/11 ARM64: mm: HugeTLB support.

In reply to Steve Capper

Add huge page support to ARM64, different huge page sizes are supported depending on the size of normal pages:

PAGE_SIZE is 4KB:
2MB - (pmds) these can be allocated at any time.
1024MB - (puds) usually allocated on bootup with the command line
with something like: hugepagesz=1G hugepages=6

PAGE_SIZE is 64KB:
512MB - (pmds) usually allocated on bootup via command line.

---
 arch/arm64/Kconfig                     |   9 +++
 arch/arm64/include/asm/hugetlb.h       | 117 +++++++++++++++++++++++++++++++++
 arch/arm64/include/asm/pgtable-hwdef.h |   8 +++
 arch/arm64/include/asm/pgtable.h       |  13 +++-
 arch/arm64/mm/Makefile                 |   1 +
 arch/arm64/mm/fault.c                  |  19 ++----
 arch/arm64/mm/hugetlbpage.c            |  70 ++++++++++++++++++++
 7 files changed, 220 insertions(+), 17 deletions(-)
 create mode 100644 arch/arm64/include/asm/hugetlb.h
 create mode 100644 arch/arm64/mm/hugetlbpage.c
arch/arm64/Kconfig +9 -0
index 56b3f6d..cd6eca8 100644
@@ -180,6 +180,15 @@ config HW_PERF_EVENTS
 	  Enable hardware performance counter support for perf events. If
 	  disabled, perf events will use software events only.
 
+config SYS_SUPPORTS_HUGETLBFS
+	def_bool y
+
+config ARCH_WANT_GENERAL_HUGETLB
+	def_bool y
+
+config ARCH_WANT_HUGE_PMD_SHARE
+	def_bool y if !ARM64_64K_PAGES
+
 source "mm/Kconfig"
 
 endmenu
arch/arm64/include/asm/hugetlb.h new file +117 -0

117 lines not on this page · raw message

arch/arm64/include/asm/pgtable-hwdef.h +8 -0
index 75fd13d..e6e0a0d 100644
@@ -25,12 +25,19 @@
 /*
  * Hardware page table definitions.
  *
+ * Level 1 descriptor (PUD).
+ */
+
+#define PUD_TABLE_BIT		(_AT(pgdval_t, 1) << 1)
+
+/*
  * Level 2 descriptor (PMD).
  */
 #define PMD_TYPE_MASK		(_AT(pmdval_t, 3) << 0)
 #define PMD_TYPE_FAULT		(_AT(pmdval_t, 0) << 0)
 #define PMD_TYPE_TABLE		(_AT(pmdval_t, 3) << 0)
 #define PMD_TYPE_SECT		(_AT(pmdval_t, 1) << 0)
+#define PMD_TABLE_BIT		(_AT(pmdval_t, 1) << 1)
 
 /*
  * Section
@@ -53,6 +60,7 @@
 #define PTE_TYPE_MASK		(_AT(pteval_t, 3) << 0)
 #define PTE_TYPE_FAULT		(_AT(pteval_t, 0) << 0)
 #define PTE_TYPE_PAGE		(_AT(pteval_t, 3) << 0)
+#define PTE_TABLE_BIT		(_AT(pteval_t, 1) << 1)
 #define PTE_USER		(_AT(pteval_t, 1) << 6)		/* AP[1] */
 #define PTE_RDONLY		(_AT(pteval_t, 1) << 7)		/* AP[2] */
 #define PTE_SHARED		(_AT(pteval_t, 3) << 8)		/* SH[1:0], inner shareable */
arch/arm64/include/asm/pgtable.h +11 -2
index 8867282..7476896 100644
@@ -173,8 +173,17 @@ static inline void set_pte_at(struct mm_struct *mm, unsigned long addr,
 /*
  * Huge pte definitions.
  */
-#define pte_huge(pte)		((pte_val(pte) & PTE_TYPE_MASK) == PTE_TYPE_HUGEPAGE)
-#define pte_mkhuge(pte)		(__pte((pte_val(pte) & ~PTE_TYPE_MASK) | PTE_TYPE_HUGEPAGE))
+#define pte_huge(pte)		(!(pte_val(pte) & PTE_TABLE_BIT))
+#define pte_mkhuge(pte)		(__pte(pte_val(pte) & ~PTE_TABLE_BIT))
+
+/*
+ * Hugetlb definitions.
+ */
+#define HUGE_MAX_HSTATE		2
+#define HPAGE_SHIFT		PMD_SHIFT
+#define HPAGE_SIZE		(_AC(1, UL) << HPAGE_SHIFT)
+#define HPAGE_MASK		(~(HPAGE_SIZE - 1))
+#define HUGETLB_PAGE_ORDER	(HPAGE_SHIFT - PAGE_SHIFT)
 
 #define __HAVE_ARCH_PTE_SPECIAL
 
arch/arm64/mm/Makefile +1 -0
index 3140a2a..b51d364 100644
@@ -2,3 +2,4 @@ obj-y				:= dma-mapping.o extable.o fault.o init.o \
 				   cache.o copypage.o flush.o \
 				   ioremap.o mmap.o pgd.o mmu.o \
 				   context.o tlb.o proc.o
+obj-$(CONFIG_HUGETLB_PAGE)	+= hugetlbpage.o
arch/arm64/mm/fault.c +4 -15
index 98af6e7..29bf387 100644
@@ -364,17 +364,6 @@ static int __kprobes do_translation_fault(unsigned long addr,
 }
 
 /*
- * Some section permission faults need to be handled gracefully.  They can
- * happen due to a __{get,put}_user during an oops.
- */
-static int do_sect_fault(unsigned long addr, unsigned int esr,
-			 struct pt_regs *regs)
-{
-	do_bad_area(addr, esr, regs);
-	return 0;
-}
-
-/*
  * This abort handler always returns "fault".
  */
 static int do_bad(unsigned long addr, unsigned int esr, struct pt_regs *regs)
@@ -397,12 +386,12 @@ static struct fault_info {
 	{ do_translation_fault,	SIGSEGV, SEGV_MAPERR,	"level 2 translation fault"	},
 	{ do_page_fault,	SIGSEGV, SEGV_MAPERR,	"level 3 translation fault"	},
 	{ do_bad,		SIGBUS,  0,		"reserved access flag fault"	},
-	{ do_bad,		SIGSEGV, SEGV_ACCERR,	"level 1 access flag fault"	},
-	{ do_bad,		SIGSEGV, SEGV_ACCERR,	"level 2 access flag fault"	},
+	{ do_page_fault,	SIGSEGV, SEGV_ACCERR,	"level 1 access flag fault"	},
+	{ do_page_fault,	SIGSEGV, SEGV_ACCERR,	"level 2 access flag fault"	},
 	{ do_page_fault,	SIGSEGV, SEGV_ACCERR,	"level 3 access flag fault"	},
 	{ do_bad,		SIGBUS,  0,		"reserved permission fault"	},
-	{ do_bad,		SIGSEGV, SEGV_ACCERR,	"level 1 permission fault"	},
-	{ do_sect_fault,	SIGSEGV, SEGV_ACCERR,	"level 2 permission fault"	},
+	{ do_page_fault,	SIGSEGV, SEGV_ACCERR,	"level 1 permission fault"	},
+	{ do_page_fault,	SIGSEGV, SEGV_ACCERR,	"level 2 permission fault"	},
 	{ do_page_fault,	SIGSEGV, SEGV_ACCERR,	"level 3 permission fault"	},
 	{ do_bad,		SIGBUS,  0,		"synchronous external abort"	},
 	{ do_bad,		SIGBUS,  0,		"asynchronous external abort"	},
arch/arm64/mm/hugetlbpage.c new file +70 -0

70 lines not on this page · raw message

Signature · 1 line
1.8.1.4
maintainer Acked-by on v1 9/11
#17 lore raw

Re: [PATCH 09/11] ARM64: mm: HugeTLB support.

Quoted 8 lines · Steve Capper

Add huge page support to ARM64, different huge page sizes are supported depending on the size of normal pages:

PAGE_SIZE is 4KB:
2MB - (pmds) these can be allocated at any time.
1024MB - (puds) usually allocated on bootup with the command line
with something like: hugepagesz=1G hugepages=6

Quoted 4 lines

PAGE_SIZE is 64KB:
512MB - (pmds) usually allocated on bootup via command line.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#18 lore raw

PATCH 10/11 ARM64: mm: Raise MAX_ORDER for 64KB pages and THP.

In reply to Steve Capper

The buddy allocator has a default MAX_ORDER of 11, which is too low to allocate enough memory for 512MB Transparent HugePages if our base page size is 64KB.

This patch introduces MAX_ZONE_ORDER and sets it to 14 when 64KB pages are used in conjuction with THP, otherwise the default value of 11 is used.

---
 arch/arm64/Kconfig | 5 +++++
 1 file changed, 5 insertions(+)
arch/arm64/Kconfig +5 -0
index cd6eca8..10607d6 100644
@@ -191,6 +191,11 @@ config ARCH_WANT_HUGE_PMD_SHARE
 
 source "mm/Kconfig"
 
+config FORCE_MAX_ZONEORDER
+	int
+	default "14" if (ARM64_64K_PAGES && TRANSPARENT_HUGEPAGE)
+	default "11"
+
 endmenu
 
 menu "Boot options"
Signature · 1 line
1.8.1.4
#19 lore raw

PATCH 11/11 ARM64: mm: THP support.

In reply to Steve Capper

Bring Transparent HugePage support to ARM. The size of a transparent huge page depends on the normal page size. A transparent huge page is always represented as a pmd.

If PAGE_SIZE is 4KB, THPs are 2MB.
If PAGE_SIZE is 64KB, THPs are 512MB.

---
 arch/arm64/Kconfig                     |  3 ++
 arch/arm64/include/asm/pgtable-hwdef.h |  4 +++
 arch/arm64/include/asm/pgtable.h       | 55 ++++++++++++++++++++++++++++++++++
 arch/arm64/include/asm/tlb.h           |  6 ++++
 arch/arm64/include/asm/tlbflush.h      |  2 ++
 5 files changed, 70 insertions(+)
arch/arm64/Kconfig +3 -0
index 10607d6..308a556 100644
@@ -189,6 +189,9 @@ config ARCH_WANT_GENERAL_HUGETLB
 config ARCH_WANT_HUGE_PMD_SHARE
 	def_bool y if !ARM64_64K_PAGES
 
+config HAVE_ARCH_TRANSPARENT_HUGEPAGE
+	def_bool y
+
 source "mm/Kconfig"
 
 config FORCE_MAX_ZONEORDER
arch/arm64/include/asm/pgtable-hwdef.h +4 -0
index e6e0a0d..63c9d0d 100644
@@ -42,6 +42,10 @@
 /*
  * Section
  */
+#define PMD_SECT_VALID		(_AT(pmdval_t, 1) << 0)
+#define PMD_SECT_PROT_NONE	(_AT(pmdval_t, 1) << 2)
+#define PMD_SECT_USER		(_AT(pmdval_t, 1) << 6)		/* AP[1] */
+#define PMD_SECT_RDONLY		(_AT(pmdval_t, 1) << 7)		/* AP[2] */
 #define PMD_SECT_S		(_AT(pmdval_t, 3) << 8)
 #define PMD_SECT_AF		(_AT(pmdval_t, 1) << 10)
 #define PMD_SECT_NG		(_AT(pmdval_t, 1) << 11)
arch/arm64/include/asm/pgtable.h +55 -0

61 lines not on this page · raw message

arch/arm64/include/asm/tlb.h +6 -0
index 654f096..46b3beb 100644
@@ -187,4 +187,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp,
 
 #define tlb_migrate_finish(mm)		do { } while (0)
 
+static inline void
+tlb_remove_pmd_tlb_entry(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr)
+{
+	tlb_add_flush(tlb, addr);
+}
+
 #endif
arch/arm64/include/asm/tlbflush.h +2 -0
index 122d632..8b48203 100644
@@ -117,6 +117,8 @@ static inline void update_mmu_cache(struct vm_area_struct *vma,
 	dsb();
 }
 
+#define update_mmu_cache_pmd(vma, address, pmd) do { } while (0)
+
 #endif
 
 #endif
Signature · 1 line
1.8.1.4
#20 lore raw
In reply to Steve Capper
Quoted 32 lines · Steve Capper

This series brings huge pages and transparent huge pages to ARM64. The functionality is very similar to x86, and a lot of code that can be used by both ARM64 and x86 is brought into mm to avoid the need for code duplication.

One notable difference from x86 is that ARM64 supports normal pages that are 64KB. When 64KB pages are enabled, huge page and transparent huge pages are 512MB only, otherwise the sizes match x86.

This series applies to 3.10-rc2.

I've tested this under the ARMv8 Fast model and the x86 code has been tested in a KVM guest. libhugetlbfs was used for testing under both architectures.

Changelog:
Patch:
* pud_large usage replaced with pud_huge for general hugetlb code imported into mm.
* comments tidied up for bit swap of PTE_FILE, PTE_PROT_NONE.

RFC v2:
* PROT_NONE support added for HugeTLB and THP.
* pmd_modify implementation fixed.
* Superfluous huge dcache flushing code removed.
* Simplified (and corrected) MAX_ORDER raise for THP && 64KB pages.
* The MAX_ORDER check in huge_mm.h has been corrected.

---

Quoted 12 lines

Steve Capper (11):
mm: hugetlb: Copy huge_pmd_share from x86 to mm.
x86: mm: Remove x86 version of huge_pmd_share.
mm: hugetlb: Copy general hugetlb code from x86 to mm.
x86: mm: Remove general hugetlb code from x86.
mm: thp: Correct the HPAGE_PMD_ORDER check.
ARM64: mm: Restore memblock limit when map_mem finished.
ARM64: mm: Make PAGE_NONE pages read only and no-execute.
ARM64: mm: Swap PTE_FILE and PTE_PROT_NONE bits.
ARM64: mm: HugeTLB support.
ARM64: mm: Raise MAX_ORDER for 64KB pages and THP.
ARM64: mm: THP support.

[ ... ]

Hello,
I was just wondering if there were any comments on the mm and x86 patches in this series, or should I send a pull request for them?

Catalin has acked the ARM64 ones but we need the x86->mm code move in place before the ARM64 code is merged. The idea behind the code move was to avoid code duplication between x86 and ARM64 (and ARM).

Thanks,

Signature · 7 lines
Steve

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#21 lore raw
Quoted 52 lines · Steve Capper
Quoted 32 lines · Steve Capper

This series brings huge pages and transparent huge pages to ARM64. The functionality is very similar to x86, and a lot of code that can be used by both ARM64 and x86 is brought into mm to avoid the need for code duplication.

One notable difference from x86 is that ARM64 supports normal pages that are 64KB. When 64KB pages are enabled, huge page and transparent huge pages are 512MB only, otherwise the sizes match x86.

This series applies to 3.10-rc2.

I've tested this under the ARMv8 Fast model and the x86 code has been tested in a KVM guest. libhugetlbfs was used for testing under both architectures.

Changelog:
Patch:
* pud_large usage replaced with pud_huge for general hugetlb code imported into mm.
* comments tidied up for bit swap of PTE_FILE, PTE_PROT_NONE.

RFC v2:
* PROT_NONE support added for HugeTLB and THP.
* pmd_modify implementation fixed.
* Superfluous huge dcache flushing code removed.
* Simplified (and corrected) MAX_ORDER raise for THP && 64KB pages.
* The MAX_ORDER check in huge_mm.h has been corrected.

---

Quoted 12 lines

Steve Capper (11):
mm: hugetlb: Copy huge_pmd_share from x86 to mm.
x86: mm: Remove x86 version of huge_pmd_share.
mm: hugetlb: Copy general hugetlb code from x86 to mm.
x86: mm: Remove general hugetlb code from x86.
mm: thp: Correct the HPAGE_PMD_ORDER check.
ARM64: mm: Restore memblock limit when map_mem finished.
ARM64: mm: Make PAGE_NONE pages read only and no-execute.
ARM64: mm: Swap PTE_FILE and PTE_PROT_NONE bits.
ARM64: mm: HugeTLB support.
ARM64: mm: Raise MAX_ORDER for 64KB pages and THP.
ARM64: mm: THP support.

[ ... ]

Hello,
I was just wondering if there were any comments on the mm and x86 patches in this series, or should I send a pull request for them?

Quoted 7 lines

Catalin has acked the ARM64 ones but we need the x86->mm code move in place before the ARM64 code is merged. The idea behind the code move was to avoid code duplication between x86 and ARM64 (and ARM).

Thanks,

Signature · 1 line
Steve

Hello,
Just a polite ping on the above. With the to: field expanded out (apologies if I was using the incorrect address before).

I was wondering if there were any comments on the x86/mm patches in this series?

For reference the series can be found archived at:
http://marc.info/?l=linux-mm&m=136932889118818&w=4

Thanks,

Signature · 7 lines
Steve

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
maintainer
#22 lore raw
In reply to Steve Capper
Quoted 7 lines · On Tue, 11 Jun 2013 10:07:15 +0100 Steve Capper <[email protected]> wrote:

Hello,
I was just wondering if there were any comments on the mm and x86 patches in this series, or should I send a pull request for them?

Catalin has acked the ARM64 ones but we need the x86->mm code move in place before the ARM64 code is merged. The idea behind the code move was to avoid code duplication between x86 and ARM64 (and ARM).

Ack from me on patches 1, 2, 3, 4 and 5. Please get the whole series into linux-next asap and merge it up at the appropriate time.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#23 lore raw
Quoted 6 lines · Andrew Morton
Quoted 4 lines · On Tue, 11 Jun 2013 10:07:15 +0100 Steve Capper <[email protected]> wrote:

Hello,
I was just wondering if there were any comments on the mm and x86 patches in this series, or should I send a pull request for them?

Quoted 6 lines
Quoted 3 lines

Catalin has acked the ARM64 ones but we need the x86->mm code move in place before the ARM64 code is merged. The idea behind the code move was to avoid code duplication between x86 and ARM64 (and ARM).

Ack from me on patches 1, 2, 3, 4 and 5. Please get the whole series into linux-next asap and merge it up at the appropriate time.

Thanks,
I've sent a request to the linux-next to pull the series.

Cheers,

Signature · 7 lines
Steve

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>