Skip to contentSkip to conversation

Fix issues with huge mapping in ioremap for ARM64

PATCH Superseded Chintan Pandya 3 participants 12 messages Apr 3, 2018 – Apr 27, 2018

Subsystems (5)
x86 Library code arm64 Generic include/asm header files +1 more subsystems

v8 of 13 Superseded Latest is v13 →

lore
Share & tools
lore

Chintan Pandya

author
#0 lore raw

v8 cover Fix issues with huge mapping in ioremap for ARM64

This series of patches are follow up work (and depends on) Toshi Kani <[email protected]>'s patches "fix memory leak/ panic in ioremap huge pages".

This series of patches are tested on 4.9 kernel with Cortex-A75 based SoC.

These patches can also go into '-stable' branch (if accepted) for 4.6 onwards.

From V7->V8:
- Properly fixed compilation issue in x86 file

From V6->V7:
- Fixed compilation issue in x86 case
- V6 patches were not properly enumarated

From V5->V6:
- Use __flush_tlb_kernel_pgtable() for both PUD and PMD. Remove "bool tlb_inv" based variance as it is not need now
- Re-naming for consistency

From V4->V5:
- Add new API __flush_tlb_kernel_pgtable(unsigned long addr) for kernel addresses

From V3->V4:
- Add header for 'addr' in x86 implementation
- Re-order pmd/pud clear and table free
- Avoid redundant TLB invalidatation in one perticular case

From V2->V3:
- Use the exisiting page table free interface to do arm64 specific things

From V1->V2:
- Rebased my patches on top of "[PATCH v2 1/2] mm/vmalloc:
Add interfaces to free unmapped page table"
- Honored BBM for ARM64

Chintan Pandya (4):
ioremap: Update pgtable free interfaces with addr
arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable
arm64: Implement page table free interfaces Revert "arm64: Enforce BBM for huge IO/VMAP mappings"

 arch/arm64/include/asm/tlbflush.h |  6 ++++++
 arch/arm64/mm/mmu.c               | 37 +++++++++++++++++++++++++------------
 arch/x86/mm/pgtable.c             |  8 +++++---
 include/asm-generic/pgtable.h     |  8 ++++----
 lib/ioremap.c                     |  4 ++--
 5 files changed, 42 insertions(+), 21 deletions(-)
Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project
author
#1 lore raw

PATCH v8 1/4 ioremap: Update pgtable free interfaces with addr

This patch ("mm/vmalloc: Add interfaces to free unmapped page table") adds following 2 interfaces to free the page table in case we implement huge mapping.

pud_free_pmd_page() and pmd_free_pte_page()

Some architectures (like arm64) needs to do proper TLB maintanance after updating pagetable entry even in map. Why ? Read this,
https://patchwork.kernel.org/patch/10134581/

Pass 'addr' in these interfaces so that proper TLB ops can be performed.

Signed-off-by Chintan Pandya <[email protected]>
---
 arch/arm64/mm/mmu.c           | 4 ++--
 arch/x86/mm/pgtable.c         | 8 +++++---
 include/asm-generic/pgtable.h | 8 ++++----
 lib/ioremap.c                 | 4 ++--
 4 files changed, 13 insertions(+), 11 deletions(-)
arch/arm64/mm/mmu.c +2 -2
index 2dbb2c9..da98828 100644
@@ -973,12 +973,12 @@ int pmd_clear_huge(pmd_t *pmdp)
 	return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
 	return pud_none(*pud);
 }
 
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
 	return pmd_none(*pmd);
 }
arch/x86/mm/pgtable.c +5 -3
index 34cda7e..cd95edc 100644
@@ -706,11 +706,12 @@ int pmd_clear_huge(pmd_t *pmd)
 /**
  * pud_free_pmd_page - Clear pud entry and free pmd page.
  * @pud: Pointer to a PUD.
+ * @addr: Virtual address associated with pud.
  *
  * Context: The pud range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
 	pmd_t *pmd;
 	int i;

@@ -721,7 +722,7 @@ int pud_free_pmd_page(pud_t *pud)
 	pmd = (pmd_t *)pud_page_vaddr(*pud);
 
 	for (i = 0; i < PTRS_PER_PMD; i++)
-		if (!pmd_free_pte_page(&pmd[i]))
+		if (!pmd_free_pte_page(&pmd[i], addr + (i * PMD_SIZE)))
 			return 0;
 
 	pud_clear(pud);

@@ -733,11 +734,12 @@ int pud_free_pmd_page(pud_t *pud)
 /**
  * pmd_free_pte_page - Clear pmd entry and free pte page.
  * @pmd: Pointer to a PMD.
+ * @addr: Virtual address associated with pmd.
  *
  * Context: The pmd range has been unmaped and TLB purged.
  * Return: 1 if clearing the entry succeeded. 0 otherwise.
  */
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
 	pte_t *pte;
include/asm-generic/pgtable.h +4 -4
index bfbb44a..e6310f4 100644
@@ -983,8 +983,8 @@ static inline int p4d_clear_huge(p4d_t *p4d)
 int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot);
 int pud_clear_huge(pud_t *pud);
 int pmd_clear_huge(pmd_t *pmd);
-int pud_free_pmd_page(pud_t *pud);
-int pmd_free_pte_page(pmd_t *pmd);
+int pud_free_pmd_page(pud_t *pud, unsigned long addr);
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr);
 #else	/* !CONFIG_HAVE_ARCH_HUGE_VMAP */
 static inline int p4d_set_huge(p4d_t *p4d, phys_addr_t addr, pgprot_t prot)
 {

@@ -1010,11 +1010,11 @@ static inline int pmd_clear_huge(pmd_t *pmd)
 {
 	return 0;
 }
-static inline int pud_free_pmd_page(pud_t *pud)
+static inline int pud_free_pmd_page(pud_t *pud, unsigned long addr)
 {
 	return 0;
 }
-static inline int pmd_free_pte_page(pmd_t *pmd)
+static inline int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 {
 	return 0;
 }
lib/ioremap.c +2 -2
index 54e5bba..517f585 100644
@@ -92,7 +92,7 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr,
 		if (ioremap_pmd_enabled() &&
 		    ((next - addr) == PMD_SIZE) &&
 		    IS_ALIGNED(phys_addr + addr, PMD_SIZE) &&
-		    pmd_free_pte_page(pmd)) {
+		    pmd_free_pte_page(pmd, addr)) {
 			if (pmd_set_huge(pmd, phys_addr + addr, prot))
 				continue;
 		}

@@ -119,7 +119,7 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr,
 		if (ioremap_pud_enabled() &&
 		    ((next - addr) == PUD_SIZE) &&
 		    IS_ALIGNED(phys_addr + addr, PUD_SIZE) &&
-		    pud_free_pmd_page(pud)) {
+		    pud_free_pmd_page(pud, addr)) {
 			if (pud_set_huge(pud, phys_addr + addr, prot))
 				continue;
 		}
Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project
author
#2 lore raw

PATCH v8 2/4 arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable

In reply to Chintan Pandya

Add an interface to invalidate intermediate page tables from TLB for kernel.

Signed-off-by Chintan Pandya <[email protected]>
---
 arch/arm64/include/asm/tlbflush.h | 6 ++++++
 1 file changed, 6 insertions(+)
arch/arm64/include/asm/tlbflush.h +6 -0
index 9e82dd7..6a4816d 100644
@@ -209,6 +209,12 @@ static inline void __flush_tlb_pgtable(struct mm_struct *mm,
 	dsb(ish);
 }
 
+static inline void __flush_tlb_kernel_pgtable(unsigned long addr)
+{
+	addr >>= 12;
+	__tlbi(vaae1is, addr);
+	dsb(ish);
+}
 #endif
 
 #endif
Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project
maintainer
#3 lore raw

Re: [PATCH v8 2/4] arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable

Quoted 13 lines · Chintan Pandya

Add an interface to invalidate intermediate page tables from TLB for kernel.

Signed-off-by Chintan Pandya <[email protected]>
---
 arch/arm64/include/asm/tlbflush.h | 6 ++++++
 1 file changed, 6 insertions(+)
arch/arm64/include/asm/tlbflush.h +0 -0
index 9e82dd7..6a4816d 100644

arch/arm64/include/asm/tlbflush.h static inline void __flush_tlb_pgtable(struct mm_struct *mm, · line 218

Quoted 10 lines
arch/arm64/include/asm/tlbflush.h +6 -0
@@ -209,6 +209,12 @@ static inline void __flush_tlb_pgtable(struct mm_struct *mm,
 	dsb(ish);
 }
 
+static inline void __flush_tlb_kernel_pgtable(unsigned long addr)
+{
+	addr >>= 12;
+	__tlbi(vaae1is, addr);
+	dsb(ish);
+}
 #endif

Please use __TLBI_VADDR here as it does some additional masking.

Signature · 1 line
Catalin
author
#4 lore raw

Re: [PATCH v8 2/4] arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable

Quoted 17 lines · Catalin Marinas
Quoted 16 lines · Chintan Pandya

Add an interface to invalidate intermediate page tables from TLB for kernel.

Signed-off-by Chintan Pandya <[email protected]>
---
  arch/arm64/include/asm/tlbflush.h | 6 ++++++
  1 file changed, 6 insertions(+)
arch/arm64/include/asm/tlbflush.h +0 -0
index 9e82dd7..6a4816d 100644
@@ -209,6 +209,12 @@ static inline void __flush_tlb_pgtable(struct mm_struct *mm,
  	dsb(ish);
  }
Quoted 10 lines

arch/arm64/include/asm/tlbflush.h static inline void __flush_tlb_pgtable(struct mm_struct *mm, · line 218

Quoted 7 lines
arch/arm64/include/asm/tlbflush.h +6 -0
+static inline void __flush_tlb_kernel_pgtable(unsigned long addr)
+{
+	addr >>= 12;
+	__tlbi(vaae1is, addr);
+	dsb(ish);
+}
  #endif

Please use __TLBI_VADDR here as it does some additional masking.

Sure.

Chintan

Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project
author
#5 lore raw

PATCH v8 3/4 arm64: Implement page table free interfaces

In reply to Chintan Pandya

Implement pud_free_pmd_page() and pmd_free_pte_page().

Implementation requires,
1) Clearing off the current pud/pmd entry
2) Invalidate TLB which could have previously
valid but not stale entry
3) Freeing of the un-used next level page tables

Signed-off-by Chintan Pandya <[email protected]>
---
 arch/arm64/mm/mmu.c | 29 +++++++++++++++++++++++++----
 1 file changed, 25 insertions(+), 4 deletions(-)
arch/arm64/mm/mmu.c +25 -4
index da98828..0f651db 100644
@@ -45,6 +45,7 @@
 #include <asm/memblock.h>
 #include <asm/mmu_context.h>
 #include <asm/ptdump.h>
+#include <asm/tlbflush.h>
 
 #define NO_BLOCK_MAPPINGS	BIT(0)
 #define NO_CONT_MAPPINGS	BIT(1)

@@ -973,12 +974,32 @@ int pmd_clear_huge(pmd_t *pmdp)
 	return 1;
 }
 
-int pud_free_pmd_page(pud_t *pud, unsigned long addr)
+int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
 {
-	return pud_none(*pud);
+	pmd_t *table;
+
+	if (pmd_present(READ_ONCE(*pmdp))) {
+		table = __va(pmd_val(*pmdp));
+		pmd_clear(pmdp);
+		__flush_tlb_kernel_pgtable(addr);
+		free_page((unsigned long) table);
+	}
+	return 1;
 }
 
-int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
+int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
 {
-	return pmd_none(*pmd);
+	pmd_t *table;
+	int i;
+
+	if (pud_present(READ_ONCE(*pudp))) {
+		table = __va(pud_val(*pudp));
+		for (i = 0; i < PTRS_PER_PMD; i++)
+			pmd_free_pte_page(&table[i], addr + (i * PMD_SIZE));
+
+		pud_clear(pudp);
+		__flush_tlb_kernel_pgtable(addr);
+		free_page((unsigned long) table);
+	}
+	return 1;
 }
Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project
author
#6 lore raw

PATCH v8 4/4 Revert "arm64: Enforce BBM for huge IO/VMAP mappings"

In reply to Chintan Pandya

This commit 15122ee2c515a ("arm64: Enforce BBM for huge IO/VMAP mappings") is a temporary work-around until the issues with CONFIG_HAVE_ARCH_HUGE_VMAP gets fixed.

Revert this change as we have fixes for the issue.

Signed-off-by Chintan Pandya <[email protected]>
---
 arch/arm64/mm/mmu.c | 8 --------
 1 file changed, 8 deletions(-)
arch/arm64/mm/mmu.c +0 -8
index 0f651db..170009b 100644
@@ -935,10 +935,6 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot)
 	pgprot_t sect_prot = __pgprot(PUD_TYPE_SECT |
 					pgprot_val(mk_sect_prot(prot)));
 
-	/* ioremap_page_range doesn't honour BBM */
-	if (pud_present(READ_ONCE(*pudp)))
-		return 0;
-
 	BUG_ON(phys & ~PUD_MASK);
 	set_pud(pudp, pfn_pud(__phys_to_pfn(phys), sect_prot));
 	return 1;

@@ -949,10 +945,6 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t prot)
 	pgprot_t sect_prot = __pgprot(PMD_TYPE_SECT |
 					pgprot_val(mk_sect_prot(prot)));
 
-	/* ioremap_page_range doesn't honour BBM */
-	if (pmd_present(READ_ONCE(*pmdp)))
-		return 0;
-
 	BUG_ON(phys & ~PMD_MASK);
 	set_pmd(pmdp, pfn_pmd(__phys_to_pfn(phys), sect_prot));
 	return 1;
Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project
#7 lore raw
In reply to Chintan Pandya

Hi Chintan,

Quoted 6 lines · Chintan Pandya

This series of patches are follow up work (and depends on) Toshi Kani <[email protected]>'s patches "fix memory leak/ panic in ioremap huge pages".

This series of patches are tested on 4.9 kernel with Cortex-A75 based SoC.

Given that this is targeting mainline, can you please test with 4.16? Basing your patches on something that is over 15 months old is not exactly reassuring.

Thanks,

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
author
#8 lore raw
Quoted 1 line · Marc Zyngier

Hi Chintan,

Hi Marc,

Quoted 6 lines · Chintan Pandya
Quoted 4 lines

This series of patches are follow up work (and depends on) Toshi Kani <[email protected]>'s patches "fix memory leak/ panic in ioremap huge pages".

Quoted 6 lines
Quoted 2 lines

This series of patches are tested on 4.9 kernel with Cortex-A75 based SoC.

Given that this is targeting mainline, can you please test with 4.16? Basing your patches on something that is over 15 months old is not exactly reassuring.

We have 4.14 ported with the devices (remotely) accessible to me. If that's sufficient, I will post test report by tomorrow.

If we really require these patches to be tested with 4.16, I would need some help here.

Thanks,

Quoted 5 lines

Thanks,

	M.

Chintan

Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project
author
#9 lore raw
Quoted 18 lines · Chintan Pandya
Quoted 1 line · Marc Zyngier

Hi Chintan,

Hi Marc,

Quoted 6 lines · Chintan Pandya
Quoted 4 lines

This series of patches are follow up work (and depends on) Toshi Kani <[email protected]>'s patches "fix memory leak/ panic in ioremap huge pages".

Quoted 6 lines
Quoted 2 lines

This series of patches are tested on 4.9 kernel with Cortex-A75 based SoC.

Given that this is targeting mainline, can you please test with 4.16? Basing your patches on something that is over 15 months old is not exactly reassuring.

Quoted 6 lines

We have 4.14 ported with the devices (remotely) accessible to me. If that's sufficient, I will post test report by tomorrow.

If we really require these patches to be tested with 4.16, I would need some help here.

Rajendra Nayak <[email protected]> helped me with testing these patches on 4.16 kernel (with all the dependent patches already part of 4.16) on SDM845.

In addition to kernel boot up, he executed my test-code which was stressing ioremap framework and was able to reproduce the bug without these patches (and Will Deacon's WA).

That passed 10hrs of run.

[36001.446355] IOREMAP_TEST: my tests ended now

Thanks Rajendra Nayak for testing these patches.

Quoted 9 lines

Thanks,

Quoted 5 lines

Thanks,

    M.

Chintan

Chintan

Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project
author
#10 lore raw
In reply to Chintan Pandya

Ping...

Quoted 46 lines · Chintan Pandya

This series of patches are follow up work (and depends on) Toshi Kani <[email protected]>'s patches "fix memory leak/ panic in ioremap huge pages".

This series of patches are tested on 4.9 kernel with Cortex-A75 based SoC.

These patches can also go into '-stable' branch (if accepted) for 4.6 onwards.

From V7->V8:
- Properly fixed compilation issue in x86 file

From V6->V7:
- Fixed compilation issue in x86 case
- V6 patches were not properly enumarated

From V5->V6:
- Use __flush_tlb_kernel_pgtable() for both PUD and PMD. Remove "bool tlb_inv" based variance as it is not need now
- Re-naming for consistency

From V4->V5:
- Add new API __flush_tlb_kernel_pgtable(unsigned long addr) for kernel addresses

From V3->V4:
- Add header for 'addr' in x86 implementation
- Re-order pmd/pud clear and table free
- Avoid redundant TLB invalidatation in one perticular case

From V2->V3:
- Use the exisiting page table free interface to do arm64 specific things

From V1->V2:
- Rebased my patches on top of "[PATCH v2 1/2] mm/vmalloc:
Add interfaces to free unmapped page table"
- Honored BBM for ARM64

Chintan Pandya (4):
ioremap: Update pgtable free interfaces with addr
arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable
arm64: Implement page table free interfaces Revert "arm64: Enforce BBM for huge IO/VMAP mappings"

Quoted 7 lines
  arch/arm64/include/asm/tlbflush.h |  6 ++++++
  arch/arm64/mm/mmu.c               | 37 +++++++++++++++++++++++++------------
  arch/x86/mm/pgtable.c             |  8 +++++---
  include/asm-generic/pgtable.h     |  8 ++++----
  lib/ioremap.c                     |  4 ++--
  5 files changed, 42 insertions(+), 21 deletions(-)

Chintan

Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation
Collaborative Project
maintainer
#11 lore raw
Quoted 6 lines · Chintan Pandya

Ping...

Quoted 3 lines · Chintan Pandya

This series of patches are follow up work (and depends on) Toshi Kani <[email protected]>'s patches "fix memory leak/ panic in ioremap huge pages".

First of all, you'd need to post patches against the latest mainline (4.17-rcX).

Second, since they depend on another series, we'd have to wait for the comments on that series to settle.

Signature · 1 line
Catalin