Skip to contentSkip to conversation

riscv: mm: execute local TLB flush after populating vmemmap

PATCH Applied Vincent Chen 4 participants 5 messages Jan 17, 2024 – Feb 8, 2024 RISC-V

lore
Share & tools
lore
#0 lore raw

PATCH v6 1/1 riscv: mm: execute local TLB flush after populating vmemmap

The spare_init() calls memmap_populate() many times to create VA to PA mapping for the VMEMMAP area, where all "struct page" are located once CONFIG_SPARSEMEM_VMEMMAP is defined. These "struct page" are later initialized in the zone_sizes_init() function. However, during this process, no sfence.vma instruction is executed for this VMEMMAP area. This omission may cause the hart to fail to perform page table walk because some data related to the address translation is invisible to the hart. To solve this issue, the local_flush_tlb_kernel_range() is called right after the sparse_init() to execute a sfence.vma instruction for this VMEMMAP area, ensuring that all data related to the address translation is visible to the hart.

Fixes d95f1a542c3d ("RISC-V: Implement sparsemem")
---
 arch/riscv/include/asm/tlbflush.h | 2 ++
 arch/riscv/mm/init.c              | 5 +++++
 arch/riscv/mm/tlbflush.c          | 6 ++++++
 3 files changed, 13 insertions(+)
arch/riscv/include/asm/tlbflush.h +2 -0
index 8f3418c5f172..525267379ccb 100644
@@ -46,6 +46,7 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end);
 void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start,
 			unsigned long end);
 #endif
+void local_flush_tlb_kernel_range(unsigned long start, unsigned long end);
 #else /* CONFIG_SMP && CONFIG_MMU */
 
 #define flush_tlb_all() local_flush_tlb_all()
@@ -66,6 +67,7 @@ static inline void flush_tlb_kernel_range(unsigned long start,
 
 #define flush_tlb_mm(mm) flush_tlb_all()
 #define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all()
+#define local_flush_tlb_kernel_range(start, end) flush_tlb_all()
 #endif /* !CONFIG_SMP || !CONFIG_MMU */
 
 #endif /* _ASM_RISCV_TLBFLUSH_H */
arch/riscv/mm/init.c +5 -0
index 2e011cbddf3a..cc56a0945120 100644
@@ -1377,6 +1377,10 @@ void __init misc_mem_init(void)
 	early_memtest(min_low_pfn << PAGE_SHIFT, max_low_pfn << PAGE_SHIFT);
 	arch_numa_init();
 	sparse_init();
+#ifdef CONFIG_SPARSEMEM_VMEMMAP
+	/* The entire VMEMMAP region has been populated. Flush TLB for this region */
+	local_flush_tlb_kernel_range(VMEMMAP_START, VMEMMAP_END);
+#endif
 	zone_sizes_init();
 	arch_reserve_crashkernel();
 	memblock_dump_all();
@@ -1386,6 +1390,7 @@ void __init misc_mem_init(void)
 int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
 			       struct vmem_altmap *altmap)
 {
+	/* Defer the required TLB flush until the entire VMEMMAP region has been populated */
 	return vmemmap_populate_basepages(start, end, node, NULL);
 }
 #endif
arch/riscv/mm/tlbflush.c +6 -0
index e6659d7368b3..d11a4ae87ec1 100644
@@ -193,6 +193,12 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end)
 	__flush_tlb_range(NULL, start, end - start, PAGE_SIZE);
 }
 
+/* Flush a range of kernel pages without broadcasting */
+void local_flush_tlb_kernel_range(unsigned long start, unsigned long end)
+{
+	local_flush_tlb_range_asid(start, end - start, PAGE_SIZE, FLUSH_TLB_NO_ASID);
+}
+
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start,
 			unsigned long end)
Signature · 7 lines
2.34.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#1 lore raw

Hi Vincent,

Thank you for the patch.

Quoted 13 lines · Vincent Chen

The spare_init() calls memmap_populate() many times to create VA to PA mapping for the VMEMMAP area, where all "struct page" are located once CONFIG_SPARSEMEM_VMEMMAP is defined. These "struct page" are later initialized in the zone_sizes_init() function. However, during this process, no sfence.vma instruction is executed for this VMEMMAP area. This omission may cause the hart to fail to perform page table walk because some data related to the address translation is invisible to the hart. To solve this issue, the local_flush_tlb_kernel_range() is called right after the sparse_init() to execute a sfence.vma instruction for this VMEMMAP area, ensuring that all data related to the address translation is visible to the hart.

Quoted 9 lines
Fixes d95f1a542c3d ("RISC-V: Implement sparsemem")
---
 arch/riscv/include/asm/tlbflush.h | 2 ++
 arch/riscv/mm/init.c              | 5 +++++
 arch/riscv/mm/tlbflush.c          | 6 ++++++
 3 files changed, 13 insertions(+)

I get below build issue with this patch applied:

arch/riscv/mm/tlbflush.c:207:6: error: redefinition of 'local_flush_tlb_kernel_range'
207 | void local_flush_tlb_kernel_range(unsigned long start,
unsigned long end)

      |      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

arch/riscv/mm/tlbflush.c:69:6: note: previous definition of 'local_flush_tlb_kernel_range' with type 'void(long unsigned int, long unsigned int)'
69 | void local_flush_tlb_kernel_range(unsigned long start,
unsigned long end)

      |      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

make[4]: *** [scripts/Makefile.build:243: arch/riscv/mm/tlbflush.o] Error 1 make[4]: *** Waiting for unfinished jobs....
CC kernel/resource.o
make[3]: *** [scripts/Makefile.build:481: arch/riscv/mm] Error 2 make[3]: *** Waiting for unfinished jobs....

I have attached the defconfig file.

Cheers,
Prabhakar

Quoted 64 lines
arch/riscv/include/asm/tlbflush.h +2 -0
index 8f3418c5f172..525267379ccb 100644
@@ -46,6 +46,7 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end);
 void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start,
                        unsigned long end);
 #endif
+void local_flush_tlb_kernel_range(unsigned long start, unsigned long end);
 #else /* CONFIG_SMP && CONFIG_MMU */
 
 #define flush_tlb_all() local_flush_tlb_all()
@@ -66,6 +67,7 @@ static inline void flush_tlb_kernel_range(unsigned long start,
 
 #define flush_tlb_mm(mm) flush_tlb_all()
 #define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all()
+#define local_flush_tlb_kernel_range(start, end) flush_tlb_all()
 #endif /* !CONFIG_SMP || !CONFIG_MMU */
 
 #endif /* _ASM_RISCV_TLBFLUSH_H */
arch/riscv/mm/init.c +5 -0
index 2e011cbddf3a..cc56a0945120 100644
@@ -1377,6 +1377,10 @@ void __init misc_mem_init(void)
        early_memtest(min_low_pfn << PAGE_SHIFT, max_low_pfn << PAGE_SHIFT);
        arch_numa_init();
        sparse_init();
+#ifdef CONFIG_SPARSEMEM_VMEMMAP
+       /* The entire VMEMMAP region has been populated. Flush TLB for this region */
+       local_flush_tlb_kernel_range(VMEMMAP_START, VMEMMAP_END);
+#endif
        zone_sizes_init();
        arch_reserve_crashkernel();
        memblock_dump_all();
@@ -1386,6 +1390,7 @@ void __init misc_mem_init(void)
 int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
                               struct vmem_altmap *altmap)
 {
+       /* Defer the required TLB flush until the entire VMEMMAP region has been populated */
        return vmemmap_populate_basepages(start, end, node, NULL);
 }
 #endif
arch/riscv/mm/tlbflush.c +6 -1
index e6659d7368b3..d11a4ae87ec1 100644
@@ -193,6 +193,12 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end)
        __flush_tlb_range(NULL, start, end - start, PAGE_SIZE);
 }
 
+/* Flush a range of kernel pages without broadcasting */
+void local_flush_tlb_kernel_range(unsigned long start, unsigned long end)
+{
+       local_flush_tlb_range_asid(start, end - start, PAGE_SIZE, FLUSH_TLB_NO_ASID);
+}
+
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start,
                        unsigned long end)
--

2.34.1

Quoted 4 lines

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#2 lore raw

Hi Prabhakar,

Quoted 40 lines · Prabhakar

Hi Vincent,

Thank you for the patch.

Quoted 13 lines · Vincent Chen

The spare_init() calls memmap_populate() many times to create VA to PA mapping for the VMEMMAP area, where all "struct page" are located once CONFIG_SPARSEMEM_VMEMMAP is defined. These "struct page" are later initialized in the zone_sizes_init() function. However, during this process, no sfence.vma instruction is executed for this VMEMMAP area. This omission may cause the hart to fail to perform page table walk because some data related to the address translation is invisible to the hart. To solve this issue, the local_flush_tlb_kernel_range() is called right after the sparse_init() to execute a sfence.vma instruction for this VMEMMAP area, ensuring that all data related to the address translation is visible to the hart.

Quoted 9 lines
Fixes d95f1a542c3d ("RISC-V: Implement sparsemem")
---
 arch/riscv/include/asm/tlbflush.h | 2 ++
 arch/riscv/mm/init.c              | 5 +++++
 arch/riscv/mm/tlbflush.c          | 6 ++++++
 3 files changed, 13 insertions(+)

I get below build issue with this patch applied:

arch/riscv/mm/tlbflush.c:207:6: error: redefinition of 'local_flush_tlb_kernel_range'
207 | void local_flush_tlb_kernel_range(unsigned long start,
unsigned long end)

      |      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

arch/riscv/mm/tlbflush.c:69:6: note: previous definition of 'local_flush_tlb_kernel_range' with type 'void(long unsigned int, long unsigned int)'
69 | void local_flush_tlb_kernel_range(unsigned long start,
unsigned long end)

Quoted 9 lines
      |      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

make[4]: *** [scripts/Makefile.build:243: arch/riscv/mm/tlbflush.o] Error 1 make[4]: *** Waiting for unfinished jobs....
CC kernel/resource.o
make[3]: *** [scripts/Makefile.build:481: arch/riscv/mm] Error 2 make[3]: *** Waiting for unfinished jobs....

I have attached the defconfig file.

Indeed, we recently added the same function, you can find a rebased version of this patch on top of 6.8 here:
https://github.com/AlexGhiti/linux-riscv/commit/756d1871f58edac455773db0bb996f8d064a6b06

Quoted 67 lines

Cheers,
Prabhakar

Quoted 64 lines
arch/riscv/include/asm/tlbflush.h +2 -0
index 8f3418c5f172..525267379ccb 100644
@@ -46,6 +46,7 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end);
 void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start,
                        unsigned long end);
 #endif
+void local_flush_tlb_kernel_range(unsigned long start, unsigned long end);
 #else /* CONFIG_SMP && CONFIG_MMU */
 
 #define flush_tlb_all() local_flush_tlb_all()
@@ -66,6 +67,7 @@ static inline void flush_tlb_kernel_range(unsigned long start,
 
 #define flush_tlb_mm(mm) flush_tlb_all()
 #define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all()
+#define local_flush_tlb_kernel_range(start, end) flush_tlb_all()
 #endif /* !CONFIG_SMP || !CONFIG_MMU */
 
 #endif /* _ASM_RISCV_TLBFLUSH_H */
arch/riscv/mm/init.c +5 -0
index 2e011cbddf3a..cc56a0945120 100644
@@ -1377,6 +1377,10 @@ void __init misc_mem_init(void)
        early_memtest(min_low_pfn << PAGE_SHIFT, max_low_pfn << PAGE_SHIFT);
        arch_numa_init();
        sparse_init();
+#ifdef CONFIG_SPARSEMEM_VMEMMAP
+       /* The entire VMEMMAP region has been populated. Flush TLB for this region */
+       local_flush_tlb_kernel_range(VMEMMAP_START, VMEMMAP_END);
+#endif
        zone_sizes_init();
        arch_reserve_crashkernel();
        memblock_dump_all();
@@ -1386,6 +1390,7 @@ void __init misc_mem_init(void)
 int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
                               struct vmem_altmap *altmap)
 {
+       /* Defer the required TLB flush until the entire VMEMMAP region has been populated */
        return vmemmap_populate_basepages(start, end, node, NULL);
 }
 #endif
arch/riscv/mm/tlbflush.c +6 -1
index e6659d7368b3..d11a4ae87ec1 100644
@@ -193,6 +193,12 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end)
        __flush_tlb_range(NULL, start, end - start, PAGE_SIZE);
 }
 
+/* Flush a range of kernel pages without broadcasting */
+void local_flush_tlb_kernel_range(unsigned long start, unsigned long end)
+{
+       local_flush_tlb_range_asid(start, end - start, PAGE_SIZE, FLUSH_TLB_NO_ASID);
+}
+
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start,
                        unsigned long end)
--

2.34.1

Quoted 4 lines
Quoted 4 lines

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#3 lore raw

Hi Alexandre,

Quoted 52 lines · Alexandre Ghiti

Hi Prabhakar,

Quoted 47 lines · Prabhakar

Hi Vincent,

Thank you for the patch.

Quoted 13 lines · Vincent Chen

The spare_init() calls memmap_populate() many times to create VA to PA mapping for the VMEMMAP area, where all "struct page" are located once CONFIG_SPARSEMEM_VMEMMAP is defined. These "struct page" are later initialized in the zone_sizes_init() function. However, during this process, no sfence.vma instruction is executed for this VMEMMAP area. This omission may cause the hart to fail to perform page table walk because some data related to the address translation is invisible to the hart. To solve this issue, the local_flush_tlb_kernel_range() is called right after the sparse_init() to execute a sfence.vma instruction for this VMEMMAP area, ensuring that all data related to the address translation is visible to the hart.

Quoted 9 lines
Fixes d95f1a542c3d ("RISC-V: Implement sparsemem")
---
 arch/riscv/include/asm/tlbflush.h | 2 ++
 arch/riscv/mm/init.c              | 5 +++++
 arch/riscv/mm/tlbflush.c          | 6 ++++++
 3 files changed, 13 insertions(+)

I get below build issue with this patch applied:

arch/riscv/mm/tlbflush.c:207:6: error: redefinition of 'local_flush_tlb_kernel_range'
207 | void local_flush_tlb_kernel_range(unsigned long start,
unsigned long end)

      |      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

arch/riscv/mm/tlbflush.c:69:6: note: previous definition of 'local_flush_tlb_kernel_range' with type 'void(long unsigned int, long unsigned int)'
69 | void local_flush_tlb_kernel_range(unsigned long start,
unsigned long end)

      |      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

make[4]: *** [scripts/Makefile.build:243: arch/riscv/mm/tlbflush.o] Error 1 make[4]: *** Waiting for unfinished jobs....
CC kernel/resource.o
make[3]: *** [scripts/Makefile.build:481: arch/riscv/mm] Error 2 make[3]: *** Waiting for unfinished jobs....

Quoted 7 lines
Quoted 2 lines

I have attached the defconfig file.

Indeed, we recently added the same function, you can find a rebased version of this patch on top of 6.8 here:
https://github.com/AlexGhiti/linux-riscv/commit/756d1871f58edac455773db0bb996f8d064a6b06

Thanks for the pointer.

Cheers,
Prabhakar

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#4 lore raw
In reply to Vincent Chen
This patch was applied to riscv/linux.git (fixes) by Palmer Dabbelt <[email protected]>:

Hello:

This patch was applied to riscv/linux.git (fixes)
by Palmer Dabbelt <[email protected]>:

Quoted 12 lines · On Wed, 17 Jan 2024 22:03:33 +0800 you wrote:

The spare_init() calls memmap_populate() many times to create VA to PA mapping for the VMEMMAP area, where all "struct page" are located once CONFIG_SPARSEMEM_VMEMMAP is defined. These "struct page" are later initialized in the zone_sizes_init() function. However, during this process, no sfence.vma instruction is executed for this VMEMMAP area. This omission may cause the hart to fail to perform page table walk because some data related to the address translation is invisible to the hart. To solve this issue, the local_flush_tlb_kernel_range() is called right after the sparse_init() to execute a sfence.vma instruction for this VMEMMAP area, ensuring that all data related to the address translation is visible to the hart.

Quoted 1 line

[...]

Here is the summary with links:
- [v6] riscv: mm: execute local TLB flush after populating vmemmap

    https://git.kernel.org/riscv/c/d9807d60c145

You are awesome, thank you!

Signature · 9 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html



_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv