Skip to contentSkip to conversation

riscv: mm: Fixup spurious fault of kernel vaddr

PATCH In review [email protected] 2 participants 8 messages Jul 21, 2023 – Jul 31, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 riscv: mm: Fixup spurious fault of kernel vaddr

From: Guo Ren <[email protected]>

RISC-V specification permits the caching of PTEs whose V (Valid) bit is clear. Operating systems must be written to cope with this possibility, but implementers are reminded that eagerly caching invalid PTEs will reduce performance by causing additional page faults.

So we must keep vmalloc_fault for the spurious page faults of kernel virtual address from an OoO machine.

---
 arch/riscv/mm/fault.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/riscv/mm/fault.c +1 -2
index 85165fe438d8..f662c9eae7d4 100644
@@ -258,8 +258,7 @@ void handle_page_fault(struct pt_regs *regs)
 	 * only copy the information from the master page table,
 	 * nothing more.
 	 */
-	if ((!IS_ENABLED(CONFIG_MMU) || !IS_ENABLED(CONFIG_64BIT)) &&
-	    unlikely(addr >= VMALLOC_START && addr < VMALLOC_END)) {
+	if (unlikely(addr >= TASK_SIZE)) {
 		vmalloc_fault(regs, code, addr);
 		return;
 	}
Signature · 1 line
2.36.1
#1 lore raw
Quoted 22 lines · On 21/07/2023 16:51, [email protected] wrote:

From: Guo Ren <[email protected]>

RISC-V specification permits the caching of PTEs whose V (Valid) bit is clear. Operating systems must be written to cope with this possibility, but implementers are reminded that eagerly caching invalid PTEs will reduce performance by causing additional page faults.

So we must keep vmalloc_fault for the spurious page faults of kernel virtual address from an OoO machine.

---
  arch/riscv/mm/fault.c | 3 +--
  1 file changed, 1 insertion(+), 2 deletions(-)
arch/riscv/mm/fault.c +0 -0
index 85165fe438d8..f662c9eae7d4 100644

arch/riscv/mm/fault.c void handle_page_fault(struct pt_regs *regs) · line 264

Quoted 9 lines
arch/riscv/mm/fault.c +1 -2
@@ -258,8 +258,7 @@ void handle_page_fault(struct pt_regs *regs)
  	 * only copy the information from the master page table,
  	 * nothing more.
  	 */
-	if ((!IS_ENABLED(CONFIG_MMU) || !IS_ENABLED(CONFIG_64BIT)) &&
-	    unlikely(addr >= VMALLOC_START && addr < VMALLOC_END)) {
+	if (unlikely(addr >= TASK_SIZE)) {
  		vmalloc_fault(regs, code, addr);
  		return;
  	}

Can you share what you are trying to fix here?

I have a fix (that's currently running our CI) for commit 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") that implements flush_cache_vmap() since we lost the vmalloc_fault.

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

author
#2 lore raw
Quoted 29 lines · Alexandre Ghiti
Quoted 26 lines · On 21/07/2023 16:51, [email protected] wrote:

From: Guo Ren <[email protected]>

RISC-V specification permits the caching of PTEs whose V (Valid) bit is clear. Operating systems must be written to cope with this possibility, but implementers are reminded that eagerly caching invalid PTEs will reduce performance by causing additional page faults.

So we must keep vmalloc_fault for the spurious page faults of kernel virtual address from an OoO machine.

---
  arch/riscv/mm/fault.c | 3 +--
  1 file changed, 1 insertion(+), 2 deletions(-)
arch/riscv/mm/fault.c +0 -1
index 85165fe438d8..f662c9eae7d4 100644
@@ -258,8 +258,7 @@ void handle_page_fault(struct pt_regs *regs)
       * only copy the information from the master page table,
       * nothing more.
       */
-     if ((!IS_ENABLED(CONFIG_MMU) || !IS_ENABLED(CONFIG_64BIT)) &&
Quoted 8 lines

arch/riscv/mm/fault.c void handle_page_fault(struct pt_regs *regs) · line 264

Quoted 5 lines
arch/riscv/mm/fault.c +1 -1
-         unlikely(addr >= VMALLOC_START && addr < VMALLOC_END)) {
+     if (unlikely(addr >= TASK_SIZE)) {
              vmalloc_fault(regs, code, addr);
              return;
      }

Can you share what you are trying to fix here?

We met a spurious page fault panic on an OoO machine.

1. The processor speculative execution brings the V=0 entries into the TLB in the kernel virtual address.
2. Linux kernel installs the kernel virtual address with the page, and V=1
3. When kernel code access the kernel virtual address, it would raise a page fault as the V=0 entry in the tlb.
4. No vmalloc_fault, then panic.

Quoted 4 lines

I have a fix (that's currently running our CI) for commit 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") that implements flush_cache_vmap() since we lost the vmalloc_fault.

Could you share that patch?

Quoted 1 line
Signature · 2 lines
Best Regards
 Guo Ren
#3 lore raw
Quoted 45 lines · Guo Ren
Quoted 19 lines · Alexandre Ghiti
Quoted 17 lines · On 21/07/2023 16:51, [email protected] wrote:

From: Guo Ren <[email protected]>

RISC-V specification permits the caching of PTEs whose V (Valid) bit is clear. Operating systems must be written to cope with this possibility, but implementers are reminded that eagerly caching invalid PTEs will reduce performance by causing additional page faults.

So we must keep vmalloc_fault for the spurious page faults of kernel virtual address from an OoO machine.

---
   arch/riscv/mm/fault.c | 3 +--
   1 file changed, 1 insertion(+), 2 deletions(-)
Quoted 16 lines
Quoted 5 lines
arch/riscv/mm/fault.c +0 -0
index 85165fe438d8..f662c9eae7d4 100644

arch/riscv/mm/fault.c void handle_page_fault(struct pt_regs *regs) · line 264

Quoted 9 lines
arch/riscv/mm/fault.c +1 -2
@@ -258,8 +258,7 @@ void handle_page_fault(struct pt_regs *regs)
        * only copy the information from the master page table,
        * nothing more.
        */
-     if ((!IS_ENABLED(CONFIG_MMU) || !IS_ENABLED(CONFIG_64BIT)) &&
-         unlikely(addr >= VMALLOC_START && addr < VMALLOC_END)) {
+     if (unlikely(addr >= TASK_SIZE)) {
               vmalloc_fault(regs, code, addr);
               return;
       }

Can you share what you are trying to fix here?

We met a spurious page fault panic on an OoO machine.

1. The processor speculative execution brings the V=0 entries into the TLB in the kernel virtual address.
2. Linux kernel installs the kernel virtual address with the page, and V=1
3. When kernel code access the kernel virtual address, it would raise a page fault as the V=0 entry in the tlb.
4. No vmalloc_fault, then panic.

Quoted 4 lines
Quoted 3 lines

I have a fix (that's currently running our CI) for commit 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") that implements flush_cache_vmap() since we lost the vmalloc_fault.

Could you share that patch?

Here we go:

Author: Alexandre Ghiti <[email protected]>
Date:   Fri Jul 21 08:43:44 2023 +0000

    riscv: Implement flush_cache_vmap()

    The RISC-V kernel needs a sfence.vma after a page table modification: we used to rely on the vmalloc fault handling to emit an sfence.vma, but commit 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") got rid of this path for 64-bit kernels, so now we need to explicitly emit a sfence.vma in flush_cache_vmap().

    Note that we don't need to implement flush_cache_vunmap() as the generic code should emit a flush tlb after unmapping a vmalloc region.

Fixes 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area")
arch/riscv/include/asm/cacheflush.h +0 -0
index 8091b8bf4883..b93ffddf8a61 100644
@@ -37,6 +37,10 @@ static inline void flush_dcache_page(struct page *page)

 #define flush_icache_user_page(vma, pg, addr, len) \
        flush_icache_mm(vma->vm_mm, 0)

+#ifdef CONFIG_64BIT
+#define flush_cache_vmap(start, end) flush_tlb_kernel_range(start, end) +#endif
+
 #ifndef CONFIG_SMP

 #define flush_icache_all() local_flush_icache_all()

Let me know if that works for you!

Quoted 2 lines

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

author
#4 lore raw
Quoted 84 lines · Alexandre Ghiti
Quoted 45 lines · Guo Ren
Quoted 19 lines · Alexandre Ghiti
Quoted 17 lines · On 21/07/2023 16:51, [email protected] wrote:

From: Guo Ren <[email protected]>

RISC-V specification permits the caching of PTEs whose V (Valid) bit is clear. Operating systems must be written to cope with this possibility, but implementers are reminded that eagerly caching invalid PTEs will reduce performance by causing additional page faults.

So we must keep vmalloc_fault for the spurious page faults of kernel virtual address from an OoO machine.

---
   arch/riscv/mm/fault.c | 3 +--
   1 file changed, 1 insertion(+), 2 deletions(-)
Quoted 16 lines
Quoted 5 lines
arch/riscv/mm/fault.c +0 -0
index 85165fe438d8..f662c9eae7d4 100644

arch/riscv/mm/fault.c void handle_page_fault(struct pt_regs *regs) · line 264

Quoted 9 lines
arch/riscv/mm/fault.c +1 -2
@@ -258,8 +258,7 @@ void handle_page_fault(struct pt_regs *regs)
        * only copy the information from the master page table,
        * nothing more.
        */
-     if ((!IS_ENABLED(CONFIG_MMU) || !IS_ENABLED(CONFIG_64BIT)) &&
-         unlikely(addr >= VMALLOC_START && addr < VMALLOC_END)) {
+     if (unlikely(addr >= TASK_SIZE)) {
               vmalloc_fault(regs, code, addr);
               return;
       }

Can you share what you are trying to fix here?

We met a spurious page fault panic on an OoO machine.

1. The processor speculative execution brings the V=0 entries into the TLB in the kernel virtual address.
2. Linux kernel installs the kernel virtual address with the page, and V=1
3. When kernel code access the kernel virtual address, it would raise a page fault as the V=0 entry in the tlb.
4. No vmalloc_fault, then panic.

Quoted 4 lines
Quoted 3 lines

I have a fix (that's currently running our CI) for commit 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") that implements flush_cache_vmap() since we lost the vmalloc_fault.

Could you share that patch?

Here we go:

Author: Alexandre Ghiti <[email protected]>
Date:   Fri Jul 21 08:43:44 2023 +0000

riscv: Implement flush_cache_vmap()

The RISC-V kernel needs a sfence.vma after a page table
modification: we
used to rely on the vmalloc fault handling to emit an sfence.vma, but commit 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") got rid of this path for 64-bit kernels, so
now we
need to explicitly emit a sfence.vma in flush_cache_vmap().

Note that we don't need to implement flush_cache_vunmap() as the
generic
code should emit a flush tlb after unmapping a vmalloc region.

Fixes 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for

vmalloc/modules area")

arch/riscv/include/asm/cacheflush.h +0 -0
index 8091b8bf4883..b93ffddf8a61 100644

arch/riscv/include/asm/cacheflush.h static inline void flush_dcache_page(struct page *page) · line 42

Quoted 6 lines
arch/riscv/include/asm/cacheflush.h +3 -0
@@ -37,6 +37,10 @@ static inline void flush_dcache_page(struct page *page)
  #define flush_icache_user_page(vma, pg, addr, len) \
         flush_icache_mm(vma->vm_mm, 0)
 
+#ifdef CONFIG_64BIT
+#define flush_cache_vmap(start, end) flush_tlb_kernel_range(start, end)
+#endif

I don't want that, and flush_tlb_kernel_range is flush_tlb_all. In addition, it would call IPI, which is a performance killer.

What's the problem of spurious fault replay? It only costs a local_tlb_flush with vaddr.

Quoted 11 lines
arch/riscv/include/asm/cacheflush.h +1 -0
+
  #ifndef CONFIG_SMP
 
  #define flush_icache_all() local_flush_icache_all()

Let me know if that works for you!

Quoted 2 lines

--
Best Regards
Guo Ren

#5 lore raw
Quoted 84 lines · Guo Ren
Quoted 83 lines · Alexandre Ghiti
Quoted 43 lines · Guo Ren
Quoted 18 lines · Alexandre Ghiti
Quoted 17 lines · On 21/07/2023 16:51, [email protected] wrote:

From: Guo Ren <[email protected]>

RISC-V specification permits the caching of PTEs whose V (Valid) bit is clear. Operating systems must be written to cope with this possibility, but implementers are reminded that eagerly caching invalid PTEs will reduce performance by causing additional page faults.

So we must keep vmalloc_fault for the spurious page faults of kernel virtual address from an OoO machine.

---
    arch/riscv/mm/fault.c | 3 +--
    1 file changed, 1 insertion(+), 2 deletions(-)
Quoted 15 lines
Quoted 5 lines
arch/riscv/mm/fault.c +0 -0
index 85165fe438d8..f662c9eae7d4 100644

arch/riscv/mm/fault.c void handle_page_fault(struct pt_regs *regs) · line 264

Quoted 9 lines
arch/riscv/mm/fault.c +1 -2
@@ -258,8 +258,7 @@ void handle_page_fault(struct pt_regs *regs)
         * only copy the information from the master page table,
         * nothing more.
         */
-     if ((!IS_ENABLED(CONFIG_MMU) || !IS_ENABLED(CONFIG_64BIT)) &&
-         unlikely(addr >= VMALLOC_START && addr < VMALLOC_END)) {
+     if (unlikely(addr >= TASK_SIZE)) {
                vmalloc_fault(regs, code, addr);
                return;
        }

Can you share what you are trying to fix here?

We met a spurious page fault panic on an OoO machine.

1. The processor speculative execution brings the V=0 entries into the TLB in the kernel virtual address.
2. Linux kernel installs the kernel virtual address with the page, and V=1
3. When kernel code access the kernel virtual address, it would raise a page fault as the V=0 entry in the tlb.
4. No vmalloc_fault, then panic.

Quoted 4 lines
Quoted 3 lines

I have a fix (that's currently running our CI) for commit 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") that implements flush_cache_vmap() since we lost the vmalloc_fault.

Could you share that patch?

Here we go:

Author: Alexandre Ghiti <[email protected]>
Date:   Fri Jul 21 08:43:44 2023 +0000

riscv: Implement flush_cache_vmap()

The RISC-V kernel needs a sfence.vma after a page table
modification: we
used to rely on the vmalloc fault handling to emit an sfence.vma, but commit 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") got rid of this path for 64-bit kernels, so
now we
need to explicitly emit a sfence.vma in flush_cache_vmap().

Note that we don't need to implement flush_cache_vunmap() as the
generic
code should emit a flush tlb after unmapping a vmalloc region.

Fixes 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for

vmalloc/modules area")

arch/riscv/include/asm/cacheflush.h +0 -0
index 8091b8bf4883..b93ffddf8a61 100644
@@ -37,6 +37,10 @@ static inline void flush_dcache_page(struct page *page)
   #define flush_icache_user_page(vma, pg, addr, len) \
          flush_icache_mm(vma->vm_mm, 0)
 
Quoted 5 lines

arch/riscv/include/asm/cacheflush.h static inline void flush_dcache_page(struct page *page) · line 42

Quoted 3 lines
arch/riscv/include/asm/cacheflush.h +3 -0
+#ifdef CONFIG_64BIT
+#define flush_cache_vmap(start, end) flush_tlb_kernel_range(start, end)
+#endif

I don't want that, and flush_tlb_kernel_range is flush_tlb_all. In addition, it would call IPI, which is a performance killer.

At the moment, flush_tlb_kernel_range() indeed calls flush_tlb_all() but that needs to be fixed, see my last patchset https://lore.kernel.org/linux-riscv/[email protected]/.

But can you at least check that this fixes your issue? It would be interesting to see if the problem comes from vmalloc or something else.

Quoted 2 lines

What's the problem of spurious fault replay? It only costs a local_tlb_flush with vaddr.

We had this exact discussion internally this week, and the fault replay seems like a solution. But that needs to be thought carefully: the vmalloc fault was removed for a reason (see Bjorn commit log), tracing functions can use vmalloc() in the path of the vmalloc fault, causing an infinite trap loop. And here you are simply re-enabling this problem. In addition, this patch makes vmalloc_fault() catch *all* kernel faults in the kernel address space, so any genuine kernel fault would loop forever in vmalloc_fault().

For now, the simplest solution is to implement flush_cache_vmap() because riscv needs a sfence.vma when adding a new mapping, and if that's a "performance killer", let's measure that and implement something like this patch is trying to do. I may be wrong, but there aren't many new kernel mappings that would require a call to flush_cache_vmap() so I disagree with the performance killer argument, but happy to be proven otherwise!

Thanks,

Alex

Quoted 14 lines
Quoted 11 lines
arch/riscv/include/asm/cacheflush.h +1 -0
+
   #ifndef CONFIG_SMP
 
   #define flush_icache_all() local_flush_icache_all()

Let me know if that works for you!

Quoted 1 line

--
Best Regards
Guo Ren

author
#6 lore raw

Sorry for the late reply, Alexandre. I'm busy with other suffs.

Quoted 94 lines · Alexandre Ghiti
Quoted 84 lines · Guo Ren
Quoted 83 lines · Alexandre Ghiti
Quoted 43 lines · Guo Ren
Quoted 18 lines · Alexandre Ghiti
Quoted 17 lines · On 21/07/2023 16:51, [email protected] wrote:

From: Guo Ren <[email protected]>

RISC-V specification permits the caching of PTEs whose V (Valid) bit is clear. Operating systems must be written to cope with this possibility, but implementers are reminded that eagerly caching invalid PTEs will reduce performance by causing additional page faults.

So we must keep vmalloc_fault for the spurious page faults of kernel virtual address from an OoO machine.

---
    arch/riscv/mm/fault.c | 3 +--
    1 file changed, 1 insertion(+), 2 deletions(-)
Quoted 15 lines
Quoted 5 lines
arch/riscv/mm/fault.c +0 -0
index 85165fe438d8..f662c9eae7d4 100644

arch/riscv/mm/fault.c void handle_page_fault(struct pt_regs *regs) · line 264

Quoted 9 lines
arch/riscv/mm/fault.c +1 -2
@@ -258,8 +258,7 @@ void handle_page_fault(struct pt_regs *regs)
         * only copy the information from the master page table,
         * nothing more.
         */
-     if ((!IS_ENABLED(CONFIG_MMU) || !IS_ENABLED(CONFIG_64BIT)) &&
-         unlikely(addr >= VMALLOC_START && addr < VMALLOC_END)) {
+     if (unlikely(addr >= TASK_SIZE)) {
                vmalloc_fault(regs, code, addr);
                return;
        }

Can you share what you are trying to fix here?

We met a spurious page fault panic on an OoO machine.

1. The processor speculative execution brings the V=0 entries into the TLB in the kernel virtual address.
2. Linux kernel installs the kernel virtual address with the page, and V=1
3. When kernel code access the kernel virtual address, it would raise a page fault as the V=0 entry in the tlb.
4. No vmalloc_fault, then panic.

Quoted 4 lines
Quoted 3 lines

I have a fix (that's currently running our CI) for commit 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") that implements flush_cache_vmap() since we lost the vmalloc_fault.

Could you share that patch?

Here we go:

Author: Alexandre Ghiti <[email protected]>
Date:   Fri Jul 21 08:43:44 2023 +0000

riscv: Implement flush_cache_vmap()

The RISC-V kernel needs a sfence.vma after a page table
modification: we
used to rely on the vmalloc fault handling to emit an sfence.vma, but commit 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") got rid of this path for 64-bit kernels, so
now we
need to explicitly emit a sfence.vma in flush_cache_vmap().

Note that we don't need to implement flush_cache_vunmap() as the
generic
code should emit a flush tlb after unmapping a vmalloc region.

Fixes 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for

vmalloc/modules area")

arch/riscv/include/asm/cacheflush.h +0 -0
index 8091b8bf4883..b93ffddf8a61 100644
@@ -37,6 +37,10 @@ static inline void flush_dcache_page(struct page *page)
   #define flush_icache_user_page(vma, pg, addr, len) \
          flush_icache_mm(vma->vm_mm, 0)
 
Quoted 5 lines

arch/riscv/include/asm/cacheflush.h static inline void flush_dcache_page(struct page *page) · line 42

Quoted 3 lines
arch/riscv/include/asm/cacheflush.h +3 -0
+#ifdef CONFIG_64BIT
+#define flush_cache_vmap(start, end) flush_tlb_kernel_range(start, end)
+#endif

I don't want that, and flush_tlb_kernel_range is flush_tlb_all. In addition, it would call IPI, which is a performance killer.

Quoted 6 lines

At the moment, flush_tlb_kernel_range() indeed calls flush_tlb_all() but that needs to be fixed, see my last patchset
https://lore.kernel.org/linux-riscv/[email protected]/.

But can you at least check that this fixes your issue? It would be interesting to see if the problem comes from vmalloc or something else.

It could solve my issue.

Quoted 11 lines
Quoted 4 lines

What's the problem of spurious fault replay? It only costs a local_tlb_flush with vaddr.

We had this exact discussion internally this week, and the fault replay seems like a solution. But that needs to be thought carefully: the vmalloc fault was removed for a reason (see Bjorn commit log), tracing functions can use vmalloc() in the path of the vmalloc fault, causing an infinite trap loop. And here you are simply re-enabling this problem.

Thx for mentioning it, and I will solve it in the next version of the patch:

-static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long addr)
+static void notrace vmalloc_fault(struct pt_regs *regs, int code, unsigned long addr)

Quoted 4 lines

In
addition, this patch makes vmalloc_fault() catch *all* kernel faults in the kernel address space, so any genuine kernel fault would loop forever in vmalloc_fault().

We check whether kernel vaddr is valid by the page_table, not range. I'm sure "the any genuine kernel fault would loop forever in vmalloc_fault()" is about what? Could you give an example?

Quoted 4 lines

For now, the simplest solution is to implement flush_cache_vmap() because riscv needs a sfence.vma when adding a new mapping, and if

It's not a local_tlb_flush, and it would ipi broadcast all harts. on_each_cpu(__ipi_flush_tlb_all, NULL, 1);

That's too horrible.

Some custom drivers or test codes may care about it.

Quoted 5 lines

that's a "performance killer", let's measure that and implement something like this patch is trying to do. I may be wrong, but there aren't many new kernel mappings that would require a call to flush_cache_vmap() so I disagree with the performance killer argument, but happy to be proven otherwise!

1. I agree to pre-allocate pgd entries. It's good for performance, but don't do that when Sv32.
2. We still need vmalloc_fault to match ISA spec requirements. (Some vendors' microarchitectures (e.g., T-HEAD c910) could prevent V=0 into TLB when PTW, then they don't need it.)
3. Only when vmalloc_fault can't solve the problem, then let's think about the flush_cache_vmap() solution.

Quoted 20 lines

Thanks,

Alex

Quoted 14 lines
Quoted 11 lines
arch/riscv/include/asm/cacheflush.h +1 -0
+
   #ifndef CONFIG_SMP
 
   #define flush_icache_all() local_flush_icache_all()

Let me know if that works for you!

Quoted 1 line

--
Best Regards
Guo Ren

--
Best Regards
Guo Ren

#7 lore raw
Quoted 110 lines · Guo Ren

Sorry for the late reply, Alexandre. I'm busy with other suffs.

Quoted 90 lines · Alexandre Ghiti
Quoted 82 lines · Guo Ren
Quoted 81 lines · Alexandre Ghiti
Quoted 43 lines · Guo Ren
Quoted 18 lines · Alexandre Ghiti
Quoted 17 lines · On 21/07/2023 16:51, [email protected] wrote:

From: Guo Ren <[email protected]>

RISC-V specification permits the caching of PTEs whose V (Valid) bit is clear. Operating systems must be written to cope with this possibility, but implementers are reminded that eagerly caching invalid PTEs will reduce performance by causing additional page faults.

So we must keep vmalloc_fault for the spurious page faults of kernel virtual address from an OoO machine.

---
     arch/riscv/mm/fault.c | 3 +--
     1 file changed, 1 insertion(+), 2 deletions(-)
Quoted 15 lines
Quoted 5 lines
arch/riscv/mm/fault.c +0 -0
index 85165fe438d8..f662c9eae7d4 100644

arch/riscv/mm/fault.c void handle_page_fault(struct pt_regs *regs) · line 264

Quoted 9 lines
arch/riscv/mm/fault.c +1 -2
@@ -258,8 +258,7 @@ void handle_page_fault(struct pt_regs *regs)
          * only copy the information from the master page table,
          * nothing more.
          */
-     if ((!IS_ENABLED(CONFIG_MMU) || !IS_ENABLED(CONFIG_64BIT)) &&
-         unlikely(addr >= VMALLOC_START && addr < VMALLOC_END)) {
+     if (unlikely(addr >= TASK_SIZE)) {
                 vmalloc_fault(regs, code, addr);
                 return;
         }

Can you share what you are trying to fix here?

We met a spurious page fault panic on an OoO machine.

1. The processor speculative execution brings the V=0 entries into the TLB in the kernel virtual address.
2. Linux kernel installs the kernel virtual address with the page, and V=1
3. When kernel code access the kernel virtual address, it would raise a page fault as the V=0 entry in the tlb.
4. No vmalloc_fault, then panic.

Quoted 4 lines
Quoted 3 lines

I have a fix (that's currently running our CI) for commit 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") that implements flush_cache_vmap() since we lost the vmalloc_fault.

Could you share that patch?

Here we go:

Author: Alexandre Ghiti <[email protected]>
Date:   Fri Jul 21 08:43:44 2023 +0000

riscv: Implement flush_cache_vmap()

The RISC-V kernel needs a sfence.vma after a page table
modification: we
used to rely on the vmalloc fault handling to emit an sfence.vma, but commit 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") got rid of this path for 64-bit kernels, so
now we
need to explicitly emit a sfence.vma in flush_cache_vmap().

Note that we don't need to implement flush_cache_vunmap() as the
generic
code should emit a flush tlb after unmapping a vmalloc region.

Fixes 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for

vmalloc/modules area")

arch/riscv/include/asm/cacheflush.h +0 -0
index 8091b8bf4883..b93ffddf8a61 100644
@@ -37,6 +37,10 @@ static inline void flush_dcache_page(struct page *page)
    #define flush_icache_user_page(vma, pg, addr, len) \
           flush_icache_mm(vma->vm_mm, 0)
 
Quoted 5 lines

arch/riscv/include/asm/cacheflush.h static inline void flush_dcache_page(struct page *page) · line 42

Quoted 3 lines
arch/riscv/include/asm/cacheflush.h +3 -0
+#ifdef CONFIG_64BIT
+#define flush_cache_vmap(start, end) flush_tlb_kernel_range(start, end)
+#endif

I don't want that, and flush_tlb_kernel_range is flush_tlb_all. In addition, it would call IPI, which is a performance killer.

Quoted 6 lines

At the moment, flush_tlb_kernel_range() indeed calls flush_tlb_all() but that needs to be fixed, see my last patchset
https://lore.kernel.org/linux-riscv/[email protected]/.

But can you at least check that this fixes your issue? It would be interesting to see if the problem comes from vmalloc or something else.

It could solve my issue.

Quoted 9 lines
Quoted 3 lines

What's the problem of spurious fault replay? It only costs a local_tlb_flush with vaddr.

We had this exact discussion internally this week, and the fault replay seems like a solution. But that needs to be thought carefully: the vmalloc fault was removed for a reason (see Bjorn commit log), tracing functions can use vmalloc() in the path of the vmalloc fault, causing an infinite trap loop. And here you are simply re-enabling this problem.

Quoted 6 lines

Thx for mentioning it, and I will solve it in the next version of the patch:

-static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long addr)
+static void notrace vmalloc_fault(struct pt_regs *regs, int code, unsigned long addr)

That does not solve the problem, since it should be any function called from the trap entry to vmalloc_fault() that needs to be marked a notrace. But it's not only tracing, it's all code that could vmalloc() something in the page fault path before reaching vmalloc_fault().

I have been rethinking about that,  it is very unlikely but the problem could happen, even if the microarchitecture does not cache invalid entries (but it's way more likely to happen when invalid entries are cached in the TLB)...And the only way I can think of to get rid of it is a preventive sfence.vma.

Quoted 8 lines
Quoted 5 lines

In
addition, this patch makes vmalloc_fault() catch *all* kernel faults in the kernel address space, so any genuine kernel fault would loop forever in vmalloc_fault().

We check whether kernel vaddr is valid by the page_table, not range. I'm sure "the any genuine kernel fault would loop forever in vmalloc_fault()" is about what? Could you give an example?

In the patch you propose, vmalloc_fault() would intercept all faults triggered on kernel addresses, even protection traps. vmalloc_fault() only checks the presence of the faulting entry in the kernel page table, not its permissions. So any protection trap on kernel addresses would loop forever in vmalloc_fault().

Quoted 10 lines
Quoted 4 lines

For now, the simplest solution is to implement flush_cache_vmap() because riscv needs a sfence.vma when adding a new mapping, and if

It's not a local_tlb_flush, and it would ipi broadcast all harts. on_each_cpu(__ipi_flush_tlb_all, NULL, 1);

That's too horrible.

Some custom drivers or test codes may care about it.

Yes, and there is vmap stack too...

Quoted 8 lines
Quoted 6 lines

that's a "performance killer", let's measure that and implement something like this patch is trying to do. I may be wrong, but there aren't many new kernel mappings that would require a call to flush_cache_vmap() so I disagree with the performance killer argument, but happy to be proven otherwise!

1. I agree to pre-allocate pgd entries. It's good for performance, but don't do that when Sv32.

Nothing changes for sv32, vmalloc_fault() is still there (but to me, there is still the problem with vmalloc() done before reaching vmalloc_fault()).

Quoted 3 lines

2. We still need vmalloc_fault to match ISA spec requirements. (Some vendors' microarchitectures (e.g., T-HEAD c910) could prevent V=0 into TLB when PTW, then they don't need it.)

I disagree, even if invalid entries are not cached in the TLB, there still exists a small window where we could take a trap on a newly created vmalloc mapping which would require vmalloc_fault(). Or to prevent that, we need flush_cache_vmap().

Quoted 2 lines

3. Only when vmalloc_fault can't solve the problem, then let's think about the flush_cache_vmap() solution.

To me, vmalloc_fault() can't solve the problem, even when invalid entries are not cached in the TLB. So I agree with you that flush_cache_vmap() is not that great, but I don't see any other straightforward solution here. Maybe a very early sfence.vma in the page fault path, before anything gets called, but we need a solution for 6.5.

Quoted 23 lines
Quoted 18 lines

Thanks,

Alex

Quoted 12 lines
Quoted 9 lines
arch/riscv/include/asm/cacheflush.h +1 -0
+
    #ifndef CONFIG_SMP
 
    #define flush_icache_all() local_flush_icache_all()

Let me know if that works for you!

--
Best Regards
Guo Ren

--
Best Regards
Guo Ren