riscv: mm: Fixup spurious fault of kernel vaddr
Subsystems (2)
PATCH 1/1 riscv: mm: Fixup spurious fault of kernel vaddr
From: Guo Ren <[email protected]>
RISC-V specification permits the caching of PTEs whose V (Valid) bit is clear. Operating systems must be written to cope with this possibility, but implementers are reminded that eagerly caching invalid PTEs will reduce performance by causing additional page faults.
So we must keep vmalloc_fault for the spurious page faults of kernel virtual address from an OoO machine.
--- arch/riscv/mm/fault.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/riscv/mm/fault.c +1 -2
@@ -258,8 +258,7 @@ void handle_page_fault(struct pt_regs *regs)* only copy the information from the master page table, * nothing more. */ -if ((!IS_ENABLED (CONFIG_MMU ) || !IS_ENABLED (CONFIG_64BIT )) && -unlikely (addr >=VMALLOC_START && addr <VMALLOC_END )) { +if (unlikely (addr >=TASK_SIZE )) {vmalloc_fault (regs, code, addr);return ; }
Signature · 1 line
2.36.1
Alexandre Ghiti
reviewerQuoted 22 lines · On 21/07/2023 16:51, [email protected] wrote:
From: Guo Ren <[email protected]>
RISC-V specification permits the caching of PTEs whose V (Valid) bit is clear. Operating systems must be written to cope with this possibility, but implementers are reminded that eagerly caching invalid PTEs will reduce performance by causing additional page faults.
So we must keep
vmalloc_faultfor the spurious page faults of kernel virtual address from an OoO machine.Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>--- arch/riscv/mm/fault.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/riscv/mm/fault.c +0
-0
arch/riscv/mm/fault.c void handle_page_fault(struct pt_regs *regs) · line 264
Quoted 9 lines
arch/riscv/mm/fault.c +1
-2@@ -258,8 +258,7 @@ void handle_page_fault(struct pt_regs *regs)* only copy the information from the master page table, * nothing more. */ -if ((!IS_ENABLED (CONFIG_MMU ) || !IS_ENABLED (CONFIG_64BIT )) && -unlikely (addr >=VMALLOC_START && addr <VMALLOC_END )) { +if (unlikely (addr >=TASK_SIZE )) {vmalloc_fault (regs, code, addr);return ; }
Can you share what you are trying to fix here?
I have a fix (that's currently running our CI) for commit 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") that implements flush_cache_vmap() since we lost the vmalloc_fault.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 29 lines · Alexandre Ghiti
Quoted 26 lines · On 21/07/2023 16:51, [email protected] wrote:
From: Guo Ren <[email protected]>
RISC-V specification permits the caching of PTEs whose V (Valid) bit is clear. Operating systems must be written to cope with this possibility, but implementers are reminded that eagerly caching invalid PTEs will reduce performance by causing additional page faults.
So we must keep
vmalloc_faultfor the spurious page faults of kernel virtual address from an OoO machine.Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>--- arch/riscv/mm/fault.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/riscv/mm/fault.c +0
-1@@ -258,8 +258,7 @@ void handle_page_fault(struct pt_regs *regs)* only copy the information from the master page table, * nothing more. */ - if ((!IS_ENABLED(CONFIG_MMU) || !IS_ENABLED(CONFIG_64BIT)) &&
Quoted 8 lines
arch/riscv/mm/fault.c void handle_page_fault(struct pt_regs *regs) · line 264
Quoted 5 lines
arch/riscv/mm/fault.c +1
-1- unlikely(addr >= VMALLOC_START && addr < VMALLOC_END)) { +if (unlikely (addr >=TASK_SIZE )) {vmalloc_fault (regs, code, addr);return ; }Can you share what you are trying to fix here?
We met a spurious page fault panic on an OoO machine.
1. The processor speculative execution brings the V=0 entries into the TLB in the kernel virtual address.
2. Linux kernel installs the kernel virtual address with the page, and V=1
3. When kernel code access the kernel virtual address, it would raise a page fault as the V=0 entry in the tlb.
4. No vmalloc_fault, then panic.
Quoted 4 lines
I have a fix (that's currently running our CI) for commit
7d3332be011e("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") that implementsflush_cache_vmap()since we lost thevmalloc_fault.
Could you share that patch?
Quoted 1 line
Signature · 2 lines
Best Regards Guo Ren
Alexandre Ghiti
reviewerQuoted 45 lines · Guo Ren
Quoted 19 lines · Alexandre Ghiti
Quoted 17 lines · On 21/07/2023 16:51, [email protected] wrote:
From: Guo Ren <[email protected]>
RISC-V specification permits the caching of PTEs whose V (Valid) bit is clear. Operating systems must be written to cope with this possibility, but implementers are reminded that eagerly caching invalid PTEs will reduce performance by causing additional page faults.
So we must keep
vmalloc_faultfor the spurious page faults of kernel virtual address from an OoO machine.Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>--- arch/riscv/mm/fault.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)Quoted 16 lines
Quoted 5 lines
arch/riscv/mm/fault.c +0
-0arch/riscv/mm/fault.c void handle_page_fault(struct pt_regs *regs) · line 264
Quoted 9 lines
arch/riscv/mm/fault.c +1
-2@@ -258,8 +258,7 @@ void handle_page_fault(struct pt_regs *regs)* only copy the information from the master page table, * nothing more. */ -if ((!IS_ENABLED (CONFIG_MMU ) || !IS_ENABLED (CONFIG_64BIT )) && -unlikely (addr >=VMALLOC_START && addr <VMALLOC_END )) { +if (unlikely (addr >=TASK_SIZE )) {vmalloc_fault (regs, code, addr);return ; }Can you share what you are trying to fix here?
We met a spurious page fault panic on an OoO machine.
1. The processor speculative execution brings the V=0 entries into the TLB in the kernel virtual address.
2. Linux kernel installs the kernel virtual address with the page, and V=1
3. When kernel code access the kernel virtual address, it would raise a page fault as the V=0 entry in the tlb.
4. Novmalloc_fault, then panic.
Quoted 4 lines
Quoted 3 lines
I have a fix (that's currently running our CI) for commit
7d3332be011e("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") that implementsflush_cache_vmap()since we lost thevmalloc_fault.Could you share that patch?
Here we go:
Author: Alexandre Ghiti <[email protected]>
Date: Fri Jul 21 08:43:44 2023 +0000
riscv: Implement flush_cache_vmap()
The RISC-V kernel needs a sfence.vma after a page table modification: we used to rely on the vmalloc fault handling to emit an sfence.vma, but commit 7d3332be011e ("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") got rid of this path for 64-bit kernels, so now we need to explicitly emit a sfence.vma in flush_cache_vmap().
Note that we don't need to implement flush_cache_vunmap() as the generic code should emit a flush tlb after unmapping a vmalloc region.
arch/riscv/include/asm/cacheflush.h +0 -0
@@ -37,6 +37,10 @@ static inline void flush_dcache_page(struct page *page)
#define flush_icache_user_page(vma, pg, addr, len) \
flush_icache_mm(vma->vm_mm, 0)
+#ifdef CONFIG_64BIT
+#define flush_cache_vmap(start, end) flush_tlb_kernel_range(start, end) +#endif
+
#ifndef CONFIG_SMP
#define flush_icache_all() local_flush_icache_all()
Let me know if that works for you!
Quoted 2 lines
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 84 lines · Alexandre Ghiti
Quoted 45 lines · Guo Ren
Quoted 19 lines · Alexandre Ghiti
Quoted 17 lines · On 21/07/2023 16:51, [email protected] wrote:
From: Guo Ren <[email protected]>
RISC-V specification permits the caching of PTEs whose V (Valid) bit is clear. Operating systems must be written to cope with this possibility, but implementers are reminded that eagerly caching invalid PTEs will reduce performance by causing additional page faults.
So we must keep
vmalloc_faultfor the spurious page faults of kernel virtual address from an OoO machine.Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>--- arch/riscv/mm/fault.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)Quoted 16 lines
Quoted 5 lines
arch/riscv/mm/fault.c +0
-0arch/riscv/mm/fault.c void handle_page_fault(struct pt_regs *regs) · line 264
Quoted 9 lines
arch/riscv/mm/fault.c +1
-2@@ -258,8 +258,7 @@ void handle_page_fault(struct pt_regs *regs)* only copy the information from the master page table, * nothing more. */ -if ((!IS_ENABLED (CONFIG_MMU ) || !IS_ENABLED (CONFIG_64BIT )) && -unlikely (addr >=VMALLOC_START && addr <VMALLOC_END )) { +if (unlikely (addr >=TASK_SIZE )) {vmalloc_fault (regs, code, addr);return ; }Can you share what you are trying to fix here?
We met a spurious page fault panic on an OoO machine.
1. The processor speculative execution brings the V=0 entries into the TLB in the kernel virtual address.
2. Linux kernel installs the kernel virtual address with the page, and V=1
3. When kernel code access the kernel virtual address, it would raise a page fault as the V=0 entry in the tlb.
4. Novmalloc_fault, then panic.Quoted 4 lines
Quoted 3 lines
I have a fix (that's currently running our CI) for commit
7d3332be011e("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") that implementsflush_cache_vmap()since we lost thevmalloc_fault.Could you share that patch?
Here we go:
Author: Alexandre Ghiti <[email protected]> Date: Fri Jul 21 08:43:44 2023 +0000riscv: Implement
flush_cache_vmap()The RISC-V kernel needs a sfence.vma after a page table
modification: we
used to rely on the vmalloc fault handling to emit an sfence.vma, but commit7d3332be011e("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") got rid of this path for 64-bit kernels, so
now we
need to explicitly emit a sfence.vma inflush_cache_vmap().Note that we don't need to implement
flush_cache_vunmap()as the
generic
code should emit a flush tlb after unmapping a vmalloc region.Fixes 7d3332be011e ("riscv: mm: Pre-allocate PGD entries forvmalloc/modules area")
Signed-off-by Alexandre Ghiti <[email protected]>arch/riscv/include/asm/cacheflush.h +0
-0
arch/riscv/include/asm/cacheflush.h static inline void flush_dcache_page(struct page *page) · line 42
Quoted 6 lines
arch/riscv/include/asm/cacheflush.h +3
-0@@ -37,6 +37,10 @@ static inline void flush_dcache_page(struct page *page)#define flush_icache_user_page (vma, pg, addr, len) \ flush_icache_mm(vma->vm_mm, 0) +#ifdef CONFIG_64BIT +#define flush_cache_vmap (start, end) flush_tlb_kernel_range(start, end) +#endif
I don't want that, and flush_tlb_kernel_range is flush_tlb_all. In addition, it would call IPI, which is a performance killer.
What's the problem of spurious fault replay? It only costs a local_tlb_flush with vaddr.
Quoted 11 lines
arch/riscv/include/asm/cacheflush.h +1
-0+#ifndef CONFIG_SMP #define flush_icache_all () local_flush_icache_all()Let me know if that works for you!
Quoted 2 lines
--
Best Regards
Guo Ren
Alexandre Ghiti
reviewerQuoted 84 lines · Guo Ren
Quoted 83 lines · Alexandre Ghiti
Quoted 43 lines · Guo Ren
Quoted 18 lines · Alexandre Ghiti
Quoted 17 lines · On 21/07/2023 16:51, [email protected] wrote:
From: Guo Ren <[email protected]>
RISC-V specification permits the caching of PTEs whose V (Valid) bit is clear. Operating systems must be written to cope with this possibility, but implementers are reminded that eagerly caching invalid PTEs will reduce performance by causing additional page faults.
So we must keep
vmalloc_faultfor the spurious page faults of kernel virtual address from an OoO machine.Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>--- arch/riscv/mm/fault.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)Quoted 15 lines
Quoted 5 lines
arch/riscv/mm/fault.c +0
-0arch/riscv/mm/fault.c void handle_page_fault(struct pt_regs *regs) · line 264
Quoted 9 lines
arch/riscv/mm/fault.c +1
-2@@ -258,8 +258,7 @@ void handle_page_fault(struct pt_regs *regs)* only copy the information from the master page table, * nothing more. */ -if ((!IS_ENABLED (CONFIG_MMU ) || !IS_ENABLED (CONFIG_64BIT )) && -unlikely (addr >=VMALLOC_START && addr <VMALLOC_END )) { +if (unlikely (addr >=TASK_SIZE )) {vmalloc_fault (regs, code, addr);return ; }Can you share what you are trying to fix here?
We met a spurious page fault panic on an OoO machine.
1. The processor speculative execution brings the V=0 entries into the TLB in the kernel virtual address.
2. Linux kernel installs the kernel virtual address with the page, and V=1
3. When kernel code access the kernel virtual address, it would raise a page fault as the V=0 entry in the tlb.
4. Novmalloc_fault, then panic.Quoted 4 lines
Quoted 3 lines
I have a fix (that's currently running our CI) for commit
7d3332be011e("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") that implementsflush_cache_vmap()since we lost thevmalloc_fault.Could you share that patch?
Here we go:
Author: Alexandre Ghiti <[email protected]> Date: Fri Jul 21 08:43:44 2023 +0000riscv: Implement
flush_cache_vmap()The RISC-V kernel needs a sfence.vma after a page table
modification: we
used to rely on the vmalloc fault handling to emit an sfence.vma, but commit7d3332be011e("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") got rid of this path for 64-bit kernels, so
now we
need to explicitly emit a sfence.vma inflush_cache_vmap().Note that we don't need to implement
flush_cache_vunmap()as the
generic
code should emit a flush tlb after unmapping a vmalloc region.Fixes 7d3332be011e ("riscv: mm: Pre-allocate PGD entries forvmalloc/modules area")
Signed-off-by Alexandre Ghiti <[email protected]>arch/riscv/include/asm/cacheflush.h +0
-0@@ -37,6 +37,10 @@ static inline void flush_dcache_page(struct page *page)#define flush_icache_user_page (vma, pg, addr, len) \ flush_icache_mm(vma->vm_mm, 0)
Quoted 5 lines
arch/riscv/include/asm/cacheflush.h static inline void flush_dcache_page(struct page *page) · line 42
Quoted 3 lines
arch/riscv/include/asm/cacheflush.h +3
-0+#ifdef CONFIG_64BIT +#define flush_cache_vmap (start, end) flush_tlb_kernel_range(start, end) +#endif I don't want that, and
flush_tlb_kernel_rangeisflush_tlb_all. In addition, it would call IPI, which is a performance killer.
At the moment, flush_tlb_kernel_range() indeed calls flush_tlb_all() but that needs to be fixed, see my last patchset https://lore.kernel.org/linux-riscv/[email protected]/.
But can you at least check that this fixes your issue? It would be interesting to see if the problem comes from vmalloc or something else.
Quoted 2 lines
What's the problem of spurious fault replay? It only costs a
local_tlb_flushwith vaddr.
We had this exact discussion internally this week, and the fault replay seems like a solution. But that needs to be thought carefully: the vmalloc fault was removed for a reason (see Bjorn commit log), tracing functions can use vmalloc() in the path of the vmalloc fault, causing an infinite trap loop. And here you are simply re-enabling this problem. In addition, this patch makes vmalloc_fault() catch *all* kernel faults in the kernel address space, so any genuine kernel fault would loop forever in vmalloc_fault().
For now, the simplest solution is to implement flush_cache_vmap() because riscv needs a sfence.vma when adding a new mapping, and if that's a "performance killer", let's measure that and implement something like this patch is trying to do. I may be wrong, but there aren't many new kernel mappings that would require a call to flush_cache_vmap() so I disagree with the performance killer argument, but happy to be proven otherwise!
Thanks,
Alex
Quoted 14 lines
Quoted 11 lines
arch/riscv/include/asm/cacheflush.h +1
-0+#ifndef CONFIG_SMP #define flush_icache_all () local_flush_icache_all()Let me know if that works for you!
Quoted 1 line
--
Best Regards
Guo Ren
Sorry for the late reply, Alexandre. I'm busy with other suffs.
Quoted 94 lines · Alexandre Ghiti
Quoted 84 lines · Guo Ren
Quoted 83 lines · Alexandre Ghiti
Quoted 43 lines · Guo Ren
Quoted 18 lines · Alexandre Ghiti
Quoted 17 lines · On 21/07/2023 16:51, [email protected] wrote:
From: Guo Ren <[email protected]>
RISC-V specification permits the caching of PTEs whose V (Valid) bit is clear. Operating systems must be written to cope with this possibility, but implementers are reminded that eagerly caching invalid PTEs will reduce performance by causing additional page faults.
So we must keep
vmalloc_faultfor the spurious page faults of kernel virtual address from an OoO machine.Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>--- arch/riscv/mm/fault.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)Quoted 15 lines
Quoted 5 lines
arch/riscv/mm/fault.c +0
-0arch/riscv/mm/fault.c void handle_page_fault(struct pt_regs *regs) · line 264
Quoted 9 lines
arch/riscv/mm/fault.c +1
-2@@ -258,8 +258,7 @@ void handle_page_fault(struct pt_regs *regs)* only copy the information from the master page table, * nothing more. */ -if ((!IS_ENABLED (CONFIG_MMU ) || !IS_ENABLED (CONFIG_64BIT )) && -unlikely (addr >=VMALLOC_START && addr <VMALLOC_END )) { +if (unlikely (addr >=TASK_SIZE )) {vmalloc_fault (regs, code, addr);return ; }Can you share what you are trying to fix here?
We met a spurious page fault panic on an OoO machine.
1. The processor speculative execution brings the V=0 entries into the TLB in the kernel virtual address.
2. Linux kernel installs the kernel virtual address with the page, and V=1
3. When kernel code access the kernel virtual address, it would raise a page fault as the V=0 entry in the tlb.
4. Novmalloc_fault, then panic.Quoted 4 lines
Quoted 3 lines
I have a fix (that's currently running our CI) for commit
7d3332be011e("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") that implementsflush_cache_vmap()since we lost thevmalloc_fault.Could you share that patch?
Here we go:
Author: Alexandre Ghiti <[email protected]> Date: Fri Jul 21 08:43:44 2023 +0000riscv: Implement
flush_cache_vmap()The RISC-V kernel needs a sfence.vma after a page table
modification: we
used to rely on the vmalloc fault handling to emit an sfence.vma, but commit7d3332be011e("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") got rid of this path for 64-bit kernels, so
now we
need to explicitly emit a sfence.vma inflush_cache_vmap().Note that we don't need to implement
flush_cache_vunmap()as the
generic
code should emit a flush tlb after unmapping a vmalloc region.Fixes 7d3332be011e ("riscv: mm: Pre-allocate PGD entries forvmalloc/modules area")
Signed-off-by Alexandre Ghiti <[email protected]>arch/riscv/include/asm/cacheflush.h +0
-0@@ -37,6 +37,10 @@ static inline void flush_dcache_page(struct page *page)#define flush_icache_user_page (vma, pg, addr, len) \ flush_icache_mm(vma->vm_mm, 0)Quoted 5 lines
arch/riscv/include/asm/cacheflush.h static inline void flush_dcache_page(struct page *page) · line 42
Quoted 3 lines
arch/riscv/include/asm/cacheflush.h +3
-0+#ifdef CONFIG_64BIT +#define flush_cache_vmap (start, end) flush_tlb_kernel_range(start, end) +#endif I don't want that, and
flush_tlb_kernel_rangeisflush_tlb_all. In addition, it would call IPI, which is a performance killer.
Quoted 6 lines
At the moment,
flush_tlb_kernel_range()indeed callsflush_tlb_all()but that needs to be fixed, see my last patchset
https://lore.kernel.org/linux-riscv/[email protected]/.But can you at least check that this fixes your issue? It would be interesting to see if the problem comes from vmalloc or something else.
It could solve my issue.
Quoted 11 lines
Quoted 4 lines
What's the problem of spurious fault replay? It only costs a
local_tlb_flushwith vaddr.We had this exact discussion internally this week, and the fault replay seems like a solution. But that needs to be thought carefully: the vmalloc fault was removed for a reason (see Bjorn commit log), tracing functions can use
vmalloc()in the path of the vmalloc fault, causing an infinite trap loop. And here you are simply re-enabling this problem.
Thx for mentioning it, and I will solve it in the next version of the patch:
-static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long addr)
+static void notrace vmalloc_fault(struct pt_regs *regs, int code, unsigned long addr)
Quoted 4 lines
In
addition, this patch makesvmalloc_fault()catch *all* kernel faults in the kernel address space, so any genuine kernel fault would loop forever invmalloc_fault().
We check whether kernel vaddr is valid by the page_table, not range. I'm sure "the any genuine kernel fault would loop forever in vmalloc_fault()" is about what? Could you give an example?
Quoted 4 lines
For now, the simplest solution is to implement
flush_cache_vmap()because riscv needs a sfence.vma when adding a new mapping, and if
It's not a local_tlb_flush, and it would ipi broadcast all harts. on_each_cpu(__ipi_flush_tlb_all, NULL, 1);
That's too horrible.
Some custom drivers or test codes may care about it.
Quoted 5 lines
that's a "performance killer", let's measure that and implement something like this patch is trying to do. I may be wrong, but there aren't many new kernel mappings that would require a call to
flush_cache_vmap()so I disagree with the performance killer argument, but happy to be proven otherwise!
1. I agree to pre-allocate pgd entries. It's good for performance, but don't do that when Sv32.
2. We still need vmalloc_fault to match ISA spec requirements. (Some vendors' microarchitectures (e.g., T-HEAD c910) could prevent V=0 into TLB when PTW, then they don't need it.)
3. Only when vmalloc_fault can't solve the problem, then let's think about the flush_cache_vmap() solution.
Quoted 20 lines
Thanks,
Alex
Quoted 14 lines
Quoted 11 lines
arch/riscv/include/asm/cacheflush.h +1
-0+#ifndef CONFIG_SMP #define flush_icache_all () local_flush_icache_all()Let me know if that works for you!
Quoted 1 line
--
Best Regards
Guo Ren
--
Best Regards
Guo Ren
Alexandre Ghiti
reviewerQuoted 110 lines · Guo Ren
Sorry for the late reply, Alexandre. I'm busy with other suffs.
Quoted 90 lines · Alexandre Ghiti
Quoted 82 lines · Guo Ren
Quoted 81 lines · Alexandre Ghiti
Quoted 43 lines · Guo Ren
Quoted 18 lines · Alexandre Ghiti
Quoted 17 lines · On 21/07/2023 16:51, [email protected] wrote:
From: Guo Ren <[email protected]>
RISC-V specification permits the caching of PTEs whose V (Valid) bit is clear. Operating systems must be written to cope with this possibility, but implementers are reminded that eagerly caching invalid PTEs will reduce performance by causing additional page faults.
So we must keep
vmalloc_faultfor the spurious page faults of kernel virtual address from an OoO machine.Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>--- arch/riscv/mm/fault.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)Quoted 15 lines
Quoted 5 lines
arch/riscv/mm/fault.c +0
-0arch/riscv/mm/fault.c void handle_page_fault(struct pt_regs *regs) · line 264
Quoted 9 lines
arch/riscv/mm/fault.c +1
-2@@ -258,8 +258,7 @@ void handle_page_fault(struct pt_regs *regs)* only copy the information from the master page table, * nothing more. */ -if ((!IS_ENABLED (CONFIG_MMU ) || !IS_ENABLED (CONFIG_64BIT )) && -unlikely (addr >=VMALLOC_START && addr <VMALLOC_END )) { +if (unlikely (addr >=TASK_SIZE )) {vmalloc_fault (regs, code, addr);return ; }Can you share what you are trying to fix here?
We met a spurious page fault panic on an OoO machine.
1. The processor speculative execution brings the V=0 entries into the TLB in the kernel virtual address.
2. Linux kernel installs the kernel virtual address with the page, and V=1
3. When kernel code access the kernel virtual address, it would raise a page fault as the V=0 entry in the tlb.
4. Novmalloc_fault, then panic.Quoted 4 lines
Quoted 3 lines
I have a fix (that's currently running our CI) for commit
7d3332be011e("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") that implementsflush_cache_vmap()since we lost thevmalloc_fault.Could you share that patch?
Here we go:
Author: Alexandre Ghiti <[email protected]> Date: Fri Jul 21 08:43:44 2023 +0000riscv: Implement
flush_cache_vmap()The RISC-V kernel needs a sfence.vma after a page table
modification: we
used to rely on the vmalloc fault handling to emit an sfence.vma, but commit7d3332be011e("riscv: mm: Pre-allocate PGD entries for vmalloc/modules area") got rid of this path for 64-bit kernels, so
now we
need to explicitly emit a sfence.vma inflush_cache_vmap().Note that we don't need to implement
flush_cache_vunmap()as the
generic
code should emit a flush tlb after unmapping a vmalloc region.Fixes 7d3332be011e ("riscv: mm: Pre-allocate PGD entries forvmalloc/modules area")
Signed-off-by Alexandre Ghiti <[email protected]>arch/riscv/include/asm/cacheflush.h +0
-0@@ -37,6 +37,10 @@ static inline void flush_dcache_page(struct page *page)#define flush_icache_user_page (vma, pg, addr, len) \ flush_icache_mm(vma->vm_mm, 0)Quoted 5 lines
arch/riscv/include/asm/cacheflush.h static inline void flush_dcache_page(struct page *page) · line 42
Quoted 3 lines
arch/riscv/include/asm/cacheflush.h +3
-0+#ifdef CONFIG_64BIT +#define flush_cache_vmap (start, end) flush_tlb_kernel_range(start, end) +#endif I don't want that, and
flush_tlb_kernel_rangeisflush_tlb_all. In addition, it would call IPI, which is a performance killer.Quoted 6 lines
At the moment,
flush_tlb_kernel_range()indeed callsflush_tlb_all()but that needs to be fixed, see my last patchset
https://lore.kernel.org/linux-riscv/[email protected]/.But can you at least check that this fixes your issue? It would be interesting to see if the problem comes from vmalloc or something else.
It could solve my issue.
Quoted 9 lines
Quoted 3 lines
What's the problem of spurious fault replay? It only costs a
local_tlb_flushwith vaddr.We had this exact discussion internally this week, and the fault replay seems like a solution. But that needs to be thought carefully: the vmalloc fault was removed for a reason (see Bjorn commit log), tracing functions can use
vmalloc()in the path of the vmalloc fault, causing an infinite trap loop. And here you are simply re-enabling this problem.
Quoted 6 lines
Thx for mentioning it, and I will solve it in the next version of the patch:
-static inline void
vmalloc_fault(struct pt_regs *regs, int code, unsigned long addr)
+static void notracevmalloc_fault(struct pt_regs *regs, int code, unsigned long addr)
That does not solve the problem, since it should be any function called from the trap entry to vmalloc_fault() that needs to be marked a notrace. But it's not only tracing, it's all code that could vmalloc() something in the page fault path before reaching vmalloc_fault().
I have been rethinking about that, it is very unlikely but the problem could happen, even if the microarchitecture does not cache invalid entries (but it's way more likely to happen when invalid entries are cached in the TLB)...And the only way I can think of to get rid of it is a preventive sfence.vma.
Quoted 8 lines
Quoted 5 lines
In
addition, this patch makesvmalloc_fault()catch *all* kernel faults in the kernel address space, so any genuine kernel fault would loop forever invmalloc_fault().We check whether kernel vaddr is valid by the
page_table, not range. I'm sure "the any genuine kernel fault would loop forever invmalloc_fault()" is about what? Could you give an example?
In the patch you propose, vmalloc_fault() would intercept all faults triggered on kernel addresses, even protection traps. vmalloc_fault() only checks the presence of the faulting entry in the kernel page table, not its permissions. So any protection trap on kernel addresses would loop forever in vmalloc_fault().
Quoted 10 lines
Quoted 4 lines
For now, the simplest solution is to implement
flush_cache_vmap()because riscv needs a sfence.vma when adding a new mapping, and ifIt's not a
local_tlb_flush, and it would ipi broadcast all harts.on_each_cpu(__ipi_flush_tlb_all, NULL, 1);That's too horrible.
Some custom drivers or test codes may care about it.
Yes, and there is vmap stack too...
Quoted 8 lines
Quoted 6 lines
that's a "performance killer", let's measure that and implement something like this patch is trying to do. I may be wrong, but there aren't many new kernel mappings that would require a call to
flush_cache_vmap()so I disagree with the performance killer argument, but happy to be proven otherwise!1. I agree to pre-allocate pgd entries. It's good for performance, but don't do that when Sv32.
Nothing changes for sv32, vmalloc_fault() is still there (but to me, there is still the problem with vmalloc() done before reaching vmalloc_fault()).
Quoted 3 lines
2. We still need
vmalloc_faultto match ISA spec requirements. (Some vendors' microarchitectures (e.g., T-HEAD c910) could prevent V=0 into TLB when PTW, then they don't need it.)
I disagree, even if invalid entries are not cached in the TLB, there still exists a small window where we could take a trap on a newly created vmalloc mapping which would require vmalloc_fault(). Or to prevent that, we need flush_cache_vmap().
Quoted 2 lines
3. Only when
vmalloc_faultcan't solve the problem, then let's think about theflush_cache_vmap()solution.
To me, vmalloc_fault() can't solve the problem, even when invalid entries are not cached in the TLB. So I agree with you that flush_cache_vmap() is not that great, but I don't see any other straightforward solution here. Maybe a very early sfence.vma in the page fault path, before anything gets called, but we need a solution for 6.5.
Quoted 23 lines
Quoted 18 lines
Thanks,
Alex
Quoted 12 lines
Quoted 9 lines
arch/riscv/include/asm/cacheflush.h +1
-0+#ifndef CONFIG_SMP #define flush_icache_all () local_flush_icache_all()Let me know if that works for you!
--
Best Regards
Guo Ren--
Best Regards
Guo Ren