riscv: Fixup _PAGE_GLOBAL in _PAGE_KERNEL
PATCH 1/3 riscv: Fixup _PAGE_GLOBAL in _PAGE_KERNEL
From: Guo Ren <[email protected]>
Kernel virtual address translation should avoid care asid or it'll cause more TLB-miss and TLB-refill. Because the current asid in satp belongs to the current process, but the target kernel va TLB entry's asid still belongs to the previous process.
--- arch/riscv/include/asm/pgtable.h | 1 + 1 file changed, 1 insertion(+)
arch/riscv/include/asm/pgtable.h +1 -0
@@ -135,6 +135,7 @@ | _PAGE_PRESENT \ | _PAGE_ACCESSED \ | _PAGE_DIRTY \ + | _PAGE_GLOBAL \ | _PAGE_CACHE)#define PAGE_KERNEL __pgprot(_PAGE_KERNEL)
Signature · 1 line
2.7.4
PATCH 2/3 riscv: Fixup PAGE_UP in asm/page.h
From: Guo Ren <[email protected]>
Current PAGE_UP implementation is wrong. PAGE_UP(0) should be 0x1000, but current implementation will give out 0.
Although the current PAGE_UP isn't used, it will soon be used in tlb_flush optimization.
--- arch/riscv/include/asm/page.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/riscv/include/asm/page.h +1 -1
@@ -37,7 +37,7 @@#ifndef __ASSEMBLY__ -#define PAGE_UP (addr) (((addr)+((PAGE_SIZE)-1))&(~((PAGE_SIZE)-1))) +#define PAGE_UP (addr) (((addr)+(PAGE_SIZE))&(~((PAGE_SIZE)-1)))#define PAGE_DOWN (addr) ((addr)&(~((PAGE_SIZE)-1)))/* align addr on a size boundary - adjust address up/down if needed */
Signature · 1 line
2.7.4
Re: [PATCH 2/3] riscv: Fixup PAGE_UP in asm/page.h
Quoted 7 lines · On Mon, May 24, 2021 at 06:51:07AM +0000, [email protected] wrote:
From: Guo Ren <[email protected]>
Current
PAGE_UPimplementation is wrong.PAGE_UP(0)should be0x1000, but current implementation will give out 0.Although the current
PAGE_UPisn't used, it will soon be used intlb_flushoptimization.
Nak. Please just remove the PAGE_UP/PAGE_DOWN macros just like they have been removed from other architectures long ago and use the generic DIV_ROUND_UP macro for your new code like everyone else.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [PATCH 2/3] riscv: Fixup PAGE_UP in asm/page.h
Quoted 7 lines · Christoph Hellwig
Quoted 5 lines · On Mon, May 24, 2021 at 06:51:07AM +0000, [email protected] wrote:
From: Guo Ren <[email protected]>
Current
PAGE_UPimplementation is wrong.PAGE_UP(0)should be0x1000, but current implementation will give out 0.
Quoted 6 lines
Quoted 2 lines
Although the current
PAGE_UPisn't used, it will soon be used intlb_flushoptimization.Nak. Please just remove the PAGE_UP/PAGE_DOWN macros just like they have been removed from other architectures long ago and use the generic
DIV_ROUND_UPmacro for your new code like everyone else.
This patch has been dropped because it's wrong, ref Anup's reply.
Remove PAGE_UP/DOWN is okay for me. How about:
static inline void local_flush_tlb_range_asid(unsigned long start,
unsigned long size,
unsigned long asid)
{
- unsigned long page_add = PAGE_DOWN(start); - unsigned long page_end = PAGE_UP(start + size); + unsigned long page_add = _ALIGN_DOWN(start, PAGE_SIZE); + unsigned long page_end = _ALIGN_UP(start + size, PAGE_SIZE);
_ALIGN_XXX are also defined in arch/riscv/include/asm/page.h.
Signature · 4 lines
Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/
Re: [PATCH 2/3] riscv: Fixup PAGE_UP in asm/page.h
Quoted 27 lines · Guo Ren
Quoted 7 lines · Christoph Hellwig
Quoted 5 lines · On Mon, May 24, 2021 at 06:51:07AM +0000, [email protected] wrote:
From: Guo Ren <[email protected]>
Current
PAGE_UPimplementation is wrong.PAGE_UP(0)should be0x1000, but current implementation will give out 0.Quoted 6 lines
Quoted 2 lines
Although the current
PAGE_UPisn't used, it will soon be used intlb_flushoptimization.Nak. Please just remove the PAGE_UP/PAGE_DOWN macros just like they have been removed from other architectures long ago and use the generic
DIV_ROUND_UPmacro for your new code like everyone else.This patch has been dropped because it's wrong, ref Anup's reply.
Remove PAGE_UP/DOWN is okay for me. How about:
static inline void local_flush_tlb_range_asid(unsigned long start,
unsigned long size,
unsigned long asid)
{(quoted diff) +2
-2- unsigned long page_add = PAGE_DOWN(start); - unsigned long page_end = PAGE_UP(start + size); + unsigned long page_add = _ALIGN_DOWN(start, PAGE_SIZE); + unsigned long page_end = _ALIGN_UP(start + size, PAGE_SIZE);
Quoted 1 line
_ALIGN_XXXare also defined inarch/riscv/include/asm/page.h.
And these also are leftovers from days gone by and should be removed.
I think this should simply be:
unsigned long page_add = start & PAGE_MASK; unsigned long page_end = PAGE_ALIGN(start + size);
(and page_add is a pretty horrible name as well..)
PATCH 3/3 riscv: Use use_asid_allocator flush TLB
In reply to [email protected]From: Guo Ren <[email protected]>
Use static_branch_unlikely(&use_asid_allocator) to keep the origin tlb flush style, so it's no effect on the existing machine. Here are the optimized functions:
- flush_tlb_mm
- flush_tlb_page
- flush_tlb_range
All above are based on the below new implement functions:
- __sbi_tlb_flush_range_asid
- local_flush_tlb_range_asid
--- arch/riscv/include/asm/mmu_context.h | 2 ++ arch/riscv/include/asm/tlbflush.h | 22 ++++++++++++++++++++ arch/riscv/mm/context.c | 2 +- arch/riscv/mm/tlbflush.c | 40 +++++++++++++++++++++++++++++++++--- 4 files changed, 62 insertions(+), 4 deletions(-)
arch/riscv/include/asm/mmu_context.h +2 -0
@@ -33,6 +33,8 @@ static inline int init_new_context(struct task_struct *tsk, return 0; } +DECLARE_STATIC_KEY_FALSE (use_asid_allocator); +#include <asm-generic/mmu_context.h> #endif /* _ASM_RISCV_MMU_CONTEXT_H */
arch/riscv/include/asm/tlbflush.h +22 -0
@@ -8,6 +8,7 @@#define _ASM_RISCV_TLBFLUSH_H#include <linux/mm_types.h> +#include <asm/page.h> #include <asm/smp.h> #include <asm/errata_list.h>
@@ -22,9 +23,30 @@ static inline void local_flush_tlb_page(unsigned long addr) {ALT_FLUSH_TLB_PAGE (__asm__ __volatile__ ("sfence.vma %0" : :"r" (addr) :"memory" )); } + +static inline void local_flush_tlb_range_asid (unsigned long start,unsigned long size, +unsigned long asid) +{ +unsigned long page_add =PAGE_DOWN (start); +unsigned long page_end =PAGE_UP (start + size); + +if (size ==-1 ) { + __asm__ __volatile__ ("sfence.vma x0, %0" : :"r" (asid) :"memory" ); +return ; + } + +while (page_add < page_end) { + __asm__ __volatile__ ("sfence.vma %0, %1" + : + :"r" (page_add),"r" (asid) + :"memory" ); + page_add +=PAGE_SIZE ; + } +}#else /* CONFIG_MMU */ #define local_flush_tlb_all () do { } while (0)#define local_flush_tlb_page (addr) do { } while (0) +#define local_flush_tlb_range_asid (addr) do { } while (0)#endif /* CONFIG_MMU */ #if defined(CONFIG_SMP ) && defined(CONFIG_MMU )
arch/riscv/mm/context.c +1 -1
@@ -18,7 +18,7 @@#ifdef CONFIG_MMU -static DEFINE_STATIC_KEY_FALSE (use_asid_allocator); +DEFINE_STATIC_KEY_FALSE (use_asid_allocator);static unsigned long asid_bits;static unsigned long num_asids;
arch/riscv/mm/tlbflush.c +37 -3
@@ -4,6 +4,7 @@#include <linux/smp.h> #include <linux/sched.h> #include <asm/sbi.h> +#include <asm/mmu_context.h> void flush_tlb_all (void ) {
@@ -39,18 +40,51 @@ static void __sbi_tlb_flush_range(struct cpumask *cmask, unsigned long start,put_cpu (); } +static void __sbi_tlb_flush_range_asid (struct cpumask *cmask,unsigned long start, +unsigned long size,unsigned long asid) +{ +struct cpumask hmask; +unsigned int cpuid; + +if (cpumask_empty (cmask)) +return ; + + cpuid =get_cpu (); + +if (cpumask_any_but (cmask, cpuid) >= nr_cpu_ids) { +local_flush_tlb_range_asid (start, size, asid); + }else { +riscv_cpuid_to_hartid_mask (cmask, &hmask); +sbi_remote_sfence_vma_asid (cpumask_bits (&hmask), start, size, asid); + } + +put_cpu (); +} +void flush_tlb_mm (struct mm_struct *mm) { -__sbi_tlb_flush_range (mm_cpumask (mm),0 ,-1 ); +if (static_branch_unlikely (&use_asid_allocator)) +__sbi_tlb_flush_range_asid (mm_cpumask (mm),0 ,-1 , +atomic_long_read (&mm->context.id)); +else +__sbi_tlb_flush_range (mm_cpumask (mm),0 ,-1 ); }void flush_tlb_page (struct vm_area_struct *vma,unsigned long addr) { -__sbi_tlb_flush_range (mm_cpumask (vma->vm_mm), addr,PAGE_SIZE ); +if (static_branch_unlikely (&use_asid_allocator)) +__sbi_tlb_flush_range_asid (mm_cpumask (vma->vm_mm), addr,PAGE_SIZE , +atomic_long_read (&vma->vm_mm->context.id)); +else +__sbi_tlb_flush_range (mm_cpumask (vma->vm_mm), addr,PAGE_SIZE ); }void flush_tlb_range (struct vm_area_struct *vma,unsigned long start,unsigned long end) { -__sbi_tlb_flush_range (mm_cpumask (vma->vm_mm), start, end - start); +if (static_branch_unlikely (&use_asid_allocator)) +__sbi_tlb_flush_range_asid (mm_cpumask (vma->vm_mm), start, end - start, +atomic_long_read (&vma->vm_mm->context.id)); +else +__sbi_tlb_flush_range (mm_cpumask (vma->vm_mm), start, end - start); }
Signature · 1 line
2.7.4
Re: [PATCH 3/3] riscv: Use use_asid_allocator flush TLB
Quoted 50 lines · On Mon, May 24, 2021 at 12:22 PM <[email protected]> wrote:
From: Guo Ren <[email protected]>
Use
static_branch_unlikely(&use_asid_allocator)to keep the origin tlb flush style, so it's no effect on the existing machine. Here are the optimized functions:
-flush_tlb_mm
-flush_tlb_page
-flush_tlb_rangeAll above are based on the below new implement functions:
-__sbi_tlb_flush_range_asid
-local_flush_tlb_range_asidSigned-off-by Guo Ren <[email protected]>Cc Anup Patel <[email protected]>Cc Palmer Dabbelt <[email protected]>--- arch/riscv/include/asm/mmu_context.h | 2 ++ arch/riscv/include/asm/tlbflush.h | 22 ++++++++++++++++++++ arch/riscv/mm/context.c | 2 +- arch/riscv/mm/tlbflush.c | 40 +++++++++++++++++++++++++++++++++--- 4 files changed, 62 insertions(+), 4 deletions(-)arch/riscv/include/asm/mmu_context.h +2
-0@@ -33,6 +33,8 @@ static inline int init_new_context(struct task_struct *tsk, return 0; } +DECLARE_STATIC_KEY_FALSE (use_asid_allocator); +#include <asm-generic/mmu_context.h> #endif /* _ASM_RISCV_MMU_CONTEXT_H */ arch/riscv/include/asm/tlbflush.h +1
-0@@ -8,6 +8,7 @@#define _ASM_RISCV_TLBFLUSH_H#include <linux/mm_types.h> +#include <asm/page.h> #include <asm/smp.h> #include <asm/errata_list.h>
arch/riscv/include/asm/tlbflush.h static inline void local_flush_tlb_page(unsigned long addr) · line 31
Quoted 9 lines
arch/riscv/include/asm/tlbflush.h +6
-0@@ -22,9 +23,30 @@ static inline void local_flush_tlb_page(unsigned long addr) {ALT_FLUSH_TLB_PAGE (__asm__ __volatile__ ("sfence.vma %0" : :"r" (addr) :"memory" )); } + +static inline void local_flush_tlb_range_asid (unsigned long start,unsigned long size, +unsigned long asid) +{ +unsigned long page_add =PAGE_DOWN (start); +unsigned long page_end =PAGE_UP (start + size);
Your PATCH2 is not correct because PAGE_UP(x) should in-fact return 0 when x == 0.
In fact, if both "start" and "size" are zero then both page_add and page_end should be zero so that no "sfence.vma" is executed.
If you want at least one TLB entry to be invalidated when size == 0 then you can simply set "size = 1" when size is zero which will force one TLB invalidation.
Please drop your PATCH2 and the rest of the things look good to me.
Regards,
Anup
Quoted 104 lines
arch/riscv/include/asm/tlbflush.h +15
-0+ +if (size ==-1 ) { + __asm__ __volatile__ ("sfence.vma x0, %0" : :"r" (asid) :"memory" ); +return ; + } + +while (page_add < page_end) { + __asm__ __volatile__ ("sfence.vma %0, %1" + : + :"r" (page_add),"r" (asid) + :"memory" ); + page_add +=PAGE_SIZE ; + } +}#else /* CONFIG_MMU */ #define local_flush_tlb_all () do { } while (0)#define local_flush_tlb_page (addr) do { } while (0) +#define local_flush_tlb_range_asid (addr) do { } while (0)#endif /* CONFIG_MMU */ #if defined(CONFIG_SMP ) && defined(CONFIG_MMU )arch/riscv/mm/context.c +1
-1@@ -18,7 +18,7 @@#ifdef CONFIG_MMU -static DEFINE_STATIC_KEY_FALSE (use_asid_allocator); +DEFINE_STATIC_KEY_FALSE (use_asid_allocator);static unsigned long asid_bits;static unsigned long num_asids;arch/riscv/mm/tlbflush.c +37
-4@@ -4,6 +4,7 @@#include <linux/smp.h> #include <linux/sched.h> #include <asm/sbi.h> +#include <asm/mmu_context.h> void flush_tlb_all (void ) {@@ -39,18 +40,51 @@ static void __sbi_tlb_flush_range(struct cpumask *cmask, unsigned long start,put_cpu (); } +static void __sbi_tlb_flush_range_asid (struct cpumask *cmask,unsigned long start, +unsigned long size,unsigned long asid) +{ +struct cpumask hmask; +unsigned int cpuid; + +if (cpumask_empty (cmask)) +return ; + + cpuid =get_cpu (); + +if (cpumask_any_but (cmask, cpuid) >= nr_cpu_ids) { +local_flush_tlb_range_asid (start, size, asid); + }else { +riscv_cpuid_to_hartid_mask (cmask, &hmask); +sbi_remote_sfence_vma_asid (cpumask_bits (&hmask), start, size, asid); + } + +put_cpu (); +} +void flush_tlb_mm (struct mm_struct *mm) { -__sbi_tlb_flush_range (mm_cpumask (mm),0 ,-1 ); +if (static_branch_unlikely (&use_asid_allocator)) +__sbi_tlb_flush_range_asid (mm_cpumask (mm),0 ,-1 , +atomic_long_read (&mm->context.id)); +else +__sbi_tlb_flush_range (mm_cpumask (mm),0 ,-1 ); }void flush_tlb_page (struct vm_area_struct *vma,unsigned long addr) { -__sbi_tlb_flush_range (mm_cpumask (vma->vm_mm), addr,PAGE_SIZE ); +if (static_branch_unlikely (&use_asid_allocator)) +__sbi_tlb_flush_range_asid (mm_cpumask (vma->vm_mm), addr,PAGE_SIZE , +atomic_long_read (&vma->vm_mm->context.id)); +else +__sbi_tlb_flush_range (mm_cpumask (vma->vm_mm), addr,PAGE_SIZE ); }void flush_tlb_range (struct vm_area_struct *vma,unsigned long start,unsigned long end) { -__sbi_tlb_flush_range (mm_cpumask (vma->vm_mm), start, end - start); +if (static_branch_unlikely (&use_asid_allocator)) +__sbi_tlb_flush_range_asid (mm_cpumask (vma->vm_mm), start, end - start, +atomic_long_read (&vma->vm_mm->context.id)); +else +__sbi_tlb_flush_range (mm_cpumask (vma->vm_mm), start, end - start); } --2.7.4
Quoted 11 lines · On Mon, May 24, 2021 at 12:22 PM <[email protected]> wrote:
From: Guo Ren <[email protected]>
Kernel virtual address translation should avoid care asid or it'll cause more TLB-miss and TLB-refill. Because the current asid in satp belongs to the current process, but the target kernel va TLB entry's asid still belongs to the previous process.
Signed-off-by Guo Ren <[email protected]>Cc Anup Patel <[email protected]>Cc Palmer Dabbelt <[email protected]>
First of all thanks for doing this series, I had similar changes in mind as follow-up to the ASID allocator.
I went through all three patches and at least I don't see any obvious issue but I think we should try testing it more on a few existing platforms.
Regards,
Anup
Quoted 19 lines
--- arch/riscv/include/asm/pgtable.h | 1 + 1 file changed, 1 insertion(+)arch/riscv/include/asm/pgtable.h +1
-1@@ -135,6 +135,7 @@ | _PAGE_PRESENT \ | _PAGE_ACCESSED \ | _PAGE_DIRTY \ + | _PAGE_GLOBAL \ | _PAGE_CACHE)#define PAGE_KERNEL __pgprot(_PAGE_KERNEL) --2.7.4
Hi Anup,
Quoted 14 lines · Anup Patel
Quoted 11 lines · On Mon, May 24, 2021 at 12:22 PM <[email protected]> wrote:
From: Guo Ren <[email protected]>
Kernel virtual address translation should avoid care asid or it'll cause more TLB-miss and TLB-refill. Because the current asid in satp belongs to the current process, but the target kernel va TLB entry's asid still belongs to the previous process.
Signed-off-by Guo Ren <[email protected]>Cc Anup Patel <[email protected]>Cc Palmer Dabbelt <[email protected]>
Quoted 6 lines
First of all thanks for doing this series, I had similar changes in mind as follow-up to the ASID allocator.
I went through all three patches and at least I don't see any obvious issue but I think we should try testing it more on a few existing platforms.
We've tested it on Allwinner D1 C906 and C910 SMP*4 for a long time, just hope it won't affect u540.
(In fact, C910 has used ASID allocator for more than two years with our own kernel-tree. I remember we've talked about it in 2019 plumber.)
Quoted 25 lines
Reviewed-by Anup Patel <[email protected]>Regards,
AnupQuoted 19 lines
--- arch/riscv/include/asm/pgtable.h | 1 + 1 file changed, 1 insertion(+)arch/riscv/include/asm/pgtable.h +1
-1@@ -135,6 +135,7 @@ | _PAGE_PRESENT \ | _PAGE_ACCESSED \ | _PAGE_DIRTY \ + | _PAGE_GLOBAL \ | _PAGE_CACHE)#define PAGE_KERNEL __pgprot(_PAGE_KERNEL) --2.7.4
Signature · 4 lines
Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/
Quoted 20 lines · On Mon, May 24, 2021 at 12:22 PM <[email protected]> wrote:
From: Guo Ren <[email protected]>
Kernel virtual address translation should avoid care asid or it'll cause more TLB-miss and TLB-refill. Because the current asid in satp belongs to the current process, but the target kernel va TLB entry's asid still belongs to the previous process.
Signed-off-by Guo Ren <[email protected]>Cc Anup Patel <[email protected]>Cc Palmer Dabbelt <[email protected]>--- arch/riscv/include/asm/pgtable.h | 1 + 1 file changed, 1 insertion(+)arch/riscv/include/asm/pgtable.h +0
-0
arch/riscv/include/asm/pgtable.h line 139
Quoted 5 lines
arch/riscv/include/asm/pgtable.h +1
-0@@ -135,6 +135,7 @@ | _PAGE_PRESENT \ | _PAGE_ACCESSED \ | _PAGE_DIRTY \ + | _PAGE_GLOBAL \ | _PAGE_CACHE)
It seems this patch is not based on the upstream kernel. The _PAGE_CACHE seems to be from your other patch series.
Please rebase these patches on the latest upstream kernel without dependency on any other patch series.
Regards,
Anup
Quoted 5 lines
arch/riscv/include/asm/pgtable.h +0
-1#define PAGE_KERNEL __pgprot(_PAGE_KERNEL) --2.7.4
Quoted 31 lines · Anup Patel
Quoted 20 lines · On Mon, May 24, 2021 at 12:22 PM <[email protected]> wrote:
From: Guo Ren <[email protected]>
Kernel virtual address translation should avoid care asid or it'll cause more TLB-miss and TLB-refill. Because the current asid in satp belongs to the current process, but the target kernel va TLB entry's asid still belongs to the previous process.
Signed-off-by Guo Ren <[email protected]>Cc Anup Patel <[email protected]>Cc Palmer Dabbelt <[email protected]>--- arch/riscv/include/asm/pgtable.h | 1 + 1 file changed, 1 insertion(+)arch/riscv/include/asm/pgtable.h +0
-0arch/riscv/include/asm/pgtable.h line 139
Quoted 5 lines
arch/riscv/include/asm/pgtable.h +1
-0@@ -135,6 +135,7 @@ | _PAGE_PRESENT \ | _PAGE_ACCESSED \ | _PAGE_DIRTY \ + | _PAGE_GLOBAL \ | _PAGE_CACHE)It seems this patch is not based on the upstream kernel. The
_PAGE_CACHEseems to be from your other patch series.
Quoted 2 lines
Please rebase these patches on the latest upstream kernel without dependency on any other patch series.
Yes, it based on DMA_COHERENT. I'll rebase in PATCH V2, thx.
Quoted 9 lines
Regards,
AnupQuoted 5 lines
arch/riscv/include/asm/pgtable.h +0
-1#define PAGE_KERNEL __pgprot(_PAGE_KERNEL) --2.7.4
Signature · 4 lines
Best Regards Guo Ren ML: https://lore.kernel.org/linux-csky/