use static key to optimize pgtable_l4_enabled
v6 cover use static key to optimize pgtable_l4_enabled
The pgtable_l4|[l5]_enabled check sits at hot code path, performance is impacted a lot. Since pgtable_l4|[l5]_enabled isn't changed after boot, so static key can be used to solve the performance issue[1].
An unified way static key was introduced in [2], but it only targets riscv isa extension. We dunno whether SV48 and SV57 will be considered as isa extension, so the unified solution isn't used for pgtable_l4[l5]_enabled now.
patch1 fixes a NULL pointer deference if static key is used a bit earlier. patch2 uses the static key to optimize pgtable_l4|[l5]_enabled.
[1] http://lists.infradead.org/pipermail/linux-riscv/2021-December/011164.html
[2] https://lore.kernel.org/linux-riscv/[email protected]/T/#t
Since v5:
- Use DECLARE_STATIC_KEY_FALSE
Since v4:
- rebased on v5.19-rcN
- collect Reviewed-by tags
- Fix kernel panic issue if SPARSEMEM is enabled by moving the riscv_finalise_pgtable_lx() after sparse_init()
Since v3:
- fix W=1 call to undeclared function 'static_branch_likely' error
Since v2:
- move the W=1 warning fix to a separate patch
- move the unified way to use static key to a new patch series.
Since v1:
- Add a W=1 warning fix
- Fix W=1 error
- Based on v5.18-rcN, since SV57 support is added, so convert pgtable_l5_enabled as well.
Jisheng Zhang (2):
riscv: move sbi_init() earlier before jump_label_init()
riscv: turn pgtable_l4|[l5]_enabled to static key for RV64
arch/riscv/include/asm/pgalloc.h | 16 ++++---- arch/riscv/include/asm/pgtable-32.h | 3 ++ arch/riscv/include/asm/pgtable-64.h | 60 ++++++++++++++++++--------- arch/riscv/include/asm/pgtable.h | 5 +-- arch/riscv/kernel/cpu.c | 4 +- arch/riscv/kernel/setup.c | 2 +- arch/riscv/mm/init.c | 64 ++++++++++++++++++----------- arch/riscv/mm/kasan_init.c | 16 ++++---- 8 files changed, 104 insertions(+), 66 deletions(-)
Signature · 1 line
2.34.1
PATCH v6 1/2 riscv: move sbi_init() earlier before jump_label_init()
We call jump_label_init() in setup_arch() is to use static key mechanism earlier, but riscv jump label relies on the sbi functions, If we enable static key before sbi_init(), the code path looks like:
static_branch_enable()
..
arch_jump_label_transform()
patch_text_nosync()
flush_icache_range()
flush_icache_all()
sbi_remote_fence_i() for CONFIG_RISCV_SBI case
__sbi_rfence()Since sbi isn't initialized, so NULL deference! Here is a typical panic log:
[ 0.000000] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000 [ 0.000000] Oops [#1] [ 0.000000] Modules linked in: [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.18.0-rc7+ #79 [ 0.000000] Hardware name: riscv-virtio,qemu (DT) [ 0.000000] epc : 0x0 [ 0.000000] ra : sbi_remote_fence_i+0x1e/0x26 [ 0.000000] epc : 0000000000000000 ra : ffffffff80005826 sp : ffffffff80c03d50 [ 0.000000] gp : ffffffff80ca6178 tp : ffffffff80c0ad80 t0 : 6200000000000000 [ 0.000000] t1 : 0000000000000000 t2 : 62203a6b746e6972 s0 : ffffffff80c03d60 [ 0.000000] s1 : ffffffff80001af6 a0 : 0000000000000000 a1 : 0000000000000000 [ 0.000000] a2 : 0000000000000000 a3 : 0000000000000000 a4 : 0000000000000000 [ 0.000000] a5 : 0000000000000000 a6 : 0000000000000000 a7 : 0000000000080200 [ 0.000000] s2 : ffffffff808b3e48 s3 : ffffffff808bf698 s4 : ffffffff80cb2818 [ 0.000000] s5 : 0000000000000001 s6 : ffffffff80c9c345 s7 : ffffffff80895aa0 [ 0.000000] s8 : 0000000000000001 s9 : 000000000000007f s10: 0000000000000000 [ 0.000000] s11: 0000000000000000 t3 : ffffffff80824d08 t4 : 0000000000000022 [ 0.000000] t5 : 000000000000003d t6 : 0000000000000000 [ 0.000000] status: 0000000000000100 badaddr: 0000000000000000 cause: 000000000000000c [ 0.000000] ---[ end trace 0000000000000000 ]--- [ 0.000000] Kernel panic - not syncing: Attempted to kill the idle task! [ 0.000000] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---
Fix this issue by moving sbi_init() earlier before jump_label_init()
--- arch/riscv/kernel/setup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/riscv/kernel/setup.c +1 -1
@@ -269,6 +269,7 @@ void __init setup_arch(char **cmdline_p) *cmdline_p = boot_command_line;early_ioremap_setup (); +sbi_init ();jump_label_init ();parse_early_param ();
@@ -285,7 +286,6 @@ void __init setup_arch(char **cmdline_p)misc_mem_init ();init_resources (); -sbi_init ();#ifdef CONFIG_KASAN kasan_init ();
Signature · 1 line
2.34.1
PATCH v6 2/2 riscv: turn pgtable_l4|[l5]_enabled to static key for RV64
In reply to Jisheng ZhangOn a specific HW platform, pgtable_l4|[l5]_enabled won't change after boot, and the check sits at hot code path, this characteristic makes it suitable for optimization with static key.
_pgtable_l4|[l5]_enabled is used very early during boot, even is used with MMU off, so the static key mechanism isn't ready. For this case, we use another static key _pgtable_lx_ready to indicate whether we have finalised pgtable_l4|[l5]_enabled or not, then fall back to _pgtable_l4|[l5]_enabled_early bool.
--- arch/riscv/include/asm/pgalloc.h | 16 ++++---- arch/riscv/include/asm/pgtable-32.h | 3 ++ arch/riscv/include/asm/pgtable-64.h | 60 ++++++++++++++++++--------- arch/riscv/include/asm/pgtable.h | 5 +-- arch/riscv/kernel/cpu.c | 4 +- arch/riscv/mm/init.c | 64 ++++++++++++++++++----------- arch/riscv/mm/kasan_init.c | 16 ++++---- 7 files changed, 103 insertions(+), 65 deletions(-)
arch/riscv/include/asm/pgalloc.h +8 -8
@@ -41,7 +41,7 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd)static inline void p4d_populate (struct mm_struct *mm,p4d_t *p4d,pud_t *pud) { -if (pgtable_l4_enabled) { +if (pgtable_l4_enabled ()) {unsigned long pfn =virt_to_pfn (pud);set_p4d (p4d,__p4d ((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE));
@@ -51,7 +51,7 @@ static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4d, pud_t *pud)static inline void p4d_populate_safe (struct mm_struct *mm,p4d_t *p4d,pud_t *pud) { -if (pgtable_l4_enabled) { +if (pgtable_l4_enabled ()) {unsigned long pfn =virt_to_pfn (pud); set_p4d_safe(p4d,
@@ -61,7 +61,7 @@ static inline void p4d_populate_safe(struct mm_struct *mm, p4d_t *p4d,static inline void pgd_populate (struct mm_struct *mm,pgd_t *pgd,p4d_t *p4d) { -if (pgtable_l5_enabled) { +if (pgtable_l5_enabled ()) {unsigned long pfn =virt_to_pfn (p4d);set_pgd (pgd,__pgd ((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE));
@@ -71,7 +71,7 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4d)static inline void pgd_populate_safe (struct mm_struct *mm,pgd_t *pgd,p4d_t *p4d) { -if (pgtable_l5_enabled) { +if (pgtable_l5_enabled ()) {unsigned long pfn =virt_to_pfn (p4d); set_pgd_safe(pgd,
@@ -82,7 +82,7 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd,#define pud_alloc_one pud_alloc_onestatic inline pud_t *pud_alloc_one (struct mm_struct *mm,unsigned long addr) { -if (pgtable_l4_enabled) +if (pgtable_l4_enabled ())return __pud_alloc_one (mm, addr);return NULL ;
@@ -91,7 +91,7 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr)#define pud_free pud_freestatic inline void pud_free (struct mm_struct *mm,pud_t *pud) { -if (pgtable_l4_enabled) +if (pgtable_l4_enabled ())__pud_free (mm, pud); }
@@ -100,7 +100,7 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud)#define p4d_alloc_one p4d_alloc_onestatic inline p4d_t *p4d_alloc_one (struct mm_struct *mm,unsigned long addr) { -if (pgtable_l5_enabled) { +if (pgtable_l5_enabled ()) {gfp_t gfp =GFP_PGTABLE_USER ;if (mm == &init_mm)
@@ -120,7 +120,7 @@ static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d)#define p4d_free p4d_freestatic inline void p4d_free (struct mm_struct *mm,p4d_t *p4d) { -if (pgtable_l5_enabled) +if (pgtable_l5_enabled ())__p4d_free (mm, p4d); }
arch/riscv/include/asm/pgtable-32.h +3 -0
@@ -17,6 +17,9 @@#define MAX_POSSIBLE_PHYSMEM_BITS 34 +#define pgtable_l5_enabled () 0 +#define pgtable_l4_enabled () 0 +/* * rv32 PTE format: * | XLEN-1 10 | 9 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0
arch/riscv/include/asm/pgtable-64.h +40 -20
169 lines not on this page · raw message
arch/riscv/include/asm/pgtable.h +2 -3
@@ -63,8 +63,8 @@* position vmemmap directly below the VMALLOC region. */ #ifdef CONFIG_64BIT -#define VA_BITS (pgtable_l5_enabled ? \ - 57 : (pgtable_l4_enabled ? 48 : 39)) +#define VA_BITS (pgtable_l5_enabled() ? \ + 57 : (pgtable_l4_enabled() ? 48 : 39))#else #define VA_BITS 32#endif
@@ -834,7 +834,6 @@ extern uintptr_t _dtb_early_pa;#define dtb_early_pa _dtb_early_pa#endif /* CONFIG_XIP_KERNEL */ extern u64 satp_mode; -extern bool pgtable_l4_enabled;void paging_init (void );void misc_mem_init (void );
arch/riscv/kernel/cpu.c +2 -2
@@ -143,9 +143,9 @@ static void print_mmu(struct seq_file *f)#if defined(CONFIG_32BIT )strncpy (sv_type,"sv32" ,5 );#elif defined(CONFIG_64BIT ) -if (pgtable_l5_enabled) +if (pgtable_l5_enabled ())strncpy (sv_type,"sv57" ,5 ); -else if (pgtable_l4_enabled) +else if (pgtable_l4_enabled ())strncpy (sv_type,"sv48" ,5 );else strncpy (sv_type,"sv39" ,5 );
arch/riscv/mm/init.c +40 -24
134 lines not on this page · raw message
arch/riscv/mm/kasan_init.c +8 -8
@@ -209,15 +209,15 @@ static void __init kasan_populate_p4d(pgd_t *pgd,set_pgd (pgd,pfn_pgd (PFN_DOWN (__pa (base_p4d)),PAGE_TABLE )); } -#define kasan_early_shadow_pgd_next (pgtable_l5_enabled ? \ +#define kasan_early_shadow_pgd_next (pgtable_l5_enabled() ? \ (uintptr_t)kasan_early_shadow_p4d : \ - (pgtable_l4_enabled ? \ + (pgtable_l4_enabled() ? \ (uintptr_t)kasan_early_shadow_pud : \ (uintptr_t)kasan_early_shadow_pmd))#define kasan_populate_pgd_next (pgdp, vaddr, next, early) \ - (pgtable_l5_enabled ? \ + (pgtable_l5_enabled() ? \ kasan_populate_p4d(pgdp, vaddr, next, early) : \ - (pgtable_l4_enabled ? \ + (pgtable_l4_enabled() ? \ kasan_populate_pud(pgdp, vaddr, next, early) : \ kasan_populate_pmd((pud_t *)pgdp, vaddr, next)))
@@ -274,7 +274,7 @@ asmlinkage void __init kasan_early_init(void) (__pa ((uintptr_t )kasan_early_shadow_pte)), PAGE_TABLE)); -if (pgtable_l4_enabled) { +if (pgtable_l4_enabled ()) {for (i =0 ; i <PTRS_PER_PUD ; ++i) set_pud(kasan_early_shadow_pud + i, pfn_pud(PFN_DOWN
@@ -282,7 +282,7 @@ asmlinkage void __init kasan_early_init(void) PAGE_TABLE)); } -if (pgtable_l5_enabled) { +if (pgtable_l5_enabled ()) {for (i =0 ; i <PTRS_PER_P4D ; ++i) set_p4d(kasan_early_shadow_p4d + i, pfn_p4d(PFN_DOWN
@@ -393,9 +393,9 @@ static void __init kasan_shallow_populate_p4d(pgd_t *pgdp, }#define kasan_shallow_populate_pgd_next (pgdp, vaddr, next) \ - (pgtable_l5_enabled ? \ + (pgtable_l5_enabled() ? \ kasan_shallow_populate_p4d(pgdp, vaddr, next) : \ - (pgtable_l4_enabled ? \ + (pgtable_l4_enabled() ? \ kasan_shallow_populate_pud(pgdp, vaddr, next) : \ kasan_shallow_populate_pmd(pgdp, vaddr, next)))
Signature · 1 line
2.34.1
Quoted 9 lines · Jisheng Zhang
The pgtable_l4|[l5]_enabled check sits at hot code path, performance is impacted a lot. Since pgtable_l4|[l5]_enabled isn't changed after boot, so static key can be used to solve the performance issue[1].
An unified way static key was introduced in [2], but it only targets riscv isa extension. We dunno whether SV48 and SV57 will be considered as isa extension, so the unified solution isn't used for pgtable_l4[l5]_enabled now.
Quoted 5 lines
patch1 fixes a NULL pointer deference if static key is used a bit earlier. patch2 uses the static key to optimize pgtable_l4|[l5]_enabled.
[1] http://lists.infradead.org/pipermail/linux-riscv/2021-December/011164.html
[2] https://lore.kernel.org/linux-riscv/[email protected]/T/#t
Hi Palmer,
I see part1 and part2 were sent out...
What I can do to make this series merged for 6.0-rc1? I'm afraid this series may miss anothe round of merge window again.
Thanks in advance
Quoted 45 lines
Since v5:
- UseDECLARE_STATIC_KEY_FALSESince v4:
- rebased on v5.19-rcN
- collect Reviewed-by tags
- Fix kernel panic issue if SPARSEMEM is enabled by moving theriscv_finalise_pgtable_lx()aftersparse_init()Since v3:
- fix W=1 call to undeclared function 'static_branch_likely' errorSince v2:
- move the W=1 warning fix to a separate patch
- move the unified way to use static key to a new patch series.Since v1:
- Add a W=1 warning fix
- Fix W=1 error
- Based on v5.18-rcN, since SV57 support is added, so convertpgtable_l5_enabledas well.Jisheng Zhang (2):
riscv: movesbi_init()earlier beforejump_label_init()
riscv: turn pgtable_l4|[l5]_enabled to static key for RV64arch/riscv/include/asm/pgalloc.h | 16 ++++---- arch/riscv/include/asm/pgtable-32.h | 3 ++ arch/riscv/include/asm/pgtable-64.h | 60 ++++++++++++++++++--------- arch/riscv/include/asm/pgtable.h | 5 +-- arch/riscv/kernel/cpu.c | 4 +- arch/riscv/kernel/setup.c | 2 +- arch/riscv/mm/init.c | 64 ++++++++++++++++++----------- arch/riscv/mm/kasan_init.c | 16 ++++---- 8 files changed, 104 insertions(+), 66 deletions(-)Signature · 7 lines
2.34.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv