I-side fixes
v2 cover I-side fixes
Hi all,
Here's version two of the patches I previously posted here:
http://lists.infradead.org/pipermail/linux-arm-kernel/2018-June/584857.html
Changes since v1 include:
* Removal of "hotpatch" mechanism for certain classes of instructions
* Avoid attempting IPI when secondaries are in stop_machine()
* Leave ISBs intact in the stop_machine() callbacks
Cheers,
Will
--->8
Will Deacon (4):
arm64: Avoid flush_icache_range() in alternatives patching code
arm64: Remove unnecessary ISBs from set_{pte,pmd,pud}
arm64: IPI each CPU after invalidating the I-cache for kernel mappings
arm64: insn: Don't fallback on nosync path for general insn patching
arch/arm64/include/asm/alternative.h | 7 +++- arch/arm64/include/asm/cacheflush.h | 13 ++++++- arch/arm64/include/asm/insn.h | 2 -- arch/arm64/include/asm/pgtable.h | 6 +--- arch/arm64/kernel/alternative.c | 51 ++++++++++++++++++++++---- arch/arm64/kernel/cpu_errata.c | 2 +- arch/arm64/kernel/insn.c | 70 ++---------------------------------- arch/arm64/kernel/module.c | 5 ++- arch/arm64/mm/cache.S | 4 +-- 9 files changed, 71 insertions(+), 89 deletions(-)
Signature · 1 line
2.1.4
PATCH v2 1/4 arm64: Avoid flush_icache_range() in alternatives patching code
The implementation of flush_icache_range() includes instruction sequences which are themselves patched at runtime, so it is not safe to call from the patching framework.
This patch reworks the alternatives cache-flushing code so that it rolls its own internal D-cache maintenance using DC CIVAC before invalidating the entire I-cache after all alternatives have been applied at boot. Modules don't cause any issues, since flush_icache_range() is safe to call by the time they are loaded.
--- arch/arm64/include/asm/alternative.h | 7 ++++- arch/arm64/kernel/alternative.c | 51 +++++++++++++++++++++++++++++++----- arch/arm64/kernel/module.c | 5 ++-- 3 files changed, 52 insertions(+), 11 deletions(-)
arch/arm64/include/asm/alternative.h +6 -1
@@ -28,7 +28,12 @@ typedef void (*alternative_cb_t)(struct alt_instr *alt, __le32 *origptr, __le32 *updptr, int nr_inst);void __init apply_alternatives_all (void ); -void apply_alternatives (void *start,size_t length); + +#ifdef CONFIG_MODULES +void apply_alternatives_module (void *start,size_t length); +#else +static inline void apply_alternatives_module (void *start,size_t length) { } +#endif #define ALTINSTR_ENTRY (feature,cb) \ " .word 661b - .\n"/* label */ \
arch/arm64/kernel/alternative.c +44 -7
85 lines not on this page · raw message
arch/arm64/kernel/module.c +2 -3
@@ -448,9 +448,8 @@ int module_finalize(const Elf_Ehdr *hdr,const char *secstrs = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset;for (s = sechdrs, se = sechdrs + hdr->e_shnum; s < se; s++) { -if (strcmp (".altinstructions" , secstrs + s->sh_name) ==0 ) { -apply_alternatives ((void *)s->sh_addr, s->sh_size); - } +if (strcmp (".altinstructions" , secstrs + s->sh_name) ==0 ) +apply_alternatives_module ((void *)s->sh_addr, s->sh_size);#ifdef CONFIG_ARM64_MODULE_PLTS if (IS_ENABLED (CONFIG_DYNAMIC_FTRACE ) && !strcmp (".text.ftrace_trampoline" , secstrs + s->sh_name))
Signature · 1 line
2.1.4
PATCH v2 2/4 arm64: Remove unnecessary ISBs from set_{pte,pmd,pud}
In reply to Will DeaconCommit 7f0b1bf04511 ("arm64: Fix barriers used for page table modifications") fixed a reported issue with fixmap page-table entries not being visible to the walker due to a missing DSB instruction. At the same time, it added ISB instructions to the arm64 set_{pte,pmd,pud} functions, which are not required by the architecture and make little sense in isolation.
Remove the redundant ISBs.
--- arch/arm64/include/asm/pgtable.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
arch/arm64/include/asm/pgtable.h +1 -5
@@ -224,10 +224,8 @@ static inline void set_pte(pte_t *ptep, pte_t pte)* Only if the new pte is valid and kernel, otherwise TLB maintenance * or update_mmu_cache() have the necessary barriers. */ -if (pte_valid_not_user (pte)) { +if (pte_valid_not_user (pte))dsb (ishst); -isb (); - } }extern void __sync_icache_dcache (pte_t pteval);
@@ -434,7 +432,6 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) {WRITE_ONCE (*pmdp, pmd);dsb (ishst); -isb (); }static inline void pmd_clear (pmd_t *pmdp)
@@ -485,7 +482,6 @@ static inline void set_pud(pud_t *pudp, pud_t pud) {WRITE_ONCE (*pudp, pud);dsb (ishst); -isb (); }static inline void pud_clear (pud_t *pudp)
Signature · 1 line
2.1.4
PATCH v2 2/4 arm64: Remove unnecessary ISBs from set_{pte,pmd,pud}
[Apologies for getting to this late]
Hi Will,
Quoted 8 lines · Will Deacon
Commit
7f0b1bf04511("arm64: Fix barriers used for page table modifications") fixed a reported issue with fixmap page-table entries not being visible to the walker due to a missing DSB instruction. At the same time, it added ISB instructions to the arm64 set_{pte,pmd,pud} functions, which are not required by the architecture and make little sense in isolation.Remove the redundant ISBs.
I agree, we don't want to update any cached local execution context information whilst setting page table entries and TLB maintenance is handled elsewhere; so the ISBs are redundant.
FWIW: Acked-by: Steve Capper <[email protected]>
Cheers,
Signature · 1 line
Steve
PATCH v2 3/4 arm64: IPI each CPU after invalidating the I-cache for kernel mappings
In reply to Will DeaconWhen invalidating the instruction cache for a kernel mapping via flush_icache_range(), it is also necessary to flush the pipeline for other CPUs so that instructions fetched into the pipeline before the I-cache invalidation are discarded. For example, if module 'foo' is unloaded and then module 'bar' is loaded into the same area of memory, a CPU could end up executing instructions from 'foo' when branching into 'bar' if these instructions were fetched into the pipeline before 'foo' was unloaded.
Whilst this is highly unlikely to occur in practice, particularly as any exception acts as a context-synchronizing operation, following the letter of the architecture requires us to execute an ISB on each CPU in order for the new instruction stream to be visible.
--- arch/arm64/include/asm/cacheflush.h | 13 ++++++++++++- arch/arm64/kernel/cpu_errata.c | 2 +- arch/arm64/kernel/insn.c | 18 ++++-------------- arch/arm64/mm/cache.S | 4 ++-- 4 files changed, 19 insertions(+), 18 deletions(-)
arch/arm64/include/asm/cacheflush.h +12 -1
@@ -71,7 +71,7 @@* - kaddr - page address * - size - region size */ -extern void flush_icache_range (unsigned long start,unsigned long end); +extern void __flush_icache_range (unsigned long start,unsigned long end);extern int invalidate_icache_range (unsigned long start,unsigned long end);extern void __flush_dcache_area (void *addr,size_t len);extern void __inval_dcache_area (void *addr,size_t len);
@@ -81,6 +81,17 @@ extern void __clean_dcache_area_pou(void *addr, size_t len);extern long __flush_cache_user_range (unsigned long start,unsigned long end);extern void sync_icache_aliases (void *kaddr,unsigned long len); +static inline void flush_icache_range (unsigned long start,unsigned long end) +{ +__flush_icache_range (start, end); + +/* +* IPI all online CPUs so that they undergo a context synchronization +* event and are forced to refetch the new instructions. +*/ +kick_all_cpus_sync (); +} +static inline void flush_cache_mm (struct mm_struct *mm) { }
arch/arm64/kernel/cpu_errata.c +1 -1
@@ -101,7 +101,7 @@ static void __copy_hyp_vect_bpi(int slot, const char *hyp_vecs_start,for (i =0 ; i <SZ_2K ; i +=0x80 )memcpy (dst + i, hyp_vecs_start, hyp_vecs_end - hyp_vecs_start); -flush_icache_range ((uintptr_t )dst, (uintptr_t )dst +SZ_2K ); +__flush_icache_range ((uintptr_t )dst, (uintptr_t )dst +SZ_2K ); } static void __install_bp_hardening_cb(bp_hardening_cb_t fn,
arch/arm64/kernel/insn.c +4 -14
@@ -216,8 +216,8 @@ int __kprobes aarch64_insn_patch_text_nosync(void *addr, u32 insn) ret =aarch64_insn_write (tp, insn);if (ret ==0 ) -flush_icache_range ((uintptr_t )tp, - (uintptr_t )tp +AARCH64_INSN_SIZE ); +__flush_icache_range ((uintptr_t )tp, + (uintptr_t )tp +AARCH64_INSN_SIZE );return ret; }
@@ -283,18 +283,8 @@ int __kprobes aarch64_insn_patch_text(void *addrs[], u32 insns[], int cnt)if (ret)return ret; -if (aarch64_insn_hotpatch_safe (insn, insns[0 ])) { -/* -* ARMv8 architecture doesn't guarantee all CPUs see -* the new instruction after returning from function -* aarch64_insn_patch_text_nosync(). So send IPIs to -* all other CPUs to achieve instruction -* synchronization. -*/ - ret =aarch64_insn_patch_text_nosync (addrs[0 ], insns[0 ]); -kick_all_cpus_sync (); -return ret; - } +if (aarch64_insn_hotpatch_safe (insn, insns[0 ])) +return aarch64_insn_patch_text_nosync (addrs[0 ], insns[0 ]); }return aarch64_insn_patch_text_sync (addrs, insns, cnt);
arch/arm64/mm/cache.S +2 -2
@@ -35,7 +35,7 @@ * - start - virtual start address of region * - end - virtual end address of region */ -ENTRY(flush_icache_range) +ENTRY(__flush_icache_range) /* FALLTHROUGH */ /*
@@ -77,7 +77,7 @@ alternative_else_nop_endif 9: mov x0, #-EFAULT b 1b -ENDPROC(flush_icache_range) +ENDPROC(__flush_icache_range) ENDPROC(__flush_cache_user_range) /*
Signature · 1 line
2.1.4
Catalin Marinas
maintainerPATCH v2 3/4 arm64: IPI each CPU after invalidating the I-cache for kernel mappings
Hi Will,
Quoted 9 lines · Will Deacon
When invalidating the instruction cache for a kernel mapping via
flush_icache_range(), it is also necessary to flush the pipeline for other CPUs so that instructions fetched into the pipeline before the I-cache invalidation are discarded. For example, if module 'foo' is unloaded and then module 'bar' is loaded into the same area of memory, a CPU could end up executing instructions from 'foo' when branching into 'bar' if these instructions were fetched into the pipeline before 'foo' was unloaded.
Quoted 6 lines
Whilst this is highly unlikely to occur in practice, particularly as any exception acts as a context-synchronizing operation, following the letter of the architecture requires us to execute an ISB on each CPU in order for the new instruction stream to be visible.
Signed-off-by Will Deacon <[email protected]>
I hit a warning via kgdb_flush_swbreak_addr() because if IRQs disabled (and it actually deadlocks for me; running as a guest under KVM on TX1):
WARNING: CPU: 3 PID: 1 at /home/cmarinas/work/Linux/linux-2.6-aarch64/kernel/smp.c:416 smp_call_function_many+0xd4/0x350 Modules linked in: CPU: 3 PID: 1 Comm: swapper/0 Not tainted 4.18.0-rc2-00006-ga5cfc8429d36 #97 Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 pstate: 200003c5 (nzCv DAIF -PAN -UAO) pc : smp_call_function_many+0xd4/0x350 lr : smp_call_function+0x38/0x68
sp : ffff0000080737f0
x29: ffff0000080737f0 x28: ffff000009169000
x27: 0000000000000003 x26: 0000000000000000
x25: ffff00000815a3d0 x24: 0000000000000001
x23: 0000000000000000 x22: ffff000008eba730
x21: ffff000009169940 x20: 0000000000000000
x19: 0000000000000003 x18: ffffffffffffffff
x17: 0000000000000001 x16: 0000000000000019
x15: ffff0000091696c8 x14: ffff00008930ef07
x13: ffff00000930ef1d x12: 0000000000000010
x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f
x9 : 656565652aff6223 x8 : ffffffffffffffff
x7 : fefefefefefefefe x6 : ffff7dfffe7fe014
x5 : ffff000009169940 x4 : ffff000009147018
x3 : 0000000000000001 x2 : 0000000000000000
x1 : ffff00000815a3d0 x0 : 0000000000000000
Call trace: smp_call_function_many+0xd4/0x350 smp_call_function+0x38/0x68 kick_all_cpus_sync+0x20/0x28 kgdb_flush_swbreak_addr+0x14/0x20 dbg_activate_sw_breakpoints+0x74/0xb0 gdb_serial_stub+0x6ec/0xc80 kgdb_cpu_enter+0x36c/0x578 kgdb_handle_exception+0x130/0x238 kgdb_compiled_brk_fn+0x28/0x38 brk_handler+0x80/0xd8 do_debug_exception+0x94/0x170 el1_dbg+0x18/0x78 kgdb_breakpoint+0x20/0x40 run_breakpoint_test+0x58/0xb0 configure_kgdbts+0x1b8/0x538 init_kgdbts+0x1c/0x2c do_one_initcall+0x58/0x170 kernel_init_freeable+0x184/0x22c kernel_init+0x10/0x108 ret_from_fork+0x10/0x18
Signature · 1 line
Catalin
PATCH v2 3/4 arm64: IPI each CPU after invalidating the I-cache for kernel mappings
Hi Catalin,
Quoted 43 lines · Catalin Marinas
Quoted 9 lines · Will Deacon
When invalidating the instruction cache for a kernel mapping via
flush_icache_range(), it is also necessary to flush the pipeline for other CPUs so that instructions fetched into the pipeline before the I-cache invalidation are discarded. For example, if module 'foo' is unloaded and then module 'bar' is loaded into the same area of memory, a CPU could end up executing instructions from 'foo' when branching into 'bar' if these instructions were fetched into the pipeline before 'foo' was unloaded.Quoted 6 lines
Whilst this is highly unlikely to occur in practice, particularly as any exception acts as a context-synchronizing operation, following the letter of the architecture requires us to execute an ISB on each CPU in order for the new instruction stream to be visible.
Signed-off-by Will Deacon <[email protected]>I hit a warning via
kgdb_flush_swbreak_addr()because if IRQs disabled (and it actually deadlocks for me; running as a guest under KVM on TX1):WARNING: CPU: 3 PID: 1 at /home/cmarinas/work/Linux/linux-2.6-aarch64/kernel/smp.c:416 smp_call_function_many+0xd4/0x350 Modules linked in: CPU: 3 PID: 1 Comm: swapper/0 Not tainted 4.18.0-rc2-00006-ga5cfc8429d36 #97 Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 pstate: 200003c5 (nzCv DAIF -PAN -UAO) pc : smp_call_function_many+0xd4/0x350 lr : smp_call_function+0x38/0x68sp :
ffff0000080737f0
x29:ffff0000080737f0x28:ffff000009169000
x27: 0000000000000003 x26: 0000000000000000
x25:ffff00000815a3d0x24: 0000000000000001
x23: 0000000000000000 x22:ffff000008eba730
x21:ffff000009169940x20: 0000000000000000
x19: 0000000000000003 x18: ffffffffffffffff
x17: 0000000000000001 x16: 0000000000000019
x15:ffff0000091696c8x14:ffff00008930ef07
x13:ffff00000930ef1dx12: 0000000000000010
x11: 0101010101010101 x10:7f7f7f7f7f7f7f7f
x9 :656565652aff6223x8 : ffffffffffffffff
x7 : fefefefefefefefe x6 :ffff7dfffe7fe014
x5 :ffff000009169940x4 :ffff000009147018
x3 : 0000000000000001 x2 : 0000000000000000
x1 :ffff00000815a3d0x0 : 0000000000000000
Quoted 8 lines
Call trace: smp_call_function_many+0xd4/0x350 smp_call_function+0x38/0x68 kick_all_cpus_sync+0x20/0x28 kgdb_flush_swbreak_addr+0x14/0x20 dbg_activate_sw_breakpoints+0x74/0xb0 gdb_serial_stub+0x6ec/0xc80 kgdb_cpu_enter+0x36c/0x578
Yuck, this is horrible. We don't actually need the IPI in this case because kgdb is using smp_call_function to roundup the secondary cores, but the core code doesn't have the flexibility for us to hook the operation like that. All it provides is a CACHE_FLUSH_IS_SAFE #define, which you can set to 0 if the maintenance isn't safe in IRQ-disabled context. However, there isn't a fall-back and the maintenance is just not performed at all in that case.
If we had a "flush the entire D-cache to PoU" instruction, we could hack something into the backend (MIPS does this) but we don't. The best I can come up with is the nasty hack below.
Will
--->8
arch/arm64/include/asm/cacheflush.h +13 -0
@@ -89,6 +89,19 @@ static inline void flush_icache_range(unsigned long start, unsigned long end)* IPI all online CPUs so that they undergo a context synchronization * event and are forced to refetch the new instructions. */ +#ifdef CONFIG_KGDB +/* +* KGDB performs cache maintenance with interrupts disabled, so we +* will deadlock trying to IPI the secondary CPUs. In theory, we can +* set CACHE_FLUSH_IS_SAFE to 0 to avoid this known issue, but that +* just means that KGDB will elide the maintenance altogether! As it +* turns out, KGDB uses IPIs to round-up the secondary CPUs during +* the patching operation, so we don't need extra IPIs here anyway. +* In which case, add a KGDB-specific bodge and return early. +*/ +if (kgdb_connected &&irqs_disabled ()) +return ; +#endif kick_all_cpus_sync (); }
Catalin Marinas
maintainerPATCH v2 3/4 arm64: IPI each CPU after invalidating the I-cache for kernel mappings
Quoted 20 lines · Will Deacon
Quoted 19 lines · Catalin Marinas
Quoted 9 lines · Will Deacon
When invalidating the instruction cache for a kernel mapping via
flush_icache_range(), it is also necessary to flush the pipeline for other CPUs so that instructions fetched into the pipeline before the I-cache invalidation are discarded. For example, if module 'foo' is unloaded and then module 'bar' is loaded into the same area of memory, a CPU could end up executing instructions from 'foo' when branching into 'bar' if these instructions were fetched into the pipeline before 'foo' was unloaded.Quoted 6 lines
Whilst this is highly unlikely to occur in practice, particularly as any exception acts as a context-synchronizing operation, following the letter of the architecture requires us to execute an ISB on each CPU in order for the new instruction stream to be visible.
Signed-off-by Will Deacon <[email protected]>I hit a warning via
kgdb_flush_swbreak_addr()because if IRQs disabled (and it actually deadlocks for me; running as a guest under KVM on TX1):
[...]
Quoted 28 lines
Yuck, this is horrible. We don't actually need the IPI in this case because kgdb is using
smp_call_functionto roundup the secondary cores, but the core code doesn't have the flexibility for us to hook the operation like that. All it provides is aCACHE_FLUSH_IS_SAFE#define, which you can set to 0 if the maintenance isn't safe in IRQ-disabled context. However, there isn't a fall-back and the maintenance is just not performed at all in that case.If we had a "flush the entire D-cache to PoU" instruction, we could hack something into the backend (MIPS does this) but we don't. The best I can come up with is the nasty hack below.
Will
--->8
arch/arm64/include/asm/cacheflush.h +5
-0@@ -89,6 +89,19 @@ static inline void flush_icache_range(unsigned long start, unsigned long end)* IPI all online CPUs so that they undergo a context synchronization * event and are forced to refetch the new instructions. */ +#ifdef CONFIG_KGDB +/* +* KGDB performs cache maintenance with interrupts disabled, so we +* will deadlock trying to IPI the secondary CPUs. In theory, we can +* set CACHE_FLUSH_IS_SAFE to 0 to avoid this known issue, but that
arch/arm64/include/asm/cacheflush.h static inline void flush_icache_range(unsigned long start, unsigned long end) · line 106
Quoted 10 lines
arch/arm64/include/asm/cacheflush.h +8
-0+* just means that KGDB will elide the maintenance altogether! As it +* turns out, KGDB uses IPIs to round-up the secondary CPUs during +* the patching operation, so we don't need extra IPIs here anyway. +* In which case, add a KGDB-specific bodge and return early. +*/ +if (kgdb_connected &&irqs_disabled ()) +return ; +#endif kick_all_cpus_sync (); }
It's indeed a hack but we can live with this. On the patch with this hunk:
PATCH v2 4/4 arm64: insn: Don't fallback on nosync path for general insn patching
In reply to Will DeaconPatching kernel instructions at runtime requires other CPUs to undergo a context synchronisation event via an explicit ISB or an IPI in order to ensure that the new instructions are visible. This is required even for "hotpatch" instructions such as NOP and BL, so avoid optimising in this case and always go via stop_machine() when performing general patching.
ftrace isn't quite as strict, so it can continue to call the nosync code directly.
--- arch/arm64/include/asm/insn.h | 2 -- arch/arm64/kernel/insn.c | 56 +------------------------------------------ 2 files changed, 1 insertion(+), 57 deletions(-)
arch/arm64/include/asm/insn.h +0 -2
@@ -446,8 +446,6 @@ u32 aarch64_insn_gen_prefetch(enum aarch64_insn_register base,s32 aarch64_get_branch_offset (u32 insn);u32 aarch64_set_branch_offset (u32 insn,s32 offset); -bool aarch64_insn_hotpatch_safe (u32 old_insn,u32 new_insn); -int aarch64_insn_patch_text_nosync (void *addr,u32 insn);int aarch64_insn_patch_text (void *addrs[],u32 insns[],int cnt);
arch/arm64/kernel/insn.c +1 -55
86 lines not on this page · raw message
Signature · 1 line
2.1.4
Catalin Marinas
maintainerv2 cover I-side fixes
In reply to Will DeaconQuoted 3 lines · Will Deacon
Will Deacon (4):
arm64: Avoidflush_icache_range()in alternatives patching code
arm64: Remove unnecessary ISBs from set_{pte,pmd,pud}
I'm merging these two patches for -rc3. I'll leave the other two to you for 4.19.
Signature · 1 line
Catalin