arm64: ftrace: Ensure module ftrace trampoline is coherent with I-side
Subsystems (3)
Will Deacon
maintainerPATCH 1/1 arm64: ftrace: Ensure module ftrace trampoline is coherent with I-side
The initial support for dynamic ftrace trampolines in modules made use of an indirect branch which loaded its target from the beginning of a special section (e71a4e1bebaf7 ("arm64: ftrace: add support for far branches to dynamic ftrace")). Since no instructions were being patched, no cache maintenance was needed. However, later in be0f272bfc83 ("arm64:
ftrace: emit ftrace-mod.o contents through code") this code was reworked to output the trampoline instructions directly into the PLT entry but, unfortunately, the necessary cache maintenance was overlooked.
Add a call to __flush_icache_range() after writing the new trampoline instructions but before patching in the branch to the trampoline.
--- arch/arm64/kernel/ftrace.c | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-)
arch/arm64/kernel/ftrace.c +13 -9
@@ -73,7 +73,7 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr)if (offset < -SZ_128M || offset >=SZ_128M ) {#ifdef CONFIG_ARM64_MODULE_PLTS -struct plt_entry trampoline; +struct plt_entry trampoline, *dst;struct module *mod;/*
@@ -106,23 +106,27 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr)* to check if the actual opcodes are in fact identical, * regardless of the offset in memory so use memcmp() instead. */ - trampoline =get_plt_entry (addr, mod->arch.ftrace_trampoline); -if (memcmp (mod->arch.ftrace_trampoline, &trampoline, -sizeof (trampoline))) { -if (plt_entry_is_initialized (mod->arch.ftrace_trampoline)) { + dst = mod->arch.ftrace_trampoline; + trampoline =get_plt_entry (addr, dst); +if (memcmp (dst, &trampoline,sizeof (trampoline))) { +if (plt_entry_is_initialized (dst)) {pr_err ("ftrace: far branches to multiple entry points unsupported inside a single module\n" );return -EINVAL ; }/* point the trampoline to our ftrace entry point */ module_disable_ro (mod); - *mod->arch.ftrace_trampoline = trampoline; + *dst = trampoline;module_enable_ro (mod, true); -/* update trampoline before patching in the branch */ -smp_wmb (); +/* +* Ensure updated trampoline is visible to instruction +* fetch before we patch in the branch. +*/ +__flush_icache_range ((unsigned long )&dst[0 ], + (unsigned long )&dst[1 ]); } - addr = (unsigned long )(void *)mod->arch.ftrace_trampoline; + addr = (unsigned long )dst;#else /* CONFIG_ARM64_MODULE_PLTS */ return -EINVAL ;#endif /* CONFIG_ARM64_MODULE_PLTS */
Signature · 1 line
2.11.0
Quoted 12 lines · Will Deacon
The initial support for dynamic ftrace trampolines in modules made use of an indirect branch which loaded its target from the beginning of a special section (
e71a4e1bebaf7("arm64: ftrace: add support for far branches to dynamic ftrace")). Since no instructions were being patched, no cache maintenance was needed. However, later inbe0f272bfc83("arm64:
ftrace: emit ftrace-mod.o contents through code") this code was reworked to output the trampoline instructions directly into the PLT entry but, unfortunately, the necessary cache maintenance was overlooked.Add a call to
__flush_icache_range()after writing the new trampoline instructions but before patching in the branch to the trampoline.
Quoted 6 lines
Cc Ard Biesheuvel <[email protected]>Cc Catalin Marinas <[email protected]>Cc James Morse <[email protected]>Cc <[email protected]>Fixes be0f272bfc83 ("arm64: ftrace: emit ftrace-mod.o contents through code")Signed-off-by Will Deacon <[email protected]>
Queued for 5.3. Thanks.
Signature · 1 line
Catalin