powerpc: ftrace updates

Early RFC.

This series attempts to address couple of issues with the existing support for ftrace on powerpc, with a view towards improving performance when ftrace is not enabled. See patch 6 for more details.

Patches 7 and 8 implement support for ftrace direct calls, through adding support for DYNAMIC_FTRACE_WITH_CALL_OPS.

The first 5 patches are minor cleanups and updates, and can go in separately.

This series depends on Benjamin Gray's series adding support for patch_ulong().

I have lightly tested this patch set and it looks to be working well. As described in patch 6, context_switch microbenchmark shows an improvement of ~6% with this series with ftrace disabled. Performance when ftrace is enabled reduces due to how DYNAMIC_FTRACE_WITH_CALL_OPS works, and due to support for direct calls. Some of that can hopefully be improved, if this approach is otherwise ok.

- Naveen

Naveen N Rao (8):
powerpc/ftrace: Fix indentation in ftrace.h powerpc/ftrace: Unify 32-bit and 64-bit ftrace entry code powerpc/ftrace: Remove nops after the call to ftrace_stub powerpc/kprobes: Use ftrace to determine if a probe is at function
entry
powerpc/ftrace: Update and move function profile instructions
out-of-line
powerpc/ftrace: Add support for DYNAMIC_FTRACE_WITH_CALL_OPS powerpc/ftrace: Add support for DYNAMIC_FTRACE_WITH_DIRECT_CALLS samples/ftrace: Add support for ftrace direct samples on powerpc

Sathvika Vasireddy (1):
powerpc/Kconfig: Select FUNCTION_ALIGNMENT_4B

 arch/powerpc/Kconfig                        |   6 +
 arch/powerpc/Makefile                       |   6 +-
 arch/powerpc/include/asm/code-patching.h    |  15 +-
 arch/powerpc/include/asm/ftrace.h           |  35 ++-
 arch/powerpc/include/asm/linkage.h          |   3 -
 arch/powerpc/kernel/asm-offsets.c           |   7 +
 arch/powerpc/kernel/kprobes.c               |  69 +++++-
 arch/powerpc/kernel/trace/ftrace.c          | 231 ++++++++++++++++----
 arch/powerpc/kernel/trace/ftrace_entry.S    | 182 +++++++++++----
 samples/ftrace/ftrace-direct-modify.c       |  94 +++++++-
 samples/ftrace/ftrace-direct-multi-modify.c | 110 +++++++++-
 samples/ftrace/ftrace-direct-multi.c        |  64 +++++-
 samples/ftrace/ftrace-direct-too.c          |  72 +++++-
 samples/ftrace/ftrace-direct.c              |  61 +++++-
 14 files changed, 845 insertions(+), 110 deletions(-)

base-commit: 9a15ae60f2c9707433b01e55815cd9142be102b2
prerequisite-patch-id: 38d3e705bf2e27cfa5e3ba369a6ded84ba6615c2
prerequisite-patch-id: 609d292e054b2396b603890522a940fa0bdfb6d8
prerequisite-patch-id: 6f7213fb77b1260defbf43be0e47bff9c80054cc
prerequisite-patch-id: f2328625ae2193c3c8e336b154b62030940cece8

2.43.0

Replace seven spaces with a tab character to fix an indentation issue reported by the kernel test robot.

---
 arch/powerpc/include/asm/ftrace.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 9e5a39b6a311..1ebd2ca97f12 100644
@@ -25,7 +25,7 @@ static inline unsigned long ftrace_call_adjust(unsigned long addr)
 	if (IS_ENABLED(CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY))
 		addr += MCOUNT_INSN_SIZE;
 
-       return addr;
+	return addr;
 }
 
 unsigned long prepare_ftrace_return(unsigned long parent, unsigned long ip,

2.43.0

On 32-bit powerpc, gcc generates a three instruction sequence for function profiling:

	mflr	r0
	stw	r0, 4(r1)
	bl	_mcount

On kernel boot, the call to _mcount() is nop-ed out, to be patched back in when ftrace is actually enabled. The 'stw' instruction therefore is not necessary unless ftrace is enabled. Nop it out during ftrace init.

When ftrace is enabled, we want the 'stw' so that stack unwinding works properly. Perform the same within the ftrace handler, similar to 64-bit powerpc.

For 64-bit powerpc, early versions of gcc used to emit a three instruction sequence for function profiling (with -mprofile-kernel) with a 'std' instruction to mimic the 'stw' above. Address that scenario also by nop-ing out the 'std' instruction during ftrace init.

---
 arch/powerpc/kernel/trace/ftrace.c       | 6 ++++--
 arch/powerpc/kernel/trace/ftrace_entry.S | 4 ++--
 2 files changed, 6 insertions(+), 4 deletions(-)

index 82010629cf88..2956196c98ff 100644
@@ -229,13 +229,15 @@ int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec)
 		/* Expected sequence: 'mflr r0', 'stw r0,4(r1)', 'bl _mcount' */
 		ret = ftrace_validate_inst(ip - 8, ppc_inst(PPC_RAW_MFLR(_R0)));
 		if (!ret)
-			ret = ftrace_validate_inst(ip - 4, ppc_inst(PPC_RAW_STW(_R0, _R1, 4)));
+			ret = ftrace_modify_code(ip - 4, ppc_inst(PPC_RAW_STW(_R0, _R1, 4)),
+						 ppc_inst(PPC_RAW_NOP()));
 	} else if (IS_ENABLED(CONFIG_MPROFILE_KERNEL)) {
 		/* Expected sequence: 'mflr r0', ['std r0,16(r1)'], 'bl _mcount' */
 		ret = ftrace_read_inst(ip - 4, &old);
 		if (!ret && !ppc_inst_equal(old, ppc_inst(PPC_RAW_MFLR(_R0)))) {
 			ret = ftrace_validate_inst(ip - 8, ppc_inst(PPC_RAW_MFLR(_R0)));
-			ret |= ftrace_validate_inst(ip - 4, ppc_inst(PPC_RAW_STD(_R0, _R1, 16)));
+			ret |= ftrace_modify_code(ip - 4, ppc_inst(PPC_RAW_STD(_R0, _R1, 16)),
+						  ppc_inst(PPC_RAW_NOP()));
 		}
 	} else {
 		return -EINVAL;

index 40677416d7b2..17d1ed3d0b40 100644
@@ -33,6 +33,8 @@
  * and then arrange for the ftrace function to be called.
  */
 .macro	ftrace_regs_entry allregs
+	/* Save the original return address in A's stack frame */
+	PPC_STL		r0, LRSAVE(r1)
 	/* Create a minimal stack frame for representing B */
 	PPC_STLU	r1, -STACK_FRAME_MIN_SIZE(r1)
 

@@ -44,8 +46,6 @@
 	SAVE_GPRS(3, 10, r1)
 
 #ifdef CONFIG_PPC64
-	/* Save the original return address in A's stack frame */
-	std	r0, LRSAVE+SWITCH_FRAME_SIZE+STACK_FRAME_MIN_SIZE(r1)
 	/* Ok to continue? */
 	lbz	r3, PACA_FTRACE_ENABLED(r13)
 	cmpdi	r3, 0

2.43.0

ftrace_stub is within the same CU, so there is no need for a subsequent nop instruction.

---
 arch/powerpc/kernel/trace/ftrace_entry.S | 2 --
 1 file changed, 2 deletions(-)

index 17d1ed3d0b40..244a1c7bb1e8 100644
@@ -162,7 +162,6 @@ _GLOBAL(ftrace_regs_caller)
 .globl ftrace_regs_call
 ftrace_regs_call:
 	bl	ftrace_stub
-	nop
 	ftrace_regs_exit 1
 
 _GLOBAL(ftrace_caller)

@@ -171,7 +170,6 @@ _GLOBAL(ftrace_caller)
 .globl ftrace_call
 ftrace_call:
 	bl	ftrace_stub
-	nop
 	ftrace_regs_exit 0
 
 _GLOBAL(ftrace_stub)

2.43.0

From: Sathvika Vasireddy <[email protected]>

Commit d49a0626216b95 ("arch: Introduce CONFIG_FUNCTION_ALIGNMENT") introduced a generic function-alignment infrastructure. Move to using FUNCTION_ALIGNMENT_4B on powerpc, to use the same alignment as that of the existing _GLOBAL macro.

---
 arch/powerpc/Kconfig               | 1 +
 arch/powerpc/include/asm/linkage.h | 3 ---
 2 files changed, 1 insertion(+), 3 deletions(-)

index 6f105ee4f3cf..318e5c1b7454 100644
@@ -189,6 +189,7 @@ config PPC
 	select EDAC_ATOMIC_SCRUB
 	select EDAC_SUPPORT
 	select FTRACE_MCOUNT_USE_PATCHABLE_FUNCTION_ENTRY if ARCH_USING_PATCHABLE_FUNCTION_ENTRY
+	select FUNCTION_ALIGNMENT_4B
 	select GENERIC_ATOMIC64			if PPC32
 	select GENERIC_CLOCKEVENTS_BROADCAST	if SMP
 	select GENERIC_CMOS_UPDATE

index b88d1d2cf304..b71b9582e754 100644
@@ -4,9 +4,6 @@
 
 #include <asm/types.h>
 
-#define __ALIGN		.align 2
-#define __ALIGN_STR	".align 2"
-
 #ifdef CONFIG_PPC64_ELF_ABI_V1
 #define cond_syscall(x) \
 	asm ("\t.weak " #x "\n\t.set " #x ", sys_ni_syscall\n"		\

2.43.0

Rather than hard-coding the offset into a function to be used to determine if a kprobe is at function entry, use ftrace_location() to determine the ftrace location within the function and categorize all instructions till that offset to be function entry.

For functions that cannot be traced, we fall back to using a fixed offset of 8 (two instructions) to categorize a probe as being at function entry for 64-bit elfv2.

---
 arch/powerpc/kernel/kprobes.c | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)

index b20ee72e873a..42665dfab59e 100644
@@ -105,24 +105,22 @@ kprobe_opcode_t *kprobe_lookup_name(const char *name, unsigned int offset)
 	return addr;
 }
 
-static bool arch_kprobe_on_func_entry(unsigned long offset)
+static bool arch_kprobe_on_func_entry(unsigned long addr, unsigned long offset)
 {
-#ifdef CONFIG_PPC64_ELF_ABI_V2
-#ifdef CONFIG_KPROBES_ON_FTRACE
-	return offset <= 16;
-#else
-	return offset <= 8;
-#endif
-#else
+	unsigned long ip = ftrace_location(addr);
+
+	if (ip)
+		return offset <= (ip - addr);
+	if (IS_ENABLED(CONFIG_PPC64_ELF_ABI_V2))
+		return offset <= 8;
 	return !offset;
-#endif
 }
 
 /* XXX try and fold the magic of kprobe_lookup_name() in this */
 kprobe_opcode_t *arch_adjust_kprobe_addr(unsigned long addr, unsigned long offset,
 					 bool *on_func_entry)
 {
-	*on_func_entry = arch_kprobe_on_func_entry(offset);
+	*on_func_entry = arch_kprobe_on_func_entry(addr, offset);
 	return (kprobe_opcode_t *)(addr + offset);
 }
 

2.43.0

OK, so this can cover both KPROBES_ON_FTRACE=y/n cases and the function is traced by ftrace or not.

Looks good to me.

Thanks,

Masami Hiramatsu (Google) <[email protected]>

Function profile sequence on powerpc includes two instructions at the beginning of each function:

	mflr	r0
	bl	ftrace_caller

The call to ftrace_caller() gets nop'ed out during kernel boot and is patched in when ftrace is enabled.

There are two issues with this:
1. The 'mflr r0' instruction at the beginning of each function remains even though ftrace is not being used.
2. When ftrace is activated, we return from ftrace_caller() with a bctr instruction to preserve r0 and LR, resulting in the link stack becoming unbalanced.

To address (1), we have tried to nop'out the 'mflr r0' instruction when nop'ing out the call to ftrace_caller() and restoring it when enabling ftrace. But, that required additional synchronization slowing down ftrace activation. It also left an additional nop instruction at the beginning of each function and that wasn't desirable on 32-bit powerpc.

Instead of that, move the function profile sequence out-of-line leaving a single nop at function entry. On ftrace activation, the nop is changed to an unconditional branch to the out-of-line sequence that in turn calls ftrace_caller(). This removes the need for complex synchronization during ftrace activation and simplifies the code. More importantly, this improves performance of the kernel when ftrace is not in use.

To address (2), change the ftrace trampoline to return with a 'blr' instruction with the original return address in r0 intact. Then, an additional 'mtlr r0' instruction in the function profile sequence can move the correct return address back to LR.

With the above two changes, the function profile sequence now looks like the following:

[func: # GEP -- 64-bit powerpc, optional

	addis	r2,r12,imm1
	addi	r2,r2,imm2]
  tramp:
	mflr	r0
	bl	ftrace_caller
	mtlr	r0
	b	func
	nop
	[nop]	# 64-bit powerpc only

func: # LEP
nop

On 32-bit powerpc, the ftrace mcount trampoline is now completely outside the function. This is also the case on 64-bit powerpc for functions that do not need a GEP. However, for functions that need a GEP, the additional instructions are inserted between the GEP and the LEP. Since we can only have a fixed number of instructions between GEP and LEP, we choose to emit 6 instructions. Four of those instructions are used for the function profile sequence and two instruction slots are reserved for implementing support for DYNAMIC_FTRACE_WITH_CALL_OPS. On 32-bit powerpc, we emit one additional nop for this purpose resulting in a total of 5 nops before function entry.

To enable ftrace, the nop at function entry is changed to an unconditional branch to 'tramp'. The call to ftrace_caller() may be updated to ftrace_regs_caller() depending on the registered ftrace ops. On 64-bit powerpc, we additionally change the instruction at 'tramp' to 'mflr r0' from an unconditional branch back to func+4. This is so that functions entered through the GEP can skip the function profile sequence unless ftrace is enabled.

With the context_switch microbenchmark on a P9 machine, there is a performance improvement of ~6% with this patch applied, going from 650k context switches to 690k context switches without ftrace enabled. With ftrace enabled, the performance was similar at 86k context switches.

The downside of this approach is the increase in vmlinux size, especially on 32-bit powerpc. We now emit 3 additional instructions for each function (excluding the one or two instructions for supporting DYNAMIC_FTRACE_WITH_CALL_OPS). On 64-bit powerpc with the current implementation of -fpatchable-function-entry though, this is not avoidable since we are forced to emit 6 instructions between the GEP and the LEP even if we are to only support DYNAMIC_FTRACE_WITH_CALL_OPS.

---
 arch/powerpc/Makefile                    |   6 +-
 arch/powerpc/include/asm/code-patching.h |  15 ++-
 arch/powerpc/include/asm/ftrace.h        |  18 ++-
 arch/powerpc/kernel/kprobes.c            |  51 +++++++-
 arch/powerpc/kernel/trace/ftrace.c       | 149 ++++++++++++++++++-----
 arch/powerpc/kernel/trace/ftrace_entry.S |  54 ++++++--
 6 files changed, 246 insertions(+), 47 deletions(-)

index f19dbaa1d541..91ef34be8eb9 100644
@@ -145,7 +145,11 @@ CFLAGS-$(CONFIG_PPC32)	+= $(call cc-option,-mno-readonly-in-sdata)
 ifdef CONFIG_FUNCTION_TRACER
 ifdef CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY
 KBUILD_CPPFLAGS	+= -DCC_USING_PATCHABLE_FUNCTION_ENTRY
-CC_FLAGS_FTRACE := -fpatchable-function-entry=2
+ifdef CONFIG_PPC32
+CC_FLAGS_FTRACE := -fpatchable-function-entry=6,5
+else
+CC_FLAGS_FTRACE := -fpatchable-function-entry=7,6
+endif
 else
 CC_FLAGS_FTRACE := -pg
 ifdef CONFIG_MPROFILE_KERNEL

index 84f6ccd7de7a..9a54bb9e0dde 100644
@@ -185,10 +185,21 @@ static inline unsigned long ppc_function_entry(void *func)
 	 */
 	if ((((*insn & OP_RT_RA_MASK) == ADDIS_R2_R12) ||
 	     ((*insn & OP_RT_RA_MASK) == LIS_R2)) &&
-	    ((*(insn+1) & OP_RT_RA_MASK) == ADDI_R2_R2))
+	    ((*(insn+1) & OP_RT_RA_MASK) == ADDI_R2_R2)) {
+#ifdef CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY
+		/*
+		 * Heuristic: look for the 'mtlr r0' instruction assuming ftrace init is done.
+		 * If it is not found, look for two consecutive nops after the GEP.
+		 * Longer term, we really should be parsing the symbol table to determine LEP.
+		 */
+		if ((*(insn+4) == PPC_RAW_MTLR(_R0)) ||
+		    ((*(insn+2) == PPC_RAW_NOP() && *(insn+3) == PPC_RAW_NOP())))
+			return (unsigned long)(insn + 8);
+#endif
 		return (unsigned long)(insn + 2);
-	else
+	} else {
 		return (unsigned long)func;
+	}
 #elif defined(CONFIG_PPC64_ELF_ABI_V1)
 	/*
 	 * On PPC64 ABIv1 the function pointer actually points to the

index 1ebd2ca97f12..d9b99781bea3 100644
@@ -11,10 +11,20 @@
 #define HAVE_FUNCTION_GRAPH_RET_ADDR_PTR
 
 /* Ignore unused weak functions which will have larger offsets */
-#if defined(CONFIG_MPROFILE_KERNEL) || defined(CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY)
-#define FTRACE_MCOUNT_MAX_OFFSET	16
+#if defined(CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY)
+#if defined(CONFIG_PPC64)
+#define FTRACE_MCOUNT_MAX_OFFSET	(MCOUNT_INSN_SIZE * 8)
+#define FTRACE_MCOUNT_TRAMP_OFFSET	(MCOUNT_INSN_SIZE * 6)
+#else
+#define FTRACE_MCOUNT_MAX_OFFSET	0
+#define FTRACE_MCOUNT_TRAMP_OFFSET	(MCOUNT_INSN_SIZE * 5)
+#endif /* CONFIG_PPC64 */
+#elif defined(CONFIG_MPROFILE_KERNEL)
+#define FTRACE_MCOUNT_MAX_OFFSET	(MCOUNT_INSN_SIZE * 4)
+#define FTRACE_MCOUNT_TRAMP_OFFSET	0
 #elif defined(CONFIG_PPC32)
-#define FTRACE_MCOUNT_MAX_OFFSET	8
+#define FTRACE_MCOUNT_MAX_OFFSET	(MCOUNT_INSN_SIZE * 2)
+#define FTRACE_MCOUNT_TRAMP_OFFSET	0
 #endif
 
 #ifndef __ASSEMBLY__

@@ -23,7 +33,7 @@ extern void _mcount(void);
 static inline unsigned long ftrace_call_adjust(unsigned long addr)
 {
 	if (IS_ENABLED(CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY))
-		addr += MCOUNT_INSN_SIZE;
+		addr += FTRACE_MCOUNT_TRAMP_OFFSET;
 
 	return addr;
 }

2.43.0

Le 08/12/2023 à 17:30, Naveen N Rao a écrit :

Wondering how significant that context_switch micorbenchmark is.

I ran it on both mpc885 and mpc8321 and I'm a bit puzzled by some of the results:

# ./context_switch --no-fp
Using threads with yield on cpus 0/0 touching FP:no altivec:no vector:no 
vdso:no

On 885, I get the following results before and after your patch.

CONFIG_FTRACE not selected : 44,9k
CONFIG_FTRACE selected, before : 32,8k
CONFIG_FTRACE selected, after : 33,6k

All this is with CONFIG_INIT_STACK_ALL_ZERO which is the default. But when I select CONFIG_INIT_STACK_NONE, the CONFIG_FTRACE not selected result is only 34,4.

On 8321:

CONFIG_FTRACE not selected : 100,3k
CONFIG_FTRACE selected, before : 72,5k
CONFIG_FTRACE selected, after : 116k

So the results look odd to me.

The increase is almost 5% on the few 32 bits defconfig I have tested. That's a lot.

On 32 bits powerpc, only the e500 has a link stack that could end up being unbalanced. Could we keep the bctr and avoid the mtlr and the jump in the trampoline ?

On 8xx a NOP is one cycle, a taken branch is 2 cycles, but the second cycle is a bubble that most of the time gets filled by following operations. On the 83xx, branches and NOPs are supposed to be seamless.

So is that out-of-line trampoline really worth it ? Maybe keeping the ftrace instructions at the begining and just replacing the mflr by an jump when ftrace is off would help reduce the ftrace insns by one more instruction.

BTW, CONFIG_FTRACE just enables the tracing system (I would like to change that to CONFIG_TRACING, but not sure if I can without breaking .configs all over the place).

The nops for ftrace is enabled with CONFIG_FUNCTION_TRACER.

-- Steve

Le 21/12/2023 à 15:25, Steven Rostedt a écrit :

Yes I selected both CONFIG_FTRACE and CONFIG_FUNCTION_TRACER

Christophe

That's indeed odd, though it looks to be showing good improvement. Are those numbers with/without the function tracer enabled?

Do you see more reasonable numbers if you use a different FUNCTION_ALIGNMENT?

Indeed. Note that one of those nops is for DYN_FTRACE_WITH_CALL_OPS, which we will need regardless.

Moving the ftrace mcount sequence out of line will alone need 2 additional nops. 'mtlr r0' for balancing the link stack costs us one more nop.

I was looking forward to your feedback w.r.t 32-bit powerpc since I couldn't test that.

We can certainly retain the existing behavior for ppc32, though it might make it harder to share the code base with ppc64.

The primary downside with the 'mflr r0' at function entry is that patching it out (or replacing it with a branch) will need additional synchronization.

I'm out on vacation till end of the year. I plan on doing more tests once I am back to understand if the performance benefit is worth the increase in vmlinux size.

Thanks,
Naveen

Implement support for DYNAMIC_FTRACE_WITH_CALL_OPS similar to the arm64 implementation.

This works by patching-in a pointer to an associated ftrace_ops structure before each traceable function. If multiple ftrace_ops are associated with a call site, then a special ftrace_list_ops is used to enable iterating over all the registered ftrace_ops. If no ftrace_ops are associated with a call site, then a special ftrace_nop_ops structure is used to render the ftrace call as a no-op. ftrace trampoline can then read the associated ftrace_ops for a call site by loading from an offset from the LR, and branch directly to the associated function.

The primary advantage with this approach is that we don't have to iterate over all the registered ftrace_ops for call sites that have a single ftrace_ops registered. This is the equivalent of implementing support for dynamic ftrace trampolines, which set up a special ftrace trampoline for each registered ftrace_ops and have individual call sites branch into those directly.

A secondary advantage is that this gives us a way to add support for direct ftrace callers without having to resort to using stubs. The address of the direct call trampoline can be loaded from the ftrace_ops structure.

To support this, we utilize the space between the existing function profile sequence and the function entry. During ftrace activation, we update this location with the associated ftrace_ops pointer. Then, on ftrace entry, we load from this location and call into ftrace_ops->func().

For 64-bit powerpc, we also select FUNCTION_ALIGNMENT_8B so that the ftrace_ops pointer is double word aligned and can be updated atomically.

---
 arch/powerpc/Kconfig                     |  2 +
 arch/powerpc/kernel/asm-offsets.c        |  4 ++
 arch/powerpc/kernel/trace/ftrace.c       | 58 ++++++++++++++++++++++++
 arch/powerpc/kernel/trace/ftrace_entry.S | 39 +++++++++++-----
 4 files changed, 91 insertions(+), 12 deletions(-)

index 318e5c1b7454..c8ecc9dcc914 100644
@@ -190,6 +190,7 @@ config PPC
 	select EDAC_SUPPORT
 	select FTRACE_MCOUNT_USE_PATCHABLE_FUNCTION_ENTRY if ARCH_USING_PATCHABLE_FUNCTION_ENTRY
 	select FUNCTION_ALIGNMENT_4B
+	select FUNCTION_ALIGNMENT_8B		if PPC64 && DYNAMIC_FTRACE_WITH_CALL_OPS
 	select GENERIC_ATOMIC64			if PPC32
 	select GENERIC_CLOCKEVENTS_BROADCAST	if SMP
 	select GENERIC_CMOS_UPDATE

@@ -233,6 +234,7 @@ config PPC
 	select HAVE_DEBUG_STACKOVERFLOW
 	select HAVE_DYNAMIC_FTRACE
 	select HAVE_DYNAMIC_FTRACE_WITH_ARGS	if ARCH_USING_PATCHABLE_FUNCTION_ENTRY || MPROFILE_KERNEL || PPC32
+	select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS if ARCH_USING_PATCHABLE_FUNCTION_ENTRY
 	select HAVE_DYNAMIC_FTRACE_WITH_REGS	if ARCH_USING_PATCHABLE_FUNCTION_ENTRY || MPROFILE_KERNEL || PPC32
 	select HAVE_EBPF_JIT
 	select HAVE_EFFICIENT_UNALIGNED_ACCESS

index 9f14d95b8b32..8b8a39b57a9f 100644
@@ -676,5 +676,9 @@ int main(void)
 	DEFINE(BPT_SIZE, BPT_SIZE);
 #endif
 
+#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS
+	OFFSET(FTRACE_OPS_FUNC, ftrace_ops, func);
+#endif
+
 	return 0;
 }

index 537c14b12904..4d1220c2e32f 100644
@@ -97,11 +97,6 @@
 	/* Save callee's TOC in the ABI compliant location */
 	std	r2, STK_GOT(r1)
 	LOAD_PACA_TOC()		/* get kernel TOC in r2 */
-	LOAD_REG_ADDR(r3, function_trace_op)
-	ld	r5,0(r3)
-#else
-	lis	r3,function_trace_op@ha
-	lwz	r5,function_trace_op@l(r3)
 #endif
 
 #ifdef CONFIG_LIVEPATCH_64

@@ -177,20 +172,40 @@
 #endif
 .endm
 
-_GLOBAL(ftrace_regs_caller)
-	ftrace_regs_entry 1
-	/* ftrace_call(r3, r4, r5, r6) */
+.macro ftrace_regs_func allregs
+#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS
+	PPC_LL	r5, -SZL(r3)
+	PPC_LL	r12, FTRACE_OPS_FUNC(r5)
+	mtctr	r12
+	bctrl
+#else
+#ifdef CONFIG_PPC64
+	LOAD_REG_ADDR(r5, function_trace_op)
+	ld	r5, 0(r5)
+#else
+	lis	r5, function_trace_op@ha
+	lwz	r5, function_trace_op@l(r5)
+#endif
+	.if \allregs == 1
 .globl ftrace_regs_call
 ftrace_regs_call:
+	.else
+.globl ftrace_call
+ftrace_call:
+	.endif
+	/* ftrace_call(r3, r4, r5, r6) */
 	bl	ftrace_stub
+#endif
+.endm
+
+_GLOBAL(ftrace_regs_caller)
+	ftrace_regs_entry 1
+	ftrace_regs_func 1
 	ftrace_regs_exit 1
 
 _GLOBAL(ftrace_caller)
 	ftrace_regs_entry 0
-	/* ftrace_call(r3, r4, r5, r6) */
-.globl ftrace_call
-ftrace_call:
-	bl	ftrace_stub
+	ftrace_regs_func 0
 	ftrace_regs_exit 0
 
 _GLOBAL(ftrace_stub)

2.43.0

Add support for DYNAMIC_FTRACE_WITH_DIRECT_CALLS similar to the arm64 implementation.

ftrace direct calls allow custom trampolines to be called into directly from function ftrace call sites, bypassing the ftrace trampoline completely. This functionality is currently utilized by BPF trampolines to hook into kernel function entries.

Since we have limited relative branch range, we support ftrace direct calls through support for DYNAMIC_FTRACE_WITH_CALL_OPS. In this approach, ftrace trampoline is not entirely bypassed. Rather, it is re-purposed into a stub that reads direct_call field from the associated ftrace_ops structure and branches into that, if it is not NULL. For this, it is sufficient if we can ensure that the ftrace trampoline is reachable from all traceable functions.

When multiple ftrace_ops are associated with a call site, we utilize a call back to set pt_regs->orig_gpr3 that can then be tested on the return path from the ftrace trampoline to branch into the direct caller.

---
 arch/powerpc/Kconfig                     |  1 +
 arch/powerpc/include/asm/ftrace.h        | 15 ++++
 arch/powerpc/kernel/asm-offsets.c        |  3 +
 arch/powerpc/kernel/trace/ftrace_entry.S | 99 ++++++++++++++++++------
 4 files changed, 93 insertions(+), 25 deletions(-)

index c8ecc9dcc914..4fe04fdca33a 100644
@@ -235,6 +235,7 @@ config PPC
 	select HAVE_DYNAMIC_FTRACE
 	select HAVE_DYNAMIC_FTRACE_WITH_ARGS	if ARCH_USING_PATCHABLE_FUNCTION_ENTRY || MPROFILE_KERNEL || PPC32
 	select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS if ARCH_USING_PATCHABLE_FUNCTION_ENTRY
+	select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS if HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS
 	select HAVE_DYNAMIC_FTRACE_WITH_REGS	if ARCH_USING_PATCHABLE_FUNCTION_ENTRY || MPROFILE_KERNEL || PPC32
 	select HAVE_EBPF_JIT
 	select HAVE_EFFICIENT_UNALIGNED_ACCESS

index d9b99781bea3..986c4fffb9ec 100644
@@ -93,6 +93,21 @@ struct ftrace_ops;
 #define ftrace_graph_func ftrace_graph_func
 void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,
 		       struct ftrace_ops *op, struct ftrace_regs *fregs);
+
+#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
+/*
+ * When an ftrace registered caller is tracing a function that is also set by a
+ * register_ftrace_direct() call, it needs to be differentiated in the
+ * ftrace_caller trampoline so that the direct call can be invoked after the
+ * other ftrace ops. To do this, place the direct caller in the orig_gpr3 field
+ * of pt_regs. This tells ftrace_caller that there's a direct caller.
+ */
+static inline void arch_ftrace_set_direct_caller(struct ftrace_regs *fregs, unsigned long addr)
+{
+	struct pt_regs *regs = &fregs->regs;
+	regs->orig_gpr3 = addr;
+}
+#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */
 #endif
 #endif /* __ASSEMBLY__ */
 

index 8b8a39b57a9f..85da10726d98 100644
@@ -678,6 +678,9 @@ int main(void)
 
 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS
 	OFFSET(FTRACE_OPS_FUNC, ftrace_ops, func);
+#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
+	OFFSET(FTRACE_OPS_DIRECT_CALL, ftrace_ops, direct_call);
+#endif
 #endif
 
 	return 0;

2.43.0

Add powerpc 32-bit and 64-bit samples for ftrace direct. This serves to show the sample instruction sequence to be used by ftrace direct calls to adhere to the ftrace ABI.

On 64-bit powerpc, TOC setup requires some additional work.

---
 arch/powerpc/Kconfig                        |   2 +
 samples/ftrace/ftrace-direct-modify.c       |  94 ++++++++++++++++-
 samples/ftrace/ftrace-direct-multi-modify.c | 110 +++++++++++++++++++-
 samples/ftrace/ftrace-direct-multi.c        |  64 +++++++++++-
 samples/ftrace/ftrace-direct-too.c          |  72 ++++++++++++-
 samples/ftrace/ftrace-direct.c              |  61 ++++++++++-
 6 files changed, 398 insertions(+), 5 deletions(-)

index 4fe04fdca33a..28de3a5f3e98 100644
@@ -274,6 +274,8 @@ config PPC
 	select HAVE_REGS_AND_STACK_ACCESS_API
 	select HAVE_RELIABLE_STACKTRACE
 	select HAVE_RSEQ
+	select HAVE_SAMPLE_FTRACE_DIRECT	if HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
+	select HAVE_SAMPLE_FTRACE_DIRECT_MULTI	if HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
 	select HAVE_SETUP_PER_CPU_AREA		if PPC64
 	select HAVE_SOFTIRQ_ON_OWN_STACK
 	select HAVE_STACKPROTECTOR		if PPC32 && $(cc-option,-mstack-protector-guard=tls -mstack-protector-guard-reg=r2)

2.43.0

Patches 1, 3 and 4 applied to powerpc/next.

[1/9] powerpc/ftrace: Fix indentation in ftrace.h

      https://git.kernel.org/powerpc/c/2ec36570c3581285d15de672eaed10ce7e9218cd
[3/9] powerpc/ftrace: Remove nops after the call to ftrace_stub
      https://git.kernel.org/powerpc/c/ae24db43b3b427eb290b58d55179c32f0a7539d1
[4/9] powerpc/Kconfig: Select FUNCTION_ALIGNMENT_4B
      https://git.kernel.org/powerpc/c/b20f98e8b3deb50247603f0242ee2d1e38726635

cheers