Fix ftrace for livepatch + BPF fexit programs

When livepatch is attached to the same function as bpf trampoline with a fexit program, bpf trampoline code calls register_ftrace_direct() twice. The first time will fail with -EAGAIN, and the second time it will succeed. This requires register_ftrace_direct() to unregister the address on the first attempt. Otherwise, the bpf trampoline cannot attach. Here is an easy way to reproduce this issue:

insmod samples/livepatch/livepatch-sample.ko
bpftrace -e 'fexit:cmdline_proc_show {}'
ERROR: Unable to attach probe: fexit:vmlinux:cmdline_proc_show...

Fix this by cleaning up the hash when register_ftrace_function_nolock hits errors.

Also, move the code that resets ops->func and ops->trampoline to the error path of register_ftrace_direct().

Fixes d05cb470663a ("ftrace: Fix modification of direct_function hash while in use")

Cc [email protected] # v6.6+

Reported-by Andrey Grodzovsky <[email protected]>

Closes https://lore.kernel.org/live-patching/[email protected]/ open here

Cc Steven Rostedt (Google) <[email protected]>

Cc Masami Hiramatsu (Google) <[email protected]>

Acked-and-tested-by Andrey Grodzovsky <[email protected]>

Signed-off-by Song Liu <[email protected]>

---
 kernel/bpf/trampoline.c | 5 -----
 kernel/trace/ftrace.c   | 6 ++++++
 2 files changed, 6 insertions(+), 5 deletions(-)

kernel/bpf/trampoline.c +0 -5

index 5949095e51c3..f2cb0b097093 100644
@@ -479,11 +479,6 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
 		 * BPF_TRAMP_F_SHARE_IPMODIFY is set, we can generate the
 		 * trampoline again, and retry register.
 		 */
-		/* reset fops->func and fops->trampoline for re-register */
-		tr->fops->func = NULL;
-		tr->fops->trampoline = 0;
-
-		/* free im memory and reallocate later */
 		bpf_tramp_image_free(im);
 		goto again;
 	}

kernel/trace/ftrace.c +0 -0

index 42bd2ba68a82..725c224fb4e6 100644

kernel/trace/ftrace.c +2 -0

+		remove_direct_functions_hash(hash, addr);
+	}
 
  out_unlock:
 	mutex_unlock(&direct_mutex);

Signature · 1 line

2.47.3

Quoted 26 lines

kernel/bpf/trampoline.c +0 -5

index 5949095e51c3..f2cb0b097093 100644
@@ -479,11 +479,6 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
 * BPF_TRAMP_F_SHARE_IPMODIFY is set, we can generate the
 * trampoline again, and retry register.
 */
- /* reset fops->func and fops->trampoline for re-register */
- tr->fops->func = NULL;
- tr->fops->trampoline = 0;
-
- /* free im memory and reallocate later */
 bpf_tramp_image_free(im);
 goto again;
 }

kernel/trace/ftrace.c +0 -0

index 42bd2ba68a82..725c224fb4e6 100644
@@ -6048,6 +6048,12 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
 ops->direct_call = addr;
 

kernel/trace/ftrace.c int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) · line 6054

Quoted 5 lines

kernel/trace/ftrace.c +4 -0

 err = register_ftrace_function_nolock(ops);
+ if (err) {
+ /* cleanup for possible another register call */
+ ops->func = NULL;
+ ops->trampoline = 0;

nit, we could cleanup also flags and direct_call just to be complete, but at the same time it does not seem to affect anything

kernel/trace/ftrace.c int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) · line 6057

Quoted 13 lines · On Sun, 26 Oct 2025 13:54:43 -0700 Song Liu <[email protected]> wrote:

kernel/trace/ftrace.c +6 -0

@@ -6048,6 +6048,12 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
 ops->direct_call = addr;
 
 err = register_ftrace_function_nolock(ops);
+ if (err) {
+ /* cleanup for possible another register call */
+ ops->func = NULL;
+ ops->trampoline = 0;
+ remove_direct_functions_hash(hash, addr);
+ }
 

As you AI bot noticed that it was missing what unregister_ftrace_direct() does, instead, can we make a helper function that both use? This way it will not get out of sync again.

static void reset_direct(struct ftrace_ops *ops, unsigned long addr) {
struct ftrace_hash *hash = ops->func_hash->filter_hash;

ops->func = NULL;
ops->trampoline = 0;
remove_direct_functions_hash(hash, addr);
}

Then we could have:

kernel/trace/ftrace.c +1 -3

index 0c91247a95ab..51c3f5d46fde 100644
@@ -6062,7 +6062,7 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
 
 err = register_ftrace_function_nolock(ops);
 if (err)
- remove_direct_functions_hash(hash, addr);
+ reset_direct(ops, addr);
 
 out_unlock:
 mutex_unlock(&direct_mutex);

@@ -6095,7 +6095,6 @@ EXPORT_SYMBOL_GPL(register_ftrace_direct);
 int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr,
    bool free_filters)
 {
- struct ftrace_hash *hash = ops->func_hash->filter_hash;
 int err;
 
 if (check_direct_multi(ops))

@@ -6105,13 +6104,9 @@ int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr,
 
 mutex_lock(&direct_mutex);
 err = unregister_ftrace_function(ops);
- remove_direct_functions_hash(hash, addr);

livepatch and BPF trampoline are two special users of ftrace. livepatch uses ftrace with IPMODIFY flag and BPF trampoline uses ftrace direct functions. When livepatch and BPF trampoline with fexit programs attach to the same kernel function, BPF trampoline needs to call into the patched version of the kernel function.

1/3 and 2/3 of this patchset fix two issues with livepatch + fexit cases, one in the register_ftrace_direct path, the other in the modify_ftrace_direct path.

3/3 adds selftests for both cases.

---

Changes v2 => v3:
1. Incorporate feedback by AI, which also fixes build error reported by Steven and kernel test robot.

v2: https://lore.kernel.org/bpf/[email protected]/

Changes v1 => v2:
1. Target bpf tree. (Alexei)
2. Bring back the FTRACE_WARN_ON in __ftrace_hash_update_ipmodify for valid code paths. (Steven)
3. Update selftests with cleaner way to find livepatch-sample.ko. (offlline discussion with Ihor)

v1: https://lore.kernel.org/bpf/[email protected]/

Song Liu (3):
ftrace: Fix BPF fexit with livepatch
ftrace: bpf: Fix IPMODIFY + DIRECT in modify_ftrace_direct() selftests/bpf: Add tests for livepatch + bpf trampoline

 kernel/bpf/trampoline.c                       |   5 -
 kernel/trace/ftrace.c                         |  46 ++++++--
 tools/testing/selftests/bpf/config            |   3 +
 .../bpf/prog_tests/livepatch_trampoline.c     | 107 ++++++++++++++++++
 .../bpf/progs/livepatch_trampoline.c          |  30 +++++
 5 files changed, 177 insertions(+), 14 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/livepatch_trampoline.c
 create mode 100644 tools/testing/selftests/bpf/progs/livepatch_trampoline.c

--
2.47.3