Skip to contentSkip to conversation

Fix ftrace for livepatch + BPF fexit programs

PATCH Applied Song Liu 4 participants 8 messages Oct 27, 2025 – Nov 4, 2025

Subsystems (8)
Stable branch Kselftest BPF Tracing +4 more subsystems
lore
Share & tools
lore

Song Liu

author maintainer
#0 lore raw

v4 cover Fix ftrace for livepatch + BPF fexit programs

livepatch and BPF trampoline are two special users of ftrace. livepatch uses ftrace with IPMODIFY flag and BPF trampoline uses ftrace direct functions. When livepatch and BPF trampoline with fexit programs attach to the same kernel function, BPF trampoline needs to call into the patched version of the kernel function.

1/3 and 2/3 of this patchset fix two issues with livepatch + fexit cases, one in the register_ftrace_direct path, the other in the modify_ftrace_direct path.

3/3 adds selftests for both cases.

---

Changes v3 => v4:
1. Add helper reset_direct. (Steven)
2. Add Reviewed-by from Jiri.
3. Fix minor typo in comments.

v3: https://lore.kernel.org/bpf/[email protected]/

Changes v2 => v3:
1. Incorporate feedback by AI, which also fixes build error reported by Steven and kernel test robot.

v2: https://lore.kernel.org/bpf/[email protected]/

Changes v1 => v2:
1. Target bpf tree. (Alexei)
2. Bring back the FTRACE_WARN_ON in __ftrace_hash_update_ipmodify for valid code paths. (Steven)
3. Update selftests with cleaner way to find livepatch-sample.ko. (offlline discussion with Ihor)

v1: https://lore.kernel.org/bpf/[email protected]/

Song Liu (3):
ftrace: Fix BPF fexit with livepatch
ftrace: bpf: Fix IPMODIFY + DIRECT in modify_ftrace_direct() selftests/bpf: Add tests for livepatch + bpf trampoline

 kernel/bpf/trampoline.c                       |   5 -
 kernel/trace/ftrace.c                         |  60 +++++++---
 tools/testing/selftests/bpf/config            |   3 +
 .../bpf/prog_tests/livepatch_trampoline.c     | 107 ++++++++++++++++++
 .../bpf/progs/livepatch_trampoline.c          |  30 +++++
 5 files changed, 185 insertions(+), 20 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/livepatch_trampoline.c
 create mode 100644 tools/testing/selftests/bpf/progs/livepatch_trampoline.c

--
2.47.3

author maintainer
#1 lore raw

PATCH v4 1/3 ftrace: Fix BPF fexit with livepatch

When livepatch is attached to the same function as bpf trampoline with a fexit program, bpf trampoline code calls register_ftrace_direct() twice. The first time will fail with -EAGAIN, and the second time it will succeed. This requires register_ftrace_direct() to unregister the address on the first attempt. Otherwise, the bpf trampoline cannot attach. Here is an easy way to reproduce this issue:

insmod samples/livepatch/livepatch-sample.ko
bpftrace -e 'fexit:cmdline_proc_show {}'
ERROR: Unable to attach probe: fexit:vmlinux:cmdline_proc_show...

Fix this by cleaning up the hash when register_ftrace_function_nolock hits errors.

Also, move the code that resets ops->func and ops->trampoline to the error path of register_ftrace_direct(); and add a helper function reset_direct() in register_ftrace_direct() and unregister_ftrace_direct().

Fixes d05cb470663a ("ftrace: Fix modification of direct_function hash while in use")
Cc [email protected] # v6.6+
Reported-by Andrey Grodzovsky <[email protected]>
Closes https://lore.kernel.org/live-patching/[email protected]/ open here
Cc Steven Rostedt (Google) <[email protected]>
Cc Masami Hiramatsu (Google) <[email protected]>
Acked-and-tested-by Andrey Grodzovsky <[email protected]>
Signed-off-by Song Liu <[email protected]>
Reviewed-by Jiri Olsa <[email protected]>
---
 kernel/bpf/trampoline.c |  5 -----
 kernel/trace/ftrace.c   | 20 ++++++++++++++------
 2 files changed, 14 insertions(+), 11 deletions(-)
kernel/bpf/trampoline.c +0 -5
index 5949095e51c3..f2cb0b097093 100644
@@ -479,11 +479,6 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
 		 * BPF_TRAMP_F_SHARE_IPMODIFY is set, we can generate the
 		 * trampoline again, and retry register.
 		 */
-		/* reset fops->func and fops->trampoline for re-register */
-		tr->fops->func = NULL;
-		tr->fops->trampoline = 0;
-
-		/* free im memory and reallocate later */
 		bpf_tramp_image_free(im);
 		goto again;
 	}
kernel/trace/ftrace.c +14 -6
index 42bd2ba68a82..cbeb7e833131 100644
@@ -5953,6 +5953,17 @@ static void register_ftrace_direct_cb(struct rcu_head *rhp)
 	free_ftrace_hash(fhp);
 }
 
+static void reset_direct(struct ftrace_ops *ops, unsigned long addr)
+{
+	struct ftrace_hash *hash = ops->func_hash->filter_hash;
+
+	remove_direct_functions_hash(hash, addr);
+
+	/* cleanup for possible another register call */
+	ops->func = NULL;
+	ops->trampoline = 0;
+}
+
 /**
  * register_ftrace_direct - Call a custom trampoline directly
  * for multiple functions registered in @ops

@@ -6048,6 +6059,8 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
 	ops->direct_call = addr;
 
 	err = register_ftrace_function_nolock(ops);
+	if (err)
+		reset_direct(ops, addr);
 
  out_unlock:
 	mutex_unlock(&direct_mutex);

@@ -6080,7 +6093,6 @@ EXPORT_SYMBOL_GPL(register_ftrace_direct);
 int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr,
 			     bool free_filters)
 {
-	struct ftrace_hash *hash = ops->func_hash->filter_hash;
 	int err;
 
 	if (check_direct_multi(ops))

@@ -6090,13 +6102,9 @@ int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr,
 
 	mutex_lock(&direct_mutex);
 	err = unregister_ftrace_function(ops);
-	remove_direct_functions_hash(hash, addr);
+	reset_direct(ops, addr);
 	mutex_unlock(&direct_mutex);
 
-	/* cleanup for possible another register call */
-	ops->func = NULL;
-	ops->trampoline = 0;
-
 	if (free_filters)
 		ftrace_free_filter(ops);
 	return err;
Signature · 1 line
2.47.3
author maintainer
#2 lore raw

PATCH v4 2/3 ftrace: bpf: Fix IPMODIFY + DIRECT in modify_ftrace_direct()

In reply to Song Liu

ftrace_hash_ipmodify_enable() checks IPMODIFY and DIRECT ftrace_ops on the same kernel function. When needed, ftrace_hash_ipmodify_enable() calls ops->ops_func() to prepare the direct ftrace (BPF trampoline) to share the same function as the IPMODIFY ftrace (livepatch).

ftrace_hash_ipmodify_enable() is called in register_ftrace_direct() path, but not called in modify_ftrace_direct() path. As a result, the following operations will break livepatch:

1. Load livepatch to a kernel function;
2. Attach fentry program to the kernel function;
3. Attach fexit program to the kernel function.

After 3, the kernel function being used will not be the livepatched version, but the original version.

Fix this by adding __ftrace_hash_update_ipmodify() to __modify_ftrace_direct() and adjust some logic around the call.

Signed-off-by Song Liu <[email protected]>
Reviewed-by Jiri Olsa <[email protected]>
---
 kernel/trace/ftrace.c | 40 +++++++++++++++++++++++++++++++---------
 1 file changed, 31 insertions(+), 9 deletions(-)
kernel/trace/ftrace.c +31 -9
index cbeb7e833131..59cfacb8a5bb 100644
@@ -1971,7 +1971,8 @@ static void ftrace_hash_rec_enable_modify(struct ftrace_ops *ops)
  */
 static int __ftrace_hash_update_ipmodify(struct ftrace_ops *ops,
 					 struct ftrace_hash *old_hash,
-					 struct ftrace_hash *new_hash)
+					 struct ftrace_hash *new_hash,
+					 bool update_target)
 {
 	struct ftrace_page *pg;
 	struct dyn_ftrace *rec, *end = NULL;

@@ -2006,10 +2007,13 @@ static int __ftrace_hash_update_ipmodify(struct ftrace_ops *ops,
 		if (rec->flags & FTRACE_FL_DISABLED)
 			continue;
 
-		/* We need to update only differences of filter_hash */
+		/*
+		 * Unless we are updating the target of a direct function,
+		 * we only need to update differences of filter_hash
+		 */
 		in_old = !!ftrace_lookup_ip(old_hash, rec->ip);
 		in_new = !!ftrace_lookup_ip(new_hash, rec->ip);
-		if (in_old == in_new)
+		if (!update_target && (in_old == in_new))
 			continue;
 
 		if (in_new) {

@@ -2020,7 +2024,16 @@ static int __ftrace_hash_update_ipmodify(struct ftrace_ops *ops,
 				if (is_ipmodify)
 					goto rollback;
 
-				FTRACE_WARN_ON(rec->flags & FTRACE_FL_DIRECT);
+				/*
+				 * If this is called by __modify_ftrace_direct()
+				 * then it is only changing where the direct
+				 * pointer is jumping to, and the record already
+				 * points to a direct trampoline. If it isn't,
+				 * then it is a bug to update ipmodify on a direct
+				 * caller.
+				 */
+				FTRACE_WARN_ON(!update_target &&
+					       (rec->flags & FTRACE_FL_DIRECT));
 
 				/*
 				 * Another ops with IPMODIFY is already

@@ -2076,7 +2089,7 @@ static int ftrace_hash_ipmodify_enable(struct ftrace_ops *ops)
 	if (ftrace_hash_empty(hash))
 		hash = NULL;
 
-	return __ftrace_hash_update_ipmodify(ops, EMPTY_HASH, hash);
+	return __ftrace_hash_update_ipmodify(ops, EMPTY_HASH, hash, false);
 }
 
 /* Disabling always succeeds */

@@ -2087,7 +2100,7 @@ static void ftrace_hash_ipmodify_disable(struct ftrace_ops *ops)
 	if (ftrace_hash_empty(hash))
 		hash = NULL;
 
-	__ftrace_hash_update_ipmodify(ops, hash, EMPTY_HASH);
+	__ftrace_hash_update_ipmodify(ops, hash, EMPTY_HASH, false);
 }
 
 static int ftrace_hash_ipmodify_update(struct ftrace_ops *ops,

@@ -2101,7 +2114,7 @@ static int ftrace_hash_ipmodify_update(struct ftrace_ops *ops,
 	if (ftrace_hash_empty(new_hash))
 		new_hash = NULL;
 
-	return __ftrace_hash_update_ipmodify(ops, old_hash, new_hash);
+	return __ftrace_hash_update_ipmodify(ops, old_hash, new_hash, false);
 }
 
 static void print_ip_ins(const char *fmt, const unsigned char *p)

@@ -6114,7 +6127,7 @@ EXPORT_SYMBOL_GPL(unregister_ftrace_direct);
 static int
 __modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
 {
-	struct ftrace_hash *hash;
+	struct ftrace_hash *hash = ops->func_hash->filter_hash;
 	struct ftrace_func_entry *entry, *iter;
 	static struct ftrace_ops tmp_ops = {
 		.func		= ftrace_stub,

@@ -6134,13 +6147,21 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
 	if (err)
 		return err;
 
+	/*
+	 * Call __ftrace_hash_update_ipmodify() here, so that we can call
+	 * ops->ops_func for the ops. This is needed because the above
+	 * register_ftrace_function_nolock() worked on tmp_ops.
+	 */
+	err = __ftrace_hash_update_ipmodify(ops, hash, hash, true);
+	if (err)
+		goto out;
+
 	/*
 	 * Now the ftrace_ops_list_func() is called to do the direct callers.
 	 * We can safely change the direct functions attached to each entry.
 	 */
 	mutex_lock(&ftrace_lock);
 
-	hash = ops->func_hash->filter_hash;
 	size = 1 << hash->size_bits;
 	for (i = 0; i < size; i++) {
 		hlist_for_each_entry(iter, &hash->buckets[i], hlist) {

@@ -6155,6 +6176,7 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
 
 	mutex_unlock(&ftrace_lock);
 
+out:
 	/* Removing the tmp_ops will add the updated direct callers to the functions */
 	unregister_ftrace_function(&tmp_ops);
Signature · 1 line
2.47.3
author maintainer
#3 lore raw

PATCH v4 3/3 selftests/bpf: Add tests for livepatch + bpf trampoline

In reply to Song Liu

Both livepatch and BPF trampoline use ftrace. Special attention is needed when livepatch and fexit program touch the same function at the same time, because livepatch updates a kernel function and the BPF trampoline need to call into the right version of the kernel function.

Use samples/livepatch/livepatch-sample.ko for the test.

The test covers two cases:
1) When a fentry program is loaded first. This exercises the modify_ftrace_direct code path.
2) When a fentry program is loaded first. This exercises the register_ftrace_direct code path.

Signed-off-by Song Liu <[email protected]>
Reviewed-by Jiri Olsa <[email protected]>
---
 tools/testing/selftests/bpf/config            |   3 +
 .../bpf/prog_tests/livepatch_trampoline.c     | 107 ++++++++++++++++++
 .../bpf/progs/livepatch_trampoline.c          |  30 +++++
 3 files changed, 140 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/livepatch_trampoline.c
 create mode 100644 tools/testing/selftests/bpf/progs/livepatch_trampoline.c
tools/testing/selftests/bpf/config +3 -0
index 70b28c1e653e..f2a2fd236ca8 100644
@@ -50,6 +50,7 @@ CONFIG_IPV6_SIT=y
 CONFIG_IPV6_TUNNEL=y
 CONFIG_KEYS=y
 CONFIG_LIRC=y
+CONFIG_LIVEPATCH=y
 CONFIG_LWTUNNEL=y
 CONFIG_MODULE_SIG=y
 CONFIG_MODULE_SRCVERSION_ALL=y

@@ -111,6 +112,8 @@ CONFIG_IP6_NF_FILTER=y
 CONFIG_NF_NAT=y
 CONFIG_PACKET=y
 CONFIG_RC_CORE=y
+CONFIG_SAMPLES=y
+CONFIG_SAMPLE_LIVEPATCH=m
 CONFIG_SECURITY=y
 CONFIG_SECURITYFS=y
 CONFIG_SYN_COOKIES=y
tools/testing/selftests/bpf/prog_tests/livepatch_trampoline.c new file +107 -0

107 lines not on this page · raw message

tools/testing/selftests/bpf/progs/livepatch_trampoline.c new file +30 -0
new file mode 100644
index 000000000000..15579d5bcd91
@@ -0,0 +1,30 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+int fentry_hit;
+int fexit_hit;
+int my_pid;
+
+SEC("fentry/cmdline_proc_show")
+int BPF_PROG(fentry_cmdline)
+{
+	if (my_pid != (bpf_get_current_pid_tgid() >> 32))
+		return 0;
+
+	fentry_hit = 1;
+	return 0;
+}
+
+SEC("fexit/cmdline_proc_show")
+int BPF_PROG(fexit_cmdline)
+{
+	if (my_pid != (bpf_get_current_pid_tgid() >> 32))
+		return 0;
+
+	fexit_hit = 1;
+	return 0;
+}
Signature · 1 line
2.47.3
#4 lore raw
In reply to Song Liu
Quoted 11 lines · Song Liu

livepatch and BPF trampoline are two special users of ftrace. livepatch uses ftrace with IPMODIFY flag and BPF trampoline uses ftrace direct functions. When livepatch and BPF trampoline with fexit programs attach to the same kernel function, BPF trampoline needs to call into the patched version of the kernel function.

1/3 and 2/3 of this patchset fix two issues with livepatch + fexit cases, one in the register_ftrace_direct path, the other in the modify_ftrace_direct path.

Quoted 8 lines

3/3 adds selftests for both cases.

---

Changes v3 => v4:
1. Add helper reset_direct. (Steven)
2. Add Reviewed-by from Jiri.
3. Fix minor typo in comments.

Steven,

can you apply the fixes or should I take them ?
If so, pls ack.

maintainer
#5 lore raw
Quoted 2 lines · On Fri, 31 Oct 2025 17:19:54 -0700 Alexei Starovoitov…

can you apply the fixes or should I take them ?
If so, pls ack.

Let me run them through my full test suite. It takes up to 13 hours to run. Then I'll give a ack for you to take them.

-- Steve

maintainer Acked-by
#6 lore raw
Quoted 9 lines · On Sat, 1 Nov 2025 09:11:16 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 2 lines · On Fri, 31 Oct 2025 17:19:54 -0700 Alexei Starovoitov…

can you apply the fixes or should I take them ?
If so, pls ack.

Let me run them through my full test suite. It takes up to 13 hours to run. Then I'll give a ack for you to take them.

They passed my tests.

Acked-by Steven Rostedt (Google) <[email protected]>

Feel free to send them through the BPF tree.

-- Steve

CI Applied
#7 lore raw
In reply to Song Liu
This series was applied to bpf/bpf.git (master) by Alexei Starovoitov <[email protected]>:

Hello:

This series was applied to bpf/bpf.git (master)
by Alexei Starovoitov <[email protected]>:

Quoted 11 lines · On Mon, 27 Oct 2025 10:50:20 -0700 you wrote:

livepatch and BPF trampoline are two special users of ftrace. livepatch uses ftrace with IPMODIFY flag and BPF trampoline uses ftrace direct functions. When livepatch and BPF trampoline with fexit programs attach to the same kernel function, BPF trampoline needs to call into the patched version of the kernel function.

1/3 and 2/3 of this patchset fix two issues with livepatch + fexit cases, one in the register_ftrace_direct path, the other in the modify_ftrace_direct path.

[...]

Here is the summary with links:
- [v4,bpf,1/3] ftrace: Fix BPF fexit with livepatch

    https://git.kernel.org/bpf/bpf/c/56b3c85e153b
  - [v4,bpf,2/3] ftrace: bpf: Fix IPMODIFY + DIRECT in modify_ftrace_direct()
    https://git.kernel.org/bpf/bpf/c/3e9a18e1c3e9
  - [v4,bpf,3/3] selftests/bpf: Add tests for livepatch + bpf trampoline
    https://git.kernel.org/bpf/bpf/c/62d2d0a33839

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html