Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

Dear Upstream Livepatch team and Ubuntu Kernel team - I included you both in this since the issue lies on the boundary between Ubuntu kernel and upstream.

According to official kernel documentation - https://docs.kernel.org/livepatch/livepatch.html#livepatch, section 7, Limitations - 1 - Kretprobes using the ftrace framework conflict with the patched functions.
2 - Kprobes in the original function are ignored when the code is redirected to the new implementation.

Yet, when testing on my Ubuntu 5.15.0-1005.7-aws (based on 5.15.30 stable kernel) machine, I have no problem applying Livepatch and then setting krpobes and kretprobes on a patched function using bpftrace (or directly by coding a BPF program with kprobe/kretprobe attachment)and can confirm both execute without issues. Also the opposite works fine, running my krpobe and kretprobe hooks doesn't prevent from livepatch to be applied successfully.

fentry/fexit probes do fail in in both directions - but this is expected according to my understanding as coexistence of livepatching and ftrace based BPF hooks are mutually exclusive until 6.0 based kernels

libbpf: prog 'begin_new_exec_fentry': failed to attach: Device or resource busy
libbpf: prog 'begin_new_exec_fentry': failed to auto-attach: -16

Please help me understand this contradiction about kprobes - is this because the KPROBES are FTRACE based or any other reason ?

Bellow a few prints to prove my point - the kernel function i am using to test is begin_new_exec

ubuntu@ip-xxxxx:~$ sudo canonical-livepatch status
last check: 19 minutes ago
kernel: 5.15.0-1005.7-aws
server check-in: succeeded
kernel state: ✗ Livepatch coverage ended 2023-05-20; please upgrade the kernel and reboot
patch state: ✓ all applicable livepatch modules inserted
patch version: 94.1
tier: stable
machine id: xxxxx

ubuntu@ip-xxxxxx:~$ sudo cat /proc/kallsyms | grep begin_new_exec
ffffffff8af7bdc0 T begin_new_exec
ffffffff8c335694 r __ksymtab_begin_new_exec
ffffffff8c369830 r __kstrtabns_begin_new_exec
ffffffff8c36cf3d r __kstrtab_begin_new_exec
ffffffffc28c1650 t begin_new_exec	[lkp_Ubuntu_5_15_0_1005_7_aws_94]
ffffffffc1a9d388 t bpf_prog_211a1dc71de113a4_begin_new_exec_kprobe	[bpf]
ffffffffc1a9f588 t bpf_prog_5522bacfbb7628bd_begin_new_exec_kretprobe	[bpf]

ubuntu@ip-xxxxx:~$ sudo cat /sys/kernel/debug/kprobes/list  | grep begin_new_exec
ffffffff8af7bdc0  r  begin_new_exec+0x0    [FTRACE]
ffffffff8af7bdc0  k  begin_new_exec+0x0    [FTRACE]

root@ip-xxxxx:~# echo 102400 > /sys/kernel/debug/tracing/buffer_size_kb && echo 1 > /sys/kernel/debug/tracing/events/bpf_trace/enable && echo 1 > /sys/kernel/debug/tracing/tracing_on && cat 
/sys/kernel/debug/tracing/trace_pipe
             cat-10406   [000] d...1 433341.003035: bpf_trace_printk: kprobe: begin_new_exec called with bprm=000000005304a36e
             cat-10406   [000] d...1 433341.003163: bpf_trace_printk: kretprobe: begin_new_exec returned 0

Heh, it seems that we have discussed this 10 years ago and I already forgot most details.

Yes, the conflict is detected when KPROBES are using FTRACE infrastructure. But it happens only when the KPROBE needs to redirect the function call, namely when it needs to modify IP address which will be used when all attached ftrace callbacks are proceed.

It is related to the FTRACE_OPS_FL_IPMODIFY flag.

More details can be found in the discussion at
https://lore.kernel.org/all/[email protected]/T/#re746846b6b16c49a55c89b4c63b7995fe5972111

I seems that I made some analyze when it worked and it did not work,
see https://lore.kernel.org/all/[email protected]/T/#mffd8c8bf4325b473d89876f2805f42f1af7c82d7
But I am not 100% sure that it was correct. Also it was before the
BPF-boom.

Also you might look at the selftest in the todays Linus' tree:

+ tools/testing/selftests/livepatch/test-kprobe.sh
+ tools/testing/selftests/livepatch/test_modules/test_klp_kprobe.c
+ tools/testing/selftests/livepatch/test_modules/test_klp_livepatch.c

The parallel load fails when the Kprobe is using a post_handler.

Sigh, we should fix the livepatch documentation. The kretprobes obviously work. register_kretprobe() even explicitely sets:

	rp->kp.post_handler = NULL;

It seems that .post_handler is called after all ftrace handlers. And it sets IP after the NOPs, see kprobe_ftrace_handler(). I am not sure about the use case.

Best Regards,
Petr

I see, that explains my case as my probes are simple, print only probes that defently don't that the ip pointer.

But i still don't understand limitation 2 above doesn't show itself - how my kprobes and especially kretprobes, continue execute even after livepatch applied to the function  i am attached to ? The code execution is redirected to a different function then to which i attached my probes...

Also - can you please confirm that as far as i checked, starting with kernel 6.0 fentry/fexit on x86 should not have any conflicts with livepatch per merge of this RFC - https://lkml.iu.edu/hypermail/linux/kernel/2207.2/00858.html ?

Thanks,
Andrey

Added Song Liu and Steven into Cc because they are more familiar with the ftrace trampolines.

The code is rather complicated and it is not obvious to me.

But I expect that kretprobes modify return addresses which are stored on the stack. They replace them
with a helper function (trampoline) which would process the kretprobe callback and jump to the original return address.

It works even with livepatching. The only requirement is that the function returns using a "ret" instruction which reads the return address from the stack.

My understanding is that the above mentioned patchset allowed to use livepatching and BPF on the same function at the same time. BPF used to redirect the function to a BPF trampoline. And the patchset allowed to call the BPF trampoline/callback directly from the ftrace callback.

Best Regards,
Petr

Yes, the patch set above allows BPF trampoline to work on the same function at the same time with livepatch. After this patchset, BPF trampoline no longer sets FTRACE_OPS_FL_IPMODIFY.

Thanks,
Song

I tested this bellow on 6.1.0 based Ubuntu kernel using bpftrace - in the end as seen bellow, I wasn't able to attach using fentry/fexit (or krpboe/kretprobe) as long as livepatch was present

ubuntu@ip-10-10-114-204:~$ uname -r
6.8.0-1008-aws
ubuntu@ip-10-10-114-204:~$ cat /proc/version_signature
Ubuntu 6.8.0-1008.8-aws 6.8.1

[AG] - See bellow the patched function I am to attach to -

ubuntu@ip-10-10-114-204:~$ sudo cat /proc/kallsyms | grep begin_new_exec
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name resolution
ffffffff81ee0420 T __pfx_begin_new_exec
ffffffff81ee0430 T begin_new_exec
ffffffff8374b3d8 r __ksymtab_begin_new_exec
ffffffffc07c9390 t begin_new_exec [lkp_Ubuntu_6_8_0_1008_8_aws_111]
ffffffffc07c9380 t __pfx_begin_new_exec [lkp_Ubuntu_6_8_0_1008_8_aws_111]

[AG] - Attaching to begin_new_exec

ubuntu@ip-10-10-114-204:~$ sudo bpftool btf dump file /sys/kernel/btf/vmlinux | grep begin_new_exec
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name resolution
[68920] FUNC 'begin_new_exec' type_id=48063 linkage=static

[AG] - , trying to attach to fentry and fexit and first error asks me to explicitly resolve the ambiguity - which i guess make sense

ubuntu@ip-10-10-114-204:~$

ubuntu@ip-10-10-114-204:~$ sudo bpftrace -v -e 'fentry:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC START (fentry): PID %d, Comm %s\n", pid, comm); } fexit:begin_new_exec { $latency = nsecs - @start[tid]; delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s, Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
stdin:1:1-22: ERROR: ambiguous attach point, please specify module containing the function 'begin_new_exec'
fentry:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC START (fentry): PID %d, Comm %s\n", pid, comm); } fexit:begin_new_exec { $latency = nsecs - @start[tid]; delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s, Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }
~~~~~~~~~~~~~~~~~~~~~
stdin:1:111-132: ERROR: ambiguous attach point, please specify module containing the function 'begin_new_exec'
fentry:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC START (fentry): PID %d, Comm %s\n", pid, comm); } fexit:begin_new_exec { $latency = nsecs - @start[tid]; delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s, Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }
                                            ~~~~~~~~~~~~~~~~~~~~~
[AG] - Trying first to point him at the original  function - but he fails on the fexit I assume  - which is strange, I assumed fexit (kretfunc) and livepatch can coexist ?

ubuntu@ip-10-10-114-204:~$ sudo bpftrace -e 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC START (fentry): PID %d, Comm %s\n", pid, comm); } fexit:vmlinux:begin_new_exec { $latency = nsecs - @start[tid]; delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s, Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
Attaching 2 probes...
ERROR: Error attaching probe: kretfunc:vmlinux:begin_new_exec

[AG] - Trying to skip the fexit and only do fentry - he still rejects

ubuntu@ip-10-10-114-204:~$ sudo bpftrace -vvv -e 'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC START (fentry): PID %d, Comm %s\n", pid, comm); }'
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name resolution
INFO: node count: 12
Attaching 1 probe...

Program ID: 295

The verifier log:
processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3 peak_states 3 mark_read 1

Attaching kfunc:vmlinux:begin_new_exec
ERROR: Error attaching probe: kfunc:vmlinux:begin_new_exec

[AG] - Now trying to point him at the livepatch function but here he complains that Uubntu doesn't have BTF info for the livepatch module

ubuntu@ip-10-10-114-204:~$

ubuntu@ip-10-10-114-204:~$ sudo bpftrace -v -e 'fentry:lkp_Ubuntu_6_8_0_1008_8_aws_111:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC START (fentry): PID %d, Comm %s\n", pid, comm); } fexit:lkp_Ubuntu_6_8_0_1008_8_aws_111:begin_new_exec { $latency = nsecs - @start[tid]; delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s, Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name resolution
INFO: node count: 33
Attaching 2 probes...
ERROR: No BTF found for lkp_Ubuntu_6_8_0_1008_8_aws_111:begin_new_exec

[AG] - Disabling livepatch, rebooting to verify all livepatch symbols are out of kalsym table and now everything fine

ubuntu@ip-10-10-114-204:~$   sudo bpftrace -v -e 'fentry:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC START (fentry): PID %d, Comm %s\n", pid, comm]; delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s, Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
INFO: node count: 33
Attaching 2 probes...
Program ID: 33
The verifier log:
processed 76 insns (limit 1000000) max_states_per_insn 0 total_states 5 peak_states 5 mark_read 3
Attaching kretfunc:vmlinux:begin_new_exec
Program ID: 34
The verifier log:
processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3 peak_states 3 mark_read 1

Attaching kfunc:vmlinux:begin_new_exec
-> EXEC START (fentry): PID 1714, Comm bash
<- EXEC END (fexit): PID 1714, Comm sudo, Retval 0, Latency 99 us

ubuntu@ip-10-10-114-204:~$ sudo bpftool link
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name resolution
1: tracing  prog 2
    prog_type tracing  attach_type modify_return
    target_obj_id 1  target_btf_id 119714
5: tracing  prog 33
    prog_type tracing  attach_type trace_fexit
    target_obj_id 1  target_btf_id 68920
6: tracing  prog 34
    prog_type tracing  attach_type trace_fentry
    target_obj_id 1  target_btf_id 68920
ubuntu@ip-10-10-114-204:~$

Thanks,
Andrey

On Thu, Oct 16, 2025 at 2:55 PM Andrey Grodzovsky
<[email protected]> wrote:
[...]

OK, I could reproduce this issue and found the issue. In my test, fexit+livepatch works on some older kernel, but fails on some newer kernel. I haven't bisected to the commit that broke it.

Something like the following make it work:

index 2e113f8b13a2..4277b4f33eb8 100644
@@ -5985,6 +5985,8 @@ int register_ftrace_direct(struct ftrace_ops

*ops, unsigned long addr)

        ops->direct_call = addr;

        err = register_ftrace_function_nolock(ops);

+ if (err)
+ remove_direct_functions_hash(direct_functions, addr);

  out_unlock:
        mutex_unlock(&direct_mutex);

Andrey, could you also test this change?

Thanks,
Song

Attached is a better version of the fix.

Thanks,
Song

From d4aa9c0058c794d6259248187bd5896cb40eea29 Mon Sep 17 00:00:00 2001
From: Song Liu <[email protected]>
Date: Fri, 17 Oct 2025 11:49:45 -0700
Subject: [PATCH] ftrace: Fix BPF fexit with livepatch

When livepatch is attached to the same function as bpf trampoline with a fexit program, bpf trampoline code calls register_ftrace_direct() twice. The first time will fail with -EAGAIN, and the second time it will succeed. This requires register_ftrace_direct() to unregister the address on the first attempt. Otherwise, the bpf trampoline cannot attach. Here is an easy way to reproduce this issue:

insmod samples/livepatch/livepatch-sample.ko
bpftrace -e 'fexit:cmdline_proc_show {}'
ERROR: Unable to attach probe: fexit:vmlinux:cmdline_proc_show...

Fix this by updating direct_functions hash after
register_ftrace_function_nolock succeeds.

---
 kernel/trace/ftrace.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

index a69067367c29..82a06b7d0268 100644
@@ -6038,16 +6038,18 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
 		}
 	}
 
-	free_hash = direct_functions;
-	rcu_assign_pointer(direct_functions, new_hash);
-	new_hash = NULL;
-
 	ops->func = call_direct_funcs;
 	ops->flags = MULTI_FLAGS;
 	ops->trampoline = FTRACE_REGS_ADDR;
 	ops->direct_call = addr;
 
 	err = register_ftrace_function_nolock(ops);
+	if (err)
+		goto out_unlock;
+
+	free_hash = direct_functions;
+	rcu_assign_pointer(direct_functions, new_hash);
+	new_hash = NULL;
 
  out_unlock:
 	mutex_unlock(&direct_mutex);

2.47.3

Thank you Song!

So, with this You say both fentry and fexit will work no issues ?

So juts to understand, as i am not familiar with live-patch generation, I get the sources for my Ubuntu kernel, I apply your patch, I also generate manually livepatch module that makes a dummy patching to my test function (begin_new_exec), and apply this patch to my running costum kernel ? Because i don't think the stadard ubuntu livepatching will agree to apply his livepatch CVEs to my ostum kenel, it will probably recognize it's not stock ubuntu kernel but amanully built one.

Thanks,
Andrey

So i need to build the stable

Yes, fentry and fexit should both work.

livepatch is a kernel module. Therefore, unless the two kernels are almost identical, livepatch built for one kernel cannot be used on the other.

If you build the kernel from source code, there are some samples in samples/livepatch that you can use for testing. PS: You need to enable

CONFIG_SAMPLE_LIVEPATCH=m

I hope this helps.

Thanks,
Song

Thanks Song, working on repro, kernel rebuilt, test module is loading but, bpftrace is refusing to attach now to fentries/fexits claiming the costum kernel is not supporting it. It did
attach in the case of stock AWS kernel i copied the .config from. So just trying to figure out now if some Kcofnig flags are missing or different . Let me know in case you manage to confirm yourself in the meanwhile the fix works for you.

Thanks,
Andrey

On Mon, Oct 20, 2025 at 7:56 AM Andrey Grodzovsky
<[email protected]> wrote:
[...]

Yes, it worked in my tests.

[root@(none) /]# kpatch load linux/samples/livepatch/livepatch-sample.ko loading patch module: linux/samples/livepatch/livepatch-sample.ko [root@(none) /]# bpftrace.real -e 'fexit:cmdline_proc_show {printf("fexit\n");}' &
[1] 388
[root@(none) /]# Attached 1 probe
[root@(none) /]# bpftrace.real -e 'fentry:cmdline_proc_show {printf("fentry\n");}' &
[2] 397
[root@(none) /]# Attached 1 probe

[root@(none) /]# cat /proc/cmdline
this has been live patched
fentry
fexit

Thanks,
Song

Verified the failures I observe when trying to attach with BPF trace are only in presence of patch you provided. Please see attached dmesg for failures. Initial warning on boot. Subsequebt warnings and errors at the point i try to run sudo bpftrace -e "fexit:cmdline_proc_show { printf(\"fexit hit\\n\"); exit(); }"

sudo: unable to resolve host ip-10-10-115-238: Temporary failure in name resolution stdin:1:1-25: ERROR: kfunc/kretfunc not available for your kernel version.

ubuntu@ip-10-10-115-238:~/linux-6.8.1$ sudo cat /sys/kernel/debug/tracing/available_filter_functions | grep cmdline_proc_show
sudo: unable to resolve host ip-10-10-115-238: Temporary failure in name resolution
cat: /sys/kernel/debug/tracing/available_filter_functions: No such device

After reboot and before trying to attacg with bpftrace, /sys/kernel/debug/tracing/available_filter_functions is available and shows all function.

Using stable kernel from https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.8.1.tar.gz for build. FTRACE related KCONFIGs bellow

ubuntu@ip-10-10-115-238:~$ cat /boot/config-6.8.1 | grep TRACE
CONFIG_TASKS_TRACE_RCU=y
CONFIG_SYSCTL_EXCEPTION_TRACE=y
CONFIG_TRACEPOINTS=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_PM_TRACE=y
CONFIG_PM_TRACE_RTC=y
CONFIG_KPROBES_ON_FTRACE=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_ARCH_CORRECT_STACKTRACE_ON_KRETPROBE=y
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_TRACE_IRQFLAGS_NMI_SUPPORT=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_RELIABLE_STACKTRACE=y
CONFIG_NETFILTER_XT_TARGET_TRACE=m
# CONFIG_DMA_FENCE_TRACE is not set
CONFIG_SCSI_IPR_TRACE=y
# CONFIG_ATH5K_TRACER is not set
# CONFIG_DRM_I915_LOW_LEVEL_TRACEPOINTS is not set
CONFIG_STM_SOURCE_FTRACE=m
# CONFIG_PSTORE_FTRACE is not set
# CONFIG_STACKTRACE_BUILD_ID is not set
CONFIG_STACKTRACE=y
# CONFIG_RCU_TRACE is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS=y
CONFIG_HAVE_DYNAMIC_FTRACE_NO_PATCHABLE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_TRACE_CLOCK=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_GENERIC_TRACER=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
CONFIG_DYNAMIC_FTRACE_WITH_ARGS=y
CONFIG_STACK_TRACER=y
# CONFIG_IRQSOFF_TRACER is not set
CONFIG_SCHED_TRACER=y
CONFIG_HWLAT_TRACER=y
CONFIG_OSNOISE_TRACER=y
CONFIG_TIMERLAT_TRACER=y
CONFIG_MMIOTRACE=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_TRACER_SNAPSHOT=y
# CONFIG_TRACER_SNAPSHOT_PER_CPU_SWAP is not set
CONFIG_BLK_DEV_IO_TRACE=y
# CONFIG_KPROBE_EVENTS_ON_NOTRACE is not set
CONFIG_FTRACE_MCOUNT_RECORD=y
CONFIG_FTRACE_MCOUNT_USE_CC=y
CONFIG_TRACE_EVENT_INJECT=y
# CONFIG_TRACEPOINT_BENCHMARK is not set
# CONFIG_TRACE_EVAL_MAP_FILE is not set
# CONFIG_FTRACE_RECORD_RECURSION is not set
# CONFIG_FTRACE_STARTUP_TEST is not set
CONFIG_FTRACE_SORT_STARTUP_TEST=y
# CONFIG_MMIOTRACE_TEST is not set
# CONFIG_SAMPLE_TRACE_EVENTS is not set
# CONFIG_SAMPLE_TRACE_CUSTOM_EVENTS is not set
CONFIG_SAMPLE_TRACE_PRINTK=m
CONFIG_SAMPLE_FTRACE_DIRECT=m
# CONFIG_SAMPLE_FTRACE_DIRECT_MULTI is not set
# CONFIG_SAMPLE_FTRACE_OPS is not set
CONFIG_SAMPLE_TRACE_ARRAY=m
CONFIG_HAVE_SAMPLE_FTRACE_DIRECT=y
CONFIG_HAVE_SAMPLE_FTRACE_DIRECT_MULTI=y
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_CC_HAS_SANCOV_TRACE_PC=y
# CONFIG_BACKTRACE_SELF_TEST is not set

Andrey

[    0.958579] NET: Registered PF_INET6 protocol family
[    0.960794] Segment Routing with IPv6
[    0.961819] In-situ OAM (IOAM) with IPv6
[    0.962927] NET: Registered PF_PACKET protocol family
[    0.964393] Key type dns_resolver registered
[    0.968676] No MBM correction factor available
[    0.969933] IPI shorthand broadcast: enabled
[    0.972554] sched_clock: Marking stable (673007499, 299279483)->(1155822036, -183535054)
[    0.974892] registered taskstats version 1
[    0.977043] Loading compiled-in X.509 certificates
[    0.979081] Loaded X.509 cert 'Build time autogenerated kernel key: 774d7f8f61628ad4e8d329abbb98ef067d7b43a9'
[    0.985242] Key type .fscrypt registered
[    0.986488] Key type fscrypt-provisioning registered
[    0.988673] Key type encrypted registered
[    0.989923] AppArmor: AppArmor sha256 policy hashing enabled
[    0.991934] ima: No TPM chip found, activating TPM-bypass!
[    0.993570] Loading compiled-in module X.509 certificates
[    0.996021] Loaded X.509 cert 'Build time autogenerated kernel key: 774d7f8f61628ad4e8d329abbb98ef067d7b43a9'
[    0.999303] ima: Allocated hash algorithm: sha256
[    1.000769] ima: No architecture policies found
[    1.002184] evm: Initialising EVM extended attributes:
[    1.003740] evm: security.selinux
[    1.004829] evm: security.SMACK64
[    1.005916] evm: security.SMACK64EXEC
[    1.007085] evm: security.SMACK64TRANSMUTE
[    1.008389] evm: security.SMACK64MMAP
[    1.009567] evm: security.apparmor
[    1.010663] evm: security.ima
[    1.011651] evm: security.capability
[    1.012788] evm: HMAC attrs: 0x1
[    1.014248] PM:   Magic number: 13:837:285
[    1.019035] ------------[ cut here ]------------
[    1.020163] WARNING: CPU: 10 PID: 1 at kernel/trace/ftrace.c:2589 ftrace_get_addr_new+0x10c/0x120
[    1.022417] Modules linked in:
[    1.023417] CPU: 10 PID: 1 Comm: swapper/0 Not tainted 6.8.1 #1
[    1.025148] Hardware name: Amazon EC2 c5.4xlarge/, BIOS 1.0 10/16/2017
[    1.027039] RIP: 0010:ftrace_get_addr_new+0x10c/0x120
[    1.028559] Code: 00 48 85 c0 74 9d 5b 41 5c 41 5d 5d 31 d2 31 c9 31 f6 31 ff c3 cc cc cc cc 48 8b 3f e8 5d fe ff ff 48 85 c0 0f 85 38 ff ff ff <0f> 0b 48 8b 43 08 e9 0f ff ff ff 66 0f 1f 84 00 00 00 00 00 90 90
[    1.033947] RSP: 0000:ffffa8bbc0027808 EFLAGS: 00010246
[    1.035521] RAX: 0000000000000000 RBX: ffff88bf800ec540 RCX: 0000000000000000
[    1.037587] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[    1.039648] RBP: ffffa8bbc0027820 R08: 0000000000000000 R09: 0000000000000000
[    1.041713] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88bf800ec540
[    1.043773] R13: 0000000000000001 R14: ffffffffa39aef08 R15: ffffffffa2206cca
[    1.045836] FS:  0000000000000000(0000) GS:ffff88c665900000(0000) knlGS:0000000000000000
[    1.048143] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    1.049851] CR2: 0000000000000000 CR3: 000000037203c001 CR4: 00000000007706f0
[    1.051926] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    1.053992] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[    1.056061] PKRU: 55555554
[    1.056978] Call Trace:
[    1.057833]  <TASK>
[    1.058584]  ? show_regs+0x6d/0x80
[    1.059676]  ? __warn+0x89/0x150
[    1.060714]  ? ftrace_get_addr_new+0x10c/0x120
[    1.062073]  ? report_bug+0x16a/0x190
[    1.063219]  ? handle_bug+0x51/0xa0
[    1.064351]  ? exc_invalid_op+0x18/0x80
[    1.065547]  ? asm_exc_invalid_op+0x1b/0x20
[    1.066840]  ? ftrace_get_addr_new+0x10c/0x120
[    1.068207]  ? ftrace_get_addr_new+0x103/0x120
[    1.069576]  ftrace_replace_code+0x11d/0x1b0
[    1.070894]  ftrace_modify_all_code+0x15e/0x1a0
[    1.072317]  arch_ftrace_update_code+0x9/0x20
[    1.073666]  ftrace_startup+0xcc/0x180
[    1.074855]  register_ftrace_function_nolock+0x24/0x70
[    1.076438]  register_ftrace_direct+0x2ed/0x370
[    1.077838]  ? 0xffffffffc010ca00
[    1.078923]  ? __pfx___hid_bpf_tail_call+0x10/0x10
[    1.080404]  bpf_trampoline_update+0x469/0x650
[    1.081775]  ? 0xffffffffc010ca00
[    1.082833]  ? 0xffffffffc010ca00
[    1.083881]  __bpf_trampoline_link_prog+0x10d/0x2d0
[    1.085359]  bpf_trampoline_link_prog+0x2b/0x50
[    1.086700]  bpf_tracing_prog_attach+0x3c2/0x5f0
[    1.088073]  bpf_raw_tp_link_attach+0x206/0x2d0
[    1.089430]  __sys_bpf+0x1120/0x2500
[    1.090531]  kern_sys_bpf+0x61/0x1e0
[    1.091635]  ? close_fd+0x58/0x80
[    1.092664]  skel_raw_tracepoint_open.constprop.0+0x43/0x70
[    1.094297]  hid_bpf_preload_skel+0x111/0x180
[    1.095596]  ? __pfx_hid_bpf_init+0x10/0x10
[    1.096849]  hid_bpf_init+0x1e/0xc0
[    1.097930]  do_one_initcall+0x5e/0x310
[    1.099092]  kernel_init_freeable+0x359/0x500
[    1.100403]  ? __pfx_kernel_init+0x10/0x10
[    1.101628]  kernel_init+0x1b/0x200
[    1.102695]  ret_from_fork+0x47/0x70
[    1.103790]  ? __pfx_kernel_init+0x10/0x10
[    1.105011]  ret_from_fork_asm+0x1b/0x30
[    1.106207]  </TASK>
[    1.106931] ---[ end trace 0000000000000000 ]---
[    1.109385] RAS: Correctable Errors collector initialized.
[    1.111113] clk: Disabling unused clocks
[    1.416156] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2
[    1.418725] md: Waiting for all devices to be available before autodetect
[    1.419892] md: If you don't use raid, use raid=noautodetect
[    1.420894] md: Autodetecting RAID arrays.
[    1.421656] md: autorun ...
[    1.422215] md: ... autorun DONE.
[    1.429543] EXT4-fs (nvme0n1p1): mounted filesystem 6387caa1-122f-48bc-b447-9f1386d06e06 ro with ordered data mode. Quota mode: none.
[    1.431922] VFS: Mounted root (ext4 filesystem) readonly on device 259:1.
[    1.434051] devtmpfs: mounted
[    1.435734] Freeing unused decrypted memory: 2028K
[    1.437643] Freeing unused kernel image (initmem) memory: 4984K
[    1.439121] Write protecting the kernel read-only data: 34816k
[    1.440674] Freeing unused kernel image (rodata/data gap) memory: 872K
[    1.486425] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    1.487559] x86/mm: Checking user space page tables
[    1.531530] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    1.532674] Run /sbin/init as init process
[    1.533460]   with arguments:
[    1.533462]     /sbin/init
[    1.533463]   with environment:
[    1.533464]     HOME=/
[    1.533465]     TERM=linux
[    1.533465]     BOOT_IMAGE=/vmlinuz-6.8.1
[    1.691160] systemd[1]: Inserted module 'autofs4'
[    1.739692] systemd[1]: systemd 255.4-1ubuntu8 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    1.745330] systemd[1]: Detected virtualization amazon.
[    1.746574] systemd[1]: Detected architecture x86-64.
[    1.751410] systemd[1]: Hostname set to <ip-10-10-115-238>.
[    2.144939] systemd[1]: Queued start job for default target graphical.target.
[    2.169782] systemd[1]: Created slice system-modprobe.slice - Slice /system/modprobe.
[    2.173549] systemd[1]: Created slice system-serial\x2dgetty.slice - Slice /system/serial-getty.
[    2.177997] systemd[1]: Created slice system-systemd\x2dfsck.slice - Slice /system/systemd-fsck.
[    2.181785] systemd[1]: Created slice user.slice - User and Session Slice.
[    2.184612] systemd[1]: Started ntpsec-systemd-netif.path.
[    2.187009] systemd[1]: Started systemd-ask-password-wall.path - Forward Password Requests to Wall Directory Watch.
[    2.191530] systemd[1]: Set up automount proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point.
[    2.196152] systemd[1]: Expecting device dev-disk-by\x2dlabel-BOOT.device - /dev/disk/by-label/BOOT...
[    2.200018] systemd[1]: Expecting device dev-disk-by\x2dlabel-UEFI.device - /dev/disk/by-label/UEFI...
[    2.203876] systemd[1]: Expecting device dev-ttyS0.device - /dev/ttyS0...
[    2.206699] systemd[1]: Reached target integritysetup.target - Local Integrity Protected Volumes.
[    2.210142] systemd[1]: Reached target slices.target - Slice Units.
[    2.213182] systemd[1]: Reached target swap.target - Swaps.
[    2.215621] systemd[1]: Reached target veritysetup.target - Local Verity Protected Volumes.
[    2.219064] systemd[1]: Listening on dm-event.socket - Device-mapper event daemon FIFOs.
[    2.222434] systemd[1]: Listening on lvm2-lvmpolld.socket - LVM2 poll daemon socket.
[    2.226036] systemd[1]: Listening on multipathd.socket - multipathd control socket.
[    2.242149] systemd[1]: Listening on rpcbind.socket - RPCbind Server Activation Socket.
[    2.246074] systemd[1]: Listening on syslog.socket - Syslog Socket.
[    2.249145] systemd[1]: Listening on systemd-fsckd.socket - fsck to fsckd communication Socket.
[    2.252891] systemd[1]: Listening on systemd-initctl.socket - initctl Compatibility Named Pipe.
[    2.256663] systemd[1]: Listening on systemd-journald-dev-log.socket - Journal Socket (/dev/log).
[    2.260465] systemd[1]: Listening on systemd-journald.socket - Journal Socket.
[    2.263756] systemd[1]: Listening on systemd-networkd.socket - Network Service Netlink Socket.
[    2.267430] systemd[1]: systemd-pcrextend.socket - TPM2 PCR Extension (Varlink) was skipped because of an unmet condition check (ConditionSecurity=measured-uki).
[    2.270426] systemd[1]: Listening on systemd-udevd-control.socket - udev Control Socket.
[    2.273935] systemd[1]: Listening on systemd-udevd-kernel.socket - udev Kernel Socket.
[    2.291382] systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
[    2.295171] systemd[1]: Mounting dev-mqueue.mount - POSIX Message Queue File System...
[    2.299230] systemd[1]: Mounting sys-kernel-debug.mount - Kernel Debug File System...
[    2.303059] systemd[1]: Mounting sys-kernel-tracing.mount - Kernel Trace File System...
[    2.306079] systemd[1]: auth-rpcgss-module.service - Kernel Module supporting RPCSEC_GSS was skipped because of an unmet condition check (ConditionPathExists=/etc/krb5.keytab).
[    2.310195] systemd[1]: Starting keyboard-setup.service - Set the console keyboard layout...
[    2.314585] systemd[1]: Starting kmod-static-nodes.service - Create List of Static Device Nodes...
[    2.319415] systemd[1]: Starting lvm2-monitor.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[    2.324999] systemd[1]: Starting [email protected] - Load Kernel Module configfs...
[    2.328934] systemd[1]: Starting modprobe@dm_mod.service - Load Kernel Module dm_mod...
[    2.332977] systemd[1]: Starting [email protected] - Load Kernel Module drm...
[    2.337481] systemd[1]: Starting modprobe@efi_pstore.service - Load Kernel Module efi_pstore...
[    2.341518] systemd[1]: Starting [email protected] - Load Kernel Module fuse...
[    2.345226] systemd[1]: Starting [email protected] - Load Kernel Module loop...
[    2.348111] systemd[1]: netplan-ovs-cleanup.service - OpenVSwitch configuration for cleanup was skipped because of an unmet condition check (ConditionFileIsExecutable=/usr/bin/ovs-vsctl).
[    2.351100] pstore: Using crash dump compression: deflate
[    2.352436] systemd[1]: Starting systemd-fsck-root.service - File System Check on Root Device...
[    2.356026] pstore: Registered efi_pstore as persistent store backend
[    2.358847] systemd[1]: Starting systemd-journald.service - Journal Service...
[    2.364360] systemd[1]: Starting systemd-modules-load.service - Load Kernel Modules...
[    2.366617] systemd[1]: systemd-pcrmachine.service - TPM2 PCR Machine ID Measurement was skipped because of an unmet condition check (ConditionSecurity=measured-uki).
[    2.370028] systemd[1]: systemd-tpm2-setup-early.service - TPM2 SRK Setup (Early) was skipped because of an unmet condition check (ConditionSecurity=measured-uki).
[    2.373850] systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices...
[    2.377223] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[    2.380691] systemd[1]: Mounted dev-mqueue.mount - POSIX Message Queue File System.
[    2.383791] systemd[1]: Mounted sys-kernel-debug.mount - Kernel Debug File System.
[    2.385162] systemd-journald[255]: Collecting audit messages is disabled.
[    2.387968] systemd[1]: Mounted sys-kernel-tracing.mount - Kernel Trace File System.
[    2.391368] systemd[1]: Finished keyboard-setup.service - Set the console keyboard layout.
[    2.394953] systemd[1]: Finished kmod-static-nodes.service - Create List of Static Device Nodes.
[    2.398483] systemd[1]: [email protected]: Deactivated successfully.
[    2.399910] systemd[1]: Finished [email protected] - Load Kernel Module configfs.
[    2.403776] systemd[1]: modprobe@dm_mod.service: Deactivated successfully.
[    2.405468] systemd[1]: Finished modprobe@dm_mod.service - Load Kernel Module dm_mod.
[    2.408695] systemd[1]: Finished lvm2-monitor.service - Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
[    2.412929] systemd[1]: [email protected]: Deactivated successfully.
[    2.414661] systemd[1]: Finished [email protected] - Load Kernel Module drm.
[    2.417973] systemd[1]: Started systemd-journald.service - Journal Service.
[    2.545456] EXT4-fs (nvme0n1p1): re-mounted 6387caa1-122f-48bc-b447-9f1386d06e06 r/w. Quota mode: none.
[    2.572734] systemd-journald[255]: Received client request to flush runtime journal.
[    2.594301] systemd-journald[255]: /var/log/journal/ec220e94df30bda465b15689b7378e3a/system.journal: Journal file uses a different sequence number ID, rotating.
[    2.594307] systemd-journald[255]: Rotating system journal.
[    2.761565] piix4_smbus 0000:00:01.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
[    2.786745] ena 0000:00:05.0: ENA device version: 0.10
[    2.786750] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    2.802370] parport_pc 00:03: reported by Plug and Play ACPI
[    2.841917] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem c0510000, mac addr 12:67:59:84:01:a5
[    2.852675] ppdev: user-space parallel port driver
[    2.853640] ena 0000:00:05.0 ens5: renamed from eth0
[    2.860372] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer
[    2.866183] cryptd: max_cpu_qlen set to 1000
[    2.899090] AVX2 version of gcm_enc/dec engaged.
[    2.899202] AES CTR mode by8 optimization enabled
[    2.984184] EXT4-fs (nvme0n1p16): mounted filesystem 9d39f3ff-b465-4c20-9ec1-06182226356c r/w with ordered data mode. Quota mode: none.
[    3.084656] audit: type=1400 audit(1760973474.672:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="busybox" pid=708 comm="apparmor_parser"
[    3.084663] audit: type=1400 audit(1760973474.672:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="crun" pid=714 comm="apparmor_parser"
[    3.084665] audit: type=1400 audit(1760973474.672:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="ch-checkns" pid=710 comm="apparmor_parser"
[    3.084668] audit: type=1400 audit(1760973474.672:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="devhelp" pid=715 comm="apparmor_parser"
[    3.084671] audit: type=1400 audit(1760973474.672:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="Discord" pid=703 comm="apparmor_parser"
[    3.084697] audit: type=1400 audit(1760973474.672:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="brave" pid=706 comm="apparmor_parser"
[    3.084727] audit: type=1400 audit(1760973474.672:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="vscode" pid=713 comm="apparmor_parser"
[    3.084763] audit: type=1400 audit(1760973474.672:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="ch-run" pid=711 comm="apparmor_parser"
[    3.084794] audit: type=1400 audit(1760973474.672:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="element-desktop" pid=716 comm="apparmor_parser"
[    3.502649] RPC: Registered named UNIX socket transport module.
[    3.502652] RPC: Registered udp transport module.
[    3.502653] RPC: Registered tcp transport module.
[    3.502654] RPC: Registered tcp-with-tls transport module.
[    3.502655] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    3.674659] input: ImPS/2 Generic Wheel Mouse as /devices/platform/i8042/serio1/input/input4
[    4.013037] 8021q: 802.1Q VLAN Support v1.8
[    4.140597] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[    4.140856] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[    4.140992] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600'
[    4.141693] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[    4.141696] cfg80211: failed to load regulatory.db
[    9.730872] kauditd_printk_skb: 94 callbacks suppressed
[    9.730875] audit: type=1400 audit(1760973481.318:105): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="rsyslogd" pid=1152 comm="apparmor_parser"
[    9.906210] NET: Registered PF_QIPCRTR protocol family
[   10.657618] audit: type=1400 audit(1760973482.245:106): apparmor="STATUS" operation="profile_load" profile="unconfined" name="docker-default" pid=1353 comm="apparmor_parser"
[   10.743033] evm: overlay not supported
[   10.843306] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   10.852812] Bridge firewalling registered
[   11.008539] Initializing XFRM netlink socket
[   49.644664] systemd-journald[255]: /var/log/journal/ec220e94df30bda465b15689b7378e3a/user-1000.journal: Journal file uses a different sequence number ID, rotating.
[ 1237.397558] ------------[ cut here ]------------
[ 1237.397562] WARNING: CPU: 12 PID: 1988 at arch/x86/kernel/ftrace.c:99 ftrace_verify_code+0x4a/0x90
[ 1237.397569] Modules linked in: xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables libcrc32c br_netfilter bridge qrtr overlay cfg80211 8021q garp mrp stp llc sunrpc binfmt_misc intel_rapl_msr intel_rapl_common nls_iso8859_1 intel_uncore_frequency_common isst_if_common nfit crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd cryptd rapl ppdev parport_pc psmouse ena parport i2c_piix4 input_leds mac_hid serio_raw dm_multipath msr efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4
[ 1237.397610] CPU: 12 PID: 1988 Comm: bpftrace Tainted: G        W          6.8.1 #1
[ 1237.397613] Hardware name: Amazon EC2 c5.4xlarge/, BIOS 1.0 10/16/2017
[ 1237.397614] RIP: 0010:ftrace_verify_code+0x4a/0x90
[ 1237.397616] Code: 00 48 89 45 f0 31 c0 c7 45 eb 00 00 00 00 c6 45 ef 00 e8 09 9c 2f 00 48 85 c0 75 41 8b 03 39 45 eb 74 2d 48 89 1d 26 01 be 02 <0f> 0b b8 ea ff ff ff 48 8b 55 f0 65 48 2b 14 25 28 00 00 00 75 26
[ 1237.397618] RSP: 0018:ffffa8bbc1d7fbd8 EFLAGS: 00010286
[ 1237.397621] RAX: 000000001d8d3be8 RBX: ffffffffa39aef08 RCX: 0000000000000000
[ 1237.397622] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 1237.397623] RBP: ffffa8bbc1d7fbf0 R08: 0000000000000000 R09: 0000000000000000
[ 1237.397625] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa39aef08
[ 1237.397626] R13: 0000000000000001 R14: ffff88bf80114d20 R15: ffffffffa39aef08
[ 1237.397627] FS:  00007fa3bcbd0780(0000) GS:ffff88c665a00000(0000) knlGS:0000000000000000
[ 1237.397629] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1237.397630] CR2: 00007ffe71155124 CR3: 0000000117032001 CR4: 00000000007706f0
[ 1237.397636] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1237.397637] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1237.397639] PKRU: 55555554
[ 1237.397640] Call Trace:
[ 1237.397641]  <TASK>
[ 1237.397643]  ? show_regs+0x6d/0x80
[ 1237.397649]  ? __warn+0x89/0x150
[ 1237.397653]  ? ftrace_verify_code+0x4a/0x90
[ 1237.397655]  ? report_bug+0x16a/0x190
[ 1237.397659]  ? handle_bug+0x51/0xa0
[ 1237.397662]  ? exc_invalid_op+0x18/0x80
[ 1237.397664]  ? asm_exc_invalid_op+0x1b/0x20
[ 1237.397668]  ? ftrace_verify_code+0x4a/0x90
[ 1237.397669]  ? ftrace_verify_code+0x37/0x90
[ 1237.397670]  ftrace_replace_code+0x71/0x1b0
[ 1237.397673]  ftrace_modify_all_code+0x125/0x1a0
[ 1237.397675]  ? 0xffffffffc01206c0
[ 1237.397684]  arch_ftrace_update_code+0x9/0x20
[ 1237.397686]  ftrace_shutdown.part.0+0x114/0x1e0
[ 1237.397688]  ? 0xffffffffc01206c0
[ 1237.397690]  ? 0xffffffffc01206c0
[ 1237.397691]  unregister_ftrace_function+0x47/0x170
[ 1237.397693]  ? 0xffffffffc01206c0
[ 1237.397694]  ? 0xffffffffc01206c0
[ 1237.397696]  unregister_ftrace_direct+0x60/0x1e0
[ 1237.397697]  ? kmalloc_trace+0x136/0x310
[ 1237.397700]  ? bpf_trampoline_update+0x68/0x650
[ 1237.397703]  ? 0xffffffffc01206c0
[ 1237.397704]  ? __pfx_sched_fork+0x10/0x10
[ 1237.397707]  bpf_trampoline_update+0x4f7/0x650
[ 1237.397709]  bpf_trampoline_unlink_prog+0x9d/0x130
[ 1237.397711]  bpf_tracing_link_release+0x16/0x50
[ 1237.397715]  bpf_link_free+0x55/0x80
[ 1237.397717]  bpf_link_release+0x26/0x40
[ 1237.397719]  __fput+0xa1/0x2e0
[ 1237.397723]  __fput_sync+0x1c/0x30
[ 1237.397725]  __x64_sys_close+0x3e/0x90
[ 1237.397728]  do_syscall_64+0x85/0x180
[ 1237.397732]  ? syscall_exit_to_user_mode+0x86/0x260
[ 1237.397736]  ? do_syscall_64+0x92/0x180
[ 1237.397738]  ? do_syscall_64+0x92/0x180
[ 1237.397740]  ? do_syscall_64+0x92/0x180
[ 1237.397743]  ? irqentry_exit+0x21/0x40
[ 1237.397745]  ? exc_page_fault+0x94/0x190
[ 1237.397748]  entry_SYSCALL_64_after_hwframe+0x6e/0x76
[ 1237.397752] RIP: 0033:0x7fa3b4f166f4
[ 1237.397755] Code: 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 80 3d 55 49 0f 00 00 74 13 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 3c c3 0f 1f 00 55 48 89 e5 48 83 ec 10 89 7d
[ 1237.397756] RSP: 002b:00007ffe7113d028 EFLAGS: 00000202 ORIG_RAX: 0000000000000003
[ 1237.397758] RAX: ffffffffffffffda RBX: 0000555e10456800 RCX: 00007fa3b4f166f4
[ 1237.397760] RDX: 0000000000000010 RSI: 00007ffe7113cf60 RDI: 0000000000000004
[ 1237.397761] RBP: 00007ffe7113d080 R08: 0000555e0f7fb86c R09: 0000000000000000
[ 1237.397762] R10: 00007ffe7113ba00 R11: 0000000000000202 R12: 0000000000000004
[ 1237.397763] R13: 0000000000000003 R14: 0000555e0f7faeac R15: 00007ffe7113d750
[ 1237.397765]  </TASK>
[ 1237.397766] ---[ end trace 0000000000000000 ]---
[ 1237.397767] ------------[ ftrace bug ]------------
[ 1237.397768] ftrace failed to modify 
[ 1237.397769] [<ffffffffa0f47980>] sched_fork+0x0/0x260
[ 1237.397771]  actual:   e8:db:38:f6:ff
[ 1237.397773]  expected: e8:3b:8d:1d:1f
[ 1237.397775] Setting ftrace call site to NOP
[ 1237.397775] ftrace record flags: e8980000
[ 1237.397776]  (0) R  	tramp: ERROR!
                expected tramp: ffffffffc01206c0
[ 1237.397787] ------------[ cut here ]------------
[ 1237.397788] WARNING: CPU: 12 PID: 1988 at kernel/trace/ftrace.c:2179 ftrace_bug+0x10d/0x2d0
[ 1237.397790] Modules linked in: xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables libcrc32c br_netfilter bridge qrtr overlay cfg80211 8021q garp mrp stp llc sunrpc binfmt_misc intel_rapl_msr intel_rapl_common nls_iso8859_1 intel_uncore_frequency_common isst_if_common nfit crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd cryptd rapl ppdev parport_pc psmouse ena parport i2c_piix4 input_leds mac_hid serio_raw dm_multipath msr efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4
[ 1237.397819] CPU: 12 PID: 1988 Comm: bpftrace Tainted: G        W          6.8.1 #1
[ 1237.397821] Hardware name: Amazon EC2 c5.4xlarge/, BIOS 1.0 10/16/2017
[ 1237.397821] RIP: 0010:ftrace_bug+0x10d/0x2d0
[ 1237.397823] Code: 07 00 e8 46 df f2 ff 41 f6 44 24 0b 08 0f 85 18 01 00 00 4c 89 e7 e8 a2 fc ff ff 48 c7 c7 7d 51 77 a2 48 89 c6 e8 23 df f2 ff <0f> 0b c7 05 af cf 3d 02 01 00 00 00 5b c7 05 b4 cf 3d 02 00 00 00
[ 1237.397824] RSP: 0018:ffffa8bbc1d7fbd8 EFLAGS: 00010246
[ 1237.397826] RAX: 0000000000000022 RBX: ffffffffa344a640 RCX: 0000000000000000
[ 1237.397827] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 1237.397828] RBP: ffffa8bbc1d7fbf0 R08: 0000000000000000 R09: 0000000000000000
[ 1237.397829] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88bf80114d20
[ 1237.397830] R13: ffffffffa0f47980 R14: ffff88bf80114d20 R15: ffffffffa39aef08
[ 1237.397831] FS:  00007fa3bcbd0780(0000) GS:ffff88c665a00000(0000) knlGS:0000000000000000
[ 1237.397833] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1237.397834] CR2: 00007ffe71155124 CR3: 0000000117032001 CR4: 00000000007706f0
[ 1237.397839] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1237.397840] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1237.397841] PKRU: 55555554
[ 1237.397842] Call Trace:
[ 1237.397843]  <TASK>
[ 1237.397844]  ? show_regs+0x6d/0x80
[ 1237.397846]  ? __warn+0x89/0x150
[ 1237.397849]  ? ftrace_bug+0x10d/0x2d0
[ 1237.397850]  ? report_bug+0x16a/0x190
[ 1237.397853]  ? handle_bug+0x51/0xa0
[ 1237.397854]  ? exc_invalid_op+0x18/0x80
[ 1237.397856]  ? asm_exc_invalid_op+0x1b/0x20
[ 1237.397858]  ? __pfx_sched_fork+0x10/0x10
[ 1237.397859]  ? ftrace_bug+0x10d/0x2d0
[ 1237.397861]  ? ftrace_bug+0x10d/0x2d0
[ 1237.397863]  ftrace_replace_code+0xc7/0x1b0
[ 1237.397864]  ftrace_modify_all_code+0x125/0x1a0
[ 1237.397866]  ? 0xffffffffc01206c0
[ 1237.397868]  arch_ftrace_update_code+0x9/0x20
[ 1237.397870]  ftrace_shutdown.part.0+0x114/0x1e0
[ 1237.397871]  ? 0xffffffffc01206c0
[ 1237.397873]  ? 0xffffffffc01206c0
[ 1237.397874]  unregister_ftrace_function+0x47/0x170
[ 1237.397876]  ? 0xffffffffc01206c0
[ 1237.397877]  ? 0xffffffffc01206c0
[ 1237.397878]  unregister_ftrace_direct+0x60/0x1e0
[ 1237.397880]  ? kmalloc_trace+0x136/0x310
[ 1237.397882]  ? bpf_trampoline_update+0x68/0x650
[ 1237.397884]  ? 0xffffffffc01206c0
[ 1237.397885]  ? __pfx_sched_fork+0x10/0x10
[ 1237.397886]  bpf_trampoline_update+0x4f7/0x650
[ 1237.397889]  bpf_trampoline_unlink_prog+0x9d/0x130
[ 1237.397891]  bpf_tracing_link_release+0x16/0x50
[ 1237.397893]  bpf_link_free+0x55/0x80
[ 1237.397895]  bpf_link_release+0x26/0x40
[ 1237.397897]  __fput+0xa1/0x2e0
[ 1237.397899]  __fput_sync+0x1c/0x30
[ 1237.397901]  __x64_sys_close+0x3e/0x90
[ 1237.397903]  do_syscall_64+0x85/0x180
[ 1237.397906]  ? syscall_exit_to_user_mode+0x86/0x260
[ 1237.397908]  ? do_syscall_64+0x92/0x180
[ 1237.397911]  ? do_syscall_64+0x92/0x180
[ 1237.397913]  ? do_syscall_64+0x92/0x180
[ 1237.397915]  ? irqentry_exit+0x21/0x40
[ 1237.397917]  ? exc_page_fault+0x94/0x190
[ 1237.397920]  entry_SYSCALL_64_after_hwframe+0x6e/0x76
[ 1237.397922] RIP: 0033:0x7fa3b4f166f4
[ 1237.397924] Code: 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 80 3d 55 49 0f 00 00 74 13 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 3c c3 0f 1f 00 55 48 89 e5 48 83 ec 10 89 7d
[ 1237.397925] RSP: 002b:00007ffe7113d028 EFLAGS: 00000202 ORIG_RAX: 0000000000000003
[ 1237.397927] RAX: ffffffffffffffda RBX: 0000555e10456800 RCX: 00007fa3b4f166f4
[ 1237.397928] RDX: 0000000000000010 RSI: 00007ffe7113cf60 RDI: 0000000000000004
[ 1237.397929] RBP: 00007ffe7113d080 R08: 0000555e0f7fb86c R09: 0000000000000000
[ 1237.397930] R10: 00007ffe7113ba00 R11: 0000000000000202 R12: 0000000000000004
[ 1237.397931] R13: 0000000000000003 R14: 0000555e0f7faeac R15: 00007ffe7113d750
[ 1237.397933]  </TASK>
[ 1237.397934] ---[ end trace 0000000000000000 ]---

I can see the similar issue with the upstream kernel. I was testing on stable 6.17 before just know because of another issue with upstream kernel, and somehow 6.17 kernel doesn't seem to have the issue.

To fix this, I think we should land a fix similar to the earlier diff:

index 42bd2ba68a82..8f320df0ac52 100644
@@ -6049,6 +6049,9 @@ int register_ftrace_direct(struct ftrace_ops

*ops, unsigned long addr)

        err = register_ftrace_function_nolock(ops);

+ if (err)
+ remove_direct_functions_hash(hash, addr);
+
out_unlock:

        mutex_unlock(&direct_mutex);

Steven,

Does this change look good to you?

Thanks,
Song

Seems reasonable to me, we are simply cleaning the entry on failure so we don't encounter it late anymore. So I will apply this patch ONLY and retest - correct ?

Another question - it seems you found where it broke ? I saw 'Cc: [email protected] # v6.6+' in your prev. patch.' If so , can you please point me to the offending patch so I add this to my records of my discovery work of bpf coexistence livepatching ?

Thanks,

Andrey

On Mon, Oct 20, 2025 at 12:10 PM Andrey Grodzovsky <[email protected]> wrote:
[...]

Right, we only need this small change.

The commit that causing the issue was in the Fixes tag:

while in use")

Thanks,
Song

Update - with latest fix work find, both after loading the livepatch .ko  no conflicts and hooks work and, before loading it, pre exsisting hooks keep working after the patch is loaded

You can add Acked-and-tested-by: Andrey Grodzovsky <[email protected]>

Once again, in case you now the exact commit that broke it, please let me know.

Thanks,
Andrey

Song, I identified another issue in pre 6.6 kernel, building ~/linux-6.5/samples/livepatch/livepatch-sample.c as ko, before insmoding it, bpftrace fentry/fexit fires as expected, after insmod, while no errors reported on attachments, the hooks stop firing, both if attaching before insmod and if attaching after insmod. If i rrmod the ko, existing hooks resume working.

ubuntu@ip-10-10-115-238:~$ cat /proc/version_signature
Ubuntu 6.5.0-1008.8-aws 6.5.3
Source obtained to build the test module for the AWS kernel from the related stable branch - https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz

Let me know what you think.

Thanks,
Andrey

On Mon, Oct 20, 2025 at 2:31 PM Andrey Grodzovsky
<[email protected]> wrote:
[...]

I tested various stable kernels. I got:

With livepatch, fentry and fexit work on 6.3 kernels.

On 6.4 and 6.5 kernels, the combination stops working since this commit:

commit 60c8971899f3b34ad24857913c0784dab08962f0

Author: Florent Revest <[email protected]>
Date:   2 years, 7 months ago

ftrace: Make DIRECT_CALLS work WITH_ARGS and !WITH_REGS

On 6.5 kernels, it got fixed by the following two commits:

commit a8b9cf62ade1bf17261a979fc97e40c2d7842353
Author: Masami Hiramatsu (Google) <[email protected]>
Date: 1 year, 9 months ago
ftrace: Fix DIRECT_CALLS to use SAVE_REGS by default

commit bdbddb109c75365d22ec4826f480c5e75869e1cb

Author: Petr Pavlu <[email protected]>
Date:   1 year, 8 months ago

tracing: Fix HAVE_DYNAMIC_FTRACE_WITH_REGS ifdef

I tried to cherry-pick 60c8971899f3b34ad24857913c0784dab08962f0 and a8b9cf62ade1bf17261a979fc97e40c2d7842353, on top of 6.5.13 kernel. Then, fentry and fexit both work with livepatch.

Thanks,
Song

Hmm, this is a work around. I wonder if we can make this work with ARGS as well? Hmm. I'll have to take a look when I get a chance.

-- Steve

Given both ftrace_caller and arch_prepare_bpf_trampoline are created per arch, I think it is possible. But I guess it may require quite some work to update and test each architecture?

Thanks,
Song

I see, thanks for testing! Is the reason it breaks so often is because this combination of having BPF and llivepatch together on a system with intersection on same functions as relatively   rate event and so regressions go easily unnoticed ? Isn't there any relevant automated testing in upstream that checks for those types of breaks ?

Thanks,
Andrey

On Tue, Oct 21, 2025 at 7:15 AM Andrey Grodzovsky
<[email protected]> wrote:
[...]

This case is not being covered because it is the intersection of tracing and livepatch. I am thinking about adding a BPF selftest to cover this case.

Thanks,
Song

With the small nit that you don't need a space between the err = and the if (err).

But yeah, looks fine to me.

-- Steve