The same symbol is printed twice when use tracepoint to get stack
Hi guys,
When using tracepoints to retrieve stack information, I observed that perf_trace_sched_migrate_task was printed twice. And the issue also occurs with tools using libbpf.
sudo bpftrace -e '
tracepoint:sched:sched_migrate_task {printf("Task %s migrated by:\n", args->comm);
print(kstack);
}'
Task kcompactd0 migrated by:
perf_trace_sched_migrate_task+9
perf_trace_sched_migrate_task+9
set_task_cpu+353
detach_task+77
detach_tasks+281
sched_balance_rq+452
sched_balance_newidle+504
pick_next_task_fair+84
__pick_next_task+66
pick_next_task+43
__schedule+332
schedule+41
schedule_hrtimeout_range+239
do_poll.constprop.0+668
do_sys_poll+499
__x64_sys_ppoll+220
x64_sys_call+5722
do_syscall_64+126
entry_SYSCALL_64_after_hwframe+118Task jbd2/sda2-8 migrated by:
perf_trace_sched_migrate_task+9
perf_trace_sched_migrate_task+9
set_task_cpu+353
try_to_wake_up+365
default_wake_function+26
autoremove_wake_function+18
__wake_up_common+118
__wake_up+55
__jbd2_log_start_commit+195env:
bpftrace v0.21.2
ubuntu24.04,6.14.0-36-generic
The issue is as follows:
https://github.com/bpftrace/bpftrace/issues/4949
It seems that there is no special handling in the kernel. Does anyone has thoughts on this issue. Thanks.
BPF_CALL_4(bpf_get_stack_raw_tp, struct bpf_raw_tracepoint_args *, args,
void *, buf, u32, size, u64, flags)
{
struct pt_regs *regs = get_bpf_raw_tp_regs();
int ret;
if (IS_ERR(regs))
return PTR_ERR(regs);
perf_fetch_caller_regs(regs);
ret = bpf_get_stack((unsigned long) regs, (unsigned long) buf,
(unsigned long) size, flags, 0);
put_bpf_raw_tp_regs();
return ret;
}Signature · 2 lines
Best Regards Tao Chen
Quoted 4 lines · Tao Chen
Hi guys,
When using tracepoints to retrieve stack information, I observed that
perf_trace_sched_migrate_taskwas printed twice. And the issue also occurs with tools using libbpf.
You may need the fix Jiri provided for x86_64 [1]. Eugene mentioned that the issue persists for arm64 however [2].
Alan
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Quoted 67 lines
sudo bpftrace -e '
tracepoint:sched:sched_migrate_task {printf("Task %s migrated by:\n", args->comm);
print(kstack);
}'Task kcompactd0 migrated by:
perf_trace_sched_migrate_task+9
perf_trace_sched_migrate_task+9
set_task_cpu+353
detach_task+77
detach_tasks+281
sched_balance_rq+452
sched_balance_newidle+504
pick_next_task_fair+84
__pick_next_task+66
pick_next_task+43
__schedule+332
schedule+41
schedule_hrtimeout_range+239
do_poll.constprop.0+668
do_sys_poll+499
__x64_sys_ppoll+220
x64_sys_call+5722
do_syscall_64+126
entry_SYSCALL_64_after_hwframe+118Task jbd2/sda2-8 migrated by:
perf_trace_sched_migrate_task+9
perf_trace_sched_migrate_task+9
set_task_cpu+353
try_to_wake_up+365
default_wake_function+26
autoremove_wake_function+18
__wake_up_common+118
__wake_up+55
__jbd2_log_start_commit+195env:
bpftrace v0.21.2
ubuntu24.04,6.14.0-36-genericThe issue is as follows:
https://github.com/bpftrace/bpftrace/issues/4949It seems that there is no special handling in the kernel. Does anyone has thoughts on this issue. Thanks.
BPF_CALL_4(bpf_get_stack_raw_tp, struct bpf_raw_tracepoint_args *, args, void *, buf, u32, size, u64, flags)
{
structpt_regs*regs =get_bpf_raw_tp_regs();
int ret;if (
IS_ERR(regs))
returnPTR_ERR(regs);
perf_fetch_caller_regs(regs);
ret =bpf_get_stack((unsigned long) regs, (unsigned long) buf, (unsigned long) size, flags, 0);put_bpf_raw_tp_regs();
return ret;
}
Quoted 8 lines · Alan Maguire
Quoted 4 lines · Tao Chen
Hi guys,
When using tracepoints to retrieve stack information, I observed that
perf_trace_sched_migrate_taskwas printed twice. And the issue also occurs with tools using libbpf.You may need the fix Jiri provided for
x86_64[1]. Eugene mentioned that the issue persists for arm64 however [2].
yep, there's also follow patchset up for kprobe multi [1]
jirka
[1] https://lore.kernel.org/bpf/[email protected]/
Quoted 74 lines
Alan
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/Quoted 67 lines
sudo bpftrace -e '
tracepoint:sched:sched_migrate_task {printf("Task %s migrated by:\n", args->comm);
print(kstack);
}'Task kcompactd0 migrated by:
perf_trace_sched_migrate_task+9
perf_trace_sched_migrate_task+9
set_task_cpu+353
detach_task+77
detach_tasks+281
sched_balance_rq+452
sched_balance_newidle+504
pick_next_task_fair+84
__pick_next_task+66
pick_next_task+43
__schedule+332
schedule+41
schedule_hrtimeout_range+239
do_poll.constprop.0+668
do_sys_poll+499
__x64_sys_ppoll+220
x64_sys_call+5722
do_syscall_64+126
entry_SYSCALL_64_after_hwframe+118Task jbd2/sda2-8 migrated by:
perf_trace_sched_migrate_task+9
perf_trace_sched_migrate_task+9
set_task_cpu+353
try_to_wake_up+365
default_wake_function+26
autoremove_wake_function+18
__wake_up_common+118
__wake_up+55
__jbd2_log_start_commit+195env:
bpftrace v0.21.2
ubuntu24.04,6.14.0-36-genericThe issue is as follows:
https://github.com/bpftrace/bpftrace/issues/4949It seems that there is no special handling in the kernel. Does anyone has thoughts on this issue. Thanks.
BPF_CALL_4(bpf_get_stack_raw_tp, struct bpf_raw_tracepoint_args *, args, void *, buf, u32, size, u64, flags)
{
structpt_regs*regs =get_bpf_raw_tp_regs();
int ret;if (
IS_ERR(regs))
returnPTR_ERR(regs);
perf_fetch_caller_regs(regs);
ret =bpf_get_stack((unsigned long) regs, (unsigned long) buf, (unsigned long) size, flags, 0);put_bpf_raw_tp_regs();
return ret;
}
Quoted 10 lines · Jiri Olsa
Quoted 8 lines · Alan Maguire
Quoted 4 lines · Tao Chen
Hi guys,
When using tracepoints to retrieve stack information, I observed that
perf_trace_sched_migrate_taskwas printed twice. And the issue also occurs with tools using libbpf.You may need the fix Jiri provided for
x86_64[1]. Eugene mentioned that the issue persists for arm64 however [2].
Quoted 81 lines
yep, there's also follow patchset up for kprobe multi [1]
jirka
[1] https://lore.kernel.org/bpf/[email protected]/
Quoted 74 lines
Alan
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/Quoted 67 lines
sudo bpftrace -e '
tracepoint:sched:sched_migrate_task {printf("Task %s migrated by:\n", args->comm);
print(kstack);
}'Task kcompactd0 migrated by:
perf_trace_sched_migrate_task+9
perf_trace_sched_migrate_task+9
set_task_cpu+353
detach_task+77
detach_tasks+281
sched_balance_rq+452
sched_balance_newidle+504
pick_next_task_fair+84
__pick_next_task+66
pick_next_task+43
__schedule+332
schedule+41
schedule_hrtimeout_range+239
do_poll.constprop.0+668
do_sys_poll+499
__x64_sys_ppoll+220
x64_sys_call+5722
do_syscall_64+126
entry_SYSCALL_64_after_hwframe+118Task jbd2/sda2-8 migrated by:
perf_trace_sched_migrate_task+9
perf_trace_sched_migrate_task+9
set_task_cpu+353
try_to_wake_up+365
default_wake_function+26
autoremove_wake_function+18
__wake_up_common+118
__wake_up+55
__jbd2_log_start_commit+195env:
bpftrace v0.21.2
ubuntu24.04,6.14.0-36-genericThe issue is as follows:
https://github.com/bpftrace/bpftrace/issues/4949It seems that there is no special handling in the kernel. Does anyone has thoughts on this issue. Thanks.
BPF_CALL_4(bpf_get_stack_raw_tp, struct
bpf_raw_tracepoint_args*, args,
void *, buf, u32, size, u64, flags)
{
structpt_regs*regs =get_bpf_raw_tp_regs();
int ret;if (
IS_ERR(regs))
returnPTR_ERR(regs);
perf_fetch_caller_regs(regs);
ret =bpf_get_stack((unsigned long) regs, (unsigned long) buf, (unsigned long) size, flags, 0);put_bpf_raw_tp_regs();
return ret;
}
Alan, Jiri
Thanks for your reply, i see.
Signature · 2 lines
Best Regards Tao Chen