Skip to contentSkip to conversation

The same symbol is printed twice when use tracepoint to get stack

DISCUSSION Tao Chen 3 participants 4 messages Jan 14 – Jan 15 BPF

lore
Share & tools
lore

Tao Chen

author
#0 lore raw

Hi guys,

When using tracepoints to retrieve stack information, I observed that perf_trace_sched_migrate_task was printed twice. And the issue also occurs with tools using libbpf.

sudo bpftrace -e '
tracepoint:sched:sched_migrate_task {
printf("Task %s migrated by:\n", args->comm);
print(kstack);
}'

Task kcompactd0 migrated by:

        perf_trace_sched_migrate_task+9
        perf_trace_sched_migrate_task+9
        set_task_cpu+353
        detach_task+77
        detach_tasks+281
        sched_balance_rq+452
        sched_balance_newidle+504
        pick_next_task_fair+84
        __pick_next_task+66
        pick_next_task+43
        __schedule+332
        schedule+41
        schedule_hrtimeout_range+239
        do_poll.constprop.0+668
        do_sys_poll+499
        __x64_sys_ppoll+220
        x64_sys_call+5722
        do_syscall_64+126
        entry_SYSCALL_64_after_hwframe+118

Task jbd2/sda2-8 migrated by:

        perf_trace_sched_migrate_task+9
        perf_trace_sched_migrate_task+9
        set_task_cpu+353
        try_to_wake_up+365
        default_wake_function+26
        autoremove_wake_function+18
        __wake_up_common+118
        __wake_up+55
        __jbd2_log_start_commit+195

env:
bpftrace v0.21.2
ubuntu24.04,6.14.0-36-generic

The issue is as follows:
https://github.com/bpftrace/bpftrace/issues/4949

It seems that there is no special handling in the kernel. Does anyone has thoughts on this issue. Thanks.

BPF_CALL_4(bpf_get_stack_raw_tp, struct bpf_raw_tracepoint_args *, args,

           void *, buf, u32, size, u64, flags)
{
        struct pt_regs *regs = get_bpf_raw_tp_regs();
        int ret;

        if (IS_ERR(regs))
                return PTR_ERR(regs);

        perf_fetch_caller_regs(regs);
        ret = bpf_get_stack((unsigned long) regs, (unsigned long) buf,
                            (unsigned long) size, flags, 0);
        put_bpf_raw_tp_regs();
        return ret;
}
Signature · 2 lines
Best Regards
Tao Chen
#1 lore raw
Quoted 4 lines · Tao Chen

Hi guys,

When using tracepoints to retrieve stack information, I observed that perf_trace_sched_migrate_task was printed twice. And the issue also occurs with tools using libbpf.

You may need the fix Jiri provided for x86_64 [1]. Eugene mentioned that the issue persists for arm64 however [2].

Alan

[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

Quoted 67 lines

sudo bpftrace -e '
tracepoint:sched:sched_migrate_task {
printf("Task %s migrated by:\n", args->comm);
print(kstack);
}'

Task kcompactd0 migrated by:

        perf_trace_sched_migrate_task+9
        perf_trace_sched_migrate_task+9
        set_task_cpu+353
        detach_task+77
        detach_tasks+281
        sched_balance_rq+452
        sched_balance_newidle+504
        pick_next_task_fair+84
        __pick_next_task+66
        pick_next_task+43
        __schedule+332
        schedule+41
        schedule_hrtimeout_range+239
        do_poll.constprop.0+668
        do_sys_poll+499
        __x64_sys_ppoll+220
        x64_sys_call+5722
        do_syscall_64+126
        entry_SYSCALL_64_after_hwframe+118

Task jbd2/sda2-8 migrated by:

        perf_trace_sched_migrate_task+9
        perf_trace_sched_migrate_task+9
        set_task_cpu+353
        try_to_wake_up+365
        default_wake_function+26
        autoremove_wake_function+18
        __wake_up_common+118
        __wake_up+55
        __jbd2_log_start_commit+195

env:
bpftrace v0.21.2
ubuntu24.04,6.14.0-36-generic

The issue is as follows:
https://github.com/bpftrace/bpftrace/issues/4949

It seems that there is no special handling in the kernel. Does anyone has thoughts on this issue. Thanks.

BPF_CALL_4(bpf_get_stack_raw_tp, struct bpf_raw_tracepoint_args *, args, void *, buf, u32, size, u64, flags)
{
        struct pt_regs *regs = get_bpf_raw_tp_regs();
        int ret;

        if (IS_ERR(regs))
                return PTR_ERR(regs);

        perf_fetch_caller_regs(regs);
        ret = bpf_get_stack((unsigned long) regs, (unsigned long) buf, (unsigned long) size, flags, 0); put_bpf_raw_tp_regs();
        return ret;
}

#2 lore raw
Quoted 8 lines · Alan Maguire
Quoted 4 lines · Tao Chen

Hi guys,

When using tracepoints to retrieve stack information, I observed that perf_trace_sched_migrate_task was printed twice. And the issue also occurs with tools using libbpf.

You may need the fix Jiri provided for x86_64 [1]. Eugene mentioned that the issue persists for arm64 however [2].

yep, there's also follow patchset up for kprobe multi [1]

jirka

[1] https://lore.kernel.org/bpf/[email protected]/

Quoted 74 lines

Alan

[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

Quoted 67 lines

sudo bpftrace -e '
tracepoint:sched:sched_migrate_task {
printf("Task %s migrated by:\n", args->comm);
print(kstack);
}'

Task kcompactd0 migrated by:

        perf_trace_sched_migrate_task+9
        perf_trace_sched_migrate_task+9
        set_task_cpu+353
        detach_task+77
        detach_tasks+281
        sched_balance_rq+452
        sched_balance_newidle+504
        pick_next_task_fair+84
        __pick_next_task+66
        pick_next_task+43
        __schedule+332
        schedule+41
        schedule_hrtimeout_range+239
        do_poll.constprop.0+668
        do_sys_poll+499
        __x64_sys_ppoll+220
        x64_sys_call+5722
        do_syscall_64+126
        entry_SYSCALL_64_after_hwframe+118

Task jbd2/sda2-8 migrated by:

        perf_trace_sched_migrate_task+9
        perf_trace_sched_migrate_task+9
        set_task_cpu+353
        try_to_wake_up+365
        default_wake_function+26
        autoremove_wake_function+18
        __wake_up_common+118
        __wake_up+55
        __jbd2_log_start_commit+195

env:
bpftrace v0.21.2
ubuntu24.04,6.14.0-36-generic

The issue is as follows:
https://github.com/bpftrace/bpftrace/issues/4949

It seems that there is no special handling in the kernel. Does anyone has thoughts on this issue. Thanks.

BPF_CALL_4(bpf_get_stack_raw_tp, struct bpf_raw_tracepoint_args *, args, void *, buf, u32, size, u64, flags)
{
        struct pt_regs *regs = get_bpf_raw_tp_regs();
        int ret;

        if (IS_ERR(regs))
                return PTR_ERR(regs);

        perf_fetch_caller_regs(regs);
        ret = bpf_get_stack((unsigned long) regs, (unsigned long) buf, (unsigned long) size, flags, 0); put_bpf_raw_tp_regs();
        return ret;
}

author
#3 lore raw
Quoted 10 lines · Jiri Olsa
Quoted 8 lines · Alan Maguire
Quoted 4 lines · Tao Chen

Hi guys,

When using tracepoints to retrieve stack information, I observed that perf_trace_sched_migrate_task was printed twice. And the issue also occurs with tools using libbpf.

You may need the fix Jiri provided for x86_64 [1]. Eugene mentioned that the issue persists for arm64 however [2].

Quoted 81 lines

yep, there's also follow patchset up for kprobe multi [1]

jirka

[1] https://lore.kernel.org/bpf/[email protected]/

Quoted 74 lines

Alan

[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

Quoted 67 lines

sudo bpftrace -e '
tracepoint:sched:sched_migrate_task {
printf("Task %s migrated by:\n", args->comm);
print(kstack);
}'

Task kcompactd0 migrated by:

        perf_trace_sched_migrate_task+9
        perf_trace_sched_migrate_task+9
        set_task_cpu+353
        detach_task+77
        detach_tasks+281
        sched_balance_rq+452
        sched_balance_newidle+504
        pick_next_task_fair+84
        __pick_next_task+66
        pick_next_task+43
        __schedule+332
        schedule+41
        schedule_hrtimeout_range+239
        do_poll.constprop.0+668
        do_sys_poll+499
        __x64_sys_ppoll+220
        x64_sys_call+5722
        do_syscall_64+126
        entry_SYSCALL_64_after_hwframe+118

Task jbd2/sda2-8 migrated by:

        perf_trace_sched_migrate_task+9
        perf_trace_sched_migrate_task+9
        set_task_cpu+353
        try_to_wake_up+365
        default_wake_function+26
        autoremove_wake_function+18
        __wake_up_common+118
        __wake_up+55
        __jbd2_log_start_commit+195

env:
bpftrace v0.21.2
ubuntu24.04,6.14.0-36-generic

The issue is as follows:
https://github.com/bpftrace/bpftrace/issues/4949

It seems that there is no special handling in the kernel. Does anyone has thoughts on this issue. Thanks.

BPF_CALL_4(bpf_get_stack_raw_tp, struct bpf_raw_tracepoint_args *, args,
           void *, buf, u32, size, u64, flags)
{
        struct pt_regs *regs = get_bpf_raw_tp_regs();
        int ret;

        if (IS_ERR(regs))
                return PTR_ERR(regs);

        perf_fetch_caller_regs(regs);
        ret = bpf_get_stack((unsigned long) regs, (unsigned long) buf, (unsigned long) size, flags, 0); put_bpf_raw_tp_regs();
        return ret;
}

Alan, Jiri

Thanks for your reply, i see.

Signature · 2 lines
Best Regards
Tao Chen