Pass external callchain entry to get_perf_callchain
Subsystems (3)
v4 cover Pass external callchain entry to get_perf_callchain
Background
==========
Alexei noted we should use preempt_disable to protect get_perf_callchain in bpf stackmap.
https://lore.kernel.org/bpf/CAADnVQ+s8B7-fvR1TNO-bniSyKv57cH_ihRszmZV7pQDyV=VDQ@mail.gmail.com
A previous patch was submitted to attempt fixing this issue. And Andrii suggested teach get_perf_callchain to let us pass that buffer directly to avoid that unnecessary copy.
https://lore.kernel.org/bpf/[email protected]
Proposed Solution
=================
Add external perf_callchain_entry parameter for get_perf_callchain to allow us to use external buffer from BPF side. The biggest advantage is that it can reduce unnecessary copies.
Todo
====
But I'm not sure if this modification is appropriate. After all, the implementation of get_callchain_entry in the perf subsystem seems much more complex than directly using an external buffer.
Comments and suggestions are always welcome.
Change list:
- v1 -> v2
From Jiri
- rebase code, fix conflict
- v1: https://lore.kernel.org/bpf/[email protected]
- v2 -> v3:
From Andrii
- entries per CPU used in a stack-like fashion
- v2: https://lore.kernel.org/bpf/[email protected]
- v3 -> v4:
From Peter
- refactor get_perf_callchain and add three new APIs to use perf callchain easily.
From Andrii
- reuse the perf callchain management.
- rename patch1 and patch2.
- v3: https://lore.kernel.org/bpf/[email protected]
Tao Chen (2):
perf: Refactor get_perf_callchain
bpf: Hold the perf callchain entry until used completely
include/linux/perf_event.h | 11 +++++- kernel/bpf/stackmap.c | 61 ++++++++++++++++++++++++------- kernel/events/callchain.c | 75 ++++++++++++++++++++++++-------------- kernel/events/core.c | 2 +- 4 files changed, 107 insertions(+), 42 deletions(-)
Signature · 1 line
2.48.1
PATCH v4 1/2 perf: Refactor get_perf_callchain
From BPF stack map, we want to use our own buffers to avoid unnecessary copy and ensure that the buffer will not be overwritten by other preemptive tasks. Peter suggested provide more flexible stack-sampling APIs, which can be used in BPF, and we can still use the perf callchain entry with the help of these APIs. The next patch will modify the BPF part.
--- include/linux/perf_event.h | 11 +++++- kernel/bpf/stackmap.c | 4 +- kernel/events/callchain.c | 75 ++++++++++++++++++++++++-------------- kernel/events/core.c | 2 +- 4 files changed, 61 insertions(+), 31 deletions(-)
include/linux/perf_event.h +10 -1
@@ -67,6 +67,7 @@ struct perf_callchain_entry_ctx {u32 nr;short contexts;bool contexts_maxed; +bool add_mark; }; typedef unsigned long (*perf_copy_f)(void *dst, const void *src,
@@ -1718,9 +1719,17 @@ DECLARE_PER_CPU(struct perf_callchain_entry, perf_callchain_entry);extern void perf_callchain_user (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs);extern void perf_callchain_kernel (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs); + +extern void __init_perf_callchain_ctx (struct perf_callchain_entry_ctx *ctx, +struct perf_callchain_entry *entry, +u32 max_stack,bool add_mark); + +extern void __get_perf_callchain_kernel (struct perf_callchain_entry_ctx *ctx,struct pt_regs *regs); +extern void __get_perf_callchain_user (struct perf_callchain_entry_ctx *ctx,struct pt_regs *regs); +extern struct perf_callchain_entry *get_perf_callchain (struct pt_regs *regs,bool kernel,bool user, -u32 max_stack,bool crosstask,bool add_mark); +u32 max_stack,bool crosstask);extern int get_callchain_buffers (int max_stack);extern void put_callchain_buffers (void );extern struct perf_callchain_entry *get_callchain_entry (int *rctx);
kernel/bpf/stackmap.c +2 -2
@@ -315,7 +315,7 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, max_depth = sysctl_perf_event_max_stack; trace =get_perf_callchain (regs, kernel, user, max_depth, - false, false); + false);if (unlikely (!trace))/* couldn't fetch the stack trace */
@@ -452,7 +452,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, trace =get_callchain_entry_for_task (task, max_depth); else trace =get_perf_callchain (regs, kernel, user, max_depth, - crosstask, false); + crosstask);if (unlikely (!trace) || trace->nr < skip) {if (may_fault)
kernel/events/callchain.c +48 -27
95 lines not on this page · raw message
kernel/events/core.c +1 -1
@@ -8218,7 +8218,7 @@ perf_callchain(struct perf_event *event, struct pt_regs *regs)return &__empty_callchain; callchain =get_perf_callchain (regs, kernel, user, - max_stack, crosstask, true); + max_stack, crosstask);return callchain ?: &__empty_callchain; }
Signature · 1 line
2.48.1
[email protected]
AI reviewRe: [PATCH bpf-next v4 1/2] perf: Refactor get_perf_callchain
AI reviewed your patch. Please fix the bug or email reply why it's not a bug. See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
``` commit 59cfc61f83ad4ec998f4242ca000c554e3001b47 Author: Tao Chen <[email protected]> perf: Refactor get_perf_callchain This patch refactors get_perf_callchain() by extracting logic into new helper functions to provide more flexible stack-sampling APIs for BPF. > diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h > index fd1d91017..14a382cad 100644 > --- a/include/linux/perf_event.h > +++ b/include/linux/perf_event.h > @@ -67,6 +67,7 @@ struct perf_callchain_entry_ctx { > u32 nr; > short contexts; > bool contexts_maxed; > + bool add_mark; > }; [ ... ] > @@ -1718,9 +1719,17 @@ DECLARE_PER_CPU(struct perf_callchain_entry, perf_callchain_entry); > > extern void perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs); > extern void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs); > + > +extern void __init_perf_callchain_ctx(struct perf_callchain_entry_ctx *ctx, > + struct perf_callchain_entry *entry, > + u32 max_stack, bool add_mark); > + > +extern void __get_perf_callchain_kernel(struct perf_callchain_entry_ctx *ctx, struct pt_regs *regs); > +extern void __get_perf_callchain_user(struct perf_callchain_entry_ctx *ctx, struct pt_regs *regs); > + > extern struct perf_callchain_entry * > get_perf_callchain(struct pt_regs *regs, bool kernel, bool user, > - u32 max_stack, bool crosstask, bool add_mark); > + u32 max_stack, bool crosstask); > diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c > index 4d53cdd13..e28b35c7e 100644 > --- a/kernel/bpf/stackmap.c > +++ b/kernel/bpf/stackmap.c > @@ -315,7 +315,7 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, > max_depth = sysctl_perf_event_max_stack; > > trace = get_perf_callchain(regs, kernel, user, max_depth, > - false, false); > + false); ^^^^^ removed last parameter Previously bpf_get_stackid() passed add_mark=false to get_perf_callchain(). Now the parameter is gone, but get_perf_callchain() hardcodes add_mark=true when calling __init_perf_callchain_ctx(). This means BPF stack traces will now include PERF_CONTEXT_KERNEL/PERF_CONTEXT_USER marker values when they didn't before. The markers get added by __get_perf_callchain_kernel() and __get_perf_callchain_user() when ctx->add_mark is true. These markers are sentinel values (-128 for KERNEL, -512 for USER) that appear in the trace IP array alongside actual instruction pointers. > > if (unlikely(!trace)) > /* couldn't fetch the stack trace */ > @@ -452,7 +452,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, > trace = get_callchain_entry_for_task(task, max_depth); > else > trace = get_perf_callchain(regs, kernel, user, max_depth, > - crosstask, false); > + crosstask); ^^^^^ same issue here Same problem in __bpf_get_stack() which was also passing add_mark=false. This changes BPF stack map behavior in user-visible ways: 1) bpf_get_stackid() hashes the entire trace->ip[] array including any markers, so identical stacks will now produce different stack IDs. 2) bpf_get_stack() copies trace->ip[] directly to userspace buffers, so BPF programs will see marker values mixed with instruction pointers. Can BPF tools handle these unexpected marker values? The commit message says "The next patch will modify the BPF part" but doesn't explain this intermediate behavior change. > diff --git a/kernel/events/callchain.c b/kernel/events/callchain.c > index 808c0d7a3..2c36e4906 100644 > --- a/kernel/events/callchain.c > +++ b/kernel/events/callchain.c [ ... ] > struct perf_callchain_entry * > get_perf_callchain(struct pt_regs *regs, bool kernel, bool user, > - u32 max_stack, bool crosstask, bool add_mark) > + u32 max_stack, bool crosstask) > { > struct perf_callchain_entry *entry; > struct perf_callchain_entry_ctx ctx; > - int rctx, start_entry_idx; > + int rctx; > > /* crosstask is not supported for user stacks */ > if (crosstask && user && !kernel) > @@ -232,34 +273,14 @@ get_perf_callchain(struct pt_regs *regs, bool kernel, bool user, > if (!entry) > return NULL; > > - ctx.entry = entry; > - ctx.max_stack = max_stack; > - ctx.nr = entry->nr = 0; > - ctx.contexts = 0; > - ctx.contexts_maxed = false; > + __init_perf_callchain_ctx(&ctx, entry, max_stack, true); ^^^^ Here's where add_mark is hardcoded to true for all callers. The old code allowed callers to control this via parameter. BPF was passing false, perf_callchain() was passing true. Now both get true unconditionally. [ ... ] > diff --git a/kernel/events/core.c b/kernel/events/core.c > index 7541f6f85..eb0f11059 100644 > --- a/kernel/events/core.c > +++ b/kernel/events/core.c > @@ -8218,7 +8218,7 @@ perf_callchain(struct perf_event *event, struct pt_regs *regs) > return &__empty_callchain; > > callchain = get_perf_callchain(regs, kernel, user, > - max_stack, crosstask, true); > + max_stack, crosstask); > return callchain ?: &__empty_callchain; > } ```
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
In-Reply-To-Subject: perf: Refactor get_perf_callchain
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/18882141979
Re: [PATCH bpf-next v4 1/2] perf: Refactor get_perf_callchain
Quoted 52 lines · 在 2025/10/29 01:09, [email protected] 写道:
```
commit59cfc61f83ad4ec998f4242ca000c554e3001b47
Author: Tao Chen <[email protected]>perf: Refactor
get_perf_callchainThis patch refactors
get_perf_callchain()by extracting logic into new helper functions to provide more flexible stack-sampling APIs for BPF.Quoted 10 lines
include/linux/perf_event.h +1
-0@@ -67,6 +67,7 @@ struct perf_callchain_entry_ctx {u32 nr;short contexts;bool contexts_maxed; +bool add_mark; };[ ... ]
Quoted 22 lines
include/linux/perf_event.h +9
-1@@ -1718,9 +1719,17 @@ DECLARE_PER_CPU(struct perf_callchain_entry, perf_callchain_entry);extern void perf_callchain_user (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs);extern void perf_callchain_kernel (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs); + +extern void __init_perf_callchain_ctx (struct perf_callchain_entry_ctx *ctx, +struct perf_callchain_entry *entry, +u32 max_stack,bool add_mark); + +extern void __get_perf_callchain_kernel (struct perf_callchain_entry_ctx *ctx,struct pt_regs *regs); +extern void __get_perf_callchain_user (struct perf_callchain_entry_ctx *ctx,struct pt_regs *regs); +extern struct perf_callchain_entry *get_perf_callchain (struct pt_regs *regs,bool kernel,bool user, -u32 max_stack,bool crosstask,bool add_mark); +u32 max_stack,bool crosstask);kernel/bpf/stackmap.c +0
-0kernel/bpf/stackmap.c BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, · line 318
Quoted 5 lines
kernel/bpf/stackmap.c +1
-1@@ -315,7 +315,7 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, max_depth = sysctl_perf_event_max_stack; trace =get_perf_callchain (regs, kernel, user, max_depth, - false, false); + false);^^^^^ removed last parameter
Quoted 6 lines
Previously
bpf_get_stackid()passed add_mark=false toget_perf_callchain(). Now the parameter is gone, butget_perf_callchain()hardcodes add_mark=true when calling__init_perf_callchain_ctx(). This means BPF stack traces will now include PERF_CONTEXT_KERNEL/PERF_CONTEXT_USER marker values when they didn't before.
The entire modification involves changes to two subsystems. To make the review easier, the get_perf_callchain has been simply adapted here. In the next patch, BPF will no longer use get_perf_callchain, and add_mark will be set to false.
Quoted 88 lines
The markers get added by
__get_perf_callchain_kernel()and__get_perf_callchain_user()whenctx->add_markis true. These markers are sentinel values (-128 for KERNEL, -512 for USER) that appear in the trace IP array alongside actual instruction pointers.kernel/bpf/stackmap.c static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, · line 455
Quoted 9 lines
kernel/bpf/stackmap.c +1
-1if (unlikely (!trace))/* couldn't fetch the stack trace */ @@ -452,7 +452,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, trace =get_callchain_entry_for_task (task, max_depth); else trace =get_perf_callchain (regs, kernel, user, max_depth, - crosstask, false); + crosstask);^^^^^ same issue here
Same problem in
__bpf_get_stack()which was also passing add_mark=false.This changes BPF stack map behavior in user-visible ways:
1)
bpf_get_stackid()hashes the entire trace->ip[] array including any markers, so identical stacks will now produce different stack IDs.2)
bpf_get_stack()copies trace->ip[] directly to userspace buffers, so BPF programs will see marker values mixed with instruction pointers.Can BPF tools handle these unexpected marker values? The commit message says "The next patch will modify the BPF part" but doesn't explain this intermediate behavior change.
Quoted 4 lines
kernel/events/callchain.c +0
-0[ ... ]
Quoted 13 lines
struct
perf_callchain_entry* get_perf_callchain(structpt_regs*regs, bool kernel, bool user,kernel/events/callchain.c +2
-2- u32 max_stack, bool crosstask, bool add_mark) + u32 max_stack, bool crosstask) {struct perf_callchain_entry *entry;struct perf_callchain_entry_ctx ctx; -int rctx, start_entry_idx; +int rctx;/* crosstask is not supported for user stacks */ if (crosstask && user && !kernel)kernel/events/callchain.c get_perf_callchain(struct pt_regs *regs, bool kernel, bool user, · line 276
Quoted 9 lines
kernel/events/callchain.c +1
-5@@ -232,34 +273,14 @@ get_perf_callchain(struct pt_regs *regs, bool kernel, bool user,if (!entry)return NULL ; - ctx.entry = entry; - ctx.max_stack = max_stack; - ctx.nr = entry->nr =0 ; - ctx.contexts =0 ; - ctx.contexts_maxed = false; +__init_perf_callchain_ctx (&ctx, entry, max_stack, true);^^^^Here's where
add_markis hardcoded to true for all callers. The old code allowed callers to control this via parameter. BPF was passing false,perf_callchain()was passing true. Now both get true unconditionally.[ ... ]
kernel/events/core.c perf_callchain(struct perf_event *event, struct pt_regs *regs) · line 8223
Quoted 12 lines
kernel/events/core.c +1
-1@@ -8218,7 +8218,7 @@ perf_callchain(struct perf_event *event, struct pt_regs *regs)return &__empty_callchain; callchain =get_perf_callchain (regs, kernel, user, - max_stack, crosstask, true); + max_stack, crosstask);return callchain ?: &__empty_callchain; }```
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdIn-Reply-To-Subject:
perf: Refactor get_perf_callchain
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/18882141979
Signature · 2 lines
Best Regards Tao Chen
Yonghong Song
reviewerRe: [PATCH bpf-next v4 1/2] perf: Refactor get_perf_callchain
In reply to Tao ChenQuoted 52 lines · Tao Chen
From BPF stack map, we want to use our own buffers to avoid
unnecessary copy and ensure that the buffer will not be overwritten by other preemptive tasks. Peter suggested provide more flexible stack-sampling APIs, which can be used in BPF, and we can still use the perf callchain entry with the help of these APIs. The next patch will modify the BPF part.Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Tao Chen <[email protected]>--- include/linux/perf_event.h | 11 +++++- kernel/bpf/stackmap.c | 4 +- kernel/events/callchain.c | 75 ++++++++++++++++++++++++-------------- kernel/events/core.c | 2 +- 4 files changed, 61 insertions(+), 31 deletions(-)include/linux/perf_event.h +10
-1@@ -67,6 +67,7 @@ struct perf_callchain_entry_ctx {u32 nr;short contexts;bool contexts_maxed; +bool add_mark; }; typedef unsigned long (*perf_copy_f)(void *dst, const void *src,@@ -1718,9 +1719,17 @@ DECLARE_PER_CPU(struct perf_callchain_entry, perf_callchain_entry);extern void perf_callchain_user (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs);extern void perf_callchain_kernel (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs); + +extern void __init_perf_callchain_ctx (struct perf_callchain_entry_ctx *ctx, +struct perf_callchain_entry *entry, +u32 max_stack,bool add_mark); + +extern void __get_perf_callchain_kernel (struct perf_callchain_entry_ctx *ctx,struct pt_regs *regs); +extern void __get_perf_callchain_user (struct perf_callchain_entry_ctx *ctx,struct pt_regs *regs); +extern struct perf_callchain_entry *get_perf_callchain (struct pt_regs *regs,bool kernel,bool user, -u32 max_stack,bool crosstask,bool add_mark); +u32 max_stack,bool crosstask);extern int get_callchain_buffers (int max_stack);extern void put_callchain_buffers (void );extern struct perf_callchain_entry *get_callchain_entry (int *rctx);kernel/bpf/stackmap.c +0
-0
kernel/bpf/stackmap.c BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, · line 318
Quoted 5 lines
kernel/bpf/stackmap.c +1
-1@@ -315,7 +315,7 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, max_depth = sysctl_perf_event_max_stack; trace =get_perf_callchain (regs, kernel, user, max_depth, - false, false); + false);
This is not a refactor. Here, the add_mark parameter is removed. The 'add_mark' value here is expected to be false, but later get_perf_callchain(...) has 'add_mark' is true in __init_perf_callchain_ctx().
Applying this patch only on top of bpf-next master branch, we will have the following crash:
[ 457.730077] bpf_testmod: oh no, recursing into test_1, recursion_misses 1 [ 460.221871] BUG: unable to handle page fault for address: fffa3bfffffff000 [ 460.221912] #PF: supervisor read access in kernel mode [ 460.221912] #PF: error_code(0x0000) - not-present page [ 460.221912] PGD 1e0ef1067 P4D 1e0ef0067 PUD 1e0eef067 PMD 1e0eee067 PTE 0 [ 460.221912] Oops: Oops: 0000 [#1] SMP KASAN NOPTI [ 460.221912] CPU: 2 UID: 0 PID: 2012 Comm: test_progs Tainted: G W OE 6.18.0-rc4-gafe2e8 [ 460.221912] Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE [ 460.221912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-pr4 [ 460.221912] RIP: 0010:kasan_check_range+0x183/0x2c0 [ 460.221912] Code: 41 bf 08 00 00 00 41 29 ef 4d 01 fb 4d 29 de 4d 89 f4 4d 8d 6c 24 07 4d 85 e4 4d 0fd [ 460.221912] RSP: 0018:ff110001193bfc78 EFLAGS: 00010206 [ 460.221912] RAX: ffd1ffffffd5b301 RBX: dffffc0000000001 RCX: ffffffff819a2ecb [ 460.221912] RDX: 0000000000000001 RSI: 00000000ffffffb0 RDI: ffd1ffffffd5b360 [ 460.221912] RBP: 0000000000000004 R08: ffd20000ffd5b30f R09: 1ffa40001ffab661 [ 460.221912] R10: dffffc0000000000 R11: fffa3bfffffab670 R12: 000000001ffffff2 [ 460.221912] R13: 0000000003ff58cc R14: 0000000000053990 R15: 0000000000000000 [ 460.221912] FS: 00007f358c6460c0(0000) GS:ff110002384b4000(0000) knlGS:0000000000000000 [ 460.221912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 460.221912] CR2: fffa3bfffffff000 CR3: 000000011468c006 CR4: 0000000000371ef0 [ 460.221912] Call Trace: [ 460.221912] <TASK> [ 460.221912] __asan_memset+0x22/0x50 [ 460.221912] __bpf_get_stack+0x6eb/0x7a0 [ 460.221912] ? bpf_perf_event_output_raw_tp+0x58c/0x6c0 [ 460.221912] bpf_get_stack+0x1d/0x30 [ 460.221912] bpf_get_stack_raw_tp+0x148/0x180 [ 460.221912] bpf_prog_40e346a03dc2914c_bpf_prog1+0x169/0x1af [ 460.221912] bpf_trace_run2+0x1bc/0x350 [ 460.221912] ? bpf_trace_run2+0x104/0x350 [ 460.221912] ? trace_sys_enter+0x6b/0xf0 [ 460.221912] __bpf_trace_sys_enter+0x38/0x60 [ 460.221912] trace_sys_enter+0xa7/0xf0 [ 460.221912] syscall_trace_enter+0xfc/0x160 [ 460.221912] ? entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 460.221912] do_syscall_64+0x5a/0xfa0 [ 460.221912] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 460.221912] entry_SYSCALL_64_after_hwframe+0x76/0x7e
Quoted 126 lines
kernel/bpf/stackmap.c +1
-1if (unlikely (!trace))/* couldn't fetch the stack trace */ @@ -452,7 +452,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, trace =get_callchain_entry_for_task (task, max_depth); else trace =get_perf_callchain (regs, kernel, user, max_depth, - crosstask, false); + crosstask);if (unlikely (!trace) || trace->nr < skip) {if (may_fault)kernel/events/callchain.c +48
-27@@ -216,13 +216,54 @@ static void fixup_uretprobe_trampoline_entries(struct perf_callchain_entry *entr#endif } +void __init_perf_callchain_ctx (struct perf_callchain_entry_ctx *ctx, +struct perf_callchain_entry *entry, +u32 max_stack,bool add_mark) + +{ + ctx->entry = entry; + ctx->max_stack = max_stack; + ctx->nr = entry->nr =0 ; + ctx->contexts =0 ; + ctx->contexts_maxed = false; + ctx->add_mark = add_mark; +} + +void __get_perf_callchain_kernel (struct perf_callchain_entry_ctx *ctx,struct pt_regs *regs) +{ +if (user_mode (regs)) +return ; + +if (ctx->add_mark) +perf_callchain_store_context (ctx,PERF_CONTEXT_KERNEL ); +perf_callchain_kernel (ctx, regs); +} + +void __get_perf_callchain_user (struct perf_callchain_entry_ctx *ctx,struct pt_regs *regs) +{ +int start_entry_idx; + +if (!user_mode (regs)) { +if (current->flags & (PF_KTHREAD |PF_USER_WORKER )) +return ; + regs =task_pt_regs (current); + } + +if (ctx->add_mark) +perf_callchain_store_context (ctx,PERF_CONTEXT_USER ); + + start_entry_idx = ctx->nr; +perf_callchain_user (ctx, regs); +fixup_uretprobe_trampoline_entries (ctx->entry, start_entry_idx); +} +struct perf_callchain_entry *get_perf_callchain (struct pt_regs *regs,bool kernel,bool user, -u32 max_stack,bool crosstask,bool add_mark) +u32 max_stack,bool crosstask) {struct perf_callchain_entry *entry;struct perf_callchain_entry_ctx ctx; -int rctx, start_entry_idx; +int rctx;/* crosstask is not supported for user stacks */ if (crosstask && user && !kernel)@@ -232,34 +273,14 @@ get_perf_callchain(struct pt_regs *regs, bool kernel, bool user,if (!entry)return NULL ; - ctx.entry = entry; - ctx.max_stack = max_stack; - ctx.nr = entry->nr =0 ; - ctx.contexts =0 ; - ctx.contexts_maxed = false; +__init_perf_callchain_ctx (&ctx, entry, max_stack, true); -if (kernel && !user_mode (regs)) { -if (add_mark) -perf_callchain_store_context (&ctx,PERF_CONTEXT_KERNEL ); -perf_callchain_kernel (&ctx, regs); - } - -if (user && !crosstask) { -if (!user_mode (regs)) { -if (current->flags & (PF_KTHREAD |PF_USER_WORKER )) - goto exit_put; - regs =task_pt_regs (current); - } +if (kernel) +__get_perf_callchain_kernel (&ctx, regs); -if (add_mark) -perf_callchain_store_context (&ctx,PERF_CONTEXT_USER ); - - start_entry_idx = entry->nr; -perf_callchain_user (&ctx, regs); -fixup_uretprobe_trampoline_entries (entry, start_entry_idx); - } +if (user && !crosstask) +__get_perf_callchain_user (&ctx, regs); -exit_put:put_callchain_entry (rctx);return entry;kernel/events/core.c +1
-1@@ -8218,7 +8218,7 @@ perf_callchain(struct perf_event *event, struct pt_regs *regs)return &__empty_callchain; callchain =get_perf_callchain (regs, kernel, user, - max_stack, crosstask, true); + max_stack, crosstask);return callchain ?: &__empty_callchain; }
Re: [PATCH bpf-next v4 1/2] perf: Refactor get_perf_callchain
Quoted 58 lines · Yonghong Song
Quoted 55 lines · Tao Chen
From BPF stack map, we want to use our own buffers to avoid unnecessary copy and ensure that the buffer will not be overwritten by other preemptive tasks. Peter suggested provide more flexible stack-sampling APIs, which can be used in BPF, and we can still use the perf callchain entry with the help of these APIs. The next patch will modify the BPF part.
Signed-off-by Peter Zijlstra <[email protected]>Signed-off-by Tao Chen <[email protected]>---
include/linux/perf_event.h| 11 +++++- kernel/bpf/stackmap.c | 4 +- kernel/events/callchain.c | 75 ++++++++++++++++++++++++-------------- kernel/events/core.c | 2 +-4 files changed, 61 insertions(+), 31 deletions(-)include/linux/perf_event.h +9
-0@@ -67,6 +67,7 @@ struct perf_callchain_entry_ctx { u32 nr; short contexts; bool contexts_maxed; + bool add_mark; }; typedef unsigned long (*perf_copy_f)(void *dst, const void *src,@@ -1718,9 +1719,17 @@ DECLARE_PER_CPU(struct perf_callchain_entry, perf_callchain_entry); extern void perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs); extern void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs); + +extern void __init_perf_callchain_ctx (struct perf_callchain_entry_ctx *ctx, + struct perf_callchain_entry *entry, + u32 max_stack, bool add_mark); + +extern void __get_perf_callchain_kernel (struct perf_callchain_entry_ctx *ctx,struct pt_regs *regs); +extern void __get_perf_callchain_user (struct perf_callchain_entry_ctx *ctx,struct pt_regs *regs); +extern struct
perf_callchain_entry* get_perf_callchain(structpt_regs*regs, bool kernel, bool user, - u32max_stack, bool crosstask, booladd_mark); + u32max_stack, bool crosstask);
extern intget_callchain_buffers(int max_stack);
extern voidput_callchain_buffers(void);
extern structperf_callchain_entry*get_callchain_entry(int *rctx);kernel/bpf/stackmap.c +0
-0@@ -315,7 +315,7 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map,
Quoted 9 lines
kernel/bpf/stackmap.c BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, · line 318
Quoted 4 lines
kernel/bpf/stackmap.c +1
-1max_depth = sysctl_perf_event_max_stack; trace = get_perf_callchain(regs, kernel, user, max_depth, - false, false); + false);This is not a refactor. Here, the
add_markparameter is removed. The 'add_mark' value here is expected to be false, but laterget_perf_callchain(...)has 'add_mark' is true in__init_perf_callchain_ctx().
Hi Yonghong,
Thanks for your report, you are right. Maybe we should keep the get_perf_callchain parameters unchanged. I will change it in v5.
Quoted 164 lines
Applying this patch only on top of bpf-next master branch, we will have the following crash:
[ 457.730077]
bpf_testmod: oh no, recursing intotest_1,recursion_misses1
[ 460.221871] BUG: unable to handle page fault for address:fffa3bfffffff000
[ 460.221912] #PF: supervisor read access in kernel mode
[ 460.221912] #PF:error_code(0x0000)- not-present page
[ 460.221912] PGD1e0ef1067P4D1e0ef0067PUD1e0eef067PMD1e0eee067PTE 0
[ 460.221912] Oops: Oops: 0000 [#1] SMP KASAN NOPTI
[ 460.221912] CPU: 2 UID: 0 PID: 2012 Comm:test_progsTainted: G W OE 6.18.0-rc4-gafe2e8 [ 460.221912] Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[ 460.221912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-pr4 [ 460.221912] RIP: 0010:kasan_check_range+0x183/0x2c0
[ 460.221912] Code: 41 bf 08 00 00 00 41 29 ef 4d 01 fb 4d 29 de 4d 89 f4 4d 8d 6c 24 07 4d 85 e4 4d 0fd [ 460.221912] RSP: 0018:ff110001193bfc78 EFLAGS: 00010206
[ 460.221912] RAX:ffd1ffffffd5b301RBX:dffffc0000000001RCX:ffffffff819a2ecb
[ 460.221912] RDX: 0000000000000001 RSI:00000000ffffffb0RDI:ffd1ffffffd5b360
[ 460.221912] RBP: 0000000000000004 R08:ffd20000ffd5b30fR09:1ffa40001ffab661
[ 460.221912] R10:dffffc0000000000R11:fffa3bfffffab670R12:000000001ffffff2
[ 460.221912] R13:0000000003ff58ccR14: 0000000000053990 R15: 0000000000000000
[ 460.221912] FS: 00007f358c6460c0(0000) GS:ff110002384b4000(0000) knlGS:0000000000000000 [ 460.221912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 460.221912] CR2:fffa3bfffffff000CR3:000000011468c006CR4:0000000000371ef0
[ 460.221912] Call Trace:
[ 460.221912] <TASK>
[ 460.221912]__asan_memset+0x22/0x50
[ 460.221912]__bpf_get_stack+0x6eb/0x7a0
[ 460.221912] ?bpf_perf_event_output_raw_tp+0x58c/0x6c0
[ 460.221912]bpf_get_stack+0x1d/0x30
[ 460.221912]bpf_get_stack_raw_tp+0x148/0x180
[ 460.221912]bpf_prog_40e346a03dc2914c_bpf_prog1+0x169/0x1af
[ 460.221912]bpf_trace_run2+0x1bc/0x350
[ 460.221912] ?bpf_trace_run2+0x104/0x350
[ 460.221912] ?trace_sys_enter+0x6b/0xf0
[ 460.221912]__bpf_trace_sys_enter+0x38/0x60
[ 460.221912]trace_sys_enter+0xa7/0xf0
[ 460.221912]syscall_trace_enter+0xfc/0x160
[ 460.221912] ?entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 460.221912]do_syscall_64+0x5a/0xfa0
[ 460.221912] ?asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 460.221912]entry_SYSCALL_64_after_hwframe+0x76/0x7eQuoted 121 lines
kernel/bpf/stackmap.c +0
-0if (unlikely(!trace)) /* couldn't fetch the stack trace */@@ -452,7 +452,7 @@ static long __bpf_get_stack(struct pt_regs *regs,struct task_struct *task, trace = get_callchain_entry_for_task(task, max_depth);else
trace =get_perf_callchain(regs, kernel, user, max_depth, - crosstask, false);
+ crosstask);
if (unlikely(!trace) ||trace->nr< skip) {
if (may_fault)kernel/events/callchain.c +0
-0@@ -216,13 +216,54 @@ static voidfixup_uretprobe_trampoline_entries(struct
perf_callchain_entry*entr #endif
}kernel/events/callchain.c +14
-0+void __init_perf_callchain_ctx (struct perf_callchain_entry_ctx *ctx, + struct perf_callchain_entry *entry, + u32 max_stack, bool add_mark) + +{ + ctx->entry = entry; + ctx->max_stack = max_stack; + ctx->nr = entry->nr = 0; + ctx->contexts = 0; + ctx->contexts_maxed = false; + ctx->add_mark = add_mark; +} + +void __get_perf_callchain_kernel(struct perf_callchain_entry_ctx*ctx, struct
pt_regs*regs)kernel/events/callchain.c +10
-0+{ + if (user_mode(regs)) + return; + + if (ctx->add_mark) + perf_callchain_store_context(ctx, PERF_CONTEXT_KERNEL); + perf_callchain_kernel(ctx, regs); +} + +void __get_perf_callchain_user(struct perf_callchain_entry_ctx *ctx,struct
pt_regs*regs)kernel/events/callchain.c +17
-0+{ + int start_entry_idx; + + if (!user_mode(regs)) { + if (current->flags & (PF_KTHREAD | PF_USER_WORKER)) + return; + regs = task_pt_regs(current); + } + + if (ctx->add_mark) + perf_callchain_store_context(ctx, PERF_CONTEXT_USER); + + start_entry_idx = ctx->nr; + perf_callchain_user(ctx, regs); + fixup_uretprobe_trampoline_entries(ctx->entry, start_entry_idx); +} +struct
perf_callchain_entry* get_perf_callchain(structpt_regs*regs, bool kernel, bool user, - u32max_stack, bool crosstask, booladd_mark) + u32max_stack, bool crosstask)
{
structperf_callchain_entry*entry;
structperf_callchain_entry_ctxctx;
- int rctx,start_entry_idx;
+ int rctx;
/* crosstask is not supported for user stacks */ if (crosstask && user && !kernel)kernel/events/callchain.c +5
-25@@ -232,34 +273,14 @@ get_perf_callchain(struct pt_regs *regs, bool kernel, bool user, if (!entry) return NULL; - ctx.entry = entry; - ctx.max_stack = max_stack; - ctx.nr = entry->nr = 0; - ctx.contexts = 0; - ctx.contexts_maxed = false; + __init_perf_callchain_ctx(&ctx, entry, max_stack, true); - if (kernel && !user_mode(regs)) { - if (add_mark) - perf_callchain_store_context(&ctx, PERF_CONTEXT_KERNEL); - perf_callchain_kernel(&ctx, regs); - } - - if (user && !crosstask) { - if (!user_mode(regs)) { - if (current->flags & (PF_KTHREAD | PF_USER_WORKER)) - goto exit_put; - regs = task_pt_regs(current); - } + if (kernel) + __get_perf_callchain_kernel(&ctx, regs); - if (add_mark) - perf_callchain_store_context(&ctx, PERF_CONTEXT_USER); - - start_entry_idx = entry->nr; - perf_callchain_user(&ctx, regs); - fixup_uretprobe_trampoline_entries(entry, start_entry_idx); - } + if (user && !crosstask) + __get_perf_callchain_user(&ctx, regs); -exit_put: put_callchain_entry(rctx); return entry;kernel/events/core.c +1
-1@@ -8218,7 +8218,7 @@ perf_callchain(struct perf_event *event, struct pt_regs *regs) return &__empty_callchain; callchain = get_perf_callchain(regs, kernel, user, - max_stack, crosstask, true); + max_stack, crosstask); return callchain ?: &__empty_callchain; }
Signature · 2 lines
Best Regards Tao Chen
PATCH v4 2/2 bpf: Hold the perf callchain entry until used completely
In reply to Tao ChenAs Alexei noted, get_perf_callchain() return values may be reused if a task is preempted after the BPF program enters migrate disable mode. The perf_callchain_entres has a small stack of entries, and we can reuse it as follows:
1. get the perf callchain entry
2. BPF use...
3. put the perf callchain entry
--- kernel/bpf/stackmap.c | 61 ++++++++++++++++++++++++++++++++++--------- 1 file changed, 48 insertions(+), 13 deletions(-)
kernel/bpf/stackmap.c +48 -13
@@ -188,13 +188,12 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, }static struct perf_callchain_entry * -get_callchain_entry_for_task (struct task_struct *task,u32 max_depth) +get_callchain_entry_for_task (int *rctx,struct task_struct *task,u32 max_depth) {#ifdef CONFIG_STACKTRACE struct perf_callchain_entry *entry; -int rctx; - entry =get_callchain_entry (&rctx); + entry =get_callchain_entry (rctx);if (!entry)return NULL ;
@@ -216,8 +215,6 @@ get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) to[i] = (u64)(from[i]); } -put_callchain_entry (rctx); -return entry;#else /* CONFIG_STACKTRACE */ return NULL ;
@@ -297,6 +294,31 @@ static long __bpf_get_stackid(struct bpf_map *map,return id; } +static struct perf_callchain_entry * +bpf_get_perf_callchain (int *rctx,struct pt_regs *regs,bool kernel,bool user, +int max_stack,bool crosstask) +{ +struct perf_callchain_entry_ctx ctx; +struct perf_callchain_entry *entry; + + entry =get_callchain_entry (rctx); +if (unlikely (!entry)) +return NULL ; + +__init_perf_callchain_ctx (&ctx, entry, max_stack, false); +if (kernel) +__get_perf_callchain_kernel (&ctx, regs); +if (user && !crosstask) +__get_perf_callchain_user (&ctx, regs); + +return entry; +} + +static void bpf_put_callchain_entry (int rctx) +{ +put_callchain_entry (rctx); +} + BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, u64, flags) {
@@ -305,6 +327,7 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map,bool user = flags &BPF_F_USER_STACK ;struct perf_callchain_entry *trace;bool kernel = !user; +int rctx, ret;if (unlikely (flags & ~(BPF_F_SKIP_FIELD_MASK |BPF_F_USER_STACK |BPF_F_FAST_STACK_CMP |BPF_F_REUSE_STACKID )))
@@ -314,14 +337,15 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map,if (max_depth > sysctl_perf_event_max_stack) max_depth = sysctl_perf_event_max_stack; - trace =get_perf_callchain (regs, kernel, user, max_depth, - false); - + trace =bpf_get_perf_callchain (&rctx, regs, kernel, user, max_depth, false);if (unlikely (!trace))/* couldn't fetch the stack trace */ return -EFAULT ; -return __bpf_get_stackid (map, trace, flags); + ret =__bpf_get_stackid (map, trace, flags); +bpf_put_callchain_entry (rctx); + +return ret; }const struct bpf_func_proto bpf_get_stackid_proto = {
@@ -415,6 +439,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,bool kernel = !user;int err = -EINVAL ;u64 *ips; +int rctx;if (unlikely (flags & ~(BPF_F_SKIP_FIELD_MASK |BPF_F_USER_STACK |BPF_F_USER_BUILD_ID )))
@@ -449,17 +474,24 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,if (trace_in) trace = trace_in;else if (kernel && task) - trace =get_callchain_entry_for_task (task, max_depth); + trace =get_callchain_entry_for_task (&rctx, task, max_depth);else - trace =get_perf_callchain (regs, kernel, user, max_depth, - crosstask); + trace =bpf_get_perf_callchain (&rctx, regs, kernel, user, max_depth, crosstask); -if (unlikely (!trace) || trace->nr < skip) { +if (unlikely (!trace)) {if (may_fault)rcu_read_unlock (); goto err_fault; } +if (trace->nr < skip) { +if (may_fault) +rcu_read_unlock (); +if (!trace_in) +bpf_put_callchain_entry (rctx); + goto err_fault; + } + trace_nr = trace->nr - skip; trace_nr = (trace_nr <= num_elem) ? trace_nr : num_elem; copy_len = trace_nr * elem_size;
@@ -479,6 +511,9 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,if (may_fault)rcu_read_unlock (); +if (!trace_in) +bpf_put_callchain_entry (rctx); +if (user_build_id)stack_map_get_build_id_offset (buf, trace_nr, user, may_fault);
Signature · 1 line
2.48.1
Yonghong Song
reviewerRe: [PATCH bpf-next v4 2/2] bpf: Hold the perf callchain entry until used completely
Quoted 45 lines · Tao Chen
As Alexei noted,
get_perf_callchain()return values may be reused if a task is preempted after the BPF program enters migrate disable mode. Theperf_callchain_entreshas a small stack of entries, and we can reuse it as follows:1. get the perf callchain entry
2. BPF use...
3. put the perf callchain entrySigned-off-by Tao Chen <[email protected]>--- kernel/bpf/stackmap.c | 61 ++++++++++++++++++++++++++++++++++--------- 1 file changed, 48 insertions(+), 13 deletions(-)kernel/bpf/stackmap.c +2
-5@@ -188,13 +188,12 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, }static struct perf_callchain_entry * -get_callchain_entry_for_task (struct task_struct *task,u32 max_depth) +get_callchain_entry_for_task (int *rctx,struct task_struct *task,u32 max_depth) {#ifdef CONFIG_STACKTRACE struct perf_callchain_entry *entry; -int rctx; - entry =get_callchain_entry (&rctx); + entry =get_callchain_entry (rctx);if (!entry)return NULL ;@@ -216,8 +215,6 @@ get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) to[i] = (u64)(from[i]); } -put_callchain_entry (rctx); -return entry;#else /* CONFIG_STACKTRACE */ return NULL ;@@ -297,6 +294,31 @@ static long __bpf_get_stackid(struct bpf_map *map,return id;
kernel/bpf/stackmap.c static long __bpf_get_stackid(struct bpf_map *map, · line 304
Quoted 10 lines
kernel/bpf/stackmap.c +8
-0} +static struct perf_callchain_entry * +bpf_get_perf_callchain (int *rctx,struct pt_regs *regs,bool kernel,bool user, +int max_stack,bool crosstask) +{ +struct perf_callchain_entry_ctx ctx; +struct perf_callchain_entry *entry; + + entry =get_callchain_entry (rctx);
I think this may not work. Let us say we have two bpf programs both pinned to a particular cpu (migrate disabled but preempt enabled). get_callchain_entry() calls get_recursion_context() to get the buffer for a particulart level.
static inline int get_recursion_context(u8 *recursion)
{
unsigned char rctx = interrupt_context_level();
if (recursion[rctx])
return -1;
recursion[rctx]++;
barrier();
return rctx;
}It is possible that both tasks (at process level) may reach right before "recursion[rctx]++;".
In such cases, both tasks will be able to get buffer and this is not right.
To fix this, we either need to have preempt disable
in bpf side, or maybe we have some kind of atomic operation (cmpxchg or similar things), or maybe
has a preempt disable between if statement and recursion[rctx]++, so only one task can get buffer?
kernel/bpf/stackmap.c static long __bpf_get_stackid(struct bpf_map *map, · line 317
Quoted 13 lines
kernel/bpf/stackmap.c +13
-0+if (unlikely (!entry)) +return NULL ; + +__init_perf_callchain_ctx (&ctx, entry, max_stack, false); +if (kernel) +__get_perf_callchain_kernel (&ctx, regs); +if (user && !crosstask) +__get_perf_callchain_user (&ctx, regs); + +return entry; +} + +static void bpf_put_callchain_entry (int rctx)
we have bpf_get_perf_callchain(), maybe rename the above to bpf_put_perf_callchain()?
Quoted 4 lines
kernel/bpf/stackmap.c +4
-0+{ +put_callchain_entry (rctx); +} +
[...]
Re: [PATCH bpf-next v4 2/2] bpf: Hold the perf callchain entry until used completely
Quoted 78 lines · Yonghong Song
Quoted 44 lines · Tao Chen
As Alexei noted,
get_perf_callchain()return values may be reused if a task is preempted after the BPF program enters migrate disable mode. Theperf_callchain_entreshas a small stack of entries, and we can reuse it as follows:1. get the perf callchain entry
2. BPF use...
3. put the perf callchain entrySigned-off-by Tao Chen <[email protected]>---
kernel/bpf/stackmap.c| 61 ++++++++++++++++++++++++++++++++++---------1 file changed, 48 insertions(+), 13 deletions(-)kernel/bpf/stackmap.c +0
-0@@ -188,13 +188,12 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, }static struct
perf_callchain_entry*kernel/bpf/stackmap.c +1
-1-get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) +get_callchain_entry_for_task(int *rctx, struct task_struct *task, u32
max_depth)
{
#ifdefCONFIG_STACKTRACE
structperf_callchain_entry*entry;
- int rctx;
- entry =get_callchain_entry(&rctx);
+ entry =get_callchain_entry(rctx);
if (!entry)
return NULL;kernel/bpf/stackmap.c +0
-2@@ -216,8 +215,6 @@ get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) to[i] = (u64)(from[i]); } - put_callchain_entry(rctx); - return entry; #else /* CONFIG_STACKTRACE */ return NULL;@@ -297,6 +294,31 @@ static long __bpf_get_stackid(struct bpf_map *map, return id;kernel/bpf/stackmap.c static long __bpf_get_stackid(struct bpf_map *map, · line 304
Quoted 10 lines
kernel/bpf/stackmap.c +8
-0} +static struct perf_callchain_entry * +bpf_get_perf_callchain (int *rctx,struct pt_regs *regs,bool kernel,bool user, + int max_stack, bool crosstask) +{ + struct perf_callchain_entry_ctx ctx; + struct perf_callchain_entry *entry; + + entry = get_callchain_entry(rctx);I think this may not work. Let us say we have two bpf programs both pinned to a particular cpu (migrate disabled but preempt enabled).
get_callchain_entry()callsget_recursion_context()to get the buffer for a particulart level.static inline int
get_recursion_context(u8 *recursion){
unsigned char rctx =interrupt_context_level(); if (recursion[rctx])
return -1;
recursion[rctx]++;
barrier();
return rctx;
}It is possible that both tasks (at process level) may reach right before "recursion[rctx]++;".
In such cases, both tasks will be able to get buffer and this is not right.
Quoted 6 lines
To fix this, we either need to have preempt disable
in bpf side, or maybe we have some kind of atomic operation (cmpxchg or similar things), or maybe
has a preempt disable between if statement and recursion[rctx]++, so only one task can get buffer?
Thanks to your reminder, can we add preempt disable before and after get_callchain_entry, avoid affecting the original functions of perf.
Regarding multiple task preemption: if the entry is not released via put_callchain_entry, it appears that perf's buffer does not support recording the second task, so it returns directly here.
if (recursion[rctx])
return -1;Quoted 18 lines
Quoted 4 lines
kernel/bpf/stackmap.c +3
-0+ if (unlikely(!entry)) + return NULL; +kernel/bpf/stackmap.c static long __bpf_get_stackid(struct bpf_map *map, · line 317
Quoted 10 lines
kernel/bpf/stackmap.c +10
-0+ __init_perf_callchain_ctx(&ctx, entry, max_stack, false); + if (kernel) + __get_perf_callchain_kernel(&ctx, regs); + if (user && !crosstask) + __get_perf_callchain_user(&ctx, regs); + + return entry; +} + +static void bpf_put_callchain_entry (int rctx)we have bpf_get_perf_callchain(), maybe rename the above to
bpf_put_perf_callchain()?
Ack, thanks.
Quoted 7 lines
Quoted 4 lines
kernel/bpf/stackmap.c +4
-0+{ + put_callchain_entry(rctx); +} +[...]
Signature · 2 lines
Best Regards Tao Chen
Yonghong Song
reviewerRe: [PATCH bpf-next v4 2/2] bpf: Hold the perf callchain entry until used completely
Quoted 80 lines · Tao Chen
Quoted 79 lines · Yonghong Song
Quoted 45 lines · Tao Chen
As Alexei noted,
get_perf_callchain()return values may be reused if a task is preempted after the BPF program enters migrate disable mode. Theperf_callchain_entreshas a small stack of entries, and we can reuse it as follows:1. get the perf callchain entry
2. BPF use...
3. put the perf callchain entrySigned-off-by Tao Chen <[email protected]>---
kernel/bpf/stackmap.c| 61
++++++++++++++++++++++++++++++++++---------1 file changed, 48 insertions(+), 13 deletions(-)kernel/bpf/stackmap.c +0
-0@@ -188,13 +188,12 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, }static struct
perf_callchain_entry*kernel/bpf/stackmap.c +1
-1-get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) +get_callchain_entry_for_task(int *rctx, struct task_struct *task,u32
max_depth)
{
#ifdefCONFIG_STACKTRACE
structperf_callchain_entry*entry;
- int rctx;
- entry =get_callchain_entry(&rctx);
+ entry =get_callchain_entry(rctx);
if (!entry)
return NULL;kernel/bpf/stackmap.c +0
-2@@ -216,8 +215,6 @@ get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) to[i] = (u64)(from[i]); } - put_callchain_entry(rctx); - return entry; #else /* CONFIG_STACKTRACE */ return NULL;@@ -297,6 +294,31 @@ static long __bpf_get_stackid(struct bpf_map *map, return id;kernel/bpf/stackmap.c static long __bpf_get_stackid(struct bpf_map *map, · line 304
Quoted 10 lines
kernel/bpf/stackmap.c +8
-0} +static struct perf_callchain_entry * +bpf_get_perf_callchain (int *rctx,struct pt_regs *regs,bool kernel,bool user, + int max_stack, bool crosstask) +{ + struct perf_callchain_entry_ctx ctx; + struct perf_callchain_entry *entry; + + entry = get_callchain_entry(rctx);I think this may not work. Let us say we have two bpf programs both pinned to a particular cpu (migrate disabled but preempt enabled).
get_callchain_entry()callsget_recursion_context()to get the buffer for a particulart level.static inline int
get_recursion_context(u8 *recursion){
unsigned char rctx =interrupt_context_level(); if (recursion[rctx])
return -1;
recursion[rctx]++;
barrier();
return rctx;
}It is possible that both tasks (at process level) may reach right before "recursion[rctx]++;".
In such cases, both tasks will be able to get buffer and this is not right.
Quoted 8 lines
Quoted 6 lines
To fix this, we either need to have preempt disable
in bpf side, or maybe we have some kind of atomic operation (cmpxchg or similar things), or maybe
has a preempt disable between if statement and recursion[rctx]++, so only one task can get buffer?Thanks to your reminder, can we add preempt disable before and after
get_callchain_entry, avoid affecting the original functions of perf.
Yes, we get two get_callchain_entry() call site:
bpf/stackmap.c: entry = get_callchain_entry(&rctx);
events/callchain.c: entry = get_callchain_entry(&rctx);
We need to have preempt_disable()/preempt_enable() around them.
Another choice maybe adds preempt_disable/enable() for get_callchain_entry() and get_perf_callchain() in stackmap.c, assuming these two function usage in other places are for interrupts (softirq, hardirq and nmi) so they are okay.
But maybe the following is better?
kernel/events/internal.h +1 -4
@@ -214,12 +214,9 @@ static inline int get_recursion_context(u8 *recursion) {unsigned char rctx =interrupt_context_level (); -if (recursion[rctx]) +if (cmpxchg (&recursion[rctx],0 ,1 ) !=0 )return -1 ; - recursion[rctx]++; -barrier (); -return rctx; }
Quoted 36 lines
Regarding multiple task preemption: if the entry is not released via
put_callchain_entry, it appears that perf's buffer does not support recording the second task, so it returns directly here.if (recursion[rctx])
return -1;Quoted 18 lines
Quoted 4 lines
kernel/bpf/stackmap.c +3
-0+ if (unlikely(!entry)) + return NULL; +kernel/bpf/stackmap.c static long __bpf_get_stackid(struct bpf_map *map, · line 317
Quoted 10 lines
kernel/bpf/stackmap.c +10
-0+ __init_perf_callchain_ctx(&ctx, entry, max_stack, false); + if (kernel) + __get_perf_callchain_kernel(&ctx, regs); + if (user && !crosstask) + __get_perf_callchain_user(&ctx, regs); + + return entry; +} + +static void bpf_put_callchain_entry (int rctx)we have bpf_get_perf_callchain(), maybe rename the above to
bpf_put_perf_callchain()?Ack, thanks.
Quoted 7 lines
Quoted 4 lines
kernel/bpf/stackmap.c +4
-0+{ + put_callchain_entry(rctx); +} +[...]
Re: [PATCH bpf-next v4 2/2] bpf: Hold the perf callchain entry until used completely
Quoted 112 lines · Yonghong Song
Quoted 80 lines · Tao Chen
Quoted 79 lines · Yonghong Song
Quoted 45 lines · Tao Chen
As Alexei noted,
get_perf_callchain()return values may be reused if a task is preempted after the BPF program enters migrate disable mode. Theperf_callchain_entreshas a small stack of entries, and we can reuse it as follows:1. get the perf callchain entry
2. BPF use...
3. put the perf callchain entrySigned-off-by Tao Chen <[email protected]>---
kernel/bpf/stackmap.c| 61 +++++++++++++++++++++++++++++++++
+---------1 file changed, 48 insertions(+), 13 deletions(-)kernel/bpf/stackmap.c +0
-0@@ -188,13 +188,12 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, }static struct
perf_callchain_entry*kernel/bpf/stackmap.c +1
-1-get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) +get_callchain_entry_for_task(int *rctx, struct task_struct *task,u32
max_depth)
{
#ifdefCONFIG_STACKTRACE
structperf_callchain_entry*entry;
- int rctx;
- entry =get_callchain_entry(&rctx);
+ entry =get_callchain_entry(rctx);
if (!entry)
return NULL;kernel/bpf/stackmap.c +0
-2@@ -216,8 +215,6 @@ get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) to[i] = (u64)(from[i]); } - put_callchain_entry(rctx); - return entry; #else /* CONFIG_STACKTRACE */ return NULL;@@ -297,6 +294,31 @@ static long __bpf_get_stackid(struct bpf_map *map, return id;kernel/bpf/stackmap.c static long __bpf_get_stackid(struct bpf_map *map, · line 304
Quoted 10 lines
kernel/bpf/stackmap.c +8
-0} +static struct perf_callchain_entry * +bpf_get_perf_callchain (int *rctx,struct pt_regs *regs,bool kernel,bool user, + int max_stack, bool crosstask) +{ + struct perf_callchain_entry_ctx ctx; + struct perf_callchain_entry *entry; + + entry = get_callchain_entry(rctx);I think this may not work. Let us say we have two bpf programs both pinned to a particular cpu (migrate disabled but preempt enabled).
get_callchain_entry()callsget_recursion_context()to get the buffer for a particulart level.static inline int
get_recursion_context(u8 *recursion){
unsigned char rctx =interrupt_context_level(); if (recursion[rctx])
return -1;
recursion[rctx]++;
barrier();
return rctx;
}It is possible that both tasks (at process level) may reach right before "recursion[rctx]++;".
In such cases, both tasks will be able to get buffer and this is not right.Quoted 8 lines
Quoted 6 lines
To fix this, we either need to have preempt disable
in bpf side, or maybe we have some kind of atomic operation (cmpxchg or similar things), or maybe
has a preempt disable between if statement and recursion[rctx]++, so only one task can get buffer?Thanks to your reminder, can we add preempt disable before and after
get_callchain_entry, avoid affecting the original functions of perf.Yes, we get two
get_callchain_entry()call site:
bpf/stackmap.c: entry =get_callchain_entry(&rctx); events/callchain.c: entry =get_callchain_entry(&rctx);
We need to have preempt_disable()/preempt_enable() around them.Another choice maybe adds preempt_disable/enable() for
get_callchain_entry()andget_perf_callchain()instackmap.c, assuming these two function usage in other places are for interrupts (softirq, hardirq and nmi) so they are okay.But maybe the following is better?
kernel/events/internal.h +0
-0@@ -214,12 +214,9 @@ static inline int get_recursion_context(u8 *recursion) { unsigned char rctx = interrupt_context_level();
kernel/events/internal.h static inline int get_recursion_context(u8 *recursion) · line 222
Quoted 10 lines
kernel/events/internal.h +1
-4- if (recursion[rctx]) + if (cmpxchg(&recursion[rctx], 0, 1) != 0) return -1; - recursion[rctx]++; - barrier(); - return rctx; }
Agree, this seems to have fewer side effects, thanks.
Quoted 37 lines
Quoted 36 lines
Regarding multiple task preemption: if the entry is not released via
put_callchain_entry, it appears that perf's buffer does not support recording the second task, so it returns directly here.if (recursion[rctx])
return -1;Quoted 18 lines
Quoted 4 lines
kernel/bpf/stackmap.c +3
-0+ if (unlikely(!entry)) + return NULL; +kernel/bpf/stackmap.c static long __bpf_get_stackid(struct bpf_map *map, · line 317
Quoted 10 lines
kernel/bpf/stackmap.c +10
-0+ __init_perf_callchain_ctx(&ctx, entry, max_stack, false); + if (kernel) + __get_perf_callchain_kernel(&ctx, regs); + if (user && !crosstask) + __get_perf_callchain_user(&ctx, regs); + + return entry; +} + +static void bpf_put_callchain_entry (int rctx)we have bpf_get_perf_callchain(), maybe rename the above to
bpf_put_perf_callchain()?Ack, thanks.
Quoted 7 lines
Quoted 4 lines
kernel/bpf/stackmap.c +4
-0+{ + put_callchain_entry(rctx); +} +[...]
Signature · 2 lines
Best Regards Tao Chen