Pass external callchain entry to get_perf_callchain
Subsystems (3)
v7 cover Pass external callchain entry to get_perf_callchain
Background
==========
Alexei noted we should use preempt_disable to protect get_perf_callchain in bpf stackmap.
https://lore.kernel.org/bpf/CAADnVQ+s8B7-fvR1TNO-bniSyKv57cH_ihRszmZV7pQDyV=VDQ@mail.gmail.com
A previous patch was submitted to attempt fixing this issue. And Andrii suggested teach get_perf_callchain to let us pass that buffer directly to avoid that unnecessary copy.
https://lore.kernel.org/bpf/[email protected]
Proposed Solution
=================
Add external perf_callchain_entry parameter for get_perf_callchain to allow us to use external buffer from BPF side. The biggest advantage is that it can reduce unnecessary copies.
Todo
====
But I'm not sure if this modification is appropriate. After all, the implementation of get_callchain_entry in the perf subsystem seems much more complex than directly using an external buffer.
Comments and suggestions are always welcome.
Change list:
- v1 -> v2
From Jiri
- rebase code, fix conflict
- v1: https://lore.kernel.org/bpf/[email protected]
- v2 -> v3:
From Andrii
- entries per CPU used in a stack-like fashion
- v2: https://lore.kernel.org/bpf/[email protected]
- v3 -> v4:
From Peter
- refactor get_perf_callchain and add three new APIs to use perf callchain easily.
From Andrii
- reuse the perf callchain management.
- rename patch1 and patch2.
- v3: https://lore.kernel.org/bpf/[email protected]
- v4 -> v5:
From Yonghong
- keep add_mark false in stackmap when refactor get_perf_callchain in
patch1.
- add atomic operation in get_recursion_context in patch2.
- rename bpf_put_callchain_entry with bpf_put_perf_callchain in
patch3. - rebase bpf-next master.
- v4: https://lore.kernel.org/bpf/[email protected]
- v5 -> v6:
From Peter
- disable preemption from BPF side in patch2.
From AI
- use ctx->entry->nr instead of ctx->nr in patch1.
- v5: https://lore.kernel.org/bpf/[email protected]
- v6 -> v7:
From yonghong
- Add ack in patch2
From AI
- resolve conflict
- v6: https://lore.kernel.org/bpf/[email protected]
Tao Chen (2):
perf: Refactor get_perf_callchain
bpf: Hold the perf callchain entry until used completely
include/linux/perf_event.h | 10 ++++ kernel/bpf/stackmap.c | 68 +++++++++++++++++++++----- kernel/events/callchain.c | 99 +++++++++++++++++++++++--------------- 3 files changed, 126 insertions(+), 51 deletions(-)
Signature · 1 line
2.48.1
PATCH v7 1/2 perf: Refactor get_perf_callchain
From BPF stack map, we want to ensure that the callchain buffer will not be overwritten by other preemptive tasks. Peter suggested provide more flexible stack-sampling APIs, which can be used in BPF, and we can still use the perf callchain entry with the help of these APIs. The next patch will modify the BPF part.
In the future, these APIs will also make it convenient for us to add stack-sampling kfuncs in the eBPF subsystem, just as Andrii and Alexei discussed earlier.
--- include/linux/perf_event.h | 10 ++++ kernel/events/callchain.c | 99 +++++++++++++++++++++++--------------- 2 files changed, 70 insertions(+), 39 deletions(-)
include/linux/perf_event.h +10 -0
@@ -67,6 +67,7 @@ struct perf_callchain_entry_ctx {u32 nr;short contexts;bool contexts_maxed; +bool add_mark; }; typedef unsigned long (*perf_copy_f)(void *dst, const void *src,
@@ -1718,6 +1719,15 @@ DECLARE_PER_CPU(struct perf_callchain_entry, perf_callchain_entry);extern void perf_callchain_user (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs);extern void perf_callchain_kernel (struct perf_callchain_entry_ctx *entry,struct pt_regs *regs); + +extern void __init_perf_callchain_ctx (struct perf_callchain_entry_ctx *ctx, +struct perf_callchain_entry *entry, +u32 max_stack,bool add_mark); + +extern void __get_perf_callchain_kernel (struct perf_callchain_entry_ctx *ctx,struct pt_regs *regs); +extern void __get_perf_callchain_user (struct perf_callchain_entry_ctx *ctx,struct pt_regs *regs, +u64 defer_cookie); +extern struct perf_callchain_entry *get_perf_callchain (struct pt_regs *regs,bool kernel,bool user,u32 max_stack,bool crosstask,bool add_mark,u64 defer_cookie);
kernel/events/callchain.c +60 -39
119 lines not on this page · raw message
Signature · 1 line
2.48.1
PATCH v7 2/2 bpf: Hold the perf callchain entry until used completely
In reply to Tao ChenAs Alexei noted, get_perf_callchain() return values may be reused if a task is preempted after the BPF program enters migrate disable mode. The perf_callchain_entres has a small stack of entries, and we can reuse it as follows:
1. get the perf callchain entry
2. BPF use...
3. put the perf callchain entry
And Peter suggested that get_recursion_context used with preemption disabled, so we should disable preemption at BPF side.
--- kernel/bpf/stackmap.c | 68 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 56 insertions(+), 12 deletions(-)
kernel/bpf/stackmap.c +56 -12
@@ -210,13 +210,14 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, }static struct perf_callchain_entry * -get_callchain_entry_for_task (struct task_struct *task,u32 max_depth) +get_callchain_entry_for_task (int *rctx,struct task_struct *task,u32 max_depth) {#ifdef CONFIG_STACKTRACE struct perf_callchain_entry *entry; -int rctx; - entry =get_callchain_entry (&rctx); +preempt_disable (); + entry =get_callchain_entry (rctx); +preempt_enable ();if (!entry)return NULL ;
@@ -238,8 +239,6 @@ get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) to[i] = (u64)(from[i]); } -put_callchain_entry (rctx); -return entry;#else /* CONFIG_STACKTRACE */ return NULL ;
@@ -320,6 +319,34 @@ static long __bpf_get_stackid(struct bpf_map *map,return id; } +static struct perf_callchain_entry * +bpf_get_perf_callchain (int *rctx,struct pt_regs *regs,bool kernel,bool user, +int max_stack,bool crosstask) +{ +struct perf_callchain_entry_ctx ctx; +struct perf_callchain_entry *entry; + +preempt_disable (); + entry =get_callchain_entry (rctx); +preempt_enable (); + +if (unlikely (!entry)) +return NULL ; + +__init_perf_callchain_ctx (&ctx, entry, max_stack, false); +if (kernel) +__get_perf_callchain_kernel (&ctx, regs); +if (user && !crosstask) +__get_perf_callchain_user (&ctx, regs,0 ); + +return entry; +} + +static void bpf_put_perf_callchain (int rctx) +{ +put_callchain_entry (rctx); +} + BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, u64, flags) {
@@ -328,20 +355,25 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map,struct perf_callchain_entry *trace;bool kernel = !user;u32 max_depth; +int rctx, ret;if (unlikely (flags & ~(BPF_F_SKIP_FIELD_MASK |BPF_F_USER_STACK |BPF_F_FAST_STACK_CMP |BPF_F_REUSE_STACKID )))return -EINVAL ; max_depth =stack_map_calculate_max_depth (map->value_size, elem_size, flags); - trace =get_perf_callchain (regs, kernel, user, max_depth, - false, false,0 ); + + trace =bpf_get_perf_callchain (&rctx, regs, kernel, user, max_depth, + false);if (unlikely (!trace))/* couldn't fetch the stack trace */ return -EFAULT ; -return __bpf_get_stackid (map, trace, flags); + ret =__bpf_get_stackid (map, trace, flags); +bpf_put_perf_callchain (rctx); + +return ret; }const struct bpf_func_proto bpf_get_stackid_proto = {
@@ -435,6 +467,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,bool kernel = !user;int err = -EINVAL ;u64 *ips; +int rctx;if (unlikely (flags & ~(BPF_F_SKIP_FIELD_MASK |BPF_F_USER_STACK |BPF_F_USER_BUILD_ID )))
@@ -467,18 +500,26 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, trace = trace_in; trace->nr =min_t (u32, trace->nr, max_depth); }else if (kernel && task) { - trace =get_callchain_entry_for_task (task, max_depth); + trace =get_callchain_entry_for_task (&rctx, task, max_depth); }else { - trace =get_perf_callchain (regs, kernel, user, max_depth, - crosstask, false,0 ); + trace =bpf_get_perf_callchain (&rctx, regs, kernel, user, max_depth, + crosstask); } -if (unlikely (!trace) || trace->nr < skip) { +if (unlikely (!trace)) {if (may_fault)rcu_read_unlock (); goto err_fault; } +if (trace->nr < skip) { +if (may_fault) +rcu_read_unlock (); +if (!trace_in) +bpf_put_perf_callchain (rctx); + goto err_fault; + } + trace_nr = trace->nr - skip; copy_len = trace_nr * elem_size;
@@ -497,6 +538,9 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,if (may_fault)rcu_read_unlock (); +if (!trace_in) +bpf_put_perf_callchain (rctx); +if (user_build_id)stack_map_get_build_id_offset (buf, trace_nr, user, may_fault);
Signature · 1 line
2.48.1
Re: [PATCH bpf-next v7 2/2] bpf: Hold the perf callchain entry until used completely
Quoted 153 lines · Tao Chen
As Alexei noted,
get_perf_callchain()return values may be reused if a task is preempted after the BPF program enters migrate disable mode. Theperf_callchain_entreshas a small stack of entries, and we can reuse it as follows:1. get the perf callchain entry
2. BPF use...
3. put the perf callchain entryAnd Peter suggested that
get_recursion_contextused with preemption disabled, so we should disable preemption at BPF side.Acked-by Yonghong Song <[email protected]>Signed-off-by Tao Chen <[email protected]>--- kernel/bpf/stackmap.c | 68 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 56 insertions(+), 12 deletions(-)kernel/bpf/stackmap.c +53
-12@@ -210,13 +210,14 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, }static struct perf_callchain_entry * -get_callchain_entry_for_task (struct task_struct *task,u32 max_depth) +get_callchain_entry_for_task (int *rctx,struct task_struct *task,u32 max_depth) {#ifdef CONFIG_STACKTRACE struct perf_callchain_entry *entry; -int rctx; - entry =get_callchain_entry (&rctx); +preempt_disable (); + entry =get_callchain_entry (rctx); +preempt_enable ();if (!entry)return NULL ;@@ -238,8 +239,6 @@ get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) to[i] = (u64)(from[i]); } -put_callchain_entry (rctx); -return entry;#else /* CONFIG_STACKTRACE */ return NULL ;@@ -320,6 +319,34 @@ static long __bpf_get_stackid(struct bpf_map *map,return id; } +static struct perf_callchain_entry * +bpf_get_perf_callchain (int *rctx,struct pt_regs *regs,bool kernel,bool user, +int max_stack,bool crosstask) +{ +struct perf_callchain_entry_ctx ctx; +struct perf_callchain_entry *entry; + +preempt_disable (); + entry =get_callchain_entry (rctx); +preempt_enable (); + +if (unlikely (!entry)) +return NULL ; + +__init_perf_callchain_ctx (&ctx, entry, max_stack, false); +if (kernel) +__get_perf_callchain_kernel (&ctx, regs); +if (user && !crosstask) +__get_perf_callchain_user (&ctx, regs,0 ); + +return entry; +} + +static void bpf_put_perf_callchain (int rctx) +{ +put_callchain_entry (rctx); +} + BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, u64, flags) {@@ -328,20 +355,25 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map,struct perf_callchain_entry *trace;bool kernel = !user;u32 max_depth; +int rctx, ret;if (unlikely (flags & ~(BPF_F_SKIP_FIELD_MASK |BPF_F_USER_STACK |BPF_F_FAST_STACK_CMP |BPF_F_REUSE_STACKID )))return -EINVAL ; max_depth =stack_map_calculate_max_depth (map->value_size, elem_size, flags); - trace =get_perf_callchain (regs, kernel, user, max_depth, - false, false,0 ); + + trace =bpf_get_perf_callchain (&rctx, regs, kernel, user, max_depth, + false);if (unlikely (!trace))/* couldn't fetch the stack trace */ return -EFAULT ; -return __bpf_get_stackid (map, trace, flags); + ret =__bpf_get_stackid (map, trace, flags); +bpf_put_perf_callchain (rctx); + +return ret; }const struct bpf_func_proto bpf_get_stackid_proto = {@@ -435,6 +467,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,bool kernel = !user;int err = -EINVAL ;u64 *ips; +int rctx;if (unlikely (flags & ~(BPF_F_SKIP_FIELD_MASK |BPF_F_USER_STACK |BPF_F_USER_BUILD_ID )))@@ -467,18 +500,26 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, trace = trace_in; trace->nr =min_t (u32, trace->nr, max_depth); }else if (kernel && task) { - trace =get_callchain_entry_for_task (task, max_depth); + trace =get_callchain_entry_for_task (&rctx, task, max_depth); }else { - trace =get_perf_callchain (regs, kernel, user, max_depth, - crosstask, false,0 ); + trace =bpf_get_perf_callchain (&rctx, regs, kernel, user, max_depth, + crosstask); } -if (unlikely (!trace) || trace->nr < skip) { +if (unlikely (!trace)) {if (may_fault)rcu_read_unlock (); goto err_fault; } +if (trace->nr < skip) { +if (may_fault) +rcu_read_unlock (); +if (!trace_in) +bpf_put_perf_callchain (rctx); + goto err_fault; + } + trace_nr = trace->nr - skip; copy_len = trace_nr * elem_size;
kernel/bpf/stackmap.c static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, · line 546
Quoted 9 lines
kernel/bpf/stackmap.c +3
-0@@ -497,6 +538,9 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,if (may_fault)rcu_read_unlock (); +if (!trace_in) +bpf_put_perf_callchain (rctx); +if (user_build_id)stack_map_get_build_id_offset (buf, trace_nr, user, may_fault);
Hi Peter,
As Alexei said, the patch needs your ack, please review again, thanks.
Signature · 2 lines
Best Regards Tao Chen
Re: [PATCH bpf-next v7 2/2] bpf: Hold the perf callchain entry until used completely
Quoted 159 lines · Tao Chen
Quoted 147 lines · Tao Chen
As Alexei noted,
get_perf_callchain()return values may be reused if a task is preempted after the BPF program enters migrate disable mode. Theperf_callchain_entreshas a small stack of entries, and we can reuse it as follows:1. get the perf callchain entry
2. BPF use...
3. put the perf callchain entryAnd Peter suggested that
get_recursion_contextused with preemption disabled, so we should disable preemption at BPF side.Acked-by Yonghong Song <[email protected]>Signed-off-by Tao Chen <[email protected]>---
kernel/bpf/stackmap.c| 68 +++++++++++++++++++++++++++++++++++--------1 file changed, 56 insertions(+), 12 deletions(-)kernel/bpf/stackmap.c +0
-0@@ -210,13 +210,14 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, }static struct
perf_callchain_entry*kernel/bpf/stackmap.c +1
-1-get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) +get_callchain_entry_for_task(int *rctx, struct task_struct *task, u32
max_depth)
{
#ifdefCONFIG_STACKTRACE
structperf_callchain_entry*entry;
- int rctx;
- entry =get_callchain_entry(&rctx);
+preempt_disable();
+ entry =get_callchain_entry(rctx);
+preempt_enable();
if (!entry)
return NULL;kernel/bpf/stackmap.c +29
-2@@ -238,8 +239,6 @@ get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) to[i] = (u64)(from[i]); } - put_callchain_entry(rctx); - return entry; #else /* CONFIG_STACKTRACE */ return NULL;@@ -320,6 +319,34 @@ static long __bpf_get_stackid(struct bpf_map *map, return id; } +static struct perf_callchain_entry * +bpf_get_perf_callchain (int *rctx,struct pt_regs *regs,bool kernel,bool user, + int max_stack, bool crosstask) +{ + struct perf_callchain_entry_ctx ctx; + struct perf_callchain_entry *entry; + + preempt_disable(); + entry = get_callchain_entry(rctx); + preempt_enable(); + + if (unlikely(!entry)) + return NULL; + + __init_perf_callchain_ctx(&ctx, entry, max_stack, false); + if (kernel) + __get_perf_callchain_kernel(&ctx, regs); + if (user && !crosstask) + __get_perf_callchain_user(&ctx, regs, 0); + + return entry; +} + +static void bpf_put_perf_callchain (int rctx) +{ + put_callchain_entry(rctx); +} + BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, u64, flags) {@@ -328,20 +355,25 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, struct perf_callchain_entry *trace; bool kernel = !user; u32 max_depth; + int rctx, ret;if (
unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK | BPF_F_FAST_STACK_CMP | BPF_F_REUSE_STACKID))) return-EINVAL;
max_depth=stack_map_calculate_max_depth(map->value_size, elem_size, flags); - trace = get_perf_callchain(regs, kernel, user,max_depth,
- false, false, 0);
+
+ trace =bpf_get_perf_callchain(&rctx, regs, kernel, user, max_depth, + false);
if (unlikely(!trace))
/* couldn't fetch the stack trace */
return-EFAULT;
- return__bpf_get_stackid(map, trace, flags);
+ ret =__bpf_get_stackid(map, trace, flags);
+bpf_put_perf_callchain(rctx);
+
+ return ret;
}
const structbpf_func_protobpf_get_stackid_proto= {kernel/bpf/stackmap.c +1
-0@@ -435,6 +467,7 @@ static long __bpf_get_stack(struct pt_regs *regs,struct task_struct *task, bool kernel = !user; int err = -EINVAL; u64 *ips; + int rctx;if (
unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK | BPF_F_USER_BUILD_ID)))kernel/bpf/stackmap.c +12
-4@@ -467,18 +500,26 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, trace = trace_in; trace->nr = min_t(u32, trace->nr, max_depth); } else if (kernel && task) { - trace = get_callchain_entry_for_task(task, max_depth); + trace = get_callchain_entry_for_task(&rctx, task, max_depth); } else { - trace = get_perf_callchain(regs, kernel, user, max_depth, - crosstask, false, 0); + trace = bpf_get_perf_callchain(&rctx, regs, kernel, user, max_depth, + crosstask); } - if (unlikely(!trace) || trace->nr < skip) { + if (unlikely(!trace)) { if (may_fault) rcu_read_unlock(); goto err_fault; } + if (trace->nr < skip) { + if (may_fault) + rcu_read_unlock(); + if (!trace_in) + bpf_put_perf_callchain(rctx); + goto err_fault; + } + trace_nr = trace->nr - skip; copy_len = trace_nr * elem_size;kernel/bpf/stackmap.c static long __bpf_get_stack(struct pt_regs *regs, · line 545
Quoted 8 lines
kernel/bpf/stackmap.c +3
-0@@ -497,6 +538,9 @@ static long __bpf_get_stack(struct pt_regs *regs,struct task_struct *task, if (may_fault) rcu_read_unlock(); + if (!trace_in) + bpf_put_perf_callchain(rctx); + if (user_build_id) stack_map_get_build_id_offset(buf, trace_nr, user, may_fault);Hi Peter,
Quoted 2 lines
As Alexei said, the patch needs your ack, please review again, thanks.
ping...
Signature · 2 lines
Best Regards Tao Chen
Re: [PATCH bpf-next v7 2/2] bpf: Hold the perf callchain entry until used completely
Quoted 163 lines · Tao Chen
Quoted 161 lines · Tao Chen
Quoted 149 lines · Tao Chen
As Alexei noted,
get_perf_callchain()return values may be reused if a task is preempted after the BPF program enters migrate disable mode. Theperf_callchain_entreshas a small stack of entries, and we can reuse it as follows:1. get the perf callchain entry
2. BPF use...
3. put the perf callchain entryAnd Peter suggested that
get_recursion_contextused with preemption disabled, so we should disable preemption at BPF side.Acked-by Yonghong Song <[email protected]>Signed-off-by Tao Chen <[email protected]>--- kernel/bpf/stackmap.c | 68 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 56 insertions(+), 12 deletions(-)kernel/bpf/stackmap.c +32
-5@@ -210,13 +210,14 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, }static struct perf_callchain_entry * -get_callchain_entry_for_task (struct task_struct *task,u32 max_depth) +get_callchain_entry_for_task (int *rctx,struct task_struct *task,u32 max_depth) {#ifdef CONFIG_STACKTRACE struct perf_callchain_entry *entry; -int rctx; - entry =get_callchain_entry (&rctx); +preempt_disable (); + entry =get_callchain_entry (rctx); +preempt_enable ();if (!entry)return NULL ;@@ -238,8 +239,6 @@ get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) to[i] = (u64)(from[i]); } -put_callchain_entry (rctx); -return entry;#else /* CONFIG_STACKTRACE */ return NULL ;@@ -320,6 +319,34 @@ static long __bpf_get_stackid(struct bpf_map *map,return id; } +static struct perf_callchain_entry * +bpf_get_perf_callchain (int *rctx,struct pt_regs *regs,bool kernel,bool user, +int max_stack,bool crosstask) +{ +struct perf_callchain_entry_ctx ctx; +struct perf_callchain_entry *entry; + +preempt_disable (); + entry =get_callchain_entry (rctx); +preempt_enable (); + +if (unlikely (!entry)) +return NULL ; + +__init_perf_callchain_ctx (&ctx, entry, max_stack, false); +if (kernel) +__get_perf_callchain_kernel (&ctx, regs); +if (user && !crosstask) +__get_perf_callchain_user (&ctx, regs,0 ); + +return entry; +} + +static void bpf_put_perf_callchain (int rctx) +{ +put_callchain_entry (rctx); +} + BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, u64, flags){kernel/bpf/stackmap.c +21
-7@@ -328,20 +355,25 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, struct perf_callchain_entry *trace;bool kernel = !user;u32 max_depth; +int rctx, ret;if (unlikely (flags & ~(BPF_F_SKIP_FIELD_MASK |BPF_F_USER_STACK |BPF_F_FAST_STACK_CMP |BPF_F_REUSE_STACKID )))return -EINVAL ; max_depth =stack_map_calculate_max_depth (map->value_size, elem_size, flags); - trace =get_perf_callchain (regs, kernel, user, max_depth, - false, false,0 ); + + trace =bpf_get_perf_callchain (&rctx, regs, kernel, user, max_depth, + false);if (unlikely (!trace))/* couldn't fetch the stack trace */ return -EFAULT ; -return __bpf_get_stackid (map, trace, flags); + ret =__bpf_get_stackid (map, trace, flags); +bpf_put_perf_callchain (rctx); + +return ret; }const struct bpf_func_proto bpf_get_stackid_proto = {@@ -435,6 +467,7 @@ static long __bpf_get_stack(struct pt_regs *regs,struct task_struct *task, bool kernel = !user;int err = -EINVAL ;u64 *ips; +int rctx;if (unlikely (flags & ~(BPF_F_SKIP_FIELD_MASK |BPF_F_USER_STACK |BPF_F_USER_BUILD_ID )))@@ -467,18 +500,26 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, trace = trace_in; trace->nr =min_t (u32, trace->nr, max_depth); }else if (kernel && task) { - trace =get_callchain_entry_for_task (task, max_depth); + trace =get_callchain_entry_for_task (&rctx, task, max_depth); }else { - trace =get_perf_callchain (regs, kernel, user, max_depth, - crosstask, false,0 ); + trace =bpf_get_perf_callchain (&rctx, regs, kernel, user, max_depth, + crosstask); } -if (unlikely (!trace) || trace->nr < skip) { +if (unlikely (!trace)) {if (may_fault)rcu_read_unlock (); goto err_fault; } +if (trace->nr < skip) { +if (may_fault) +rcu_read_unlock (); +if (!trace_in) +bpf_put_perf_callchain (rctx); + goto err_fault; + } + trace_nr = trace->nr - skip; copy_len = trace_nr * elem_size;kernel/bpf/stackmap.c static long __bpf_get_stack(struct pt_regs *regs, · line 545
Quoted 8 lines
kernel/bpf/stackmap.c +3
-0@@ -497,6 +538,9 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, if (may_fault) rcu_read_unlock(); +if (!trace_in) +bpf_put_perf_callchain (rctx); +if (user_build_id)stack_map_get_build_id_offset (buf, trace_nr, user, may_fault);Hi Peter,
Quoted 4 lines
Quoted 2 lines
As Alexei said, the patch needs your ack, please review again, thanks.
ping...
Peter, if I understand correctly, this will go through bpf-next tree, but it would be great if you could take a look and confirm this overall is not broken. Thanks!
Quoted 4 lines
--
Best Regards
Tao Chen
Re: [PATCH bpf-next v7 2/2] bpf: Hold the perf callchain entry until used completely
Quoted 164 lines · Andrii Nakryiko
Quoted 163 lines · Tao Chen
Quoted 161 lines · Tao Chen
Quoted 149 lines · Tao Chen
As Alexei noted,
get_perf_callchain()return values may be reused if a task is preempted after the BPF program enters migrate disable mode. Theperf_callchain_entreshas a small stack of entries, and we can reuse it as follows:1. get the perf callchain entry
2. BPF use...
3. put the perf callchain entryAnd Peter suggested that
get_recursion_contextused with preemption disabled, so we should disable preemption at BPF side.Acked-by Yonghong Song <[email protected]>Signed-off-by Tao Chen <[email protected]>--- kernel/bpf/stackmap.c | 68 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 56 insertions(+), 12 deletions(-)kernel/bpf/stackmap.c +32
-5@@ -210,13 +210,14 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, }static struct perf_callchain_entry * -get_callchain_entry_for_task (struct task_struct *task,u32 max_depth) +get_callchain_entry_for_task (int *rctx,struct task_struct *task,u32 max_depth) {#ifdef CONFIG_STACKTRACE struct perf_callchain_entry *entry; -int rctx; - entry =get_callchain_entry (&rctx); +preempt_disable (); + entry =get_callchain_entry (rctx); +preempt_enable ();if (!entry)return NULL ;@@ -238,8 +239,6 @@ get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) to[i] = (u64)(from[i]); } -put_callchain_entry (rctx); -return entry;#else /* CONFIG_STACKTRACE */ return NULL ;@@ -320,6 +319,34 @@ static long __bpf_get_stackid(struct bpf_map *map,return id; } +static struct perf_callchain_entry * +bpf_get_perf_callchain (int *rctx,struct pt_regs *regs,bool kernel,bool user, +int max_stack,bool crosstask) +{ +struct perf_callchain_entry_ctx ctx; +struct perf_callchain_entry *entry; + +preempt_disable (); + entry =get_callchain_entry (rctx); +preempt_enable (); + +if (unlikely (!entry)) +return NULL ; + +__init_perf_callchain_ctx (&ctx, entry, max_stack, false); +if (kernel) +__get_perf_callchain_kernel (&ctx, regs); +if (user && !crosstask) +__get_perf_callchain_user (&ctx, regs,0 ); + +return entry; +} + +static void bpf_put_perf_callchain (int rctx) +{ +put_callchain_entry (rctx); +} + BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, u64, flags){kernel/bpf/stackmap.c +21
-7@@ -328,20 +355,25 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, struct perf_callchain_entry *trace;bool kernel = !user;u32 max_depth; +int rctx, ret;if (unlikely (flags & ~(BPF_F_SKIP_FIELD_MASK |BPF_F_USER_STACK |BPF_F_FAST_STACK_CMP |BPF_F_REUSE_STACKID )))return -EINVAL ; max_depth =stack_map_calculate_max_depth (map->value_size, elem_size, flags); - trace =get_perf_callchain (regs, kernel, user, max_depth, - false, false,0 ); + + trace =bpf_get_perf_callchain (&rctx, regs, kernel, user, max_depth, + false);if (unlikely (!trace))/* couldn't fetch the stack trace */ return -EFAULT ; -return __bpf_get_stackid (map, trace, flags); + ret =__bpf_get_stackid (map, trace, flags); +bpf_put_perf_callchain (rctx); + +return ret; }const struct bpf_func_proto bpf_get_stackid_proto = {@@ -435,6 +467,7 @@ static long __bpf_get_stack(struct pt_regs *regs,struct task_struct *task, bool kernel = !user;int err = -EINVAL ;u64 *ips; +int rctx;if (unlikely (flags & ~(BPF_F_SKIP_FIELD_MASK |BPF_F_USER_STACK |BPF_F_USER_BUILD_ID )))@@ -467,18 +500,26 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, trace = trace_in; trace->nr =min_t (u32, trace->nr, max_depth); }else if (kernel && task) { - trace =get_callchain_entry_for_task (task, max_depth); + trace =get_callchain_entry_for_task (&rctx, task, max_depth); }else { - trace =get_perf_callchain (regs, kernel, user, max_depth, - crosstask, false,0 ); + trace =bpf_get_perf_callchain (&rctx, regs, kernel, user, max_depth, + crosstask); } -if (unlikely (!trace) || trace->nr < skip) { +if (unlikely (!trace)) {if (may_fault)rcu_read_unlock (); goto err_fault; } +if (trace->nr < skip) { +if (may_fault) +rcu_read_unlock (); +if (!trace_in) +bpf_put_perf_callchain (rctx); + goto err_fault; + } + trace_nr = trace->nr - skip; copy_len = trace_nr * elem_size;kernel/bpf/stackmap.c static long __bpf_get_stack(struct pt_regs *regs, · line 545
Quoted 8 lines
kernel/bpf/stackmap.c +3
-0@@ -497,6 +538,9 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, if (may_fault) rcu_read_unlock(); +if (!trace_in) +bpf_put_perf_callchain (rctx); +if (user_build_id)stack_map_get_build_id_offset (buf, trace_nr, user, may_fault);Hi Peter,
Quoted 13 lines
Quoted 4 lines
Quoted 2 lines
As Alexei said, the patch needs your ack, please review again, thanks.
ping...
Peter, if I understand correctly, this will go through bpf-next tree, but it would be great if you could take a look and confirm this overall is not broken. Thanks!
Quoted 4 lines
--
Best Regards
Tao Chen
Hi Andrii, Peter
It appears that the code does not require a rebase, and the latest CI build is valid. Looking forward to your response. Thanks.
CI has tested the following submission:
Status: SUCCESS Name: [RESEND,bpf-next,v7,0/2] Pass external callchain entry to get_perf_callchain Patchwork: https://patchwork.kernel.org/project/netdevbpf/list/?series=1034091&state=* Matrix: https://github.com/kernel-patches/bpf/actions/runs/21051611369
Signature · 2 lines
Best Regards Tao Chen
Re: [PATCH bpf-next v7 2/2] bpf: Hold the perf callchain entry until used completely
In reply to Tao ChenQuoted 10 lines · Tao Chen
As Alexei noted,
get_perf_callchain()return values may be reused if a task is preempted after the BPF program enters migrate disable mode. Theperf_callchain_entreshas a small stack of entries, and we can reuse it as follows:1. get the perf callchain entry
2. BPF use...
3. put the perf callchain entry
Quoted 9 lines
And Peter suggested that
get_recursion_contextused with preemption disabled, so we should disable preemption at BPF side.Acked-by Yonghong Song <[email protected]>Signed-off-by Tao Chen <[email protected]>--- kernel/bpf/stackmap.c | 68 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 56 insertions(+), 12 deletions(-)
I took a bit closer look at these changes and I'm a fan of the particular implementation, tbh. It's a bit of a maze how all these different call chain cases are handled, so I might be missing something, but I'd address this a bit differently.
First, instead of manipulating this obscure rctx as part of interface, I'd record rctx inside the perf_callchain_entry itself, and make sure that get_callchain_entry does have any output arguments. put_callchain_entry() would then accept perf_callchain_entry reference and just fetch rctx from inside it.
Then instead of open-coding get_perf_callchain by exposing __init_perf_callchain_ctx, __get_perf_callchain_kernel, and __get_perf_callchain_user, can't we have __get_perf_callchain() which will accept perf_callchain_entry as an input and won't do get/put internally. And then existing get_perf_callchain() will just do get + __get_perf_callchain + put, while BPF-side code will do it's own get (with preemption temporarily disabled), will fetch callstack in one of a few possible ways, and then put it (unless callchain_entry is coming from outside, that trace_in thing).
It's close to what you are doing, but I don't think anyone likes those exposed __init_perf_callchain_ctx + __get_perf_callchain_kernel + __get_perf_callchain_user. Can't we avoid that? (and also not sure we need add_mark inside the ctx itself, do we?)
pw-bot: cr
Quoted 147 lines
kernel/bpf/stackmap.c +56
-13@@ -210,13 +210,14 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, }static struct perf_callchain_entry * -get_callchain_entry_for_task (struct task_struct *task,u32 max_depth) +get_callchain_entry_for_task (int *rctx,struct task_struct *task,u32 max_depth) {#ifdef CONFIG_STACKTRACE struct perf_callchain_entry *entry; -int rctx; - entry =get_callchain_entry (&rctx); +preempt_disable (); + entry =get_callchain_entry (rctx); +preempt_enable ();if (!entry)return NULL ;@@ -238,8 +239,6 @@ get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) to[i] = (u64)(from[i]); } -put_callchain_entry (rctx); -return entry;#else /* CONFIG_STACKTRACE */ return NULL ;@@ -320,6 +319,34 @@ static long __bpf_get_stackid(struct bpf_map *map,return id; } +static struct perf_callchain_entry * +bpf_get_perf_callchain (int *rctx,struct pt_regs *regs,bool kernel,bool user, +int max_stack,bool crosstask) +{ +struct perf_callchain_entry_ctx ctx; +struct perf_callchain_entry *entry; + +preempt_disable (); + entry =get_callchain_entry (rctx); +preempt_enable (); + +if (unlikely (!entry)) +return NULL ; + +__init_perf_callchain_ctx (&ctx, entry, max_stack, false); +if (kernel) +__get_perf_callchain_kernel (&ctx, regs); +if (user && !crosstask) +__get_perf_callchain_user (&ctx, regs,0 ); + +return entry; +} + +static void bpf_put_perf_callchain (int rctx) +{ +put_callchain_entry (rctx); +} + BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, u64, flags) {@@ -328,20 +355,25 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map,struct perf_callchain_entry *trace;bool kernel = !user;u32 max_depth; +int rctx, ret;if (unlikely (flags & ~(BPF_F_SKIP_FIELD_MASK |BPF_F_USER_STACK |BPF_F_FAST_STACK_CMP |BPF_F_REUSE_STACKID )))return -EINVAL ; max_depth =stack_map_calculate_max_depth (map->value_size, elem_size, flags); - trace =get_perf_callchain (regs, kernel, user, max_depth, - false, false,0 ); + + trace =bpf_get_perf_callchain (&rctx, regs, kernel, user, max_depth, + false);if (unlikely (!trace))/* couldn't fetch the stack trace */ return -EFAULT ; -return __bpf_get_stackid (map, trace, flags); + ret =__bpf_get_stackid (map, trace, flags); +bpf_put_perf_callchain (rctx); + +return ret; }const struct bpf_func_proto bpf_get_stackid_proto = {@@ -435,6 +467,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,bool kernel = !user;int err = -EINVAL ;u64 *ips; +int rctx;if (unlikely (flags & ~(BPF_F_SKIP_FIELD_MASK |BPF_F_USER_STACK |BPF_F_USER_BUILD_ID )))@@ -467,18 +500,26 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, trace = trace_in; trace->nr =min_t (u32, trace->nr, max_depth); }else if (kernel && task) { - trace =get_callchain_entry_for_task (task, max_depth); + trace =get_callchain_entry_for_task (&rctx, task, max_depth); }else { - trace =get_perf_callchain (regs, kernel, user, max_depth, - crosstask, false,0 ); + trace =bpf_get_perf_callchain (&rctx, regs, kernel, user, max_depth, + crosstask); } -if (unlikely (!trace) || trace->nr < skip) { +if (unlikely (!trace)) {if (may_fault)rcu_read_unlock (); goto err_fault; } +if (trace->nr < skip) { +if (may_fault) +rcu_read_unlock (); +if (!trace_in) +bpf_put_perf_callchain (rctx); + goto err_fault; + } + trace_nr = trace->nr - skip; copy_len = trace_nr * elem_size;@@ -497,6 +538,9 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,if (may_fault)rcu_read_unlock (); +if (!trace_in) +bpf_put_perf_callchain (rctx); +if (user_build_id)stack_map_get_build_id_offset (buf, trace_nr, user, may_fault); --2.48.1
Re: [PATCH bpf-next v7 2/2] bpf: Hold the perf callchain entry until used completely
Quoted 26 lines · Andrii Nakryiko
Quoted 10 lines · Tao Chen
As Alexei noted,
get_perf_callchain()return values may be reused if a task is preempted after the BPF program enters migrate disable mode. Theperf_callchain_entreshas a small stack of entries, and we can reuse it as follows:1. get the perf callchain entry
2. BPF use...
3. put the perf callchain entryQuoted 9 lines
And Peter suggested that
get_recursion_contextused with preemption disabled, so we should disable preemption at BPF side.Acked-by Yonghong Song <[email protected]>Signed-off-by Tao Chen <[email protected]>--- kernel/bpf/stackmap.c | 68 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 56 insertions(+), 12 deletions(-)I took a bit closer look at these changes and I'm a fan of the particular implementation, tbh. It's a bit of a maze how all these different call chain cases are handled, so I might be missing something, but I'd address this a bit differently.
Quoted 6 lines
First, instead of manipulating this obscure rctx as part of interface, I'd record rctx inside the
perf_callchain_entryitself, and make sure thatget_callchain_entrydoes have any output arguments.put_callchain_entry()would then acceptperf_callchain_entryreference and just fetch rctx from inside it.
Hi Andrri,
Try to implement this briefly with code, is my understanding correct?
struct perf_callchain_entry *get_callchain_entry(void)
{
int cpu;
int rctx;
struct perf_callchain_entry *entry;
struct callchain_cpus_entries *entries;
rctx = get_recursion_context(this_cpu_ptr(callchain_recursion));
if (rctx == -1)
return NULL;
entries = rcu_dereference(callchain_cpus_entries);
if (!entries) {put_recursion_context(this_cpu_ptr(callchain_recursion), *rctx);
return NULL;
}
cpu = smp_processor_id();
entry = ((void *)entries->cpu_entries[cpu]) +
(rctx * perf_callchain_entry__sizeof());
entry->rctx = rctx;
return entry;
}voidput_callchain_entry(struct perf_callchain_entry *entry) {
put_recursion_context(this_cpu_ptr(callchain_recursion),entry->rctx);
}
And then no need rtcx in bpf_get_perf_callchain.
bpf_get_perf_callchain(struct pt_regs *regs, bool kernel, bool user,
int max_stack, bool crosstask)
Functionally, this seems fine. The only concern is whether the perf maintainer will approve it after all, this change involves modifying their core interfaces.
Peter, Yonghong, can we do this?
Quoted 10 lines
Then instead of open-coding
get_perf_callchainby exposing__init_perf_callchain_ctx,__get_perf_callchain_kernel, and__get_perf_callchain_user, can't we have__get_perf_callchain()which will acceptperf_callchain_entryas an input and won't do get/put internally. And then existingget_perf_callchain()will just do get +__get_perf_callchain+ put, while BPF-side code will do it's own get (with preemption temporarily disabled), will fetch callstack in one of a few possible ways, and then put it (unlesscallchain_entryis coming from outside, thattrace_inthing).
Exposing only __get_perf_callchain will make it much easier for BPF callers to understand and use. And following Peter's suggestion, we can mark _init_perf_callchain_ctx, __get_perf_callchain_kernel, and __get_perf_callchain_user as static, then have __get_perf_callchain encapsulate the logic of _init_perf_callchain_ctx, __get_perf_callchain_kernel, and __get_perf_callchain_user. What do you think?
Quoted 156 lines
It's close to what you are doing, but I don't think anyone likes those exposed
__init_perf_callchain_ctx+__get_perf_callchain_kernel+__get_perf_callchain_user. Can't we avoid that? (and also not sure we needadd_markinside the ctx itself, do we?)pw-bot: cr
Quoted 147 lines
kernel/bpf/stackmap.c +56
-13@@ -210,13 +210,14 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, }static struct perf_callchain_entry * -get_callchain_entry_for_task (struct task_struct *task,u32 max_depth) +get_callchain_entry_for_task (int *rctx,struct task_struct *task,u32 max_depth) {#ifdef CONFIG_STACKTRACE struct perf_callchain_entry *entry; -int rctx; - entry =get_callchain_entry (&rctx); +preempt_disable (); + entry =get_callchain_entry (rctx); +preempt_enable ();if (!entry)return NULL ;@@ -238,8 +239,6 @@ get_callchain_entry_for_task(struct task_struct *task, u32 max_depth) to[i] = (u64)(from[i]); } -put_callchain_entry (rctx); -return entry;#else /* CONFIG_STACKTRACE */ return NULL ;@@ -320,6 +319,34 @@ static long __bpf_get_stackid(struct bpf_map *map,return id; } +static struct perf_callchain_entry * +bpf_get_perf_callchain (int *rctx,struct pt_regs *regs,bool kernel,bool user, +int max_stack,bool crosstask) +{ +struct perf_callchain_entry_ctx ctx; +struct perf_callchain_entry *entry; + +preempt_disable (); + entry =get_callchain_entry (rctx); +preempt_enable (); + +if (unlikely (!entry)) +return NULL ; + +__init_perf_callchain_ctx (&ctx, entry, max_stack, false); +if (kernel) +__get_perf_callchain_kernel (&ctx, regs); +if (user && !crosstask) +__get_perf_callchain_user (&ctx, regs,0 ); + +return entry; +} + +static void bpf_put_perf_callchain (int rctx) +{ +put_callchain_entry (rctx); +} + BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, u64, flags) {@@ -328,20 +355,25 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map,struct perf_callchain_entry *trace;bool kernel = !user;u32 max_depth; +int rctx, ret;if (unlikely (flags & ~(BPF_F_SKIP_FIELD_MASK |BPF_F_USER_STACK |BPF_F_FAST_STACK_CMP |BPF_F_REUSE_STACKID )))return -EINVAL ; max_depth =stack_map_calculate_max_depth (map->value_size, elem_size, flags); - trace =get_perf_callchain (regs, kernel, user, max_depth, - false, false,0 ); + + trace =bpf_get_perf_callchain (&rctx, regs, kernel, user, max_depth, + false);if (unlikely (!trace))/* couldn't fetch the stack trace */ return -EFAULT ; -return __bpf_get_stackid (map, trace, flags); + ret =__bpf_get_stackid (map, trace, flags); +bpf_put_perf_callchain (rctx); + +return ret; }const struct bpf_func_proto bpf_get_stackid_proto = {@@ -435,6 +467,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,bool kernel = !user;int err = -EINVAL ;u64 *ips; +int rctx;if (unlikely (flags & ~(BPF_F_SKIP_FIELD_MASK |BPF_F_USER_STACK |BPF_F_USER_BUILD_ID )))@@ -467,18 +500,26 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, trace = trace_in; trace->nr =min_t (u32, trace->nr, max_depth); }else if (kernel && task) { - trace =get_callchain_entry_for_task (task, max_depth); + trace =get_callchain_entry_for_task (&rctx, task, max_depth); }else { - trace =get_perf_callchain (regs, kernel, user, max_depth, - crosstask, false,0 ); + trace =bpf_get_perf_callchain (&rctx, regs, kernel, user, max_depth, + crosstask); } -if (unlikely (!trace) || trace->nr < skip) { +if (unlikely (!trace)) {if (may_fault)rcu_read_unlock (); goto err_fault; } +if (trace->nr < skip) { +if (may_fault) +rcu_read_unlock (); +if (!trace_in) +bpf_put_perf_callchain (rctx); + goto err_fault; + } + trace_nr = trace->nr - skip; copy_len = trace_nr * elem_size;@@ -497,6 +538,9 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,if (may_fault)rcu_read_unlock (); +if (!trace_in) +bpf_put_perf_callchain (rctx); +if (user_build_id)stack_map_get_build_id_offset (buf, trace_nr, user, may_fault); --2.48.1
Signature · 2 lines
Best Regards Tao Chen
Re: [PATCH bpf-next v7 2/2] bpf: Hold the perf callchain entry until used completely
Quoted 66 lines · Tao Chen
Quoted 26 lines · Andrii Nakryiko
Quoted 10 lines · Tao Chen
As Alexei noted,
get_perf_callchain()return values may be reused if a task is preempted after the BPF program enters migrate disable mode. Theperf_callchain_entreshas a small stack of entries, and we can reuse it as follows:1. get the perf callchain entry
2. BPF use...
3. put the perf callchain entryQuoted 9 lines
And Peter suggested that
get_recursion_contextused with preemption disabled, so we should disable preemption at BPF side.Acked-by Yonghong Song <[email protected]>Signed-off-by Tao Chen <[email protected]>--- kernel/bpf/stackmap.c | 68 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 56 insertions(+), 12 deletions(-)I took a bit closer look at these changes and I'm a fan of the particular implementation, tbh. It's a bit of a maze how all these different call chain cases are handled, so I might be missing something, but I'd address this a bit differently.
Quoted 6 lines
First, instead of manipulating this obscure rctx as part of interface, I'd record rctx inside the
perf_callchain_entryitself, and make sure thatget_callchain_entrydoes have any output arguments.put_callchain_entry()would then acceptperf_callchain_entryreference and just fetch rctx from inside it.Hi Andrri,
Try to implement this briefly with code, is my understanding correct?
struct
perf_callchain_entry*get_callchain_entry(void){ int cpu; int rctx; struct perf_callchain_entry *entry; struct callchain_cpus_entries *entries; rctx = get_recursion_context(this_cpu_ptr(callchain_recursion)); if (rctx == -1) return NULL; entries = rcu_dereference(callchain_cpus_entries); if (!entries) {put_recursion_context(this_cpu_ptr(callchain_recursion), *rctx); return NULL; } cpu = smp_processor_id(); entry = ((void *)entries->cpu_entries[cpu]) + (rctx * perf_callchain_entry__sizeof()); entry->rctx = rctx; return entry; }
Quoted 8 lines
void
put_callchain_entry(struct perf_callchain_entry *entry){
put_recursion_context(this_cpu_ptr(callchain_recursion),entry->rctx);
}And then no need rtcx in
bpf_get_perf_callchain.
yes, exactly
Quoted 21 lines
bpf_get_perf_callchain(struct
pt_regs*regs, bool kernel, bool user,
intmax_stack, bool crosstask)Functionally, this seems fine. The only concern is whether the perf maintainer will approve it after all, this change involves modifying their core interfaces.
Peter, Yonghong, can we do this?
Quoted 10 lines
Then instead of open-coding
get_perf_callchainby exposing__init_perf_callchain_ctx,__get_perf_callchain_kernel, and__get_perf_callchain_user, can't we have__get_perf_callchain()which will acceptperf_callchain_entryas an input and won't do get/put internally. And then existingget_perf_callchain()will just do get +__get_perf_callchain+ put, while BPF-side code will do it's own get (with preemption temporarily disabled), will fetch callstack in one of a few possible ways, and then put it (unlesscallchain_entryis coming from outside, thattrace_inthing).
Quoted 7 lines
Exposing only
__get_perf_callchainwill make it much easier for BPF callers to understand and use. And following Peter's suggestion, we can mark_init_perf_callchain_ctx,__get_perf_callchain_kernel, and__get_perf_callchain_useras static, then have__get_perf_callchainencapsulate the logic of_init_perf_callchain_ctx,__get_perf_callchain_kernel, and__get_perf_callchain_user. What do you think?
That's exactly what I proposed. Except I don't think we even need to have __get_perf_callchain_user and __get_perf_callchain_kernel, keep __get_perf_callchain almost identical to current get_perf_callchain(), except for getting/putting callchain entry
Quoted 10 lines
Quoted 10 lines
It's close to what you are doing, but I don't think anyone likes those exposed
__init_perf_callchain_ctx+__get_perf_callchain_kernel+__get_perf_callchain_user. Can't we avoid that? (and also not sure we needadd_markinside the ctx itself, do we?)pw-bot: cr
[please trim irrelevant parts]