bpf: Remove RCU lock for perf callchain buffer
Subsystems (2)
PATCH 1/1 bpf: Remove RCU lock for perf callchain buffer
As Andrii suggested, BPF can guarantee perf callchain buffer won't be released during use, for bpf_get_stack_id, BPF stack map will keep them alive by delaying put_callchain_buffer() until freeing time or for bpf_get_stack/bpf_get_task_stack, BPF program itself will hold these buffers alive again, until freeing time which is delayed until after RCU Tasks Trace + RCU grace period.
--- kernel/bpf/stackmap.c | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-)
kernel/bpf/stackmap.c +1 -11
@@ -460,9 +460,6 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, max_depth =stack_map_calculate_max_depth (size, elem_size, flags); -if (may_fault) -rcu_read_lock ();/* need RCU for perf's callchain below */ -if (trace_in) { trace = trace_in; trace->nr =min_t (u32, trace->nr, max_depth);
@@ -473,11 +470,8 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, crosstask, false, 0); } -if (unlikely (!trace) || trace->nr < skip) { -if (may_fault) -rcu_read_unlock (); +if (unlikely (!trace) || trace->nr < skip) goto err_fault; - } trace_nr = trace->nr - skip; copy_len = trace_nr * elem_size;
@@ -493,10 +487,6 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,memcpy (buf, ips, copy_len); } -/* trace/ips should not be dereferenced after this point */ -if (may_fault) -rcu_read_unlock (); -if (user_build_id)stack_map_get_build_id_offset (buf, trace_nr, user, may_fault);
Signature · 1 line
2.48.1
Quoted 41 lines · Tao Chen
As Andrii suggested, BPF can guarantee perf callchain buffer won't be released during use, for
bpf_get_stack_id, BPF stack map will keep them alive by delayingput_callchain_buffer()until freeing time or for bpf_get_stack/bpf_get_task_stack, BPF program itself will hold these buffers alive again, until freeing time which is delayed until after RCU Tasks Trace + RCU grace period.Suggested-by Andrii Nakryiko <[email protected]>Signed-off-by Tao Chen <[email protected]>--- kernel/bpf/stackmap.c | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-)kernel/bpf/stackmap.c +1
-7@@ -460,9 +460,6 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, max_depth =stack_map_calculate_max_depth (size, elem_size, flags); -if (may_fault) -rcu_read_lock ();/* need RCU for perf's callchain below */ -if (trace_in) { trace = trace_in; trace->nr =min_t (u32, trace->nr, max_depth);@@ -473,11 +470,8 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, crosstask, false, 0); } -if (unlikely (!trace) || trace->nr < skip) { -if (may_fault) -rcu_read_unlock (); +if (unlikely (!trace) || trace->nr < skip) goto err_fault; - } trace_nr = trace->nr - skip; copy_len = trace_nr * elem_size;
kernel/bpf/stackmap.c static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, · line 492
Quoted 10 lines
kernel/bpf/stackmap.c +0
-4@@ -493,10 +487,6 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,memcpy (buf, ips, copy_len); } -/* trace/ips should not be dereferenced after this point */ -if (may_fault) -rcu_read_unlock (); -if (user_build_id)stack_map_get_build_id_offset (buf, trace_nr, user, may_fault);
Hi Andrii,
I send this patch as a separate submission, and I'm not sure when the patchset will be merged. After this merged, i will update v9. Is it ok?
The previous patchset:
https://lore.kernel.org/bpf/CAEf4BzbubXxLHsO_2aVBG_oYE1Sc9iafLyvcX-PEjcQ7EXWCqA@mail.gmail.com/
And i also saw Qing reported a syzbot issue recently.
https://lore.kernel.org/bpf/[email protected]/
Signature · 2 lines
Best Regards Tao Chen
Quoted 56 lines · Tao Chen
Quoted 41 lines · Tao Chen
As Andrii suggested, BPF can guarantee perf callchain buffer won't be released during use, for
bpf_get_stack_id, BPF stack map will keep them alive by delayingput_callchain_buffer()until freeing time or for bpf_get_stack/bpf_get_task_stack, BPF program itself will hold these buffers alive again, until freeing time which is delayed until after RCU Tasks Trace + RCU grace period.Suggested-by Andrii Nakryiko <[email protected]>Signed-off-by Tao Chen <[email protected]>--- kernel/bpf/stackmap.c | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-)kernel/bpf/stackmap.c +1
-7@@ -460,9 +460,6 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, max_depth =stack_map_calculate_max_depth (size, elem_size, flags); -if (may_fault) -rcu_read_lock ();/* need RCU for perf's callchain below */ -if (trace_in) { trace = trace_in; trace->nr =min_t (u32, trace->nr, max_depth);@@ -473,11 +470,8 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, crosstask, false, 0); } -if (unlikely (!trace) || trace->nr < skip) { -if (may_fault) -rcu_read_unlock (); +if (unlikely (!trace) || trace->nr < skip) goto err_fault; - } trace_nr = trace->nr - skip; copy_len = trace_nr * elem_size;kernel/bpf/stackmap.c static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, · line 492
Quoted 10 lines
kernel/bpf/stackmap.c +0
-4@@ -493,10 +487,6 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,memcpy (buf, ips, copy_len); } -/* trace/ips should not be dereferenced after this point */ -if (may_fault) -rcu_read_unlock (); -if (user_build_id)stack_map_get_build_id_offset (buf, trace_nr, user, may_fault);Hi Andrii,
Quoted 2 lines
I send this patch as a separate submission, and I'm not sure when the patchset will be merged. After this merged, i will update v9. Is it ok?
See my reply to your other patch set. We should probably just replace RCU manipulations with preempt_disable/enable and be done with all of this. (and RCU Tasks Trace is implying RCU right now and that is going to be guaranteed moving forward, we already make this assumption in recent bpf_timer patch set).
Quoted 10 lines
The previous patchset:
https://lore.kernel.org/bpf/CAEf4BzbubXxLHsO_2aVBG_oYE1Sc9iafLyvcX-PEjcQ7EXWCqA@mail.gmail.com/And i also saw Qing reported a syzbot issue recently.
https://lore.kernel.org/bpf/[email protected]/--
Best Regards
Tao Chen