bpf: Add trampoline helpers
Subsystems (9)
Jiri Olsa
reviewerv3 cover bpf: Add trampoline helpers
hi,
adding helpers for trampolines and 2 other fixes to have kernel support for loading trampoline programs in bcc/bpftrace.
Original rfc post [1].
Speedup output of perf bench while running klockstat.py on kprobes vs trampolines:
Without:
$ perf bench sched messaging -l 50000
...
Total time: 18.571 [sec]
With current kprobe tracing:
$ perf bench sched messaging -l 50000
...
Total time: 183.395 [sec]
With kfunc tracing:
$ perf bench sched messaging -l 50000
...
Total time: 39.773 [sec]v3 changes:
- added ack from John Fastabend for patch 1
- move out is_bpf_image_address from is_bpf_text_address call [David]
v2 changes:
- make the unwind work for dispatcher as well
- added test for allowed trampolines count
- used raw tp pt_regs nest-arrays for trampoline helpers
thanks,
jirka
[1] https://lore.kernel.org/netdev/[email protected]/
---
Jiri Olsa (6):
bpf: Allow ctx access for pointers to scalar
bpf: Add bpf_perf_event_output_kfunc
bpf: Add bpf_get_stackid_kfunc
bpf: Add bpf_get_stack_kfunc
bpf: Allow to resolve bpf trampoline and dispatcher in unwind selftest/bpf: Add test for allowed trampolines count
include/linux/bpf.h | 12 +++++++++++- kernel/bpf/btf.c | 13 ++++++++++++- kernel/bpf/dispatcher.c | 4 ++-- kernel/bpf/trampoline.c | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------- kernel/extable.c | 7 +++++-- kernel/trace/bpf_trace.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/testing/selftests/bpf/prog_tests/trampoline_count.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/testing/selftests/bpf/progs/test_trampoline_count.c | 21 +++++++++++++++++++++ 8 files changed, 334 insertions(+), 13 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/trampoline_count.c create mode 100644 tools/testing/selftests/bpf/progs/test_trampoline_count.c
Jiri Olsa
reviewerPATCH v3 1/6 bpf: Allow ctx access for pointers to scalar
When accessing the context we allow access to arguments with scalar type and pointer to struct. But we omit pointer to scalar type, which is the case for many functions and same case as when accessing scalar.
Adding the check if the pointer is to scalar type and allow it.
--- kernel/bpf/btf.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
kernel/bpf/btf.c +12 -1
@@ -3668,7 +3668,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,const struct bpf_prog *prog, struct bpf_insn_access_aux *info) { - const struct btf_type *t = prog->aux->attach_func_proto; + const struct btf_type *tp, *t = prog->aux->attach_func_proto;struct bpf_prog *tgt_prog = prog->aux->linked_prog;struct btf *btf =bpf_prog_get_target_btf (prog);const char *tname = prog->aux->attach_func_name;
@@ -3730,6 +3730,17 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,*/ return true; + tp =btf_type_by_id (btf, t->type); +/* skip modifiers */ +while (btf_type_is_modifier (tp)) + tp =btf_type_by_id (btf, tp->type); + +if (btf_type_is_int (tp) ||btf_type_is_enum (tp)) +/* This is a pointer scalar. +* It is the same as scalar from the verifier safety pov. +*/ +return true; +/* this is a pointer to another type */ info->reg_type =PTR_TO_BTF_ID ;
Signature · 1 line
2.24.1
Re: [PATCH 1/6] bpf: Allow ctx access for pointers to scalar
Quoted 31 lines · Jiri Olsa
When accessing the context we allow access to arguments with scalar type and pointer to struct. But we omit pointer to scalar type, which is the case for many functions and same case as when accessing scalar.
Adding the check if the pointer is to scalar type and allow it.
Acked-by John Fastabend <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- kernel/bpf/btf.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)kernel/bpf/btf.c +1
-1@@ -3668,7 +3668,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,const struct bpf_prog *prog, struct bpf_insn_access_aux *info) { - const struct btf_type *t = prog->aux->attach_func_proto; + const struct btf_type *tp, *t = prog->aux->attach_func_proto;struct bpf_prog *tgt_prog = prog->aux->linked_prog;struct btf *btf =bpf_prog_get_target_btf (prog);const char *tname = prog->aux->attach_func_name;@@ -3730,6 +3730,17 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,*/ return true;
kernel/bpf/btf.c bool btf_ctx_access(int off, int size, enum bpf_access_type type, · line 3742
Quoted 10 lines
kernel/bpf/btf.c +10
-0+ tp =btf_type_by_id (btf, t->type); +/* skip modifiers */ +while (btf_type_is_modifier (tp)) + tp =btf_type_by_id (btf, tp->type); + +if (btf_type_is_int (tp) ||btf_type_is_enum (tp)) +/* This is a pointer scalar. +* It is the same as scalar from the verifier safety pov. +*/ +return true;
The reason I didn't do it earlier is I was thinking to represent it as PTR_TO_BTF_ID as well, so that corresponding u8..u64 access into this memory would still be possible.
I'm trying to analyze the situation that returning a scalar now and converting to PTR_TO_BTF_ID in the future will keep progs passing the verifier. Is it really the case?
Could you give a specific example that needs this support? It will help me understand this backward compatibility concern. What prog is doing with that 'u32 *' that is seen as scalar ? It cannot dereference it. Use it as what?
Re: [PATCH 1/6] bpf: Allow ctx access for pointers to scalar
Quoted 45 lines · Alexei Starovoitov
Quoted 31 lines · Jiri Olsa
When accessing the context we allow access to arguments with scalar type and pointer to struct. But we omit pointer to scalar type, which is the case for many functions and same case as when accessing scalar.
Adding the check if the pointer is to scalar type and allow it.
Acked-by John Fastabend <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- kernel/bpf/btf.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)kernel/bpf/btf.c +1
-1@@ -3668,7 +3668,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,const struct bpf_prog *prog, struct bpf_insn_access_aux *info) { - const struct btf_type *t = prog->aux->attach_func_proto; + const struct btf_type *tp, *t = prog->aux->attach_func_proto;struct bpf_prog *tgt_prog = prog->aux->linked_prog;struct btf *btf =bpf_prog_get_target_btf (prog);const char *tname = prog->aux->attach_func_name;@@ -3730,6 +3730,17 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,*/ return true;kernel/bpf/btf.c bool btf_ctx_access(int off, int size, enum bpf_access_type type, · line 3742
Quoted 10 lines
kernel/bpf/btf.c +10
-0+ tp =btf_type_by_id (btf, t->type); +/* skip modifiers */ +while (btf_type_is_modifier (tp)) + tp =btf_type_by_id (btf, tp->type); + +if (btf_type_is_int (tp) ||btf_type_is_enum (tp)) +/* This is a pointer scalar. +* It is the same as scalar from the verifier safety pov. +*/ +return true;The reason I didn't do it earlier is I was thinking to represent it as
PTR_TO_BTF_IDas well, so that corresponding u8..u64
Quoted 8 lines
access into this memory would still be possible.
I'm trying to analyze the situation that returning a scalar now and converting toPTR_TO_BTF_IDin the future will keep progs passing the verifier. Is it really the case?
Could you give a specific example that needs this support? It will help me understand this backward compatibility concern. What prog is doing with that 'u32 *' that is seen as scalar ? It cannot dereference it. Use it as what?
If this is from original bcc code, it will use bpf_probe_read for dereference. This is what I understand when I first reviewed this patch. But it will be good to get Jiri's confirmation.
Re: [PATCH 1/6] bpf: Allow ctx access for pointers to scalar
Quoted 57 lines · Yonghong Song
Quoted 45 lines · Alexei Starovoitov
Quoted 31 lines · Jiri Olsa
When accessing the context we allow access to arguments with scalar type and pointer to struct. But we omit pointer to scalar type, which is the case for many functions and same case as when accessing scalar.
Adding the check if the pointer is to scalar type and allow it.
Acked-by John Fastabend <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- kernel/bpf/btf.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)kernel/bpf/btf.c +1
-1@@ -3668,7 +3668,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,const struct bpf_prog *prog, struct bpf_insn_access_aux *info) { - const struct btf_type *t = prog->aux->attach_func_proto; + const struct btf_type *tp, *t = prog->aux->attach_func_proto;struct bpf_prog *tgt_prog = prog->aux->linked_prog;struct btf *btf =bpf_prog_get_target_btf (prog);const char *tname = prog->aux->attach_func_name;@@ -3730,6 +3730,17 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,*/ return true;kernel/bpf/btf.c bool btf_ctx_access(int off, int size, enum bpf_access_type type, · line 3742
Quoted 10 lines
kernel/bpf/btf.c +10
-0+ tp =btf_type_by_id (btf, t->type); +/* skip modifiers */ +while (btf_type_is_modifier (tp)) + tp =btf_type_by_id (btf, tp->type); + +if (btf_type_is_int (tp) ||btf_type_is_enum (tp)) +/* This is a pointer scalar. +* It is the same as scalar from the verifier safety pov. +*/ +return true;The reason I didn't do it earlier is I was thinking to represent it as
PTR_TO_BTF_IDas well, so that corresponding u8..u64Quoted 8 lines
access into this memory would still be possible.
I'm trying to analyze the situation that returning a scalar now and converting toPTR_TO_BTF_IDin the future will keep progs passing the verifier. Is it really the case?
Could you give a specific example that needs this support? It will help me understand this backward compatibility concern. What prog is doing with that 'u32 *' that is seen as scalar ? It cannot dereference it. Use it as what?
Quoted 3 lines
If this is from original bcc code, it will use
bpf_probe_readfor dereference. This is what I understand when I first reviewed this patch. But it will be good to get Jiri's confirmation.
it blocked me from accessing 'filename' argument when I probed do_sys_open via trampoline in bcc, like:
KRETFUNC_PROBE(do_sys_open)
{
const char *filename = (const char *) args[1];AFAICS the current code does not allow for trampoline arguments being other pointers than to void or struct, the patch should detect that the argument is pointer to scalar type and let it pass
jirka
Re: [PATCH 1/6] bpf: Allow ctx access for pointers to scalar
Quoted 65 lines · Jiri Olsa
Quoted 57 lines · Yonghong Song
Quoted 45 lines · Alexei Starovoitov
Quoted 31 lines · Jiri Olsa
When accessing the context we allow access to arguments with scalar type and pointer to struct. But we omit pointer to scalar type, which is the case for many functions and same case as when accessing scalar.
Adding the check if the pointer is to scalar type and allow it.
Acked-by John Fastabend <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- kernel/bpf/btf.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)kernel/bpf/btf.c +1
-1@@ -3668,7 +3668,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,const struct bpf_prog *prog, struct bpf_insn_access_aux *info) { - const struct btf_type *t = prog->aux->attach_func_proto; + const struct btf_type *tp, *t = prog->aux->attach_func_proto;struct bpf_prog *tgt_prog = prog->aux->linked_prog;struct btf *btf =bpf_prog_get_target_btf (prog);const char *tname = prog->aux->attach_func_name;@@ -3730,6 +3730,17 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,*/ return true;kernel/bpf/btf.c bool btf_ctx_access(int off, int size, enum bpf_access_type type, · line 3742
Quoted 10 lines
kernel/bpf/btf.c +10
-0+ tp =btf_type_by_id (btf, t->type); +/* skip modifiers */ +while (btf_type_is_modifier (tp)) + tp =btf_type_by_id (btf, tp->type); + +if (btf_type_is_int (tp) ||btf_type_is_enum (tp)) +/* This is a pointer scalar. +* It is the same as scalar from the verifier safety pov. +*/ +return true;The reason I didn't do it earlier is I was thinking to represent it as
PTR_TO_BTF_IDas well, so that corresponding u8..u64Quoted 8 lines
access into this memory would still be possible.
I'm trying to analyze the situation that returning a scalar now and converting toPTR_TO_BTF_IDin the future will keep progs passing the verifier. Is it really the case?
Could you give a specific example that needs this support? It will help me understand this backward compatibility concern. What prog is doing with that 'u32 *' that is seen as scalar ? It cannot dereference it. Use it as what?Quoted 3 lines
If this is from original bcc code, it will use
bpf_probe_readfor dereference. This is what I understand when I first reviewed this patch. But it will be good to get Jiri's confirmation.it blocked me from accessing 'filename' argument when I probed
do_sys_openvia trampoline in bcc, like:
Quoted 8 lines
KRETFUNC_PROBE(do_sys_open) { const char *filename = (const char *) args[1];AFAICS the current code does not allow for trampoline arguments being other pointers than to void or struct, the patch should detect that the argument is pointer to scalar type and let it pass
Got it. I've looked up your bcc patches and I agree that there is no way to workaround. BTF type argument of that kernel function is 'const char *' and the verifier will enforce that if bpf program tries to cast it the verifier will still see 'const char *'. (It's done this way by design). How about we special case 'char *' in the verifier? Then my concern regarding future extensibility of 'int *' and 'long *' will go away.
Compilers have a long history special casing 'char *'. In particular signed char because it's a pointer to null terminated string. I think it's still a special pointer from pointer aliasing point of view. I think the verifier can treat it as scalar here too. In the future the verifier will get smarter and will recognize it as PTR_TO_NULL_STRING while 'u8 *', 'u32 *' will be PTR_TO_BTF_ID. I think it will solve this particular issue. I like conservative approach to the verifier improvements: start with strict checking and relax it on case-by-case. Instead of accepting wide range of cases and cause potential compatibility issues.
Re: [PATCH 1/6] bpf: Allow ctx access for pointers to scalar
On Wed, Jan 22, 2020 at 08:09:59AM -0800, Alexei Starovoitov wrote:
SNIP
Quoted 18 lines
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
It cannot dereference it. Use it as what?
If this is from original bcc code, it will use
bpf_probe_readfor dereference. This is what I understand when I first reviewed this patch. But it will be good to get Jiri's confirmation.it blocked me from accessing 'filename' argument when I probed
do_sys_openvia trampoline in bcc, like:Quoted 8 lines
KRETFUNC_PROBE(do_sys_open) { const char *filename = (const char *) args[1];AFAICS the current code does not allow for trampoline arguments being other pointers than to void or struct, the patch should detect that the argument is pointer to scalar type and let it pass
Quoted 15 lines
Got it. I've looked up your bcc patches and I agree that there is no way to workaround. BTF type argument of that kernel function is 'const char *' and the verifier will enforce that if bpf program tries to cast it the verifier will still see 'const char *'. (It's done this way by design). How about we special case 'char *' in the verifier? Then my concern regarding future extensibility of 'int *' and 'long *' will go away.
Compilers have a long history special casing 'char *'. In particular signed char because it's a pointer to null terminated string. I think it's still a special pointer from pointer aliasing point of view. I think the verifier can treat it as scalar here too. In the future the verifier will get smarter and will recognize it asPTR_TO_NULL_STRINGwhile 'u8 *', 'u32 *' will bePTR_TO_BTF_ID. I think it will solve this particular issue. I like conservative approach to the verifier improvements: start with strict checking and relax it on case-by-case. Instead of accepting wide range of cases and cause potential compatibility issues.
ok, so something like below?
jirka
---
kernel/bpf/btf.c +16 -0
@@ -3664,6 +3664,19 @@ struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog) } } +static bool is_string_ptr (struct btf *btf,const struct btf_type *t) +{ +/* t comes in already as a pointer */ + t =btf_type_by_id (btf, t->type); + +/* allow const */ +if (BTF_INFO_KIND (t->info) ==BTF_KIND_CONST ) + t =btf_type_by_id (btf, t->type); + +/* char, signed char, unsigned char */ +return btf_type_is_int (t) && t->size ==1 ; +} +bool btf_ctx_access (int off,int size,enum bpf_access_type type,const struct bpf_prog *prog,struct bpf_insn_access_aux *info)
@@ -3730,6 +3743,9 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,*/ return true; +if (is_string_ptr (btf, t)) +return true; +/* this is a pointer to another type */ info->reg_type =PTR_TO_BTF_ID ;
Re: [PATCH 1/6] bpf: Allow ctx access for pointers to scalar
Quoted 55 lines · Jiri Olsa
On Wed, Jan 22, 2020 at 08:09:59AM -0800, Alexei Starovoitov wrote:
SNIP
Quoted 18 lines
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
It cannot dereference it. Use it as what?
If this is from original bcc code, it will use
bpf_probe_readfor dereference. This is what I understand when I first reviewed this patch. But it will be good to get Jiri's confirmation.it blocked me from accessing 'filename' argument when I probed
do_sys_openvia trampoline in bcc, like:Quoted 8 lines
KRETFUNC_PROBE(do_sys_open) { const char *filename = (const char *) args[1];AFAICS the current code does not allow for trampoline arguments being other pointers than to void or struct, the patch should detect that the argument is pointer to scalar type and let it pass
Quoted 15 lines
Got it. I've looked up your bcc patches and I agree that there is no way to workaround. BTF type argument of that kernel function is 'const char *' and the verifier will enforce that if bpf program tries to cast it the verifier will still see 'const char *'. (It's done this way by design). How about we special case 'char *' in the verifier? Then my concern regarding future extensibility of 'int *' and 'long *' will go away.
Compilers have a long history special casing 'char *'. In particular signed char because it's a pointer to null terminated string. I think it's still a special pointer from pointer aliasing point of view. I think the verifier can treat it as scalar here too. In the future the verifier will get smarter and will recognize it asPTR_TO_NULL_STRINGwhile 'u8 *', 'u32 *' will bePTR_TO_BTF_ID. I think it will solve this particular issue. I like conservative approach to the verifier improvements: start with strict checking and relax it on case-by-case. Instead of accepting wide range of cases and cause potential compatibility issues.ok, so something like below?
jirka
---
kernel/bpf/btf.c +2
-0@@ -3664,6 +3664,19 @@ struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog) } } +static bool is_string_ptr (struct btf *btf,const struct btf_type *t) +{
kernel/bpf/btf.c struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog) · line 3678
Quoted 10 lines
kernel/bpf/btf.c +10
-0+/* t comes in already as a pointer */ + t =btf_type_by_id (btf, t->type); + +/* allow const */ +if (BTF_INFO_KIND (t->info) ==BTF_KIND_CONST ) + t =btf_type_by_id (btf, t->type); + +/* char, signed char, unsigned char */ +return btf_type_is_int (t) && t->size ==1 ; +}
yep. looks like btf doesn't distinguish signedness for chars. So above is good.
Jiri Olsa
reviewerPATCH v3 2/6 bpf: Add bpf_perf_event_output_kfunc
In reply to Jiri OlsaAdding support to use perf_event_output in
BPF_TRACE_FENTRY/BPF_TRACE_FEXIT programs.
Using nesting regs array from raw tracepoint helpers.
--- kernel/trace/bpf_trace.c | 41 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)
kernel/trace/bpf_trace.c +41 -0
@@ -1172,6 +1172,43 @@ raw_tp_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) } } +BPF_CALL_5(bpf_perf_event_output_kfunc, void *, ctx, struct bpf_map *, map, + u64, flags, void *, data, u64, size) +{ +struct pt_regs *regs =get_bpf_raw_tp_regs (); +int ret; + +if (IS_ERR (regs)) +return PTR_ERR (regs); + +perf_fetch_caller_regs (regs); + ret =____bpf_perf_event_output (regs, map, flags, data, size); +put_bpf_raw_tp_regs (); +return ret; +} + +static const struct bpf_func_proto bpf_perf_event_output_proto_kfunc = { + .func = bpf_perf_event_output_kfunc, + .gpl_only = true, + .ret_type =RET_INTEGER , + .arg1_type =ARG_PTR_TO_CTX , + .arg2_type =ARG_CONST_MAP_PTR , + .arg3_type =ARG_ANYTHING , + .arg4_type =ARG_PTR_TO_MEM , + .arg5_type =ARG_CONST_SIZE_OR_ZERO , +}; + +static const struct bpf_func_proto * +kfunc_prog_func_proto (enum bpf_func_id func_id,const struct bpf_prog *prog) +{ +switch (func_id) { +case BPF_FUNC_perf_event_output: +return &bpf_perf_event_output_proto_kfunc; +default : +return tracing_func_proto (func_id, prog); + } +} +static const struct bpf_func_proto *tracing_prog_func_proto (enum bpf_func_id func_id,const struct bpf_prog *prog) {
@@ -1181,6 +1218,10 @@ tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)return &bpf_skb_output_proto;#endif default : +if (prog->expected_attach_type ==BPF_TRACE_FENTRY || + prog->expected_attach_type ==BPF_TRACE_FEXIT ) +return kfunc_prog_func_proto (func_id, prog); +return raw_tp_prog_func_proto (func_id, prog); } }
Signature · 1 line
2.24.1
Re: [PATCH 2/6] bpf: Add bpf_perf_event_output_kfunc
Quoted 22 lines · Jiri Olsa
Adding support to use
perf_event_outputin
BPF_TRACE_FENTRY/BPF_TRACE_FEXIT programs.Using nesting regs array from raw tracepoint helpers.
Signed-off-by Jiri Olsa <[email protected]>--- kernel/trace/bpf_trace.c | 41 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)kernel/trace/bpf_trace.c +4
-0@@ -1172,6 +1172,43 @@ raw_tp_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) } } +BPF_CALL_5(bpf_perf_event_output_kfunc, void *, ctx, struct bpf_map *, map, + u64, flags, void *, data, u64, size) +{ +struct pt_regs *regs =get_bpf_raw_tp_regs ();
kernel/trace/bpf_trace.c raw_tp_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · line 1188
Quoted 10 lines
kernel/trace/bpf_trace.c +10
-0+int ret; + +if (IS_ERR (regs)) +return PTR_ERR (regs); + +perf_fetch_caller_regs (regs); + ret =____bpf_perf_event_output (regs, map, flags, data, size); +put_bpf_raw_tp_regs (); +return ret; +}
I'm not sure why copy paste bpf_perf_event_output_raw_tp() into new function.
kernel/trace/bpf_trace.c tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · line 1225
Quoted 9 lines
kernel/trace/bpf_trace.c +4
-0@@ -1181,6 +1218,10 @@ tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)return &bpf_skb_output_proto;#endif default : +if (prog->expected_attach_type ==BPF_TRACE_FENTRY || + prog->expected_attach_type ==BPF_TRACE_FEXIT ) +return kfunc_prog_func_proto (func_id, prog); +return raw_tp_prog_func_proto (func_id, prog);
Are you saying bpf_perf_event_output_raw_tp() for some reason didn't work for fentry/fexit?
But above is exact copy-paste and it somehow worked?
Ditto for patches 3,4.
Re: [PATCH 2/6] bpf: Add bpf_perf_event_output_kfunc
Quoted 50 lines · Alexei Starovoitov
Quoted 22 lines · Jiri Olsa
Adding support to use
perf_event_outputin
BPF_TRACE_FENTRY/BPF_TRACE_FEXIT programs.Using nesting regs array from raw tracepoint helpers.
Signed-off-by Jiri Olsa <[email protected]>--- kernel/trace/bpf_trace.c | 41 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)kernel/trace/bpf_trace.c +4
-0@@ -1172,6 +1172,43 @@ raw_tp_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) } } +BPF_CALL_5(bpf_perf_event_output_kfunc, void *, ctx, struct bpf_map *, map, + u64, flags, void *, data, u64, size) +{ +struct pt_regs *regs =get_bpf_raw_tp_regs ();kernel/trace/bpf_trace.c raw_tp_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · line 1188
Quoted 10 lines
kernel/trace/bpf_trace.c +10
-0+int ret; + +if (IS_ERR (regs)) +return PTR_ERR (regs); + +perf_fetch_caller_regs (regs); + ret =____bpf_perf_event_output (regs, map, flags, data, size); +put_bpf_raw_tp_regs (); +return ret; +}I'm not sure why copy paste
bpf_perf_event_output_raw_tp()into new function.kernel/trace/bpf_trace.c tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · line 1225
Quoted 9 lines
kernel/trace/bpf_trace.c +4
-0@@ -1181,6 +1218,10 @@ tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)return &bpf_skb_output_proto;#endif default : +if (prog->expected_attach_type ==BPF_TRACE_FENTRY || + prog->expected_attach_type ==BPF_TRACE_FEXIT ) +return kfunc_prog_func_proto (func_id, prog); +return raw_tp_prog_func_proto (func_id, prog);Are you saying
bpf_perf_event_output_raw_tp()for some reason didn't work for fentry/fexit?
But above is exact copy-paste and it somehow worked?
Quoted 1 line
Ditto for patches 3,4.
ugh right.. did not realize that after switching to the rawtp regs nest arrays it's identical and we don't need that
jirka
Jiri Olsa
reviewerPATCH v3 3/6 bpf: Add bpf_get_stackid_kfunc
In reply to Jiri OlsaAdding support to use bpf_get_stackid_kfunc in
BPF_TRACE_FENTRY/BPF_TRACE_FEXIT programs.
--- kernel/trace/bpf_trace.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
kernel/trace/bpf_trace.c +27 -0
@@ -1198,12 +1198,39 @@ static const struct bpf_func_proto bpf_perf_event_output_proto_kfunc = { .arg5_type = ARG_CONST_SIZE_OR_ZERO, }; +BPF_CALL_3(bpf_get_stackid_kfunc, void*, args, + struct bpf_map *, map, u64, flags) +{ +struct pt_regs *regs =get_bpf_raw_tp_regs (); +int ret; + +if (IS_ERR (regs)) +return PTR_ERR (regs); + +perf_fetch_caller_regs (regs); + ret =bpf_get_stackid ((unsigned long ) regs, (unsigned long ) map, + flags,0 ,0 ); +put_bpf_raw_tp_regs (); +return ret; +} + +static const struct bpf_func_proto bpf_get_stackid_proto_kfunc = { + .func = bpf_get_stackid_kfunc, + .gpl_only = true, + .ret_type =RET_INTEGER , + .arg1_type =ARG_PTR_TO_CTX , + .arg2_type =ARG_CONST_MAP_PTR , + .arg3_type =ARG_ANYTHING , +}; +static const struct bpf_func_proto *kfunc_prog_func_proto (enum bpf_func_id func_id,const struct bpf_prog *prog) {switch (func_id) {case BPF_FUNC_perf_event_output:return &bpf_perf_event_output_proto_kfunc; +case BPF_FUNC_get_stackid: +return &bpf_get_stackid_proto_kfunc;default :return tracing_func_proto (func_id, prog); }
Signature · 1 line
2.24.1
Jiri Olsa
reviewerPATCH v3 4/6 bpf: Add bpf_get_stack_kfunc
In reply to Jiri OlsaAdding support to use bpf_get_stack_kfunc in
BPF_TRACE_FENTRY/BPF_TRACE_FEXIT programs.
--- kernel/trace/bpf_trace.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
kernel/trace/bpf_trace.c +28 -0
@@ -1223,6 +1223,32 @@ static const struct bpf_func_proto bpf_get_stackid_proto_kfunc = { .arg3_type = ARG_ANYTHING, }; +BPF_CALL_4(bpf_get_stack_kfunc, void*, args, + void *, buf, u32, size, u64, flags) +{ +struct pt_regs *regs =get_bpf_raw_tp_regs (); +int ret; + +if (IS_ERR (regs)) +return PTR_ERR (regs); + +perf_fetch_caller_regs (regs); + ret =bpf_get_stack ((unsigned long ) regs, (unsigned long ) buf, + (unsigned long ) size, flags,0 ); +put_bpf_raw_tp_regs (); +return ret; +} + +static const struct bpf_func_proto bpf_get_stack_proto_kfunc = { + .func = bpf_get_stack_raw_tp, + .gpl_only = true, + .ret_type =RET_INTEGER , + .arg1_type =ARG_PTR_TO_CTX , + .arg2_type =ARG_PTR_TO_MEM , + .arg3_type =ARG_CONST_SIZE_OR_ZERO , + .arg4_type =ARG_ANYTHING , +}; +static const struct bpf_func_proto *kfunc_prog_func_proto (enum bpf_func_id func_id,const struct bpf_prog *prog) {
@@ -1231,6 +1257,8 @@ kfunc_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)return &bpf_perf_event_output_proto_kfunc;case BPF_FUNC_get_stackid:return &bpf_get_stackid_proto_kfunc; +case BPF_FUNC_get_stack: +return &bpf_get_stack_proto_kfunc;default :return tracing_func_proto (func_id, prog); }
Signature · 1 line
2.24.1
Jiri Olsa
reviewerPATCH v3 5/6 bpf: Allow to resolve bpf trampoline and dispatcher in unwind
In reply to Jiri OlsaWhen unwinding the stack we need to identify each address to successfully continue. Adding latch tree to keep trampolines for quick lookup during the unwind.
The patch uses first 48 bytes for latch tree node, leaving 4048 bytes from the rest of the page for trampoline or dispatcher generated code.
It's still enough not to affect trampoline and dispatcher progs maximum counts.
--- include/linux/bpf.h | 12 +++++- kernel/bpf/dispatcher.c | 4 +- kernel/bpf/trampoline.c | 82 +++++++++++++++++++++++++++++++++++++---- kernel/extable.c | 7 +++- 4 files changed, 93 insertions(+), 12 deletions(-)
include/linux/bpf.h +11 -1
@@ -519,7 +519,6 @@ struct bpf_trampoline *bpf_trampoline_lookup(u64 key);int bpf_trampoline_link_prog (struct bpf_prog *prog);int bpf_trampoline_unlink_prog (struct bpf_prog *prog);void bpf_trampoline_put (struct bpf_trampoline *tr); -void *bpf_jit_alloc_exec_page (void );#define BPF_DISPATCHER_INIT (name) { \ .mutex = __MUTEX_INITIALIZER(name.mutex), \ .func = &name##func, \
@@ -551,6 +550,13 @@ void *bpf_jit_alloc_exec_page(void);#define BPF_DISPATCHER_PTR (name) (&name)void bpf_dispatcher_change_prog (struct bpf_dispatcher *d,struct bpf_prog *from,struct bpf_prog *to); +struct bpf_image { +struct latch_tree_node tnode; +unsigned char data[]; +}; +#define BPF_IMAGE_SIZE (PAGE_SIZE - sizeof(struct bpf_image)) +bool is_bpf_image_address (unsigned long address); +void *bpf_image_alloc (void );#else static inline struct bpf_trampoline *bpf_trampoline_lookup (u64 key) {
@@ -572,6 +578,10 @@ static inline void bpf_trampoline_put(struct bpf_trampoline *tr) {}static inline void bpf_dispatcher_change_prog (struct bpf_dispatcher *d,struct bpf_prog *from,struct bpf_prog *to) {} +static inline bool is_bpf_image_address (unsigned long address) +{ +return false; +}#endif struct bpf_func_info_aux {
kernel/bpf/dispatcher.c +2 -2
@@ -113,7 +113,7 @@ static void bpf_dispatcher_update(struct bpf_dispatcher *d, int prev_num_progs) noff =0 ; }else { old = d->image + d->image_off; - noff = d->image_off ^ (PAGE_SIZE /2 ); + noff = d->image_off ^ (BPF_IMAGE_SIZE /2 ); } new = d->num_progs ? d->image + noff :NULL ;
@@ -140,7 +140,7 @@ void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from,mutex_lock (&d->mutex);if (!d->image) { - d->image =bpf_jit_alloc_exec_page (); + d->image =bpf_image_alloc ();if (!d->image) goto out; }
kernel/bpf/trampoline.c +75 -7
129 lines not on this page · raw message
kernel/extable.c +5 -2
@@ -131,8 +131,9 @@ int kernel_text_address(unsigned long addr)* triggers a stack trace, or a WARN() that happens during * coming back from idle, or cpu on or offlining. * -* is_module_text_address() as well as the kprobe slots -* and is_bpf_text_address() require RCU to be watching. +* is_module_text_address() as well as the kprobe slots, +* is_bpf_text_address() and is_bpf_image_address require +* RCU to be watching. */ no_rcu = !rcu_is_watching ();
@@ -148,6 +149,8 @@ int kernel_text_address(unsigned long addr) goto out;if (is_bpf_text_address (addr)) goto out; +if (is_bpf_image_address (addr)) + goto out; ret =0 ; out:if (no_rcu)
Signature · 1 line
2.24.1
Jiri Olsa
reviewerPATCH v3 6/6 selftest/bpf: Add test for allowed trampolines count
In reply to Jiri OlsaThere's limit of 40 programs tht can be attached to trampoline for one function. Adding test that tries to attach that many plus one extra that needs to fail.
--- .../bpf/prog_tests/trampoline_count.c | 112 ++++++++++++++++++ .../bpf/progs/test_trampoline_count.c | 21 ++++ 2 files changed, 133 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/trampoline_count.c create mode 100644 tools/testing/selftests/bpf/progs/test_trampoline_count.c
tools/testing/selftests/bpf/prog_tests/trampoline_count.c new file +112 -0
@@ -0,0 +1,112 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include <sched.h> +#include <sys/prctl.h> +#include <test_progs.h> + +#define MAX_TRAMP_PROGS 40 + +struct inst { +struct bpf_object *obj; +struct bpf_link *link_fentry; +struct bpf_link *link_fexit; +}; + +static int test_task_rename (void ) +{ +int fd, duration =0 , err; +char buf[] ="test_overhead" ; + + fd =open ("/proc/self/comm" ,O_WRONLY |O_TRUNC ); +if (CHECK (fd <0 ,"open /proc" ,"err %d" , errno)) +return -1 ; + err =write (fd, buf,sizeof (buf)); +if (err <0 ) { +CHECK (err <0 ,"task rename" ,"err %d" , errno); +close (fd); +return -1 ; + } +close (fd); +return 0 ; +} + +static struct bpf_link *load (struct bpf_object *obj,const char *name) +{ +struct bpf_program *prog; +int duration =0 ; + + prog =bpf_object__find_program_by_title (obj, name); +if (CHECK (!prog,"find_probe" ,"prog '%s' not found\n" , name)) +return ERR_PTR (-EINVAL ); +return bpf_program__attach_trace (prog); +} + +void test_trampoline_count (void ) +{ +const char *fentry_name ="fentry/__set_task_comm" ; +const char *fexit_name ="fexit/__set_task_comm" ; +const char *object ="test_trampoline_count.o" ; +struct inst inst[MAX_TRAMP_PROGS ] = {0 }; +int err, i =0 , duration =0 ; +struct bpf_object *obj; +struct bpf_link *link; +char comm[16 ] = {}; + +/* attach 'allowed' 40 trampoline programs */ +for (i =0 ; i <MAX_TRAMP_PROGS ; i++) { + obj =bpf_object__open_file (object,NULL ); +if (CHECK (IS_ERR (obj),"obj_open_file" ,"err %ld\n" ,PTR_ERR (obj))) + goto cleanup; + + err =bpf_object__load (obj); +if (CHECK (err,"obj_load" ,"err %d\n" , err)) + goto cleanup; + inst[i].obj = obj; + +if (rand () %2 ) { + link =load (obj, fentry_name); +if (CHECK (IS_ERR (link),"attach prog" ,"err %ld\n" ,PTR_ERR (link))) + goto cleanup; + inst[i].link_fentry = link; + }else { + link =load (obj, fexit_name); +if (CHECK (IS_ERR (link),"attach prog" ,"err %ld\n" ,PTR_ERR (link))) + goto cleanup; + inst[i].link_fexit = link; + } + } + +/* and try 1 extra.. */ + obj =bpf_object__open_file (object,NULL ); +if (CHECK (IS_ERR (obj),"obj_open_file" ,"err %ld\n" ,PTR_ERR (obj))) + goto cleanup; + + err =bpf_object__load (obj); +if (CHECK (err,"obj_load" ,"err %d\n" , err)) + goto cleanup_extra; + +/* ..that needs to fail */ + link =load (obj, fentry_name); +if (CHECK (!IS_ERR (link),"cannot attach over the limit" ,"err %ld\n" ,PTR_ERR (link))) { +bpf_link__destroy (link); + goto cleanup_extra; + } + +/* with E2BIG error */ +CHECK (PTR_ERR (link) != -E2BIG ,"proper error check" ,"err %ld\n" ,PTR_ERR (link)); + +/* and finaly execute the probe */ +if (CHECK_FAIL (prctl (PR_GET_NAME , comm,0L ,0L ,0L ))) + goto cleanup_extra; +CHECK_FAIL (test_task_rename ()); +CHECK_FAIL (prctl (PR_SET_NAME , comm,0L ,0L ,0L )); + +cleanup_extra: +bpf_object__close (obj); +cleanup: +while (--i) { +bpf_link__destroy (inst[i].link_fentry); +bpf_link__destroy (inst[i].link_fexit); +bpf_object__close (inst[i].obj); + } +}
tools/testing/selftests/bpf/progs/test_trampoline_count.c new file +21 -0
@@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <stdbool.h> +#include <stddef.h> +#include <linux/bpf.h> +#include "bpf_trace_helpers.h" + +struct task_struct ; + +SEC("fentry/__set_task_comm") +int BPF_PROG (prog1 ,struct task_struct *tsk,const char *buf,bool exec) +{ +return 0 ; +} + +SEC("fexit/__set_task_comm") +int BPF_PROG (prog2 ,struct task_struct *tsk,const char *buf,bool exec) +{ +return 0 ; +} + +char _license[] SEC("license") = "GPL";
Signature · 1 line
2.24.1
Re: [PATCH 6/6] selftest/bpf: Add test for allowed trampolines count
Quoted 7 lines · Jiri Olsa
There's limit of 40 programs tht can be attached to trampoline for one function. Adding test that tries to attach that many plus one extra that needs to fail.
Signed-off-by Jiri Olsa <[email protected]>
I don't mind another test. Just pointing out that there is one for this purpose already :) prog_tests/fexit_stress.c
Yours is better. Mine wasn't that sophisticated. :)
Re: [PATCH 6/6] selftest/bpf: Add test for allowed trampolines count
Quoted 7 lines · Alexei Starovoitov
Quoted 6 lines · Jiri Olsa
There's limit of 40 programs tht can be attached to trampoline for one function. Adding test that tries to attach that many plus one extra that needs to fail.
Quoted 6 lines
Quoted 1 line
Signed-off-by Jiri Olsa <[email protected]>I don't mind another test. Just pointing out that there is one for this purpose already :)
prog_tests/fexit_stress.c
Yours is better. Mine wasn't that sophisticated. :)
ok ;-) did not notice that one.. just wanted to be sure the unwind change won't screw that
jirka