bpf: implement bpf_send_signal() helper
Subsystems (8)
v2 cover bpf: implement bpf_send_signal() helper
Currently, bpf program can already collect stack traces when certain events happens (e.g., cache miss counter or cpu clock counter overflows). These stack traces can be used for performance analysis. For jitted programs, e.g., hhvm (jited php), it is very hard to get the true stack trace in the bpf program due to jit complexity.
To resolve this issue, hhvm implements a signal handler, e.g. for SIGALARM, and a set of program locations which it can dump stack traces. When it receives a signal, it will dump the stack in next such program location.
The following is the current way to handle this use case:
. profiler installs a bpf program and polls on a map.
When certain event happens, bpf program writes to a map.
. Once receiving the information from the map, the profiler
sends a signal to hhvm.
This method could have large delays and cause profiling results skewed.
This patch implements bpf_send_signal() helper to send a signal to hhvm in real time, resulting in intended stack traces.
The patch is sent out as RFC as (1). I have not found a simple solution to return error code from irq_work, and (2). I would like some general feedback about the approach.
Changelogs:
v1 -> v2:
. fixed a compilation error/warning (missing return value in one
error branch) discovered by kbuild bot.
Yonghong Song (3):
bpf: implement bpf_send_signal() helper tools/bpf: sync bpf uapi header bpf.h tools/bpf: add a selftest for bpf_send_signal() helper
include/uapi/linux/bpf.h | 15 +- kernel/trace/bpf_trace.c | 85 ++++++++ tools/include/uapi/linux/bpf.h | 15 +- tools/testing/selftests/bpf/Makefile | 5 +- tools/testing/selftests/bpf/bpf_helpers.h | 2 + .../bpf/progs/test_send_signal_kern.c | 50 +++++ .../selftests/bpf/test_send_signal_user.c | 186 ++++++++++++++++++ 7 files changed, 354 insertions(+), 4 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/test_send_signal_kern.c create mode 100644 tools/testing/selftests/bpf/test_send_signal_user.c
Signature · 1 line
2.17.1
RFC PATCH v2 1/3 bpf: implement bpf_send_signal() helper
This patch tries to solve the following specific use case.
Currently, bpf program can already collect stack traces when certain events happens (e.g., cache miss counter or cpu clock counter overflows). These stack traces can be used for performance analysis. For jitted programs, e.g., hhvm (jited php), it is very hard to get the true stack trace in the bpf program due to jit complexity.
To resolve this issue, hhvm implements a signal handler, e.g. for SIGALARM, and a set of program locations which it can dump stack traces. When it receives a signal, it will dump the stack in next such program location.
The following is the current way to handle this use case:
. profiler installs a bpf program and polls on a map.
When certain event happens, bpf program writes to a map.
. Once receiving the information from the map, the profiler
sends a signal to hhvm.
This method could have large delays and causing profiling results skewed.
This patch implements bpf_send_signal() helper to send a signal to hhvm in real time, resulting in intended stack traces.
--- include/uapi/linux/bpf.h | 15 ++++++- kernel/trace/bpf_trace.c | 85 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+), 1 deletion(-)
v1 -> v2:
fixed a compilation warning (missing return value in case) reported by kbuild test robot.
added Reported-by in the above to notify the bot.
include/uapi/linux/bpf.h +14 -1
@@ -2667,6 +2667,18 @@ union bpf_attr {* 0 on success. * * **-ENOENT** if the bpf-local-storage cannot be found. +* +* int bpf_send_signal(u32 pid, u32 sig) +* Description +* Send signal *sig* to *pid*. +* Return +* 0 on success or successfully queued. +* +* **-ENOENT** if *pid* cannot be found. +* +* **-EBUSY** if work queue under nmi is full. +* +* Other negative values for actual signal sending errors. */ #define __BPF_FUNC_MAPPER (FN ) \ FN(unspec), \
@@ -2777,7 +2789,8 @@ union bpf_attr {FN (strtol), \FN (strtoul), \FN (sk_storage_get), \ -FN (sk_storage_delete), +FN (sk_storage_delete), \ +FN (send_signal),/* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call
kernel/trace/bpf_trace.c +85 -0
102 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 23 lines · Yonghong Song
This patch tries to solve the following specific use case.
Currently, bpf program can already collect stack traces when certain events happens (e.g., cache miss counter or cpu clock counter overflows). These stack traces can be used for performance analysis. For jitted programs, e.g., hhvm (jited php), it is very hard to get the true stack trace in the bpf program due to jit complexity.
To resolve this issue, hhvm implements a signal handler, e.g. for SIGALARM, and a set of program locations which it can dump stack traces. When it receives a signal, it will dump the stack in next such program location.
The following is the current way to handle this use case:
. profiler installs a bpf program and polls on a map.
When certain event happens, bpf program writes to a map.
. Once receiving the information from the map, the profiler
sends a signal to hhvm.
This method could have large delays and causing profiling results skewed.
Quoted 4 lines
This patch implements
bpf_send_signal()helper to send a signal to hhvm in real time, resulting in intended stack traces.Reported-by kbuild test robot <[email protected]>
v2 addressing buildbot issue doesn't need to have such tag.
Quoted 56 lines
Signed-off-by Yonghong Song <[email protected]>--- include/uapi/linux/bpf.h | 15 ++++++- kernel/trace/bpf_trace.c | 85 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+), 1 deletion(-)v1 -> v2:
fixed a compilation warning (missing return value in case) reported by kbuild test robot.
added Reported-by in the above to notify the bot.include/uapi/linux/bpf.h +14
-1@@ -2667,6 +2667,18 @@ union bpf_attr {* 0 on success. * * **-ENOENT** if the bpf-local-storage cannot be found. +* +* int bpf_send_signal(u32 pid, u32 sig) +* Description +* Send signal *sig* to *pid*. +* Return +* 0 on success or successfully queued. +* +* **-ENOENT** if *pid* cannot be found. +* +* **-EBUSY** if work queue under nmi is full. +* +* Other negative values for actual signal sending errors. */ #define __BPF_FUNC_MAPPER (FN ) \ FN(unspec), \@@ -2777,7 +2789,8 @@ union bpf_attr {FN (strtol), \FN (strtoul), \FN (sk_storage_get), \ -FN (sk_storage_delete), +FN (sk_storage_delete), \ +FN (send_signal),/* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call kernel/trace/bpf_trace.c +4
-0@@ -559,6 +559,76 @@ static const struct bpf_func_proto bpf_probe_read_str_proto = { .arg3_type = ARG_ANYTHING, }; +struct send_signal_irq_work { +struct irq_work irq_work; +u32 pid; +u32 sig;
kernel/trace/bpf_trace.c static const struct bpf_func_proto bpf_probe_read_str_proto = { · line 575
Quoted 10 lines
kernel/trace/bpf_trace.c +10
-0+}; + +static DEFINE_PER_CPU(struct send_signal_irq_work, send_signal_work); + +static int do_bpf_send_signal_pid (u32 pid,u32 sig) +{ +struct task_struct *task; +struct pid *pidp; + + pidp =find_vpid (pid);
it takes pidns from current which should be valid
which makes bpf prog depend on current, but from nmi
there are no guarantees.
I think find_pid_ns() would be cleaner, but then the question would be how to pass pidns...
Another issue is instability of pid as an integer...
upcoming pidfd could be the answer.
At this point I think it's cleaner to make such api to send signal to existing process only under the same conditions as in bpf_probe_write_user. Would that work for your use case?
Quoted 24 lines · Alexei Starovoitov
Quoted 23 lines · Yonghong Song
This patch tries to solve the following specific use case.
Currently, bpf program can already collect stack traces when certain events happens (e.g., cache miss counter or cpu clock counter overflows). These stack traces can be used for performance analysis. For jitted programs, e.g., hhvm (jited php), it is very hard to get the true stack trace in the bpf program due to jit complexity.
To resolve this issue, hhvm implements a signal handler, e.g. for SIGALARM, and a set of program locations which it can dump stack traces. When it receives a signal, it will dump the stack in next such program location.
The following is the current way to handle this use case:
. profiler installs a bpf program and polls on a map.
When certain event happens, bpf program writes to a map.
. Once receiving the information from the map, the profiler
sends a signal to hhvm.
This method could have large delays and causing profiling results skewed.
Quoted 6 lines
Quoted 4 lines
This patch implements
bpf_send_signal()helper to send a signal to hhvm in real time, resulting in intended stack traces.Reported-by kbuild test robot <[email protected]>v2 addressing buildbot issue doesn't need to have such tag.
I will drop this in the next revision. The intention is, as suggested by buildbot email, is to give credit to buildbot and let it know the bug is taken care of.
Quoted 63 lines
Quoted 63 lines
Signed-off-by Yonghong Song <[email protected]>--- include/uapi/linux/bpf.h | 15 ++++++- kernel/trace/bpf_trace.c | 85 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+), 1 deletion(-)v1 -> v2:
fixed a compilation warning (missing return value in case) reported by kbuild test robot.
added Reported-by in the above to notify the bot.include/uapi/linux/bpf.h +14
-1@@ -2667,6 +2667,18 @@ union bpf_attr {* 0 on success. * * **-ENOENT** if the bpf-local-storage cannot be found. +* +* int bpf_send_signal(u32 pid, u32 sig) +* Description +* Send signal *sig* to *pid*. +* Return +* 0 on success or successfully queued. +* +* **-ENOENT** if *pid* cannot be found. +* +* **-EBUSY** if work queue under nmi is full. +* +* Other negative values for actual signal sending errors. */ #define __BPF_FUNC_MAPPER (FN ) \ FN(unspec), \@@ -2777,7 +2789,8 @@ union bpf_attr {FN (strtol), \FN (strtoul), \FN (sk_storage_get), \ -FN (sk_storage_delete), +FN (sk_storage_delete), \ +FN (send_signal),/* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call kernel/trace/bpf_trace.c +10
-0@@ -559,6 +559,76 @@ static const struct bpf_func_proto bpf_probe_read_str_proto = { .arg3_type = ARG_ANYTHING, }; +struct send_signal_irq_work { +struct irq_work irq_work; +u32 pid; +u32 sig; +}; + +static DEFINE_PER_CPU(struct send_signal_irq_work, send_signal_work); + +static int do_bpf_send_signal_pid (u32 pid,u32 sig) +{
Quoted 8 lines
kernel/trace/bpf_trace.c static const struct bpf_func_proto bpf_probe_read_str_proto = { · line 575
Quoted 4 lines
kernel/trace/bpf_trace.c +4
-0+struct task_struct *task; +struct pid *pidp; + + pidp =find_vpid (pid);it takes pidns from current which should be valid which makes bpf prog depend on current, but from nmi there are no guarantees.
This is right. For nmi, I have found some fields of "current" may not be valid.
Quoted 2 lines
I think
find_pid_ns()would be cleaner, but then the question would be how to pass pidns...
The "current" is mostly used to find pid namespace task_active_pid_ns(current). Yes, we can use find_pid_ns(), but we need to get pid_namespace in advance then pid, maybe by introducing a new map or something, or may be directly using the below pidfd based approach to hold the reference count in a map.
Quoted 2 lines
Another issue is instability of pid as an integer... upcoming pidfd could be the answer.
Will take a look at the details. Thanks for referencing.
Quoted 3 lines
At this point I think it's cleaner to make such api to send signal to existing process only under the same conditions as in
bpf_probe_write_user. Would that work for your use case?
Let me double check this.
RFC PATCH v2 2/3 tools/bpf: sync bpf uapi header bpf.h
In reply to Yonghong Songsync bpf uapi header bpf.h to tools directory.
--- tools/include/uapi/linux/bpf.h | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
tools/include/uapi/linux/bpf.h +14 -1
@@ -2667,6 +2667,18 @@ union bpf_attr {* 0 on success. * * **-ENOENT** if the bpf-local-storage cannot be found. +* +* int bpf_send_signal(u32 pid, u32 sig) +* Description +* Send signal *sig* to *pid*. +* Return +* 0 on success or successfully queued. +* +* **-ENOENT** if *pid* cannot be found. +* +* **-EBUSY** if work queue under nmi is full. +* +* Other negative values for actual signal sending errors. */ #define __BPF_FUNC_MAPPER (FN ) \ FN(unspec), \
@@ -2777,7 +2789,8 @@ union bpf_attr {FN (strtol), \FN (strtoul), \FN (sk_storage_get), \ -FN (sk_storage_delete), +FN (sk_storage_delete), \ +FN (send_signal),/* integer value in 'imm' field of BPF_CALL instruction selects which helper * function eBPF program intends to call
Signature · 1 line
2.17.1
RFC PATCH v2 3/3 tools/bpf: add a selftest for bpf_send_signal() helper
In reply to Yonghong SongThe test covered both nmi and tracepoint perf events.
$ ./test_send_signal_user test_send_signal (tracepoint): OK test_send_signal (perf_event): OK
--- tools/testing/selftests/bpf/Makefile | 5 +- tools/testing/selftests/bpf/bpf_helpers.h | 2 + .../bpf/progs/test_send_signal_kern.c | 50 +++++ .../selftests/bpf/test_send_signal_user.c | 186 ++++++++++++++++++ 4 files changed, 241 insertions(+), 2 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/test_send_signal_kern.c create mode 100644 tools/testing/selftests/bpf/test_send_signal_user.c
tools/testing/selftests/bpf/Makefile +3 -2
@@ -23,7 +23,8 @@ TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test test_align test_verifier_log test_dev_cgroup test_tcpbpf_user \ test_sock test_btf test_sockmap test_lirc_mode2_user get_cgroup_id_user \ test_socket_cookie test_cgroup_storage test_select_reuseport test_section_names \ - test_netcnt test_tcpnotify_user test_sock_fields test_sysctl + test_netcnt test_tcpnotify_user test_sock_fields test_sysctl \ + test_send_signal_user BPF_OBJ_FILES = $(patsubst %.c,%.o, $(notdir $(wildcard progs/*.c))) TEST_GEN_FILES = $(BPF_OBJ_FILES)
@@ -124,7 +125,7 @@ endif CLANG_SYS_INCLUDES := $(shell $(CLANG) -v -E - </dev/null 2>&1 \ | sed -n '/<...> search starts here:/,/End of search list./{ s| \(/.*\)|-idirafter \1|p }') -CLANG_FLAGS = -I. -I./include/uapi -I../../../include/uapi \ +CLANG_FLAGS = -I. -I./include/uapi -I../../../include/uapi -I../../../include \ $(CLANG_SYS_INCLUDES) \ -Wno-compare-distinct-pointer-types
tools/testing/selftests/bpf/bpf_helpers.h +2 -0
@@ -216,6 +216,8 @@ static void *(*bpf_sk_storage_get)(void *map, struct bpf_sock *sk, (void *) BPF_FUNC_sk_storage_get;static int (*bpf_sk_storage_delete)(void *map,struct bpf_sock *sk) = (void *)BPF_FUNC_sk_storage_delete; +static int (*bpf_send_signal)(unsigned pid,unsigned sig) = + (void *)BPF_FUNC_send_signal;/* llvm builtin functions that eBPF C program may use to * emit BPF_LD_ABS and BPF_LD_IND instructions
tools/testing/selftests/bpf/progs/test_send_signal_kern.c new file +50 -0
50 lines not on this page · raw message
tools/testing/selftests/bpf/test_send_signal_user.c new file +186 -0
186 lines not on this page · raw message
Signature · 1 line
2.17.1