samples:bpf: introduce task detector
Subsystems (3)
RFC PATCH 1/1 samples:bpf: introduce task detector
This is a tool to trace the related schedule events of a specified task, eg the migration, sched in/out, wakeup and sleep/block.
The event was translated into sentence to be more readable, by execute command 'task_detector -p 49870' we continually tracing the schedule events related to 'top' like:
----------------------------
923455517688 CPU=23 PID=49870 COMM=top ENQUEUE 923455519633 CPU=23 PID=0 COMM=IDLE PREEMPTED 1945ns 923455519868 CPU=23 PID=49870 COMM=top EXECUTE AFTER WAITED 2180ns 923468279019 CPU=23 PID=49870 COMM=top WAIT AFTER EXECUTED 12ms 923468279220 CPU=23 PID=128 COMM=ksoftirqd/23 PREEMPT 923468283051 CPU=23 PID=128 COMM=ksoftirqd/23 DEQUEUE AFTER PREEMPTED 3831ns 923468283216 CPU=23 PID=49870 COMM=top EXECUTE AFTER WAITED 4197ns 923476280180 CPU=23 PID=49870 COMM=top WAIT AFTER EXECUTED 7996us 923476280350 CPU=23 PID=128 COMM=ksoftirqd/23 PREEMPT 923476322029 CPU=23 PID=128 COMM=ksoftirqd/23 DEQUEUE AFTER PREEMPTED 41us 923476322150 CPU=23 PID=49870 COMM=top EXECUTE AFTER WAITED 41us 923479726879 CPU=23 PID=49870 COMM=top DEQUEUE AFTER EXECUTED 3404us
----------------------------
This could be helpful on debugging the competition on CPU resource, to find out who has stolen the CPU and how much it stolen.
It can also tracing the syscall by append option -s.
--- samples/bpf/Makefile | 3 + samples/bpf/task_detector.h | 382 +++++++++++++++++++++++++++++++++++++++ samples/bpf/task_detector_kern.c | 329 +++++++++++++++++++++++++++++++++ samples/bpf/task_detector_user.c | 314 ++++++++++++++++++++++++++++++++ 4 files changed, 1028 insertions(+) create mode 100644 samples/bpf/task_detector.h create mode 100644 samples/bpf/task_detector_kern.c create mode 100644 samples/bpf/task_detector_user.c
samples/bpf/Makefile +3 -0
@@ -53,6 +53,7 @@ tprogs-y += task_fd_query tprogs-y += xdp_sample_pkts tprogs-y += ibumad tprogs-y += hbm +tprogs-y += task_detector # Libbpf dependencies LIBBPF = $(TOOLS_PATH)/lib/bpf/libbpf.a
@@ -109,6 +110,7 @@ task_fd_query-objs := bpf_load.o task_fd_query_user.o $(TRACE_HELPERS) xdp_sample_pkts-objs := xdp_sample_pkts_user.o $(TRACE_HELPERS) ibumad-objs := bpf_load.o ibumad_user.o $(TRACE_HELPERS) hbm-objs := bpf_load.o hbm.o $(CGROUP_HELPERS) +task_detector-objs := bpf_load.o task_detector_user.o $(TRACE_HELPERS) # Tell kbuild to always build the programs always-y := $(tprogs-y)
@@ -170,6 +172,7 @@ always-y += ibumad_kern.o always-y += hbm_out_kern.o always-y += hbm_edt_kern.o always-y += xdpsock_kern.o +always-y += task_detector_kern.o ifeq ($(ARCH), arm) # Strip all except -D__LINUX_ARM_ARCH__ option needed to handle linux
samples/bpf/task_detector.h new file +382 -0
382 lines not on this page · raw message
samples/bpf/task_detector_kern.c new file +329 -0
329 lines not on this page · raw message
samples/bpf/task_detector_user.c new file +314 -0
314 lines not on this page · raw message
Signature · 1 line
2.14.4.44.g2045bb6
Quoted 24 lines · 王贇
This is a tool to trace the related schedule events of a specified task, eg the migration, sched in/out, wakeup and sleep/block.
The event was translated into sentence to be more readable, by execute command '
task_detector-p 49870' we continually tracing the schedule events related to 'top' like:----------------------------
923455517688 CPU=23 PID=49870 COMM=top ENQUEUE 923455519633 CPU=23 PID=0 COMM=IDLE PREEMPTED 1945ns 923455519868 CPU=23 PID=49870 COMM=top EXECUTE AFTER WAITED 2180ns 923468279019 CPU=23 PID=49870 COMM=top WAIT AFTER EXECUTED 12ms 923468279220 CPU=23 PID=128 COMM=ksoftirqd/23 PREEMPT 923468283051 CPU=23 PID=128 COMM=ksoftirqd/23 DEQUEUE AFTER PREEMPTED 3831ns 923468283216 CPU=23 PID=49870 COMM=top EXECUTE AFTER WAITED 4197ns 923476280180 CPU=23 PID=49870 COMM=top WAIT AFTER EXECUTED 7996us 923476280350 CPU=23 PID=128 COMM=ksoftirqd/23 PREEMPT 923476322029 CPU=23 PID=128 COMM=ksoftirqd/23 DEQUEUE AFTER PREEMPTED 41us 923476322150 CPU=23 PID=49870 COMM=top EXECUTE AFTER WAITED 41us 923479726879 CPU=23 PID=49870 COMM=top DEQUEUE AFTER EXECUTED 3404us----------------------------
Quoted 8 lines
This could be helpful on debugging the competition on CPU resource, to find out who has stolen the CPU and how much it stolen.
It can also tracing the syscall by append option -s.
Signed-off-by Michael Wang <[email protected]>---
I haven't looked through implementation thoroughly yet. But I have few general remarks.
This looks like a useful and generic tool. I think it will get most attention and be most useful if it will be part of BCC tools. There is already a set of generic tools that use libbpf and CO-RE, see [0]. It feels like this belongs there.
Some of the annoying parts (e.g., syscall name translation) is already generalized as part of syscount tool PR (to be hopefully merged soon), so you'll be able to save quite a lot of code with this. There is also a common build infra that takes care of things like vmlinux.h, which would provide definitions for all those xxx_args structs that you had to manually define.
With CO-RE, it also will allow to compile this tool once and run it on many different kernels without recompilation. Please do take a look and submit a PR there, it will be a good addition to the toolkit (and will force you write a bit of README explaining use of this tool as well ;).
As for the code itself, I haven't gone through it much, but please convert map definition syntax to BTF-defined one. The one you are using is a legacy one. Thanks!
[0] https://github.com/iovisor/bcc/tree/master/libbpf-tools
Quoted 9 lines
samples/bpf/Makefile | 3 + samples/bpf/task_detector.h | 382 +++++++++++++++++++++++++++++++++++++++ samples/bpf/task_detector_kern.c | 329 +++++++++++++++++++++++++++++++++ samples/bpf/task_detector_user.c | 314 ++++++++++++++++++++++++++++++++ 4 files changed, 1028 insertions(+) create mode 100644 samples/bpf/task_detector.h create mode 100644 samples/bpf/task_detector_kern.c create mode 100644 samples/bpf/task_detector_user.c
[...]
Hi, Andrii
Thanks for your comments :-)
On 2020/5/28 下午2:36, Andrii Nakryiko wrote:
[snip]
Quoted 17 lines
Quoted 1 line
---
I haven't looked through implementation thoroughly yet. But I have few general remarks.
This looks like a useful and generic tool. I think it will get most attention and be most useful if it will be part of BCC tools. There is already a set of generic tools that use libbpf and CO-RE, see [0]. It feels like this belongs there.
Some of the annoying parts (e.g., syscall name translation) is already generalized as part of syscount tool PR (to be hopefully merged soon), so you'll be able to save quite a lot of code with this. There is also a common build infra that takes care of things like
vmlinux.h, which would provide definitions for all thosexxx_argsstructs that you had to manually define.
Quoted 5 lines
With CO-RE, it also will allow to compile this tool once and run it on many different kernels without recompilation. Please do take a look and submit a PR there, it will be a good addition to the toolkit (and will force you write a bit of README explaining use of this tool as well ;).
Aha, I used to think bcc only support python and cpp :-P
I'll try to rework it and submit PR, I'm glad to know that you think this tool as a helpful one, we do solved some tough issue with it already.
Quoted 6 lines
As for the code itself, I haven't gone through it much, but please convert map definition syntax to BTF-defined one. The one you are using is a legacy one. Thanks!
Will check the example there :-)
Regards,
Michael Wang
Quoted 13 lines
Quoted 10 lines
samples/bpf/Makefile | 3 + samples/bpf/task_detector.h | 382 +++++++++++++++++++++++++++++++++++++++ samples/bpf/task_detector_kern.c | 329 +++++++++++++++++++++++++++++++++ samples/bpf/task_detector_user.c | 314 ++++++++++++++++++++++++++++++++ 4 files changed, 1028 insertions(+) create mode 100644 samples/bpf/task_detector.h create mode 100644 samples/bpf/task_detector_kern.c create mode 100644 samples/bpf/task_detector_user.c[...]
Quoted 24 lines · 王贇
Hi, Andrii
Thanks for your comments :-)
On 2020/5/28 下午2:36, Andrii Nakryiko wrote:
[snip]Quoted 17 lines
Quoted 1 line
---
I haven't looked through implementation thoroughly yet. But I have few general remarks.
This looks like a useful and generic tool. I think it will get most attention and be most useful if it will be part of BCC tools. There is already a set of generic tools that use libbpf and CO-RE, see [0]. It feels like this belongs there.
Some of the annoying parts (e.g., syscall name translation) is already generalized as part of syscount tool PR (to be hopefully merged soon), so you'll be able to save quite a lot of code with this. There is also a common build infra that takes care of things like
vmlinux.h, which would provide definitions for all thosexxx_argsstructs that you had to manually define.
Quoted 8 lines
Quoted 5 lines
With CO-RE, it also will allow to compile this tool once and run it on many different kernels without recompilation. Please do take a look and submit a PR there, it will be a good addition to the toolkit (and will force you write a bit of README explaining use of this tool as well ;).
Aha, I used to think bcc only support python and cpp :-P
libbpf-tools don't use BCC at all, they are just co-located with BCC and BCC tools in the same repository and are lightweight alternatives to BCC-based tools. But it needs kernel with BTF built-in, which is the only (temporary) downside.
Quoted 29 lines
I'll try to rework it and submit PR, I'm glad to know that you think this tool as a helpful one, we do solved some tough issue with it already.
Quoted 6 lines
As for the code itself, I haven't gone through it much, but please convert map definition syntax to BTF-defined one. The one you are using is a legacy one. Thanks!
Will check the example there :-)
Regards,
Michael WangQuoted 13 lines
Quoted 10 lines
samples/bpf/Makefile | 3 + samples/bpf/task_detector.h | 382 +++++++++++++++++++++++++++++++++++++++ samples/bpf/task_detector_kern.c | 329 +++++++++++++++++++++++++++++++++ samples/bpf/task_detector_user.c | 314 ++++++++++++++++++++++++++++++++ 4 files changed, 1028 insertions(+) create mode 100644 samples/bpf/task_detector.h create mode 100644 samples/bpf/task_detector_kern.c create mode 100644 samples/bpf/task_detector_user.c[...]
On 2020/5/29 上午2:34, Andrii Nakryiko wrote:
[snip]
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
With CO-RE, it also will allow to compile this tool once and run it on many different kernels without recompilation. Please do take a look and submit a PR there, it will be a good addition to the toolkit (and will force you write a bit of README explaining use of this tool as well ;).
Quoted 7 lines
Quoted 2 lines
Aha, I used to think bcc only support python and cpp :-P
libbpf-tools don't use BCC at all, they are just co-located with BCC and BCC tools in the same repository and are lightweight alternatives to BCC-based tools. But it needs kernel with BTF built-in, which is the only (temporary) downside.
I see, thanks for the explain :-)
We prefer libbpf since we feel it's more friendly for kernel folks to do develop, less suspects when meet any BUG, while BCC is a good way to package and distribute, glad to know that we can have them in one piece now.
Regards,
Michael Wang
Quoted 30 lines
Quoted 30 lines
I'll try to rework it and submit PR, I'm glad to know that you think this tool as a helpful one, we do solved some tough issue with it already.
Quoted 6 lines
As for the code itself, I haven't gone through it much, but please convert map definition syntax to BTF-defined one. The one you are using is a legacy one. Thanks!
Will check the example there :-)
Regards,
Michael WangQuoted 13 lines
Quoted 10 lines
samples/bpf/Makefile | 3 + samples/bpf/task_detector.h | 382 +++++++++++++++++++++++++++++++++++++++ samples/bpf/task_detector_kern.c | 329 +++++++++++++++++++++++++++++++++ samples/bpf/task_detector_user.c | 314 ++++++++++++++++++++++++++++++++ 4 files changed, 1028 insertions(+) create mode 100644 samples/bpf/task_detector.h create mode 100644 samples/bpf/task_detector_kern.c create mode 100644 samples/bpf/task_detector_user.c[...]