Skip to contentSkip to conversation

samples:bpf: introduce task detector

RFC PATCH In review 王贇 2 participants 5 messages May 28, 2020 – May 29, 2020

Subsystems (3)
lore
Share & tools
lore

王贇

author
#0 lore raw

RFC PATCH 1/1 samples:bpf: introduce task detector

This is a tool to trace the related schedule events of a specified task, eg the migration, sched in/out, wakeup and sleep/block.

The event was translated into sentence to be more readable, by execute command 'task_detector -p 49870' we continually tracing the schedule events related to 'top' like:

----------------------------

923455517688  CPU=23  PID=49870  COMM=top          ENQUEUE
923455519633  CPU=23  PID=0      COMM=IDLE         PREEMPTED                1945ns
923455519868  CPU=23  PID=49870  COMM=top          EXECUTE AFTER WAITED     2180ns
923468279019  CPU=23  PID=49870  COMM=top          WAIT AFTER EXECUTED      12ms
923468279220  CPU=23  PID=128    COMM=ksoftirqd/23 PREEMPT
923468283051  CPU=23  PID=128    COMM=ksoftirqd/23 DEQUEUE AFTER PREEMPTED  3831ns
923468283216  CPU=23  PID=49870  COMM=top          EXECUTE AFTER WAITED     4197ns
923476280180  CPU=23  PID=49870  COMM=top          WAIT AFTER EXECUTED      7996us
923476280350  CPU=23  PID=128    COMM=ksoftirqd/23 PREEMPT
923476322029  CPU=23  PID=128    COMM=ksoftirqd/23 DEQUEUE AFTER PREEMPTED  41us
923476322150  CPU=23  PID=49870  COMM=top          EXECUTE AFTER WAITED     41us
923479726879  CPU=23  PID=49870  COMM=top          DEQUEUE AFTER EXECUTED   3404us

----------------------------

This could be helpful on debugging the competition on CPU resource, to find out who has stolen the CPU and how much it stolen.

It can also tracing the syscall by append option -s.

---
 samples/bpf/Makefile             |   3 +
 samples/bpf/task_detector.h      | 382 +++++++++++++++++++++++++++++++++++++++
 samples/bpf/task_detector_kern.c | 329 +++++++++++++++++++++++++++++++++
 samples/bpf/task_detector_user.c | 314 ++++++++++++++++++++++++++++++++
 4 files changed, 1028 insertions(+)
 create mode 100644 samples/bpf/task_detector.h
 create mode 100644 samples/bpf/task_detector_kern.c
 create mode 100644 samples/bpf/task_detector_user.c
samples/bpf/Makefile +3 -0
index 8403e4762306..a8e468e0eac9 100644
@@ -53,6 +53,7 @@ tprogs-y += task_fd_query
 tprogs-y += xdp_sample_pkts
 tprogs-y += ibumad
 tprogs-y += hbm
+tprogs-y += task_detector
 
 # Libbpf dependencies
 LIBBPF = $(TOOLS_PATH)/lib/bpf/libbpf.a
@@ -109,6 +110,7 @@ task_fd_query-objs := bpf_load.o task_fd_query_user.o $(TRACE_HELPERS)
 xdp_sample_pkts-objs := xdp_sample_pkts_user.o $(TRACE_HELPERS)
 ibumad-objs := bpf_load.o ibumad_user.o $(TRACE_HELPERS)
 hbm-objs := bpf_load.o hbm.o $(CGROUP_HELPERS)
+task_detector-objs := bpf_load.o task_detector_user.o $(TRACE_HELPERS)
 
 # Tell kbuild to always build the programs
 always-y := $(tprogs-y)
@@ -170,6 +172,7 @@ always-y += ibumad_kern.o
 always-y += hbm_out_kern.o
 always-y += hbm_edt_kern.o
 always-y += xdpsock_kern.o
+always-y += task_detector_kern.o
 
 ifeq ($(ARCH), arm)
 # Strip all except -D__LINUX_ARM_ARCH__ option needed to handle linux
samples/bpf/task_detector.h new file +382 -0

382 lines not on this page · raw message

samples/bpf/task_detector_kern.c new file +329 -0

329 lines not on this page · raw message

samples/bpf/task_detector_user.c new file +314 -0

314 lines not on this page · raw message

Signature · 1 line
2.14.4.44.g2045bb6
#1 lore raw
Quoted 24 lines · 王贇

This is a tool to trace the related schedule events of a specified task, eg the migration, sched in/out, wakeup and sleep/block.

The event was translated into sentence to be more readable, by execute command 'task_detector -p 49870' we continually tracing the schedule events related to 'top' like:

----------------------------

923455517688  CPU=23  PID=49870  COMM=top          ENQUEUE
923455519633  CPU=23  PID=0      COMM=IDLE         PREEMPTED                1945ns
923455519868  CPU=23  PID=49870  COMM=top          EXECUTE AFTER WAITED     2180ns
923468279019  CPU=23  PID=49870  COMM=top          WAIT AFTER EXECUTED      12ms
923468279220  CPU=23  PID=128    COMM=ksoftirqd/23 PREEMPT
923468283051  CPU=23  PID=128    COMM=ksoftirqd/23 DEQUEUE AFTER PREEMPTED  3831ns
923468283216  CPU=23  PID=49870  COMM=top          EXECUTE AFTER WAITED     4197ns
923476280180  CPU=23  PID=49870  COMM=top          WAIT AFTER EXECUTED      7996us
923476280350  CPU=23  PID=128    COMM=ksoftirqd/23 PREEMPT
923476322029  CPU=23  PID=128    COMM=ksoftirqd/23 DEQUEUE AFTER PREEMPTED  41us
923476322150  CPU=23  PID=49870  COMM=top          EXECUTE AFTER WAITED     41us
923479726879  CPU=23  PID=49870  COMM=top          DEQUEUE AFTER EXECUTED   3404us

----------------------------

Quoted 8 lines

This could be helpful on debugging the competition on CPU resource, to find out who has stolen the CPU and how much it stolen.

It can also tracing the syscall by append option -s.

---

I haven't looked through implementation thoroughly yet. But I have few general remarks.

This looks like a useful and generic tool. I think it will get most attention and be most useful if it will be part of BCC tools. There is already a set of generic tools that use libbpf and CO-RE, see [0]. It feels like this belongs there.

Some of the annoying parts (e.g., syscall name translation) is already generalized as part of syscount tool PR (to be hopefully merged soon), so you'll be able to save quite a lot of code with this. There is also a common build infra that takes care of things like vmlinux.h, which would provide definitions for all those xxx_args structs that you had to manually define.

With CO-RE, it also will allow to compile this tool once and run it on many different kernels without recompilation. Please do take a look and submit a PR there, it will be a good addition to the toolkit (and will force you write a bit of README explaining use of this tool as well ;).

As for the code itself, I haven't gone through it much, but please convert map definition syntax to BTF-defined one. The one you are using is a legacy one. Thanks!

[0] https://github.com/iovisor/bcc/tree/master/libbpf-tools

Quoted 9 lines
 samples/bpf/Makefile             |   3 +
 samples/bpf/task_detector.h      | 382 +++++++++++++++++++++++++++++++++++++++
 samples/bpf/task_detector_kern.c | 329 +++++++++++++++++++++++++++++++++
 samples/bpf/task_detector_user.c | 314 ++++++++++++++++++++++++++++++++
 4 files changed, 1028 insertions(+)
 create mode 100644 samples/bpf/task_detector.h
 create mode 100644 samples/bpf/task_detector_kern.c
 create mode 100644 samples/bpf/task_detector_user.c

[...]

author
#2 lore raw

Hi, Andrii

Thanks for your comments :-)

On 2020/5/28 下午2:36, Andrii Nakryiko wrote:
[snip]

Quoted 17 lines
Quoted 1 line

---

I haven't looked through implementation thoroughly yet. But I have few general remarks.

This looks like a useful and generic tool. I think it will get most attention and be most useful if it will be part of BCC tools. There is already a set of generic tools that use libbpf and CO-RE, see [0]. It feels like this belongs there.

Some of the annoying parts (e.g., syscall name translation) is already generalized as part of syscount tool PR (to be hopefully merged soon), so you'll be able to save quite a lot of code with this. There is also a common build infra that takes care of things like vmlinux.h, which would provide definitions for all those xxx_args structs that you had to manually define.

Quoted 5 lines

With CO-RE, it also will allow to compile this tool once and run it on many different kernels without recompilation. Please do take a look and submit a PR there, it will be a good addition to the toolkit (and will force you write a bit of README explaining use of this tool as well ;).

Aha, I used to think bcc only support python and cpp :-P

I'll try to rework it and submit PR, I'm glad to know that you think this tool as a helpful one, we do solved some tough issue with it already.

Quoted 6 lines

As for the code itself, I haven't gone through it much, but please convert map definition syntax to BTF-defined one. The one you are using is a legacy one. Thanks!

[0] https://github.com/iovisor/bcc/tree/master/libbpf-tools

Will check the example there :-)

Regards,
Michael Wang

Quoted 13 lines
Quoted 10 lines
 samples/bpf/Makefile             |   3 +
 samples/bpf/task_detector.h      | 382 +++++++++++++++++++++++++++++++++++++++
 samples/bpf/task_detector_kern.c | 329 +++++++++++++++++++++++++++++++++
 samples/bpf/task_detector_user.c | 314 ++++++++++++++++++++++++++++++++
 4 files changed, 1028 insertions(+)
 create mode 100644 samples/bpf/task_detector.h
 create mode 100644 samples/bpf/task_detector_kern.c
 create mode 100644 samples/bpf/task_detector_user.c

[...]

#3 lore raw
Quoted 24 lines · 王贇

Hi, Andrii

Thanks for your comments :-)

On 2020/5/28 下午2:36, Andrii Nakryiko wrote:
[snip]

Quoted 17 lines
Quoted 1 line

---

I haven't looked through implementation thoroughly yet. But I have few general remarks.

This looks like a useful and generic tool. I think it will get most attention and be most useful if it will be part of BCC tools. There is already a set of generic tools that use libbpf and CO-RE, see [0]. It feels like this belongs there.

Some of the annoying parts (e.g., syscall name translation) is already generalized as part of syscount tool PR (to be hopefully merged soon), so you'll be able to save quite a lot of code with this. There is also a common build infra that takes care of things like vmlinux.h, which would provide definitions for all those xxx_args structs that you had to manually define.

Quoted 8 lines
Quoted 5 lines

With CO-RE, it also will allow to compile this tool once and run it on many different kernels without recompilation. Please do take a look and submit a PR there, it will be a good addition to the toolkit (and will force you write a bit of README explaining use of this tool as well ;).

Aha, I used to think bcc only support python and cpp :-P

libbpf-tools don't use BCC at all, they are just co-located with BCC and BCC tools in the same repository and are lightweight alternatives to BCC-based tools. But it needs kernel with BTF built-in, which is the only (temporary) downside.

Quoted 29 lines

I'll try to rework it and submit PR, I'm glad to know that you think this tool as a helpful one, we do solved some tough issue with it already.

Quoted 6 lines

As for the code itself, I haven't gone through it much, but please convert map definition syntax to BTF-defined one. The one you are using is a legacy one. Thanks!

[0] https://github.com/iovisor/bcc/tree/master/libbpf-tools

Will check the example there :-)

Regards,
Michael Wang

Quoted 13 lines
Quoted 10 lines
 samples/bpf/Makefile             |   3 +
 samples/bpf/task_detector.h      | 382 +++++++++++++++++++++++++++++++++++++++
 samples/bpf/task_detector_kern.c | 329 +++++++++++++++++++++++++++++++++
 samples/bpf/task_detector_user.c | 314 ++++++++++++++++++++++++++++++++
 4 files changed, 1028 insertions(+)
 create mode 100644 samples/bpf/task_detector.h
 create mode 100644 samples/bpf/task_detector_kern.c
 create mode 100644 samples/bpf/task_detector_user.c

[...]

author
#4 lore raw

On 2020/5/29 上午2:34, Andrii Nakryiko wrote:
[snip]

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

With CO-RE, it also will allow to compile this tool once and run it on many different kernels without recompilation. Please do take a look and submit a PR there, it will be a good addition to the toolkit (and will force you write a bit of README explaining use of this tool as well ;).

Quoted 7 lines
Quoted 2 lines

Aha, I used to think bcc only support python and cpp :-P

libbpf-tools don't use BCC at all, they are just co-located with BCC and BCC tools in the same repository and are lightweight alternatives to BCC-based tools. But it needs kernel with BTF built-in, which is the only (temporary) downside.

I see, thanks for the explain :-)

We prefer libbpf since we feel it's more friendly for kernel folks to do develop, less suspects when meet any BUG, while BCC is a good way to package and distribute, glad to know that we can have them in one piece now.

Regards,
Michael Wang

Quoted 30 lines
Quoted 30 lines

I'll try to rework it and submit PR, I'm glad to know that you think this tool as a helpful one, we do solved some tough issue with it already.

Quoted 6 lines

As for the code itself, I haven't gone through it much, but please convert map definition syntax to BTF-defined one. The one you are using is a legacy one. Thanks!

[0] https://github.com/iovisor/bcc/tree/master/libbpf-tools

Will check the example there :-)

Regards,
Michael Wang

Quoted 13 lines
Quoted 10 lines
 samples/bpf/Makefile             |   3 +
 samples/bpf/task_detector.h      | 382 +++++++++++++++++++++++++++++++++++++++
 samples/bpf/task_detector_kern.c | 329 +++++++++++++++++++++++++++++++++
 samples/bpf/task_detector_user.c | 314 ++++++++++++++++++++++++++++++++
 4 files changed, 1028 insertions(+)
 create mode 100644 samples/bpf/task_detector.h
 create mode 100644 samples/bpf/task_detector_kern.c
 create mode 100644 samples/bpf/task_detector_user.c

[...]