perf, bpf: add support for bpf in sw/hw perf_events
Subsystems (8)
v3 cover perf, bpf: add support for bpf in sw/hw perf_events
Hi Peter, Dave,
this patch set is a follow up to the discussion:
https://lkml.kernel.org/r/20160804142853.GO6862%20()%20twins%20!%20programming%20!%20kicks-ass%20!%20net
It turned out to be simpler than what we discussed.
Patches 1-3 is bpf-side prep for the main patch 4
that adds bpf program as an overflow_handler to sw and hw perf_events.
Patches 5 and 6 are examples from myself and Brendan.
Peter,
to implement your suggestion to add ifdef CONFIG_BPF_SYSCALL inside struct perf_event, I had to shuffle ifdefs in events/core.c Please double check whether that is what you wanted to see.
v2->v3: fixed few more minor issuesv1->v2: fixed issues spotted by Peter and Daniel.
Thanks!
Alexei Starovoitov (5):
bpf: support 8-byte metafield access
bpf: introduce BPF_PROG_TYPE_PERF_EVENT program type
bpf: perf_event progs should only use preallocated maps perf, bpf: add perf events core support for BPF_PROG_TYPE_PERF_EVENT
programs
samples/bpf: add perf_event+bpf example
Brendan Gregg (1):
samples/bpf: add sampleip example
include/linux/bpf.h | 4 + include/linux/perf_event.h | 9 ++ include/uapi/linux/Kbuild | 1 + include/uapi/linux/bpf.h | 1 + include/uapi/linux/bpf_perf_event.h | 18 +++ kernel/bpf/verifier.c | 31 +++++- kernel/events/core.c | 89 ++++++++++++++- kernel/trace/bpf_trace.c | 61 +++++++++++ samples/bpf/Makefile | 8 ++ samples/bpf/bpf_helpers.h | 2 + samples/bpf/bpf_load.c | 7 +- samples/bpf/sampleip_kern.c | 38 +++++++ samples/bpf/sampleip_user.c | 196 +++++++++++++++++++++++++++++++++ samples/bpf/trace_event_kern.c | 65 +++++++++++ samples/bpf/trace_event_user.c | 213 ++++++++++++++++++++++++++++++++++++ 15 files changed, 737 insertions(+), 6 deletions(-) create mode 100644 include/uapi/linux/bpf_perf_event.h create mode 100644 samples/bpf/sampleip_kern.c create mode 100644 samples/bpf/sampleip_user.c create mode 100644 samples/bpf/trace_event_kern.c create mode 100644 samples/bpf/trace_event_user.c
Signature · 1 line
2.8.0
PATCH v3 1/6 bpf: support 8-byte metafield access
The verifier supported only 4-byte metafields in
struct __sk_buff and struct xdp_md. The metafields in upcoming struct bpf_perf_event are 8-byte to match register width in struct pt_regs. Teach verifier to recognize 8-byte metafield access.
The patch doesn't affect safety of sockets and xdp programs. They check for 4-byte only ctx access before these conditions are hit.
--- kernel/bpf/verifier.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
kernel/bpf/verifier.c +6 -3
@@ -2333,7 +2333,8 @@ static int do_check(struct verifier_env *env)if (err)return err; -if (BPF_SIZE (insn->code) !=BPF_W ) { +if (BPF_SIZE (insn->code) !=BPF_W && +BPF_SIZE (insn->code) !=BPF_DW ) { insn_idx++;continue ; }
@@ -2642,9 +2643,11 @@ static int convert_ctx_accesses(struct verifier_env *env)for (i =0 ; i < insn_cnt; i++, insn++) {u32 insn_delta, cnt; -if (insn->code == (BPF_LDX |BPF_MEM |BPF_W )) +if (insn->code == (BPF_LDX |BPF_MEM |BPF_W ) || + insn->code == (BPF_LDX |BPF_MEM |BPF_DW )) type =BPF_READ ; -else if (insn->code == (BPF_STX |BPF_MEM |BPF_W )) +else if (insn->code == (BPF_STX |BPF_MEM |BPF_W ) || + insn->code == (BPF_STX |BPF_MEM |BPF_DW )) type =BPF_WRITE ;else continue ;
Signature · 1 line
2.8.0
PATCH v3 2/6 bpf: introduce BPF_PROG_TYPE_PERF_EVENT program type
In reply to Alexei StarovoitovIntroduce BPF_PROG_TYPE_PERF_EVENT programs that can be attached to HW and SW perf events (PERF_TYPE_HARDWARE and PERF_TYPE_SOFTWARE correspondingly in uapi/linux/perf_event.h)
The program visible context meta structure is
struct bpf_perf_event_data {
struct pt_regs regs;
__u64 sample_period;
};which is accessible directly from the program:
int bpf_prog(struct bpf_perf_event_data *ctx)
{
... ctx->sample_period ...
... ctx->regs.ip ...
}
The bpf verifier rewrites the accesses into kernel internal struct bpf_perf_event_data_kern which allows changing struct perf_sample_data without affecting bpf programs. New fields can be added to the end of struct bpf_perf_event_data in the future.
--- include/linux/perf_event.h | 5 +++ include/uapi/linux/Kbuild | 1 + include/uapi/linux/bpf.h | 1 + include/uapi/linux/bpf_perf_event.h | 18 +++++++++++ kernel/trace/bpf_trace.c | 61 +++++++++++++++++++++++++++++++++++++ 5 files changed, 86 insertions(+) create mode 100644 include/uapi/linux/bpf_perf_event.h
include/linux/perf_event.h +5 -0
@@ -788,6 +788,11 @@ struct perf_output_handle {int page; }; +struct bpf_perf_event_data_kern { +struct pt_regs *regs; +struct perf_sample_data *data; +}; +#ifdef CONFIG_CGROUP_PERF /*
include/uapi/linux/Kbuild +1 -0
@@ -71,6 +71,7 @@ header-y += binfmts.h header-y += blkpg.h header-y += blktrace_api.h header-y += bpf_common.h +header-y += bpf_perf_event.h header-y += bpf.h header-y += bpqether.h header-y += bsg.h
include/uapi/linux/bpf.h +1 -0
@@ -95,6 +95,7 @@ enum bpf_prog_type {BPF_PROG_TYPE_SCHED_ACT ,BPF_PROG_TYPE_TRACEPOINT ,BPF_PROG_TYPE_XDP , + BPF_PROG_TYPE_PERF_EVENT, };#define BPF_PSEUDO_MAP_FD 1
include/uapi/linux/bpf_perf_event.h new file +18 -0
@@ -0,0 +1,18 @@ +/* Copyright (c) 2016 Facebook +* +* This program is free software; you can redistribute it and/or +* modify it under the terms of version 2 of the GNU General Public +* License as published by the Free Software Foundation. +*/ +#ifndef _UAPI__LINUX_BPF_PERF_EVENT_H__ +#define _UAPI__LINUX_BPF_PERF_EVENT_H__ + +#include <linux/types.h> +#include <linux/ptrace.h> + +struct bpf_perf_event_data { +struct pt_regs regs; +__u64 sample_period; +}; + +#endif /* _UAPI__LINUX_BPF_PERF_EVENT_H__ */
kernel/trace/bpf_trace.c +61 -0
81 lines not on this page · raw message
Signature · 1 line
2.8.0
PATCH v3 3/6 bpf: perf_event progs should only use preallocated maps
In reply to Alexei StarovoitovMake sure that BPF_PROG_TYPE_PERF_EVENT programs only use preallocated hash maps, since doing memory allocation
in overflow_handler can crash depending on where nmi got triggered.
--- kernel/bpf/verifier.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)
kernel/bpf/verifier.c +21 -1
@@ -2511,6 +2511,20 @@ static int do_check(struct verifier_env *env)return 0 ; } +static int check_map_prog_compatibility (struct bpf_map *map, +struct bpf_prog *prog) + +{ +if (prog->type ==BPF_PROG_TYPE_PERF_EVENT && + (map->map_type ==BPF_MAP_TYPE_HASH || + map->map_type ==BPF_MAP_TYPE_PERCPU_HASH ) && + (map->map_flags &BPF_F_NO_PREALLOC )) { +verbose ("perf_event programs can only use preallocated hash map\n" ); +return -EINVAL ; + } +return 0 ; +} +/* look for pseudo eBPF instructions that access map FDs and * replace them with actual map pointers */
@@ -2518,7 +2532,7 @@ static int replace_map_fd_with_map_ptr(struct verifier_env *env) {struct bpf_insn *insn = env->prog->insnsi;int insn_cnt = env->prog->len; -int i, j; +int i, j, err;for (i =0 ; i < insn_cnt; i++, insn++) { if (BPF_CLASS(insn->code) == BPF_LDX &&
@@ -2562,6 +2576,12 @@ static int replace_map_fd_with_map_ptr(struct verifier_env *env)return PTR_ERR (map); } + err =check_map_prog_compatibility (map, env->prog); +if (err) { +fdput (f); +return err; + } +/* store map pointer inside BPF_LD_IMM64 instruction */ insn[0 ].imm = (u32 ) (unsigned long ) map; insn[1 ].imm = ((u64 ) (unsigned long ) map) >>32 ;
Signature · 1 line
2.8.0
PATCH v3 4/6 perf, bpf: add perf events core support for BPF_PROG_TYPE_PERF_EVENT programs
In reply to Alexei StarovoitovAllow attaching BPF_PROG_TYPE_PERF_EVENT programs to sw and hw perf events via overflow_handler mechanism.
When program is attached the overflow_handlers become stacked.
The program acts as a filter.
Returning zero from the program means that the normal perf_event_output handler will not be called and sampling event won't be stored in the ring buffer.
The overflow_handler_context==NULL is an additional safety check to make sure programs are not attached to hw breakpoints and watchdog in case other checks (that prevent that now anyway) get accidentally relaxed in the future.
The program refcnt is incremented in case perf_events are inhereted when target task is forked.
Similar to kprobe and tracepoint programs there is no ioctl to detach the program or swap already attached program. The user space expected to close(perf_event_fd) like it does right now for kprobe+bpf. That restriction simplifies the code quite a bit.
The invocation of overflow_handler in __perf_event_overflow() is now done via READ_ONCE, since that pointer can be replaced when the program is attached while perf_event itself could have been active already. There is no need to do similar treatment for event->prog, since it's assigned only once before it's accessed.
--- include/linux/bpf.h | 4 +++ include/linux/perf_event.h | 4 +++ kernel/events/core.c | 89 +++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 96 insertions(+), 1 deletion(-)
include/linux/bpf.h +4 -0
@@ -297,6 +297,10 @@ static inline struct bpf_prog *bpf_prog_add(struct bpf_prog *prog, int i)static inline void bpf_prog_put (struct bpf_prog *prog) { } +static inline struct bpf_prog *bpf_prog_inc (struct bpf_prog *prog) +{ +return ERR_PTR (-EOPNOTSUPP ); +}#endif /* CONFIG_BPF_SYSCALL */ /* verifier prototypes for helper functions called from eBPF programs */
include/linux/perf_event.h +4 -0
@@ -679,6 +679,10 @@ struct perf_event {u64 (*clock)(void);perf_overflow_handler_t overflow_handler;void *overflow_handler_context; +#ifdef CONFIG_BPF_SYSCALL +perf_overflow_handler_t orig_overflow_handler; +struct bpf_prog *prog; +#endif #ifdef CONFIG_EVENT_TRACING struct trace_event_call *tp_event;
kernel/events/core.c +88 -1
118 lines not on this page · raw message
Signature · 1 line
2.8.0
PATCH v3 5/6 samples/bpf: add perf_event+bpf example
In reply to Alexei StarovoitovThe bpf program is called 50 times a second and does hashmap[kern&user_stackid]++ It's primary purpose to check that key bpf helpers like map lookup, update, get_stackid, trace_printk and ctx access are all working.
It checks:
- PERF_COUNT_HW_CPU_CYCLES on all cpus
- PERF_COUNT_HW_CPU_CYCLES for current process and inherited perf_events to children
- PERF_COUNT_SW_CPU_CLOCK on all cpus
- PERF_COUNT_SW_CPU_CLOCK for current process
--- samples/bpf/Makefile | 4 + samples/bpf/bpf_helpers.h | 2 + samples/bpf/bpf_load.c | 7 +- samples/bpf/trace_event_kern.c | 65 +++++++++++++ samples/bpf/trace_event_user.c | 213 +++++++++++++++++++++++++++++++++++++++++ 5 files changed, 290 insertions(+), 1 deletion(-) create mode 100644 samples/bpf/trace_event_kern.c create mode 100644 samples/bpf/trace_event_user.c
samples/bpf/Makefile +4 -0
@@ -25,6 +25,7 @@ hostprogs-y += test_cgrp2_array_pin hostprogs-y += xdp1 hostprogs-y += xdp2 hostprogs-y += test_current_task_under_cgroup +hostprogs-y += trace_event test_verifier-objs := test_verifier.o libbpf.o test_maps-objs := test_maps.o libbpf.o
@@ -52,6 +53,7 @@ xdp1-objs := bpf_load.o libbpf.o xdp1_user.o xdp2-objs := bpf_load.o libbpf.o xdp1_user.o test_current_task_under_cgroup-objs := bpf_load.o libbpf.o \ test_current_task_under_cgroup_user.o +trace_event-objs := bpf_load.o libbpf.o trace_event_user.o # Tell kbuild to always build the programs always := $(hostprogs-y)
@@ -79,6 +81,7 @@ always += test_cgrp2_tc_kern.o always += xdp1_kern.o always += xdp2_kern.o always += test_current_task_under_cgroup_kern.o +always += trace_event_kern.o HOSTCFLAGS += -I$(objtree)/usr/include
@@ -103,6 +106,7 @@ HOSTLOADLIBES_test_overhead += -lelf -lrt HOSTLOADLIBES_xdp1 += -lelf HOSTLOADLIBES_xdp2 += -lelf HOSTLOADLIBES_test_current_task_under_cgroup += -lelf +HOSTLOADLIBES_trace_event += -lelf # Allows pointing LLC/CLANG to a LLVM backend with bpf support, redefine on cmdline: # make samples/bpf/ LLC=~/git/llvm/build/bin/llc CLANG=~/git/llvm/build/bin/clang
samples/bpf/bpf_helpers.h +2 -0
@@ -55,6 +55,8 @@ static int (*bpf_skb_get_tunnel_opt)(void *ctx, void *md, int size) = (void *) BPF_FUNC_skb_get_tunnel_opt;static int (*bpf_skb_set_tunnel_opt)(void *ctx,void *md,int size) = (void *) BPF_FUNC_skb_set_tunnel_opt; +static unsigned long long (*bpf_get_prandom_u32)(void ) = + (void *) BPF_FUNC_get_prandom_u32;/* llvm builtin functions that eBPF C program may use to * emit BPF_LD_ABS and BPF_LD_IND instructions
samples/bpf/bpf_load.c +6 -1
@@ -51,6 +51,7 @@ static int load_and_attach(const char *event, struct bpf_insn *prog, int size)bool is_kretprobe =strncmp (event,"kretprobe/" ,10 ) ==0 ;bool is_tracepoint =strncmp (event,"tracepoint/" ,11 ) ==0 ;bool is_xdp =strncmp (event,"xdp" ,3 ) ==0 ; +bool is_perf_event =strncmp (event,"perf_event" ,10 ) ==0 ;enum bpf_prog_type prog_type;char buf[256 ];int fd, efd, err, id;
@@ -69,6 +70,8 @@ static int load_and_attach(const char *event, struct bpf_insn *prog, int size) prog_type =BPF_PROG_TYPE_TRACEPOINT ; }else if (is_xdp) { prog_type =BPF_PROG_TYPE_XDP ; + }else if (is_perf_event) { + prog_type =BPF_PROG_TYPE_PERF_EVENT ; }else {printf ("Unknown event '%s'\n" , event);return -1 ;
@@ -82,7 +85,7 @@ static int load_and_attach(const char *event, struct bpf_insn *prog, int size) prog_fd[prog_cnt++] = fd; -if (is_xdp) +if (is_xdp || is_perf_event)return 0 ;if (is_socket) {
@@ -326,6 +329,7 @@ int load_bpf_file(char *path) memcmp(shname_prog, "kretprobe/", 10) == 0 || memcmp(shname_prog, "tracepoint/", 11) == 0 || memcmp(shname_prog, "xdp", 3) == 0 || + memcmp(shname_prog, "perf_event", 10) == 0 || memcmp(shname_prog, "socket", 6) == 0)load_and_attach (shname_prog, insns, data_prog->d_size); }
@@ -344,6 +348,7 @@ int load_bpf_file(char *path) memcmp(shname, "kretprobe/", 10) == 0 || memcmp(shname, "tracepoint/", 11) == 0 || memcmp(shname, "xdp", 3) == 0 || + memcmp(shname, "perf_event", 10) == 0 || memcmp(shname, "socket", 6) == 0)load_and_attach (shname, data->d_buf, data->d_size); }
samples/bpf/trace_event_kern.c new file +65 -0
65 lines not on this page · raw message
samples/bpf/trace_event_user.c new file +213 -0
213 lines not on this page · raw message
Signature · 1 line
2.8.0
PATCH v3 6/6 samples/bpf: add sampleip example
In reply to Alexei StarovoitovFrom: Brendan Gregg <[email protected]>
sample instruction pointer and frequency count in a BPF map
--- samples/bpf/Makefile | 4 + samples/bpf/sampleip_kern.c | 38 +++++++++ samples/bpf/sampleip_user.c | 196 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 238 insertions(+) create mode 100644 samples/bpf/sampleip_kern.c create mode 100644 samples/bpf/sampleip_user.c
samples/bpf/Makefile +4 -0
@@ -26,6 +26,7 @@ hostprogs-y += xdp1 hostprogs-y += xdp2 hostprogs-y += test_current_task_under_cgroup hostprogs-y += trace_event +hostprogs-y += sampleip test_verifier-objs := test_verifier.o libbpf.o test_maps-objs := test_maps.o libbpf.o
@@ -54,6 +55,7 @@ xdp2-objs := bpf_load.o libbpf.o xdp1_user.o test_current_task_under_cgroup-objs := bpf_load.o libbpf.o \ test_current_task_under_cgroup_user.o trace_event-objs := bpf_load.o libbpf.o trace_event_user.o +sampleip-objs := bpf_load.o libbpf.o sampleip_user.o # Tell kbuild to always build the programs always := $(hostprogs-y)
@@ -82,6 +84,7 @@ always += xdp1_kern.o always += xdp2_kern.o always += test_current_task_under_cgroup_kern.o always += trace_event_kern.o +always += sampleip_kern.o HOSTCFLAGS += -I$(objtree)/usr/include
@@ -107,6 +110,7 @@ HOSTLOADLIBES_xdp1 += -lelf HOSTLOADLIBES_xdp2 += -lelf HOSTLOADLIBES_test_current_task_under_cgroup += -lelf HOSTLOADLIBES_trace_event += -lelf +HOSTLOADLIBES_sampleip += -lelf # Allows pointing LLC/CLANG to a LLVM backend with bpf support, redefine on cmdline: # make samples/bpf/ LLC=~/git/llvm/build/bin/llc CLANG=~/git/llvm/build/bin/clang
samples/bpf/sampleip_kern.c new file +38 -0
@@ -0,0 +1,38 @@ +/* Copyright 2016 Netflix, Inc. +* +* This program is free software; you can redistribute it and/or +* modify it under the terms of version 2 of the GNU General Public +* License as published by the Free Software Foundation. +*/ +#include <linux/version.h> +#include <linux/ptrace.h> +#include <uapi/linux/bpf.h> +#include <uapi/linux/bpf_perf_event.h> +#include "bpf_helpers.h" + +#define MAX_IPS 8192 + +struct bpf_map_def SEC("maps") ip_map = { + .type = BPF_MAP_TYPE_HASH, + .key_size = sizeof(u64), + .value_size = sizeof(u32), + .max_entries = MAX_IPS, +}; + +SEC("perf_event") +int do_sample (struct bpf_perf_event_data *ctx) +{ +u64 ip; +u32 *value, init_val =1 ; + + ip = ctx->regs.ip; + value =bpf_map_lookup_elem (&ip_map, &ip); +if (value) + *value +=1 ; +else +/* E2BIG not tested for this example only */ +bpf_map_update_elem (&ip_map, &ip, &init_val,BPF_NOEXIST ); + +return 0 ; +} +char _license[] SEC("license") = "GPL";
samples/bpf/sampleip_user.c new file +196 -0
196 lines not on this page · raw message
Signature · 1 line
2.8.0
Quoted 4 lines · Alexei Starovoitov
Peter,
to implement your suggestion to add ifdefCONFIG_BPF_SYSCALLinside structperf_event, I had to shuffle ifdefs inevents/core.cPlease double check whether that is what you wanted to see.
Looks fine, Thanks!
David Miller
maintainerFrom: Alexei Starovoitov <[email protected]>
Date: Thu, 1 Sep 2016 18:37:20 -0700
Quoted 11 lines
Hi Peter, Dave,
this patch set is a follow up to the discussion:
https://lkml.kernel.org/r/20160804142853.GO6862%20()%20twins%20!%20programming%20!%20kicks-ass%20!%20net
It turned out to be simpler than what we discussed.Patches 1-3 is bpf-side prep for the main patch 4
that adds bpf program as anoverflow_handlerto sw and hwperf_events.Patches 5 and 6 are examples from myself and Brendan.
Quoted 7 lines
Peter,
to implement your suggestion to add ifdefCONFIG_BPF_SYSCALLinside structperf_event, I had to shuffle ifdefs inevents/core.cPlease double check whether that is what you wanted to see.
v2->v3: fixed few more minor issuesv1->v2: fixed issues spotted by Peter and Daniel.
Series applied, thanks.