kprobe/bpf: Add support to attach multiple kprobes
Subsystems (16)
cover kprobe/bpf: Add support to attach multiple kprobes
hi,
adding support to attach multiple kprobes within single syscall and speed up attachment of many kprobes.
The previous attempt [1] wasn't fast enough, so coming with new approach that adds new kprobe interface.
The attachment speed of of this approach (tested in bpftrace) is now comparable to ftrace tracer attachment speed.. fast ;-)
The limit of this approach is forced by using ftrace as attach layer, so it allows only kprobes on function's entry (plus return probes).
This patchset contains:
- kprobes support to register multiple kprobes with current kprobe API (patches 1 - 8)
- bpf support ot create new kprobe link allowing to attach multiple addresses (patches 9 - 14)
We don't need to care about multiple probes on same functions because it's taken care on the ftrace_ops layer.
Also available at:
https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
kprobe/multi
thanks,
jirka
[1] https://lore.kernel.org/bpf/[email protected]/
---
Jiri Olsa (13):
ftrace: Add ftrace_set_filter_ips function
kprobe: Keep traced function address
kprobe: Add support to register multiple ftrace kprobes
kprobe: Add support to register multiple ftrace kretprobes
kprobe: Allow to get traced function address for multi ftrace kprobes samples/kprobes: Add support for multi kprobe interface samples/kprobes: Add support for multi kretprobe interface
bpf: Add kprobe link for attaching raw kprobes
libbpf: Add libbpf__kallsyms_parse function
libbpf: Add bpf_link_create support for multi kprobes
libbpf: Add bpf_program__attach_kprobe_opts for multi kprobes selftest/bpf: Add raw kprobe attach test
selftest/bpf: Add bpf_cookie test for raw_k[ret]probe
arch/Kconfig | 3 ++ arch/x86/Kconfig | 1 + arch/x86/kernel/kprobes/ftrace.c | 51 +++++++++++++----- include/linux/bpf_types.h | 1 + include/linux/ftrace.h | 3 ++ include/linux/kprobes.h | 55 ++++++++++++++++++++ include/uapi/linux/bpf.h | 12 +++++ kernel/bpf/syscall.c | 191 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- kernel/kprobes.c | 264 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------- kernel/trace/bpf_trace.c | 7 ++- kernel/trace/ftrace.c | 53 +++++++++++++++---- samples/kprobes/kprobe_example.c | 47 +++++++++++++++-- samples/kprobes/kretprobe_example.c | 43 +++++++++++++++- tools/include/uapi/linux/bpf.h | 12 +++++ tools/lib/bpf/bpf.c | 5 ++ tools/lib/bpf/bpf.h | 7 ++- tools/lib/bpf/libbpf.c | 186 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------- tools/lib/bpf/libbpf_internal.h | 5 ++ tools/testing/selftests/bpf/prog_tests/bpf_cookie.c | 42 +++++++++++++++ tools/testing/selftests/bpf/prog_tests/raw_kprobe_test.c | 92 +++++++++++++++++++++++++++++++++ tools/testing/selftests/bpf/progs/get_func_ip_test.c | 4 +- tools/testing/selftests/bpf/progs/raw_kprobe.c | 58 +++++++++++++++++++++ tools/testing/selftests/bpf/progs/test_bpf_cookie.c | 24 ++++++++- 23 files changed, 1062 insertions(+), 104 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/raw_kprobe_test.c create mode 100644 tools/testing/selftests/bpf/progs/raw_kprobe.c
PATCH 1/13 ftrace: Add ftrace_set_filter_ips function
Adding ftrace_set_filter_ips function to be able to set filter on multiple ip addresses at once.
With the kprobe multi attach interface we have cases where we need to initialize ftrace_ops object with thousands of functions, so having single function diving into ftrace_hash_move_and_update_ops with ftrace_lock is faster.
The functions ips are passed as unsigned long array with count.
--- include/linux/ftrace.h | 3 +++ kernel/trace/ftrace.c | 53 +++++++++++++++++++++++++++++++++++------- 2 files changed, 47 insertions(+), 9 deletions(-)
include/linux/ftrace.h +3 -0
@@ -512,6 +512,8 @@ struct dyn_ftrace {int ftrace_set_filter_ip (struct ftrace_ops *ops,unsigned long ip,int remove,int reset); +int ftrace_set_filter_ips (struct ftrace_ops *ops,unsigned long *ips, +unsigned int cnt,int remove,int reset);int ftrace_set_filter (struct ftrace_ops *ops,unsigned char *buf,int len,int reset); int ftrace_set_notrace(struct ftrace_ops *ops, unsigned char *buf,
@@ -802,6 +804,7 @@ static inline unsigned long ftrace_location(unsigned long ip)#define ftrace_regex_open (ops, flag, inod, file) ({ -ENODEV; })#define ftrace_set_early_filter (ops, buf, enable) do { } while (0)#define ftrace_set_filter_ip (ops, ip, remove, reset) ({ -ENODEV; }) +#define ftrace_set_filter_ips (ops, ips, cnt, remove, reset) ({ -ENODEV; })#define ftrace_set_filter (ops, buf, len, reset) ({ -ENODEV; })#define ftrace_set_notrace (ops, buf, len, reset) ({ -ENODEV; })#define ftrace_free_filter (ops) do { } while (0)
kernel/trace/ftrace.c +44 -9
95 lines not on this page · raw message
Signature · 1 line
2.33.1
PATCH 2/13 kprobe: Keep traced function address
In reply to Jiri OlsaThe bpf_get_func_ip_kprobe helper should return traced function address, but it's doing so only for kprobes that are placed on the function entry.
If kprobe is placed within the function, bpf_get_func_ip_kprobe returns that address instead of function entry.
Storing the function entry directly in kprobe object, so it could be used in bpf_get_func_ip_kprobe helper.
--- include/linux/kprobes.h | 3 +++ kernel/kprobes.c | 12 ++++++++++++ kernel/trace/bpf_trace.c | 2 +- tools/testing/selftests/bpf/progs/get_func_ip_test.c | 4 ++-- 4 files changed, 18 insertions(+), 3 deletions(-)
include/linux/kprobes.h +3 -0
@@ -74,6 +74,9 @@ struct kprobe {/* Offset into the symbol */ unsigned int offset; +/* traced function address */ +unsigned long func_addr; +/* Called before addr is executed. */ kprobe_pre_handler_t pre_handler;
kernel/kprobes.c +12 -0
@@ -1310,6 +1310,7 @@ static void init_aggr_kprobe(struct kprobe *ap, struct kprobe *p)copy_kprobe (p, ap);flush_insn_slot (ap); ap->addr = p->addr; + ap->func_addr = p->func_addr; ap->flags = p->flags & ~KPROBE_FLAG_OPTIMIZED ; ap->pre_handler = aggr_pre_handler;/* We don't care the kprobe which has gone. */
@@ -1588,6 +1589,16 @@ static int check_kprobe_address_safe(struct kprobe *p,return ret; } +static unsigned long resolve_func_addr (kprobe_opcode_t *addr) +{ +char str[KSYM_SYMBOL_LEN ]; +unsigned long offset; + +if (kallsyms_lookup ((unsigned long ) addr,NULL , &offset,NULL , str)) +return (unsigned long ) addr - offset; +return 0 ; +} +int register_kprobe (struct kprobe *p) {int ret;
@@ -1600,6 +1611,7 @@ int register_kprobe(struct kprobe *p)if (IS_ERR (addr))return PTR_ERR (addr); p->addr = addr; + p->func_addr =resolve_func_addr (addr); ret =warn_kprobe_rereg (p);if (ret)
kernel/trace/bpf_trace.c +1 -1
@@ -1026,7 +1026,7 @@ BPF_CALL_1(bpf_get_func_ip_kprobe, struct pt_regs *, regs) {struct kprobe *kp =kprobe_running (); -return kp ? (uintptr_t )kp->addr :0 ; +return kp ? (uintptr_t )kp->func_addr :0 ; }static const struct bpf_func_proto bpf_get_func_ip_proto_kprobe = {
tools/testing/selftests/bpf/progs/get_func_ip_test.c +2 -2
@@ -69,7 +69,7 @@ int test6(struct pt_regs *ctx) {__u64 addr =bpf_get_func_ip (ctx); - test6_result = (const void *) addr == &bpf_fentry_test6 +5 ; + test6_result = (const void *) addr == &bpf_fentry_test6;return 0 ; }
@@ -79,6 +79,6 @@ int test7(struct pt_regs *ctx) {__u64 addr =bpf_get_func_ip (ctx); - test7_result = (const void *) addr == &bpf_fentry_test7 +5 ; + test7_result = (const void *) addr == &bpf_fentry_test7;return 0 ; }
Signature · 1 line
2.33.1
PATCH 3/13 kprobe: Add support to register multiple ftrace kprobes
In reply to Jiri OlsaAdding support to register kprobe on multiple addresses within single kprobe object instance.
It uses the CONFIG_KPROBES_ON_FTRACE feature (so it's only available for function entry addresses) and separated ftrace_ops object placed directly in the kprobe object.
There's new CONFIG_HAVE_KPROBES_MULTI_ON_FTRACE config option, enabled for archs that support multi kprobes.
It registers all the provided addresses in the ftrace_ops object filter and adds separate ftrace_ops callback.
To register multi kprobe user provides array of addresses or symbols with their count, like:
struct kprobe kp = {};
kp.multi.symbols = (const char **) symbols;
kp.multi.cnt = cnt;
...
err = register_kprobe(&kp);--- arch/Kconfig | 3 + arch/x86/Kconfig | 1 + arch/x86/kernel/kprobes/ftrace.c | 48 ++++++-- include/linux/kprobes.h | 25 ++++ kernel/kprobes.c | 204 +++++++++++++++++++++++++------ 5 files changed, 235 insertions(+), 46 deletions(-)
arch/Kconfig +3 -0
@@ -191,6 +191,9 @@ config HAVE_OPTPROBES config HAVE_KPROBES_ON_FTRACE bool +config HAVE_KPROBES_MULTI_ON_FTRACE + bool + config ARCH_CORRECT_STACKTRACE_ON_KRETPROBE bool help
arch/x86/Kconfig +1 -0
@@ -217,6 +217,7 @@ config X86 select HAVE_KERNEL_ZSTD select HAVE_KPROBES select HAVE_KPROBES_ON_FTRACE + select HAVE_KPROBES_MULTI_ON_FTRACE select HAVE_FUNCTION_ERROR_INJECTION select HAVE_KRETPROBES select HAVE_KVM
arch/x86/kernel/kprobes/ftrace.c +36 -12
69 lines not on this page · raw message
include/linux/kprobes.h +25 -0
@@ -68,6 +68,16 @@ struct kprobe {/* location of the probe point */ kprobe_opcode_t *addr; +#ifdef CONFIG_HAVE_KPROBES_MULTI_ON_FTRACE +/* location of the multi probe points */ +struct { +const char **symbols; +kprobe_opcode_t **addrs; +unsigned int cnt; +struct ftrace_ops ops; + } multi; +#endif +/* Allow user to indicate symbol name of the probe point */ const char *symbol_name;
@@ -105,6 +115,7 @@ struct kprobe {* this flag is only for optimized_kprobe. */ #define KPROBE_FLAG_FTRACE 8/* probe is using ftrace */ +#define KPROBE_FLAG_MULTI 16/* probe multi addresses */ /* Has this kprobe gone ? */ static inline bool kprobe_gone (struct kprobe *p)
@@ -130,6 +141,18 @@ static inline bool kprobe_ftrace(struct kprobe *p)return p->flags &KPROBE_FLAG_FTRACE ; } +/* Is this ftrace multi kprobe ? */ +static inline bool kprobe_ftrace_multi (struct kprobe *p) +{ +return kprobe_ftrace (p) && (p->flags &KPROBE_FLAG_MULTI ); +} + +/* Is this single kprobe ? */ +static inline bool kprobe_single (struct kprobe *p) +{ +return !(p->flags &KPROBE_FLAG_MULTI ); +} +/* * Function-return probe - * Note:
@@ -365,6 +388,8 @@ static inline void wait_for_kprobe_optimizer(void) { }#ifdef CONFIG_KPROBES_ON_FTRACE extern void kprobe_ftrace_handler (unsigned long ip,unsigned long parent_ip,struct ftrace_ops *ops,struct ftrace_regs *fregs); +extern void kprobe_ftrace_multi_handler (unsigned long ip,unsigned long parent_ip, +struct ftrace_ops *ops,struct ftrace_regs *fregs);extern int arch_prepare_kprobe_ftrace (struct kprobe *p);#else static inline int arch_prepare_kprobe_ftrace (struct kprobe *p)
kernel/kprobes.c +170 -34
294 lines not on this page · raw message
Signature · 1 line
2.33.1
PATCH 4/13 kprobe: Add support to register multiple ftrace kretprobes
In reply to Jiri OlsaAdding support to register kretprobe on multiple addresses within single kprobe object instance.
The interface stays same as for kprobes, the patch just adds blacklist check for each address, which is special for kretprobes.
--- include/linux/kprobes.h | 1 + kernel/kprobes.c | 46 ++++++++++++++++++++++++++++++++--------- 2 files changed, 37 insertions(+), 10 deletions(-)
include/linux/kprobes.h +1 -0
@@ -75,6 +75,7 @@ struct kprobe {kprobe_opcode_t **addrs;unsigned int cnt;struct ftrace_ops ops; + bool check_kretprobe; } multi;#endif
kernel/kprobes.c +36 -10
87 lines not on this page · raw message
Signature · 1 line
2.33.1
PATCH 5/13 kprobe: Allow to get traced function address for multi ftrace kprobes
In reply to Jiri OlsaThe current bpf_get_func_ip_kprobe helper does not work properly, when used in ebpf program triggered by the new multi kprobes.
We can't use kprobe's func_addr in bpf_get_func_ip_kprobe helper, because there are multiple functions registered for single kprobe object.
Adding new per cpu variable current_ftrace_multi_addr and extra address in kretprobe_instance object to keep current traced function address for each cpu for both kprobe handler and kretprobe trampoline.
The address value is set/passed as follows, for kprobe:
kprobe_ftrace_multi_handler
{
old = kprobe_ftrace_multi_addr_set(ip);
handler..
kprobe_ftrace_multi_addr_set(old);
}For kretprobe:
kprobe_ftrace_multi_handler
{
old = kprobe_ftrace_multi_addr_set(ip);
...
pre_handler_kretprobe
{
ri->ftrace_multi_addr = kprobe_ftrace_multi_addr
}
...
kprobe_ftrace_multi_addr_set(old);
}
__kretprobe_trampoline_handler
{
prev_func_addr = kprobe_ftrace_multi_addr_set(ri->ftrace_multi_addr);
handler..
kprobe_ftrace_multi_addr_set(prev_func_addr);
}--- arch/x86/kernel/kprobes/ftrace.c | 3 +++ include/linux/kprobes.h | 26 ++++++++++++++++++++++++++ kernel/kprobes.c | 6 ++++++ kernel/trace/bpf_trace.c | 7 ++++++- 4 files changed, 41 insertions(+), 1 deletion(-)
arch/x86/kernel/kprobes/ftrace.c +3 -0
@@ -72,6 +72,7 @@ NOKPROBE_SYMBOL(kprobe_ftrace_handler);void kprobe_ftrace_multi_handler (unsigned long ip,unsigned long parent_ip,struct ftrace_ops *ops,struct ftrace_regs *fregs) { +unsigned long old;struct kprobe *p;int bit;
@@ -79,8 +80,10 @@ void kprobe_ftrace_multi_handler(unsigned long ip, unsigned long parent_ip,if (bit <0 )return ; + old =kprobe_ftrace_multi_addr_set (ip); p = container_of(ops, struct kprobe, multi.ops);ftrace_handler (p, ip, fregs); +kprobe_ftrace_multi_addr_set (old);ftrace_test_recursion_unlock (bit); }
include/linux/kprobes.h +26 -0
@@ -191,6 +191,7 @@ struct kretprobe_instance {struct kretprobe_holder *rph;kprobe_opcode_t *ret_addr;void *fp; +unsigned long ftrace_multi_addr;char data[]; };
@@ -387,16 +388,37 @@ static inline void wait_for_kprobe_optimizer(void) { }#endif /* CONFIG_OPTPROBES */ #ifdef CONFIG_KPROBES_ON_FTRACE +DECLARE_PER_CPU(unsigned long, current_ftrace_multi_addr);extern void kprobe_ftrace_handler (unsigned long ip,unsigned long parent_ip,struct ftrace_ops *ops,struct ftrace_regs *fregs);extern void kprobe_ftrace_multi_handler (unsigned long ip,unsigned long parent_ip,struct ftrace_ops *ops,struct ftrace_regs *fregs);extern int arch_prepare_kprobe_ftrace (struct kprobe *p); + +static inline unsigned long kprobe_ftrace_multi_addr (void ) +{ +return __this_cpu_read (current_ftrace_multi_addr); +} +static inline unsigned long kprobe_ftrace_multi_addr_set (unsigned long addr) +{ +unsigned long old =__this_cpu_read (current_ftrace_multi_addr); + +__this_cpu_write (current_ftrace_multi_addr, addr); +return old; +}#else static inline int arch_prepare_kprobe_ftrace (struct kprobe *p) {return -EINVAL ; } +static inline unsigned long kprobe_ftrace_multi_addr_set (unsigned long addr) +{ +return 0 ; +} +static inline unsigned long kprobe_ftrace_multi_addr (void ) +{ +return 0 ; +}#endif /* CONFIG_KPROBES_ON_FTRACE */ /* Get the kprobe at this addr (if any) - called with preemption disabled */
@@ -514,6 +536,10 @@ static inline int kprobe_get_kallsym(unsigned int symnum, unsigned long *value, {return -ERANGE ; } +static inline unsigned long kprobe_ftrace_multi_addr (void ) +{ +return 0 ; +}#endif /* CONFIG_KPROBES */ static inline int disable_kretprobe (struct kretprobe *rp)
kernel/kprobes.c +6 -0
@@ -1064,6 +1064,8 @@ static bool in_kretprobe_blacklist(void *addr) }#ifdef CONFIG_KPROBES_ON_FTRACE +DEFINE_PER_CPU(unsigned long, current_ftrace_multi_addr); +static struct ftrace_ops kprobe_ftrace_ops__read_mostly = { .func = kprobe_ftrace_handler, .flags = FTRACE_OPS_FL_SAVE_REGS,
@@ -2106,11 +2108,14 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, rp =get_kretprobe (ri);if (rp && rp->handler) {struct kprobe *prev =kprobe_running (); +unsigned long prev_func_addr; + prev_func_addr =kprobe_ftrace_multi_addr_set (ri->ftrace_multi_addr);__this_cpu_write (current_kprobe, &rp->kp); ri->ret_addr = correct_ret_addr; rp->handler (ri, regs);__this_cpu_write (current_kprobe, prev); +kprobe_ftrace_multi_addr_set (prev_func_addr); }if (first == node)break ;
@@ -2161,6 +2166,7 @@ static int pre_handler_kretprobe(struct kprobe *p, struct pt_regs *regs) }arch_prepare_kretprobe (ri, regs); + ri->ftrace_multi_addr =kprobe_ftrace_multi_addr ();__llist_add (&ri->llist, ¤t->kretprobe_instances);
kernel/trace/bpf_trace.c +6 -1
@@ -1026,7 +1026,12 @@ BPF_CALL_1(bpf_get_func_ip_kprobe, struct pt_regs *, regs) {struct kprobe *kp =kprobe_running (); -return kp ? (uintptr_t )kp->func_addr :0 ; +if (!kp) +return 0 ; +if (kprobe_ftrace_multi (kp)) +return (uintptr_t )kprobe_ftrace_multi_addr (); +else +return (uintptr_t ) kp->func_addr; }static const struct bpf_func_proto bpf_get_func_ip_proto_kprobe = {
Signature · 1 line
2.33.1
PATCH 6/13 samples/kprobes: Add support for multi kprobe interface
In reply to Jiri OlsaAdding support to test multi kprobe interface. It's now possible to register multiple functions for the module handler, like:
# modprobe kprobe_example symbol='sched_fork,kernel_clone'
--- samples/kprobes/kprobe_example.c | 47 +++++++++++++++++++++++++++++--- 1 file changed, 43 insertions(+), 4 deletions(-)
samples/kprobes/kprobe_example.c +43 -4
@@ -15,8 +15,10 @@#include <linux/kernel.h> #include <linux/module.h> #include <linux/kprobes.h> +#include <linux/slab.h> +#include <linux/string.h> -#define MAX_SYMBOL_LEN 64 +#define MAX_SYMBOL_LEN 4096static char symbol[MAX_SYMBOL_LEN ] ="kernel_clone" ;module_param_string (symbol, symbol,sizeof (symbol),0644 );
@@ -97,17 +99,54 @@ static void __kprobes handler_post(struct kprobe *p, struct pt_regs *regs,static int __init kprobe_init (void ) { +char **symbols =NULL ;int ret; + kp.pre_handler = handler_pre; kp.post_handler = handler_post; +#ifdef CONFIG_HAVE_KPROBES_MULTI_ON_FTRACE +if (strchr (symbol,',' )) { +char *p, *tmp; +int cnt; + + tmp =kstrdup (symbol,GFP_KERNEL ); +if (!tmp) +return -ENOMEM ; + + p =strchr (tmp,',' ); +while (p) { + *p =' ' ; + p =strchr (p +1 ,',' ); + } + + symbols =argv_split (GFP_KERNEL , tmp, &cnt); +kfree (tmp); +if (!symbols) { + ret = -ENOMEM ; + goto out; + } + + kp.multi.symbols = (const char **) symbols; + kp.multi.cnt = cnt; + } +#endif + ret =register_kprobe (&kp);if (ret <0 ) {pr_err ("register_kprobe failed, returned %d\n" , ret); -return ret; + goto out; } -pr_info ("Planted kprobe at %p\n" , kp.addr); -return 0 ; + +if (symbols) +pr_info ("Planted multi kprobe to %s\n" , symbol); +else +pr_info ("Planted kprobe at %p\n" , kp.addr); + +out: +if (symbols) +argv_free (symbols); +return ret; } static void __exit kprobe_exit(void)
Signature · 1 line
2.33.1
PATCH 7/13 samples/kprobes: Add support for multi kretprobe interface
In reply to Jiri OlsaAdding support to test multi kprobe interface. It's now possible to register multiple functions for the module handler, like:
# modprobe kretprobe_example.ko func='sched_fork,kernel_clone'
--- samples/kprobes/kretprobe_example.c | 43 +++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-)
samples/kprobes/kretprobe_example.c +41 -2
@@ -25,6 +25,8 @@#include <linux/ktime.h> #include <linux/limits.h> #include <linux/sched.h> +#include <linux/slab.h> +#include <linux/string.h> static char func_name[NAME_MAX ] ="kernel_clone" ;module_param_string (func, func_name,NAME_MAX ,S_IRUGO );
@@ -80,17 +82,54 @@ static struct kretprobe my_kretprobe = {static int __init kretprobe_init (void ) { +char **symbols =NULL ;int ret; my_kretprobe.kp.symbol_name = func_name; + +#ifdef CONFIG_HAVE_KPROBES_MULTI_ON_FTRACE +if (strchr (func_name,',' )) { +char *p, *tmp; +int cnt; + + tmp =kstrdup (func_name,GFP_KERNEL ); +if (!tmp) +return -ENOMEM ; + + p =strchr (tmp,',' ); +while (p) { + *p =' ' ; + p =strchr (p +1 ,',' ); + } + + symbols =argv_split (GFP_KERNEL , tmp, &cnt); +kfree (tmp); +if (!symbols) { + ret = -ENOMEM ; + goto out; + } + + my_kretprobe.kp.multi.symbols = (const char **) symbols; + my_kretprobe.kp.multi.cnt = cnt; + } +#endif ret =register_kretprobe (&my_kretprobe);if (ret <0 ) {pr_err ("register_kretprobe failed, returned %d\n" , ret);return ret; } -pr_info ("Planted return probe at %s: %p\n" , + +if (symbols) { +pr_info ("Planted multi return kprobe to %s\n" , func_name); + }else { +pr_info ("Planted return probe at %s: %p\n" , my_kretprobe.kp.symbol_name, my_kretprobe.kp.addr); -return 0 ; + } + +out: +if (symbols) +argv_free (symbols); +return ret; } static void __exit kretprobe_exit(void)
Signature · 1 line
2.33.1
PATCH 8/13 bpf: Add kprobe link for attaching raw kprobes
In reply to Jiri OlsaAdding new link type BPF_LINK_TYPE_KPROBE to attach kprobes directly through register_kprobe/kretprobe API.
Adding new attach type BPF_TRACE_RAW_KPROBE that enables such link for kprobe program.
The new link allows to create multiple kprobes link by using new link_create interface:
struct {
__aligned_u64 addrs;
__u32 cnt;
__u64 bpf_cookie;
} kprobe;Plus new flag BPF_F_KPROBE_RETURN for link_create.flags to create return probe.
--- include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 12 +++ kernel/bpf/syscall.c | 191 ++++++++++++++++++++++++++++++++- tools/include/uapi/linux/bpf.h | 12 +++ 4 files changed, 211 insertions(+), 5 deletions(-)
include/linux/bpf_types.h +1 -0
@@ -140,3 +140,4 @@ BPF_LINK_TYPE(BPF_LINK_TYPE_XDP, xdp)#ifdef CONFIG_PERF_EVENTS BPF_LINK_TYPE(BPF_LINK_TYPE_PERF_EVENT, perf)#endif +BPF_LINK_TYPE (BPF_LINK_TYPE_KPROBE , kprobe)
include/uapi/linux/bpf.h +12 -0
@@ -995,6 +995,7 @@ enum bpf_attach_type {BPF_SK_REUSEPORT_SELECT ,BPF_SK_REUSEPORT_SELECT_OR_MIGRATE ,BPF_PERF_EVENT , +BPF_TRACE_RAW_KPROBE , __MAX_BPF_ATTACH_TYPE };
@@ -1009,6 +1010,7 @@ enum bpf_link_type {BPF_LINK_TYPE_NETNS =5 ,BPF_LINK_TYPE_XDP =6 ,BPF_LINK_TYPE_PERF_EVENT =7 , +BPF_LINK_TYPE_KPROBE =8 , MAX_BPF_LINK_TYPE, };
@@ -1111,6 +1113,11 @@ enum bpf_link_type {*/ #define BPF_F_SLEEPABLE (1U << 4) +/* link_create flags used in LINK_CREATE command for BPF_TRACE_RAW_KPROBE +* attach type. +*/ +#define BPF_F_KPROBE_RETURN (1U << 0) +/* When BPF ldimm64's insn[0].src_reg != 0 then this can have * the following extensions: *
@@ -1465,6 +1472,11 @@ union bpf_attr {*/ __u64 bpf_cookie; } perf_event; +struct { +__aligned_u64 addrs; +__u32 cnt; +__u64 bpf_cookie; + } kprobe; }; } link_create;
kernel/bpf/syscall.c +186 -5
233 lines not on this page · raw message
tools/include/uapi/linux/bpf.h +12 -0
@@ -995,6 +995,7 @@ enum bpf_attach_type {BPF_SK_REUSEPORT_SELECT ,BPF_SK_REUSEPORT_SELECT_OR_MIGRATE ,BPF_PERF_EVENT , +BPF_TRACE_RAW_KPROBE , __MAX_BPF_ATTACH_TYPE };
@@ -1009,6 +1010,7 @@ enum bpf_link_type {BPF_LINK_TYPE_NETNS =5 ,BPF_LINK_TYPE_XDP =6 ,BPF_LINK_TYPE_PERF_EVENT =7 , +BPF_LINK_TYPE_KPROBE =8 , MAX_BPF_LINK_TYPE, };
@@ -1111,6 +1113,11 @@ enum bpf_link_type {*/ #define BPF_F_SLEEPABLE (1U << 4) +/* link_create flags used in LINK_CREATE command for BPF_TRACE_RAW_KPROBE +* attach type. +*/ +#define BPF_F_KPROBE_RETURN (1U << 0) +/* When BPF ldimm64's insn[0].src_reg != 0 then this can have * the following extensions: *
@@ -1465,6 +1472,11 @@ union bpf_attr {*/ __u64 bpf_cookie; } perf_event; +struct { +__aligned_u64 addrs; +__u32 cnt; +__u64 bpf_cookie; + } kprobe; }; } link_create;
Signature · 1 line
2.33.1
PATCH 9/13 libbpf: Add libbpf__kallsyms_parse function
In reply to Jiri OlsaMove the kallsyms parsing in internal libbpf__kallsyms_parse function, so it can be used from other places.
It will be used in following changes.
--- tools/lib/bpf/libbpf.c | 62 ++++++++++++++++++++------------- tools/lib/bpf/libbpf_internal.h | 5 +++ 2 files changed, 43 insertions(+), 24 deletions(-)
tools/lib/bpf/libbpf.c +38 -24
@@ -7146,12 +7146,10 @@ static int bpf_object__sanitize_maps(struct bpf_object *obj)return 0 ; } -static int bpf_object__read_kallsyms_file (struct bpf_object *obj) +int libbpf__kallsyms_parse (void *arg,kallsyms_cb_t cb) {char sym_type, sym_name[500 ];unsigned long long sym_addr; -const struct btf_type *t; -struct extern_desc *ext;int ret, err =0 ;FILE *f;
@@ -7170,35 +7168,51 @@ static int bpf_object__read_kallsyms_file(struct bpf_object *obj)if (ret !=3 ) {pr_warn ("failed to read kallsyms entry: %d\n" , ret); err = -EINVAL ; - goto out; +break ; } - ext =find_extern_by_name (obj, sym_name); -if (!ext || ext->type !=EXT_KSYM ) -continue ; - - t =btf__type_by_id (obj->btf, ext->btf_id); -if (!btf_is_var (t)) -continue ; - -if (ext->is_set && ext->ksym.addr != sym_addr) { -pr_warn ("extern (ksym) '%s' resolution is ambiguous: 0x%llx or 0x%llx\n" , - sym_name, ext->ksym.addr, sym_addr); - err = -EINVAL ; - goto out; - } -if (!ext->is_set) { - ext->is_set = true; - ext->ksym.addr = sym_addr; -pr_debug ("extern (ksym) %s=0x%llx\n" , sym_name, sym_addr); - } + err =cb (arg, sym_addr, sym_type, sym_name); +if (err) +break ; } -out:fclose (f);return err; } +static int kallsyms_cb (void *arg,unsigned long long sym_addr, +char sym_type,const char *sym_name) +{ +struct bpf_object *obj = arg; +const struct btf_type *t; +struct extern_desc *ext; + + ext =find_extern_by_name (obj, sym_name); +if (!ext || ext->type !=EXT_KSYM ) +return 0 ; + + t =btf__type_by_id (obj->btf, ext->btf_id); +if (!btf_is_var (t)) +return 0 ; + +if (ext->is_set && ext->ksym.addr != sym_addr) { +pr_warn ("extern (ksym) '%s' resolution is ambiguous: 0x%llx or 0x%llx\n" , + sym_name, ext->ksym.addr, sym_addr); +return -EINVAL ; + } +if (!ext->is_set) { + ext->is_set = true; + ext->ksym.addr = sym_addr; +pr_debug ("extern (ksym) %s=0x%llx\n" , sym_name, sym_addr); + } +return 0 ; +} + +static int bpf_object__read_kallsyms_file (struct bpf_object *obj) +{ +return libbpf__kallsyms_parse (obj, kallsyms_cb); +} +static int find_ksym_btf_id (struct bpf_object *obj,const char *ksym_name,__u16 kind,struct btf **res_btf,struct module_btf **res_mod_btf)
tools/lib/bpf/libbpf_internal.h +5 -0
@@ -446,6 +446,11 @@ __s32 btf__find_by_name_kind_own(const struct btf *btf, const char *type_name,extern enum libbpf_strict_mode libbpf_mode; +typedef int (*kallsyms_cb_t )(void *arg,unsigned long long sym_addr, +char sym_type,const char *sym_name); + +int libbpf__kallsyms_parse (void *arg,kallsyms_cb_t cb); +/* handle direct returned errors */ static inline int libbpf_err (int ret) {
Signature · 1 line
2.33.1
PATCH 10/13 libbpf: Add bpf_link_create support for multi kprobes
In reply to Jiri OlsaAdding new kprobe struct in bpf_link_create_opts object to pass multi kprobe data to link_create attr API.
--- tools/lib/bpf/bpf.c | 5 +++++ tools/lib/bpf/bpf.h | 7 ++++++- 2 files changed, 11 insertions(+), 1 deletion(-)
tools/lib/bpf/bpf.c +5 -0
@@ -848,6 +848,11 @@ int bpf_link_create(int prog_fd, int target_fd,if (!OPTS_ZEROED (opts, perf_event))return libbpf_err (-EINVAL );break ; +case BPF_TRACE_RAW_KPROBE : + attr.link_create.kprobe.addrs =OPTS_GET (opts, kprobe.addrs,0 ); + attr.link_create.kprobe.cnt =OPTS_GET (opts, kprobe.cnt,0 ); + attr.link_create.kprobe.bpf_cookie =OPTS_GET (opts, kprobe.bpf_cookie,0 ); +break ;default :if (!OPTS_ZEROED (opts, flags))return libbpf_err (-EINVAL );
tools/lib/bpf/bpf.h +6 -1
@@ -298,10 +298,15 @@ struct bpf_link_create_opts {struct {__u64 bpf_cookie; } perf_event; +struct { +__u64 addrs; +__u32 cnt; +__u64 bpf_cookie; + } kprobe; }; size_t :0; }; -#define bpf_link_create_opts__last_field perf_event +#define bpf_link_create_opts__last_field kprobe.bpf_cookie LIBBPF_API int bpf_link_create(int prog_fd, int target_fd, enum bpf_attach_type attach_type,
Signature · 1 line
2.33.1
PATCH 11/13 libbpf: Add bpf_program__attach_kprobe_opts for multi kprobes
In reply to Jiri Olsa SEC("raw_kprobe/bpf_fentry_test*")
to attach program to all bpf_fentry_test* functions.
--- tools/lib/bpf/libbpf.c | 124 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 123 insertions(+), 1 deletion(-)
tools/lib/bpf/libbpf.c +123 -1
@@ -10000,6 +10000,125 @@ static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe,return pfd; } +struct kprobe_resolve_raw_kprobe { +const char *name; +__u64 *addrs; +__u32 alloc; +__u32 cnt; +}; + +static bool glob_matches (const char *glob,const char *s) +{ +int n =strlen (glob); + +if (n ==1 && glob[0 ] =='*' ) +return true; + +if (glob[0 ] =='*' && glob[n -1 ] =='*' ) { +const char *subs; +/* substring match */ + +/* this is hacky, but we don't want to allocate +* for no good reason +*/ + ((char *)glob)[n -1 ] ='\0' ; + subs =strstr (s, glob +1 ); + ((char *)glob)[n -1 ] ='*' ; + +return subs !=NULL ; + }else if (glob[0 ] =='*' ) { +size_t nn =strlen (s); +/* suffix match */ + +/* too short for a given suffix */ +if (nn < n -1 ) +return false; +return strcmp (s + nn - (n -1 ), glob +1 ) ==0 ; + }else if (glob[n -1 ] =='*' ) { +/* prefix match */ +return strncmp (s, glob, n -1 ) ==0 ; + }else { +/* exact match */ +return strcmp (glob, s) ==0 ; + } +} + +static int +kprobe_resolve_raw_kprobe_cb (void *arg,unsigned long long sym_addr, +char sym_type,const char *sym_name) +{ +struct kprobe_resolve_raw_kprobe *res = arg; +__u64 *p; + +if (!glob_matches (res->name, sym_name)) +return 0 ; + +if (res->cnt == res->alloc) { + res->alloc =max ((__u32 )16 , res->alloc *3 /2 ); + p =libbpf_reallocarray (res->addrs, res->alloc,sizeof (__u32)); +if (!p) +return -ENOMEM ; + res->addrs = p; + } + res->addrs[res->cnt++] = sym_addr; +return 0 ; +} + +static struct bpf_link * +attach_raw_kprobe_opts (const struct bpf_program *prog, +const char *func_name, +const struct bpf_kprobe_opts *kopts) +{ +DECLARE_LIBBPF_OPTS (bpf_link_create_opts, opts); +struct kprobe_resolve_raw_kprobe res = { + .name = func_name, + }; +struct bpf_link *link =NULL ; +char errmsg[STRERR_BUFSIZE ]; +int err, link_fd, prog_fd; +__u64 bpf_cookie; +bool retprobe; + + err =libbpf__kallsyms_parse (&res, kprobe_resolve_raw_kprobe_cb); +if (err) + goto error; +if (!res.cnt) { + err = -ENOENT ; + goto error; + } + + retprobe =OPTS_GET (kopts, retprobe, false); + bpf_cookie =OPTS_GET (kopts, bpf_cookie,0 ); + + opts.kprobe.addrs = (__u64 ) res.addrs; + opts.kprobe.cnt = res.cnt; + opts.kprobe.bpf_cookie = bpf_cookie; + opts.flags = retprobe ?BPF_F_KPROBE_RETURN :0 ; + + link =calloc (1 ,sizeof (*link)); +if (!link) +return libbpf_err_ptr (-ENOMEM ); + link->detach = &bpf_link__detach_fd; + + prog_fd =bpf_program__fd (prog); + link_fd =bpf_link_create (prog_fd,0 ,BPF_TRACE_RAW_KPROBE , &opts); +if (link_fd <0 ) { + err = -errno; +pr_warn ("prog '%s': failed to attach to %s: %s\n" , + prog->name, res.name, +libbpf_strerror_r (err, errmsg,sizeof (errmsg))); + goto error; + } + link->fd = link_fd; +free (res.addrs); +return link; + +error: +free (link); +free (res.addrs); +return libbpf_err_ptr (err); +} +struct bpf_link * bpf_program__attach_kprobe_opts(const struct bpf_program *prog, const char *func_name,
@@ -10016,6 +10135,9 @@ bpf_program__attach_kprobe_opts(const struct bpf_program *prog,if (!OPTS_VALID (opts, bpf_kprobe_opts))return libbpf_err_ptr (-EINVAL ); +if (strchr (func_name,'*' )) +return attach_raw_kprobe_opts (prog, func_name, opts); + retprobe =OPTS_GET (opts, retprobe, false); offset =OPTS_GET (opts, offset,0 ); pe_opts.bpf_cookie =OPTS_GET (opts, bpf_cookie,0 );
@@ -10096,7 +10218,7 @@ static struct bpf_link *attach_kprobe(const struct bpf_program *prog, long cookielse func_name = prog->sec_name +sizeof ("kprobe/" ) -1 ; - n =sscanf (func_name,"%m[a-zA-Z0-9_.]+%li" , &func, &offset); + n =sscanf (func_name,"%m[a-zA-Z0-9_.*]+%li" , &func, &offset);if (n <1 ) { err = -EINVAL ;pr_warn ("kprobe name is invalid: %s\n" , func_name);
Signature · 1 line
2.33.1
PATCH 12/13 selftest/bpf: Add raw kprobe attach test
In reply to Jiri OlsaAdding raw kprobe attach test that uses new interface to attach multiple kprobes with new kprobe link.
The test is attaching to bpf_fentry_test* functions and single trampoline program to use bpf_prog_test_run to trigger bpf_fentry_test* functions.
--- .../bpf/prog_tests/raw_kprobe_test.c | 92 +++++++++++++++++++ .../testing/selftests/bpf/progs/raw_kprobe.c | 58 ++++++++++++ 2 files changed, 150 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/raw_kprobe_test.c create mode 100644 tools/testing/selftests/bpf/progs/raw_kprobe.c
tools/testing/selftests/bpf/prog_tests/raw_kprobe_test.c new file +92 -0
@@ -0,0 +1,92 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <test_progs.h> +#include "raw_kprobe.skel.h" +#include "trace_helpers.h" + +static void test_skel_api (void ) +{ +struct raw_kprobe *skel =NULL ; +__u32 duration =0 , retval; +int err, prog_fd; + + skel =raw_kprobe__open_and_load (); +if (!ASSERT_OK_PTR (skel,"raw_kprobe__open_and_load" )) + goto cleanup; + + err =raw_kprobe__attach (skel); +if (!ASSERT_OK (err,"raw_kprobe__attach" )) + goto cleanup; + + prog_fd =bpf_program__fd (skel->progs.test1); + err =bpf_prog_test_run (prog_fd,1 ,NULL ,0 , +NULL ,NULL , &retval, &duration); +ASSERT_OK (err,"test_run" ); +ASSERT_EQ (retval,0 ,"test_run" ); + +ASSERT_EQ (skel->bss->test2_result,8 ,"test2_result" ); +ASSERT_EQ (skel->bss->test3_result,8 ,"test3_result" ); + +cleanup: +raw_kprobe__destroy (skel); +} + +static void test_link_api (void ) +{ +DECLARE_LIBBPF_OPTS (bpf_link_create_opts, opts); +int err, prog_fd, link1_fd =-1 , link2_fd =-1 ; +struct raw_kprobe *skel =NULL ; +__u32 duration =0 , retval; +__u64 addrs[8 ]; + + skel =raw_kprobe__open_and_load (); +if (!ASSERT_OK_PTR (skel,"fentry_raw_skel_load" )) + goto cleanup; + +kallsyms_find ("bpf_fentry_test1" , &addrs[0 ]); +kallsyms_find ("bpf_fentry_test2" , &addrs[1 ]); +kallsyms_find ("bpf_fentry_test3" , &addrs[2 ]); +kallsyms_find ("bpf_fentry_test4" , &addrs[3 ]); +kallsyms_find ("bpf_fentry_test5" , &addrs[4 ]); +kallsyms_find ("bpf_fentry_test6" , &addrs[5 ]); +kallsyms_find ("bpf_fentry_test7" , &addrs[6 ]); +kallsyms_find ("bpf_fentry_test8" , &addrs[7 ]); + + opts.kprobe.addrs = (__u64 ) addrs; + opts.kprobe.cnt =8 ; + + prog_fd =bpf_program__fd (skel->progs.test2); + link1_fd =bpf_link_create (prog_fd,0 ,BPF_TRACE_RAW_KPROBE , &opts); +if (!ASSERT_GE (link1_fd,0 ,"link_fd" )) + goto cleanup; + + opts.flags =BPF_F_KPROBE_RETURN ; + prog_fd =bpf_program__fd (skel->progs.test3); + link2_fd =bpf_link_create (prog_fd,0 ,BPF_TRACE_RAW_KPROBE , &opts); +if (!ASSERT_GE (link2_fd,0 ,"link_fd" )) + goto cleanup; + + skel->bss->test2_result =0 ; + skel->bss->test3_result =0 ; + + prog_fd =bpf_program__fd (skel->progs.test1); + err =bpf_prog_test_run (prog_fd,1 ,NULL ,0 , +NULL ,NULL , &retval, &duration); +ASSERT_OK (err,"test_run" ); +ASSERT_EQ (retval,0 ,"test_run" ); + +ASSERT_EQ (skel->bss->test2_result,8 ,"test2_result" ); +ASSERT_EQ (skel->bss->test3_result,8 ,"test3_result" ); + +cleanup: +if (link1_fd !=-1 ) +close (link1_fd); +if (link2_fd !=-1 ) +close (link2_fd); +raw_kprobe__destroy (skel); +} + +void test_raw_kprobe_test (void ) +{ +test_skel_api (); +test_link_api (); +}
tools/testing/selftests/bpf/progs/raw_kprobe.c new file +58 -0
58 lines not on this page · raw message
Signature · 1 line
2.33.1
PATCH 13/13 selftest/bpf: Add bpf_cookie test for raw_k[ret]probe
In reply to Jiri OlsaAdding bpf_cookie test for raw_k[ret]probes.
Using the bpf_link_create interface directly and single trampoline program to trigger the bpf_fentry_test1 execution.
--- .../selftests/bpf/prog_tests/bpf_cookie.c | 42 +++++++++++++++++++ .../selftests/bpf/progs/test_bpf_cookie.c | 24 ++++++++++- 2 files changed, 65 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/prog_tests/bpf_cookie.c +42 -0
@@ -57,6 +57,46 @@ static void kprobe_subtest(struct test_bpf_cookie *skel)bpf_link__destroy (retlink2); } +static void rawkprobe_subtest (struct test_bpf_cookie *skel) +{ +DECLARE_LIBBPF_OPTS (bpf_link_create_opts, opts); +int err, prog_fd, link1_fd =-1 , link2_fd =-1 ; +__u32 duration =0 , retval; +__u64 addr; + +kallsyms_find ("bpf_fentry_test1" , &addr); + + opts.kprobe.addrs = (__u64) &addr; + opts.kprobe.cnt =1 ; + opts.kprobe.bpf_cookie =0x1 ; + prog_fd =bpf_program__fd (skel->progs.handle_raw_kprobe); + + link1_fd =bpf_link_create (prog_fd,0 ,BPF_TRACE_RAW_KPROBE , &opts); +if (!ASSERT_GE (link1_fd,0 ,"link1_fd" )) +return ; + + opts.flags =BPF_F_KPROBE_RETURN ; + opts.kprobe.bpf_cookie =0x2 ; + prog_fd =bpf_program__fd (skel->progs.handle_raw_kretprobe); + + link2_fd =bpf_link_create (prog_fd,0 ,BPF_TRACE_RAW_KPROBE , &opts); +if (!ASSERT_GE (link2_fd,0 ,"link2_fd" )) + goto cleanup; + + prog_fd =bpf_program__fd (skel->progs.raw_trigger); + err =bpf_prog_test_run (prog_fd,1 ,NULL ,0 , +NULL ,NULL , &retval, &duration); +ASSERT_OK (err,"test_run" ); +ASSERT_EQ (retval,0 ,"test_run" ); + +ASSERT_EQ (skel->bss->raw_kprobe_res,0x1 ,"raw_kprobe_res" ); +ASSERT_EQ (skel->bss->raw_kretprobe_res,0x2 ,"raw_kretprobe_res" ); + +cleanup: +close (link1_fd); +close (link2_fd); +} +static void uprobe_subtest (struct test_bpf_cookie *skel) {DECLARE_LIBBPF_OPTS (bpf_uprobe_opts, opts);
@@ -243,6 +283,8 @@ void test_bpf_cookie(void)if (test__start_subtest ("kprobe" ))kprobe_subtest (skel); +if (test__start_subtest ("rawkprobe" )) +rawkprobe_subtest (skel);if (test__start_subtest ("uprobe" ))uprobe_subtest (skel);if (test__start_subtest ("tracepoint" ))
tools/testing/selftests/bpf/progs/test_bpf_cookie.c +23 -1
@@ -8,8 +8,9 @@int my_tid;int kprobe_res; -int kprobe_multi_res;int kretprobe_res; +int raw_kprobe_res; +int raw_kretprobe_res;int uprobe_res;int uretprobe_res;int tp_res;
@@ -37,6 +38,27 @@ int handle_kretprobe(struct pt_regs *ctx)return 0 ; } +SEC("kprobe/bpf_fentry_test1") +int handle_raw_kprobe (struct pt_regs *ctx) +{ +update (ctx, &raw_kprobe_res); +return 0 ; +} + +SEC("kretprobe/bpf_fentry_test1") +int handle_raw_kretprobe (struct pt_regs *ctx) +{ +update (ctx, &raw_kretprobe_res); +return 0 ; +} + +/* just to trigger bpf_fentry_test1 through tracing test_run */ +SEC("fentry/bpf_modify_return_test") +int BPF_PROG (raw_trigger ) +{ +return 0 ; +} + SEC("uprobe/trigger_func")int handle_uprobe (struct pt_regs *ctx) {
Signature · 1 line
2.33.1
Quoted 10 lines · Jiri Olsa
hi,
adding support to attach multiple kprobes within single syscall and speed up attachment of many kprobes.The previous attempt [1] wasn't fast enough, so coming with new approach that adds new kprobe interface.
The attachment speed of of this approach (tested in bpftrace) is now comparable to ftrace tracer attachment speed.. fast ;-)
What are the absolute numbers?
How quickly a single bpf prog can attach to 1k kprobes?
Quoted 6 lines · Alexei Starovoitov
Quoted 5 lines · Jiri Olsa
hi,
adding support to attach multiple kprobes within single syscall and speed up attachment of many kprobes.
Quoted 9 lines
Quoted 5 lines
The previous attempt [1] wasn't fast enough, so coming with new approach that adds new kprobe interface.
The attachment speed of of this approach (tested in bpftrace) is now comparable to ftrace tracer attachment speed.. fast ;-)
What are the absolute numbers?
How quickly a single bpf prog can attach to 1k kprobes?
I'd need to write special tool for 1k kprobes exactly, we could do some benchmark selftest for that
I tested following counts with current bpftrace interface for now (note it includes both attach and detach)
2 seconds for 673 kprobes:
# perf stat -e cycles:u,cycles:k ./src/bpftrace -e 'kprobe:kvm* { } i:ms:10 { printf("KRAVA\n"); exit() }'
Attaching 2 probes...
Attaching 673 functions
KRAVA Performance counter stats for './src/bpftrace -e kprobe:kvm* { } i:ms:10 { printf("KRAVA\n"); exit() }':
1,695,142,901 cycles:u
1,909,616,944 cycles:k
1.990434019 seconds time elapsed
0.767746000 seconds user
0.921166000 seconds sys5 seconds for 3337 kprobes:
# perf stat -e cycles:u,cycles:k ./src/bpftrace -e 'kprobe:x* { } i:ms:10 { printf("KRAVA\n"); exit() }'
Attaching 2 probes...
Attaching 3337 functions
KRAVA Performance counter stats for './src/bpftrace -e kprobe:x* { } i:ms:10 { printf("KRAVA\n"); exit() }':
1,731,646,061 cycles:u
9,815,306,940 cycles:k
5.196176904 seconds time elapsed
0.780508000 seconds user
4.078170000 seconds syslot of the time above is spent in kallsyms:
42.70% bpftrace [kernel.kallsyms] [k] kallsyms_expand_symbol.constprop.0 5.11% bpftrace [kernel.kallsyms] [k] insn_get_prefixes.part.0 3.91% bpftrace [kernel.kallsyms] [k] insn_decode 3.09% bpftrace [kernel.kallsyms] [k] arch_jump_entry_size 1.98% bpftrace [kernel.kallsyms] [k] __lock_acquire 1.51% bpftrace [kernel.kallsyms] [k] static_call_text_reserved
by checking if the address is on the kprobe blacklist:
42.70% bpftrace [kernel.kallsyms] [k] kallsyms_expand_symbol.constprop.0 | ---kallsyms_expand_symbol.constprop.0 | --42.22%--kallsyms_lookup_name within_kprobe_blacklist.part.0 check_kprobe_address register_kprobe bpf_kprobe_link_attach __sys_bpf __x64_sys_bpf do_syscall_64 entry_SYSCALL_64_after_hwframe syscall bpftrace::AttachedProbe::attach_kprobe
I could revive that patch that did bsearch on kallsyms or we could add 'do-not-check-kprobe-blacklist' unsafe mode to get more speed
jirka
Masami Hiramatsu
maintainerRe: [PATCH 02/13] kprobe: Keep traced function address
In reply to Jiri OlsaQuoted 9 lines · On Tue, 4 Jan 2022 09:09:32 +0100 Jiri Olsa <[email protected]> wrote:
The
bpf_get_func_ip_kprobehelper should return traced function address, but it's doing so only for kprobes that are placed on the function entry.If kprobe is placed within the function,
bpf_get_func_ip_kprobereturns that address instead of function entry.Storing the function entry directly in kprobe object, so it could be used in
bpf_get_func_ip_kprobehelper.
Hmm, please do this in bpf side, which should have some data structure around the kprobe itself. Do not add this "specialized" field to the kprobe data structure.
Thank you,
Quoted 97 lines
Signed-off-by Jiri Olsa <[email protected]>--- include/linux/kprobes.h | 3 +++ kernel/kprobes.c | 12 ++++++++++++ kernel/trace/bpf_trace.c | 2 +- tools/testing/selftests/bpf/progs/get_func_ip_test.c | 4 ++-- 4 files changed, 18 insertions(+), 3 deletions(-)include/linux/kprobes.h +3
-0@@ -74,6 +74,9 @@ struct kprobe {/* Offset into the symbol */ unsigned int offset; +/* traced function address */ +unsigned long func_addr; +/* Called before addr is executed. */ kprobe_pre_handler_t pre_handler;kernel/kprobes.c +12
-0@@ -1310,6 +1310,7 @@ static void init_aggr_kprobe(struct kprobe *ap, struct kprobe *p)copy_kprobe (p, ap);flush_insn_slot (ap); ap->addr = p->addr; + ap->func_addr = p->func_addr; ap->flags = p->flags & ~KPROBE_FLAG_OPTIMIZED ; ap->pre_handler = aggr_pre_handler;/* We don't care the kprobe which has gone. */ @@ -1588,6 +1589,16 @@ static int check_kprobe_address_safe(struct kprobe *p,return ret; } +static unsigned long resolve_func_addr (kprobe_opcode_t *addr) +{ +char str[KSYM_SYMBOL_LEN ]; +unsigned long offset; + +if (kallsyms_lookup ((unsigned long ) addr,NULL , &offset,NULL , str)) +return (unsigned long ) addr - offset; +return 0 ; +} +int register_kprobe (struct kprobe *p) {int ret;@@ -1600,6 +1611,7 @@ int register_kprobe(struct kprobe *p)if (IS_ERR (addr))return PTR_ERR (addr); p->addr = addr; + p->func_addr =resolve_func_addr (addr); ret =warn_kprobe_rereg (p);if (ret)kernel/trace/bpf_trace.c +1
-1@@ -1026,7 +1026,7 @@ BPF_CALL_1(bpf_get_func_ip_kprobe, struct pt_regs *, regs) {struct kprobe *kp =kprobe_running (); -return kp ? (uintptr_t )kp->addr :0 ; +return kp ? (uintptr_t )kp->func_addr :0 ; }static const struct bpf_func_proto bpf_get_func_ip_proto_kprobe = {tools/testing/selftests/bpf/progs/get_func_ip_test.c +2
-2@@ -69,7 +69,7 @@ int test6(struct pt_regs *ctx) {__u64 addr =bpf_get_func_ip (ctx); - test6_result = (const void *) addr == &bpf_fentry_test6 +5 ; + test6_result = (const void *) addr == &bpf_fentry_test6;return 0 ; }@@ -79,6 +79,6 @@ int test7(struct pt_regs *ctx) {__u64 addr =bpf_get_func_ip (ctx); - test7_result = (const void *) addr == &bpf_fentry_test7 +5 ; + test7_result = (const void *) addr == &bpf_fentry_test7;return 0 ; }Signature · 1 line
2.33.1
Signature · 1 line
Masami Hiramatsu <[email protected]>
Masami Hiramatsu
maintainerRe: [PATCH 03/13] kprobe: Add support to register multiple ftrace kprobes
In reply to Jiri OlsaQuoted 16 lines · On Tue, 4 Jan 2022 09:09:33 +0100 Jiri Olsa <[email protected]> wrote:
Adding support to register kprobe on multiple addresses within single kprobe object instance.
It uses the
CONFIG_KPROBES_ON_FTRACEfeature (so it's only available for function entry addresses) and separatedftrace_opsobject placed directly in the kprobe object.There's new
CONFIG_HAVE_KPROBES_MULTI_ON_FTRACEconfig option, enabled for archs that support multi kprobes.It registers all the provided addresses in the
ftrace_opsobject filter and adds separateftrace_opscallback.To register multi kprobe user provides array of addresses or symbols with their count, like:
Quoted 7 lines
struct kprobe kp = {};
kp.multi.symbols = (const char **) symbols;
kp.multi.cnt = cnt;
...
err = register_kprobe(&kp);I would like to keep the kprobe itself as simple as possible, which also should provide a consistent probe handling model.
I understand that you consider the overhead of having multiple probes, but as far as I can see, this implementation design smells no good, sorry.
The worst point is that the multi kprobe only supports function entry (and only available if FTRACE is enabled). Then, that is FTRACE, not kprobes.
Yes, kprobe supports probing on FTRACE by using FTRACE, but that does not mean kprobes wrapps FTRACE. That is just for "avoidance" of the limitation (because we can not put a breakpoint on self-modified code.)
If you need a probe which support multiple address but only on function entry, that should be something like 'fprobe', not kprobes.
IMHO, that should be implemented as similar but different APIs because those are simply different.
So, can't we use ftrace directly from bpf? I don't think there is no reason that the bpf sticks on kprobes APIs.
Thank you,
Quoted 480 lines
Signed-off-by Jiri Olsa <[email protected]>--- arch/Kconfig | 3 + arch/x86/Kconfig | 1 + arch/x86/kernel/kprobes/ftrace.c | 48 ++++++-- include/linux/kprobes.h | 25 ++++ kernel/kprobes.c | 204 +++++++++++++++++++++++++------ 5 files changed, 235 insertions(+), 46 deletions(-)arch/Kconfig +3
-0@@ -191,6 +191,9 @@ config HAVE_OPTPROBES config HAVE_KPROBES_ON_FTRACE bool +config HAVE_KPROBES_MULTI_ON_FTRACE + bool + config ARCH_CORRECT_STACKTRACE_ON_KRETPROBE bool helparch/x86/Kconfig +1
-0@@ -217,6 +217,7 @@ config X86 select HAVE_KERNEL_ZSTD select HAVE_KPROBES select HAVE_KPROBES_ON_FTRACE + select HAVE_KPROBES_MULTI_ON_FTRACE select HAVE_FUNCTION_ERROR_INJECTION select HAVE_KRETPROBES select HAVE_KVMarch/x86/kernel/kprobes/ftrace.c +36
-12@@ -12,22 +12,14 @@#include "common.h" -/* Ftrace callback handler for kprobes -- called under preempt disabled */ -void kprobe_ftrace_handler (unsigned long ip,unsigned long parent_ip, -struct ftrace_ops *ops,struct ftrace_regs *fregs) +static void ftrace_handler (struct kprobe *p,unsigned long ip, +struct ftrace_regs *fregs) {struct pt_regs *regs =ftrace_get_regs (fregs); -struct kprobe *p;struct kprobe_ctlblk *kcb; -int bit; - bit =ftrace_test_recursion_trylock (ip, parent_ip); -if (bit <0 ) -return ; - - p =get_kprobe ((kprobe_opcode_t *)ip);if (unlikely (!p) ||kprobe_disabled (p)) - goto out; +return ; kcb =get_kprobe_ctlblk ();if (kprobe_running ()) {@@ -57,11 +49,43 @@ void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip,*/ __this_cpu_write (current_kprobe,NULL ); } -out: +} + +/* Ftrace callback handler for kprobes -- called under preempt disabled */ +void kprobe_ftrace_handler (unsigned long ip,unsigned long parent_ip, +struct ftrace_ops *ops,struct ftrace_regs *fregs) +{ +struct kprobe *p; +int bit; + + bit =ftrace_test_recursion_trylock (ip, parent_ip); +if (bit <0 ) +return ; + + p =get_kprobe ((kprobe_opcode_t *)ip); +ftrace_handler (p, ip, fregs); +ftrace_test_recursion_unlock (bit); }NOKPROBE_SYMBOL (kprobe_ftrace_handler); +void kprobe_ftrace_multi_handler (unsigned long ip,unsigned long parent_ip, +struct ftrace_ops *ops,struct ftrace_regs *fregs) +{ +struct kprobe *p; +int bit; + + bit =ftrace_test_recursion_trylock (ip, parent_ip); +if (bit <0 ) +return ; + + p = container_of(ops, struct kprobe, multi.ops); +ftrace_handler (p, ip, fregs); + +ftrace_test_recursion_unlock (bit); +} +NOKPROBE_SYMBOL (kprobe_ftrace_multi_handler); +int arch_prepare_kprobe_ftrace (struct kprobe *p) { p->ainsn.insn =NULL ;include/linux/kprobes.h +25
-0@@ -68,6 +68,16 @@ struct kprobe {/* location of the probe point */ kprobe_opcode_t *addr; +#ifdef CONFIG_HAVE_KPROBES_MULTI_ON_FTRACE +/* location of the multi probe points */ +struct { +const char **symbols; +kprobe_opcode_t **addrs; +unsigned int cnt; +struct ftrace_ops ops; + } multi; +#endif +/* Allow user to indicate symbol name of the probe point */ const char *symbol_name;@@ -105,6 +115,7 @@ struct kprobe {* this flag is only for optimized_kprobe. */ #define KPROBE_FLAG_FTRACE 8/* probe is using ftrace */ +#define KPROBE_FLAG_MULTI 16/* probe multi addresses */ /* Has this kprobe gone ? */ static inline bool kprobe_gone (struct kprobe *p)@@ -130,6 +141,18 @@ static inline bool kprobe_ftrace(struct kprobe *p)return p->flags &KPROBE_FLAG_FTRACE ; } +/* Is this ftrace multi kprobe ? */ +static inline bool kprobe_ftrace_multi (struct kprobe *p) +{ +return kprobe_ftrace (p) && (p->flags &KPROBE_FLAG_MULTI ); +} + +/* Is this single kprobe ? */ +static inline bool kprobe_single (struct kprobe *p) +{ +return !(p->flags &KPROBE_FLAG_MULTI ); +} +/* * Function-return probe - * Note: @@ -365,6 +388,8 @@ static inline void wait_for_kprobe_optimizer(void) { }#ifdef CONFIG_KPROBES_ON_FTRACE extern void kprobe_ftrace_handler (unsigned long ip,unsigned long parent_ip,struct ftrace_ops *ops,struct ftrace_regs *fregs); +extern void kprobe_ftrace_multi_handler (unsigned long ip,unsigned long parent_ip, +struct ftrace_ops *ops,struct ftrace_regs *fregs);extern int arch_prepare_kprobe_ftrace (struct kprobe *p);#else static inline int arch_prepare_kprobe_ftrace (struct kprobe *p)kernel/kprobes.c +170
-34@@ -44,6 +44,7 @@#include <asm/cacheflush.h> #include <asm/errno.h> #include <linux/uaccess.h> +#include <linux/ftrace.h> #define KPROBE_HASH_BITS 6#define KPROBE_TABLE_SIZE (1 << KPROBE_HASH_BITS)@@ -1022,6 +1023,35 @@ static struct kprobe *alloc_aggr_kprobe(struct kprobe *p) }#endif /* CONFIG_OPTPROBES */ +static int check_kprobe_address (unsigned long addr) +{ +/* Ensure it is not in reserved area nor out of text */ +return !kernel_text_address (addr) || +within_kprobe_blacklist (addr) || +jump_label_text_reserved ((void *) addr, (void *) addr) || +static_call_text_reserved ((void *) addr, (void *) addr) || +find_bug (addr); +} + +static int check_ftrace_location (unsigned long addr,struct kprobe *p) +{ +unsigned long ftrace_addr; + + ftrace_addr =ftrace_location (addr); +if (ftrace_addr) { +#ifdef CONFIG_KPROBES_ON_FTRACE +/* Given address is not on the instruction boundary */ +if (addr != ftrace_addr) +return -EILSEQ ; +if (p) + p->flags |=KPROBE_FLAG_FTRACE ; +#else /* !CONFIG_KPROBES_ON_FTRACE */ +return -EINVAL ; +#endif + } +return 0 ; +} +#ifdef CONFIG_KPROBES_ON_FTRACE static struct ftrace_ops kprobe_ftrace_ops__read_mostly = { .func = kprobe_ftrace_handler,@@ -1043,6 +1073,13 @@ static int __arm_kprobe_ftrace(struct kprobe *p, struct ftrace_ops *ops,lockdep_assert_held (&kprobe_mutex); +#ifdef CONFIG_HAVE_KPROBES_MULTI_ON_FTRACE +if (kprobe_ftrace_multi (p)) { + ret =register_ftrace_function (&p->multi.ops); +WARN (ret <0 ,"Failed to register kprobe-multi-ftrace (error %d)\n" , ret); +return ret; + } +#endif ret =ftrace_set_filter_ip (ops, (unsigned long )p->addr,0 ,0 );if (WARN_ONCE (ret <0 ,"Failed to arm kprobe-ftrace at %pS (error %d)\n" , p->addr, ret))return ret;@@ -1081,6 +1118,13 @@ static int __disarm_kprobe_ftrace(struct kprobe *p, struct ftrace_ops *ops,lockdep_assert_held (&kprobe_mutex); +#ifdef CONFIG_HAVE_KPROBES_MULTI_ON_FTRACE +if (kprobe_ftrace_multi (p)) { + ret =unregister_ftrace_function (&p->multi.ops); +WARN (ret <0 ,"Failed to unregister kprobe-ftrace (error %d)\n" , ret); +return ret; + } +#endif if (*cnt ==1 ) { ret =unregister_ftrace_function (ops);if (WARN (ret <0 ,"Failed to unregister kprobe-ftrace (error %d)\n" , ret))@@ -1103,6 +1147,94 @@ static int disarm_kprobe_ftrace(struct kprobe *p) ipmodify ? &kprobe_ipmodify_ops : &kprobe_ftrace_ops, ipmodify ? &kprobe_ipmodify_enabled : &kprobe_ftrace_enabled); } + +#ifdef CONFIG_HAVE_KPROBES_MULTI_ON_FTRACE +/* +* In addition to standard kprobe address check for multi +* ftrace kprobes we also allow only: +* - ftrace managed function entry address +* - kernel core only address +*/ +static unsigned long check_ftrace_addr (unsigned long addr) +{ +int err; + +if (!addr) +return -EINVAL ; + err =check_ftrace_location (addr,NULL ); +if (err) +return err; +if (check_kprobe_address (addr)) +return -EINVAL ; +if (__module_text_address (addr)) +return -EINVAL ; +return 0 ; +} + +static int check_ftrace_multi (struct kprobe *p) +{ +kprobe_opcode_t **addrs = p->multi.addrs; +const char **symbols = p->multi.symbols; +unsigned int i, cnt = p->multi.cnt; +unsigned long addr, *ips; +int err; + +if ((symbols && addrs) || (!symbols && !addrs)) +return -EINVAL ; + +/* do we want sysctl for this? */ +if (cnt >=20000 ) +return -E2BIG ; + + ips =kmalloc (sizeof (*ips) * cnt,GFP_KERNEL ); +if (!ips) +return -ENOMEM ; + +for (i =0 ; i < cnt; i++) { +if (symbols) + addr = (unsigned long )kprobe_lookup_name (symbols[i],0 ); +else + addr = (unsigned long ) addrs[i]; + ips[i] = addr; + } + +jump_label_lock (); +preempt_disable (); + +for (i =0 ; i < cnt; i++) { + err =check_ftrace_addr (ips[i]); +if (err) +break ; + } + +preempt_enable (); +jump_label_unlock (); + +if (err) + goto out; + + err =ftrace_set_filter_ips (&p->multi.ops, ips, cnt,0 ,0 ); +if (err) + goto out; + + p->multi.ops.func = kprobe_ftrace_multi_handler; + p->multi.ops.flags =FTRACE_OPS_FL_SAVE_REGS |FTRACE_OPS_FL_DYNAMIC ; + + p->flags |=KPROBE_FLAG_MULTI |KPROBE_FLAG_FTRACE ; +if (p->post_handler) + p->multi.ops.flags |=FTRACE_OPS_FL_IPMODIFY ; + +out: +kfree (ips); +return err; +} + +static void free_ftrace_multi (struct kprobe *p) +{ +ftrace_free_filter (&p->multi.ops); +} +#endif +#else /* !CONFIG_KPROBES_ON_FTRACE */ static inline int arm_kprobe_ftrace (struct kprobe *p) {@@ -1489,6 +1621,9 @@ static struct kprobe *__get_valid_kprobe(struct kprobe *p)lockdep_assert_held (&kprobe_mutex); +if (kprobe_ftrace_multi (p)) +return p; + ap =get_kprobe (p->addr);if (unlikely (!ap))return NULL ;@@ -1520,41 +1655,18 @@ static inline int warn_kprobe_rereg(struct kprobe *p)return ret; } -static int check_ftrace_location (struct kprobe *p) -{ -unsigned long ftrace_addr; - - ftrace_addr =ftrace_location ((unsigned long )p->addr); -if (ftrace_addr) { -#ifdef CONFIG_KPROBES_ON_FTRACE -/* Given address is not on the instruction boundary */ -if ((unsigned long )p->addr != ftrace_addr) -return -EILSEQ ; - p->flags |=KPROBE_FLAG_FTRACE ; -#else /* !CONFIG_KPROBES_ON_FTRACE */ -return -EINVAL ; -#endif - } -return 0 ; -} -static int check_kprobe_address_safe (struct kprobe *p,struct module **probed_mod) {int ret; - ret =check_ftrace_location (p); + ret =check_ftrace_location ((unsigned long ) p->addr, p);if (ret)return ret;jump_label_lock ();preempt_disable (); -/* Ensure it is not in reserved area nor out of text */ -if (!kernel_text_address ((unsigned long ) p->addr) || -within_kprobe_blacklist ((unsigned long ) p->addr) || -jump_label_text_reserved (p->addr, p->addr) || -static_call_text_reserved (p->addr, p->addr) || -find_bug ((unsigned long )p->addr)) { +if (check_kprobe_address ((unsigned long ) p->addr)) { ret = -EINVAL ; goto out; }@@ -1599,13 +1711,16 @@ static unsigned long resolve_func_addr(kprobe_opcode_t *addr)return 0 ; } -int register_kprobe (struct kprobe *p) +static int check_addr (struct kprobe *p,struct module **probed_mod) {int ret; -struct kprobe *old_p; -struct module *probed_mod;kprobe_opcode_t *addr; +#ifdef CONFIG_HAVE_KPROBES_MULTI_ON_FTRACE +if (p->multi.cnt) +return check_ftrace_multi (p); +#endif +/* Adjust probe address from symbol */ addr =kprobe_addr (p);if (IS_ERR (addr))@@ -1616,13 +1731,21 @@ int register_kprobe(struct kprobe *p) ret =warn_kprobe_rereg (p);if (ret)return ret; +return check_kprobe_address_safe (p, probed_mod); +} + +int register_kprobe (struct kprobe *p) +{ +struct module *probed_mod =NULL ; +struct kprobe *old_p; +int ret;/* User can pass only KPROBE_FLAG_DISABLED to register_kprobe */ p->flags &=KPROBE_FLAG_DISABLED ; p->nmissed =0 ;INIT_LIST_HEAD (&p->list); - ret =check_kprobe_address_safe (p, &probed_mod); + ret =check_addr (p, &probed_mod);if (ret)return ret;@@ -1644,14 +1767,21 @@ int register_kprobe(struct kprobe *p)if (ret) goto out; -INIT_HLIST_NODE (&p->hlist); -hlist_add_head_rcu (&p->hlist, - &kprobe_table[hash_ptr (p->addr,KPROBE_HASH_BITS )]); +/* +* Multi ftrace kprobes do not have single address, +* so they are not stored in the kprobe_table hash. +*/ +if (kprobe_single (p)) { +INIT_HLIST_NODE (&p->hlist); +hlist_add_head_rcu (&p->hlist, + &kprobe_table[hash_ptr (p->addr,KPROBE_HASH_BITS )]); + }if (!kprobes_all_disarmed && !kprobe_disabled (p)) { ret =arm_kprobe (p);if (ret) { -hlist_del_rcu (&p->hlist); +if (kprobe_single (p)) +hlist_del_rcu (&p->hlist);synchronize_rcu (); goto out; }@@ -1778,7 +1908,13 @@ static int __unregister_kprobe_top(struct kprobe *p)return 0 ; disarmed: -hlist_del_rcu (&ap->hlist); +if (kprobe_single (ap)) +hlist_del_rcu (&ap->hlist); + +#ifdef CONFIG_HAVE_KPROBES_MULTI_ON_FTRACE +if (kprobe_ftrace_multi (ap)) +free_ftrace_multi (ap); +#endif return 0 ; }Signature · 1 line
2.33.1
Signature · 1 line
Masami Hiramatsu <[email protected]>
Masami Hiramatsu
maintainerQuoted 6 lines · On Tue, 4 Jan 2022 09:09:30 +0100 Jiri Olsa <[email protected]> wrote:
hi,
adding support to attach multiple kprobes within single syscall and speed up attachment of many kprobes.The previous attempt [1] wasn't fast enough, so coming with new approach that adds new kprobe interface.
Yes, since register_kprobes() just registers multiple kprobes on array. This is designed for dozens of kprobes.
Quoted 2 lines
The attachment speed of of this approach (tested in bpftrace) is now comparable to ftrace tracer attachment speed.. fast ;-)
Yes, because that if ftrace, not kprobes.
Quoted 3 lines
The limit of this approach is forced by using ftrace as attach layer, so it allows only kprobes on function's entry (plus return probes).
Note that you also need to multiply the number of instances.
Quoted 9 lines
This patchset contains:
- kprobes support to register multiple kprobes with current kprobe API (patches 1 - 8)
- bpf support ot create new kprobe link allowing to attach multiple addresses (patches 9 - 14)We don't need to care about multiple probes on same functions because it's taken care on the
ftrace_opslayer.
Hmm, I think there may be a time to split the "kprobe as an interface for the software breakpoint" and "kprobe as a wrapper interface for the callbacks of various instrumentations", like 'raw_kprobe'(or kswbp) and 'kprobes'.
And this may be called as 'fprobe' as ftrace_ops wrapper. (But if the bpf is enough flexible, this kind of intermediate layer
may not be needed, it can use ftrace_ops directly, eventually)
Jiri, have you already considered to use ftrace_ops from the bpf directly? Are there any issues?
(bpf depends on 'kprobe' widely?)
Thank you,
Signature · 1 line
Masami Hiramatsu <[email protected]>
Re: [PATCH 02/13] kprobe: Keep traced function address
In reply to Jiri OlsaQuoted 24 lines · Jiri Olsa
The
bpf_get_func_ip_kprobehelper should return traced function address, but it's doing so only for kprobes that are placed on the function entry.If kprobe is placed within the function,
bpf_get_func_ip_kprobereturns that address instead of function entry.Storing the function entry directly in kprobe object, so it could be used in
bpf_get_func_ip_kprobehelper.Signed-off-by Jiri Olsa <[email protected]>--- include/linux/kprobes.h | 3 +++ kernel/kprobes.c | 12 ++++++++++++ kernel/trace/bpf_trace.c | 2 +- tools/testing/selftests/bpf/progs/get_func_ip_test.c | 4 ++-- 4 files changed, 18 insertions(+), 3 deletions(-)include/linux/kprobes.h +0
-0
include/linux/kprobes.h struct kprobe { · line 79
Quoted 6 lines
include/linux/kprobes.h +3
-0@@ -74,6 +74,9 @@ struct kprobe {/* Offset into the symbol */ unsigned int offset; +/* traced function address */ +unsigned long func_addr; +
keep in mind that we'll also need (maybe in a follow up series) to store bpf_cookie somewhere close to this func_addr as well. Just mentioning to keep in mind as you decide with Masami where to put it.
Quoted 67 lines
include/linux/kprobes.h +0
-0/* Called before addr is executed. */ kprobe_pre_handler_t pre_handler;kernel/kprobes.c +12
-0@@ -1310,6 +1310,7 @@ static void init_aggr_kprobe(struct kprobe *ap, struct kprobe *p)copy_kprobe (p, ap);flush_insn_slot (ap); ap->addr = p->addr; + ap->func_addr = p->func_addr; ap->flags = p->flags & ~KPROBE_FLAG_OPTIMIZED ; ap->pre_handler = aggr_pre_handler;/* We don't care the kprobe which has gone. */ @@ -1588,6 +1589,16 @@ static int check_kprobe_address_safe(struct kprobe *p,return ret; } +static unsigned long resolve_func_addr (kprobe_opcode_t *addr) +{ +char str[KSYM_SYMBOL_LEN ]; +unsigned long offset; + +if (kallsyms_lookup ((unsigned long ) addr,NULL , &offset,NULL , str)) +return (unsigned long ) addr - offset; +return 0 ; +} +int register_kprobe (struct kprobe *p) {int ret;@@ -1600,6 +1611,7 @@ int register_kprobe(struct kprobe *p)if (IS_ERR (addr))return PTR_ERR (addr); p->addr = addr; + p->func_addr =resolve_func_addr (addr); ret =warn_kprobe_rereg (p);if (ret)kernel/trace/bpf_trace.c +1
-1@@ -1026,7 +1026,7 @@ BPF_CALL_1(bpf_get_func_ip_kprobe, struct pt_regs *, regs) {struct kprobe *kp =kprobe_running (); -return kp ? (uintptr_t )kp->addr :0 ; +return kp ? (uintptr_t )kp->func_addr :0 ; }static const struct bpf_func_proto bpf_get_func_ip_proto_kprobe = {tools/testing/selftests/bpf/progs/get_func_ip_test.c +1
-1@@ -69,7 +69,7 @@ int test6(struct pt_regs *ctx) {__u64 addr =bpf_get_func_ip (ctx); - test6_result = (const void *) addr == &bpf_fentry_test6 +5 ; + test6_result = (const void *) addr == &bpf_fentry_test6;return 0 ; }
tools/testing/selftests/bpf/progs/get_func_ip_test.c int test7(struct pt_regs *ctx) · line 82
Quoted 5 lines
tools/testing/selftests/bpf/progs/get_func_ip_test.c +1
-1@@ -79,6 +79,6 @@ int test7(struct pt_regs *ctx) {__u64 addr =bpf_get_func_ip (ctx); - test7_result = (const void *) addr == &bpf_fentry_test7 +5 ; + test7_result = (const void *) addr == &bpf_fentry_test7;
we can treat this as a bug fix for bpf_get_func_ip() for kprobes, right? I think "Fixes: " tag is in order then.
Quoted 6 lines
tools/testing/selftests/bpf/progs/get_func_ip_test.c +0
-1return 0 ; } --2.33.1
Re: [PATCH 05/13] kprobe: Allow to get traced function address for multi ftrace kprobes
In reply to Jiri OlsaQuoted 33 lines · Jiri Olsa
The current
bpf_get_func_ip_kprobehelper does not work properly, when used in ebpf program triggered by the new multi kprobes.We can't use kprobe's
func_addrinbpf_get_func_ip_kprobehelper, because there are multiple functions registered for single kprobe object.Adding new per cpu variable
current_ftrace_multi_addrand extra address inkretprobe_instanceobject to keep current traced function address for each cpu for both kprobe handler and kretprobe trampoline.The address value is set/passed as follows, for kprobe:
kprobe_ftrace_multi_handler { old = kprobe_ftrace_multi_addr_set(ip); handler.. kprobe_ftrace_multi_addr_set(old); }For kretprobe:
kprobe_ftrace_multi_handler { old = kprobe_ftrace_multi_addr_set(ip); ... pre_handler_kretprobe { ri->ftrace_multi_addr = kprobe_ftrace_multi_addr } ... kprobe_ftrace_multi_addr_set(old);
Quoted 9 lines
}
__kretprobe_trampoline_handler
{
prev_func_addr = kprobe_ftrace_multi_addr_set(ri->ftrace_multi_addr);
handler..
kprobe_ftrace_multi_addr_set(prev_func_addr);
}Is it possible to record or calculate the multi-kprobe "instance index" (i.e., which position in addrs array did we get triggered for)? If yes, then storing that index would allow to fetch both IP and cookie value with just one per-cpu variable.
Quoted 8 lines
Signed-off-by Jiri Olsa <[email protected]>--- arch/x86/kernel/kprobes/ftrace.c | 3 +++ include/linux/kprobes.h | 26 ++++++++++++++++++++++++++ kernel/kprobes.c | 6 ++++++ kernel/trace/bpf_trace.c | 7 ++++++- 4 files changed, 41 insertions(+), 1 deletion(-)
[...]
Re: [PATCH 08/13] bpf: Add kprobe link for attaching raw kprobes
In reply to Jiri OlsaQuoted 7 lines · Jiri Olsa
Adding new link type
BPF_LINK_TYPE_KPROBEto attach kprobes directly through register_kprobe/kretprobe API.Adding new attach type
BPF_TRACE_RAW_KPROBEthat enables such link for kprobe program.
Quoted 7 lines
The new link allows to create multiple kprobes link by using new
link_createinterface:struct { __aligned_u64 addrs; __u32 cnt; __u64 bpf_cookie;
I'm afraid bpf_cookie has to be different for each addr, otherwise it's severely limiting. So it would be an array of cookies alongside an array of addresses
Quoted 13 lines
} kprobe;Plus new flag
BPF_F_KPROBE_RETURNfor link_create.flags to create return probe.Signed-off-by Jiri Olsa <[email protected]>--- include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 12 +++ kernel/bpf/syscall.c | 191 ++++++++++++++++++++++++++++++++- tools/include/uapi/linux/bpf.h | 12 +++ 4 files changed, 211 insertions(+), 5 deletions(-)
[...]
include/uapi/linux/bpf.h enum bpf_link_type { · line 1120
Quoted 9 lines
include/uapi/linux/bpf.h +5
-0@@ -1111,6 +1113,11 @@ enum bpf_link_type {*/ #define BPF_F_SLEEPABLE (1U << 4) +/* link_create flags used in LINK_CREATE command for BPF_TRACE_RAW_KPROBE +* attach type. +*/ +#define BPF_F_KPROBE_RETURN (1U << 0) +
we have plenty of flexibility to have per-link type fields, so why not add bool is_retprobe next to addrs and cnt?
Quoted 15 lines
include/uapi/linux/bpf.h +5
-0/* When BPF ldimm64's insn[0].src_reg != 0 then this can have * the following extensions: * @@ -1465,6 +1472,11 @@ union bpf_attr {*/ __u64 bpf_cookie; } perf_event; +struct { +__aligned_u64 addrs; +__u32 cnt; +__u64 bpf_cookie; + } kprobe; }; } link_create;
[...]
Quoted 42 lines · Masami Hiramatsu
Quoted 6 lines · On Tue, 4 Jan 2022 09:09:30 +0100 Jiri Olsa <[email protected]> wrote:
hi,
adding support to attach multiple kprobes within single syscall and speed up attachment of many kprobes.The previous attempt [1] wasn't fast enough, so coming with new approach that adds new kprobe interface.
Yes, since
register_kprobes()just registers multiple kprobes on array. This is designed for dozens of kprobes.Quoted 2 lines
The attachment speed of of this approach (tested in bpftrace) is now comparable to ftrace tracer attachment speed.. fast ;-)
Yes, because that if ftrace, not kprobes.
Quoted 3 lines
The limit of this approach is forced by using ftrace as attach layer, so it allows only kprobes on function's entry (plus return probes).
Note that you also need to multiply the number of instances.
Quoted 9 lines
This patchset contains:
- kprobes support to register multiple kprobes with current kprobe API (patches 1 - 8)
- bpf support ot create new kprobe link allowing to attach multiple addresses (patches 9 - 14)We don't need to care about multiple probes on same functions because it's taken care on the
ftrace_opslayer.Hmm, I think there may be a time to split the "kprobe as an interface for the software breakpoint" and "kprobe as a wrapper interface for the callbacks of various instrumentations", like 'raw_kprobe'(or kswbp) and 'kprobes'.
And this may be called as 'fprobe' asftrace_opswrapper. (But if the bpf is enough flexible, this kind of intermediate layer
may not be needed, it can useftrace_opsdirectly, eventually)
Quoted 3 lines
Jiri, have you already considered to use
ftrace_opsfrom the bpf directly? Are there any issues?
(bpf depends on 'kprobe' widely?)
at the moment there's not ftrace public interface for the return probe merged in, so to get the kretprobe working I had to use kprobe interface
but.. there are patches Steven shared some time ago, that do that and make graph_ops available as kernel interface
I recall we considered graph_ops interface before as common attach layer for trampolines, which was bad, but it might actually make sense for kprobes
I'll need to check it in more details but I think both graph_ops and kprobe do about similar thing wrt hooking return probe, so it should be comparable.. and they are already doing the same for the entry hook, because kprobe is mostly using ftrace for that
we would not need to introduce new program type - kprobe programs should be able to run from ftrace callbacks just fine
so we would have:
- kprobe type programs attaching to:
- new BPF_LINK_TYPE_FPROBE link using the graph_ops as attachment layer
jirka
Re: [PATCH 02/13] kprobe: Keep traced function address
In reply to Masami HiramatsuQuoted 10 lines · Masami Hiramatsu
Quoted 7 lines · On Tue, 4 Jan 2022 09:09:32 +0100 Jiri Olsa <[email protected]> wrote:
The
bpf_get_func_ip_kprobehelper should return traced function address, but it's doing so only for kprobes that are placed on the function entry.If kprobe is placed within the function,
bpf_get_func_ip_kprobereturns that address instead of function entry.
Quoted 6 lines
Quoted 2 lines
Storing the function entry directly in kprobe object, so it could be used in
bpf_get_func_ip_kprobehelper.Hmm, please do this in bpf side, which should have some data structure around the kprobe itself. Do not add this "specialized" field to the kprobe data structure.
ok, will check
thanks,
jirka
Quoted 105 lines
Thank you,
Quoted 97 lines
Signed-off-by Jiri Olsa <[email protected]>--- include/linux/kprobes.h | 3 +++ kernel/kprobes.c | 12 ++++++++++++ kernel/trace/bpf_trace.c | 2 +- tools/testing/selftests/bpf/progs/get_func_ip_test.c | 4 ++-- 4 files changed, 18 insertions(+), 3 deletions(-)include/linux/kprobes.h +3
-0@@ -74,6 +74,9 @@ struct kprobe {/* Offset into the symbol */ unsigned int offset; +/* traced function address */ +unsigned long func_addr; +/* Called before addr is executed. */ kprobe_pre_handler_t pre_handler;kernel/kprobes.c +12
-0@@ -1310,6 +1310,7 @@ static void init_aggr_kprobe(struct kprobe *ap, struct kprobe *p)copy_kprobe (p, ap);flush_insn_slot (ap); ap->addr = p->addr; + ap->func_addr = p->func_addr; ap->flags = p->flags & ~KPROBE_FLAG_OPTIMIZED ; ap->pre_handler = aggr_pre_handler;/* We don't care the kprobe which has gone. */ @@ -1588,6 +1589,16 @@ static int check_kprobe_address_safe(struct kprobe *p,return ret; } +static unsigned long resolve_func_addr (kprobe_opcode_t *addr) +{ +char str[KSYM_SYMBOL_LEN ]; +unsigned long offset; + +if (kallsyms_lookup ((unsigned long ) addr,NULL , &offset,NULL , str)) +return (unsigned long ) addr - offset; +return 0 ; +} +int register_kprobe (struct kprobe *p) {int ret;@@ -1600,6 +1611,7 @@ int register_kprobe(struct kprobe *p)if (IS_ERR (addr))return PTR_ERR (addr); p->addr = addr; + p->func_addr =resolve_func_addr (addr); ret =warn_kprobe_rereg (p);if (ret)kernel/trace/bpf_trace.c +1
-1@@ -1026,7 +1026,7 @@ BPF_CALL_1(bpf_get_func_ip_kprobe, struct pt_regs *, regs) {struct kprobe *kp =kprobe_running (); -return kp ? (uintptr_t )kp->addr :0 ; +return kp ? (uintptr_t )kp->func_addr :0 ; }static const struct bpf_func_proto bpf_get_func_ip_proto_kprobe = {tools/testing/selftests/bpf/progs/get_func_ip_test.c +2
-2@@ -69,7 +69,7 @@ int test6(struct pt_regs *ctx) {__u64 addr =bpf_get_func_ip (ctx); - test6_result = (const void *) addr == &bpf_fentry_test6 +5 ; + test6_result = (const void *) addr == &bpf_fentry_test6;return 0 ; }@@ -79,6 +79,6 @@ int test7(struct pt_regs *ctx) {__u64 addr =bpf_get_func_ip (ctx); - test7_result = (const void *) addr == &bpf_fentry_test7 +5 ; + test7_result = (const void *) addr == &bpf_fentry_test7;return 0 ; }Signature · 1 line
2.33.1Signature · 1 line
Masami Hiramatsu <[email protected]>
Re: [PATCH 02/13] kprobe: Keep traced function address
In reply to Andrii NakryikoQuoted 28 lines · Andrii Nakryiko
Quoted 27 lines · Jiri Olsa
The
bpf_get_func_ip_kprobehelper should return traced function address, but it's doing so only for kprobes that are placed on the function entry.If kprobe is placed within the function,
bpf_get_func_ip_kprobereturns that address instead of function entry.Storing the function entry directly in kprobe object, so it could be used in
bpf_get_func_ip_kprobehelper.Signed-off-by Jiri Olsa <[email protected]>--- include/linux/kprobes.h | 3 +++ kernel/kprobes.c | 12 ++++++++++++ kernel/trace/bpf_trace.c | 2 +- tools/testing/selftests/bpf/progs/get_func_ip_test.c | 4 ++-- 4 files changed, 18 insertions(+), 3 deletions(-)include/linux/kprobes.h +0
-0@@ -74,6 +74,9 @@ struct kprobe {/* Offset into the symbol */ unsigned int offset;
Quoted 7 lines
include/linux/kprobes.h struct kprobe { · line 79
Quoted 3 lines
include/linux/kprobes.h +3
-0+/* traced function address */ +unsigned long func_addr; +keep in mind that we'll also need (maybe in a follow up series) to store
bpf_cookiesomewhere close to thisfunc_addras well. Just mentioning to keep in mind as you decide with Masami where to put it.
ok
SNIP
Quoted 30 lines
Quoted 30 lines
kernel/trace/bpf_trace.c +1
-1@@ -1026,7 +1026,7 @@ BPF_CALL_1(bpf_get_func_ip_kprobe, struct pt_regs *, regs) {struct kprobe *kp =kprobe_running (); -return kp ? (uintptr_t )kp->addr :0 ; +return kp ? (uintptr_t )kp->func_addr :0 ; }static const struct bpf_func_proto bpf_get_func_ip_proto_kprobe = {tools/testing/selftests/bpf/progs/get_func_ip_test.c +1
-1@@ -69,7 +69,7 @@ int test6(struct pt_regs *ctx) {__u64 addr =bpf_get_func_ip (ctx); - test6_result = (const void *) addr == &bpf_fentry_test6 +5 ; + test6_result = (const void *) addr == &bpf_fentry_test6;return 0 ; }@@ -79,6 +79,6 @@ int test7(struct pt_regs *ctx) {__u64 addr =bpf_get_func_ip (ctx);
Quoted 5 lines
tools/testing/selftests/bpf/progs/get_func_ip_test.c int test7(struct pt_regs *ctx) · line 82
Quoted 2 lines
tools/testing/selftests/bpf/progs/get_func_ip_test.c +1
-1- test7_result = (const void *) addr == &bpf_fentry_test7 +5 ; + test7_result = (const void *) addr == &bpf_fentry_test7;we can treat this as a bug fix for
bpf_get_func_ip()for kprobes, right? I think "Fixes: " tag is in order then.
true, will add that in next version
thanks,
jirka
Re: [PATCH 08/13] bpf: Add kprobe link for attaching raw kprobes
In reply to Andrii NakryikoQuoted 11 lines · Andrii Nakryiko
Quoted 10 lines · Jiri Olsa
Adding new link type
BPF_LINK_TYPE_KPROBEto attach kprobes directly through register_kprobe/kretprobe API.Adding new attach type
BPF_TRACE_RAW_KPROBEthat enables such link for kprobe program.The new link allows to create multiple kprobes link by using new
link_createinterface:
Quoted 8 lines
Quoted 4 lines
struct { __aligned_u64 addrs; __u32 cnt; __u64 bpf_cookie;I'm afraid
bpf_cookiehas to be different for each addr, otherwise it's severely limiting. So it would be an array of cookies alongside an array of addresses
ok
Quoted 21 lines
Quoted 14 lines
} kprobe;Plus new flag
BPF_F_KPROBE_RETURNfor link_create.flags to create return probe.Signed-off-by Jiri Olsa <[email protected]>--- include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 12 +++ kernel/bpf/syscall.c | 191 ++++++++++++++++++++++++++++++++- tools/include/uapi/linux/bpf.h | 12 +++ 4 files changed, 211 insertions(+), 5 deletions(-)[...]
Quoted 4 lines
include/uapi/linux/bpf.h +0
-0@@ -1111,6 +1113,11 @@ enum bpf_link_type {*/ #define BPF_F_SLEEPABLE (1U << 4)
Quoted 8 lines
include/uapi/linux/bpf.h enum bpf_link_type { · line 1120
Quoted 5 lines
include/uapi/linux/bpf.h +5
-0+/* link_create flags used in LINK_CREATE command for BPF_TRACE_RAW_KPROBE +* attach type. +*/ +#define BPF_F_KPROBE_RETURN (1U << 0) +we have plenty of flexibility to have per-link type fields, so why not add
bool is_retprobenext to addrs and cnt?
well I thought if I do that, people would suggest to use the empty flags field instead ;-)
we can move it there as you suggest, but I wonder it's good idea to use bool in uapi headers, because the bool size definition is vague
jirka
Quoted 19 lines
Quoted 16 lines
include/uapi/linux/bpf.h +5
-0/* When BPF ldimm64's insn[0].src_reg != 0 then this can have * the following extensions: * @@ -1465,6 +1472,11 @@ union bpf_attr {*/ __u64 bpf_cookie; } perf_event; +struct { +__aligned_u64 addrs; +__u32 cnt; +__u64 bpf_cookie; + } kprobe; }; } link_create;[...]
Masami Hiramatsu
maintainerQuoted 43 lines · On Thu, 6 Jan 2022 09:29:02 +0100 Jiri Olsa <[email protected]> wrote:
Quoted 42 lines · Masami Hiramatsu
Quoted 6 lines · On Tue, 4 Jan 2022 09:09:30 +0100 Jiri Olsa <[email protected]> wrote:
hi,
adding support to attach multiple kprobes within single syscall and speed up attachment of many kprobes.The previous attempt [1] wasn't fast enough, so coming with new approach that adds new kprobe interface.
Yes, since
register_kprobes()just registers multiple kprobes on array. This is designed for dozens of kprobes.Quoted 2 lines
The attachment speed of of this approach (tested in bpftrace) is now comparable to ftrace tracer attachment speed.. fast ;-)
Yes, because that if ftrace, not kprobes.
Quoted 3 lines
The limit of this approach is forced by using ftrace as attach layer, so it allows only kprobes on function's entry (plus return probes).
Note that you also need to multiply the number of instances.
Quoted 9 lines
This patchset contains:
- kprobes support to register multiple kprobes with current kprobe API (patches 1 - 8)
- bpf support ot create new kprobe link allowing to attach multiple addresses (patches 9 - 14)We don't need to care about multiple probes on same functions because it's taken care on the
ftrace_opslayer.Hmm, I think there may be a time to split the "kprobe as an interface for the software breakpoint" and "kprobe as a wrapper interface for the callbacks of various instrumentations", like 'raw_kprobe'(or kswbp) and 'kprobes'.
And this may be called as 'fprobe' asftrace_opswrapper. (But if the bpf is enough flexible, this kind of intermediate layer
may not be needed, it can useftrace_opsdirectly, eventually)
Quoted 7 lines
Quoted 3 lines
Jiri, have you already considered to use
ftrace_opsfrom the bpf directly? Are there any issues?
(bpf depends on 'kprobe' widely?)at the moment there's not ftrace public interface for the return probe merged in, so to get the kretprobe working I had to use kprobe interface
Yeah, I found that too. We have to ask Steve to salvage it ;)
Quoted 6 lines
but.. there are patches Steven shared some time ago, that do that and make
graph_opsavailable as kernel interfaceI recall we considered
graph_opsinterface before as common attach layer for trampolines, which was bad, but it might actually make sense for kprobes
I started working on making 'fprobe' which will provide multiple function probe with similar interface of kprobes. See attached patch. Then you can use it in bpf, maybe with an union like
union {
struct kprobe kp; // for function body
struct fprobe fp; // for function entry and return
};
At this moment, fprobe only support entry_handler, but when we re-start the generic graph_ops interface, it is easy to expand to support exit_handler.
If this works, I think kretprobe can be phased out, since at that moment, kprobe_event can replace it with the fprobe exit_handler. (This is a benefit of decoupling the instrumentation layer from the event layer. It can choose the best way without changing user interface.)
Quoted 7 lines
I'll need to check it in more details but I think both
graph_opsand kprobe do about similar thing wrt hooking return probe, so it should be comparable.. and they are already doing the same for the entry hook, because kprobe is mostly using ftrace for thatwe would not need to introduce new program type - kprobe programs should be able to run from ftrace callbacks just fine
That seems to bind your mind. The program type is just a programing 'model' of the bpf. You can choose the best implementation to provide equal functionality. 'kprobe' in bpf is just a name that you call some instrumentations which can probe kernel code.
Thank you,
Quoted 7 lines
so we would have:
- kprobe type programs attaching to:
- newBPF_LINK_TYPE_FPROBElink using thegraph_opsas attachment layerjirka
Signature · 1 line
Masami Hiramatsu <[email protected]>
From 269b86597c166d6d4c5dd564168237603533165a Mon Sep 17 00:00:00 2001
From: Masami Hiramatsu <[email protected]>
Date: Thu, 6 Jan 2022 15:40:36 +0900
Subject: [PATCH] fprobe: Add ftrace based probe APIs
The fprobe is a wrapper API for ftrace function tracer. Unlike kprobes, this probes only supports the function entry, but it can probe multiple functions by one fprobe. The usage is almost same as the kprobe, user will specify the function names by fprobe::syms, the number of syms by fprobe::nsyms, and the user handler by fprobe::handler.
struct fprobe = { 0 };
const char *targets[] = {"func1", "func2", "func3"};
fprobe.handler = user_handler;
fprobe.nsyms = ARRAY_SIZE(targets);
fprobe.syms = targets;
ret = register_fprobe(&fprobe);
...
--- include/linux/fprobes.h | 52 ++++++++++++++++ kernel/trace/Kconfig | 10 ++++ kernel/trace/Makefile | 1 + kernel/trace/fprobes.c | 128 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 191 insertions(+) create mode 100644 include/linux/fprobes.h create mode 100644 kernel/trace/fprobes.c
include/linux/fprobes.h new file +52 -0
@@ -0,0 +1,52 @@ +#ifndef _LINUX_FPROBES_H +#define _LINUX_FPROBES_H +/* Simple ftrace probe wrapper */ + +#include <linux/compiler.h> +#include <linux/ftrace.h> + +struct fprobe { +const char **syms; +unsigned long *addrs; +unsigned int nsyms; + +struct ftrace_ops ftrace; +unsigned long nmissed; +unsigned int flags; +void (*handler) (struct fprobe *,struct pt_regs *); +}; + +#define FPROBE_FL_DISABLED 1 + +static inline bool fprobe_disabled (struct fprobe *fp) +{ +return (fp) ? fp->flags &FPROBE_FL_DISABLED : false; +} + +#ifdef CONFIG_FPROBES +int register_fprobe (struct fprobe *fp); +int unregister_fprobe (struct fprobe *fp); +#else +static inline int register_fprobe (struct fprobe *fp) +{ +return -ENOTSUPP ; +} +static inline int unregister_fprobe (struct fprobe *fp) +{ +return -ENOTSUPP ; +} +#endif + +static inline void disable_fprobe (struct fprobe *fp) +{ +if (fp) + fp->flags |=FPROBE_FL_DISABLED ; +} + +static inline void enable_fprobe (struct fprobe *fp) +{ +if (fp) + fp->flags &= ~FPROBE_FL_DISABLED ; +} + +#endif
kernel/trace/Kconfig +10 -0
@@ -223,6 +223,16 @@ config DYNAMIC_FTRACE_WITH_ARGS depends on DYNAMIC_FTRACE depends on HAVE_DYNAMIC_FTRACE_WITH_ARGS +config FPROBES + bool "Kernel Function Probe (fprobe)" + depends on FUNCTION_TRACER + depends on DYNAMIC_FTRACE_WITH_REGS + default n + help + This option enables kernel function probe feature, which is + similar to kprobes, but probes only for kernel function entries + and it can probe multiple functions by one fprobe. + config FUNCTION_PROFILER bool "Kernel function profiler" depends on FUNCTION_TRACER
kernel/trace/Makefile +1 -0
@@ -97,6 +97,7 @@ obj-$(CONFIG_PROBE_EVENTS) += trace_probe.o obj-$(CONFIG_UPROBE_EVENTS) += trace_uprobe.o obj-$(CONFIG_BOOTTIME_TRACING) += trace_boot.o obj-$(CONFIG_FTRACE_RECORD_RECURSION) += trace_recursion_record.o +obj-$(CONFIG_FPROBES) += fprobes.o obj-$(CONFIG_TRACEPOINT_BENCHMARK) += trace_benchmark.o
kernel/trace/fprobes.c new file +128 -0
128 lines not on this page · raw message
Signature · 1 line
2.25.1
On Thu, Jan 06, 2022 at 10:59:43PM +0900, Masami Hiramatsu wrote:
SNIP
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Hmm, I think there may be a time to split the "kprobe as an interface for the software breakpoint" and "kprobe as a wrapper interface for the callbacks of various instrumentations", like 'raw_kprobe'(or kswbp) and 'kprobes'.
And this may be called as 'fprobe' asftrace_opswrapper. (But if the bpf is enough flexible, this kind of intermediate layer
may not be needed, it can useftrace_opsdirectly, eventually)Jiri, have you already considered to use
ftrace_opsfrom the bpf directly? Are there any issues?
(bpf depends on 'kprobe' widely?)
Quoted 5 lines
Quoted 3 lines
at the moment there's not ftrace public interface for the return probe merged in, so to get the kretprobe working I had to use kprobe interface
Yeah, I found that too. We have to ask Steve to salvage it ;)
I got those patches rebased like half a year ago upstream code, so should be easy to revive them
Quoted 17 lines
Quoted 7 lines
but.. there are patches Steven shared some time ago, that do that and make
graph_opsavailable as kernel interfaceI recall we considered
graph_opsinterface before as common attach layer for trampolines, which was bad, but it might actually make sense for kprobesI started working on making 'fprobe' which will provide multiple function probe with similar interface of kprobes. See attached patch. Then you can use it in bpf, maybe with an union like
union {
struct kprobe kp; // for function body
struct fprobe fp; // for function entry and return
};
Quoted 9 lines
At this moment, fprobe only support
entry_handler, but when we re-start the genericgraph_opsinterface, it is easy to expand to supportexit_handler.
If this works, I think kretprobe can be phased out, since at that moment,kprobe_eventcan replace it with the fprobeexit_handler. (This is a benefit of decoupling the instrumentation layer from the event layer. It can choose the best way without changing user interface.)
I can resend out graph_ops patches if you want to base it directly on that
Quoted 5 lines
Quoted 5 lines
I'll need to check it in more details but I think both
graph_opsand kprobe do about similar thing wrt hooking return probe, so it should be comparable.. and they are already doing the same for the entry hook, because kprobe is mostly using ftrace for that
Quoted 7 lines
Quoted 2 lines
we would not need to introduce new program type - kprobe programs should be able to run from ftrace callbacks just fine
That seems to bind your mind. The program type is just a programing 'model' of the bpf. You can choose the best implementation to provide equal functionality. 'kprobe' in bpf is just a name that you call some instrumentations which can probe kernel code.
I don't want to introduce new type, there's some dependencies in bpf verifier and helpers code we'd need to handle for that
I'm looking for solution for current kprobe bpf program type to be registered for multiple addresses quickly
Quoted 54 lines
Thank you,
Quoted 7 lines
so we would have:
- kprobe type programs attaching to:
- newBPF_LINK_TYPE_FPROBElink using thegraph_opsas attachment layerjirka
Signature · 34 lines
Masami Hiramatsu <[email protected]> From 269b86597c166d6d4c5dd564168237603533165a Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu <[email protected]> Date: Thu, 6 Jan 2022 15:40:36 +0900 Subject: [PATCH] fprobe: Add ftrace based probe APIs The fprobe is a wrapper API for ftrace function tracer. Unlike kprobes, this probes only supports the function entry, but it can probe multiple functions by one fprobe. The usage is almost same as the kprobe, user will specify the function names by fprobe::syms, the number of syms by fprobe::nsyms, and the user handler by fprobe::handler. struct fprobe = { 0 }; const char *targets[] = {"func1", "func2", "func3"}; fprobe.handler = user_handler; fprobe.nsyms = ARRAY_SIZE(targets); fprobe.syms = targets; ret = register_fprobe(&fprobe); ... Signed-off-by: Masami Hiramatsu <[email protected]> --- include/linux/fprobes.h | 52 ++++++++++++++++ kernel/trace/Kconfig | 10 ++++ kernel/trace/Makefile | 1 + kernel/trace/fprobes.c | 128 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 191 insertions(+) create mode 100644 include/linux/fprobes.h create mode 100644 kernel/trace/fprobes.cinclude/linux/fprobes.h new file +0
-0
include/linux/fprobes.h line 10
Quoted 10 lines
include/linux/fprobes.h new file +10
-0@@ -0,0 +1,52 @@ +#ifndef _LINUX_FPROBES_H +#define _LINUX_FPROBES_H +/* Simple ftrace probe wrapper */ + +#include <linux/compiler.h> +#include <linux/ftrace.h> + +struct fprobe { + const char **syms; + unsigned long *addrs;
could you add array of user data for each addr/sym?
SNIP
Quoted 43 lines
include/linux/fprobes.h new file +43
-0+static int populate_func_addresses (struct fprobe *fp) +{ +unsigned int i; + + fp->addrs =kmalloc (sizeof (void *) * fp->nsyms,GFP_KERNEL ); +if (!fp->addrs) +return -ENOMEM ; + +for (i =0 ; i < fp->nsyms; i++) { + fp->addrs[i] =kallsyms_lookup_name (fp->syms[i]); +if (!fp->addrs[i]) { +kfree (fp->addrs); + fp->addrs =NULL ; +return -ENOENT ; + } + } + +return 0 ; +} + +/** +* register_fprobe - Register fprobe to ftrace +* @fp: A fprobe data structure to be registered. +* +* This expects the user set @fp::syms or @fp::addrs (not both), +* @fp::nsyms (number of entries of @fp::syms or @fp::addrs) and +* @fp::handler. Other fields are initialized by this function. +*/ +int register_fprobe (struct fprobe *fp) +{ +unsigned int i; +int ret; + +if (!fp) +return -EINVAL ; + +if (!fp->nsyms || (!fp->syms && !fp->addrs) || (fp->syms && fp->addrs)) +return -EINVAL ; + +if (fp->syms) { + ret =populate_func_addresses (fp); +if (ret <0 ) +return ret;
include/linux/fprobes.h
Quoted 10 lines
include/linux/fprobes.h new file +10
-0+ } + + fp->ftrace.func = fprobe_handler; + fp->ftrace.flags =FTRACE_OPS_FL_SAVE_REGS ; + +for (i =0 ; i < fp->nsyms; i++) { + ret =ftrace_set_filter_ip (&fp->ftrace, fp->addrs[i],0 ,0 ); +if (ret <0 ) + goto error; + }
I introduced ftrace_set_filter_ips, because loop like above was slow:
https://lore.kernel.org/bpf/[email protected]/
thanks,
jirka
Quoted 42 lines
include/linux/fprobes.h new file +39
-0+ + fp->nmissed =0 ; + ret =register_ftrace_function (&fp->ftrace); +if (!ret) +return ret; + +error: +if (fp->syms) { +kfree (fp->addrs); + fp->addrs =NULL ; + } + +return ret; +} + +/** +* unregister_fprobe - Unregister fprobe from ftrace +* @fp: A fprobe data structure to be unregistered. +*/ +int unregister_fprobe (struct fprobe *fp) +{ +int ret; + +if (!fp) +return -EINVAL ; + +if (!fp->nsyms || !fp->addrs) +return -EINVAL ; + + ret =unregister_ftrace_function (&fp->ftrace); + +if (fp->syms) { +/* fp->addrs is allocated by register_fprobe() */ +kfree (fp->addrs); + fp->addrs =NULL ; + } + +return ret; +}Signature · 1 line
2.25.1
Steven Rostedt
maintainerQuoted 5 lines · On Thu, 6 Jan 2022 22:59:43 +0900 Masami Hiramatsu <[email protected]> wrote:
Quoted 3 lines
at the moment there's not ftrace public interface for the return probe merged in, so to get the kretprobe working I had to use kprobe interface
Yeah, I found that too. We have to ask Steve to salvage it ;)
I have one more week of being unemployed (and I'm done with my office renovation), so perhaps I'll start looking into this.
This was the work to merge function graph tracer with kretprobes, right?
-- Steve
Re: [PATCH 08/13] bpf: Add kprobe link for attaching raw kprobes
In reply to Jiri OlsaQuoted 54 lines · Jiri Olsa
Quoted 11 lines · Andrii Nakryiko
Quoted 10 lines · Jiri Olsa
Adding new link type
BPF_LINK_TYPE_KPROBEto attach kprobes directly through register_kprobe/kretprobe API.Adding new attach type
BPF_TRACE_RAW_KPROBEthat enables such link for kprobe program.The new link allows to create multiple kprobes link by using new
link_createinterface:Quoted 8 lines
Quoted 4 lines
struct { __aligned_u64 addrs; __u32 cnt; __u64 bpf_cookie;I'm afraid
bpf_cookiehas to be different for each addr, otherwise it's severely limiting. So it would be an array of cookies alongside an array of addressesok
Quoted 21 lines
Quoted 14 lines
} kprobe;Plus new flag
BPF_F_KPROBE_RETURNfor link_create.flags to create return probe.Signed-off-by Jiri Olsa <[email protected]>--- include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 12 +++ kernel/bpf/syscall.c | 191 ++++++++++++++++++++++++++++++++- tools/include/uapi/linux/bpf.h | 12 +++ 4 files changed, 211 insertions(+), 5 deletions(-)[...]
Quoted 4 lines
include/uapi/linux/bpf.h +0
-0@@ -1111,6 +1113,11 @@ enum bpf_link_type {*/ #define BPF_F_SLEEPABLE (1U << 4)Quoted 8 lines
include/uapi/linux/bpf.h enum bpf_link_type { · line 1120
Quoted 5 lines
include/uapi/linux/bpf.h +5
-0+/* link_create flags used in LINK_CREATE command for BPF_TRACE_RAW_KPROBE +* attach type. +*/ +#define BPF_F_KPROBE_RETURN (1U << 0) +we have plenty of flexibility to have per-link type fields, so why not add
bool is_retprobenext to addrs and cnt?
Quoted 5 lines
well I thought if I do that, people would suggest to use the empty flags field instead ;-)
we can move it there as you suggest, but I wonder it's good idea to use bool in uapi headers, because the bool size definition is vague
Good point. No 'bool' please.
grep bool include/uapi/linux/
Only gives openvswitch.h and it's guarded by ifdef KERNEL So not a single uapi header has bool in it.
Quoted 5 lines · Masami Hiramatsu
That seems to bind your mind. The program type is just a programing 'model' of the bpf. You can choose the best implementation to provide equal functionality. 'kprobe' in bpf is just a name that you call some instrumentations which can probe kernel code.
No. We're not going to call it "fprobe" or any other name. From bpf user's pov it's going to be "multi attach kprobe", because this is how everyone got to know kprobes.
The 99% usage is at the beginning of the funcs.
When users say "kprobe" they don't care how kernel attaches it. The func entry limitation for "multi attach kprobe" is a no-brainer.
And we need both "multi attach kprobe" and "multi attach kretprobe" at the same time. It's no go to implement one first and the other some time later.
Re: [PATCH 08/13] bpf: Add kprobe link for attaching raw kprobes
In reply to Alexei StarovoitovQuoted 59 lines · Alexei Starovoitov
Quoted 57 lines · Jiri Olsa
Quoted 11 lines · Andrii Nakryiko
Quoted 10 lines · Jiri Olsa
Adding new link type
BPF_LINK_TYPE_KPROBEto attach kprobes directly through register_kprobe/kretprobe API.Adding new attach type
BPF_TRACE_RAW_KPROBEthat enables such link for kprobe program.The new link allows to create multiple kprobes link by using new
link_createinterface:Quoted 8 lines
Quoted 4 lines
struct { __aligned_u64 addrs; __u32 cnt; __u64 bpf_cookie;I'm afraid
bpf_cookiehas to be different for each addr, otherwise it's severely limiting. So it would be an array of cookies alongside an array of addressesok
Quoted 21 lines
Quoted 14 lines
} kprobe;Plus new flag
BPF_F_KPROBE_RETURNfor link_create.flags to create return probe.Signed-off-by Jiri Olsa <[email protected]>--- include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 12 +++ kernel/bpf/syscall.c | 191 ++++++++++++++++++++++++++++++++- tools/include/uapi/linux/bpf.h | 12 +++ 4 files changed, 211 insertions(+), 5 deletions(-)[...]
Quoted 4 lines
include/uapi/linux/bpf.h +0
-0@@ -1111,6 +1113,11 @@ enum bpf_link_type {*/ #define BPF_F_SLEEPABLE (1U << 4)Quoted 8 lines
include/uapi/linux/bpf.h enum bpf_link_type { · line 1120
Quoted 5 lines
include/uapi/linux/bpf.h +5
-0+/* link_create flags used in LINK_CREATE command for BPF_TRACE_RAW_KPROBE +* attach type. +*/ +#define BPF_F_KPROBE_RETURN (1U << 0) +we have plenty of flexibility to have per-link type fields, so why not add
bool is_retprobenext to addrs and cnt?well I thought if I do that, people would suggest to use the empty flags field instead ;-)
Quoted 7 lines
Quoted 2 lines
we can move it there as you suggest, but I wonder it's good idea to use bool in uapi headers, because the bool size definition is vague
Good point. No 'bool' please.
grep boolinclude/uapi/linux/
Only givesopenvswitch.hand it's guarded by ifdef KERNEL So not a single uapi header has bool in it.
Yeah, I don't insist on bool specifically.
But I was trying to avoid link_create.flags to become map_flags where we have various flags, each taking a separate bit, but most flags don't apply to most map types. Ideally link_create.flags would have few flags that apply to all or most link types (i.e., something orthogonal to a specific link type), and for this case we could have kprobe-specific flags (link_create.kprobe.flags) to adjust kprobe link creation behavior.
But I don't know, maybe I'm overthinking this.
Masami Hiramatsu
maintainerQuoted 7 lines · On Thu, 6 Jan 2022 09:40:17 -0800 Alexei Starovoitov…
Quoted 5 lines · Masami Hiramatsu
That seems to bind your mind. The program type is just a programing 'model' of the bpf. You can choose the best implementation to provide equal functionality. 'kprobe' in bpf is just a name that you call some instrumentations which can probe kernel code.
Quoted 6 lines
No. We're not going to call it "fprobe" or any other name. From bpf user's pov it's going to be "multi attach kprobe", because this is how everyone got to know kprobes.
The 99% usage is at the beginning of the funcs.
When users say "kprobe" they don't care how kernel attaches it. The func entry limitation for "multi attach kprobe" is a no-brainer.
Agreed. I think I might mislead you. From the bpf user pov, it always be shown as 'multi attached kprobes (but only for the function entry)' the 'fprobe' is kernel internal API name.
Quoted 3 lines
And we need both "multi attach kprobe" and "multi attach kretprobe" at the same time. It's no go to implement one first and the other some time later.
You can provide the interface to user space, but the kernel implementation is optimized step by step. We can start it with using real multiple kretprobes, and then, switch to 'fprobe' after integrating fgraph callback. :)
Thank you,
Signature · 1 line
Masami Hiramatsu <[email protected]>
Quoted 22 lines · Masami Hiramatsu
Quoted 7 lines · On Thu, 6 Jan 2022 09:40:17 -0800 Alexei Starovoitov…
Quoted 5 lines · Masami Hiramatsu
That seems to bind your mind. The program type is just a programing 'model' of the bpf. You can choose the best implementation to provide equal functionality. 'kprobe' in bpf is just a name that you call some instrumentations which can probe kernel code.
Quoted 6 lines
No. We're not going to call it "fprobe" or any other name. From bpf user's pov it's going to be "multi attach kprobe", because this is how everyone got to know kprobes.
The 99% usage is at the beginning of the funcs.
When users say "kprobe" they don't care how kernel attaches it. The func entry limitation for "multi attach kprobe" is a no-brainer.Agreed. I think I might mislead you. From the bpf user pov, it always be shown as 'multi attached kprobes (but only for the function entry)' the 'fprobe' is kernel internal API name.
Quoted 8 lines
Quoted 3 lines
And we need both "multi attach kprobe" and "multi attach kretprobe" at the same time. It's no go to implement one first and the other some time later.
You can provide the interface to user space, but the kernel implementation is optimized step by step. We can start it with using real multiple kretprobes, and then, switch to 'fprobe' after integrating fgraph callback. :)
Sounds good to me.
My point was that users often want to say:
"profile speed of all foo* functions".
To perform such a command a tracer would need to attach kprobes and kretprobes to all such functions. The speed of attach/detach has to be fast. Currently tracers artificially limit the regex just because attach/detach is so slow that the user will likely Ctrl-C instead of waiting for many seconds.
Masami Hiramatsu
maintainerQuoted 17 lines · On Thu, 6 Jan 2022 15:57:10 +0100 Jiri Olsa <[email protected]> wrote:
On Thu, Jan 06, 2022 at 10:59:43PM +0900, Masami Hiramatsu wrote:
SNIP
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Hmm, I think there may be a time to split the "kprobe as an interface for the software breakpoint" and "kprobe as a wrapper interface for the callbacks of various instrumentations", like 'raw_kprobe'(or kswbp) and 'kprobes'.
And this may be called as 'fprobe' asftrace_opswrapper. (But if the bpf is enough flexible, this kind of intermediate layer
may not be needed, it can useftrace_opsdirectly, eventually)Jiri, have you already considered to use
ftrace_opsfrom the bpf directly? Are there any issues?
(bpf depends on 'kprobe' widely?)
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
at the moment there's not ftrace public interface for the return probe merged in, so to get the kretprobe working I had to use kprobe interface
Yeah, I found that too. We have to ask Steve to salvage it ;)
I got those patches rebased like half a year ago upstream code, so should be easy to revive them
Nice! :)
Quoted 28 lines
Quoted 18 lines
Quoted 8 lines
but.. there are patches Steven shared some time ago, that do that and make
graph_opsavailable as kernel interfaceI recall we considered
graph_opsinterface before as common attach layer for trampolines, which was bad, but it might actually make sense for kprobesI started working on making 'fprobe' which will provide multiple function probe with similar interface of kprobes. See attached patch. Then you can use it in bpf, maybe with an union like
union {
struct kprobe kp; // for function body
struct fprobe fp; // for function entry and return
};Quoted 9 lines
At this moment, fprobe only support
entry_handler, but when we re-start the genericgraph_opsinterface, it is easy to expand to supportexit_handler.
If this works, I think kretprobe can be phased out, since at that moment,kprobe_eventcan replace it with the fprobeexit_handler. (This is a benefit of decoupling the instrumentation layer from the event layer. It can choose the best way without changing user interface.)
Quoted 2 lines
I can resend out
graph_opspatches if you want to base it directly on that
Yes, that's very helpful. Now I'm considering to use it (or via fprobe) from kretprobes like ftrace-based kprobe.
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
I'll need to check it in more details but I think both
graph_opsand kprobe do about similar thing wrt hooking return probe, so it should be comparable.. and they are already doing the same for the entry hook, because kprobe is mostly using ftrace for thatQuoted 7 lines
Quoted 2 lines
we would not need to introduce new program type - kprobe programs should be able to run from ftrace callbacks just fine
That seems to bind your mind. The program type is just a programing 'model' of the bpf. You can choose the best implementation to provide equal functionality. 'kprobe' in bpf is just a name that you call some instrumentations which can probe kernel code.
Quoted 5 lines
I don't want to introduce new type, there's some dependencies in bpf verifier and helpers code we'd need to handle for that
I'm looking for solution for current kprobe bpf program type to be registered for multiple addresses quickly
Yes, as I replied to Alex, the bpf program type itself keeps 'kprobe'. For example, you've introduced bpf_kprobe_link at [8/13],
struct bpf_kprobe_link {
struct bpf_link link;
union {
struct kretprobe rp;
struct fprobe fp;
};
bool is_return;
bool is_fentry;
kprobe_opcode_t **addrs;
u32 cnt;
u64 bpf_cookie;
};If all "addrs" are function entry, ::fp will be used. If cnt == 1 then use ::rp.
Quoted 54 lines
Quoted 54 lines
Quoted 6 lines
so we would have:
- kprobe type programs attaching to:
- newBPF_LINK_TYPE_FPROBElink using thegraph_opsas attachment layerjirka
Signature · 34 lines
Masami Hiramatsu <[email protected]> From 269b86597c166d6d4c5dd564168237603533165a Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu <[email protected]> Date: Thu, 6 Jan 2022 15:40:36 +0900 Subject: [PATCH] fprobe: Add ftrace based probe APIs The fprobe is a wrapper API for ftrace function tracer. Unlike kprobes, this probes only supports the function entry, but it can probe multiple functions by one fprobe. The usage is almost same as the kprobe, user will specify the function names by fprobe::syms, the number of syms by fprobe::nsyms, and the user handler by fprobe::handler. struct fprobe = { 0 }; const char *targets[] = {"func1", "func2", "func3"}; fprobe.handler = user_handler; fprobe.nsyms = ARRAY_SIZE(targets); fprobe.syms = targets; ret = register_fprobe(&fprobe); ... Signed-off-by: Masami Hiramatsu <[email protected]> --- include/linux/fprobes.h | 52 ++++++++++++++++ kernel/trace/Kconfig | 10 ++++ kernel/trace/Makefile | 1 + kernel/trace/fprobes.c | 128 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 191 insertions(+) create mode 100644 include/linux/fprobes.h create mode 100644 kernel/trace/fprobes.cinclude/linux/fprobes.h new file +4
-0@@ -0,0 +1,52 @@ +#ifndef _LINUX_FPROBES_H +#define _LINUX_FPROBES_H +/* Simple ftrace probe wrapper */ +
Quoted 8 lines
include/linux/fprobes.h line 10
Quoted 6 lines
include/linux/fprobes.h new file +6
-0+#include <linux/compiler.h> +#include <linux/ftrace.h> + +struct fprobe { +const char **syms; +unsigned long *addrs;could you add array of user data for each addr/sym?
OK, something like this?
void **user_data;
But note that you need O(N) to search the entry corresponding to a specific address. To reduce the overhead, we may need to sort the array in advance (e.g. when registering it).
Quoted 51 lines
SNIP
Quoted 48 lines
include/linux/fprobes.h new file +48
-0+static int populate_func_addresses (struct fprobe *fp) +{ +unsigned int i; + + fp->addrs =kmalloc (sizeof (void *) * fp->nsyms,GFP_KERNEL ); +if (!fp->addrs) +return -ENOMEM ; + +for (i =0 ; i < fp->nsyms; i++) { + fp->addrs[i] =kallsyms_lookup_name (fp->syms[i]); +if (!fp->addrs[i]) { +kfree (fp->addrs); + fp->addrs =NULL ; +return -ENOENT ; + } + } + +return 0 ; +} + +/** +* register_fprobe - Register fprobe to ftrace +* @fp: A fprobe data structure to be registered. +* +* This expects the user set @fp::syms or @fp::addrs (not both), +* @fp::nsyms (number of entries of @fp::syms or @fp::addrs) and +* @fp::handler. Other fields are initialized by this function. +*/ +int register_fprobe (struct fprobe *fp) +{ +unsigned int i; +int ret; + +if (!fp) +return -EINVAL ; + +if (!fp->nsyms || (!fp->syms && !fp->addrs) || (fp->syms && fp->addrs)) +return -EINVAL ; + +if (fp->syms) { + ret =populate_func_addresses (fp); +if (ret <0 ) +return ret; + } + + fp->ftrace.func = fprobe_handler; + fp->ftrace.flags =FTRACE_OPS_FL_SAVE_REGS ; +
Quoted 8 lines
include/linux/fprobes.h
Quoted 5 lines
include/linux/fprobes.h new file +5
-0+for (i =0 ; i < fp->nsyms; i++) { + ret =ftrace_set_filter_ip (&fp->ftrace, fp->addrs[i],0 ,0 ); +if (ret <0 ) + goto error; + }I introduced
ftrace_set_filter_ips, because loop like above was slow:
https://lore.kernel.org/bpf/[email protected]/
Ah, thanks for noticing!
Thank you,
Quoted 47 lines
thanks,
jirkaQuoted 42 lines
include/linux/fprobes.h new file +39
-0+ + fp->nmissed =0 ; + ret =register_ftrace_function (&fp->ftrace); +if (!ret) +return ret; + +error: +if (fp->syms) { +kfree (fp->addrs); + fp->addrs =NULL ; + } + +return ret; +} + +/** +* unregister_fprobe - Unregister fprobe from ftrace +* @fp: A fprobe data structure to be unregistered. +*/ +int unregister_fprobe (struct fprobe *fp) +{ +int ret; + +if (!fp) +return -EINVAL ; + +if (!fp->nsyms || !fp->addrs) +return -EINVAL ; + + ret =unregister_ftrace_function (&fp->ftrace); + +if (fp->syms) { +/* fp->addrs is allocated by register_fprobe() */ +kfree (fp->addrs); + fp->addrs =NULL ; + } + +return ret; +}Signature · 1 line
2.25.1
Signature · 1 line
Masami Hiramatsu <[email protected]>
Masami Hiramatsu
maintainerQuoted 32 lines · On Thu, 6 Jan 2022 16:20:05 -0800 Alexei Starovoitov…
Quoted 22 lines · Masami Hiramatsu
Quoted 7 lines · On Thu, 6 Jan 2022 09:40:17 -0800 Alexei Starovoitov…
Quoted 5 lines · Masami Hiramatsu
That seems to bind your mind. The program type is just a programing 'model' of the bpf. You can choose the best implementation to provide equal functionality. 'kprobe' in bpf is just a name that you call some instrumentations which can probe kernel code.
Quoted 6 lines
No. We're not going to call it "fprobe" or any other name. From bpf user's pov it's going to be "multi attach kprobe", because this is how everyone got to know kprobes.
The 99% usage is at the beginning of the funcs.
When users say "kprobe" they don't care how kernel attaches it. The func entry limitation for "multi attach kprobe" is a no-brainer.Agreed. I think I might mislead you. From the bpf user pov, it always be shown as 'multi attached kprobes (but only for the function entry)' the 'fprobe' is kernel internal API name.
Quoted 8 lines
Quoted 3 lines
And we need both "multi attach kprobe" and "multi attach kretprobe" at the same time. It's no go to implement one first and the other some time later.
You can provide the interface to user space, but the kernel implementation is optimized step by step. We can start it with using real multiple kretprobes, and then, switch to 'fprobe' after integrating fgraph callback. :)
Quoted 5 lines
Sounds good to me.
My point was that users often want to say:
"profile speed of all foo* functions".
To perform such a command a tracer would need to attach kprobes and kretprobes to all such functions.
Yeah, I know. That is more than 10 years issue since systemtap. :)
Quoted 1 line
The speed of attach/detach has to be fast.
Yes, that's why I provided register/unregister_kprobes() but it sounds not enough (and maybe not optimized enough because all handlers are same)
Quoted 3 lines
Currently tracers artificially limit the regex just because attach/detach is so slow that the user will likely Ctrl-C instead of waiting for many seconds.
Ah, OK.
Anyway I also would like to fix that issue. If user wants only function entry/exit, it should be done by ftrace. But since the syntax (and user's mind model) it should be done via 'kprobe', so transparently converting such request to ftrace is needed.
Thank you,
Signature · 1 line
Masami Hiramatsu <[email protected]>
Masami Hiramatsu
maintainercover fprobe: Introduce fprobe function entry/exit probe
In reply to Jiri OlsaHi Jiri,
Here is a short series of patches, which shows what I replied to your series.
This introduces the fprobe, the function entry/exit probe with multiple probe point support. This also introduces the rethook for hooking function return, which I cloned from kretprobe.
I also rewrite your [08/13] bpf patch to use this fprobe instead of kprobes. I didn't tested that one, but the sample module seems to work. Please test bpf part with your libbpf updates.
BTW, while implementing the fprobe, I introduced the per-probe point private data, but I'm not sure why you need it. It seems that data is not used from bpf...
If this is good for you, I would like to proceed this with the rethook and rewrite the kretprobe to use the rethook to hook the functions. That should be much cleaner (and easy to prepare for the fgraph tracer integration)
Thank you,
---
Jiri Olsa (1):
bpf: Add kprobe link for attaching raw kprobes
Masami Hiramatsu (5):
fprobe: Add ftrace based probe APIs
rethook: Add a generic return hook
rethook: x86: Add rethook x86 implementation
fprobe: Add exit_handler support
fprobe: Add sample program for fprobe
arch/x86/Kconfig | 1 arch/x86/kernel/Makefile | 1 arch/x86/kernel/rethook.c | 115 ++++++++++++++++++++ include/linux/bpf_types.h | 1 include/linux/fprobes.h | 75 +++++++++++++ include/linux/rethook.h | 74 +++++++++++++ include/linux/sched.h | 3 + include/uapi/linux/bpf.h | 12 ++ kernel/bpf/syscall.c | 199 +++++++++++++++++++++++++++++++++- kernel/exit.c | 2 kernel/fork.c | 3 + kernel/trace/Kconfig | 22 ++++ kernel/trace/Makefile | 2 kernel/trace/fprobes.c | 187 ++++++++++++++++++++++++++++++++ kernel/trace/rethook.c | 226 +++++++++++++++++++++++++++++++++++++++ samples/Kconfig | 6 + samples/Makefile | 1 samples/fprobe/Makefile | 3 + samples/fprobe/fprobe_example.c | 103 ++++++++++++++++++ tools/include/uapi/linux/bpf.h | 12 ++ 20 files changed, 1043 insertions(+), 5 deletions(-) create mode 100644 arch/x86/kernel/rethook.c create mode 100644 include/linux/fprobes.h create mode 100644 include/linux/rethook.h create mode 100644 kernel/trace/fprobes.c create mode 100644 kernel/trace/rethook.c create mode 100644 samples/fprobe/Makefile create mode 100644 samples/fprobe/fprobe_example.c
--
Masami Hiramatsu (Linaro) <[email protected]>
Masami Hiramatsu
maintainerRFC PATCH 1/6 fprobe: Add ftrace based probe APIs
The fprobe is a wrapper API for ftrace function tracer. Unlike kprobes, this probes only supports the function entry, but it can probe multiple functions by one fprobe. The usage is almost same as the kprobe, user will specify the function names by fprobe::entries[].syms, the number of syms by fprobe::nentry, and the user handler by fprobe::entry_handler.
struct fprobe fp = { 0 };
struct fprobe_entry targets[] =
{{.sym = "func1"}, {.sym = "func2"}, {.sym = "func3"}};fp.handler = user_handler;
fp.nentry = ARRAY_SIZE(targets);
fp.entries = targets;
ret = register_fprobe(&fp);
Note that the fp::entries will be sorted by the converted function address.
--- include/linux/fprobes.h | 71 +++++++++++++++++++++++++ kernel/trace/Kconfig | 10 ++++ kernel/trace/Makefile | 1 kernel/trace/fprobes.c | 132 +++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 214 insertions(+) create mode 100644 include/linux/fprobes.h create mode 100644 kernel/trace/fprobes.c
include/linux/fprobes.h new file +71 -0
@@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Simple ftrace probe wrapper */ +#ifndef _LINUX_FPROBES_H +#define _LINUX_FPROBES_H + +#include <linux/compiler.h> +#include <linux/ftrace.h> + +/* +* fprobe_entry - function entry for fprobe +* @sym: The symbol name of the function. +* @addr: The address of @sym. +* @data: per-entry data +* +* User must specify either @sym or @addr (not both). @data is optional. +*/ +struct fprobe_entry { +const char *sym; +unsigned long addr; +void *data; +}; + +struct fprobe { +struct fprobe_entry *entries; +unsigned int nentry; + +struct ftrace_ops ftrace; +unsigned long nmissed; +unsigned int flags; +void (*entry_handler) (struct fprobe *,unsigned long ,struct pt_regs *); +}; + +#define FPROBE_FL_DISABLED 1 + +static inline bool fprobe_disabled (struct fprobe *fp) +{ +return (fp) ? fp->flags &FPROBE_FL_DISABLED : false; +} + +#ifdef CONFIG_FPROBES +int register_fprobe (struct fprobe *fp); +int unregister_fprobe (struct fprobe *fp); +struct fprobe_entry *fprobe_find_entry (struct fprobe *fp,unsigned long addr); +#else +static inline int register_fprobe (struct fprobe *fp) +{ +return -ENOTSUPP ; +} +static inline int unregister_fprobe (struct fprobe *fp) +{ +return -ENOTSUPP ; +} +struct fprobe_entry *fprobe_find_entry (struct fprobe *fp,unsigned long addr) +{ +return NULL ; +} +#endif + +static inline void disable_fprobe (struct fprobe *fp) +{ +if (fp) + fp->flags |=FPROBE_FL_DISABLED ; +} + +static inline void enable_fprobe (struct fprobe *fp) +{ +if (fp) + fp->flags &= ~FPROBE_FL_DISABLED ; +} + +#endif
kernel/trace/Kconfig +10 -0
@@ -223,6 +223,16 @@ config DYNAMIC_FTRACE_WITH_ARGS depends on DYNAMIC_FTRACE depends on HAVE_DYNAMIC_FTRACE_WITH_ARGS +config FPROBES + bool "Kernel Function Probe (fprobe)" + depends on FUNCTION_TRACER + depends on DYNAMIC_FTRACE_WITH_REGS + default n + help + This option enables kernel function probe feature, which is + similar to kprobes, but probes only for kernel function entries + and it can probe multiple functions by one fprobe. + config FUNCTION_PROFILER bool "Kernel function profiler" depends on FUNCTION_TRACER
kernel/trace/Makefile +1 -0
@@ -97,6 +97,7 @@ obj-$(CONFIG_PROBE_EVENTS) += trace_probe.o obj-$(CONFIG_UPROBE_EVENTS) += trace_uprobe.o obj-$(CONFIG_BOOTTIME_TRACING) += trace_boot.o obj-$(CONFIG_FTRACE_RECORD_RECURSION) += trace_recursion_record.o +obj-$(CONFIG_FPROBES) += fprobes.o obj-$(CONFIG_TRACEPOINT_BENCHMARK) += trace_benchmark.o
kernel/trace/fprobes.c new file +132 -0
132 lines not on this page · raw message
Masami Hiramatsu
maintainerRFC PATCH 2/6 rethook: Add a generic return hook
In reply to Masami HiramatsuAdd a return hook framework which hooks the function return. Most of the idea came from the kretprobe, but this is independent from kretprobe.
Note that this is expected to be used with other function entry hooking feature, like ftrace, fprobe, adn kprobes. Eventually this will replace the kretprobe (e.g. kprobe + rethook = kretprobe), but at this moment, this is just a additional hook.
--- include/linux/rethook.h | 74 +++++++++++++++ include/linux/sched.h | 3 + kernel/exit.c | 2 kernel/fork.c | 3 + kernel/trace/Kconfig | 11 ++ kernel/trace/Makefile | 1 kernel/trace/rethook.c | 226 +++++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 320 insertions(+) create mode 100644 include/linux/rethook.h create mode 100644 kernel/trace/rethook.c
include/linux/rethook.h new file +74 -0
@@ -0,0 +1,74 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Return hooking with list-based shadow stack. +*/ +#ifndef _LINUX_RETHOOK_H +#define _LINUX_RETHOOK_H + +#include <linux/compiler.h> +#include <linux/freelist.h> +#include <linux/llist.h> +#include <linux/rcupdate.h> +#include <linux/refcount.h> + +struct rethook_node ; + +typedef void (*rethook_handler_t ) (struct rethook_node *,void *,struct pt_regs *); + +struct rethook { +void *data; +rethook_handler_t handler; +struct freelist_head pool; +refcount_t ref; +struct rcu_head rcu; +}; + +struct rethook_node { +union { +struct freelist_node freelist; +struct rcu_head rcu; + }; +struct llist_node llist; +struct rethook *rethook; +unsigned long ret_addr; +unsigned long frame; +}; + +int rethook_node_init (struct rethook_node *node); + +struct rethook *rethook_alloc (void *data,rethook_handler_t handler); +void rethook_free (struct rethook *rh); +void rethook_add_node (struct rethook *rh,struct rethook_node *node); + +struct rethook_node *rethook_try_get (struct rethook *rh); +void rethook_node_recycle (struct rethook_node *node); +void rethook_hook_current (struct rethook_node *node,struct pt_regs *regs); + +unsigned long rethook_find_ret_addr (struct task_struct *tsk,unsigned long frame, +struct llist_node **cur); + +/* Arch dependent code must implement this and trampoline code */ +void arch_rethook_prepare (struct rethook_node *node,struct pt_regs *regs); +void arch_rethook_trampoline (void ); + +static inline bool is_rethook_trampoline (unsigned long addr) +{ +return addr == (unsigned long )arch_rethook_trampoline; +} + +/* If the architecture needs a fixup the return address, implement it. */ +void arch_rethook_fixup_return (struct pt_regs *regs, +unsigned long correct_ret_addr); + +/* Generic trampoline handler, arch code must prepare asm stub */ +unsigned long rethook_trampoline_handler (struct pt_regs *regs, +unsigned long frame); + +#ifdef CONFIG_RETHOOK +void rethook_flush_task (struct task_struct *tk); +#else +#define rethook_flush_task (tsk) do { } while (0) +#endif + +#endif +
include/linux/sched.h +3 -0
@@ -1473,6 +1473,9 @@ struct task_struct {#ifdef CONFIG_KRETPROBES struct llist_head kretprobe_instances;#endif +#ifdef CONFIG_RETHOOK +struct llist_head rethooks; +#endif #ifdef CONFIG_ARCH_HAS_PARANOID_L1D_FLUSH /*
kernel/exit.c +2 -0
@@ -64,6 +64,7 @@#include <linux/compat.h> #include <linux/io_uring.h> #include <linux/kprobes.h> +#include <linux/rethook.h> #include <linux/uaccess.h> #include <asm/unistd.h>
@@ -169,6 +170,7 @@ static void delayed_put_task_struct(struct rcu_head *rhp) struct task_struct *tsk = container_of(rhp, struct task_struct, rcu);kprobe_flush_task (tsk); +rethook_flush_task (tsk);perf_event_delayed_put (tsk);trace_sched_process_free (tsk);put_task_struct (tsk);
kernel/fork.c +3 -0
@@ -2282,6 +2282,9 @@ static __latent_entropy struct task_struct *copy_process(#ifdef CONFIG_KRETPROBES p->kretprobe_instances.first =NULL ;#endif +#ifdef CONFIG_RETHOOK + p->rethooks.first =NULL ; +#endif /* * Ensure that the cgroup subsystem policies allow the new process to be
kernel/trace/Kconfig +11 -0
@@ -10,6 +10,17 @@ config USER_STACKTRACE_SUPPORT config NOP_TRACER bool +config HAVE_RETHOOK + bool + +config RETHOOK + bool + depends on HAVE_RETHOOK + help + Enable generic return hooking feature. This is an internal + API, which will be used by other function-entry hooking + feature like fprobe and kprobes. + config HAVE_FUNCTION_TRACER bool help
kernel/trace/Makefile +1 -0
@@ -98,6 +98,7 @@ obj-$(CONFIG_UPROBE_EVENTS) += trace_uprobe.o obj-$(CONFIG_BOOTTIME_TRACING) += trace_boot.o obj-$(CONFIG_FTRACE_RECORD_RECURSION) += trace_recursion_record.o obj-$(CONFIG_FPROBES) += fprobes.o +obj-$(CONFIG_RETHOOK) += rethook.o obj-$(CONFIG_TRACEPOINT_BENCHMARK) += trace_benchmark.o
kernel/trace/rethook.c new file +226 -0
226 lines not on this page · raw message
Masami Hiramatsu
maintainerRFC PATCH 3/6 rethook: x86: Add rethook x86 implementation
In reply to Masami HiramatsuAdd rethook for x86 implementation. Most of the code has been copied from kretprobes on x86.
--- arch/x86/Kconfig | 1 arch/x86/kernel/Makefile | 1 arch/x86/kernel/rethook.c | 115 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 117 insertions(+) create mode 100644 arch/x86/kernel/rethook.c
arch/x86/Kconfig +1 -0
@@ -219,6 +219,7 @@ config X86 select HAVE_KPROBES_ON_FTRACE select HAVE_FUNCTION_ERROR_INJECTION select HAVE_KRETPROBES + select HAVE_RETHOOK select HAVE_KVM select HAVE_LIVEPATCH if X86_64 select HAVE_MIXED_BREAKPOINTS_REGS
arch/x86/kernel/Makefile +1 -0
@@ -106,6 +106,7 @@ obj-$(CONFIG_FUNCTION_GRAPH_TRACER) += ftrace.o obj-$(CONFIG_FTRACE_SYSCALLS) += ftrace.o obj-$(CONFIG_X86_TSC) += trace_clock.o obj-$(CONFIG_TRACING) += trace.o +obj-$(CONFIG_RETHOOK) += rethook.o obj-$(CONFIG_CRASH_CORE) += crash_core_$(BITS).o obj-$(CONFIG_KEXEC_CORE) += machine_kexec_$(BITS).o obj-$(CONFIG_KEXEC_CORE) += relocate_kernel_$(BITS).o crash.o
arch/x86/kernel/rethook.c new file +115 -0
115 lines not on this page · raw message
Masami Hiramatsu
maintainerRFC PATCH 4/6 fprobe: Add exit_handler support
In reply to Masami HiramatsuAdd exit_handler to fprobe. fprobe + rethook allows us to hook the kernel function return without fgraph tracer. Eventually, the fgraph tracer will be generic array based return hooking and fprobe may use it if user requests. Since both array-based approach and list-based approach have Pros and Cons, (e.g. memory consumption v.s. less missing events) it is better to keep both but fprobe will provide the same exit-handler interface.
--- include/linux/fprobes.h | 4 +++ kernel/trace/Kconfig | 1 + kernel/trace/fprobes.c | 59 +++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 62 insertions(+), 2 deletions(-)
include/linux/fprobes.h +4 -0
@@ -5,6 +5,7 @@#include <linux/compiler.h> #include <linux/ftrace.h> +#include <linux/rethook.h> /* * fprobe_entry - function entry for fprobe
@@ -27,7 +28,10 @@ struct fprobe {struct ftrace_ops ftrace;unsigned long nmissed;unsigned int flags; +struct rethook *rethook; +void (*entry_handler) (struct fprobe *,unsigned long ,struct pt_regs *); +void (*exit_handler) (struct fprobe *,unsigned long ,struct pt_regs *); };#define FPROBE_FL_DISABLED 1
kernel/trace/Kconfig +1 -0
@@ -238,6 +238,7 @@ config FPROBES bool "Kernel Function Probe (fprobe)" depends on FUNCTION_TRACER depends on DYNAMIC_FTRACE_WITH_REGS + select RETHOOK default n help This option enables kernel function probe feature, which is
kernel/trace/fprobes.c +57 -2
101 lines not on this page · raw message
Masami Hiramatsu
maintainerRFC PATCH 5/6 fprobe: Add sample program for fprobe
In reply to Masami HiramatsuAdd a sample program for the fprobe.
--- samples/Kconfig | 6 ++ samples/Makefile | 1 samples/fprobe/Makefile | 3 + samples/fprobe/fprobe_example.c | 103 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 113 insertions(+) create mode 100644 samples/fprobe/Makefile create mode 100644 samples/fprobe/fprobe_example.c
samples/Kconfig +6 -0
@@ -73,6 +73,12 @@ config SAMPLE_HW_BREAKPOINT help This builds kernel hardware breakpoint example modules. +config SAMPLE_FPROBE + tristate "Build fprobe examples -- loadable modules only" + depends on FPROBES && m + help + This build several fprobe example modules. + config SAMPLE_KFIFO tristate "Build kfifo examples -- loadable modules only" depends on m
samples/Makefile +1 -0
@@ -32,3 +32,4 @@ obj-$(CONFIG_SAMPLE_INTEL_MEI) += mei/ subdir-$(CONFIG_SAMPLE_WATCHDOG) += watchdog subdir-$(CONFIG_SAMPLE_WATCH_QUEUE) += watch_queue obj-$(CONFIG_DEBUG_KMEMLEAK_TEST) += kmemleak/ +obj-$(CONFIG_SAMPLE_FPROBE) += fprobe/
samples/fprobe/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0-only + +obj-$(CONFIG_SAMPLE_FPROBE) += fprobe_example.o
samples/fprobe/fprobe_example.c new file +103 -0
103 lines not on this page · raw message
Masami Hiramatsu
maintainerRFC PATCH 6/6 bpf: Add kprobe link for attaching raw kprobes
In reply to Masami HiramatsuFrom: Jiri Olsa <[email protected]>
Adding new link type BPF_LINK_TYPE_KPROBE to attach so called "kprobes" directly through fprobe API. Note that since the using kprobes with multiple same handler is not efficient, this uses the fprobe which natively support multiple probe points for one same handler, but limited on function entry and exit.
Adding new attach type BPF_TRACE_RAW_KPROBE that enables such link for kprobe program.
The new link allows to create multiple kprobes link by using new link_create interface:
struct {
__aligned_u64 addrs;
__u32 cnt;
__u64 bpf_cookie;
} kprobe;Plus new flag BPF_F_KPROBE_RETURN for link_create.flags to create return probe.
--- include/linux/bpf_types.h | 1 include/uapi/linux/bpf.h | 12 ++ kernel/bpf/syscall.c | 199 +++++++++++++++++++++++++++++++++++++++- tools/include/uapi/linux/bpf.h | 12 ++ 4 files changed, 219 insertions(+), 5 deletions(-)
include/linux/bpf_types.h +1 -0
@@ -140,3 +140,4 @@ BPF_LINK_TYPE(BPF_LINK_TYPE_XDP, xdp)#ifdef CONFIG_PERF_EVENTS BPF_LINK_TYPE(BPF_LINK_TYPE_PERF_EVENT, perf)#endif +BPF_LINK_TYPE (BPF_LINK_TYPE_KPROBE , kprobe)
include/uapi/linux/bpf.h +12 -0
@@ -995,6 +995,7 @@ enum bpf_attach_type {BPF_SK_REUSEPORT_SELECT ,BPF_SK_REUSEPORT_SELECT_OR_MIGRATE ,BPF_PERF_EVENT , +BPF_TRACE_RAW_KPROBE , __MAX_BPF_ATTACH_TYPE };
@@ -1009,6 +1010,7 @@ enum bpf_link_type {BPF_LINK_TYPE_NETNS =5 ,BPF_LINK_TYPE_XDP =6 ,BPF_LINK_TYPE_PERF_EVENT =7 , +BPF_LINK_TYPE_KPROBE =8 , MAX_BPF_LINK_TYPE, };
@@ -1111,6 +1113,11 @@ enum bpf_link_type {*/ #define BPF_F_SLEEPABLE (1U << 4) +/* link_create flags used in LINK_CREATE command for BPF_TRACE_RAW_KPROBE +* attach type. +*/ +#define BPF_F_KPROBE_RETURN (1U << 0) +/* When BPF ldimm64's insn[0].src_reg != 0 then this can have * the following extensions: *
@@ -1463,6 +1470,11 @@ union bpf_attr {*/ __u64 bpf_cookie; } perf_event; +struct { +__aligned_u64 addrs; +__u32 cnt; +__u64 bpf_cookie; + } kprobe; }; } link_create;
kernel/bpf/syscall.c +194 -5
241 lines not on this page · raw message
tools/include/uapi/linux/bpf.h +12 -0
@@ -995,6 +995,7 @@ enum bpf_attach_type {BPF_SK_REUSEPORT_SELECT ,BPF_SK_REUSEPORT_SELECT_OR_MIGRATE ,BPF_PERF_EVENT , +BPF_TRACE_RAW_KPROBE , __MAX_BPF_ATTACH_TYPE };
@@ -1009,6 +1010,7 @@ enum bpf_link_type {BPF_LINK_TYPE_NETNS =5 ,BPF_LINK_TYPE_XDP =6 ,BPF_LINK_TYPE_PERF_EVENT =7 , +BPF_LINK_TYPE_KPROBE =8 , MAX_BPF_LINK_TYPE, };
@@ -1111,6 +1113,11 @@ enum bpf_link_type {*/ #define BPF_F_SLEEPABLE (1U << 4) +/* link_create flags used in LINK_CREATE command for BPF_TRACE_RAW_KPROBE +* attach type. +*/ +#define BPF_F_KPROBE_RETURN (1U << 0) +/* When BPF ldimm64's insn[0].src_reg != 0 then this can have * the following extensions: *
@@ -1463,6 +1470,11 @@ union bpf_attr {*/ __u64 bpf_cookie; } perf_event; +struct { +__aligned_u64 addrs; +__u32 cnt; +__u64 bpf_cookie; + } kprobe; }; } link_create;
Re: [RFC PATCH 0/6] fprobe: Introduce fprobe function entry/exit probe
In reply to Masami HiramatsuQuoted 13 lines · Masami Hiramatsu
Hi Jiri,
Here is a short series of patches, which shows what I replied to your series.
This introduces the fprobe, the function entry/exit probe with multiple probe point support. This also introduces the rethook for hooking function return, which I cloned from kretprobe.
I also rewrite your [08/13] bpf patch to use this fprobe instead of kprobes. I didn't tested that one, but the sample module seems to work. Please test bpf part with your libbpf updates.
Quoted 8 lines
BTW, while implementing the fprobe, I introduced the per-probe point private data, but I'm not sure why you need it. It seems that data is not used from bpf...
If this is good for you, I would like to proceed this with the rethook and rewrite the kretprobe to use the rethook to hook the functions. That should be much cleaner (and easy to prepare for the fgraph tracer integration)
What is the speed of attach/detach of thousands fprobes?
Masami Hiramatsu
maintainerRe: [RFC PATCH 0/6] fprobe: Introduce fprobe function entry/exit probe
Hi Alexei,
Quoted 18 lines · On Tue, 11 Jan 2022 14:39:44 -0800 Alexei Starovoitov…
Quoted 17 lines · Masami Hiramatsu
Hi Jiri,
Here is a short series of patches, which shows what I replied to your series.
This introduces the fprobe, the function entry/exit probe with multiple probe point support. This also introduces the rethook for hooking function return, which I cloned from kretprobe.
I also rewrite your [08/13] bpf patch to use this fprobe instead of kprobes. I didn't tested that one, but the sample module seems to work. Please test bpf part with your libbpf updates.
BTW, while implementing the fprobe, I introduced the per-probe point private data, but I'm not sure why you need it. It seems that data is not used from bpf...
Quoted 6 lines
Quoted 4 lines
If this is good for you, I would like to proceed this with the rethook and rewrite the kretprobe to use the rethook to hook the functions. That should be much cleaner (and easy to prepare for the fgraph tracer integration)
What is the speed of attach/detach of thousands fprobes?
I've treaked my example module and it shows below result;
/lib/modules/5.16.0-rc4+/kernel/samples/fprobe # time insmod ./fprobe_example.ko
symbol='btrfs_*'
[ 187.095925] fprobe_init: 1028 symbols found [ 188.521694] fprobe_init: Planted fprobe at btrfs_*
real 0m 1.47s
user 0m 0.00s
sys 0m 1.36s
I think using ftrace_set_filter_ips() can make it faster. (maybe it needs to drop per-probe point private data, that prevents fprobe to use that interface)
Thank you,
Signature · 1 line
Masami Hiramatsu <[email protected]>
Masami Hiramatsu
maintainerRe: [RFC PATCH 0/6] fprobe: Introduce fprobe function entry/exit probe
Quoted 36 lines · On Wed, 12 Jan 2022 16:33:53 +0900 Masami Hiramatsu <[email protected]> wrote:
Hi Alexei,
Quoted 18 lines · On Tue, 11 Jan 2022 14:39:44 -0800 Alexei Starovoitov…
Quoted 17 lines · Masami Hiramatsu
Hi Jiri,
Here is a short series of patches, which shows what I replied to your series.
This introduces the fprobe, the function entry/exit probe with multiple probe point support. This also introduces the rethook for hooking function return, which I cloned from kretprobe.
I also rewrite your [08/13] bpf patch to use this fprobe instead of kprobes. I didn't tested that one, but the sample module seems to work. Please test bpf part with your libbpf updates.
BTW, while implementing the fprobe, I introduced the per-probe point private data, but I'm not sure why you need it. It seems that data is not used from bpf...
Quoted 6 lines
Quoted 4 lines
If this is good for you, I would like to proceed this with the rethook and rewrite the kretprobe to use the rethook to hook the functions. That should be much cleaner (and easy to prepare for the fgraph tracer integration)
What is the speed of attach/detach of thousands fprobes?
I've treaked my example module and it shows below result;
/lib/modules/5.16.0-rc4+/kernel/samples/fprobe# time insmod ./fprobe_example.ko
symbol='btrfs_*'[ 187.095925] fprobe_init: 1028 symbols found [ 188.521694] fprobe_init: Planted fprobe at btrfs_*
Quoted 7 lines
real 0m 1.47s
user 0m 0.00s
sys 0m 1.36sI think using
ftrace_set_filter_ips()can make it faster. (maybe it needs to drop per-probe point private data, that prevents fprobe to use that interface)
OK, I've updated fprobes to use the ftrace_set_filter_ips() and got below result.
/lib/modules/5.16.0-rc4+/kernel/samples/fprobe # time insmod fprobe_example.ko s ymbol='btrfs_*'
[ 36.130947] fprobe_init: 1028 symbols found [ 36.177901] fprobe_init: Planted fprobe at btrfs_*
real 0m 0.08s
user 0m 0.00s
sys 0m 0.07s
Let me update the series :)
Thank you,
Signature · 1 line
Masami Hiramatsu <[email protected]>
kernel test robot
build botRe: [RFC PATCH 6/6] bpf: Add kprobe link for attaching raw kprobes
In reply to Masami Hiramatsu[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi Masami,
[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on rostedt-trace/for-next]
[also build test WARNING on bpf-next/master bpf/master linus/master v5.16] [cannot apply to next-20220112]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Masami-Hiramatsu/fprobe-Add-ftrace-based-probe-APIs/20220112-000050 base: https://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git for-next config: x86_64-randconfig-a011 (https://download.01.org/0day-ci/archive/20220113/202201130621.vPqexP9P-lkp(a)intel.com/config) compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
reproduce (this is a W=1 build):
# https://github.com/0day-ci/linux/commit/ba002e6556acd9e0af16442f4226ddb2a8759567
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Masami-Hiramatsu/fprobe-Add-ftrace-based-probe-APIs/20220112-000050 git checkout ba002e6556acd9e0af16442f4226ddb2a8759567
# save the config file to linux build tree
mkdir build_dir
make W=1 O=build_dir ARCH=x86_64 SHELL=/bin/bash kernel/bpf/
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
In file included from kernel/bpf/syscall.c:34:
>> include/linux/fprobes.h:57:22: warning: no previous prototype for 'fprobe_find_entry' [-Wmissing-prototypes]
57 | struct fprobe_entry *fprobe_find_entry(struct fprobe *fp, unsigned long addr)
| ^~~~~~~~~~~~~~~~~
vim +/fprobe_find_entry +57 include/linux/fprobes.h
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 43
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 44 #ifdef CONFIG_FPROBES
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 45 int register_fprobe(struct fprobe *fp);
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 46 int unregister_fprobe(struct fprobe *fp);
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 47 struct fprobe_entry *fprobe_find_entry(struct fprobe *fp, unsigned long addr);
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 48 #else
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 49 static inline int register_fprobe(struct fprobe *fp)
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 50 {
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 51 return -ENOTSUPP;
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 52 }
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 53 static inline int unregister_fprobe(struct fprobe *fp)
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 54 {
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 55 return -ENOTSUPP;
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 56 }
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 @57 struct fprobe_entry *fprobe_find_entry(struct fprobe *fp, unsigned long addr)
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 58 {
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 59 return NULL;
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 60 }
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 61 #endif7b8ed3653f3755 Masami Hiramatsu 2022-01-12 62
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build botRe: [RFC PATCH 4/6] fprobe: Add exit_handler support
In reply to Masami Hiramatsu[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi Masami,
[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on rostedt-trace/for-next]
[also build test WARNING on bpf-next/master bpf/master linus/master v5.16 next-20220112] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Masami-Hiramatsu/fprobe-Add-ftrace-based-probe-APIs/20220112-000050 base: https://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git for-next config: i386-allyesconfig (https://download.01.org/0day-ci/archive/20220113/202201130629.IyZ9vaDV-lkp(a)intel.com/config) compiler: gcc-9 (Debian 9.3.0-22) 9.3.0 reproduce (this is a W=1 build): # https://github.com/0day-ci/linux/commit/c0e0471b58c3c9122bbff7523f97a363558284eb git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review Masami-Hiramatsu/fprobe-Add-ftrace-based-probe-APIs/20220112-000050 git checkout c0e0471b58c3c9122bbff7523f97a363558284eb # save the config file to linux build tree mkdir build_dir make W=1 O=build_dir ARCH=i386 SHELL=/bin/bash arch/x86/kernel/
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
>> arch/x86/kernel/rethook.c:14:23: warning: no previous prototype for 'arch_rethook_trampoline_callback' [-Wmissing-prototypes]
14 | __used __visible void arch_rethook_trampoline_callback(struct pt_regs *regs)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
vim +/arch_rethook_trampoline_callback +14 arch/x86/kernel/rethook.c
3727c0ee2be25c Masami Hiramatsu 2022-01-12 10
3727c0ee2be25c Masami Hiramatsu 2022-01-12 11 /*
3727c0ee2be25c Masami Hiramatsu 2022-01-12 12 * Called from arch_rethook_trampoline
3727c0ee2be25c Masami Hiramatsu 2022-01-12 13 */
3727c0ee2be25c Masami Hiramatsu 2022-01-12 @14 __used __visible void arch_rethook_trampoline_callback(struct pt_regs *regs)
3727c0ee2be25c Masami Hiramatsu 2022-01-12 15 {
3727c0ee2be25c Masami Hiramatsu 2022-01-12 16 unsigned long *frame_pointer;
3727c0ee2be25c Masami Hiramatsu 2022-01-12 17
3727c0ee2be25c Masami Hiramatsu 2022-01-12 18 /* fixup registers */
3727c0ee2be25c Masami Hiramatsu 2022-01-12 19 regs->cs = __KERNEL_CS;
3727c0ee2be25c Masami Hiramatsu 2022-01-12 20 #ifdef CONFIG_X86_32
3727c0ee2be25c Masami Hiramatsu 2022-01-12 21 regs->gs = 0;
3727c0ee2be25c Masami Hiramatsu 2022-01-12 22 #endif
3727c0ee2be25c Masami Hiramatsu 2022-01-12 23 regs->ip = (unsigned long)&arch_rethook_trampoline;
3727c0ee2be25c Masami Hiramatsu 2022-01-12 24 regs->orig_ax = ~0UL;
3727c0ee2be25c Masami Hiramatsu 2022-01-12 25 regs->sp += sizeof(long);
3727c0ee2be25c Masami Hiramatsu 2022-01-12 26 frame_pointer = ®s->sp + 1;
3727c0ee2be25c Masami Hiramatsu 2022-01-12 27
3727c0ee2be25c Masami Hiramatsu 2022-01-12 28 /*
3727c0ee2be25c Masami Hiramatsu 2022-01-12 29 * The return address at 'frame_pointer' is recovered by the
3727c0ee2be25c Masami Hiramatsu 2022-01-12 30 * arch_rethook_fixup_return() which called from this
3727c0ee2be25c Masami Hiramatsu 2022-01-12 31 * rethook_trampoline_handler().
3727c0ee2be25c Masami Hiramatsu 2022-01-12 32 */
3727c0ee2be25c Masami Hiramatsu 2022-01-12 33 rethook_trampoline_handler(regs, (unsigned long)frame_pointer);
3727c0ee2be25c Masami Hiramatsu 2022-01-12 34
3727c0ee2be25c Masami Hiramatsu 2022-01-12 35 /*
3727c0ee2be25c Masami Hiramatsu 2022-01-12 36 * Copy FLAGS to 'pt_regs::sp' so that arch_rethook_trapmoline()
3727c0ee2be25c Masami Hiramatsu 2022-01-12 37 * can do RET right after POPF.
3727c0ee2be25c Masami Hiramatsu 2022-01-12 38 */
3727c0ee2be25c Masami Hiramatsu 2022-01-12 39 regs->sp = regs->flags;
3727c0ee2be25c Masami Hiramatsu 2022-01-12 40 }
3727c0ee2be25c Masami Hiramatsu 2022-01-12 41 NOKPROBE_SYMBOL(arch_rethook_trampoline_callback);
3727c0ee2be25c Masami Hiramatsu 2022-01-12 42 ---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build botRe: [RFC PATCH 6/6] bpf: Add kprobe link for attaching raw kprobes
In reply to Masami Hiramatsu[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi Masami,
[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on rostedt-trace/for-next]
[also build test WARNING on bpf-next/master bpf/master linus/master v5.16] [cannot apply to next-20220112]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Masami-Hiramatsu/fprobe-Add-ftrace-based-probe-APIs/20220112-000050 base: https://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git for-next config: hexagon-randconfig-r001-20220112 (https://download.01.org/0day-ci/archive/20220113/202201130747.yNR0J8tO-lkp(a)intel.com/config) compiler: clang version 14.0.0 (https://github.com/llvm/llvm-project 244dd2913a43a200f5a6544d424cdc37b771028b)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/ba002e6556acd9e0af16442f4226ddb2a8759567
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Masami-Hiramatsu/fprobe-Add-ftrace-based-probe-APIs/20220112-000050 git checkout ba002e6556acd9e0af16442f4226ddb2a8759567
# save the config file to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=hexagon SHELL=/bin/bash kernel/bpf/
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
In file included from kernel/bpf/syscall.c:34:
include/linux/fprobes.h:28:20: error: field has incomplete type 'struct ftrace_ops'
struct ftrace_ops ftrace;
^
include/linux/ftrace.h:332:8: note: forward declaration of 'struct ftrace_ops'
struct ftrace_ops;
^
In file included from kernel/bpf/syscall.c:34:
>> include/linux/fprobes.h:57:22: warning: no previous prototype for function 'fprobe_find_entry' [-Wmissing-prototypes]
struct fprobe_entry *fprobe_find_entry(struct fprobe *fp, unsigned long addr)
^ include/linux/fprobes.h:57:1: note: declare 'static' if the function is not intended to be used outside of this translation unit struct fprobe_entry *fprobe_find_entry(struct fprobe *fp, unsigned long addr)
^
static
1 warning and 1 error generated.
vim +/fprobe_find_entry +57 include/linux/fprobes.h
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 43
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 44 #ifdef CONFIG_FPROBES
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 45 int register_fprobe(struct fprobe *fp);
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 46 int unregister_fprobe(struct fprobe *fp);
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 47 struct fprobe_entry *fprobe_find_entry(struct fprobe *fp, unsigned long addr);
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 48 #else
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 49 static inline int register_fprobe(struct fprobe *fp)
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 50 {
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 51 return -ENOTSUPP;
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 52 }
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 53 static inline int unregister_fprobe(struct fprobe *fp)
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 54 {
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 55 return -ENOTSUPP;
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 56 }
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 @57 struct fprobe_entry *fprobe_find_entry(struct fprobe *fp, unsigned long addr)
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 58 {
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 59 return NULL;
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 60 }
7b8ed3653f37559 Masami Hiramatsu 2022-01-12 61 #endif7b8ed3653f37559 Masami Hiramatsu 2022-01-12 62
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build botRe: [RFC PATCH 6/6] bpf: Add kprobe link for attaching raw kprobes
In reply to Masami Hiramatsu[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi Masami,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on rostedt-trace/for-next]
[also build test ERROR on bpf-next/master bpf/master linus/master v5.16] [cannot apply to next-20220112]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Masami-Hiramatsu/fprobe-Add-ftrace-based-probe-APIs/20220112-000050 base: https://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git for-next config: i386-randconfig-m021 (https://download.01.org/0day-ci/archive/20220113/202201130737.O0QNCKhe-lkp(a)intel.com/config) compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
reproduce (this is a W=1 build):
# https://github.com/0day-ci/linux/commit/ba002e6556acd9e0af16442f4226ddb2a8759567
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Masami-Hiramatsu/fprobe-Add-ftrace-based-probe-APIs/20220112-000050 git checkout ba002e6556acd9e0af16442f4226ddb2a8759567
# save the config file to linux build tree
mkdir build_dir
make W=1 O=build_dir ARCH=i386 SHELL=/bin/bash kernel/bpf/
If you fix the issue, kindly add following tag as appropriate
All error/warnings (new ones prefixed by >>):
In file included from kernel/bpf/syscall.c:34:
>> include/linux/fprobes.h:28:20: error: field 'ftrace' has incomplete type
28 | struct ftrace_ops ftrace;
| ^~~~~~
>> include/linux/fprobes.h:57:22: warning: no previous prototype for 'fprobe_find_entry' [-Wmissing-prototypes] 57 | struct fprobe_entry *fprobe_find_entry(struct fprobe *fp, unsigned long addr)
| ^~~~~~~~~~~~~~~~~
vim +/ftrace +28 include/linux/fprobes.h
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 23
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 24 struct fprobe {
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 25 struct fprobe_entry *entries;
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 26 unsigned int nentry;
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 27
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 @28 struct ftrace_ops ftrace;
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 29 unsigned long nmissed;
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 30 unsigned int flags;
c0e0471b58c3c9 Masami Hiramatsu 2022-01-12 31 struct rethook *rethook;
c0e0471b58c3c9 Masami Hiramatsu 2022-01-12 32
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 33 void (*entry_handler) (struct fprobe *, unsigned long, struct pt_regs *);
c0e0471b58c3c9 Masami Hiramatsu 2022-01-12 34 void (*exit_handler) (struct fprobe *, unsigned long, struct pt_regs *);
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 35 };
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 36
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 37 #define FPROBE_FL_DISABLED 1
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 38
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 39 static inline bool fprobe_disabled(struct fprobe *fp)
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 40 {
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 41 return (fp) ? fp->flags & FPROBE_FL_DISABLED : false;
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 42 }
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 43
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 44 #ifdef CONFIG_FPROBES
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 45 int register_fprobe(struct fprobe *fp);
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 46 int unregister_fprobe(struct fprobe *fp);
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 47 struct fprobe_entry *fprobe_find_entry(struct fprobe *fp, unsigned long addr);
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 48 #else
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 49 static inline int register_fprobe(struct fprobe *fp)
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 50 {
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 51 return -ENOTSUPP;
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 52 }
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 53 static inline int unregister_fprobe(struct fprobe *fp)
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 54 {
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 55 return -ENOTSUPP;
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 56 }
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 @57 struct fprobe_entry *fprobe_find_entry(struct fprobe *fp, unsigned long addr)
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 58 {
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 59 return NULL;
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 60 }
7b8ed3653f3755 Masami Hiramatsu 2022-01-12 61 #endif7b8ed3653f3755 Masami Hiramatsu 2022-01-12 62
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build botRe: [RFC PATCH 4/6] fprobe: Add exit_handler support
In reply to Masami Hiramatsu[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi Masami,
[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on rostedt-trace/for-next]
[also build test WARNING on bpf-next/master bpf/master linus/master v5.16 next-20220112] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Masami-Hiramatsu/fprobe-Add-ftrace-based-probe-APIs/20220112-000050 base: https://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git for-next config: i386-allmodconfig (https://download.01.org/0day-ci/archive/20220113/202201131009.oKKmEVVq-lkp(a)intel.com/config) compiler: clang version 14.0.0 (https://github.com/llvm/llvm-project 244dd2913a43a200f5a6544d424cdc37b771028b)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/c0e0471b58c3c9122bbff7523f97a363558284eb
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Masami-Hiramatsu/fprobe-Add-ftrace-based-probe-APIs/20220112-000050 git checkout c0e0471b58c3c9122bbff7523f97a363558284eb
# save the config file to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=i386 SHELL=/bin/bash arch/x86/kernel/
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
>> arch/x86/kernel/rethook.c:14:23: warning: no previous prototype for function 'arch_rethook_trampoline_callback' [-Wmissing-prototypes]
__used __visible void arch_rethook_trampoline_callback(struct pt_regs *regs)
^
arch/x86/kernel/rethook.c:14:18: note: declare 'static' if the function is not intended to be used outside of this translation unit
__used __visible void arch_rethook_trampoline_callback(struct pt_regs *regs)
^
static 1 warning generated.
vim +/arch_rethook_trampoline_callback +14 arch/x86/kernel/rethook.c
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 10
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 11 /*
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 12 * Called from arch_rethook_trampoline
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 13 */
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 @14 __used __visible void arch_rethook_trampoline_callback(struct pt_regs *regs)
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 15 {
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 16 unsigned long *frame_pointer;
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 17
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 18 /* fixup registers */
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 19 regs->cs = __KERNEL_CS;
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 20 #ifdef CONFIG_X86_32
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 21 regs->gs = 0;
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 22 #endif
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 23 regs->ip = (unsigned long)&arch_rethook_trampoline;
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 24 regs->orig_ax = ~0UL;
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 25 regs->sp += sizeof(long);
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 26 frame_pointer = ®s->sp + 1;
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 27
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 28 /*
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 29 * The return address at 'frame_pointer' is recovered by the
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 30 * arch_rethook_fixup_return() which called from this
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 31 * rethook_trampoline_handler().
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 32 */
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 33 rethook_trampoline_handler(regs, (unsigned long)frame_pointer);
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 34
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 35 /*
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 36 * Copy FLAGS to 'pt_regs::sp' so that arch_rethook_trapmoline()
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 37 * can do RET right after POPF.
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 38 */
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 39 regs->sp = regs->flags;
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 40 }
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 41 NOKPROBE_SYMBOL(arch_rethook_trampoline_callback);
3727c0ee2be25cf Masami Hiramatsu 2022-01-12 42 ---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build botRe: [RFC PATCH 4/6] fprobe: Add exit_handler support
In reply to Masami Hiramatsu[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]
Hi Masami,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on rostedt-trace/for-next]
[also build test ERROR on bpf-next/master bpf/master linus/master v5.17-rc1 next-20220124] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Masami-Hiramatsu/fprobe-Add-ftrace-based-probe-APIs/20220112-000050 base: https://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git for-next config: s390-allmodconfig (https://download.01.org/0day-ci/archive/20220125/202201250537.lwxse0qH-lkp(a)intel.com/config) compiler: s390-linux-gcc (GCC) 11.2.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/c0e0471b58c3c9122bbff7523f97a363558284eb
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Masami-Hiramatsu/fprobe-Add-ftrace-based-probe-APIs/20220112-000050 git checkout c0e0471b58c3c9122bbff7523f97a363558284eb
# save the config file to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=s390 SHELL=/bin/bash
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
s390-linux-ld: kernel/dma/coherent.o: in function `dma_init_coherent_memory':
coherent.c:(.text+0x13e): undefined reference to `memremap'
s390-linux-ld: coherent.c:(.text+0x274): undefined reference to `memunmap'
s390-linux-ld: kernel/dma/coherent.o: in function `dma_declare_coherent_memory':
coherent.c:(.text+0x820): undefined reference to `memunmap'
s390-linux-ld: kernel/trace/rethook.o: in function `__rethook_find_ret_addr':
>> rethook.c:(.text+0xa8): undefined reference to `arch_rethook_trampoline' s390-linux-ld: kernel/trace/rethook.o: in function `rethook_hook_current': >> rethook.c:(.text+0x1a7e): undefined reference to `arch_rethook_prepare'
s390-linux-ld: kernel/trace/rethook.o: in function `rethook_trampoline_handler':
rethook.c:(.text+0x1bb2): undefined reference to `arch_rethook_trampoline'
s390-linux-ld: drivers/irqchip/irq-al-fic.o: in function `al_fic_init_dt':
irq-al-fic.c:(.init.text+0x98): undefined reference to `of_iomap'
s390-linux-ld: irq-al-fic.c:(.init.text+0x5a8): undefined reference to `iounmap'
s390-linux-ld: drivers/clk/clk-fixed-mmio.o: in function `fixed_mmio_clk_setup':
clk-fixed-mmio.c:(.text+0x9a): undefined reference to `of_iomap'
s390-linux-ld: clk-fixed-mmio.c:(.text+0xe6): undefined reference to `iounmap'
s390-linux-ld: drivers/clocksource/timer-of.o: in function `timer_of_init':
timer-of.c:(.init.text+0x184): undefined reference to `of_iomap'
s390-linux-ld: timer-of.c:(.init.text+0x9bc): undefined reference to `iounmap'
s390-linux-ld: drivers/clocksource/timer-of.o: in function `timer_of_cleanup':
timer-of.c:(.init.text+0xc46): undefined reference to `iounmap'
s390-linux-ld: drivers/clocksource/timer-microchip-pit64b.o: in function `mchp_pit64b_dt_init_timer':
timer-microchip-pit64b.c:(.init.text+0x8a8): undefined reference to `of_iomap'
s390-linux-ld: timer-microchip-pit64b.c:(.init.text+0x1072): undefined reference to `iounmap'
Kconfig warnings: (for reference only)
WARNING: unmet direct dependencies detected for RETHOOK Depends on HAVE_RETHOOK
Selected by
- FPROBES && FTRACE && FUNCTION_TRACER && DYNAMIC_FTRACE_WITH_REGS
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org