bpftool: Fix skeletons compilation for older kernels
cover bpftool: Fix skeletons compilation for older kernels
At runtime, bpftool may run its own BPF programs to get the pids of processes referencing BPF programs, or to profile programs. The skeletons for these programs rely on a vmlinux.h header and may fail to compile when building bpftool on hosts running older kernels, where some structs or enums are not defined. In this set, we address this issue by using local definitions for struct perf_event, struct bpf_perf_link, BPF_LINK_TYPE_PERF_EVENT (pids.bpf.c) and struct bpf_perf_event_value (profiler.bpf.c).
This set contains patches 1 to 3 from Alexander Lobakin's series, "bpf:
random unpopular userspace fixes (32 bit et al)" (v2) [0], from April 2022. An additional patch defines a local version of BPF_LINK_TYPE_PERF_EVENT in bpftool's pids.bpf.c.
[0] https://lore.kernel.org/bpf/[email protected]/
Alexander Lobakin (3):
bpftool: use a local copy of perf_event to fix accessing ::bpf_cookie
bpftool: define a local bpf_perf_link to fix accessing its fields
bpftool: use a local bpf_perf_event_value to fix accessing its fields
Quentin Monnet (1):
bpftool: Use a local copy of BPF_LINK_TYPE_PERF_EVENT in
pid_iter.bpf.c
tools/bpf/bpftool/skeleton/pid_iter.bpf.c | 26 +++++++++++++++++----- tools/bpf/bpftool/skeleton/profiler.bpf.c | 27 ++++++++++++++--------- 2 files changed, 38 insertions(+), 15 deletions(-)
Signature · 1 line
2.34.1
PATCH 1/4 bpftool: use a local copy of perf_event to fix accessing ::bpf_cookie
From: Alexander Lobakin <[email protected]>
When CONFIG_PERF_EVENTS is not set, struct perf_event remains empty. However, the structure is being used by bpftool indirectly via BTF. This leads to:
skeleton/pid_iter.bpf.c:49:30: error: no member named 'bpf_cookie' in 'struct perf_event'
return BPF_CORE_READ(event, bpf_cookie);
~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~...
skeleton/pid_iter.bpf.c:49:9: error: returning 'void' from a function with incompatible result type '__u64' (aka 'unsigned long long')
return BPF_CORE_READ(event, bpf_cookie);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Tools and samples can't use any CONFIG_ definitions, so the fields used there should always be present.
Define struct perf_event___local with the preserve_access_index attribute inside the pid_iter BPF prog to allow compiling on any configs. CO-RE will substitute it with the real struct perf_event accesses later on.
--- tools/bpf/bpftool/skeleton/pid_iter.bpf.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
tools/bpf/bpftool/skeleton/pid_iter.bpf.c +5 -1
@@ -15,6 +15,10 @@ enum bpf_obj_type { BPF_OBJ_BTF, }; +struct perf_event___local { +u64 bpf_cookie; +} __attribute__((preserve_access_index)); + extern const void bpf_link_fops __ksym; extern const void bpf_map_fops __ksym; extern const void bpf_prog_fops __ksym;
@@ -41,8 +45,8 @@ static __always_inline __u32 get_obj_id(void *ent, enum bpf_obj_type type)/* could be used only with BPF_LINK_TYPE_PERF_EVENT links */ static __u64 get_bpf_cookie (struct bpf_link *link) { +struct perf_event___local *event;struct bpf_perf_link *perf_link; -struct perf_event *event; perf_link = container_of(link, struct bpf_perf_link, link); event =BPF_CORE_READ (perf_link, perf_file, private_data);
Signature · 1 line
2.34.1
PATCH 2/4 bpftool: define a local bpf_perf_link to fix accessing its fields
In reply to Quentin MonnetFrom: Alexander Lobakin <[email protected]>
When building bpftool with !CONFIG_PERF_EVENTS:
skeleton/pid_iter.bpf.c:47:14: error: incomplete definition of type 'struct bpf_perf_link'
perf_link = container_of(link, struct bpf_perf_link, link);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helpers.h:74:22: note: expanded from macro 'container_of'
((type *)(__mptr - offsetof(type, member))); \
^~~~~~~~~~~~~~~~~~~~~~
tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helpers.h:68:60: note: expanded from macro 'offsetof'
#define offsetof(TYPE, MEMBER) ((unsigned long)&((TYPE *)0)->MEMBER)
~~~~~~~~~~~^
skeleton/pid_iter.bpf.c:44:9: note: forward declaration of 'struct bpf_perf_link'
struct bpf_perf_link *perf_link;
^&bpf_perf_link is being defined and used only under the ifdef. Define struct bpf_perf_link___local with the preserve_access_index attribute inside the pid_iter BPF prog to allow compiling on any configs. CO-RE will substitute it with the real struct bpf_perf_link accesses later on.container_of() is not CO-REd, but it is a noop for bpf_perf_link <-> bpf_link and the local copy is a full mirror of the original structure.
--- tools/bpf/bpftool/skeleton/pid_iter.bpf.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
tools/bpf/bpftool/skeleton/pid_iter.bpf.c +7 -2
@@ -15,6 +15,11 @@ enum bpf_obj_type { BPF_OBJ_BTF, }; +struct bpf_perf_link___local { +struct bpf_link link; +struct file *perf_file; +} __attribute__((preserve_access_index)); +struct perf_event___local {u64 bpf_cookie; } __attribute__((preserve_access_index));
@@ -45,10 +50,10 @@ static __always_inline __u32 get_obj_id(void *ent, enum bpf_obj_type type)/* could be used only with BPF_LINK_TYPE_PERF_EVENT links */ static __u64 get_bpf_cookie (struct bpf_link *link) { +struct bpf_perf_link___local *perf_link;struct perf_event___local *event; -struct bpf_perf_link *perf_link; - perf_link = container_of(link, struct bpf_perf_link, link); + perf_link = container_of(link, struct bpf_perf_link___local, link); event =BPF_CORE_READ (perf_link, perf_file, private_data);return BPF_CORE_READ (event, bpf_cookie); }
Signature · 1 line
2.34.1
Re: [PATCH bpf-next 2/4] bpftool: define a local bpf_perf_link to fix accessing its fields
Quoted 17 lines · Quentin Monnet
From: Alexander Lobakin <[email protected]>
When building bpftool with !CONFIG_PERF_EVENTS:
skeleton/pid_iter.bpf.c:47:14: error: incomplete definition of type 'structbpf_perf_link'perf_link = container_of(link, struct bpf_perf_link, link); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helpers.h:74:22: note: expanded from macro 'container_of' ((type *)(__mptr - offsetof(type, member))); \ ^~~~~~~~~~~~~~~~~~~~~~ tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helpers.h:68:60: note: expanded from macro 'offsetof' #define offsetof(TYPE, MEMBER) ((unsigned long)&((TYPE *)0)->MEMBER) ~~~~~~~~~~~^ skeleton/pid_iter.bpf.c:44:9: note: forward declaration of 'struct bpf_perf_link' struct bpf_perf_link *perf_link; ^
Quoted 6 lines
&bpf_perf_linkis being defined and used only under the ifdef. Define structbpf_perf_link___localwith thepreserve_access_indexattribute inside thepid_iterBPF prog to allow compiling on any configs. CO-RE will substitute it with the real structbpf_perf_linkaccesses later on.container_of()is not CO-REd, but it is a noop for
'container_of() is not CO-REd' is incorrect.
#define container_of(ptr, type, member) \
({ \
void *__mptr = (void *)(ptr); \
((type *)(__mptr - offsetof(type, member))); \
})offsetof() will do necessary CO-RE relocation if the field is specified with preserve_access_index attribute. So container_of will actually do CO-RE relocation as well.
Quoted 40 lines
bpf_perf_link<->bpf_linkand the local copy is a full mirror of the original structure.Fixes cbdaf71f7e65 ("bpftool: Add bpf_cookie to link output")Suggested-by Andrii Nakryiko <[email protected]>Signed-off-by Alexander Lobakin <[email protected]>Signed-off-by Quentin Monnet <[email protected]>--- tools/bpf/bpftool/skeleton/pid_iter.bpf.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)tools/bpf/bpftool/skeleton/pid_iter.bpf.c +7
-2@@ -15,6 +15,11 @@ enum bpf_obj_type { BPF_OBJ_BTF, }; +struct bpf_perf_link___local { +struct bpf_link link; +struct file *perf_file; +} __attribute__((preserve_access_index)); +struct perf_event___local {u64 bpf_cookie; } __attribute__((preserve_access_index));@@ -45,10 +50,10 @@ static __always_inline __u32 get_obj_id(void *ent, enum bpf_obj_type type)/* could be used only with BPF_LINK_TYPE_PERF_EVENT links */ static __u64 get_bpf_cookie (struct bpf_link *link) { +struct bpf_perf_link___local *perf_link;struct perf_event___local *event; -struct bpf_perf_link *perf_link; - perf_link = container_of(link, struct bpf_perf_link, link); + perf_link = container_of(link, struct bpf_perf_link___local, link); event =BPF_CORE_READ (perf_link, perf_file, private_data);return BPF_CORE_READ (event, bpf_cookie); }
Re: [PATCH bpf-next 2/4] bpftool: define a local bpf_perf_link to fix accessing its fields
2023-05-12 07:47 UTC-0700 ~ Yonghong Song <[email protected]>
Quoted 40 lines · Quentin Monnet
Quoted 21 lines
From: Alexander Lobakin <[email protected]>
When building bpftool with !CONFIG_PERF_EVENTS:
skeleton/pid_iter.bpf.c:47:14: error: incomplete definition of type 'structbpf_perf_link'
perf_link=container_of(link, struct bpf_perf_link, link); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helpers.h:74:22:
note: expanded from macro 'container_of'
((type *)(__mptr -offsetof(type, member))); \
^~~~~~~~~~~~~~~~~~~~~~tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helpers.h:68:60:
note: expanded from macro 'offsetof'
#defineoffsetof(TYPE, MEMBER)((unsigned long)&((TYPE *)0)->MEMBER) ~~~~~~~~~~~^skeleton/pid_iter.bpf.c:44:9: note: forward declaration of 'structbpf_perf_link'
structbpf_perf_link*perf_link;
^Quoted 6 lines
&bpf_perf_linkis being defined and used only under the ifdef. Define structbpf_perf_link___localwith thepreserve_access_indexattribute inside thepid_iterBPF prog to allow compiling on any configs. CO-RE will substitute it with the real structbpf_perf_linkaccesses later on.container_of()is not CO-REd, but it is a noop for'
container_of()is not CO-REd' is incorrect.#define
container_of(ptr, type, member)\
({ \
void*__mptr= (void *)(ptr); \
((type *)(__mptr -offsetof(type, member))); \
})
Quoted 3 lines
offsetof()will do necessary CO-RE relocation if the field is specified withpreserve_access_indexattribute. Socontainer_ofwill actually do CO-RE relocation as well.
Thanks! I'll amend the description for the next iteration.
Quentin
PATCH 3/4 bpftool: Use a local copy of BPF_LINK_TYPE_PERF_EVENT in pid_iter.bpf.c
In reply to Quentin MonnetIn order to allow the BPF program in bpftool's pid_iter.bpf.c to compile correctly on hosts where vmlinux.h does not define BPF_LINK_TYPE_PERF_EVENT (running kernel versions lower than 5.15, for example), define and use a local copy of the enum value. This requires LLVM 12 or newer to build the BPF program.
--- tools/bpf/bpftool/skeleton/pid_iter.bpf.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
tools/bpf/bpftool/skeleton/pid_iter.bpf.c +9 -2
@@ -24,6 +24,10 @@ struct perf_event___local {u64 bpf_cookie; } __attribute__((preserve_access_index)); +enum bpf_link_type___local { + BPF_LINK_TYPE_PERF_EVENT___local =7 , +}; + extern const void bpf_link_fops __ksym; extern const void bpf_map_fops __ksym; extern const void bpf_prog_fops __ksym;
@@ -93,10 +97,13 @@ int iter(struct bpf_iter__task_file *ctx) e.pid = task->tgid; e.id =get_obj_id (file->private_data, obj_type); -if (obj_type ==BPF_OBJ_LINK ) { +if (obj_type ==BPF_OBJ_LINK && + bpf_core_enum_value_exists(enum bpf_link_type___local, + BPF_LINK_TYPE_PERF_EVENT___local)) {struct bpf_link *link = (struct bpf_link *) file->private_data; -if (BPF_CORE_READ (link, type) ==BPF_LINK_TYPE_PERF_EVENT ) { + if (link->type == bpf_core_enum_value(enum bpf_link_type___local, + BPF_LINK_TYPE_PERF_EVENT___local)) { e.has_bpf_cookie = true; e.bpf_cookie =get_bpf_cookie (link); }
Signature · 1 line
2.34.1
PATCH 4/4 bpftool: use a local bpf_perf_event_value to fix accessing its fields
In reply to Quentin MonnetFrom: Alexander Lobakin <[email protected]>
Fix the following error when building bpftool:
CLANG profiler.bpf.o
CLANG pid_iter.bpf.oskeleton/profiler.bpf.c:18:21: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_perf_event_value'
__uint(value_size, sizeof(struct bpf_perf_event_value));
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helpers.h:13:39: note: expanded from macro '__uint'
tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helper_defs.h:7:8: note: forward declaration of 'struct bpf_perf_event_value'
struct bpf_perf_event_value;
^struct bpf_perf_event_value is being used in the kernel only when CONFIG_BPF_EVENTS is enabled, so it misses a BTF entry then. Define struct bpf_perf_event_value___local with the preserve_access_index attribute inside the pid_iter BPF prog to allow compiling on any configs. It is a full mirror of a UAPI structure, so is compatible both with and w/o CO-RE. bpf_perf_event_read_value() requires a pointer of the original type, so a cast is needed.
--- tools/bpf/bpftool/skeleton/profiler.bpf.c | 27 ++++++++++++++--------- 1 file changed, 17 insertions(+), 10 deletions(-)
tools/bpf/bpftool/skeleton/profiler.bpf.c +17 -10
@@ -4,6 +4,12 @@#include <bpf/bpf_helpers.h> #include <bpf/bpf_tracing.h> +struct bpf_perf_event_value___local { +__u64 counter; +__u64 enabled; +__u64 running; +} __attribute__((preserve_access_index)); +/* map of perf event fds, num_cpu * num_metric entries */ struct { __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
@@ -15,14 +21,14 @@ struct {struct { __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); __uint(key_size, sizeof(u32)); - __uint(value_size, sizeof(struct bpf_perf_event_value)); + __uint(value_size, sizeof(struct bpf_perf_event_value___local)); } fentry_readings SEC(".maps");/* accumulated readings */ struct { __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); __uint(key_size, sizeof(u32)); - __uint(value_size, sizeof(struct bpf_perf_event_value)); + __uint(value_size, sizeof(struct bpf_perf_event_value___local)); } accum_readings SEC(".maps");/* sample counts, one per cpu */
@@ -39,7 +45,7 @@ const volatile __u32 num_metric = 1; SEC("fentry/XXX")int BPF_PROG (fentry_XXX ) { -struct bpf_perf_event_value *ptrs[MAX_NUM_MATRICS ]; +struct bpf_perf_event_value___local *ptrs[MAX_NUM_MATRICS ];u32 key =bpf_get_smp_processor_id ();u32 i;
@@ -53,10 +59,10 @@ int BPF_PROG(fentry_XXX) }for (i =0 ; i < num_metric && i <MAX_NUM_MATRICS ; i++) { -struct bpf_perf_event_value reading; +struct bpf_perf_event_value___local reading;int err; - err =bpf_perf_event_read_value (&events, key, &reading, + err =bpf_perf_event_read_value (&events, key, (void *)&reading,sizeof (reading));if (err)return 0 ;
@@ -68,14 +74,14 @@ int BPF_PROG(fentry_XXX) }static inline void -fexit_update_maps (u32 id,struct bpf_perf_event_value *after) +fexit_update_maps (u32 id,struct bpf_perf_event_value___local *after) { -struct bpf_perf_event_value *before, diff; +struct bpf_perf_event_value___local *before, diff; before =bpf_map_lookup_elem (&fentry_readings, &id);/* only account samples with a valid fentry_reading */ if (before && before->counter) { -struct bpf_perf_event_value *accum; +struct bpf_perf_event_value___local *accum; diff.counter = after->counter - before->counter; diff.enabled = after->enabled - before->enabled;
@@ -93,7 +99,7 @@ fexit_update_maps(u32 id, struct bpf_perf_event_value *after) SEC("fexit/XXX")int BPF_PROG (fexit_XXX ) { -struct bpf_perf_event_value readings[MAX_NUM_MATRICS ]; +struct bpf_perf_event_value___local readings[MAX_NUM_MATRICS ];u32 cpu =bpf_get_smp_processor_id ();u32 i, zero =0 ;int err;
@@ -102,7 +108,8 @@ int BPF_PROG(fexit_XXX)/* read all events before updating the maps, to reduce error */ for (i =0 ; i < num_metric && i <MAX_NUM_MATRICS ; i++) { err =bpf_perf_event_read_value (&events, cpu + i * num_cpu, - readings + i,sizeof (*readings)); + (void *)(readings + i), +sizeof (*readings));if (err)return 0 ; }
Signature · 1 line
2.34.1
Re: [PATCH bpf-next 4/4] bpftool: use a local bpf_perf_event_value to fix accessing its fields
Quoted 14 lines · Quentin Monnet
From: Alexander Lobakin <[email protected]>
Fix the following error when building bpftool:
CLANG profiler.bpf.o
CLANG pid_iter.bpf.oskeleton/profiler.bpf.c:18:21: error: invalid application of 'sizeof' to an incomplete type 'structbpf_perf_event_value'__uint(value_size, sizeof(struct bpf_perf_event_value)); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helpers.h:13:39: note: expanded from macro '__uint' tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helper_defs.h:7:8: note: forward declaration of 'struct bpf_perf_event_value' struct bpf_perf_event_value; ^
Quoted 2 lines
struct
bpf_perf_event_valueis being used in the kernel only whenCONFIG_BPF_EVENTSis enabled, so it misses a BTF entry then.
hi,
when I switch off CONFIG_BPF_EVENTS the bpftool build fails for me with missing BTF error:
GEN vmlinux.h
libbpf: failed to find '.BTF' ELF section in /home/jolsa/kernel/linux-qemu/vmlinux
Error: failed to load BTF from /home/jolsa/kernel/linux-qemu/vmlinux: No data available
make: *** [Makefile:208: vmlinux.h] Error 195
make: *** Deleting file 'vmlinux.h'
so I wonder you need to care about bpf_perf_event_value in that case
jirka
Quoted 111 lines
Define struct
bpf_perf_event_value___localwith thepreserve_access_indexattribute inside thepid_iterBPF prog to allow compiling on any configs. It is a full mirror of a UAPI structure, so is compatible both with and w/o CO-RE.bpf_perf_event_read_value()requires a pointer of the original type, so a cast is needed.Fixes 47c09d6a9f67 ("bpftool: Introduce "prog profile" command")Suggested-by Andrii Nakryiko <[email protected]>Signed-off-by Alexander Lobakin <[email protected]>Signed-off-by Quentin Monnet <[email protected]>--- tools/bpf/bpftool/skeleton/profiler.bpf.c | 27 ++++++++++++++--------- 1 file changed, 17 insertions(+), 10 deletions(-)tools/bpf/bpftool/skeleton/profiler.bpf.c +17
-10@@ -4,6 +4,12 @@#include <bpf/bpf_helpers.h> #include <bpf/bpf_tracing.h> +struct bpf_perf_event_value___local { +__u64 counter; +__u64 enabled; +__u64 running; +} __attribute__((preserve_access_index)); +/* map of perf event fds, num_cpu * num_metric entries */ struct { __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);@@ -15,14 +21,14 @@ struct {struct { __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); __uint(key_size, sizeof(u32)); - __uint(value_size, sizeof(struct bpf_perf_event_value)); + __uint(value_size, sizeof(struct bpf_perf_event_value___local)); } fentry_readings SEC(".maps");/* accumulated readings */ struct { __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); __uint(key_size, sizeof(u32)); - __uint(value_size, sizeof(struct bpf_perf_event_value)); + __uint(value_size, sizeof(struct bpf_perf_event_value___local)); } accum_readings SEC(".maps");/* sample counts, one per cpu */ @@ -39,7 +45,7 @@ const volatile __u32 num_metric = 1; SEC("fentry/XXX")int BPF_PROG (fentry_XXX ) { -struct bpf_perf_event_value *ptrs[MAX_NUM_MATRICS ]; +struct bpf_perf_event_value___local *ptrs[MAX_NUM_MATRICS ];u32 key =bpf_get_smp_processor_id ();u32 i;@@ -53,10 +59,10 @@ int BPF_PROG(fentry_XXX) }for (i =0 ; i < num_metric && i <MAX_NUM_MATRICS ; i++) { -struct bpf_perf_event_value reading; +struct bpf_perf_event_value___local reading;int err; - err =bpf_perf_event_read_value (&events, key, &reading, + err =bpf_perf_event_read_value (&events, key, (void *)&reading,sizeof (reading));if (err)return 0 ;@@ -68,14 +74,14 @@ int BPF_PROG(fentry_XXX) }static inline void -fexit_update_maps (u32 id,struct bpf_perf_event_value *after) +fexit_update_maps (u32 id,struct bpf_perf_event_value___local *after) { -struct bpf_perf_event_value *before, diff; +struct bpf_perf_event_value___local *before, diff; before =bpf_map_lookup_elem (&fentry_readings, &id);/* only account samples with a valid fentry_reading */ if (before && before->counter) { -struct bpf_perf_event_value *accum; +struct bpf_perf_event_value___local *accum; diff.counter = after->counter - before->counter; diff.enabled = after->enabled - before->enabled;@@ -93,7 +99,7 @@ fexit_update_maps(u32 id, struct bpf_perf_event_value *after) SEC("fexit/XXX")int BPF_PROG (fexit_XXX ) { -struct bpf_perf_event_value readings[MAX_NUM_MATRICS ]; +struct bpf_perf_event_value___local readings[MAX_NUM_MATRICS ];u32 cpu =bpf_get_smp_processor_id ();u32 i, zero =0 ;int err;@@ -102,7 +108,8 @@ int BPF_PROG(fexit_XXX)/* read all events before updating the maps, to reduce error */ for (i =0 ; i < num_metric && i <MAX_NUM_MATRICS ; i++) { err =bpf_perf_event_read_value (&events, cpu + i * num_cpu, - readings + i,sizeof (*readings)); + (void *)(readings + i), +sizeof (*readings));if (err)return 0 ; }Signature · 1 line
2.34.1
Re: [PATCH bpf-next 4/4] bpftool: use a local bpf_perf_event_value to fix accessing its fields
2023-05-12 14:59 UTC+0200 ~ Jiri Olsa <[email protected]>
Quoted 28 lines · Quentin Monnet
Quoted 14 lines
From: Alexander Lobakin <[email protected]>
Fix the following error when building bpftool:
CLANG profiler.bpf.o
CLANG pid_iter.bpf.oskeleton/profiler.bpf.c:18:21: error: invalid application of 'sizeof' to an incomplete type 'structbpf_perf_event_value'__uint(value_size, sizeof(struct bpf_perf_event_value)); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helpers.h:13:39: note: expanded from macro '__uint' tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helper_defs.h:7:8: note: forward declaration of 'struct bpf_perf_event_value' struct bpf_perf_event_value; ^Quoted 2 lines
struct
bpf_perf_event_valueis being used in the kernel only whenCONFIG_BPF_EVENTSis enabled, so it misses a BTF entry then.hi,
when I switch offCONFIG_BPF_EVENTSthe bpftool build fails for me with missing BTF error:GEN
vmlinux.h
libbpf: failed to find '.BTF' ELF section in/home/jolsa/kernel/linux-qemu/vmlinux
Error: failed to load BTF from/home/jolsa/kernel/linux-qemu/vmlinux: No data available
make: *** [Makefile:208:vmlinux.h] Error 195
make: *** Deleting file 'vmlinux.h'
Quoted 4 lines
so I wonder you need to care about
bpf_perf_event_valuein that casejirka
Thanks for testing, Jiri! As I understand, this is the vmlinux.h generation failing, not the compilation from the updated skeletons, so I'm not sure this is related to this patch, could be an existing bug. Have you tried the same build without the patches, by any chance?
I'll try to reproduce on my side to try and figure out why libbpf's bpf_parse_elf() fails to find the .BTF section.
Quentin
Re: [PATCH bpf-next 4/4] bpftool: use a local bpf_perf_event_value to fix accessing its fields
In reply to Jiri Olsa2023-05-12 14:59 UTC+0200 ~ Jiri Olsa <[email protected]>
Quoted 28 lines · Quentin Monnet
Quoted 14 lines
From: Alexander Lobakin <[email protected]>
Fix the following error when building bpftool:
CLANG profiler.bpf.o
CLANG pid_iter.bpf.oskeleton/profiler.bpf.c:18:21: error: invalid application of 'sizeof' to an incomplete type 'structbpf_perf_event_value'__uint(value_size, sizeof(struct bpf_perf_event_value)); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helpers.h:13:39: note: expanded from macro '__uint' tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helper_defs.h:7:8: note: forward declaration of 'struct bpf_perf_event_value' struct bpf_perf_event_value; ^Quoted 2 lines
struct
bpf_perf_event_valueis being used in the kernel only whenCONFIG_BPF_EVENTSis enabled, so it misses a BTF entry then.hi,
when I switch offCONFIG_BPF_EVENTSthe bpftool build fails for me with missing BTF error:GEN
vmlinux.h
libbpf: failed to find '.BTF' ELF section in/home/jolsa/kernel/linux-qemu/vmlinux
Error: failed to load BTF from/home/jolsa/kernel/linux-qemu/vmlinux: No data available
make: *** [Makefile:208:vmlinux.h] Error 195
make: *** Deleting file 'vmlinux.h'
Quoted 4 lines
so I wonder you need to care about
bpf_perf_event_valuein that casejirka
Coming back to this - I haven't been able to reproduce. I turned CONFIG_BPF_EVENTS off but had the vmlinux.h building fine in my case, and applying the patchset fixes the build for bpftool. Is there any chance your .../vmlinux was not generated correctly?
I'll re-submit the series after addressing Yonghong's feedback on the commit log.
Thanks,
Quentin
Re: [PATCH bpf-next 4/4] bpftool: use a local bpf_perf_event_value to fix accessing its fields
In reply to Quentin MonnetQuoted 24 lines · Quentin Monnet
From: Alexander Lobakin <[email protected]>
Fix the following error when building bpftool:
CLANG profiler.bpf.o
CLANG pid_iter.bpf.oskeleton/profiler.bpf.c:18:21: error: invalid application of 'sizeof' to an incomplete type 'structbpf_perf_event_value'__uint(value_size, sizeof(struct bpf_perf_event_value)); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helpers.h:13:39: note: expanded from macro '__uint' tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helper_defs.h:7:8: note: forward declaration of 'struct bpf_perf_event_value' struct bpf_perf_event_value; ^struct
bpf_perf_event_valueis being used in the kernel only whenCONFIG_BPF_EVENTSis enabled, so it misses a BTF entry then. Define structbpf_perf_event_value___localwith thepreserve_access_indexattribute inside thepid_iterBPF prog to allow compiling on any configs. It is a full mirror of a UAPI structure, so is compatible both with and w/o CO-RE.bpf_perf_event_read_value()requires a pointer of the original type, so a cast is needed.
Quoted 5 lines
Fixes 47c09d6a9f67 ("bpftool: Introduce "prog profile" command")Suggested-by Andrii Nakryiko <[email protected]>Signed-off-by Alexander Lobakin <[email protected]>Signed-off-by Quentin Monnet <[email protected]>---
What's the point of using vmlinux.h at all if we redefine every single type? bpf_perf_event_value is part of BPF UAPI, so if we included linux/bpf.h header we'd get it.
This feels a bit split-brained. We either drop vmlinux.h completely and use UAPI headers + CO-RE-relocatable definitions of internal types, or we make sure that vmlinux.h does work (e.g., by pre-checking in a very small version of it). Both using vmlinux.h and not relying on it having necessary types seems like the worst of both worlds?...
Quentin, can you see if you can come up with some simple way to use vmlinux.h if building from inside kernel repo, but using a minimized vmlinux.h generated using bpftool gen min_core_btf when building from Github mirror? Sync script could also generate this minimal vmlinux.h automatically, presumably?
Quoted 3 lines
tools/bpf/bpftool/skeleton/profiler.bpf.c | 27 ++++++++++++++--------- 1 file changed, 17 insertions(+), 10 deletions(-)
[...]
Re: [PATCH bpf-next 4/4] bpftool: use a local bpf_perf_event_value to fix accessing its fields
2023-05-16 14:30 UTC-0700 ~ Andrii Nakryiko <[email protected]>
Quoted 31 lines · Quentin Monnet
Quoted 24 lines
From: Alexander Lobakin <[email protected]>
Fix the following error when building bpftool:
CLANG profiler.bpf.o
CLANG pid_iter.bpf.oskeleton/profiler.bpf.c:18:21: error: invalid application of 'sizeof' to an incomplete type 'structbpf_perf_event_value'__uint(value_size, sizeof(struct bpf_perf_event_value)); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helpers.h:13:39: note: expanded from macro '__uint' tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helper_defs.h:7:8: note: forward declaration of 'struct bpf_perf_event_value' struct bpf_perf_event_value; ^struct
bpf_perf_event_valueis being used in the kernel only whenCONFIG_BPF_EVENTSis enabled, so it misses a BTF entry then. Define structbpf_perf_event_value___localwith thepreserve_access_indexattribute inside thepid_iterBPF prog to allow compiling on any configs. It is a full mirror of a UAPI structure, so is compatible both with and w/o CO-RE.bpf_perf_event_read_value()requires a pointer of the original type, so a cast is needed.Quoted 5 lines
Fixes 47c09d6a9f67 ("bpftool: Introduce "prog profile" command")Suggested-by Andrii Nakryiko <[email protected]>Signed-off-by Alexander Lobakin <[email protected]>Signed-off-by Quentin Monnet <[email protected]>---
Quoted 3 lines
What's the point of using
vmlinux.hat all if we redefine every single type?bpf_perf_event_valueis part of BPF UAPI, so if we includedlinux/bpf.hheader we'd get it.
I gave a quick try at the UAPI header before posting this patch, but it was an Ubuntu box and I got the "asm/types.h not found" error. If I remember correctly, one way to fix this is to have the gcc-multilib, which I'd rather avoid to add as a dependency; or adding the correct include path for x86_64 at least, which I haven't tried for bpftool yet.
Quoted 6 lines
This feels a bit split-brained. We either drop
vmlinux.hcompletely and use UAPI headers + CO-RE-relocatable definitions of internal types, or we make sure thatvmlinux.hdoes work (e.g., by pre-checking in a very small version of it). Both usingvmlinux.hand not relying on it having necessary types seems like the worst of both worlds?...
Yeah I do feel like I'm missing something in this set and the approach is not optimal. What do you mean exactly by "pre-checking in a very small version of it"? Checking for the availability of a few types and exit early from the functions if they're missing, because we're assuming we won't have support for the feature?
Quoted 5 lines
Quentin, can you see if you can come up with some simple way to use
vmlinux.hif building from inside kernel repo, but using a minimizedvmlinux.hgenerated usingbpftool gen min_core_btfwhen building from Github mirror? Sync script could also generate this minimalvmlinux.hautomatically, presumably?
Sure, I can look into it. But not right now - I'd like to get the current issue, and the (unrelated) LLVM feature detection, sorted before starting on this.
Thanks for your feedback!
Quentin
Re: [PATCH bpf-next 4/4] bpftool: use a local bpf_perf_event_value to fix accessing its fields
Quoted 50 lines · Quentin Monnet
2023-05-16 14:30 UTC-0700 ~ Andrii Nakryiko <[email protected]>
Quoted 31 lines · Quentin Monnet
Quoted 24 lines
From: Alexander Lobakin <[email protected]>
Fix the following error when building bpftool:
CLANG profiler.bpf.o
CLANG pid_iter.bpf.oskeleton/profiler.bpf.c:18:21: error: invalid application of 'sizeof' to an incomplete type 'structbpf_perf_event_value'__uint(value_size, sizeof(struct bpf_perf_event_value)); ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helpers.h:13:39: note: expanded from macro '__uint' tools/bpf/bpftool/bootstrap/libbpf/include/bpf/bpf_helper_defs.h:7:8: note: forward declaration of 'struct bpf_perf_event_value' struct bpf_perf_event_value; ^struct
bpf_perf_event_valueis being used in the kernel only whenCONFIG_BPF_EVENTSis enabled, so it misses a BTF entry then. Define structbpf_perf_event_value___localwith thepreserve_access_indexattribute inside thepid_iterBPF prog to allow compiling on any configs. It is a full mirror of a UAPI structure, so is compatible both with and w/o CO-RE.bpf_perf_event_read_value()requires a pointer of the original type, so a cast is needed.Quoted 5 lines
Fixes 47c09d6a9f67 ("bpftool: Introduce "prog profile" command")Suggested-by Andrii Nakryiko <[email protected]>Signed-off-by Alexander Lobakin <[email protected]>Signed-off-by Quentin Monnet <[email protected]>---
Quoted 3 lines
What's the point of using
vmlinux.hat all if we redefine every single type?bpf_perf_event_valueis part of BPF UAPI, so if we includedlinux/bpf.hheader we'd get it.I gave a quick try at the UAPI header before posting this patch, but it was an Ubuntu box and I got the "
asm/types.hnot found" error. If I remember correctly, one way to fix this is to have the gcc-multilib, which I'd rather avoid to add as a dependency; or adding the correct include path forx86_64at least, which I haven't tried for bpftool yet.Quoted 6 lines
This feels a bit split-brained. We either drop
vmlinux.hcompletely and use UAPI headers + CO-RE-relocatable definitions of internal types, or we make sure thatvmlinux.hdoes work (e.g., by pre-checking in a very small version of it). Both usingvmlinux.hand not relying on it having necessary types seems like the worst of both worlds?...
Quoted 5 lines
Yeah I do feel like I'm missing something in this set and the approach is not optimal. What do you mean exactly by "pre-checking in a very small version of it"? Checking for the availability of a few types and exit early from the functions if they're missing, because we're assuming we won't have support for the feature?
So I was thinking that we'll keep relying on vmlinux BTF and proper Kconfig for bpftool build inside kernel repo, no changes there. But then we can use bpftool gen min_core_btf on resulting .bpf.o files to dump only types that are actually used/referenced by bpftool's BPF object files, and then we can generate vmlinux.h from that minimized BTF.
I haven't tried it, and I'm sure there will be some hiccups along the way, but that was the idea.
Quoted 10 lines
Quoted 6 lines
Quentin, can you see if you can come up with some simple way to use
vmlinux.hif building from inside kernel repo, but using a minimizedvmlinux.hgenerated usingbpftool gen min_core_btfwhen building from Github mirror? Sync script could also generate this minimalvmlinux.hautomatically, presumably?Sure, I can look into it. But not right now - I'd like to get the current issue, and the (unrelated) LLVM feature detection, sorted before starting on this.
Sounds good. In general, your patches look good to me, I was hoping we can avoid unnecessary definitions of UAPI types, but if that doesn't work out of the box, then I'm fine with it.
Quoted 3 lines
Thanks for your feedback!
Quentin