libbpf: Sync with latest libbpf repo
Subsystems (2)
PATCH 1/2 libbpf: Sync with latest libbpf repo
Sync up to commit 87dff0a2c775 (vmtest: allow building foreign debian rootfs).
--- lib/bpf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bpf +1 -1
@@ -1 +1 @@ -Subproject commit 393a058d061d49d5c3055fa9eefafb4c0c31ccc3 +Subproject commit 87dff0a2c775c5943ca9233e69c81a25f2ed1a77
Signature · 1 line
2.30.2
PATCH 2/2 btf_encoder: Normalize array index type for parallel dwarf loading case
With latest llvm15 built kernel (make -j LLVM=1), I hit the following error when build selftests (make -C tools/testing/selftests/bpf -j LLVM=1):
In file included from skeleton/pid_iter.bpf.c:3:
.../selftests/bpf/tools/build/bpftool/vmlinux.h:84050:9: error: unknown type name
'__builtin_va_list___2'; did you mean '__builtin_va_list'?
typedef __builtin_va_list___2 va_list___2;
^~~~~~~~~~~~~~~~~~~~~
__builtin_va_list
note: '__builtin_va_list' declared here
In file included from skeleton/profiler.bpf.c:3:
.../selftests/bpf/tools/build/bpftool/vmlinux.h:84050:9: error: unknown type name
'__builtin_va_list__ _2'; did you mean '__builtin_va_list'?
typedef __builtin_va_list___2 va_list___2;
^~~~~~~~~~~~~~~~~~~~~
__builtin_va_list note: '__builtin_va_list' declared here
The error can be easily explained with after-dedup vmlinux btf:
[21] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED [2300] STRUCT '__va_list_tag' size=24 vlen=4
'gp_offset' type_id=2 bits_offset=0
'fp_offset' type_id=2 bits_offset=32
'overflow_arg_area' type_id=32 bits_offset=64
'reg_save_area' type_id=32 bits_offset=128 [2308] TYPEDEF 'va_list' type_id=2309
[2309] TYPEDEF '__builtin_va_list' type_id=2310 [2310] ARRAY '(anon)' type_id=2300 index_type_id=21 nr_elems=1
[5289] PTR '(anon)' type_id=2308
[158520] STRUCT 'warn_args' size=32 vlen=2
'fmt' type_id=14 bits_offset=0
'args' type_id=2308 bits_offset=64 [27299] INT '__ARRAY_SIZE_TYPE__' size=4 bits_offset=0 nr_bits=32 encoding=(none) [34590] TYPEDEF '__builtin_va_list' type_id=34591
[34591] ARRAY '(anon)' type_id=2300 index_type_id=27299 nr_elems=1
The typedef __builtin_va_list is a builtin type for the compiler. In the above case, two typedef __builtin_va_list are generated. The reason is due to different array index_type_id. This happened when pahole is running with more than one jobs when parsing dwarf and generating btfs.
Function btf_encoder__encode_cu() is used to do btf encoding for each cu. The function will try to find an "int" type for the cu if it is available, otherwise, it will create a special type with name __ARRAY_SIZE_TYPE__. For example,
file1: yes 'int' type
file2: no 'int' type
In serial mode, file1 is processed first, followed by file2. both will have 'int' type as the array index type since file2 will inherit the index type from file1.
In parallel mode though, arrays in file1 will have index type 'int', and arrays in file2 wil have index type '__ARRAY_SIZE_TYPE__'. This will prevent some legitimate dedup and may have generated vmlinux.h having compilation error.
This patch fixed the issue by normalizing all array_index types to be the first array_index type in the whole btf.
--- btf_encoder.c | 24 +++++++++++++++++++++--- btf_encoder.h | 2 +- pahole.c | 2 +- 3 files changed, 23 insertions(+), 5 deletions(-)
btf_encoder.c +21 -3
@@ -1056,17 +1056,35 @@ out:return err; } -int btf_encoder__encode (struct btf_encoder *encoder) +int btf_encoder__encode (struct btf_encoder *encoder,bool normalize_array_index_tid) { -int err; +int i, err, nr_types, index_type_id =0 ;if (gobuffer__size (&encoder->percpu_secinfo) !=0 )btf_encoder__add_datasec (encoder,PERCPU_SECTION );/* Empty file, nothing to do, so... done! */ -if (btf__type_cnt (encoder->btf) ==1 ) + nr_types =btf__type_cnt (encoder->btf); +if (nr_types ==1 )return 0 ; +if (normalize_array_index_tid) { +for (i =1 ; i < nr_types; i++) { +/* remove the 'const' qualifier so the index_type can be changed. */ +struct btf_type *t = (struct btf_type *)btf__type_by_id (encoder->btf, i); +struct btf_array *arr_info; + +if (!btf_is_array (t)) +continue ; + + arr_info =btf_array (t); +if (index_type_id ==0 ) + index_type_id = arr_info->index_type; +else + arr_info->index_type = index_type_id; + } + } +if (btf__dedup (encoder->btf,NULL )) {fprintf (stderr,"%s: btf__dedup failed!\n" , __func__);return -1 ;
btf_encoder.h +1 -1
@@ -19,7 +19,7 @@ struct list_head;struct btf_encoder *btf_encoder__new (struct cu *cu,const char *detached_filename,struct btf *base_btf,bool skip_encoding_vars,bool force,bool gen_floats,bool verbose);void btf_encoder__delete (struct btf_encoder *encoder); -int btf_encoder__encode (struct btf_encoder *encoder); +int btf_encoder__encode (struct btf_encoder *encoder,bool normalize_array_index_tid);int btf_encoder__encode_cu (struct btf_encoder *encoder,struct cu *cu);
pahole.c +1 -1
@@ -3530,7 +3530,7 @@ try_sole_arg_as_class_names: header =NULL ;if (btf_encode && btf_encoder) {// maybe all CUs were filtered out and thus we don't have an encoder? - err =btf_encoder__encode (btf_encoder); + err =btf_encoder__encode (btf_encoder, conf_load.nr_jobs >1 );if (err) {fputs ("Failed to encode BTF\n" , stderr); goto out_cus_delete;
Signature · 1 line
2.30.2
Re: [PATCH dwarves 2/2] btf_encoder: Normalize array index type for parallel dwarf loading case
Quoted 50 lines · Yonghong Song
With latest llvm15 built kernel (make -j LLVM=1), I hit the following error when build selftests (make -C
tools/testing/selftests/bpf-j LLVM=1):
In file included fromskeleton/pid_iter.bpf.c:3:
.../selftests/bpf/tools/build/bpftool/vmlinux.h:84050:9: error: unknown type name'__builtin_va_list___2'; did you mean '__builtin_va_list'? typedef __builtin_va_list___2 va_list___2; ^~~~~~~~~~~~~~~~~~~~~ __builtin_va_list note: '__builtin_va_list' declared here In file included from skeleton/profiler.bpf.c:3: .../selftests/bpf/tools/build/bpftool/vmlinux.h:84050:9: error: unknown type name '__builtin_va_list__ _2'; did you mean '__builtin_va_list'? typedef __builtin_va_list___2 va_list___2; ^~~~~~~~~~~~~~~~~~~~~ __builtin_va_listnote: '
__builtin_va_list' declared hereThe error can be easily explained with after-dedup vmlinux btf:
[21] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED [2300] STRUCT '__va_list_tag' size=24 vlen=4'gp_offset' type_id=2 bits_offset=0 'fp_offset' type_id=2 bits_offset=32 'overflow_arg_area' type_id=32 bits_offset=64 'reg_save_area' type_id=32 bits_offset=128[2308] TYPEDEF '
va_list' type_id=2309
[2309] TYPEDEF '__builtin_va_list' type_id=2310 [2310] ARRAY '(anon)' type_id=2300 index_type_id=21 nr_elems=1[5289] PTR '(anon)' type_id=2308
[158520] STRUCT 'warn_args' size=32 vlen=2'fmt' type_id=14 bits_offset=0 'args' type_id=2308 bits_offset=64[27299] INT '
__ARRAY_SIZE_TYPE__' size=4 bits_offset=0 nr_bits=32 encoding=(none) [34590] TYPEDEF '__builtin_va_list' type_id=34591
[34591] ARRAY '(anon)' type_id=2300 index_type_id=27299 nr_elems=1The typedef
__builtin_va_listis a builtin type for the compiler. In the above case, two typedef__builtin_va_listare generated. The reason is due to different arrayindex_type_id. This happened when pahole is running with more than one jobs when parsing dwarf and generating btfs.Function
btf_encoder__encode_cu()is used to do btf encoding for each cu. The function will try to find an "int" type for the cu if it is available, otherwise, it will create a special type with name__ARRAY_SIZE_TYPE__. For example,
file1: yes 'int' type
file2: no 'int' type
Quoted 9 lines
In serial mode, file1 is processed first, followed by file2. both will have 'int' type as the array index type since file2 will inherit the index type from file1.
In parallel mode though, arrays in file1 will have index type 'int', and arrays in file2 wil have index type '
__ARRAY_SIZE_TYPE__'. This will prevent some legitimate dedup and may have generatedvmlinux.hhaving compilation error.
I think it is two separate problems.
1. Maybe instead of this generating __ARRAY_SIZE_TYPE__ we should generate proper 'int' type?
2. __builtin_va_list___2 shouldn't have happened, it's libbpf bug. Libbpf handles __builtin_va_list specially (see btf_dump_is_blacklisted()), so we need to fix libbpf to not get confused if there are two __builtin_va_list copies in BTF.
Quoted 10 lines
This patch fixed the issue by normalizing all
array_indextypes to be the firstarray_indextype in the whole btf.Signed-off-by Yonghong Song <[email protected]>--- btf_encoder.c | 24 +++++++++++++++++++++--- btf_encoder.h | 2 +- pahole.c | 2 +- 3 files changed, 23 insertions(+), 5 deletions(-)
[...]
Re: [PATCH dwarves 2/2] btf_encoder: Normalize array index type for parallel dwarf loading case
Quoted 61 lines · Andrii Nakryiko
Quoted 50 lines · Yonghong Song
With latest llvm15 built kernel (make -j LLVM=1), I hit the following error when build selftests (make -C
tools/testing/selftests/bpf-j LLVM=1):
In file included fromskeleton/pid_iter.bpf.c:3:.../selftests/bpf/tools/build/bpftool/vmlinux.h:84050:9: error: unknown type name '__builtin_va_list___2'; did you mean '__builtin_va_list'? typedef __builtin_va_list___2 va_list___2; ^~~~~~~~~~~~~~~~~~~~~ __builtin_va_list note: '__builtin_va_list' declared here In file included from skeleton/profiler.bpf.c:3: .../selftests/bpf/tools/build/bpftool/vmlinux.h:84050:9: error: unknown type name '__builtin_va_list__ _2'; did you mean '__builtin_va_list'? typedef __builtin_va_list___2 va_list___2; ^~~~~~~~~~~~~~~~~~~~~ __builtin_va_list note: '__builtin_va_list' declared hereThe error can be easily explained with after-dedup vmlinux btf:
[21] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED [2300] STRUCT '__va_list_tag' size=24 vlen=4 'gp_offset' type_id=2 bits_offset=0 'fp_offset' type_id=2 bits_offset=32 'overflow_arg_area' type_id=32 bits_offset=64 'reg_save_area' type_id=32 bits_offset=128 [2308] TYPEDEF 'va_list' type_id=2309 [2309] TYPEDEF '__builtin_va_list' type_id=2310 [2310] ARRAY '(anon)' type_id=2300 index_type_id=21 nr_elems=1 [5289] PTR '(anon)' type_id=2308 [158520] STRUCT 'warn_args' size=32 vlen=2 'fmt' type_id=14 bits_offset=0 'args' type_id=2308 bits_offset=64 [27299] INT '__ARRAY_SIZE_TYPE__' size=4 bits_offset=0 nr_bits=32 encoding=(none) [34590] TYPEDEF '__builtin_va_list' type_id=34591 [34591] ARRAY '(anon)' type_id=2300 index_type_id=27299 nr_elems=1The typedef
__builtin_va_listis a builtin type for the compiler. In the above case, two typedef__builtin_va_listare generated. The reason is due to different arrayindex_type_id. This happened when pahole is running with more than one jobs when parsing dwarf and generating btfs.Function
btf_encoder__encode_cu()is used to do btf encoding for each cu. The function will try to find an "int" type for the cu if it is available, otherwise, it will create a special type with name__ARRAY_SIZE_TYPE__. For example,
file1: yes 'int' type
file2: no 'int' typeQuoted 9 lines
In serial mode, file1 is processed first, followed by file2. both will have 'int' type as the array index type since file2 will inherit the index type from file1.
In parallel mode though, arrays in file1 will have index type 'int', and arrays in file2 wil have index type '
__ARRAY_SIZE_TYPE__'. This will prevent some legitimate dedup and may have generatedvmlinux.hhaving compilation error.
Quoted 4 lines
I think it is two separate problems.
1. Maybe instead of this generating
__ARRAY_SIZE_TYPE__we should generate proper 'int' type?
This should work. Will post v2 with this.
Quoted 5 lines
2.
__builtin_va_list___2shouldn't have happened, it's libbpf bug. Libbpf handles__builtin_va_listspecially (seebtf_dump_is_blacklisted()), so we need to fix libbpf to not get confused if there are two__builtin_va_listcopies in BTF.
I checked code. the libbpf prevents generating
typedef <...> __builtin_va_list
since __builtin_va_list is a builtin type.
Here, due to __ARRAY_SIZE_TYPE__ problem, the following are generated in vmlinux.h.
typedef __builtin_va_list va_list;
typedef __builtin_va_list___2 va_list___2;
since __builtin_va_list appears twice in the BTF. But due to the libbpf implementation to skip
typedef <...> __builtin_va_list
We don't have __builtin_va_list___2 defined and this caused compilation error.
Although we could workaround the issue in libbpf such that if the typedef is in the format of
typedef __builtin_va_list<...> <other_type>
we should just emit
typedef __builtin_va_list <other_type>
But fixing the issue in pahole is much better since we won't have va_list___2 any more.
Quoted 13 lines
Quoted 11 lines
This patch fixed the issue by normalizing all
array_indextypes to be the firstarray_indextype in the whole btf.Signed-off-by Yonghong Song <[email protected]>--- btf_encoder.c | 24 +++++++++++++++++++++--- btf_encoder.h | 2 +- pahole.c | 2 +- 3 files changed, 23 insertions(+), 5 deletions(-)[...]
Re: [PATCH dwarves 2/2] btf_encoder: Normalize array index type for parallel dwarf loading case
Quoted 95 lines · Yonghong Song
Quoted 61 lines · Andrii Nakryiko
Quoted 50 lines · Yonghong Song
With latest llvm15 built kernel (make -j LLVM=1), I hit the following error when build selftests (make -C
tools/testing/selftests/bpf-j LLVM=1):
In file included fromskeleton/pid_iter.bpf.c:3:.../selftests/bpf/tools/build/bpftool/vmlinux.h:84050:9: error: unknown type name '__builtin_va_list___2'; did you mean '__builtin_va_list'? typedef __builtin_va_list___2 va_list___2; ^~~~~~~~~~~~~~~~~~~~~ __builtin_va_list note: '__builtin_va_list' declared here In file included from skeleton/profiler.bpf.c:3: .../selftests/bpf/tools/build/bpftool/vmlinux.h:84050:9: error: unknown type name '__builtin_va_list__ _2'; did you mean '__builtin_va_list'? typedef __builtin_va_list___2 va_list___2; ^~~~~~~~~~~~~~~~~~~~~ __builtin_va_list note: '__builtin_va_list' declared hereThe error can be easily explained with after-dedup vmlinux btf:
[21] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED [2300] STRUCT '__va_list_tag' size=24 vlen=4 'gp_offset' type_id=2 bits_offset=0 'fp_offset' type_id=2 bits_offset=32 'overflow_arg_area' type_id=32 bits_offset=64 'reg_save_area' type_id=32 bits_offset=128 [2308] TYPEDEF 'va_list' type_id=2309 [2309] TYPEDEF '__builtin_va_list' type_id=2310 [2310] ARRAY '(anon)' type_id=2300 index_type_id=21 nr_elems=1 [5289] PTR '(anon)' type_id=2308 [158520] STRUCT 'warn_args' size=32 vlen=2 'fmt' type_id=14 bits_offset=0 'args' type_id=2308 bits_offset=64 [27299] INT '__ARRAY_SIZE_TYPE__' size=4 bits_offset=0 nr_bits=32 encoding=(none) [34590] TYPEDEF '__builtin_va_list' type_id=34591 [34591] ARRAY '(anon)' type_id=2300 index_type_id=27299 nr_elems=1The typedef
__builtin_va_listis a builtin type for the compiler. In the above case, two typedef__builtin_va_listare generated. The reason is due to different arrayindex_type_id. This happened when pahole is running with more than one jobs when parsing dwarf and generating btfs.Function
btf_encoder__encode_cu()is used to do btf encoding for each cu. The function will try to find an "int" type for the cu if it is available, otherwise, it will create a special type with name__ARRAY_SIZE_TYPE__. For example,
file1: yes 'int' type
file2: no 'int' typeQuoted 9 lines
In serial mode, file1 is processed first, followed by file2. both will have 'int' type as the array index type since file2 will inherit the index type from file1.
In parallel mode though, arrays in file1 will have index type 'int', and arrays in file2 wil have index type '
__ARRAY_SIZE_TYPE__'. This will prevent some legitimate dedup and may have generatedvmlinux.hhaving compilation error.Quoted 4 lines
I think it is two separate problems.
1. Maybe instead of this generating
__ARRAY_SIZE_TYPE__we should generate proper 'int' type?This should work. Will post v2 with this.
Quoted 5 lines
2.
__builtin_va_list___2shouldn't have happened, it's libbpf bug. Libbpf handles__builtin_va_listspecially (seebtf_dump_is_blacklisted()), so we need to fix libbpf to not get confused if there are two__builtin_va_listcopies in BTF.I checked code. the libbpf prevents generating
typedef <...> __builtin_va_listsince
__builtin_va_listis a builtin type.Here, due to
__ARRAY_SIZE_TYPE__problem, the following are generated invmlinux.h.typedef
__builtin_va_listva_list;
typedef__builtin_va_list___2va_list___2;since
__builtin_va_listappears twice in the BTF. But due to the libbpf implementation to skiptypedef <...> __builtin_va_listWe don't have
__builtin_va_list___2defined and this caused compilation error.
Quoted 8 lines
Although we could workaround the issue in libbpf such that if the typedef is in the format of
typedef __builtin_va_list<...> <other_type>
we should just emit
typedef__builtin_va_list<other_type>But fixing the issue in pahole is much better since we won't have
va_list___2any more.
Sounds good, let's do it in pahole, thanks!
Quoted 14 lines
Quoted 14 lines
Quoted 12 lines
This patch fixed the issue by normalizing all
array_indextypes to be the firstarray_indextype in the whole btf.Signed-off-by Yonghong Song <[email protected]>--- btf_encoder.c | 24 +++++++++++++++++++++--- btf_encoder.h | 2 +- pahole.c | 2 +- 3 files changed, 23 insertions(+), 5 deletions(-)[...]