btf_encoder: handle .BTF_ids section endianness when cross-compiling
Subsystems (2)
Eduard Zingerman
maintainerPATCH 1/1 btf_encoder: handle .BTF_ids section endianness when cross-compiling
btf_encoder__tag_kfuncs() reads .BTF_ids section to identify a set of kfuncs present in the ELF being processed. This section consists of records of the following shape:
struct btf_id_and_flag {
uint32_t id;
uint32_t flags;
};When endianness of binary operated by pahole differs from the host endianness these fields require byte swap before using.
At the moment such byte swap does not happen and kfuncs are not marked with decl tags when e.g. s390 kernel is compiled on x86.
To reproduces the bug:
- follow instructions from [0] to build an s390 vmlinux;
- execute:
pahole --btf_features_strict=decl_tag_kfuncs,decl_tag \
--btf_encode_detached=test.btf vmlinux
- observe no kfuncs generated:
bpftool btf dump test.btf format c | grep __ksym
This commit fixes the issue by adding an endianness conversion step for .BTF_ids section data before main processing step, modifying the Elf_Data object in-place.
The choice is such in order to:
- minimize changes;
- keep using Elf_Data, as it provides fields {d_size,d_off} used by kfunc processing routines;
- avoid sprinkling bswap_32 at each 'struct btf_id_and_flag' field access in fear of forgetting to add new ones when code is modified.
[0] https://docs.kernel.org/bpf/s390.html
--- btf_encoder.c | 42 ++++++++++++++++++++++++++++++++++++++++++ lib/bpf | 2 +- 2 files changed, 43 insertions(+), 1 deletion(-)
btf_encoder.c +42 -0
@@ -33,6 +33,7 @@#include <stdint.h> #include <search.h> /* for tsearch(), tfind() and tdestroy() */ #include <pthread.h> +#include <byteswap.h> #define BTF_IDS_SECTION ".BTF_ids"#define BTF_ID_FUNC_PFX "__BTF_ID__func__"
@@ -1847,11 +1848,47 @@ static int btf_encoder__tag_kfunc(struct btf_encoder *encoder, struct gobuffer *return 0 ; } +/* If byte order of 'elf' differs from current byte order, convert the data->d_buf. +* ELF file is opened in a readonly mode, so data->d_buf cannot be modified in place. +* Instead, allocate a new buffer if modification is necessary. +*/ +static int convert_idlist_endianness (Elf *elf,Elf_Data *data,bool *copied) +{ +int byteorder, i; +char *elf_ident; +uint32_t *tmp; + + *copied = false; + elf_ident =elf_getident (elf,NULL ); +if (elf_ident ==NULL ) { +fprintf (stderr,"Cannot get ELF identification from header\n" ); +return -EINVAL ; + } + byteorder = elf_ident[EI_DATA ]; +if ((BYTE_ORDER ==LITTLE_ENDIAN && byteorder ==ELFDATA2LSB ) + || (BYTE_ORDER ==BIG_ENDIAN && byteorder ==ELFDATA2MSB )) +return 0 ; + tmp =malloc (data->d_size); +if (tmp ==NULL ) { +fprintf (stderr,"Cannot allocate %lu bytes of memory\n" , data->d_size); +return -ENOMEM ; + } +memcpy (tmp, data->d_buf, data->d_size); + data->d_buf = tmp; + *copied = true; + +/* .BTF_ids sections consist of u32 objects */ +for (i =0 ; i < data->d_size /sizeof (uint32_t ); i++) + tmp[i] =bswap_32 (tmp[i]); +return 0 ; +} +static int btf_encoder__tag_kfuncs (struct btf_encoder *encoder) {const char *filename = encoder->source_filename;struct gobuffer btf_kfunc_ranges = {};struct gobuffer btf_funcs = {}; +bool free_idlist = false;Elf_Data *symbols =NULL ;Elf_Data *idlist =NULL ;Elf_Scn *symscn =NULL ;
@@ -1919,6 +1956,9 @@ static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder) idlist_shndx = i; idlist_addr = shdr.sh_addr; idlist = data; + err =convert_idlist_endianness (elf, idlist, &free_idlist); +if (err <0 ) + goto out; } }
@@ -2031,6 +2071,8 @@ static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder) out:__gobuffer__delete (&btf_funcs);__gobuffer__delete (&btf_kfunc_ranges); +if (free_idlist) +free (idlist->d_buf);if (elf)elf_end (elf);if (fd !=-1 )
lib/bpf +1 -1
@@ -1 +1 @@ -Subproject commit 09b9e83102eb8ab9e540d36b4559c55f3bcdb95d +Subproject commit caa17bdcbfc58e68eaf4d017c058e6577606bf56
Signature · 1 line
2.47.0
Quoted 32 lines · Eduard Zingerman
btf_encoder__tag_kfuncs()reads .BTF_ids section to identify a set of kfuncs present in the ELF being processed. This section consists of records of the following shape:struct btf_id_and_flag { uint32_t id; uint32_t flags; };When endianness of binary operated by pahole differs from the host endianness these fields require byte swap before using.
At the moment such byte swap does not happen and kfuncs are not marked with decl tags when e.g. s390 kernel is compiled on x86.
To reproduces the bug:
- follow instructions from [0] to build an s390 vmlinux;
- execute:pahole --btf_features_strict=decl_tag_kfuncs,decl_tag \ --btf_encode_detached=test.btf vmlinux- observe no kfuncs generated:
bpftool btf dump test.btf format c | grep__ksymThis commit fixes the issue by adding an endianness conversion step for .BTF_ids section data before main processing step, modifying the
Elf_Dataobject in-place.
The choice is such in order to:
- minimize changes;
- keep usingElf_Data, as it provides fields {d_size,d_off} used by kfunc processing routines;
- avoid sprinklingbswap_32at each 'structbtf_id_and_flag' field access in fear of forgetting to add new ones when code is modified.
Quoted 2 lines
LGTM, Thanks!
Quoted 8 lines · Eduard Zingerman
btf_encoder__tag_kfuncs()reads .BTF_ids section to identify a set of kfuncs present in the ELF being processed. This section consists of records of the following shape:struct btf_id_and_flag { uint32_t id; uint32_t flags; };
it contains pairs like above and also just id arrays with no flags, but that does not matter for the patch functionality, because you swap by u32 values anyway
Quoted 14 lines
When endianness of binary operated by pahole differs from the host endianness these fields require byte swap before using.
At the moment such byte swap does not happen and kfuncs are not marked with decl tags when e.g. s390 kernel is compiled on x86.
To reproduces the bug:
- follow instructions from [0] to build an s390 vmlinux;
- execute:
pahole --btf_features_strict=decl_tag_kfuncs,decl_tag \
--btf_encode_detached=test.btf vmlinux
- observe no kfuncs generated:
bpftool btf dump test.btf format c | grep__ksym
Quoted 9 lines
This commit fixes the issue by adding an endianness conversion step for .BTF_ids section data before main processing step, modifying the
Elf_Dataobject in-place.
The choice is such in order to:
- minimize changes;
- keep usingElf_Data, as it provides fields {d_size,d_off} used by kfunc processing routines;
- avoid sprinklingbswap_32at each 'structbtf_id_and_flag' field access in fear of forgetting to add new ones when code is modified.
lgtm, some questions below
Quoted 47 lines
[0] https://docs.kernel.org/bpf/s390.html
Cc Alan Maguire <[email protected]>Cc Daniel Xu <[email protected]>Cc Kumar Kartikeya Dwivedi <[email protected]>Cc Vadim Fedorenko <[email protected]>Fixes 72e88f29c6f7 ("pahole: Inject kfunc decl tags into BTF")Signed-off-by Eduard Zingerman <[email protected]>--- btf_encoder.c | 42 ++++++++++++++++++++++++++++++++++++++++++ lib/bpf | 2 +- 2 files changed, 43 insertions(+), 1 deletion(-)btf_encoder.c +18
-0@@ -33,6 +33,7 @@#include <stdint.h> #include <search.h> /* for tsearch(), tfind() and tdestroy() */ #include <pthread.h> +#include <byteswap.h> #define BTF_IDS_SECTION ".BTF_ids"#define BTF_ID_FUNC_PFX "__BTF_ID__func__"@@ -1847,11 +1848,47 @@ static int btf_encoder__tag_kfunc(struct btf_encoder *encoder, struct gobuffer *return 0 ; } +/* If byte order of 'elf' differs from current byte order, convert the data->d_buf. +* ELF file is opened in a readonly mode, so data->d_buf cannot be modified in place. +* Instead, allocate a new buffer if modification is necessary. +*/ +static int convert_idlist_endianness (Elf *elf,Elf_Data *data,bool *copied) +{ +int byteorder, i; +char *elf_ident; +uint32_t *tmp; + + *copied = false; + elf_ident =elf_getident (elf,NULL ); +if (elf_ident ==NULL ) { +fprintf (stderr,"Cannot get ELF identification from header\n" ); +return -EINVAL ; + } + byteorder = elf_ident[EI_DATA ];
btf_encoder.c static int btf_encoder__tag_kfunc(struct btf_encoder *encoder, struct gobuffer * · line 1877
Quoted 10 lines
btf_encoder.c +10
-0+if ((BYTE_ORDER ==LITTLE_ENDIAN && byteorder ==ELFDATA2LSB ) + || (BYTE_ORDER ==BIG_ENDIAN && byteorder ==ELFDATA2MSB )) +return 0 ; + tmp =malloc (data->d_size); +if (tmp ==NULL ) { +fprintf (stderr,"Cannot allocate %lu bytes of memory\n" , data->d_size); +return -ENOMEM ; + } +memcpy (tmp, data->d_buf, data->d_size); + data->d_buf = tmp;
will the original data->d_buf be leaked? are we allowed to assign d_buf like that? ;-)
Quoted 28 lines
btf_encoder.c +12
-0+ *copied = true; + +/* .BTF_ids sections consist of u32 objects */ +for (i =0 ; i < data->d_size /sizeof (uint32_t ); i++) + tmp[i] =bswap_32 (tmp[i]); +return 0 ; +} +static int btf_encoder__tag_kfuncs (struct btf_encoder *encoder) {const char *filename = encoder->source_filename;struct gobuffer btf_kfunc_ranges = {};struct gobuffer btf_funcs = {}; +bool free_idlist = false;Elf_Data *symbols =NULL ;Elf_Data *idlist =NULL ;Elf_Scn *symscn =NULL ;@@ -1919,6 +1956,9 @@ static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder) idlist_shndx = i; idlist_addr = shdr.sh_addr; idlist = data; + err =convert_idlist_endianness (elf, idlist, &free_idlist); +if (err <0 ) + goto out; } }
btf_encoder.c static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder) · line 2077
Quoted 7 lines
btf_encoder.c +2
-0@@ -2031,6 +2071,8 @@ static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder) out:__gobuffer__delete (&btf_funcs);__gobuffer__delete (&btf_kfunc_ranges); +if (free_idlist) +free (idlist->d_buf);if (elf)elf_end (elf);
curious, would elf_end try to free the d_buf at some point?
lib/bpf line 1
Quoted 8 lines
btf_encoder.c +0
-0if (fd !=-1 )lib/bpf +1
-1@@ -1 +1 @@ -Subproject commit 09b9e83102eb8ab9e540d36b4559c55f3bcdb95d +Subproject commit caa17bdcbfc58e68eaf4d017c058e6577606bf56
I think this should not be part of the patch
thanks,
jirka
Eduard Zingerman
maintainerQuoted 5 lines · Jiri Olsa
Quoted 4 lines · Eduard Zingerman
btf_encoder__tag_kfuncs()reads .BTF_ids section to identify a set of kfuncs present in the ELF being processed. This section consists of records of the following shape:
Quoted 8 lines
Quoted 4 lines
struct btf_id_and_flag { uint32_t id; uint32_t flags; };it contains pairs like above and also just id arrays with no flags, but that does not matter for the patch functionality, because you swap by u32 values anyway
Right, I'll update the description, thank you.
[...]
Quoted 25 lines
Quoted 25 lines
btf_encoder.c +21
-0@@ -1847,11 +1848,47 @@ static int btf_encoder__tag_kfunc(struct btf_encoder *encoder, struct gobuffer *return 0 ; } +/* If byte order of 'elf' differs from current byte order, convert the data->d_buf. +* ELF file is opened in a readonly mode, so data->d_buf cannot be modified in place. +* Instead, allocate a new buffer if modification is necessary. +*/ +static int convert_idlist_endianness (Elf *elf,Elf_Data *data,bool *copied) +{ +int byteorder, i; +char *elf_ident; +uint32_t *tmp; + + *copied = false; + elf_ident =elf_getident (elf,NULL ); +if (elf_ident ==NULL ) { +fprintf (stderr,"Cannot get ELF identification from header\n" ); +return -EINVAL ; + } + byteorder = elf_ident[EI_DATA ]; +if ((BYTE_ORDER ==LITTLE_ENDIAN && byteorder ==ELFDATA2LSB ) + || (BYTE_ORDER ==BIG_ENDIAN && byteorder ==ELFDATA2MSB )) +return 0 ; + tmp =malloc (data->d_size);
Quoted 8 lines
btf_encoder.c static int btf_encoder__tag_kfunc(struct btf_encoder *encoder, struct gobuffer * · line 1877
Quoted 6 lines
btf_encoder.c +6
-0+if (tmp ==NULL ) { +fprintf (stderr,"Cannot allocate %lu bytes of memory\n" , data->d_size); +return -ENOMEM ; + } +memcpy (tmp, data->d_buf, data->d_size); + data->d_buf = tmp;will the original
data->d_bufbe leaked? are we allowed to assignd_buflike that? ;-)
Well, before sending I checked using address sanitizer, and it did not complain. As far as I understand elfutils elf_getdata.c / elf_end.c [0]:
- elf_getdata() allocates memory for full section (elf_getdata.c:333), before setting d_buf field of Elf_Data;
- elf_end() frees memory for full section (elf_end.c:174).
So I assumed that this is hacky but not that bad.
Given that current patch depends on implementation details it is probably better to switch to one of the alternatives:
a. allocate new Elf_Data object using elf_newdata() API;
b. just allocate a fake instance of Elf_Data on stack in btf_encoder__tag_kfuncs().
(a) seems to be an Ok option, wdyt?
[0] b2f225d6bff8 ("Consolidate and add files to clean target variables")
git://sourceware.org/git/elfutils.git
[...]
Quoted 10 lines
lib/bpf line 1
Quoted 8 lines
btf_encoder.c +0
-0if (fd !=-1 )lib/bpf +1
-1@@ -1 +1 @@ -Subproject commit 09b9e83102eb8ab9e540d36b4559c55f3bcdb95d +Subproject commit caa17bdcbfc58e68eaf4d017c058e6577606bf56I think this should not be part of the patch
Sorry, didn't notice this thing.
Eduard Zingerman
maintainerOn Fri, 2024-11-22 at 10:08 -0800, Eduard Zingerman wrote:
[...]
Quoted 7 lines
So I assumed that this is hacky but not that bad.
Given that current patch depends on implementation details it is probably better to switch to one of the alternatives:
a. allocate newElf_Dataobject usingelf_newdata()API;
b. just allocate a fake instance ofElf_Dataon stack inbtf_encoder__tag_kfuncs().(a) seems to be an Ok option, wdyt?
Meh, any of these raise some questions about validity of API usage. I'll just add a custom wrapper:
struct converted_elf_data {
void *d_buf;
size_t d_size;
size_t d_off;
bool owns_buf;};
[...]
Quoted 10 lines · Eduard Zingerman
btf_encoder__tag_kfuncs()reads .BTF_ids section to identify a set of kfuncs present in the ELF being processed. This section consists of records of the following shape:struct btf_id_and_flag { uint32_t id; uint32_t flags; };
Can we just set data->d_type to ELF_T_WORD and let libelf handle the byte swap?
Quoted 36 lines
When endianness of binary operated by pahole differs from the host endianness these fields require byte swap before using.
At the moment such byte swap does not happen and kfuncs are not marked with decl tags when e.g. s390 kernel is compiled on x86.
To reproduces the bug:
- follow instructions from [0] to build an s390 vmlinux;
- execute:
pahole --btf_features_strict=decl_tag_kfuncs,decl_tag \
--btf_encode_detached=test.btf vmlinux
- observe no kfuncs generated:
bpftool btf dump test.btf format c | grep__ksymThis commit fixes the issue by adding an endianness conversion step for .BTF_ids section data before main processing step, modifying the
Elf_Dataobject in-place.
The choice is such in order to:
- minimize changes;
- keep usingElf_Data, as it provides fields {d_size,d_off} used by kfunc processing routines;
- avoid sprinklingbswap_32at each 'structbtf_id_and_flag' field access in fear of forgetting to add new ones when code is modified.[0] https://docs.kernel.org/bpf/s390.html
Cc Alan Maguire <[email protected]>Cc Daniel Xu <[email protected]>Cc Kumar Kartikeya Dwivedi <[email protected]>Cc Vadim Fedorenko <[email protected]>Fixes 72e88f29c6f7 ("pahole: Inject kfunc decl tags into BTF")Signed-off-by Eduard Zingerman <[email protected]>--- btf_encoder.c | 42 ++++++++++++++++++++++++++++++++++++++++++ lib/bpf | 2 +- 2 files changed, 43 insertions(+), 1 deletion(-)
[...]
Eduard Zingerman
maintainerQuoted 6 lines · Andrii Nakryiko
Quoted 5 lines · Eduard Zingerman
btf_encoder__tag_kfuncs()reads .BTF_ids section to identify a set of kfuncs present in the ELF being processed. This section consists of records of the following shape:
Quoted 7 lines
Quoted 5 lines
struct btf_id_and_flag { uint32_t id; uint32_t flags; };Can we just set
data->d_typetoELF_T_WORDand let libelf handle the byte swap?
When I tried 'data->d_type = ELF_T_WORD' + gelf_xlatetom() snippet suggested by Tony Ambardar some time ago, I got a write protection error. Concluded that this is so, because file is opened in O_RDONLY mode.
(Also please note v2 of this patch).
[...]
Quoted 14 lines · Eduard Zingerman
Quoted 12 lines · Andrii Nakryiko
Quoted 10 lines · Eduard Zingerman
btf_encoder__tag_kfuncs()reads .BTF_ids section to identify a set of kfuncs present in the ELF being processed. This section consists of records of the following shape:struct btf_id_and_flag { uint32_t id; uint32_t flags; };
Quoted 5 lines
Quoted 1 line
Can we just set
data->d_typetoELF_T_WORDand let libelf handle the byte swap?When I tried '
data->d_type=ELF_T_WORD' +gelf_xlatetom()snippet suggested by Tony Ambardar some time ago, I got a write protection error. Concluded that this is so, because file is opened inO_RDONLYmode.
Ok, maybe don't follow my words *exactly*, just in spirit ;) I see there is elf_getdata_rawchunk() API in libelf, which seems useful:
/* Get data translated from a chunk of the file contents as section data
would be for TYPE. The resulting Elf_Data pointer is valid until elf_end (ELF) is called. */
extern Elf_Data *elf_getdata_rawchunk (Elf *__elf,
int64_t __offset, size_t __size,
Elf_Type __type);I don't know the surrounding code, I was just hoping to leverage libelf's byte swapping support (which I think I learned from Tony as well). But if it's too inconvenient, so be it.
Quoted 5 lines
(Also please note v2 of this patch).
[...]
Eduard Zingerman
maintainerOn Tue, 2024-11-26 at 13:51 -0800, Andrii Nakryiko wrote:
[...]
Quoted 11 lines
Quoted 3 lines
When I tried '
data->d_type=ELF_T_WORD' +gelf_xlatetom()snippet suggested by Tony Ambardar some time ago, I got a write protection error. Concluded that this is so, because file is opened inO_RDONLYmode.Ok, maybe don't follow my words *exactly*, just in spirit ;) I see there is
elf_getdata_rawchunk()API in libelf, which seems useful:/* Get data translated from a chunk of the file contents as section data
would be for TYPE. The resultingElf_Datapointer is valid untilelf_end(ELF) is called. */
externElf_Data*elf_getdata_rawchunk(Elf*__elf,
Quoted 6 lines
int64_t __offset, size_t __size, Elf_Type __type);I don't know the surrounding code, I was just hoping to leverage libelf's byte swapping support (which I think I learned from Tony as well). But if it's too inconvenient, so be it.
I missed this API, it works and makes the patch much smaller, thank you for finding it.