Skip to contentSkip to conversation

btf_encoder: handle .BTF_ids section endianness when cross-compiling

PATCH Superseded Eduard Zingerman 4 participants 9 messages Nov 22, 2024 – Nov 27, 2024

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Eduard Zingerman

author maintainer
#0 lore raw

PATCH 1/1 btf_encoder: handle .BTF_ids section endianness when cross-compiling

btf_encoder__tag_kfuncs() reads .BTF_ids section to identify a set of kfuncs present in the ELF being processed. This section consists of records of the following shape:

  struct btf_id_and_flag {
      uint32_t id;
      uint32_t flags;
  };

When endianness of binary operated by pahole differs from the host endianness these fields require byte swap before using.

At the moment such byte swap does not happen and kfuncs are not marked with decl tags when e.g. s390 kernel is compiled on x86.
To reproduces the bug:
- follow instructions from [0] to build an s390 vmlinux;
- execute:
pahole --btf_features_strict=decl_tag_kfuncs,decl_tag \
--btf_encode_detached=test.btf vmlinux
- observe no kfuncs generated:
bpftool btf dump test.btf format c | grep __ksym

This commit fixes the issue by adding an endianness conversion step for .BTF_ids section data before main processing step, modifying the Elf_Data object in-place.
The choice is such in order to:
- minimize changes;
- keep using Elf_Data, as it provides fields {d_size,d_off} used by kfunc processing routines;
- avoid sprinkling bswap_32 at each 'struct btf_id_and_flag' field access in fear of forgetting to add new ones when code is modified.

[0] https://docs.kernel.org/bpf/s390.html

Cc Alan Maguire <[email protected]>
Cc Daniel Xu <[email protected]>
Cc Kumar Kartikeya Dwivedi <[email protected]>
Cc Vadim Fedorenko <[email protected]>
Fixes 72e88f29c6f7 ("pahole: Inject kfunc decl tags into BTF")
---
 btf_encoder.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 lib/bpf       |  2 +-
 2 files changed, 43 insertions(+), 1 deletion(-)
btf_encoder.c +42 -0
index e1adddf..3bdb73b 100644
@@ -33,6 +33,7 @@
 #include <stdint.h>
 #include <search.h> /* for tsearch(), tfind() and tdestroy() */
 #include <pthread.h>
+#include <byteswap.h>
 
 #define BTF_IDS_SECTION		".BTF_ids"
 #define BTF_ID_FUNC_PFX		"__BTF_ID__func__"
@@ -1847,11 +1848,47 @@ static int btf_encoder__tag_kfunc(struct btf_encoder *encoder, struct gobuffer *
 	return 0;
 }
 
+/* If byte order of 'elf' differs from current byte order, convert the data->d_buf.
+ * ELF file is opened in a readonly mode, so data->d_buf cannot be modified in place.
+ * Instead, allocate a new buffer if modification is necessary.
+ */
+static int convert_idlist_endianness(Elf *elf, Elf_Data *data, bool *copied)
+{
+	int byteorder, i;
+	char *elf_ident;
+	uint32_t *tmp;
+
+	*copied = false;
+	elf_ident = elf_getident(elf, NULL);
+	if (elf_ident == NULL) {
+		fprintf(stderr, "Cannot get ELF identification from header\n");
+		return -EINVAL;
+	}
+	byteorder = elf_ident[EI_DATA];
+	if ((BYTE_ORDER == LITTLE_ENDIAN && byteorder == ELFDATA2LSB)
+	    || (BYTE_ORDER == BIG_ENDIAN && byteorder == ELFDATA2MSB))
+		return 0;
+	tmp = malloc(data->d_size);
+	if (tmp == NULL) {
+		fprintf(stderr, "Cannot allocate %lu bytes of memory\n", data->d_size);
+		return -ENOMEM;
+	}
+	memcpy(tmp, data->d_buf, data->d_size);
+	data->d_buf = tmp;
+	*copied = true;
+
+	/* .BTF_ids sections consist of u32 objects */
+	for (i = 0; i < data->d_size / sizeof(uint32_t); i++)
+		tmp[i] = bswap_32(tmp[i]);
+	return 0;
+}
+
 static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder)
 {
 	const char *filename = encoder->source_filename;
 	struct gobuffer btf_kfunc_ranges = {};
 	struct gobuffer btf_funcs = {};
+	bool free_idlist = false;
 	Elf_Data *symbols = NULL;
 	Elf_Data *idlist = NULL;
 	Elf_Scn *symscn = NULL;
@@ -1919,6 +1956,9 @@ static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder)
 			idlist_shndx = i;
 			idlist_addr = shdr.sh_addr;
 			idlist = data;
+			err = convert_idlist_endianness(elf, idlist, &free_idlist);
+			if (err < 0)
+				goto out;
 		}
 	}
 
@@ -2031,6 +2071,8 @@ static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder)
 out:
 	__gobuffer__delete(&btf_funcs);
 	__gobuffer__delete(&btf_kfunc_ranges);
+	if (free_idlist)
+		free(idlist->d_buf);
 	if (elf)
 		elf_end(elf);
 	if (fd != -1)
lib/bpf +1 -1
index 09b9e83..caa17bd 160000
@@ -1 +1 @@
-Subproject commit 09b9e83102eb8ab9e540d36b4559c55f3bcdb95d
+Subproject commit caa17bdcbfc58e68eaf4d017c058e6577606bf56
Signature · 1 line
2.47.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 32 lines · Eduard Zingerman

btf_encoder__tag_kfuncs() reads .BTF_ids section to identify a set of kfuncs present in the ELF being processed. This section consists of records of the following shape:

   struct btf_id_and_flag {
       uint32_t id;
       uint32_t flags;
   };

When endianness of binary operated by pahole differs from the host endianness these fields require byte swap before using.

At the moment such byte swap does not happen and kfuncs are not marked with decl tags when e.g. s390 kernel is compiled on x86.
To reproduces the bug:
- follow instructions from [0] to build an s390 vmlinux;
- execute:

   pahole --btf_features_strict=decl_tag_kfuncs,decl_tag \
          --btf_encode_detached=test.btf vmlinux

- observe no kfuncs generated:
bpftool btf dump test.btf format c | grep __ksym

This commit fixes the issue by adding an endianness conversion step for .BTF_ids section data before main processing step, modifying the Elf_Data object in-place.
The choice is such in order to:
- minimize changes;
- keep using Elf_Data, as it provides fields {d_size,d_off} used by kfunc processing routines;
- avoid sprinkling bswap_32 at each 'struct btf_id_and_flag' field access in fear of forgetting to add new ones when code is modified.

Quoted 2 lines

[0] https://docs.kernel.org/bpf/s390.html

LGTM, Thanks!

#2 lore raw
In reply to Eduard Zingerman
Quoted 8 lines · Eduard Zingerman

btf_encoder__tag_kfuncs() reads .BTF_ids section to identify a set of kfuncs present in the ELF being processed. This section consists of records of the following shape:

  struct btf_id_and_flag {
      uint32_t id;
      uint32_t flags;
  };

it contains pairs like above and also just id arrays with no flags, but that does not matter for the patch functionality, because you swap by u32 values anyway

Quoted 14 lines

When endianness of binary operated by pahole differs from the host endianness these fields require byte swap before using.

At the moment such byte swap does not happen and kfuncs are not marked with decl tags when e.g. s390 kernel is compiled on x86.
To reproduces the bug:
- follow instructions from [0] to build an s390 vmlinux;
- execute:
pahole --btf_features_strict=decl_tag_kfuncs,decl_tag \
--btf_encode_detached=test.btf vmlinux
- observe no kfuncs generated:
bpftool btf dump test.btf format c | grep __ksym

Quoted 9 lines

This commit fixes the issue by adding an endianness conversion step for .BTF_ids section data before main processing step, modifying the Elf_Data object in-place.
The choice is such in order to:
- minimize changes;
- keep using Elf_Data, as it provides fields {d_size,d_off} used by kfunc processing routines;
- avoid sprinkling bswap_32 at each 'struct btf_id_and_flag' field access in fear of forgetting to add new ones when code is modified.

lgtm, some questions below

Quoted 47 lines

[0] https://docs.kernel.org/bpf/s390.html

Cc Alan Maguire <[email protected]>
Cc Daniel Xu <[email protected]>
Cc Kumar Kartikeya Dwivedi <[email protected]>
Cc Vadim Fedorenko <[email protected]>
Fixes 72e88f29c6f7 ("pahole: Inject kfunc decl tags into BTF")
---
 btf_encoder.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 lib/bpf       |  2 +-
 2 files changed, 43 insertions(+), 1 deletion(-)
btf_encoder.c +18 -0
index e1adddf..3bdb73b 100644
@@ -33,6 +33,7 @@
 #include <stdint.h>
 #include <search.h> /* for tsearch(), tfind() and tdestroy() */
 #include <pthread.h>
+#include <byteswap.h>
 
 #define BTF_IDS_SECTION		".BTF_ids"
 #define BTF_ID_FUNC_PFX		"__BTF_ID__func__"
@@ -1847,11 +1848,47 @@ static int btf_encoder__tag_kfunc(struct btf_encoder *encoder, struct gobuffer *
 	return 0;
 }
 
+/* If byte order of 'elf' differs from current byte order, convert the data->d_buf.
+ * ELF file is opened in a readonly mode, so data->d_buf cannot be modified in place.
+ * Instead, allocate a new buffer if modification is necessary.
+ */
+static int convert_idlist_endianness(Elf *elf, Elf_Data *data, bool *copied)
+{
+	int byteorder, i;
+	char *elf_ident;
+	uint32_t *tmp;
+
+	*copied = false;
+	elf_ident = elf_getident(elf, NULL);
+	if (elf_ident == NULL) {
+		fprintf(stderr, "Cannot get ELF identification from header\n");
+		return -EINVAL;
+	}
+	byteorder = elf_ident[EI_DATA];

btf_encoder.c static int btf_encoder__tag_kfunc(struct btf_encoder *encoder, struct gobuffer * · line 1877

Quoted 10 lines
btf_encoder.c +10 -0
+	if ((BYTE_ORDER == LITTLE_ENDIAN && byteorder == ELFDATA2LSB)
+	    || (BYTE_ORDER == BIG_ENDIAN && byteorder == ELFDATA2MSB))
+		return 0;
+	tmp = malloc(data->d_size);
+	if (tmp == NULL) {
+		fprintf(stderr, "Cannot allocate %lu bytes of memory\n", data->d_size);
+		return -ENOMEM;
+	}
+	memcpy(tmp, data->d_buf, data->d_size);
+	data->d_buf = tmp;

will the original data->d_buf be leaked? are we allowed to assign d_buf like that? ;-)

Quoted 28 lines
btf_encoder.c +12 -0
+	*copied = true;
+
+	/* .BTF_ids sections consist of u32 objects */
+	for (i = 0; i < data->d_size / sizeof(uint32_t); i++)
+		tmp[i] = bswap_32(tmp[i]);
+	return 0;
+}
+
 static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder)
 {
 	const char *filename = encoder->source_filename;
 	struct gobuffer btf_kfunc_ranges = {};
 	struct gobuffer btf_funcs = {};
+	bool free_idlist = false;
 	Elf_Data *symbols = NULL;
 	Elf_Data *idlist = NULL;
 	Elf_Scn *symscn = NULL;
@@ -1919,6 +1956,9 @@ static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder)
 			idlist_shndx = i;
 			idlist_addr = shdr.sh_addr;
 			idlist = data;
+			err = convert_idlist_endianness(elf, idlist, &free_idlist);
+			if (err < 0)
+				goto out;
 		}
 	}
 

btf_encoder.c static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder) · line 2077

Quoted 7 lines
btf_encoder.c +2 -0
@@ -2031,6 +2071,8 @@ static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder)
 out:
 	__gobuffer__delete(&btf_funcs);
 	__gobuffer__delete(&btf_kfunc_ranges);
+	if (free_idlist)
+		free(idlist->d_buf);
 	if (elf)
 		elf_end(elf);

curious, would elf_end try to free the d_buf at some point?

lib/bpf line 1

Quoted 8 lines
btf_encoder.c +0 -0
 	if (fd != -1)
lib/bpf +1 -1
index 09b9e83..caa17bd 160000
@@ -1 +1 @@
-Subproject commit 09b9e83102eb8ab9e540d36b4559c55f3bcdb95d
+Subproject commit caa17bdcbfc58e68eaf4d017c058e6577606bf56

I think this should not be part of the patch

thanks,
jirka

author maintainer
#3 lore raw
Quoted 5 lines · Jiri Olsa
Quoted 4 lines · Eduard Zingerman

btf_encoder__tag_kfuncs() reads .BTF_ids section to identify a set of kfuncs present in the ELF being processed. This section consists of records of the following shape:

Quoted 8 lines
Quoted 4 lines
  struct btf_id_and_flag {
      uint32_t id;
      uint32_t flags;
  };

it contains pairs like above and also just id arrays with no flags, but that does not matter for the patch functionality, because you swap by u32 values anyway

Right, I'll update the description, thank you.

[...]

Quoted 25 lines
Quoted 25 lines
btf_encoder.c +21 -0
@@ -1847,11 +1848,47 @@ static int btf_encoder__tag_kfunc(struct btf_encoder *encoder, struct gobuffer *
 	return 0;
 }
 
+/* If byte order of 'elf' differs from current byte order, convert the data->d_buf.
+ * ELF file is opened in a readonly mode, so data->d_buf cannot be modified in place.
+ * Instead, allocate a new buffer if modification is necessary.
+ */
+static int convert_idlist_endianness(Elf *elf, Elf_Data *data, bool *copied)
+{
+	int byteorder, i;
+	char *elf_ident;
+	uint32_t *tmp;
+
+	*copied = false;
+	elf_ident = elf_getident(elf, NULL);
+	if (elf_ident == NULL) {
+		fprintf(stderr, "Cannot get ELF identification from header\n");
+		return -EINVAL;
+	}
+	byteorder = elf_ident[EI_DATA];
+	if ((BYTE_ORDER == LITTLE_ENDIAN && byteorder == ELFDATA2LSB)
+	    || (BYTE_ORDER == BIG_ENDIAN && byteorder == ELFDATA2MSB))
+		return 0;
+	tmp = malloc(data->d_size);
Quoted 8 lines

btf_encoder.c static int btf_encoder__tag_kfunc(struct btf_encoder *encoder, struct gobuffer * · line 1877

Quoted 6 lines
btf_encoder.c +6 -0
+	if (tmp == NULL) {
+		fprintf(stderr, "Cannot allocate %lu bytes of memory\n", data->d_size);
+		return -ENOMEM;
+	}
+	memcpy(tmp, data->d_buf, data->d_size);
+	data->d_buf = tmp;

will the original data->d_buf be leaked? are we allowed to assign d_buf like that? ;-)

Well, before sending I checked using address sanitizer, and it did not complain. As far as I understand elfutils elf_getdata.c / elf_end.c [0]:
- elf_getdata() allocates memory for full section (elf_getdata.c:333), before setting d_buf field of Elf_Data;
- elf_end() frees memory for full section (elf_end.c:174).

So I assumed that this is hacky but not that bad.
Given that current patch depends on implementation details it is probably better to switch to one of the alternatives:
a. allocate new Elf_Data object using elf_newdata() API;
b. just allocate a fake instance of Elf_Data on stack in btf_encoder__tag_kfuncs().

(a) seems to be an Ok option, wdyt?

[0] b2f225d6bff8 ("Consolidate and add files to clean target variables")
git://sourceware.org/git/elfutils.git

[...]

Quoted 10 lines

lib/bpf line 1

Quoted 8 lines
btf_encoder.c +0 -0
 	if (fd != -1)
lib/bpf +1 -1
index 09b9e83..caa17bd 160000
@@ -1 +1 @@
-Subproject commit 09b9e83102eb8ab9e540d36b4559c55f3bcdb95d
+Subproject commit caa17bdcbfc58e68eaf4d017c058e6577606bf56

I think this should not be part of the patch

Sorry, didn't notice this thing.

author maintainer
#4 lore raw

On Fri, 2024-11-22 at 10:08 -0800, Eduard Zingerman wrote:

[...]

Quoted 7 lines

So I assumed that this is hacky but not that bad.
Given that current patch depends on implementation details it is probably better to switch to one of the alternatives:
a. allocate new Elf_Data object using elf_newdata() API;
b. just allocate a fake instance of Elf_Data on stack in btf_encoder__tag_kfuncs().

(a) seems to be an Ok option, wdyt?

Meh, any of these raise some questions about validity of API usage. I'll just add a custom wrapper:

  struct converted_elf_data {
	void *d_buf;
        size_t d_size;
        size_t d_off;
        bool owns_buf;

};

[...]

#5 lore raw
In reply to Eduard Zingerman
Quoted 10 lines · Eduard Zingerman

btf_encoder__tag_kfuncs() reads .BTF_ids section to identify a set of kfuncs present in the ELF being processed. This section consists of records of the following shape:

  struct btf_id_and_flag {
      uint32_t id;
      uint32_t flags;
  };

Can we just set data->d_type to ELF_T_WORD and let libelf handle the byte swap?

Quoted 36 lines

When endianness of binary operated by pahole differs from the host endianness these fields require byte swap before using.

At the moment such byte swap does not happen and kfuncs are not marked with decl tags when e.g. s390 kernel is compiled on x86.
To reproduces the bug:
- follow instructions from [0] to build an s390 vmlinux;
- execute:
pahole --btf_features_strict=decl_tag_kfuncs,decl_tag \
--btf_encode_detached=test.btf vmlinux
- observe no kfuncs generated:
bpftool btf dump test.btf format c | grep __ksym

This commit fixes the issue by adding an endianness conversion step for .BTF_ids section data before main processing step, modifying the Elf_Data object in-place.
The choice is such in order to:
- minimize changes;
- keep using Elf_Data, as it provides fields {d_size,d_off} used by kfunc processing routines;
- avoid sprinkling bswap_32 at each 'struct btf_id_and_flag' field access in fear of forgetting to add new ones when code is modified.

[0] https://docs.kernel.org/bpf/s390.html

Cc Alan Maguire <[email protected]>
Cc Daniel Xu <[email protected]>
Cc Kumar Kartikeya Dwivedi <[email protected]>
Cc Vadim Fedorenko <[email protected]>
Fixes 72e88f29c6f7 ("pahole: Inject kfunc decl tags into BTF")
---
 btf_encoder.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 lib/bpf       |  2 +-
 2 files changed, 43 insertions(+), 1 deletion(-)

[...]

author maintainer
#6 lore raw
Quoted 6 lines · Andrii Nakryiko
Quoted 5 lines · Eduard Zingerman

btf_encoder__tag_kfuncs() reads .BTF_ids section to identify a set of kfuncs present in the ELF being processed. This section consists of records of the following shape:

Quoted 7 lines
Quoted 5 lines
  struct btf_id_and_flag {
      uint32_t id;
      uint32_t flags;
  };

Can we just set data->d_type to ELF_T_WORD and let libelf handle the byte swap?

When I tried 'data->d_type = ELF_T_WORD' + gelf_xlatetom() snippet suggested by Tony Ambardar some time ago, I got a write protection error. Concluded that this is so, because file is opened in O_RDONLY mode.

(Also please note v2 of this patch).

[...]

#7 lore raw
Quoted 14 lines · Eduard Zingerman
Quoted 12 lines · Andrii Nakryiko
Quoted 10 lines · Eduard Zingerman

btf_encoder__tag_kfuncs() reads .BTF_ids section to identify a set of kfuncs present in the ELF being processed. This section consists of records of the following shape:

  struct btf_id_and_flag {
      uint32_t id;
      uint32_t flags;
  };
Quoted 5 lines
Quoted 1 line

Can we just set data->d_type to ELF_T_WORD and let libelf handle the byte swap?

When I tried 'data->d_type = ELF_T_WORD' + gelf_xlatetom() snippet suggested by Tony Ambardar some time ago, I got a write protection error. Concluded that this is so, because file is opened in O_RDONLY mode.

Ok, maybe don't follow my words *exactly*, just in spirit ;) I see there is elf_getdata_rawchunk() API in libelf, which seems useful:

/* Get data translated from a chunk of the file contents as section data
would be for TYPE. The resulting Elf_Data pointer is valid until elf_end (ELF) is called. */
extern Elf_Data *elf_getdata_rawchunk (Elf *__elf,

                                       int64_t __offset, size_t __size,
                                       Elf_Type __type);

I don't know the surrounding code, I was just hoping to leverage libelf's byte swapping support (which I think I learned from Tony as well). But if it's too inconvenient, so be it.

Quoted 5 lines

(Also please note v2 of this patch).

[...]

author maintainer
#8 lore raw

On Tue, 2024-11-26 at 13:51 -0800, Andrii Nakryiko wrote:

[...]

Quoted 11 lines
Quoted 3 lines

When I tried 'data->d_type = ELF_T_WORD' + gelf_xlatetom() snippet suggested by Tony Ambardar some time ago, I got a write protection error. Concluded that this is so, because file is opened in O_RDONLY mode.

Ok, maybe don't follow my words *exactly*, just in spirit ;) I see there is elf_getdata_rawchunk() API in libelf, which seems useful:

/* Get data translated from a chunk of the file contents as section data
would be for TYPE. The resulting Elf_Data pointer is valid until elf_end (ELF) is called. */
extern Elf_Data *elf_getdata_rawchunk (Elf *__elf,

Quoted 6 lines
                                       int64_t __offset, size_t __size,
                                       Elf_Type __type);

I don't know the surrounding code, I was just hoping to leverage libelf's byte swapping support (which I think I learned from Tony as well). But if it's too inconvenient, so be it.

I missed this API, it works and makes the patch much smaller, thank you for finding it.