Skip to contentSkip to conversation

Add split BTF support to pahole

PATCH In review Andrii Nakryiko 4 participants 17 messages Nov 6, 2020 – Nov 13, 2020

Subsystems (2)
lore
Share & tools
lore

Andrii Nakryiko

author maintainer
#0 lore raw

cover Add split BTF support to pahole

Add ability to generate split BTF (for kernel modules), as well as load split BTF. --btf_base argument is added to specify base BTF for split BTF. This works for both btf_loader and btf_encoder.

Andrii Nakryiko (4):
btf_encoder: fix array index type numbering
libbtf: improve variable naming and error reporting when writing out
BTF
libbpf: update libbpf submodlue reference to latest master
btf: add support for split BTF loading and encoding

 btf_encoder.c | 15 ++++++++-------
 btf_loader.c  |  2 +-
 lib/bpf       |  2 +-
 libbtf.c      | 43 +++++++++++++++++++++++++++----------------
 libbtf.h      |  4 +++-
 pahole.c      | 23 +++++++++++++++++++++++
 6 files changed, 63 insertions(+), 26 deletions(-)
Signature · 1 line
2.24.1
author maintainer
#1 lore raw

PATCH 1/4 btf_encoder: fix array index type numbering

Take into account type ID offset, accumulated from previous CUs, when calculating a new type ID for the generated array index type.

---
 btf_encoder.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
btf_encoder.c +6 -6
index 4c92908beab2..b3e47f172bb3 100644
@@ -358,22 +358,22 @@ int cu__encode_btf(struct cu *cu, int verbose, bool force,
 			printf("File %s:\n", btfe->filename);
 	}
 
+	btf_elf__verbose = verbose;
+	btf_elf__force = force;
+	type_id_off = btf__get_nr_types(btfe->btf);
+
 	if (!has_index_type) {
 		/* cu__find_base_type_by_name() takes "type_id_t *id" */
 		type_id_t id;
 		if (cu__find_base_type_by_name(cu, "int", &id)) {
 			has_index_type = true;
-			array_index_id = id;
+			array_index_id = type_id_off + id;
 		} else {
 			has_index_type = false;
-			array_index_id = cu->types_table.nr_entries;
+			array_index_id = type_id_off + cu->types_table.nr_entries;
 		}
 	}
 
-	btf_elf__verbose = verbose;
-	btf_elf__force = force;
-	type_id_off = btf__get_nr_types(btfe->btf);
-
 	cu__for_each_type(cu, core_id, pos) {
 		int32_t btf_type_id = tag__encode_btf(cu, pos, core_id, btfe, array_index_id, type_id_off);
 
Signature · 1 line
2.24.1
author maintainer
#2 lore raw

PATCH 2/4 libbtf: improve variable naming and error reporting when writing out BTF

In reply to Andrii Nakryiko

Rename few local variables to reflects the purpose a bit better. Also separate writing out BTF raw data and objcopy invocation into two separate steps and improve error reporting for each.

---
 libbtf.c | 33 +++++++++++++++++++++------------
 1 file changed, 21 insertions(+), 12 deletions(-)
libbtf.c +21 -12
index babf4fe8cd9e..b6ddd7599395 100644
@@ -679,11 +679,11 @@ static int btf_elf__write(const char *filename, struct btf *btf)
 {
 	GElf_Shdr shdr_mem, *shdr;
 	GElf_Ehdr ehdr_mem, *ehdr;
-	Elf_Data *btf_elf = NULL;
+	Elf_Data *btf_data = NULL;
 	Elf_Scn *scn = NULL;
 	Elf *elf = NULL;
-	const void *btf_data;
-	uint32_t btf_size;
+	const void *raw_btf_data;
+	uint32_t raw_btf_size;
 	int fd, err = -1;
 	size_t strndx;
 
@@ -735,18 +735,18 @@ static int btf_elf__write(const char *filename, struct btf *btf)
 			continue;
 		char *secname = elf_strptr(elf, strndx, shdr->sh_name);
 		if (strcmp(secname, ".BTF") == 0) {
-			btf_elf = elf_getdata(scn, btf_elf);
+			btf_data = elf_getdata(scn, btf_data);
 			break;
 		}
 	}
 
-	btf_data = btf__get_raw_data(btf, &btf_size);
+	raw_btf_data = btf__get_raw_data(btf, &raw_btf_size);
 
-	if (btf_elf) {
+	if (btf_data) {
 		/* Exisiting .BTF section found */
-		btf_elf->d_buf = (void *)btf_data;
-		btf_elf->d_size = btf_size;
-		elf_flagdata(btf_elf, ELF_C_SET, ELF_F_DIRTY);
+		btf_data->d_buf = (void *)raw_btf_data;
+		btf_data->d_size = raw_btf_size;
+		elf_flagdata(btf_data, ELF_C_SET, ELF_F_DIRTY);
 
 		if (elf_update(elf, ELF_C_NULL) >= 0 &&
 		    elf_update(elf, ELF_C_WRITE) >= 0)
@@ -770,12 +770,21 @@ static int btf_elf__write(const char *filename, struct btf *btf)
 			goto out;
 		}
 
+		if (write(fd, raw_btf_data, raw_btf_size) != raw_btf_size) {
+			fprintf(stderr, "%s: write of %d bytes to '%s' failed: %d!\n",
+				__func__, raw_btf_size, tmp_fn, errno);
+			goto out;
+		}
+
 		snprintf(cmd, sizeof(cmd), "%s --add-section .BTF=%s %s",
 			 llvm_objcopy, tmp_fn, filename);
+		if (system(cmd)) {
+			fprintf(stderr, "%s: failed to add .BTF section to '%s': %d!\n",
+				__func__, tmp_fn, errno);
+			goto out;
+		}
 
-		if (write(fd, btf_data, btf_size) == btf_size && !system(cmd))
-			err = 0;
-
+		err = 0;
 		unlink(tmp_fn);
 	}
 
Signature · 1 line
2.24.1
author maintainer
#3 lore raw

PATCH 3/4 libbpf: update libbpf submodlue reference to latest master

In reply to Andrii Nakryiko

Pull in latest libbpf changes, containing split BTF APIs.

---
 lib/bpf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bpf +1 -1
index ff797cc905d9..5af3d86b5a2c 160000
@@ -1 +1 @@
-Subproject commit ff797cc905d9c5fe9acab92d2da127342b20f80f
+Subproject commit 5af3d86b5a2c5fecdc3ab83822d083edd32b4396
Signature · 1 line
2.24.1
author maintainer
#4 lore raw

PATCH 4/4 btf: add support for split BTF loading and encoding

In reply to Andrii Nakryiko

Add support for generating split BTF, in which there is a designated base BTF, containing a base set of types, and a split BTF, which extends main BTF with extra types, that can reference types and strings from the main BTF.

This is going to be used to generate compact BTFs for kernel modules, with vmlinux BTF being a main BTF, which all kernel modules are based off of.

These changes rely on patch set [0] to be present in libbpf submodule.

[0] https://patchwork.kernel.org/project/netdevbpf/list/?series=377859&state=*

---
 btf_encoder.c |  3 ++-
 btf_loader.c  |  2 +-
 libbtf.c      | 10 ++++++----
 libbtf.h      |  4 +++-
 pahole.c      | 23 +++++++++++++++++++++++
 5 files changed, 35 insertions(+), 7 deletions(-)
btf_encoder.c +2 -1
index b3e47f172bb3..d67e29b9cbee 100644
@@ -12,6 +12,7 @@
 #include "dwarves.h"
 #include "libbtf.h"
 #include "lib/bpf/include/uapi/linux/btf.h"
+#include "lib/bpf/src/libbpf.h"
 #include "hash.h"
 #include "elf_symtab.h"
 #include "btf_encoder.h"
@@ -343,7 +344,7 @@ int cu__encode_btf(struct cu *cu, int verbose, bool force,
 	}
 
 	if (!btfe) {
-		btfe = btf_elf__new(cu->filename, cu->elf);
+		btfe = btf_elf__new(cu->filename, cu->elf, base_btf);
 		if (!btfe)
 			return -1;
 
btf_loader.c +1 -1
index 6ea207ea65b4..ec286f413f36 100644
@@ -534,7 +534,7 @@ struct debug_fmt_ops btf_elf__ops;
 int btf_elf__load_file(struct cus *cus, struct conf_load *conf, const char *filename)
 {
 	int err;
-	struct btf_elf *btfe = btf_elf__new(filename, NULL);
+	struct btf_elf *btfe = btf_elf__new(filename, NULL, base_btf);
 
 	if (btfe == NULL)
 		return -1;
libbtf.c +6 -4
index b6ddd7599395..3c52aa0d482b 100644
@@ -27,6 +27,7 @@
 #include "dwarves.h"
 #include "elf_symtab.h"
 
+struct btf *base_btf;
 uint8_t btf_elf__verbose;
 uint8_t btf_elf__force;
 
@@ -52,9 +53,9 @@ int btf_elf__load(struct btf_elf *btfe)
 	/* free initial empty BTF */
 	btf__free(btfe->btf);
 	if (btfe->raw_btf)
-		btfe->btf = btf__parse_raw(btfe->filename);
+		btfe->btf = btf__parse_raw_split(btfe->filename, btfe->base_btf);
 	else
-		btfe->btf = btf__parse_elf(btfe->filename, NULL);
+		btfe->btf = btf__parse_elf_split(btfe->filename, btfe->base_btf);
 
 	err = libbpf_get_error(btfe->btf);
 	if (err)
@@ -63,7 +64,7 @@ int btf_elf__load(struct btf_elf *btfe)
 	return 0;
 }
 
-struct btf_elf *btf_elf__new(const char *filename, Elf *elf)
+struct btf_elf *btf_elf__new(const char *filename, Elf *elf, struct btf *base_btf)
 {
 	struct btf_elf *btfe = zalloc(sizeof(*btfe));
 	GElf_Shdr shdr;
@@ -77,7 +78,8 @@ struct btf_elf *btf_elf__new(const char *filename, Elf *elf)
 	if (btfe->filename == NULL)
 		goto errout;
 
-	btfe->btf = btf__new_empty();
+	btfe->base_btf = base_btf;
+	btfe->btf = btf__new_empty_split(base_btf);
 	if (libbpf_get_error(btfe->btf)) {
 		fprintf(stderr, "%s: failed to create empty BTF.\n", __func__);
 		goto errout;
libbtf.h +3 -1
index 887b5bc55c8e..71f6cecbea93 100644
@@ -27,8 +27,10 @@ struct btf_elf {
 	uint32_t	  percpu_shndx;
 	uint64_t	  percpu_base_addr;
 	struct btf	  *btf;
+	struct btf	  *base_btf;
 };
 
+extern struct btf *base_btf;
 extern uint8_t btf_elf__verbose;
 extern uint8_t btf_elf__force;
 #define btf_elf__verbose_log(fmt, ...) { if (btf_elf__verbose) printf(fmt, __VA_ARGS__); }
@@ -39,7 +41,7 @@ struct cu;
 struct base_type;
 struct ftype;
 
-struct btf_elf *btf_elf__new(const char *filename, Elf *elf);
+struct btf_elf *btf_elf__new(const char *filename, Elf *elf, struct btf *base_btf);
 void btf_elf__delete(struct btf_elf *btf);
 
 int32_t btf_elf__add_base_type(struct btf_elf *btf, const struct base_type *bt,
pahole.c +23 -0
index bd9b993777ee..d18092c1212c 100644
@@ -22,12 +22,15 @@
 #include "dutil.h"
 #include "ctf_encoder.h"
 #include "btf_encoder.h"
+#include "libbtf.h"
+#include "lib/bpf/src/libbpf.h"
 
 static bool btf_encode;
 static bool ctf_encode;
 static bool first_obj_only;
 static bool skip_encoding_btf_vars;
 static bool btf_encode_force;
+static const char *base_btf_file;
 
 static uint8_t class__include_anonymous;
 static uint8_t class__include_nested_anonymous;
@@ -820,6 +823,7 @@ ARGP_PROGRAM_VERSION_HOOK_DEF = dwarves_print_version;
 #define ARGP_skip_encoding_btf_vars 317
 #define ARGP_btf_encode_force	   318
 #define ARGP_just_packed_structs   319
+#define ARGP_btf_base		   320
 
 static const struct argp_option pahole__options[] = {
 	{
@@ -1093,6 +1097,12 @@ static const struct argp_option pahole__options[] = {
 		.key  = ARGP_hex_fmt,
 		.doc  = "Print offsets and sizes in hexadecimal",
 	},
+	{
+		.name = "btf_base",
+		.key  = ARGP_btf_base,
+		.arg  = "SIZE",
+		.doc  = "Path to the base BTF file",
+	},
 	{
 		.name = "btf_encode",
 		.key  = 'J',
@@ -1234,6 +1244,9 @@ static error_t pahole__options_parser(int key, char *arg,
 		skip_encoding_btf_vars = true;		break;
 	case ARGP_btf_encode_force:
 		btf_encode_force = true;		break;
+	case ARGP_btf_base:
+		base_btf_file = arg;
+		break;
 	default:
 		return ARGP_ERR_UNKNOWN;
 	}
@@ -2682,6 +2695,15 @@ int main(int argc, char *argv[])
 		goto out;
 	}
 
+	if (base_btf_file) {
+		base_btf = btf__parse(base_btf_file, NULL);
+		if (libbpf_get_error(base_btf)) {
+			fprintf(stderr, "Failed to parse base BTF '%s': %ld\n",
+				base_btf_file, libbpf_get_error(base_btf));
+			goto out;
+		}
+	}
+
 	struct cus *cus = cus__new();
 	if (cus == NULL) {
 		fputs("pahole: insufficient memory\n", stderr);
@@ -2766,6 +2788,7 @@ out_cus_delete:
 #ifdef DEBUG_CHECK_LEAKS
 	cus__delete(cus);
 	structures__delete();
+	btf__free(base_btf);
 #endif
 out_dwarves_exit:
 #ifdef DEBUG_CHECK_LEAKS
Signature · 1 line
2.24.1
#5 lore raw

Re: [PATCH dwarves 4/4] btf: add support for split BTF loading and encoding

Em Thu, Nov 05, 2020 at 09:25:49PM -0800, Andrii Nakryiko escreveu:

Quoted 10 lines

Add support for generating split BTF, in which there is a designated base BTF, containing a base set of types, and a split BTF, which extends main BTF with extra types, that can reference types and strings from the main BTF.

This is going to be used to generate compact BTFs for kernel modules, with vmlinux BTF being a main BTF, which all kernel modules are based off of.

These changes rely on patch set [0] to be present in libbpf submodule.

[0] https://patchwork.kernel.org/project/netdevbpf/list/?series=377859&state=*

So, applied and added this:

man-pages/pahole.1 +4 -0
index 4b5e0a1bf5462b28..20ee91fc911d4b39 100644
@@ -185,6 +185,10 @@ Do not encode VARs in BTF.
 .B \-\-btf_encode_force
 Ignore those symbols found invalid when encoding BTF.
 
+.TP
+.B \-\-btf_base
+Path to the base BTF file, for instance: vmlinux when encoding kernel module BTF information.
+
 .TP
 .B \-l, \-\-show_first_biggest_size_base_type_member
 Show first biggest size base_type member.

---------------

The entry for btf_encode/-J is missing, I'll add in a followup patch.

Also I had to fixup ARGP_btf_base to 321 as I added this, to simplify the kernel scripts and Makefiles:

  $ pahole --numeric_version
  118
  $

Now to test this all by applying the kernel patches and the encoding module BTF, looking at it, etc.

- Arnaldo

Quoted 188 lines
---
 btf_encoder.c |  3 ++-
 btf_loader.c  |  2 +-
 libbtf.c      | 10 ++++++----
 libbtf.h      |  4 +++-
 pahole.c      | 23 +++++++++++++++++++++++
 5 files changed, 35 insertions(+), 7 deletions(-)
btf_encoder.c +2 -1
index b3e47f172bb3..d67e29b9cbee 100644
@@ -12,6 +12,7 @@
 #include "dwarves.h"
 #include "libbtf.h"
 #include "lib/bpf/include/uapi/linux/btf.h"
+#include "lib/bpf/src/libbpf.h"
 #include "hash.h"
 #include "elf_symtab.h"
 #include "btf_encoder.h"
@@ -343,7 +344,7 @@ int cu__encode_btf(struct cu *cu, int verbose, bool force,
 	}
 
 	if (!btfe) {
-		btfe = btf_elf__new(cu->filename, cu->elf);
+		btfe = btf_elf__new(cu->filename, cu->elf, base_btf);
 		if (!btfe)
 			return -1;
 
btf_loader.c +1 -1
index 6ea207ea65b4..ec286f413f36 100644
@@ -534,7 +534,7 @@ struct debug_fmt_ops btf_elf__ops;
 int btf_elf__load_file(struct cus *cus, struct conf_load *conf, const char *filename)
 {
 	int err;
-	struct btf_elf *btfe = btf_elf__new(filename, NULL);
+	struct btf_elf *btfe = btf_elf__new(filename, NULL, base_btf);
 
 	if (btfe == NULL)
 		return -1;
libbtf.c +6 -4
index b6ddd7599395..3c52aa0d482b 100644
@@ -27,6 +27,7 @@
 #include "dwarves.h"
 #include "elf_symtab.h"
 
+struct btf *base_btf;
 uint8_t btf_elf__verbose;
 uint8_t btf_elf__force;
 
@@ -52,9 +53,9 @@ int btf_elf__load(struct btf_elf *btfe)
 	/* free initial empty BTF */
 	btf__free(btfe->btf);
 	if (btfe->raw_btf)
-		btfe->btf = btf__parse_raw(btfe->filename);
+		btfe->btf = btf__parse_raw_split(btfe->filename, btfe->base_btf);
 	else
-		btfe->btf = btf__parse_elf(btfe->filename, NULL);
+		btfe->btf = btf__parse_elf_split(btfe->filename, btfe->base_btf);
 
 	err = libbpf_get_error(btfe->btf);
 	if (err)
@@ -63,7 +64,7 @@ int btf_elf__load(struct btf_elf *btfe)
 	return 0;
 }
 
-struct btf_elf *btf_elf__new(const char *filename, Elf *elf)
+struct btf_elf *btf_elf__new(const char *filename, Elf *elf, struct btf *base_btf)
 {
 	struct btf_elf *btfe = zalloc(sizeof(*btfe));
 	GElf_Shdr shdr;
@@ -77,7 +78,8 @@ struct btf_elf *btf_elf__new(const char *filename, Elf *elf)
 	if (btfe->filename == NULL)
 		goto errout;
 
-	btfe->btf = btf__new_empty();
+	btfe->base_btf = base_btf;
+	btfe->btf = btf__new_empty_split(base_btf);
 	if (libbpf_get_error(btfe->btf)) {
 		fprintf(stderr, "%s: failed to create empty BTF.\n", __func__);
 		goto errout;
libbtf.h +3 -1
index 887b5bc55c8e..71f6cecbea93 100644
@@ -27,8 +27,10 @@ struct btf_elf {
 	uint32_t	  percpu_shndx;
 	uint64_t	  percpu_base_addr;
 	struct btf	  *btf;
+	struct btf	  *base_btf;
 };
 
+extern struct btf *base_btf;
 extern uint8_t btf_elf__verbose;
 extern uint8_t btf_elf__force;
 #define btf_elf__verbose_log(fmt, ...) { if (btf_elf__verbose) printf(fmt, __VA_ARGS__); }
@@ -39,7 +41,7 @@ struct cu;
 struct base_type;
 struct ftype;
 
-struct btf_elf *btf_elf__new(const char *filename, Elf *elf);
+struct btf_elf *btf_elf__new(const char *filename, Elf *elf, struct btf *base_btf);
 void btf_elf__delete(struct btf_elf *btf);
 
 int32_t btf_elf__add_base_type(struct btf_elf *btf, const struct base_type *bt,
pahole.c +23 -0
index bd9b993777ee..d18092c1212c 100644
@@ -22,12 +22,15 @@
 #include "dutil.h"
 #include "ctf_encoder.h"
 #include "btf_encoder.h"
+#include "libbtf.h"
+#include "lib/bpf/src/libbpf.h"
 
 static bool btf_encode;
 static bool ctf_encode;
 static bool first_obj_only;
 static bool skip_encoding_btf_vars;
 static bool btf_encode_force;
+static const char *base_btf_file;
 
 static uint8_t class__include_anonymous;
 static uint8_t class__include_nested_anonymous;
@@ -820,6 +823,7 @@ ARGP_PROGRAM_VERSION_HOOK_DEF = dwarves_print_version;
 #define ARGP_skip_encoding_btf_vars 317
 #define ARGP_btf_encode_force	   318
 #define ARGP_just_packed_structs   319
+#define ARGP_btf_base		   320
 
 static const struct argp_option pahole__options[] = {
 	{
@@ -1093,6 +1097,12 @@ static const struct argp_option pahole__options[] = {
 		.key  = ARGP_hex_fmt,
 		.doc  = "Print offsets and sizes in hexadecimal",
 	},
+	{
+		.name = "btf_base",
+		.key  = ARGP_btf_base,
+		.arg  = "SIZE",
+		.doc  = "Path to the base BTF file",
+	},
 	{
 		.name = "btf_encode",
 		.key  = 'J',
@@ -1234,6 +1244,9 @@ static error_t pahole__options_parser(int key, char *arg,
 		skip_encoding_btf_vars = true;		break;
 	case ARGP_btf_encode_force:
 		btf_encode_force = true;		break;
+	case ARGP_btf_base:
+		base_btf_file = arg;
+		break;
 	default:
 		return ARGP_ERR_UNKNOWN;
 	}
@@ -2682,6 +2695,15 @@ int main(int argc, char *argv[])
 		goto out;
 	}
 
+	if (base_btf_file) {
+		base_btf = btf__parse(base_btf_file, NULL);
+		if (libbpf_get_error(base_btf)) {
+			fprintf(stderr, "Failed to parse base BTF '%s': %ld\n",
+				base_btf_file, libbpf_get_error(base_btf));
+			goto out;
+		}
+	}
+
 	struct cus *cus = cus__new();
 	if (cus == NULL) {
 		fputs("pahole: insufficient memory\n", stderr);
@@ -2766,6 +2788,7 @@ out_cus_delete:
 #ifdef DEBUG_CHECK_LEAKS
 	cus__delete(cus);
 	structures__delete();
+	btf__free(base_btf);
 #endif
 out_dwarves_exit:
 #ifdef DEBUG_CHECK_LEAKS
Signature · 1 line
2.24.1
Signature · 1 line
- Arnaldo
#6 lore raw

Re: [PATCH dwarves 4/4] btf: add support for split BTF loading and encoding

Em Wed, Nov 11, 2020 at 08:56:27AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 9 lines

The entry for btf_encode/-J is missing, I'll add in a followup patch.

Also I had to fixup ARGP_btf_base to 321 as I added this, to simplify the kernel scripts and Makefiles:

  $ pahole --numeric_version
  118
  $

Added this:

[acme@five pahole]$ git diff

man-pages/pahole.1 +8 -0
index 20ee91fc911d4b39..f44c649924383a32 100644
@@ -181,6 +181,14 @@ the debugging information.
 .B \-\-skip_encoding_btf_vars
 Do not encode VARs in BTF.
 
+.TP
+.B \-J, \-\-btf_encode
+Encode BTF information from DWARF, used in the Linux kernel build process when
+CONFIG_DEBUG_INFO_BTF=y is present, introduced in Linux v5.2. Used to implement
+features such as BPF CO-RE (Compile Once - Run Everywhere).
+
+See \fIhttp://vger.kernel.org/bpfconf2019_talks/bpf-core.pdf\fR.
+
 .TP
 .B \-\-btf_encode_force
 Ignore those symbols found invalid when encoding BTF.

[acme@five pahole]$

#7 lore raw

Re: [PATCH dwarves 4/4] btf: add support for split BTF loading and encoding

Quoted 20 lines · Arnaldo Carvalho de Melo

Em Wed, Nov 11, 2020 at 08:56:27AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 9 lines

The entry for btf_encode/-J is missing, I'll add in a followup patch.

Also I had to fixup ARGP_btf_base to 321 as I added this, to simplify the kernel scripts and Makefiles:

  $ pahole --numeric_version
  118
  $

Added this:

[acme@five pahole]$ git diff

man-pages/pahole.1 +0 -0
index 20ee91fc911d4b39..f44c649924383a32 100644

man-pages/pahole.1 the debugging information. · line 190

Quoted 10 lines
man-pages/pahole.1 +7 -0
@@ -181,6 +181,14 @@ the debugging information.
 .B \-\-skip_encoding_btf_vars
 Do not encode VARs in BTF.
 
+.TP
+.B \-J, \-\-btf_encode
+Encode BTF information from DWARF, used in the Linux kernel build process when
+CONFIG_DEBUG_INFO_BTF=y is present, introduced in Linux v5.2. Used to implement
+features such as BPF CO-RE (Compile Once - Run Everywhere).
+
+See \fIhttp://vger.kernel.org/bpfconf2019_talks/bpf-core.pdf\fR.

Can you please point to [0] instead? That linked presentation is already a bit out of date and will bitrot much faster. Blog post has at least a chance at being updated with relevant important stuff. Plus it has links both to the bpfconf2019 presentation, as well as some other resources (including your presentation).

[0] https://nakryiko.com/posts/bpf-portability-and-co-re/

Quoted 5 lines
man-pages/pahole.1 +1 -0
+
 .TP
 .B \-\-btf_encode_force
 Ignore those symbols found invalid when encoding BTF.

[acme@five pahole]$

#8 lore raw

Re: [PATCH dwarves 4/4] btf: add support for split BTF loading and encoding

Quoted 32 lines · Andrii Nakryiko
Quoted 2 lines · Arnaldo Carvalho de Melo

Em Wed, Nov 11, 2020 at 08:56:27AM -0300, Arnaldo Carvalho de Melo

escreveu:

Quoted 2 lines
Quoted 2 lines

The entry for btf_encode/-J is missing, I'll add in a followup

patch.

Quoted 2 lines
Quoted 2 lines

Also I had to fixup ARGP_btf_base to 321 as I added this, to

simplify

Quoted 14 lines
Quoted 5 lines

the kernel scripts and Makefiles:

  $ pahole --numeric_version
  118
  $

Added this:

[acme@five pahole]$ git diff

man-pages/pahole.1 +0 -0
index 20ee91fc911d4b39..f44c649924383a32 100644

man-pages/pahole.1 the debugging information. · line 186

Quoted 6 lines
man-pages/pahole.1 +3 -0
@@ -181,6 +181,14 @@ the debugging information.
 .B \-\-skip_encoding_btf_vars
 Do not encode VARs in BTF.
 
+.TP
+.B \-J, \-\-btf_encode
+Encode BTF information from DWARF, used in the Linux kernel build

process when

Quoted 7 lines

man-pages/pahole.1 the debugging information. · line 187

Quoted 1 line
man-pages/pahole.1 +1 -0
+CONFIG_DEBUG_INFO_BTF=y is present, introduced in Linux v5.2. Used

to implement

man-pages/pahole.1 the debugging information. · line 190

Quoted 3 lines
man-pages/pahole.1 +3 -0
+features such as BPF CO-RE (Compile Once - Run Everywhere).
+
+See \fIhttp://vger.kernel.org/bpfconf2019_talks/bpf-core.pdf\fR.

Can you please point to [0] instead? That linked

Sure.

Quoted 7 lines

presentation is
already a bit out of date and will bitrot much faster. Blog post has at least a chance at being updated with relevant important stuff. Plus it has links both to the bpfconf2019 presentation, as well as some other resources (including your presentation).

[0] https://nakryiko.com/posts/bpf-portability-and-co-re/

Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
#9 lore raw

Re: [PATCH dwarves 4/4] btf: add support for split BTF loading and encoding

In reply to Andrii Nakryiko

Em Wed, Nov 11, 2020 at 10:29:32AM -0800, Andrii Nakryiko escreveu:

Quoted 39 lines
Quoted 20 lines · Arnaldo Carvalho de Melo

Em Wed, Nov 11, 2020 at 08:56:27AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 9 lines

The entry for btf_encode/-J is missing, I'll add in a followup patch.

Also I had to fixup ARGP_btf_base to 321 as I added this, to simplify the kernel scripts and Makefiles:

  $ pahole --numeric_version
  118
  $

Added this:

[acme@five pahole]$ git diff

man-pages/pahole.1 +0 -0
index 20ee91fc911d4b39..f44c649924383a32 100644

man-pages/pahole.1 the debugging information. · line 190

Quoted 10 lines
man-pages/pahole.1 +7 -0
@@ -181,6 +181,14 @@ the debugging information.
 .B \-\-skip_encoding_btf_vars
 Do not encode VARs in BTF.
 
+.TP
+.B \-J, \-\-btf_encode
+Encode BTF information from DWARF, used in the Linux kernel build process when
+CONFIG_DEBUG_INFO_BTF=y is present, introduced in Linux v5.2. Used to implement
+features such as BPF CO-RE (Compile Once - Run Everywhere).
+
+See \fIhttp://vger.kernel.org/bpfconf2019_talks/bpf-core.pdf\fR.

Can you please point to [0] instead? That linked presentation is already a bit out of date and will bitrot much faster. Blog post has at least a chance at being updated with relevant important stuff. Plus it has links both to the bpfconf2019 presentation, as well as some other resources (including your presentation).

Quoted 1 line

[0] https://nakryiko.com/posts/bpf-portability-and-co-re/

Done, thanks for the pinter.

- Arnaldo

#10 lore raw

Re: [PATCH dwarves 4/4] btf: add support for split BTF loading and encoding

In reply to Arnaldo Carvalho de Melo
Quoted 6 lines · Arnaldo Carvalho de Melo

Em Thu, Nov 05, 2020 at 09:25:49PM -0800, Andrii Nakryiko escreveu:

Quoted 3 lines

Add support for generating split BTF, in which there is a designated base BTF, containing a base set of types, and a split BTF, which extends main BTF with extra types, that can reference types and strings from the main BTF.

Quoted 8 lines
Quoted 6 lines

This is going to be used to generate compact BTFs for kernel modules, with vmlinux BTF being a main BTF, which all kernel modules are based off of.

These changes rely on patch set [0] to be present in libbpf submodule.

[0] https://patchwork.kernel.org/project/netdevbpf/list/?series=377859&state=*

So, applied and added this:

Awesome, thanks! Do you plan to release v1.19 soon?

Quoted 19 lines
man-pages/pahole.1 +4 -1
index 4b5e0a1bf5462b28..20ee91fc911d4b39 100644
@@ -185,6 +185,10 @@ Do not encode VARs in BTF.
 .B \-\-btf_encode_force
 Ignore those symbols found invalid when encoding BTF.
 
+.TP
+.B \-\-btf_base
+Path to the base BTF file, for instance: vmlinux when encoding kernel module BTF information.
+
 .TP
 .B \-l, \-\-show_first_biggest_size_base_type_member
 Show first biggest size base_type member.
 
---------------
Quoted 8 lines

The entry for btf_encode/-J is missing, I'll add in a followup patch.

Also I had to fixup ARGP_btf_base to 321 as I added this, to simplify the kernel scripts and Makefiles:

  $ pahole --numeric_version
  118
  $

Oh, this is nice! Can't really use it with Kbuild now due to backwards compatibility, but maybe someday.

Quoted 6 lines

Now to test this all by applying the kernel patches and the encoding module BTF, looking at it, etc.

- Arnaldo

[...]

#11 lore raw

Re: [PATCH dwarves 4/4] btf: add support for split BTF loading and encoding

Quoted 17 lines · Andrii Nakryiko
Quoted 3 lines · Arnaldo Carvalho de Melo

Em Thu, Nov 05, 2020 at 09:25:49PM -0800, Andrii Nakryiko escreveu:

Quoted 1 line

Add support for generating split BTF, in which there is a

designated base

Quoted 1 line
Quoted 1 line

BTF, containing a base set of types, and a split BTF, which extends

main BTF

Quoted 1 line
Quoted 1 line

with extra types, that can reference types and strings from the

main BTF.

Quoted 2 lines
Quoted 2 lines

This is going to be used to generate compact BTFs for kernel

modules, with

Quoted 1 line
Quoted 1 line

vmlinux BTF being a main BTF, which all kernel modules are based

off of.

Quoted 2 lines
Quoted 2 lines

These changes rely on patch set [0] to be present in libbpf

Quoted 8 lines

submodule.

Quoted 2 lines
Quoted 2 lines

[0]

https://patchwork.kernel.org/project/netdevbpf/list/?series=377859&state=*

Quoted 2 lines

So, applied and added this:

Awesome, thanks! Do you plan to release v1.19 soon?

Yes

Quoted 26 lines

man-pages/pahole.1 Do not encode VARs in BTF. · line 190

Quoted 13 lines
man-pages/pahole.1 +3 -0
index 4b5e0a1bf5462b28..20ee91fc911d4b39 100644
@@ -185,6 +185,10 @@ Do not encode VARs in BTF.
 .B \-\-btf_encode_force
 Ignore those symbols found invalid when encoding BTF.
 
+.TP
+.B \-\-btf_base
+Path to the base BTF file, for instance: vmlinux when encoding

kernel module BTF information.

Quoted 12 lines
man-pages/pahole.1 +1 -1
+
 .TP
 .B \-l, \-\-show_first_biggest_size_base_type_member
 Show first biggest size base_type member.
 
---------------

The entry for btf_encode/-J is missing, I'll add in a followup patch.

Also I had to fixup ARGP_btf_base to 321 as I added this, to simplify the kernel scripts and Makefiles:

Quoted 6 lines
Quoted 3 lines
  $ pahole --numeric_version
  118
  $

Oh, this is nice! Can't really use it with Kbuild now due to backwards compatibility, but maybe someday.

Well, if it fails with --numeric_version, then it is old and the warning about the minimal version being v1.19 should be emitted :)

- Arnaldo

Quoted 9 lines
Quoted 7 lines

Now to test this all by applying the kernel patches and the encoding module BTF, looking at it, etc.

- Arnaldo

[...]

Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
#12 lore raw

Re: [PATCH dwarves 4/4] btf: add support for split BTF loading and encoding

Quoted 58 lines · Arnaldo
Quoted 17 lines · Andrii Nakryiko
Quoted 3 lines · Arnaldo Carvalho de Melo

Em Thu, Nov 05, 2020 at 09:25:49PM -0800, Andrii Nakryiko escreveu:

Quoted 1 line

Add support for generating split BTF, in which there is a

designated base

Quoted 1 line
Quoted 1 line

BTF, containing a base set of types, and a split BTF, which extends

main BTF

Quoted 1 line
Quoted 1 line

with extra types, that can reference types and strings from the

main BTF.

Quoted 2 lines
Quoted 2 lines

This is going to be used to generate compact BTFs for kernel

modules, with

Quoted 1 line
Quoted 1 line

vmlinux BTF being a main BTF, which all kernel modules are based

off of.

Quoted 2 lines
Quoted 2 lines

These changes rely on patch set [0] to be present in libbpf

Quoted 8 lines

submodule.

Quoted 2 lines
Quoted 2 lines

[0]

https://patchwork.kernel.org/project/netdevbpf/list/?series=377859&state=*

Quoted 2 lines

So, applied and added this:

Awesome, thanks! Do you plan to release v1.19 soon?

Yes

Quoted 26 lines

man-pages/pahole.1 Do not encode VARs in BTF. · line 190

Quoted 13 lines
man-pages/pahole.1 +3 -0
index 4b5e0a1bf5462b28..20ee91fc911d4b39 100644
@@ -185,6 +185,10 @@ Do not encode VARs in BTF.
 .B \-\-btf_encode_force
 Ignore those symbols found invalid when encoding BTF.
 
+.TP
+.B \-\-btf_base
+Path to the base BTF file, for instance: vmlinux when encoding

kernel module BTF information.

Quoted 12 lines
man-pages/pahole.1 +1 -1
+
 .TP
 .B \-l, \-\-show_first_biggest_size_base_type_member
 Show first biggest size base_type member.
 
---------------

The entry for btf_encode/-J is missing, I'll add in a followup patch.

Also I had to fixup ARGP_btf_base to 321 as I added this, to simplify the kernel scripts and Makefiles:

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
  $ pahole --numeric_version
  118
  $

Oh, this is nice! Can't really use it with Kbuild now due to backwards compatibility, but maybe someday.

Well, if it fails with --numeric_version, then it is old and the warning about the minimal version being v1.19 should be emitted :)

Right, but for CONFIG_DEBUG_INFO_BTF v1.13 is still adequate enough and will generates useful BTF (no functions and no module BTF, but all the CO-RE stuff is there), I wouldn't want to force everyone to get v1.19 immediately...

Quoted 14 lines

- Arnaldo

Quoted 9 lines
Quoted 7 lines

Now to test this all by applying the kernel patches and the encoding module BTF, looking at it, etc.

- Arnaldo

[...]

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

#13 lore raw
In reply to Andrii Nakryiko
Quoted 4 lines · Andrii Nakryiko

Add ability to generate split BTF (for kernel modules), as well as load split BTF. --btf_base argument is added to specify base BTF for split BTF. This works for both btf_loader and btf_encoder.

Arnaldo, can you please take a look at these patches? Would be nice to get them landed ASAP so that we can start testing out kernel module BTFs without locally applying patches first. Thanks!

Quoted 19 lines

Andrii Nakryiko (4):
btf_encoder: fix array index type numbering
libbtf: improve variable naming and error reporting when writing out
BTF
libbpf: update libbpf submodlue reference to latest master
btf: add support for split BTF loading and encoding

 btf_encoder.c | 15 ++++++++-------
 btf_loader.c  |  2 +-
 lib/bpf       |  2 +-
 libbtf.c      | 43 +++++++++++++++++++++++++++----------------
 libbtf.h      |  4 +++-
 pahole.c      | 23 +++++++++++++++++++++++
 6 files changed, 63 insertions(+), 26 deletions(-)

--
2.24.1

#14 lore raw
Quoted 4 lines · Andrii Nakryiko
Quoted 2 lines · Andrii Nakryiko

Add ability to generate split BTF (for kernel modules), as well as

Quoted 8 lines

load split

Quoted 1 line

BTF. --btf_base argument is added to specify base BTF for split BTF.

This

Quoted 1 line

works for both btf_loader and btf_encoder.

Arnaldo, can you please take a look at these patches? Would be nice to get them landed ASAP so that we can start testing out kernel module BTFs without locally applying patches first. Thanks!

I've been working on prepping up v1.19, will process these patches first thing in the morning, tomorrow,

Thanks,

- Arnaldo

Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
#15 lore raw
Quoted 11 lines · Arnaldo
Quoted 7 lines · Andrii Nakryiko
Quoted 2 lines · Andrii Nakryiko

Add ability to generate split BTF (for kernel modules), as well as

load split

Quoted 1 line

BTF. --btf_base argument is added to specify base BTF for split BTF.

This

Quoted 9 lines
Quoted 5 lines
Quoted 1 line

works for both btf_loader and btf_encoder.

Arnaldo, can you please take a look at these patches? Would be nice to get them landed ASAP so that we can start testing out kernel module BTFs without locally applying patches first. Thanks!

I've been working on prepping up v1.19, will process these patches first thing in the morning, tomorrow,

Thanks! Do you plan to include these changes into v1.19 as well?

Quoted 5 lines

Thanks,

- Arnaldo
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

#16 lore raw
Quoted 19 lines · Andrii Nakryiko
Quoted 1 line · Arnaldo

On November 10, 2020 8:34:18 PM GMT-03:00, Andrii Nakryiko

Quoted 6 lines · <[email protected]> wrote:
Quoted 6 lines
Quoted 2 lines · Andrii Nakryiko

Add ability to generate split BTF (for kernel modules), as well as

load split

Quoted 1 line

BTF. --btf_base argument is added to specify base BTF for split

BTF.

Quoted 4 lines
Quoted 4 lines

This

Quoted 1 line

works for both btf_loader and btf_encoder.

Arnaldo, can you please take a look at these patches? Would be nice

to

Quoted 4 lines
Quoted 2 lines

get them landed ASAP so that we can start testing out kernel module BTFs without locally applying patches first. Thanks!

Quoted 5 lines
Quoted 1 line

I've been working on prepping up v1.19, will process these patches

first thing in the morning, tomorrow,

Quoted 1 line

Thanks! Do you plan to include these changes into v1.19 as well?

Lemme test it tomorrow morning.

- Arnaldo

Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.