Add split BTF support to pahole
Subsystems (2)
Andrii Nakryiko
maintainercover Add split BTF support to pahole
Add ability to generate split BTF (for kernel modules), as well as load split BTF. --btf_base argument is added to specify base BTF for split BTF. This works for both btf_loader and btf_encoder.
Andrii Nakryiko (4):
btf_encoder: fix array index type numbering
libbtf: improve variable naming and error reporting when writing out
BTF
libbpf: update libbpf submodlue reference to latest master
btf: add support for split BTF loading and encoding
btf_encoder.c | 15 ++++++++------- btf_loader.c | 2 +- lib/bpf | 2 +- libbtf.c | 43 +++++++++++++++++++++++++++---------------- libbtf.h | 4 +++- pahole.c | 23 +++++++++++++++++++++++ 6 files changed, 63 insertions(+), 26 deletions(-)
Signature · 1 line
2.24.1
Andrii Nakryiko
maintainerPATCH 1/4 btf_encoder: fix array index type numbering
Take into account type ID offset, accumulated from previous CUs, when calculating a new type ID for the generated array index type.
--- btf_encoder.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
btf_encoder.c +6 -6
@@ -358,22 +358,22 @@ int cu__encode_btf(struct cu *cu, int verbose, bool force,printf ("File %s:\n" , btfe->filename); } + btf_elf__verbose = verbose; + btf_elf__force = force; + type_id_off =btf__get_nr_types (btfe->btf); +if (!has_index_type) {/* cu__find_base_type_by_name() takes "type_id_t *id" */ type_id_t id;if (cu__find_base_type_by_name (cu,"int" , &id)) { has_index_type = true; - array_index_id = id; + array_index_id = type_id_off + id; }else { has_index_type = false; - array_index_id = cu->types_table.nr_entries; + array_index_id = type_id_off + cu->types_table.nr_entries; } } - btf_elf__verbose = verbose; - btf_elf__force = force; - type_id_off =btf__get_nr_types (btfe->btf); -cu__for_each_type (cu, core_id, pos) {int32_t btf_type_id =tag__encode_btf (cu, pos, core_id, btfe, array_index_id, type_id_off);
Signature · 1 line
2.24.1
Andrii Nakryiko
maintainerPATCH 2/4 libbtf: improve variable naming and error reporting when writing out BTF
In reply to Andrii NakryikoRename few local variables to reflects the purpose a bit better. Also separate writing out BTF raw data and objcopy invocation into two separate steps and improve error reporting for each.
--- libbtf.c | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-)
libbtf.c +21 -12
@@ -679,11 +679,11 @@ static int btf_elf__write(const char *filename, struct btf *btf) {GElf_Shdr shdr_mem, *shdr;GElf_Ehdr ehdr_mem, *ehdr; -Elf_Data *btf_elf =NULL ; +Elf_Data *btf_data =NULL ;Elf_Scn *scn =NULL ;Elf *elf =NULL ; -const void *btf_data; -uint32_t btf_size; +const void *raw_btf_data; +uint32_t raw_btf_size;int fd, err =-1 ;size_t strndx;
@@ -735,18 +735,18 @@ static int btf_elf__write(const char *filename, struct btf *btf)continue ;char *secname =elf_strptr (elf, strndx, shdr->sh_name);if (strcmp (secname,".BTF" ) ==0 ) { - btf_elf =elf_getdata (scn, btf_elf); + btf_data =elf_getdata (scn, btf_data);break ; } } - btf_data =btf__get_raw_data (btf, &btf_size); + raw_btf_data =btf__get_raw_data (btf, &raw_btf_size); -if (btf_elf) { +if (btf_data) {/* Exisiting .BTF section found */ - btf_elf->d_buf = (void *)btf_data; - btf_elf->d_size = btf_size; -elf_flagdata (btf_elf,ELF_C_SET ,ELF_F_DIRTY ); + btf_data->d_buf = (void *)raw_btf_data; + btf_data->d_size = raw_btf_size; +elf_flagdata (btf_data,ELF_C_SET ,ELF_F_DIRTY );if (elf_update (elf,ELF_C_NULL ) >=0 &&elf_update (elf,ELF_C_WRITE ) >=0 )
@@ -770,12 +770,21 @@ static int btf_elf__write(const char *filename, struct btf *btf) goto out; } +if (write (fd, raw_btf_data, raw_btf_size) != raw_btf_size) { +fprintf (stderr,"%s: write of %d bytes to '%s' failed: %d!\n" , + __func__, raw_btf_size, tmp_fn, errno); + goto out; + } +snprintf (cmd,sizeof (cmd),"%s --add-section .BTF=%s %s" , llvm_objcopy, tmp_fn, filename); +if (system (cmd)) { +fprintf (stderr,"%s: failed to add .BTF section to '%s': %d!\n" , + __func__, tmp_fn, errno); + goto out; + } -if (write (fd, btf_data, btf_size) == btf_size && !system (cmd)) - err =0 ; - + err =0 ;unlink (tmp_fn); }
Signature · 1 line
2.24.1
Andrii Nakryiko
maintainerPATCH 3/4 libbpf: update libbpf submodlue reference to latest master
In reply to Andrii NakryikoPull in latest libbpf changes, containing split BTF APIs.
--- lib/bpf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bpf +1 -1
@@ -1 +1 @@ -Subproject commit ff797cc905d9c5fe9acab92d2da127342b20f80f +Subproject commit 5af3d86b5a2c5fecdc3ab83822d083edd32b4396
Signature · 1 line
2.24.1
Andrii Nakryiko
maintainerPATCH 4/4 btf: add support for split BTF loading and encoding
In reply to Andrii NakryikoAdd support for generating split BTF, in which there is a designated base BTF, containing a base set of types, and a split BTF, which extends main BTF with extra types, that can reference types and strings from the main BTF.
This is going to be used to generate compact BTFs for kernel modules, with vmlinux BTF being a main BTF, which all kernel modules are based off of.
These changes rely on patch set [0] to be present in libbpf submodule.
[0] https://patchwork.kernel.org/project/netdevbpf/list/?series=377859&state=*
--- btf_encoder.c | 3 ++- btf_loader.c | 2 +- libbtf.c | 10 ++++++---- libbtf.h | 4 +++- pahole.c | 23 +++++++++++++++++++++++ 5 files changed, 35 insertions(+), 7 deletions(-)
btf_encoder.c +2 -1
@@ -12,6 +12,7 @@#include "dwarves.h" #include "libbtf.h" #include "lib/bpf/include/uapi/linux/btf.h" +#include "lib/bpf/src/libbpf.h" #include "hash.h" #include "elf_symtab.h" #include "btf_encoder.h"
@@ -343,7 +344,7 @@ int cu__encode_btf(struct cu *cu, int verbose, bool force, }if (!btfe) { - btfe =btf_elf__new (cu->filename, cu->elf); + btfe =btf_elf__new (cu->filename, cu->elf, base_btf);if (!btfe)return -1 ;
btf_loader.c +1 -1
@@ -534,7 +534,7 @@ struct debug_fmt_ops btf_elf__ops;int btf_elf__load_file (struct cus *cus,struct conf_load *conf,const char *filename) {int err; -struct btf_elf *btfe =btf_elf__new (filename,NULL ); +struct btf_elf *btfe =btf_elf__new (filename,NULL , base_btf);if (btfe ==NULL )return -1 ;
libbtf.c +6 -4
@@ -27,6 +27,7 @@#include "dwarves.h" #include "elf_symtab.h" +struct btf *base_btf;uint8_t btf_elf__verbose;uint8_t btf_elf__force;
@@ -52,9 +53,9 @@ int btf_elf__load(struct btf_elf *btfe)/* free initial empty BTF */ btf__free (btfe->btf);if (btfe->raw_btf) - btfe->btf =btf__parse_raw (btfe->filename); + btfe->btf =btf__parse_raw_split (btfe->filename, btfe->base_btf);else - btfe->btf =btf__parse_elf (btfe->filename,NULL ); + btfe->btf =btf__parse_elf_split (btfe->filename, btfe->base_btf); err =libbpf_get_error (btfe->btf);if (err)
@@ -63,7 +64,7 @@ int btf_elf__load(struct btf_elf *btfe)return 0 ; } -struct btf_elf *btf_elf__new (const char *filename,Elf *elf) +struct btf_elf *btf_elf__new (const char *filename,Elf *elf,struct btf *base_btf) {struct btf_elf *btfe =zalloc (sizeof (*btfe));GElf_Shdr shdr;
@@ -77,7 +78,8 @@ struct btf_elf *btf_elf__new(const char *filename, Elf *elf)if (btfe->filename ==NULL ) goto errout; - btfe->btf =btf__new_empty (); + btfe->base_btf = base_btf; + btfe->btf =btf__new_empty_split (base_btf);if (libbpf_get_error (btfe->btf)) {fprintf (stderr,"%s: failed to create empty BTF.\n" , __func__); goto errout;
libbtf.h +3 -1
@@ -27,8 +27,10 @@ struct btf_elf {uint32_t percpu_shndx;uint64_t percpu_base_addr;struct btf *btf; +struct btf *base_btf; }; +extern struct btf *base_btf;extern uint8_t btf_elf__verbose;extern uint8_t btf_elf__force;#define btf_elf__verbose_log (fmt, ...) { if (btf_elf__verbose) printf(fmt, __VA_ARGS__); }
@@ -39,7 +41,7 @@ struct cu;struct base_type ;struct ftype ; -struct btf_elf *btf_elf__new (const char *filename,Elf *elf); +struct btf_elf *btf_elf__new (const char *filename,Elf *elf,struct btf *base_btf);void btf_elf__delete (struct btf_elf *btf); int32_t btf_elf__add_base_type(struct btf_elf *btf, const struct base_type *bt,
pahole.c +23 -0
@@ -22,12 +22,15 @@#include "dutil.h" #include "ctf_encoder.h" #include "btf_encoder.h" +#include "libbtf.h" +#include "lib/bpf/src/libbpf.h" static bool btf_encode;static bool ctf_encode;static bool first_obj_only;static bool skip_encoding_btf_vars;static bool btf_encode_force; +static const char *base_btf_file;static uint8_t class__include_anonymous;static uint8_t class__include_nested_anonymous;
@@ -820,6 +823,7 @@ ARGP_PROGRAM_VERSION_HOOK_DEF = dwarves_print_version;#define ARGP_skip_encoding_btf_vars 317#define ARGP_btf_encode_force 318#define ARGP_just_packed_structs 319 +#define ARGP_btf_base 320static const struct argp_option pahole__options[] = { {
@@ -1093,6 +1097,12 @@ static const struct argp_option pahole__options[] = { .key = ARGP_hex_fmt, .doc = "Print offsets and sizes in hexadecimal", }, + { + .name = "btf_base", + .key = ARGP_btf_base, + .arg = "SIZE", + .doc = "Path to the base BTF file", + }, { .name = "btf_encode", .key = 'J',
@@ -1234,6 +1244,9 @@ static error_t pahole__options_parser(int key, char *arg, skip_encoding_btf_vars = true;break ;case ARGP_btf_encode_force: btf_encode_force = true;break ; +case ARGP_btf_base: + base_btf_file = arg; +break ;default :return ARGP_ERR_UNKNOWN ; }
@@ -2682,6 +2695,15 @@ int main(int argc, char *argv[]) goto out; } +if (base_btf_file) { + base_btf =btf__parse (base_btf_file,NULL ); +if (libbpf_get_error (base_btf)) { +fprintf (stderr,"Failed to parse base BTF '%s': %ld\n" , + base_btf_file,libbpf_get_error (base_btf)); + goto out; + } + } +struct cus *cus =cus__new ();if (cus ==NULL ) {fputs ("pahole: insufficient memory\n" , stderr);
@@ -2766,6 +2788,7 @@ out_cus_delete:#ifdef DEBUG_CHECK_LEAKS cus__delete (cus);structures__delete (); +btf__free (base_btf);#endif out_dwarves_exit: #ifdef DEBUG_CHECK_LEAKS
Signature · 1 line
2.24.1
Re: [PATCH dwarves 4/4] btf: add support for split BTF loading and encoding
Em Thu, Nov 05, 2020 at 09:25:49PM -0800, Andrii Nakryiko escreveu:
Quoted 10 lines
Add support for generating split BTF, in which there is a designated base BTF, containing a base set of types, and a split BTF, which extends main BTF with extra types, that can reference types and strings from the main BTF.
This is going to be used to generate compact BTFs for kernel modules, with vmlinux BTF being a main BTF, which all kernel modules are based off of.
These changes rely on patch set [0] to be present in libbpf submodule.
[0] https://patchwork.kernel.org/project/netdevbpf/list/?series=377859&state=*
So, applied and added this:
man-pages/pahole.1 +4 -0
@@ -185,6 +185,10 @@ Do not encode VARs in BTF. .B \-\-btf_encode_force Ignore those symbols found invalid when encoding BTF. +.TP +.B \-\-btf_base +Path to the base BTF file, for instance: vmlinux when encoding kernel module BTF information. + .TP .B \-l, \-\-show_first_biggest_size_base_type_member Show first biggest size base_type member.
---------------
The entry for btf_encode/-J is missing, I'll add in a followup patch.
Also I had to fixup ARGP_btf_base to 321 as I added this, to simplify the kernel scripts and Makefiles:
$ pahole --numeric_version 118 $
Now to test this all by applying the kernel patches and the encoding module BTF, looking at it, etc.
- Arnaldo
Quoted 188 lines
Signed-off-by Andrii Nakryiko <[email protected]>--- btf_encoder.c | 3 ++- btf_loader.c | 2 +- libbtf.c | 10 ++++++---- libbtf.h | 4 +++- pahole.c | 23 +++++++++++++++++++++++ 5 files changed, 35 insertions(+), 7 deletions(-)btf_encoder.c +2
-1@@ -12,6 +12,7 @@#include "dwarves.h" #include "libbtf.h" #include "lib/bpf/include/uapi/linux/btf.h" +#include "lib/bpf/src/libbpf.h" #include "hash.h" #include "elf_symtab.h" #include "btf_encoder.h" @@ -343,7 +344,7 @@ int cu__encode_btf(struct cu *cu, int verbose, bool force, }if (!btfe) { - btfe =btf_elf__new (cu->filename, cu->elf); + btfe =btf_elf__new (cu->filename, cu->elf, base_btf);if (!btfe)return -1 ;btf_loader.c +1
-1@@ -534,7 +534,7 @@ struct debug_fmt_ops btf_elf__ops;int btf_elf__load_file (struct cus *cus,struct conf_load *conf,const char *filename) {int err; -struct btf_elf *btfe =btf_elf__new (filename,NULL ); +struct btf_elf *btfe =btf_elf__new (filename,NULL , base_btf);if (btfe ==NULL )return -1 ;libbtf.c +6
-4@@ -27,6 +27,7 @@#include "dwarves.h" #include "elf_symtab.h" +struct btf *base_btf;uint8_t btf_elf__verbose;uint8_t btf_elf__force;@@ -52,9 +53,9 @@ int btf_elf__load(struct btf_elf *btfe)/* free initial empty BTF */ btf__free (btfe->btf);if (btfe->raw_btf) - btfe->btf =btf__parse_raw (btfe->filename); + btfe->btf =btf__parse_raw_split (btfe->filename, btfe->base_btf);else - btfe->btf =btf__parse_elf (btfe->filename,NULL ); + btfe->btf =btf__parse_elf_split (btfe->filename, btfe->base_btf); err =libbpf_get_error (btfe->btf);if (err)@@ -63,7 +64,7 @@ int btf_elf__load(struct btf_elf *btfe)return 0 ; } -struct btf_elf *btf_elf__new (const char *filename,Elf *elf) +struct btf_elf *btf_elf__new (const char *filename,Elf *elf,struct btf *base_btf) {struct btf_elf *btfe =zalloc (sizeof (*btfe));GElf_Shdr shdr;@@ -77,7 +78,8 @@ struct btf_elf *btf_elf__new(const char *filename, Elf *elf)if (btfe->filename ==NULL ) goto errout; - btfe->btf =btf__new_empty (); + btfe->base_btf = base_btf; + btfe->btf =btf__new_empty_split (base_btf);if (libbpf_get_error (btfe->btf)) {fprintf (stderr,"%s: failed to create empty BTF.\n" , __func__); goto errout;libbtf.h +3
-1@@ -27,8 +27,10 @@ struct btf_elf {uint32_t percpu_shndx;uint64_t percpu_base_addr;struct btf *btf; +struct btf *base_btf; }; +extern struct btf *base_btf;extern uint8_t btf_elf__verbose;extern uint8_t btf_elf__force;#define btf_elf__verbose_log (fmt, ...) { if (btf_elf__verbose) printf(fmt, __VA_ARGS__); }@@ -39,7 +41,7 @@ struct cu;struct base_type ;struct ftype ; -struct btf_elf *btf_elf__new (const char *filename,Elf *elf); +struct btf_elf *btf_elf__new (const char *filename,Elf *elf,struct btf *base_btf);void btf_elf__delete (struct btf_elf *btf); int32_t btf_elf__add_base_type(struct btf_elf *btf, const struct base_type *bt,pahole.c +23
-0@@ -22,12 +22,15 @@#include "dutil.h" #include "ctf_encoder.h" #include "btf_encoder.h" +#include "libbtf.h" +#include "lib/bpf/src/libbpf.h" static bool btf_encode;static bool ctf_encode;static bool first_obj_only;static bool skip_encoding_btf_vars;static bool btf_encode_force; +static const char *base_btf_file;static uint8_t class__include_anonymous;static uint8_t class__include_nested_anonymous;@@ -820,6 +823,7 @@ ARGP_PROGRAM_VERSION_HOOK_DEF = dwarves_print_version;#define ARGP_skip_encoding_btf_vars 317#define ARGP_btf_encode_force 318#define ARGP_just_packed_structs 319 +#define ARGP_btf_base 320static const struct argp_option pahole__options[] = { {@@ -1093,6 +1097,12 @@ static const struct argp_option pahole__options[] = { .key = ARGP_hex_fmt, .doc = "Print offsets and sizes in hexadecimal", }, + { + .name = "btf_base", + .key = ARGP_btf_base, + .arg = "SIZE", + .doc = "Path to the base BTF file", + }, { .name = "btf_encode", .key = 'J',@@ -1234,6 +1244,9 @@ static error_t pahole__options_parser(int key, char *arg, skip_encoding_btf_vars = true;break ;case ARGP_btf_encode_force: btf_encode_force = true;break ; +case ARGP_btf_base: + base_btf_file = arg; +break ;default :return ARGP_ERR_UNKNOWN ; }@@ -2682,6 +2695,15 @@ int main(int argc, char *argv[]) goto out; } +if (base_btf_file) { + base_btf =btf__parse (base_btf_file,NULL ); +if (libbpf_get_error (base_btf)) { +fprintf (stderr,"Failed to parse base BTF '%s': %ld\n" , + base_btf_file,libbpf_get_error (base_btf)); + goto out; + } + } +struct cus *cus =cus__new ();if (cus ==NULL ) {fputs ("pahole: insufficient memory\n" , stderr);@@ -2766,6 +2788,7 @@ out_cus_delete:#ifdef DEBUG_CHECK_LEAKS cus__delete (cus);structures__delete (); +btf__free (base_btf);#endif out_dwarves_exit: #ifdef DEBUG_CHECK_LEAKSSignature · 1 line
2.24.1
Signature · 1 line
- Arnaldo
Re: [PATCH dwarves 4/4] btf: add support for split BTF loading and encoding
Em Wed, Nov 11, 2020 at 08:56:27AM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 9 lines
The entry for btf_encode/-J is missing, I'll add in a followup patch.
Also I had to fixup
ARGP_btf_baseto 321 as I added this, to simplify the kernel scripts and Makefiles:$ pahole --numeric_version 118 $
Added this:
[acme@five pahole]$ git diff
man-pages/pahole.1 +8 -0
@@ -181,6 +181,14 @@ the debugging information. .B \-\-skip_encoding_btf_vars Do not encode VARs in BTF. +.TP +.B \-J, \-\-btf_encode +Encode BTF information from DWARF, used in the Linux kernel build process when +CONFIG_DEBUG_INFO_BTF=y is present, introduced in Linux v5.2. Used to implement +features such as BPF CO-RE (Compile Once - Run Everywhere). + +See \fIhttp://vger.kernel.org/bpfconf2019_talks/bpf-core.pdf\fR. + .TP .B \-\-btf_encode_force Ignore those symbols found invalid when encoding BTF.
[acme@five pahole]$
Re: [PATCH dwarves 4/4] btf: add support for split BTF loading and encoding
Quoted 20 lines · Arnaldo Carvalho de Melo
Em Wed, Nov 11, 2020 at 08:56:27AM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 9 lines
The entry for btf_encode/-J is missing, I'll add in a followup patch.
Also I had to fixup
ARGP_btf_baseto 321 as I added this, to simplify the kernel scripts and Makefiles:$ pahole --numeric_version 118 $Added this:
[acme@five pahole]$ git diff
man-pages/pahole.1 +0
-0
man-pages/pahole.1 the debugging information. · line 190
Quoted 10 lines
man-pages/pahole.1 +7
-0@@ -181,6 +181,14 @@ the debugging information. .B \-\-skip_encoding_btf_vars Do not encode VARs in BTF. +.TP +.B \-J, \-\-btf_encode +Encode BTF information from DWARF, used in the Linux kernel build process when +CONFIG_DEBUG_INFO_BTF=y is present, introduced in Linux v5.2. Used to implement +features such as BPF CO-RE (Compile Once - Run Everywhere). + +See \fIhttp://vger.kernel.org/bpfconf2019_talks/bpf-core.pdf\fR.
Can you please point to [0] instead? That linked presentation is already a bit out of date and will bitrot much faster. Blog post has at least a chance at being updated with relevant important stuff. Plus it has links both to the bpfconf2019 presentation, as well as some other resources (including your presentation).
Quoted 5 lines
man-pages/pahole.1 +1
-0+ .TP .B \-\-btf_encode_force Ignore those symbols found invalid when encoding BTF.[acme@five pahole]$
Re: [PATCH dwarves 4/4] btf: add support for split BTF loading and encoding
Quoted 32 lines · Andrii Nakryiko
Quoted 2 lines · Arnaldo Carvalho de Melo
Em Wed, Nov 11, 2020 at 08:56:27AM -0300, Arnaldo Carvalho de Melo
escreveu:
Quoted 2 lines
Quoted 2 lines
The entry for btf_encode/-J is missing, I'll add in a followup
patch.
Quoted 2 lines
Quoted 2 lines
Also I had to fixup
ARGP_btf_baseto 321 as I added this, tosimplify
Quoted 14 lines
Quoted 5 lines
the kernel scripts and Makefiles:
$ pahole --numeric_version 118 $Added this:
[acme@five pahole]$ git diff
man-pages/pahole.1 +0
-0man-pages/pahole.1 the debugging information. · line 186
Quoted 6 lines
man-pages/pahole.1 +3
-0@@ -181,6 +181,14 @@ the debugging information. .B \-\-skip_encoding_btf_vars Do not encode VARs in BTF. +.TP +.B \-J, \-\-btf_encode +Encode BTF information from DWARF, used in the Linux kernel buildprocess when
Quoted 7 lines
man-pages/pahole.1 the debugging information. · line 187
Quoted 1 line
man-pages/pahole.1 +1
-0+CONFIG_DEBUG_INFO_BTF=y is present, introduced in Linux v5.2. Usedto implement
man-pages/pahole.1 the debugging information. · line 190
Quoted 3 lines
man-pages/pahole.1 +3
-0+features such as BPF CO-RE (Compile Once - Run Everywhere). + +See \fIhttp://vger.kernel.org/bpfconf2019_talks/bpf-core.pdf\fR.Can you please point to [0] instead? That linked
Sure.
Quoted 7 lines
presentation is
already a bit out of date and will bitrot much faster. Blog post has at least a chance at being updated with relevant important stuff. Plus it has links both to the bpfconf2019 presentation, as well as some other resources (including your presentation).
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: [PATCH dwarves 4/4] btf: add support for split BTF loading and encoding
In reply to Andrii NakryikoEm Wed, Nov 11, 2020 at 10:29:32AM -0800, Andrii Nakryiko escreveu:
Quoted 39 lines
Quoted 20 lines · Arnaldo Carvalho de Melo
Em Wed, Nov 11, 2020 at 08:56:27AM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 9 lines
The entry for btf_encode/-J is missing, I'll add in a followup patch.
Also I had to fixup
ARGP_btf_baseto 321 as I added this, to simplify the kernel scripts and Makefiles:$ pahole --numeric_version 118 $Added this:
[acme@five pahole]$ git diff
man-pages/pahole.1 +0
-0man-pages/pahole.1 the debugging information. · line 190
Quoted 10 lines
man-pages/pahole.1 +7
-0@@ -181,6 +181,14 @@ the debugging information. .B \-\-skip_encoding_btf_vars Do not encode VARs in BTF. +.TP +.B \-J, \-\-btf_encode +Encode BTF information from DWARF, used in the Linux kernel build process when +CONFIG_DEBUG_INFO_BTF=y is present, introduced in Linux v5.2. Used to implement +features such as BPF CO-RE (Compile Once - Run Everywhere). + +See \fIhttp://vger.kernel.org/bpfconf2019_talks/bpf-core.pdf\fR.Can you please point to [0] instead? That linked presentation is already a bit out of date and will bitrot much faster. Blog post has at least a chance at being updated with relevant important stuff. Plus it has links both to the bpfconf2019 presentation, as well as some other resources (including your presentation).
Quoted 1 line
Done, thanks for the pinter.
- Arnaldo
Re: [PATCH dwarves 4/4] btf: add support for split BTF loading and encoding
In reply to Arnaldo Carvalho de MeloQuoted 6 lines · Arnaldo Carvalho de Melo
Em Thu, Nov 05, 2020 at 09:25:49PM -0800, Andrii Nakryiko escreveu:
Quoted 3 lines
Add support for generating split BTF, in which there is a designated base BTF, containing a base set of types, and a split BTF, which extends main BTF with extra types, that can reference types and strings from the main BTF.
Quoted 8 lines
Quoted 6 lines
This is going to be used to generate compact BTFs for kernel modules, with vmlinux BTF being a main BTF, which all kernel modules are based off of.
These changes rely on patch set [0] to be present in libbpf submodule.
[0] https://patchwork.kernel.org/project/netdevbpf/list/?series=377859&state=*
So, applied and added this:
Awesome, thanks! Do you plan to release v1.19 soon?
Quoted 19 lines
man-pages/pahole.1 +4
-1@@ -185,6 +185,10 @@ Do not encode VARs in BTF. .B \-\-btf_encode_force Ignore those symbols found invalid when encoding BTF. +.TP +.B \-\-btf_base +Path to the base BTF file, for instance: vmlinux when encoding kernel module BTF information. + .TP .B \-l, \-\-show_first_biggest_size_base_type_member Show first biggest size base_type member. ---------------
Quoted 8 lines
The entry for btf_encode/-J is missing, I'll add in a followup patch.
Also I had to fixup
ARGP_btf_baseto 321 as I added this, to simplify the kernel scripts and Makefiles:$ pahole --numeric_version 118 $
Oh, this is nice! Can't really use it with Kbuild now due to backwards compatibility, but maybe someday.
Quoted 6 lines
Now to test this all by applying the kernel patches and the encoding module BTF, looking at it, etc.
- Arnaldo
[...]
Re: [PATCH dwarves 4/4] btf: add support for split BTF loading and encoding
Quoted 17 lines · Andrii Nakryiko
Quoted 3 lines · Arnaldo Carvalho de Melo
Em Thu, Nov 05, 2020 at 09:25:49PM -0800, Andrii Nakryiko escreveu:
Quoted 1 line
Add support for generating split BTF, in which there is a
designated base
Quoted 1 line
Quoted 1 line
BTF, containing a base set of types, and a split BTF, which extends
main BTF
Quoted 1 line
Quoted 1 line
with extra types, that can reference types and strings from the
main BTF.
Quoted 2 lines
Quoted 2 lines
This is going to be used to generate compact BTFs for kernel
modules, with
Quoted 1 line
Quoted 1 line
vmlinux BTF being a main BTF, which all kernel modules are based
off of.
Quoted 2 lines
Quoted 2 lines
These changes rely on patch set [0] to be present in libbpf
Quoted 8 lines
submodule.
Quoted 2 lines
Quoted 2 lines
[0]
https://patchwork.kernel.org/project/netdevbpf/list/?series=377859&state=*
Quoted 2 lines
So, applied and added this:
Awesome, thanks! Do you plan to release v1.19 soon?
Yes
Quoted 26 lines
man-pages/pahole.1 Do not encode VARs in BTF. · line 190
Quoted 13 lines
man-pages/pahole.1 +3
-0@@ -185,6 +185,10 @@ Do not encode VARs in BTF. .B \-\-btf_encode_force Ignore those symbols found invalid when encoding BTF. +.TP +.B \-\-btf_base +Path to the base BTF file, for instance: vmlinux when encodingkernel module BTF information.
Quoted 12 lines
man-pages/pahole.1 +1
-1+ .TP .B \-l, \-\-show_first_biggest_size_base_type_member Show first biggest size base_type member. ---------------The entry for btf_encode/-J is missing, I'll add in a followup patch.
Also I had to fixup
ARGP_btf_baseto 321 as I added this, to simplify the kernel scripts and Makefiles:
Quoted 6 lines
Quoted 3 lines
$ pahole --numeric_version 118 $Oh, this is nice! Can't really use it with
Kbuildnow due to backwards compatibility, but maybe someday.
Well, if it fails with --numeric_version, then it is old and the warning about the minimal version being v1.19 should be emitted :)
- Arnaldo
Quoted 9 lines
Quoted 7 lines
Now to test this all by applying the kernel patches and the encoding module BTF, looking at it, etc.
- Arnaldo
[...]
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: [PATCH dwarves 4/4] btf: add support for split BTF loading and encoding
Quoted 58 lines · Arnaldo
Quoted 17 lines · Andrii Nakryiko
Quoted 3 lines · Arnaldo Carvalho de Melo
Em Thu, Nov 05, 2020 at 09:25:49PM -0800, Andrii Nakryiko escreveu:
Quoted 1 line
Add support for generating split BTF, in which there is a
designated base
Quoted 1 line
Quoted 1 line
BTF, containing a base set of types, and a split BTF, which extends
main BTF
Quoted 1 line
Quoted 1 line
with extra types, that can reference types and strings from the
main BTF.
Quoted 2 lines
Quoted 2 lines
This is going to be used to generate compact BTFs for kernel
modules, with
Quoted 1 line
Quoted 1 line
vmlinux BTF being a main BTF, which all kernel modules are based
off of.
Quoted 2 lines
Quoted 2 lines
These changes rely on patch set [0] to be present in libbpf
Quoted 8 lines
submodule.
Quoted 2 lines
Quoted 2 lines
[0]
https://patchwork.kernel.org/project/netdevbpf/list/?series=377859&state=*
Quoted 2 lines
So, applied and added this:
Awesome, thanks! Do you plan to release v1.19 soon?
Yes
Quoted 26 lines
man-pages/pahole.1 Do not encode VARs in BTF. · line 190
Quoted 13 lines
man-pages/pahole.1 +3
-0@@ -185,6 +185,10 @@ Do not encode VARs in BTF. .B \-\-btf_encode_force Ignore those symbols found invalid when encoding BTF. +.TP +.B \-\-btf_base +Path to the base BTF file, for instance: vmlinux when encodingkernel module BTF information.
Quoted 12 lines
man-pages/pahole.1 +1
-1+ .TP .B \-l, \-\-show_first_biggest_size_base_type_member Show first biggest size base_type member. ---------------The entry for btf_encode/-J is missing, I'll add in a followup patch.
Also I had to fixup
ARGP_btf_baseto 321 as I added this, to simplify the kernel scripts and Makefiles:
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
$ pahole --numeric_version 118 $Oh, this is nice! Can't really use it with
Kbuildnow due to backwards compatibility, but maybe someday.Well, if it fails with --numeric_version, then it is old and the warning about the minimal version being v1.19 should be emitted :)
Right, but for CONFIG_DEBUG_INFO_BTF v1.13 is still adequate enough and will generates useful BTF (no functions and no module BTF, but all the CO-RE stuff is there), I wouldn't want to force everyone to get v1.19 immediately...
Quoted 14 lines
- Arnaldo
Quoted 9 lines
Quoted 7 lines
Now to test this all by applying the kernel patches and the encoding module BTF, looking at it, etc.
- Arnaldo
[...]
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Quoted 4 lines · Andrii Nakryiko
Add ability to generate split BTF (for kernel modules), as well as load split BTF. --btf_base argument is added to specify base BTF for split BTF. This works for both
btf_loaderandbtf_encoder.
Arnaldo, can you please take a look at these patches? Would be nice to get them landed ASAP so that we can start testing out kernel module BTFs without locally applying patches first. Thanks!
Quoted 19 lines
Andrii Nakryiko (4):
btf_encoder: fix array index type numbering
libbtf: improve variable naming and error reporting when writing out
BTF
libbpf: update libbpf submodlue reference to latest master
btf: add support for split BTF loading and encodingbtf_encoder.c | 15 ++++++++------- btf_loader.c | 2 +- lib/bpf | 2 +- libbtf.c | 43 +++++++++++++++++++++++++++---------------- libbtf.h | 4 +++- pahole.c | 23 +++++++++++++++++++++++ 6 files changed, 63 insertions(+), 26 deletions(-)--
2.24.1
Quoted 4 lines · Andrii Nakryiko
Quoted 2 lines · Andrii Nakryiko
Add ability to generate split BTF (for kernel modules), as well as
Quoted 8 lines
load split
Quoted 1 line
BTF. --btf_base argument is added to specify base BTF for split BTF.
This
Quoted 1 line
works for both
btf_loaderandbtf_encoder.Arnaldo, can you please take a look at these patches? Would be nice to get them landed ASAP so that we can start testing out kernel module BTFs without locally applying patches first. Thanks!
I've been working on prepping up v1.19, will process these patches first thing in the morning, tomorrow,
Thanks,
- Arnaldo
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Quoted 11 lines · Arnaldo
Quoted 7 lines · Andrii Nakryiko
Quoted 2 lines · Andrii Nakryiko
Add ability to generate split BTF (for kernel modules), as well as
load split
Quoted 1 line
BTF. --btf_base argument is added to specify base BTF for split BTF.
This
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
works for both
btf_loaderandbtf_encoder.Arnaldo, can you please take a look at these patches? Would be nice to get them landed ASAP so that we can start testing out kernel module BTFs without locally applying patches first. Thanks!
I've been working on prepping up v1.19, will process these patches first thing in the morning, tomorrow,
Thanks! Do you plan to include these changes into v1.19 as well?
Quoted 5 lines
Thanks,
- Arnaldo
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Quoted 19 lines · Andrii Nakryiko
Quoted 1 line · Arnaldo
On November 10, 2020 8:34:18 PM GMT-03:00, Andrii Nakryiko
Quoted 6 lines · <[email protected]> wrote:
Quoted 6 lines
Quoted 2 lines · Andrii Nakryiko
Add ability to generate split BTF (for kernel modules), as well as
load split
Quoted 1 line
BTF. --btf_base argument is added to specify base BTF for split
BTF.
Quoted 4 lines
Quoted 4 lines
This
Quoted 1 line
works for both
btf_loaderandbtf_encoder.Arnaldo, can you please take a look at these patches? Would be nice
to
Quoted 4 lines
Quoted 2 lines
get them landed ASAP so that we can start testing out kernel module BTFs without locally applying patches first. Thanks!
Quoted 5 lines
Quoted 1 line
I've been working on prepping up v1.19, will process these patches
first thing in the morning, tomorrow,
Quoted 1 line
Thanks! Do you plan to include these changes into v1.19 as well?
Lemme test it tomorrow morning.
- Arnaldo
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.