pahole: Make encoding percpu vars into BTF optional.
cover pahole: Make encoding percpu vars into BTF optional.
Previous commit f3d9054ba8ff ("btf_encoder: Teach pahole to store percpu variables in vmlinux BTF.") introduced a feature in btf_encoder that encodes global symbols in BTF. However, this feature is not protected by any flag. In order to avoid surprises after Pahole v1.18 rolls out, make this feature off by default and enable only upon request. On the kernel side, we can add a Kconfig to enable this new capability on an opt-in pattern.
Also as a refactor, introduce '--btf_encode_force' to replace the old '--force' and '-j' option that is used to forcefully emit BTF.
Hao Luo (2):
btf_encoder: Make encoding vars into BTF optional.
btf_encoder: Introduce option '--btf_encode_force'
btf_encoder.c | 5 ++++- btf_encoder.h | 2 +- pahole.c | 22 +++++++++++++++++----- 3 files changed, 22 insertions(+), 7 deletions(-)
Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
PATCH 1/2 btf_encoder: Make encoding vars into BTF optional.
Introduce an option --btf_encode_symbols to pahole so that pahole will only encode global symbols upon request. On the kernel side, we can add a Kconfig to turn on this feature. This would make the rollout of pahole v1.18 safe against potential bugs in the feature introduced in commit f3d9054ba8ff ("btf_encoder: Teach pahole to store percpu variables in vmlinux BTF.").
--- btf_encoder.c | 5 ++++- btf_encoder.h | 2 +- pahole.c | 11 ++++++++++- 3 files changed, 15 insertions(+), 3 deletions(-)
btf_encoder.c +4 -1
@@ -237,7 +237,7 @@ static struct variable *hashaddr__find_variable(const struct hlist_head hashtablreturn NULL ; } -int cu__encode_btf (struct cu *cu,int verbose,bool force) +int cu__encode_btf (struct cu *cu,int verbose,bool force,bool encode_symbols) {bool add_index_type = false;uint32_t type_id_off;
@@ -314,6 +314,9 @@ int cu__encode_btf(struct cu *cu, int verbose, bool force) } } +/* if there is no request to encode symbols, just finish. */ +if (!encode_symbols) + goto out;if (btfe->percpu_shndx ==0 || !btfe->symtab) goto out;
btf_encoder.h +1 -1
@@ -13,6 +13,6 @@ struct cu;int btf_encoder__encode (); -int cu__encode_btf (struct cu *cu,int verbose,bool force); +int cu__encode_btf (struct cu *cu,int verbose,bool force,bool encode_symbols);#endif /* _BTF_ENCODER_H_ */
pahole.c +10 -1
@@ -26,6 +26,7 @@static bool btf_encode;static bool ctf_encode;static bool first_obj_only; +static bool btf_encode_symbols;static uint8_t class__include_anonymous;static uint8_t class__include_nested_anonymous;
@@ -809,6 +810,7 @@ ARGP_PROGRAM_VERSION_HOOK_DEF = dwarves_print_version;#define ARGP_header_type 314#define ARGP_size_bytes 315#define ARGP_range 316 +#define ARGP_btf_encode_symbols 317static const struct argp_option pahole__options[] = { {
@@ -1087,6 +1089,11 @@ static const struct argp_option pahole__options[] = { .key = 'J', .doc = "Encode as BTF", }, + { + .name = "btf_encode_symbols", + .key = ARGP_btf_encode_symbols, + .doc ="Encode global symbols in BTF. Off by default." + }, { .name = "force", .key = 'j',
@@ -1207,6 +1214,8 @@ static error_t pahole__options_parser(int key, char *arg, conf.range = arg;break ;case ARGP_header_type: conf.header_type = arg;break ; +case ARGP_btf_encode_symbols: + btf_encode_symbols = true;break ;default :return ARGP_ERR_UNKNOWN ; }
@@ -2352,7 +2361,7 @@ static enum load_steal_kind pahole_stealer(struct cu *cu, goto filter_it;if (btf_encode) { -cu__encode_btf (cu, global_verbose, force); +cu__encode_btf (cu, global_verbose, force, btf_encode_symbols);return LSK__KEEPIT ; }
Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
Re: [PATCH 1/2] btf_encoder: Make encoding vars into BTF optional.
Em Fri, Sep 18, 2020 at 01:40:57PM -0700, Hao Luo escreveu:
Quoted 6 lines
Introduce an option --btf_encode_symbols to pahole so that pahole will only encode global symbols upon request. On the kernel side, we can add a
Kconfigto turn on this feature. This would make the rollout of pahole v1.18 safe against potential bugs in the feature introduced in commitf3d9054ba8ff("btf_encoder: Teach pahole to store percpu variables in vmlinux BTF.").
'encode_symbols' is too generic, please rename it to encode_global_symbols or something like that.
- Arnaldo
Quoted 95 lines
Signed-off-by Hao Luo <[email protected]>--- btf_encoder.c | 5 ++++- btf_encoder.h | 2 +- pahole.c | 11 ++++++++++- 3 files changed, 15 insertions(+), 3 deletions(-)btf_encoder.c +4
-1@@ -237,7 +237,7 @@ static struct variable *hashaddr__find_variable(const struct hlist_head hashtablreturn NULL ; } -int cu__encode_btf (struct cu *cu,int verbose,bool force) +int cu__encode_btf (struct cu *cu,int verbose,bool force,bool encode_symbols) {bool add_index_type = false;uint32_t type_id_off;@@ -314,6 +314,9 @@ int cu__encode_btf(struct cu *cu, int verbose, bool force) } } +/* if there is no request to encode symbols, just finish. */ +if (!encode_symbols) + goto out;if (btfe->percpu_shndx ==0 || !btfe->symtab) goto out;btf_encoder.h +1
-1@@ -13,6 +13,6 @@ struct cu;int btf_encoder__encode (); -int cu__encode_btf (struct cu *cu,int verbose,bool force); +int cu__encode_btf (struct cu *cu,int verbose,bool force,bool encode_symbols);#endif /* _BTF_ENCODER_H_ */ pahole.c +10
-1@@ -26,6 +26,7 @@static bool btf_encode;static bool ctf_encode;static bool first_obj_only; +static bool btf_encode_symbols;static uint8_t class__include_anonymous;static uint8_t class__include_nested_anonymous;@@ -809,6 +810,7 @@ ARGP_PROGRAM_VERSION_HOOK_DEF = dwarves_print_version;#define ARGP_header_type 314#define ARGP_size_bytes 315#define ARGP_range 316 +#define ARGP_btf_encode_symbols 317static const struct argp_option pahole__options[] = { {@@ -1087,6 +1089,11 @@ static const struct argp_option pahole__options[] = { .key = 'J', .doc = "Encode as BTF", }, + { + .name = "btf_encode_symbols", + .key = ARGP_btf_encode_symbols, + .doc ="Encode global symbols in BTF. Off by default." + }, { .name = "force", .key = 'j',@@ -1207,6 +1214,8 @@ static error_t pahole__options_parser(int key, char *arg, conf.range = arg;break ;case ARGP_header_type: conf.header_type = arg;break ; +case ARGP_btf_encode_symbols: + btf_encode_symbols = true;break ;default :return ARGP_ERR_UNKNOWN ; }@@ -2352,7 +2361,7 @@ static enum load_steal_kind pahole_stealer(struct cu *cu, goto filter_it;if (btf_encode) { -cu__encode_btf (cu, global_verbose, force); +cu__encode_btf (cu, global_verbose, force, btf_encode_symbols);return LSK__KEEPIT ; }Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
Signature · 1 line
- Arnaldo
Re: [PATCH 1/2] btf_encoder: Make encoding vars into BTF optional.
How about '--btf_encode_vars'? It is actually encoding BTF_KIND_VAR.
Hao
Quoted 113 lines · Arnaldo Carvalho de Melo
Em Fri, Sep 18, 2020 at 01:40:57PM -0700, Hao Luo escreveu:
Quoted 6 lines
Introduce an option --btf_encode_symbols to pahole so that pahole will only encode global symbols upon request. On the kernel side, we can add a
Kconfigto turn on this feature. This would make the rollout of pahole v1.18 safe against potential bugs in the feature introduced in commitf3d9054ba8ff("btf_encoder: Teach pahole to store percpu variables in vmlinux BTF.").'
encode_symbols' is too generic, please rename it toencode_global_symbolsor something like that.- Arnaldo
Quoted 93 lines
Signed-off-by Hao Luo <[email protected]>--- btf_encoder.c | 5 ++++- btf_encoder.h | 2 +- pahole.c | 11 ++++++++++- 3 files changed, 15 insertions(+), 3 deletions(-)btf_encoder.c +4
-1@@ -237,7 +237,7 @@ static struct variable *hashaddr__find_variable(const struct hlist_head hashtablreturn NULL ; } -int cu__encode_btf (struct cu *cu,int verbose,bool force) +int cu__encode_btf (struct cu *cu,int verbose,bool force,bool encode_symbols) {bool add_index_type = false;uint32_t type_id_off;@@ -314,6 +314,9 @@ int cu__encode_btf(struct cu *cu, int verbose, bool force) } } +/* if there is no request to encode symbols, just finish. */ +if (!encode_symbols) + goto out;if (btfe->percpu_shndx ==0 || !btfe->symtab) goto out;btf_encoder.h +1
-1@@ -13,6 +13,6 @@ struct cu;int btf_encoder__encode (); -int cu__encode_btf (struct cu *cu,int verbose,bool force); +int cu__encode_btf (struct cu *cu,int verbose,bool force,bool encode_symbols);#endif /* _BTF_ENCODER_H_ */ pahole.c +10
-2@@ -26,6 +26,7 @@static bool btf_encode;static bool ctf_encode;static bool first_obj_only; +static bool btf_encode_symbols;static uint8_t class__include_anonymous;static uint8_t class__include_nested_anonymous;@@ -809,6 +810,7 @@ ARGP_PROGRAM_VERSION_HOOK_DEF = dwarves_print_version;#define ARGP_header_type 314#define ARGP_size_bytes 315#define ARGP_range 316 +#define ARGP_btf_encode_symbols 317static const struct argp_option pahole__options[] = { {@@ -1087,6 +1089,11 @@ static const struct argp_option pahole__options[] = { .key = 'J', .doc = "Encode as BTF", }, + { + .name = "btf_encode_symbols", + .key = ARGP_btf_encode_symbols, + .doc ="Encode global symbols in BTF. Off by default." + }, { .name = "force", .key = 'j',@@ -1207,6 +1214,8 @@ static error_t pahole__options_parser(int key, char *arg, conf.range = arg;break ;case ARGP_header_type: conf.header_type = arg;break ; +case ARGP_btf_encode_symbols: + btf_encode_symbols = true;break ;default :return ARGP_ERR_UNKNOWN ; }@@ -2352,7 +2361,7 @@ static enum load_steal_kind pahole_stealer(struct cu *cu, goto filter_it;if (btf_encode) { -cu__encode_btf (cu, global_verbose, force); +cu__encode_btf (cu, global_verbose, force, btf_encode_symbols);return LSK__KEEPIT ; } --Quoted 2 lines
2.28.0.681.g6f77f65b4e-goog
--
- Arnaldo
Re: [PATCH 1/2] btf_encoder: Make encoding vars into BTF optional.
Em Fri, Sep 18, 2020 at 02:26:14PM -0700, Hao Luo escreveu:
Quoted 1 line
How about '--btf_encode_vars'? It is actually encoding
BTF_KIND_VAR.
Sounds good as it references both its for BTF and that is for KIND_VAR.
- Arnaldo
Quoted 117 lines
Hao
Quoted 113 lines · Arnaldo Carvalho de Melo
Em Fri, Sep 18, 2020 at 01:40:57PM -0700, Hao Luo escreveu:
Quoted 6 lines
Introduce an option --btf_encode_symbols to pahole so that pahole will only encode global symbols upon request. On the kernel side, we can add a
Kconfigto turn on this feature. This would make the rollout of pahole v1.18 safe against potential bugs in the feature introduced in commitf3d9054ba8ff("btf_encoder: Teach pahole to store percpu variables in vmlinux BTF.").'
encode_symbols' is too generic, please rename it toencode_global_symbolsor something like that.- Arnaldo
Quoted 93 lines
Signed-off-by Hao Luo <[email protected]>--- btf_encoder.c | 5 ++++- btf_encoder.h | 2 +- pahole.c | 11 ++++++++++- 3 files changed, 15 insertions(+), 3 deletions(-)btf_encoder.c +4
-1@@ -237,7 +237,7 @@ static struct variable *hashaddr__find_variable(const struct hlist_head hashtablreturn NULL ; } -int cu__encode_btf (struct cu *cu,int verbose,bool force) +int cu__encode_btf (struct cu *cu,int verbose,bool force,bool encode_symbols) {bool add_index_type = false;uint32_t type_id_off;@@ -314,6 +314,9 @@ int cu__encode_btf(struct cu *cu, int verbose, bool force) } } +/* if there is no request to encode symbols, just finish. */ +if (!encode_symbols) + goto out;if (btfe->percpu_shndx ==0 || !btfe->symtab) goto out;btf_encoder.h +1
-1@@ -13,6 +13,6 @@ struct cu;int btf_encoder__encode (); -int cu__encode_btf (struct cu *cu,int verbose,bool force); +int cu__encode_btf (struct cu *cu,int verbose,bool force,bool encode_symbols);#endif /* _BTF_ENCODER_H_ */ pahole.c +10
-2@@ -26,6 +26,7 @@static bool btf_encode;static bool ctf_encode;static bool first_obj_only; +static bool btf_encode_symbols;static uint8_t class__include_anonymous;static uint8_t class__include_nested_anonymous;@@ -809,6 +810,7 @@ ARGP_PROGRAM_VERSION_HOOK_DEF = dwarves_print_version;#define ARGP_header_type 314#define ARGP_size_bytes 315#define ARGP_range 316 +#define ARGP_btf_encode_symbols 317static const struct argp_option pahole__options[] = { {@@ -1087,6 +1089,11 @@ static const struct argp_option pahole__options[] = { .key = 'J', .doc = "Encode as BTF", }, + { + .name = "btf_encode_symbols", + .key = ARGP_btf_encode_symbols, + .doc ="Encode global symbols in BTF. Off by default." + }, { .name = "force", .key = 'j',@@ -1207,6 +1214,8 @@ static error_t pahole__options_parser(int key, char *arg, conf.range = arg;break ;case ARGP_header_type: conf.header_type = arg;break ; +case ARGP_btf_encode_symbols: + btf_encode_symbols = true;break ;default :return ARGP_ERR_UNKNOWN ; }@@ -2352,7 +2361,7 @@ static enum load_steal_kind pahole_stealer(struct cu *cu, goto filter_it;if (btf_encode) { -cu__encode_btf (cu, global_verbose, force); +cu__encode_btf (cu, global_verbose, force, btf_encode_symbols);return LSK__KEEPIT ; } --Quoted 2 lines
2.28.0.681.g6f77f65b4e-goog
--
- Arnaldo
Signature · 1 line
- Arnaldo
PATCH 2/2 btf_encoder: Introduce option '--btf_encode_force'
In reply to Hao LuoCommit f3d9054ba8ff ("btf_encoder: Teach pahole to store percpu variables in vmlinux BTF.") introduced an option '-j' that makes best effort in emitting VAR entries in BTF. Before no one has been using this flag, replace the one-letter option '-j' with a full flag name '--btf_encode_force' to save '-j' for future uses.
--- pahole.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
pahole.c +8 -5
@@ -27,6 +27,7 @@ static bool btf_encode;static bool ctf_encode;static bool first_obj_only;static bool btf_encode_symbols; +static bool btf_encode_force;static uint8_t class__include_anonymous;static uint8_t class__include_nested_anonymous;
@@ -62,7 +63,6 @@ static int show_reorg_steps;static const char *class_name;static LIST_HEAD (class_names);static char separator ='\t' ; -static bool force;static struct conf_fprintf conf = { .emit_stats =1 ,
@@ -811,6 +811,7 @@ ARGP_PROGRAM_VERSION_HOOK_DEF = dwarves_print_version;#define ARGP_size_bytes 315#define ARGP_range 316#define ARGP_btf_encode_symbols 317 +#define ARGP_btf_encode_force 318static const struct argp_option pahole__options[] = { {
@@ -1095,8 +1096,8 @@ static const struct argp_option pahole__options[] = { .doc = "Encode global symbols in BTF. Off by default." }, { - .name = "force", - .key = 'j', + .name = "btf_encode_force", + .key = ARGP_btf_encode_force, .doc ="Ignore those symbols found invalid when encoding BTF." }, {
@@ -1143,7 +1144,6 @@ static error_t pahole__options_parser(int key, char *arg,case 'J' : btf_encode =1 ; conf_load.get_addr_info = true; no_bitfield_type_recode = true;break ; -case 'j' : force = true;break ;case 'l' : conf.show_first_biggest_size_base_type_member =1 ;break ;case 'M' : conf.show_only_data_members =1 ;break ;case 'm' : stats_formatter = nr_methods_formatter;break ;
@@ -1216,6 +1216,8 @@ static error_t pahole__options_parser(int key, char *arg, conf.header_type = arg;break ;case ARGP_btf_encode_symbols: btf_encode_symbols = true;break ; +case ARGP_btf_encode_force: + btf_encode_force = true;break ;default :return ARGP_ERR_UNKNOWN ; }
@@ -2361,7 +2363,8 @@ static enum load_steal_kind pahole_stealer(struct cu *cu, goto filter_it;if (btf_encode) { -cu__encode_btf (cu, global_verbose, force, btf_encode_symbols); +cu__encode_btf (cu, global_verbose, btf_encode_force, + btf_encode_symbols);return LSK__KEEPIT ; }
Signature · 1 line
2.28.0.681.g6f77f65b4e-goog
Quoted 8 lines · Hao Luo
Previous commit
f3d9054ba8ff("btf_encoder: Teach pahole to store percpu variables in vmlinux BTF.") introduced a feature inbtf_encoderthat encodes global symbols in BTF. However, this feature is not protected by any flag. In order to avoid surprises after Pahole v1.18 rolls out, make this feature off by default and enable only upon request. On the kernel side, we can add aKconfigto enable this new capability on an opt-in pattern.
I think I've missed some earlier discussions and motivation for opt-in. Why opt-out is a problem?
I would think the kernel build should always include percpu vars.
Hi Alexei,
I was syncing with Arnaldo offline on whether the change in Pahole was enough for supporting percpu vars in BPF. While I think about it, I felt it's better to have a flag in pahole to toggle the feature on and off. My concern is mainly malformed BTF caused by the newly introduced VAR entries. I've tested over some kernel configurations and compilers but not confident enough.
I haven't thought about opt-out. Maybe we should use opt-out instead of opt-in?
Hao
Quoted 14 lines · Alexei Starovoitov
Quoted 8 lines · Hao Luo
Previous commit
f3d9054ba8ff("btf_encoder: Teach pahole to store percpu variables in vmlinux BTF.") introduced a feature inbtf_encoderthat encodes global symbols in BTF. However, this feature is not protected by any flag. In order to avoid surprises after Pahole v1.18 rolls out, make this feature off by default and enable only upon request. On the kernel side, we can add aKconfigto enable this new capability on an opt-in pattern.I think I've missed some earlier discussions and motivation for opt-in. Why opt-out is a problem?
I would think the kernel build should always include percpu vars.
Em Fri, Sep 18, 2020 at 09:00:10PM -0700, Hao Luo escreveu:
Quoted 10 lines
Hi Alexei,
I was syncing with Arnaldo offline on whether the change in Pahole was enough for supporting percpu vars in BPF. While I think about it, I felt it's better to have a flag in pahole to toggle the feature on and off. My concern is mainly malformed BTF caused by the newly introduced VAR entries. I've tested over some kernel configurations and compilers but not confident enough.
I haven't thought about opt-out. Maybe we should use opt-out instead of opt-in?
Yeah, having an opt-out should be useful as a debugging technique.
- Arnaldo
Quoted 18 lines
Hao
Quoted 14 lines · Alexei Starovoitov
Quoted 8 lines · Hao Luo
Previous commit
f3d9054ba8ff("btf_encoder: Teach pahole to store percpu variables in vmlinux BTF.") introduced a feature inbtf_encoderthat encodes global symbols in BTF. However, this feature is not protected by any flag. In order to avoid surprises after Pahole v1.18 rolls out, make this feature off by default and enable only upon request. On the kernel side, we can add aKconfigto enable this new capability on an opt-in pattern.I think I've missed some earlier discussions and motivation for opt-in. Why opt-out is a problem?
I would think the kernel build should always include percpu vars.
Signature · 1 line
- Arnaldo