veristat: memory accounting for bpf programs
Subsystems (4)
Eduard Zingerman
maintainerv3 cover veristat: memory accounting for bpf programs
When working on the verifier, it is sometimes interesting to know how a particular change affects memory consumption. This patch-set modifies veristat to provide such information. As a collateral, kernel needs an update to make allocations reachable from BPF program load accountable in memcg statistics.
Here is a sample output:
Program Peak states Peak memory (MiB) --------------- ----------- ----------------- lavd_select_cpu 2153 43 lavd_enqueue 1982 41 lavd_dispatch 3480 28
Technically, this is implemented by creating and entering a new cgroup at the start of veristat execution. The difference between values from cgroup "memory.peak" file before and after bpf_object__load() is used as a metric.
To minimize measurements jitter data is collected in megabytes.
Changelog:
v2: https://lore.kernel.org/bpf/[email protected]/
v2 -> v3:
- bpf_verifier_state->jmp_history and bpf_verifier_env->explored_states allocations are switched from GFP_USER to GFP_KERNEL_ACCOUNT (Andrii, Alexei);
- veristat.c:STR macro removed, PATH_MAX-1 == 4095 is hard-coded in scanf format strings (Andrii);
- env->{orig,stat}_cgroup size changed to PATH_MAX (Andrii);
- snprintf_trunc() is removed, flag -Wno-format-truncation
is added to CFLAGS for veristat.o when compiled with gcc;
v1: https://lore.kernel.org/bpf/[email protected]/
v1 -> v2:
- a single cgroup, created at the beginning of execution, is now used for measurements (Andrii, Mykyta);
- cgroup namespace is not created, as it turned out to be useless (Andrii);
- veristat no longer mounts cgroup fs or changes subtree_control, instead it looks for an existing mount point and reports an error if memory.peak file can't be opened (Andrii, Alexei);
- if 'mem_peak' statistics is not enabled, veristat skips cgroup setup;
- code sharing with cgroup_helpers.c was considered but was decided against to simplify veristat github sync.
Eduard Zingerman (2):
bpf: include verifier memory allocations in memcg statistics
veristat: memory accounting for bpf programs
kernel/bpf/btf.c | 15 +- kernel/bpf/verifier.c | 69 +++---- tools/testing/selftests/bpf/Makefile | 8 + tools/testing/selftests/bpf/veristat.c | 248 ++++++++++++++++++++++++- 4 files changed, 292 insertions(+), 48 deletions(-)
Signature · 1 line
2.47.1
Eduard Zingerman
maintainerPATCH v3 1/2 bpf: include verifier memory allocations in memcg statistics
This commit adds __GFP_ACCOUNT flag to verifier induced memory allocations. The intent is to account for all allocations reachable from BPF_PROG_LOAD command, which is needed to track verifier memory consumption in veristat. This includes allocations done in verifier.c, and some allocations in btf.c, functions in log.c do not allocate.
There is also a utility function bpf_memcg_flags() which selectively adds GFP_ACCOUNT flag depending on the cgroup.memory=nobpf option. As far as I understand [1], the idea is to remove bpf_prog instances and maps from memcg accounting as these objects do not strictly belong to cgroup, hence it should not apply here.
(btf_parse_fields() is reachable from both program load and map
creation, but allocated record is not persistent as is freed as soon as map_check_btf() exits).
[1] https://lore.kernel.org/all/[email protected]/
--- kernel/bpf/btf.c | 15 +++++----- kernel/bpf/verifier.c | 69 ++++++++++++++++++++++--------------------- 2 files changed, 43 insertions(+), 41 deletions(-)
kernel/bpf/btf.c +8 -7
@@ -3443,7 +3443,8 @@ btf_find_graph_root(const struct btf *btf, const struct btf_type *pt, node_field_name =strstr (value_type,":" );if (!node_field_name)return -EINVAL ; - value_type =kstrndup (value_type, node_field_name - value_type,GFP_KERNEL | __GFP_NOWARN); + value_type =kstrndup (value_type, node_field_name - value_type, +GFP_KERNEL_ACCOUNT | __GFP_NOWARN);if (!value_type)return -ENOMEM ; id =btf_find_by_name_kind (btf, value_type,BTF_KIND_STRUCT );
@@ -3958,7 +3959,7 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type/* This needs to be kzalloc to zero out padding and unused fields, see * comment in btf_record_equal. */ - rec =kzalloc (struct_size (rec, fields, cnt),GFP_KERNEL | __GFP_NOWARN); + rec =kzalloc (struct_size (rec, fields, cnt),GFP_KERNEL_ACCOUNT | __GFP_NOWARN);if (!rec)return ERR_PTR (-ENOMEM );
@@ -9019,7 +9020,7 @@ static struct bpf_cand_cache *populate_cand_cache(struct bpf_cand_cache *cands,bpf_free_cands_from_cache (*cc); *cc =NULL ; } - new_cands =kmemdup (cands,sizeof_cands (cands->cnt),GFP_KERNEL ); + new_cands =kmemdup (cands,sizeof_cands (cands->cnt),GFP_KERNEL_ACCOUNT );if (!new_cands) {bpf_free_cands (cands);return ERR_PTR (-ENOMEM );
@@ -9027,7 +9028,7 @@ static struct bpf_cand_cache *populate_cand_cache(struct bpf_cand_cache *cands,/* strdup the name, since it will stay in cache. * the cands->name points to strings in prog's BTF and the prog can be unloaded. */ - new_cands->name =kmemdup_nul (cands->name, cands->name_len,GFP_KERNEL ); + new_cands->name =kmemdup_nul (cands->name, cands->name_len,GFP_KERNEL_ACCOUNT );bpf_free_cands (cands);if (!new_cands->name) {kfree (new_cands);
@@ -9111,7 +9112,7 @@ bpf_core_add_cands(struct bpf_cand_cache *cands, const struct btf *targ_btf,continue ;/* most of the time there is only one candidate for a given kind+name pair */ - new_cands =kmalloc (sizeof_cands (cands->cnt +1 ),GFP_KERNEL ); + new_cands =kmalloc (sizeof_cands (cands->cnt +1 ),GFP_KERNEL_ACCOUNT );if (!new_cands) {bpf_free_cands (cands);return ERR_PTR (-ENOMEM );
@@ -9228,7 +9229,7 @@ int bpf_core_apply(struct bpf_core_ctx *ctx, const struct bpf_core_relo *relo,/* ~4k of temp memory necessary to convert LLVM spec like "0:1:0:5" * into arrays of btf_ids of struct fields and array indices. */ - specs =kcalloc (3 ,sizeof (*specs),GFP_KERNEL ); + specs =kcalloc (3 ,sizeof (*specs),GFP_KERNEL_ACCOUNT );if (!specs)return -ENOMEM ;
@@ -9253,7 +9254,7 @@ int bpf_core_apply(struct bpf_core_ctx *ctx, const struct bpf_core_relo *relo, goto out; }if (cc->cnt) { - cands.cands =kcalloc (cc->cnt,sizeof (*cands.cands),GFP_KERNEL ); + cands.cands =kcalloc (cc->cnt,sizeof (*cands.cands),GFP_KERNEL_ACCOUNT );if (!cands.cands) { err = -ENOMEM ; goto out;
kernel/bpf/verifier.c +35 -34
250 lines not on this page · raw message
Signature · 1 line
2.47.1
Eduard Zingerman
maintainerPATCH v3 2/2 veristat: memory accounting for bpf programs
In reply to Eduard ZingermanThis commit adds a new field mem_peak / "Peak memory (MiB)" field to a set of gathered statistics. The field is intended as an estimate for peak verifier memory consumption for processing of a given program.
Mechanically stat is collected as follows:
- At the beginning of handle_verif_mode() a new cgroup is created and veristat process is moved into this cgroup.
- At each program load:
- bpf_object__load() is split into bpf_object__prepare() and bpf_object__load() to avoid accounting for memory allocated for maps;
- before bpf_object__load():
- a write to "memory.peak" file of the new cgroup is used to reset cgroup statistics;
- updated value is read from "memory.peak" file and stashed;
- after bpf_object__load() "memory.peak" is read again and difference between new and stashed values is used as a metric.
If any of the above steps fails veristat proceeds w/o collecting mem_peak information for a program, reporting mem_peak as -1.
While memcg provides data in bytes (converted from pages), veristat converts it to megabytes to avoid jitter when comparing results of different executions.
The change has no measurable impact on veristat running time.
A correlation between "Peak states" and "Peak memory" fields provides a sanity check for gathered statistics, e.g. a sample of data for sched_ext programs:
Program Peak states Peak memory (MiB) ------------------------ ----------- ----------------- lavd_select_cpu 2153 44 lavd_enqueue 1982 41 lavd_dispatch 3480 28 layered_dispatch 1417 17 layered_enqueue 760 11 lavd_cpu_offline 349 6 lavd_cpu_online 349 6 lavd_init 394 6 rusty_init 350 5 layered_select_cpu 391 4 ... rusty_stopping 134 1 arena_topology_node_init 170 0
--- tools/testing/selftests/bpf/Makefile | 8 + tools/testing/selftests/bpf/veristat.c | 248 ++++++++++++++++++++++++- 2 files changed, 249 insertions(+), 7 deletions(-)
tools/testing/selftests/bpf/Makefile +8 -0
@@ -841,6 +841,14 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \ $(call msg,BINARY,,$@) $(Q)$(CC) $(CFLAGS) $(LDFLAGS) $(filter %.a %.o,$^) $(LDLIBS) -o $@ +# This works around GCC warning about snprintf truncating strings like: +# +# char a[PATH_MAX], b[PATH_MAX]; +# snprintf(a, "%s/foo", b); // triggers -Wformat-truncation +ifeq ($(LLVM),) +$(OUTPUT)/veristat.o: CFLAGS+=-Wno-format-truncation +endif + $(OUTPUT)/veristat.o: $(BPFOBJ) $(OUTPUT)/veristat: $(OUTPUT)/veristat.o $(call msg,BINARY,,$@)
tools/testing/selftests/bpf/veristat.c +241 -7
350 lines not on this page · raw message
Signature · 1 line
2.47.1
Andrii Nakryiko
AppliedQuoted 59 lines · Eduard Zingerman
This commit adds a new field
mem_peak/ "Peak memory (MiB)" field to a set of gathered statistics. The field is intended as an estimate for peak verifier memory consumption for processing of a given program.Mechanically stat is collected as follows:
- At the beginning ofhandle_verif_mode()a new cgroup is created and veristat process is moved into this cgroup.
- At each program load:
-bpf_object__load()is split intobpf_object__prepare()andbpf_object__load()to avoid accounting for memory allocated for maps;
- beforebpf_object__load():
- a write to "memory.peak" file of the new cgroup is used to reset cgroup statistics;
- updated value is read from "memory.peak" file and stashed;
- afterbpf_object__load()"memory.peak" is read again and difference between new and stashed values is used as a metric.If any of the above steps fails veristat proceeds w/o collecting
mem_peakinformation for a program, reportingmem_peakas -1.While memcg provides data in bytes (converted from pages), veristat converts it to megabytes to avoid jitter when comparing results of different executions.
The change has no measurable impact on veristat running time.
A correlation between "Peak states" and "Peak memory" fields provides a sanity check for gathered statistics, e.g. a sample of data for
sched_extprograms:Program Peak states Peak memory (MiB) ------------------------ ----------- ----------------- lavd_select_cpu 2153 44 lavd_enqueue 1982 41 lavd_dispatch 3480 28 layered_dispatch 1417 17 layered_enqueue 760 11 lavd_cpu_offline 349 6 lavd_cpu_online 349 6 lavd_init 394 6 rusty_init 350 5 layered_select_cpu 391 4 ... rusty_stopping 134 1 arena_topology_node_init 170 0Signed-off-by Eduard Zingerman <[email protected]>--- tools/testing/selftests/bpf/Makefile | 8 + tools/testing/selftests/bpf/veristat.c | 248 ++++++++++++++++++++++++- 2 files changed, 249 insertions(+), 7 deletions(-)tools/testing/selftests/bpf/Makefile +0
-0
tools/testing/selftests/bpf/Makefile $(OUTPUT)/bench: $(OUTPUT)/bench.o \ · line 849
Quoted 9 lines
tools/testing/selftests/bpf/Makefile +6
-0@@ -841,6 +841,14 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \ $(call msg,BINARY,,$@) $(Q)$(CC) $(CFLAGS) $(LDFLAGS) $(filter %.a %.o,$^) $(LDLIBS) -o $@ +# This works around GCC warning about snprintf truncating strings like: +# +# char a[PATH_MAX], b[PATH_MAX]; +# snprintf(a, "%s/foo", b); // triggers -Wformat-truncation +ifeq ($(LLVM),) +$(OUTPUT)/veristat.o: CFLAGS+=-Wno-format-truncation
fixed formatting around +=, and dropped LLVM check, I think that should be handled by -Wno-unused-command-line-argument we set earlier (make LLVM=1 veristat succeeded without any warnings for me, FWIW)
applied to bpf-next, thanks
Quoted 5 lines
tools/testing/selftests/bpf/Makefile +2
-0+endif + $(OUTPUT)/veristat.o: $(BPFOBJ) $(OUTPUT)/veristat: $(OUTPUT)/veristat.o $(call msg,BINARY,,$@)
[...]
tools/testing/selftests/bpf/Makefile
Quoted 8 lines
tools/testing/selftests/bpf/Makefile +8
-0+/* + * Creates a cgroup at /sys/fs/cgroup/veristat-accounting-<pid>, + * moves current process to this cgroup. + */ +static void create_stat_cgroup(void) +{ + char cgroup_fs_mount[PATH_MAX]; + char buf[PATH_MAX];
I hard-coded these to 4096 given we hard-coded 4095 below.
[...]
This series was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:
Quoted 9 lines · On Fri, 13 Jun 2025 00:21:45 -0700 you wrote:
When working on the verifier, it is sometimes interesting to know how a particular change affects memory consumption. This patch-set modifies veristat to provide such information. As a collateral, kernel needs an update to make allocations reachable from BPF program load accountable in memcg statistics.
Here is a sample output:
[...]
Here is the summary with links:
- [bpf-next,v3,1/2] bpf: include verifier memory allocations in memcg statistics
https://git.kernel.org/bpf/bpf-next/c/43736ec3e027
- [bpf-next,v3,2/2] veristat: memory accounting for bpf programs (no matching commit)
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html