Skip to contentSkip to conversation

veristat: memory accounting for bpf programs

PATCH Applied Eduard Zingerman 3 participants 5 messages Jun 13, 2025

Subsystems (4)
lore
Share & tools
lore

Eduard Zingerman

author maintainer
#0 lore raw

v3 cover veristat: memory accounting for bpf programs

When working on the verifier, it is sometimes interesting to know how a particular change affects memory consumption. This patch-set modifies veristat to provide such information. As a collateral, kernel needs an update to make allocations reachable from BPF program load accountable in memcg statistics.

Here is a sample output:

  Program          Peak states  Peak memory (MiB)
  ---------------  -----------  -----------------
  lavd_select_cpu         2153                 43
  lavd_enqueue            1982                 41
  lavd_dispatch           3480                 28

Technically, this is implemented by creating and entering a new cgroup at the start of veristat execution. The difference between values from cgroup "memory.peak" file before and after bpf_object__load() is used as a metric.

To minimize measurements jitter data is collected in megabytes.

Changelog:
v2: https://lore.kernel.org/bpf/[email protected]/
v2 -> v3:
- bpf_verifier_state->jmp_history and bpf_verifier_env->explored_states allocations are switched from GFP_USER to GFP_KERNEL_ACCOUNT (Andrii, Alexei);
- veristat.c:STR macro removed, PATH_MAX-1 == 4095 is hard-coded in scanf format strings (Andrii);
- env->{orig,stat}_cgroup size changed to PATH_MAX (Andrii);
- snprintf_trunc() is removed, flag -Wno-format-truncation
is added to CFLAGS for veristat.o when compiled with gcc;

v1: https://lore.kernel.org/bpf/[email protected]/
v1 -> v2:
- a single cgroup, created at the beginning of execution, is now used for measurements (Andrii, Mykyta);
- cgroup namespace is not created, as it turned out to be useless (Andrii);
- veristat no longer mounts cgroup fs or changes subtree_control, instead it looks for an existing mount point and reports an error if memory.peak file can't be opened (Andrii, Alexei);
- if 'mem_peak' statistics is not enabled, veristat skips cgroup setup;
- code sharing with cgroup_helpers.c was considered but was decided against to simplify veristat github sync.

Eduard Zingerman (2):
bpf: include verifier memory allocations in memcg statistics
veristat: memory accounting for bpf programs

 kernel/bpf/btf.c                       |  15 +-
 kernel/bpf/verifier.c                  |  69 +++----
 tools/testing/selftests/bpf/Makefile   |   8 +
 tools/testing/selftests/bpf/veristat.c | 248 ++++++++++++++++++++++++-
 4 files changed, 292 insertions(+), 48 deletions(-)
Signature · 1 line
2.47.1
author maintainer
#1 lore raw

PATCH v3 1/2 bpf: include verifier memory allocations in memcg statistics

This commit adds __GFP_ACCOUNT flag to verifier induced memory allocations. The intent is to account for all allocations reachable from BPF_PROG_LOAD command, which is needed to track verifier memory consumption in veristat. This includes allocations done in verifier.c, and some allocations in btf.c, functions in log.c do not allocate.

There is also a utility function bpf_memcg_flags() which selectively adds GFP_ACCOUNT flag depending on the cgroup.memory=nobpf option. As far as I understand [1], the idea is to remove bpf_prog instances and maps from memcg accounting as these objects do not strictly belong to cgroup, hence it should not apply here.

(btf_parse_fields() is reachable from both program load and map
creation, but allocated record is not persistent as is freed as soon as map_check_btf() exits).

[1] https://lore.kernel.org/all/[email protected]/

---
 kernel/bpf/btf.c      | 15 +++++-----
 kernel/bpf/verifier.c | 69 ++++++++++++++++++++++---------------------
 2 files changed, 43 insertions(+), 41 deletions(-)
kernel/bpf/btf.c +8 -7
index 1d2cf898e21e..682acb1ed234 100644
@@ -3443,7 +3443,8 @@ btf_find_graph_root(const struct btf *btf, const struct btf_type *pt,
 	node_field_name = strstr(value_type, ":");
 	if (!node_field_name)
 		return -EINVAL;
-	value_type = kstrndup(value_type, node_field_name - value_type, GFP_KERNEL | __GFP_NOWARN);
+	value_type = kstrndup(value_type, node_field_name - value_type,
+			      GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
 	if (!value_type)
 		return -ENOMEM;
 	id = btf_find_by_name_kind(btf, value_type, BTF_KIND_STRUCT);
@@ -3958,7 +3959,7 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type
 	/* This needs to be kzalloc to zero out padding and unused fields, see
 	 * comment in btf_record_equal.
 	 */
-	rec = kzalloc(struct_size(rec, fields, cnt), GFP_KERNEL | __GFP_NOWARN);
+	rec = kzalloc(struct_size(rec, fields, cnt), GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
 	if (!rec)
 		return ERR_PTR(-ENOMEM);
 
@@ -9019,7 +9020,7 @@ static struct bpf_cand_cache *populate_cand_cache(struct bpf_cand_cache *cands,
 		bpf_free_cands_from_cache(*cc);
 		*cc = NULL;
 	}
-	new_cands = kmemdup(cands, sizeof_cands(cands->cnt), GFP_KERNEL);
+	new_cands = kmemdup(cands, sizeof_cands(cands->cnt), GFP_KERNEL_ACCOUNT);
 	if (!new_cands) {
 		bpf_free_cands(cands);
 		return ERR_PTR(-ENOMEM);
@@ -9027,7 +9028,7 @@ static struct bpf_cand_cache *populate_cand_cache(struct bpf_cand_cache *cands,
 	/* strdup the name, since it will stay in cache.
 	 * the cands->name points to strings in prog's BTF and the prog can be unloaded.
 	 */
-	new_cands->name = kmemdup_nul(cands->name, cands->name_len, GFP_KERNEL);
+	new_cands->name = kmemdup_nul(cands->name, cands->name_len, GFP_KERNEL_ACCOUNT);
 	bpf_free_cands(cands);
 	if (!new_cands->name) {
 		kfree(new_cands);
@@ -9111,7 +9112,7 @@ bpf_core_add_cands(struct bpf_cand_cache *cands, const struct btf *targ_btf,
 			continue;
 
 		/* most of the time there is only one candidate for a given kind+name pair */
-		new_cands = kmalloc(sizeof_cands(cands->cnt + 1), GFP_KERNEL);
+		new_cands = kmalloc(sizeof_cands(cands->cnt + 1), GFP_KERNEL_ACCOUNT);
 		if (!new_cands) {
 			bpf_free_cands(cands);
 			return ERR_PTR(-ENOMEM);
@@ -9228,7 +9229,7 @@ int bpf_core_apply(struct bpf_core_ctx *ctx, const struct bpf_core_relo *relo,
 	/* ~4k of temp memory necessary to convert LLVM spec like "0:1:0:5"
 	 * into arrays of btf_ids of struct fields and array indices.
 	 */
-	specs = kcalloc(3, sizeof(*specs), GFP_KERNEL);
+	specs = kcalloc(3, sizeof(*specs), GFP_KERNEL_ACCOUNT);
 	if (!specs)
 		return -ENOMEM;
 
@@ -9253,7 +9254,7 @@ int bpf_core_apply(struct bpf_core_ctx *ctx, const struct bpf_core_relo *relo,
 			goto out;
 		}
 		if (cc->cnt) {
-			cands.cands = kcalloc(cc->cnt, sizeof(*cands.cands), GFP_KERNEL);
+			cands.cands = kcalloc(cc->cnt, sizeof(*cands.cands), GFP_KERNEL_ACCOUNT);
 			if (!cands.cands) {
 				err = -ENOMEM;
 				goto out;
kernel/bpf/verifier.c +35 -34

250 lines not on this page · raw message

Signature · 1 line
2.47.1
author maintainer
#2 lore raw

PATCH v3 2/2 veristat: memory accounting for bpf programs

In reply to Eduard Zingerman

This commit adds a new field mem_peak / "Peak memory (MiB)" field to a set of gathered statistics. The field is intended as an estimate for peak verifier memory consumption for processing of a given program.

Mechanically stat is collected as follows:
- At the beginning of handle_verif_mode() a new cgroup is created and veristat process is moved into this cgroup.
- At each program load:
- bpf_object__load() is split into bpf_object__prepare() and bpf_object__load() to avoid accounting for memory allocated for maps;
- before bpf_object__load():
- a write to "memory.peak" file of the new cgroup is used to reset cgroup statistics;
- updated value is read from "memory.peak" file and stashed;
- after bpf_object__load() "memory.peak" is read again and difference between new and stashed values is used as a metric.

If any of the above steps fails veristat proceeds w/o collecting mem_peak information for a program, reporting mem_peak as -1.

While memcg provides data in bytes (converted from pages), veristat converts it to megabytes to avoid jitter when comparing results of different executions.

The change has no measurable impact on veristat running time.

A correlation between "Peak states" and "Peak memory" fields provides a sanity check for gathered statistics, e.g. a sample of data for sched_ext programs:

Program                   Peak states  Peak memory (MiB)
------------------------  -----------  -----------------
lavd_select_cpu                  2153                 44
lavd_enqueue                     1982                 41
lavd_dispatch                    3480                 28
layered_dispatch                 1417                 17
layered_enqueue                   760                 11
lavd_cpu_offline                  349                  6
lavd_cpu_online                   349                  6
lavd_init                         394                  6
rusty_init                        350                  5
layered_select_cpu                391                  4
...
rusty_stopping                    134                  1
arena_topology_node_init          170                  0
---
 tools/testing/selftests/bpf/Makefile   |   8 +
 tools/testing/selftests/bpf/veristat.c | 248 ++++++++++++++++++++++++-
 2 files changed, 249 insertions(+), 7 deletions(-)
tools/testing/selftests/bpf/Makefile +8 -0
index cf5ed3bee573..dd598ca771c5 100644
@@ -841,6 +841,14 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \
 	$(call msg,BINARY,,$@)
 	$(Q)$(CC) $(CFLAGS) $(LDFLAGS) $(filter %.a %.o,$^) $(LDLIBS) -o $@
 
+# This works around GCC warning about snprintf truncating strings like:
+#
+#   char a[PATH_MAX], b[PATH_MAX];
+#   snprintf(a, "%s/foo", b);      // triggers -Wformat-truncation
+ifeq ($(LLVM),)
+$(OUTPUT)/veristat.o: CFLAGS+=-Wno-format-truncation
+endif
+
 $(OUTPUT)/veristat.o: $(BPFOBJ)
 $(OUTPUT)/veristat: $(OUTPUT)/veristat.o
 	$(call msg,BINARY,,$@)
tools/testing/selftests/bpf/veristat.c +241 -7

350 lines not on this page · raw message

Signature · 1 line
2.47.1
#3 lore raw
Quoted 59 lines · Eduard Zingerman

This commit adds a new field mem_peak / "Peak memory (MiB)" field to a set of gathered statistics. The field is intended as an estimate for peak verifier memory consumption for processing of a given program.

Mechanically stat is collected as follows:
- At the beginning of handle_verif_mode() a new cgroup is created and veristat process is moved into this cgroup.
- At each program load:
- bpf_object__load() is split into bpf_object__prepare() and bpf_object__load() to avoid accounting for memory allocated for maps;
- before bpf_object__load():
- a write to "memory.peak" file of the new cgroup is used to reset cgroup statistics;
- updated value is read from "memory.peak" file and stashed;
- after bpf_object__load() "memory.peak" is read again and difference between new and stashed values is used as a metric.

If any of the above steps fails veristat proceeds w/o collecting mem_peak information for a program, reporting mem_peak as -1.

While memcg provides data in bytes (converted from pages), veristat converts it to megabytes to avoid jitter when comparing results of different executions.

The change has no measurable impact on veristat running time.

A correlation between "Peak states" and "Peak memory" fields provides a sanity check for gathered statistics, e.g. a sample of data for sched_ext programs:

Program                   Peak states  Peak memory (MiB)
------------------------  -----------  -----------------
lavd_select_cpu                  2153                 44
lavd_enqueue                     1982                 41
lavd_dispatch                    3480                 28
layered_dispatch                 1417                 17
layered_enqueue                   760                 11
lavd_cpu_offline                  349                  6
lavd_cpu_online                   349                  6
lavd_init                         394                  6
rusty_init                        350                  5
layered_select_cpu                391                  4
...
rusty_stopping                    134                  1
arena_topology_node_init          170                  0
---
 tools/testing/selftests/bpf/Makefile   |   8 +
 tools/testing/selftests/bpf/veristat.c | 248 ++++++++++++++++++++++++-
 2 files changed, 249 insertions(+), 7 deletions(-)
tools/testing/selftests/bpf/Makefile +0 -0
index cf5ed3bee573..dd598ca771c5 100644

tools/testing/selftests/bpf/Makefile $(OUTPUT)/bench: $(OUTPUT)/bench.o \ · line 849

Quoted 9 lines
tools/testing/selftests/bpf/Makefile +6 -0
@@ -841,6 +841,14 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \
        $(call msg,BINARY,,$@)
        $(Q)$(CC) $(CFLAGS) $(LDFLAGS) $(filter %.a %.o,$^) $(LDLIBS) -o $@
 
+# This works around GCC warning about snprintf truncating strings like:
+#
+#   char a[PATH_MAX], b[PATH_MAX];
+#   snprintf(a, "%s/foo", b);      // triggers -Wformat-truncation
+ifeq ($(LLVM),)
+$(OUTPUT)/veristat.o: CFLAGS+=-Wno-format-truncation

fixed formatting around +=, and dropped LLVM check, I think that should be handled by -Wno-unused-command-line-argument we set earlier (make LLVM=1 veristat succeeded without any warnings for me, FWIW)

applied to bpf-next, thanks

Quoted 5 lines
tools/testing/selftests/bpf/Makefile +2 -0
+endif
+
 $(OUTPUT)/veristat.o: $(BPFOBJ)
 $(OUTPUT)/veristat: $(OUTPUT)/veristat.o
        $(call msg,BINARY,,$@)

[...]

tools/testing/selftests/bpf/Makefile

Quoted 8 lines
tools/testing/selftests/bpf/Makefile +8 -0
+/*
+ * Creates a cgroup at /sys/fs/cgroup/veristat-accounting-<pid>,
+ * moves current process to this cgroup.
+ */
+static void create_stat_cgroup(void)
+{
+       char cgroup_fs_mount[PATH_MAX];
+       char buf[PATH_MAX];

I hard-coded these to 4096 given we hard-coded 4095 below.

[...]

#4 lore raw
In reply to Eduard Zingerman
This series was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:

Hello:

This series was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:

Quoted 9 lines · On Fri, 13 Jun 2025 00:21:45 -0700 you wrote:

When working on the verifier, it is sometimes interesting to know how a particular change affects memory consumption. This patch-set modifies veristat to provide such information. As a collateral, kernel needs an update to make allocations reachable from BPF program load accountable in memcg statistics.

Here is a sample output:

[...]

Here is the summary with links:
- [bpf-next,v3,1/2] bpf: include verifier memory allocations in memcg statistics

    https://git.kernel.org/bpf/bpf-next/c/43736ec3e027

- [bpf-next,v3,2/2] veristat: memory accounting for bpf programs (no matching commit)

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html