Skip to contentSkip to conversation

bpftool: Add recursive map dumping

PATCH Superseded Tianyi Chen 3 participants 7 messages Sep 11

Subsystems (4)

v5 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v5 cover bpftool: Add recursive map dumping

Add -r/--recursive to dump referenced inner maps after the selected roots, with deduplication and bounded descriptor use. The second patch checks output formats and operation under a low descriptor limit.

Changes in v5:
- Name the child-process exit codes for descriptor cleanup, limit setup, command failure, entry counts, map IDs and JSON framing. Preserve the existing values and failure paths.

The implementation is unchanged. Keep the explicit expected JSON documents in the tests so each case's output structure is visible where it is checked; the suggested formatter helper is not needed for this revision.

v4: https://lore.kernel.org/r/[email protected]
Review: https://lore.kernel.org/r/43836e91c05ebc17a327d26c28c018d8e09ed2dcaca6fd02b562e37a17a55dd4@mail.kernel.org

Validation:
- Rebuilt bpftool, its manual pages and the focused selftest runner.
- All 12 bpftool_map_dump subtests passed. The three revised series were tested together: 52 subtests passed with no skips or failures in an x86-64 KVM guest running Linux 7.3.0-rc2 from bpf/master, with LLVM 20-built BPF test objects.
- Bash syntax, bpftool synchronization checks and diff checks passed.
- The generated patches apply cleanly to current bpf-next.

This was a focused run. Unrelated selftests requiring unavailable kernel features were excluded from the build with PERMISSIVE=1.

Tianyi Chen (2):
bpftool: Add recursive map dumping selftests/bpf: Cover recursive bpftool map dumps

 .../bpf/bpftool/Documentation/bpftool-map.rst |  22 +-
 tools/bpf/bpftool/bash-completion/bpftool     |   2 +-
 tools/bpf/bpftool/main.c                      |   7 +-
 tools/bpf/bpftool/main.h                      |   1 +
 tools/bpf/bpftool/map.c                       | 131 ++++-
 .../bpf/prog_tests/bpftool_map_dump.c         | 474 ++++++++++++++++++
 6 files changed, 626 insertions(+), 11 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/bpftool_map_dump.c

base-commit: af0b84a9215d951d16f26b7ee34353b970cf5d4e

Signature · 1 line
2.55.0
#1 lore raw

PATCH v5 1/2 bpftool: Add recursive map dumping

From: Tianyi Chen <[email protected]>

Dumping a map-of-maps currently shows inner map IDs without their contents. Add -r/--recursive to dump referenced inner maps as well, leaving the default output unchanged. Show the selected maps followed by distinct discovered inner maps, preserving plain and BTF formatting and using an array of map objects for JSON output.

Holding every discovered inner-map FD open would make descriptor use grow with the number of maps and could exhaust RLIMIT_NOFILE. Keep the selected map FDs open, queue distinct inner map IDs, and open, dump and close each queued map in turn. This needs only one additional map FD. The tradeoff is deferred ID resolution: concurrently removed inner maps can disappear before they are opened, so the dump is not atomic.

Report failure when an inner map cannot be opened, retain per-entry lookup errors, and close JSON containers before returning.

---
 .../bpf/bpftool/Documentation/bpftool-map.rst |  22 ++-
 tools/bpf/bpftool/bash-completion/bpftool     |   2 +-
 tools/bpf/bpftool/main.c                      |   7 +-
 tools/bpf/bpftool/main.h                      |   1 +
 tools/bpf/bpftool/map.c                       | 131 ++++++++++++++++--
 5 files changed, 152 insertions(+), 11 deletions(-)
tools/bpf/bpftool/Documentation/bpftool-map.rst +21 -1
index 5daf3de5c74..23faa672685 100644
@@ -16,7 +16,8 @@ SYNOPSIS
 
 **bpftool** [*OPTIONS*] **map** *COMMAND*
 
-*OPTIONS* := { |COMMON_OPTIONS| | { **-f** | **--bpffs** } | { **-n** | **--nomount** } }
+*OPTIONS* := { |COMMON_OPTIONS| | { **-f** | **--bpffs** } |
+{ **-n** | **--nomount** } | { **-r** | **--recursive** } }
 
 *COMMANDS* :=
 { **show** | **list** | **create** | **dump** | **update** | **lookup** | **getnext** |
@@ -170,6 +171,25 @@ OPTIONS
     Do not automatically attempt to mount any virtual file system (such as
     tracefs or BPF virtual file system) when necessary.
 
+-r, --recursive
+    Also dump the inner maps referenced by **array_of_maps** and **hash_of_maps**
+    entries when running **map dump**. Each map ID is visited once, even if
+    several entries refer to it. Selected maps are followed by their inner maps.
+
+    Plain output includes a header identifying each map. On success, JSON output
+    is always an array of map objects, each containing an **id** and an
+    **elements** array, including when only one map is dumped. Outer map entries
+    retain their **inner_map_id** field, which identifies the corresponding inner
+    map object.
+
+    Inner map IDs are resolved when the maps are visited. The dump is not an
+    atomic snapshot: concurrent updates can change map contents or remove a
+    referenced inner map before it is visited. Failure to open a referenced
+    inner map stops the dump and returns a nonzero exit status. Output may
+    contain maps or entries printed before the error.
+
+    In JSON mode, the enclosing arrays and objects are closed before returning.
+
 EXAMPLES
 ========
 **# bpftool map show**
tools/bpf/bpftool/bash-completion/bpftool +1 -1
index 75cbcb512eb..45c336d63be 100644
@@ -261,7 +261,7 @@ _bpftool()
 
     # Deal with options
     if [[ ${words[cword]} == -* ]]; then
-        local c='--version --json --pretty --bpffs --mapcompat --debug \
+        local c='--version --json --pretty --recursive --bpffs --mapcompat --debug \
             --use-loader --base-btf --sign -i -k'
         COMPREPLY=( $( compgen -W "$c" -- "$cur" ) )
         return 0
tools/bpf/bpftool/main.c +6 -1
index 5ababd8f7d0..c0f4df10102 100644
@@ -26,6 +26,7 @@ static int (*last_do_help)(int argc, char **argv);
 json_writer_t *json_wtr;
 bool pretty_output;
 bool json_output;
+bool recursive_dump;
 bool show_pinned;
 bool block_mount;
 bool verifier_logs;
@@ -456,6 +457,7 @@ int main(int argc, char **argv)
 		{ "json",	no_argument,	NULL,	'j' },
 		{ "help",	no_argument,	NULL,	'h' },
 		{ "pretty",	no_argument,	NULL,	'p' },
+		{ "recursive",	no_argument,	NULL,	'r' },
 		{ "version",	no_argument,	NULL,	'V' },
 		{ "bpffs",	no_argument,	NULL,	'f' },
 		{ "mapcompat",	no_argument,	NULL,	'm' },
@@ -479,7 +481,7 @@ int main(int argc, char **argv)
 	bin_name = "bpftool";
 
 	opterr = 0;
-	while ((opt = getopt_long(argc, argv, "VhpjfLmndSi:k:B:l",
+	while ((opt = getopt_long(argc, argv, "VhpjrfLmndSi:k:B:l",
 				  options, NULL)) >= 0) {
 		switch (opt) {
 		case 'V':
@@ -501,6 +503,9 @@ int main(int argc, char **argv)
 			}
 			jsonw_pretty(json_wtr, pretty_output);
 			break;
+		case 'r':
+			recursive_dump = true;
+			break;
 		case 'f':
 			show_pinned = true;
 			break;
tools/bpf/bpftool/main.h +1 -0
index 9315a1db1f7..43ef0aa592d 100644
@@ -83,6 +83,7 @@ extern const char *bin_name;
 
 extern json_writer_t *json_wtr;
 extern bool json_output;
+extern bool recursive_dump;
 extern bool show_pinned;
 extern bool show_pids;
 extern bool block_mount;
tools/bpf/bpftool/map.c +123 -8

220 lines not on this page · raw message

Signature · 1 line
2.55.0
#2 lore raw
This isn't a bug, but since libbpf_internal.h is already being pulled in here, would libbpf_ensure_mem() (with a cap field in struct map_dump_ctx) be a better fit than reallocating by one element…
Quoted 4 lines
tools/bpf/bpftool/map.c +0 -0
index 684a8fb72414..c0505f504967 100644

[ ... ]

Quoted 15 lines
tools/bpf/bpftool/map.c +11 -0
@@ -826,12 +827,44 @@ static void free_map_kv_btf(struct btf *btf)
 		btf__free(btf);
 }
 
+struct map_dump_ctx {
+	struct hashmap *seen;
+	__u32 *pending_ids;
+	size_t pending_cnt;
+};
+
+static int collect_inner_map(struct map_dump_ctx *ctx, __u32 id)
+{
+	__u32 *ids;
+	int err;
+

tools/bpf/bpftool/map.c static void free_map_kv_btf(struct btf *btf) · line 850

Quoted 10 lines
tools/bpf/bpftool/map.c +10 -0
+	if (hashmap__find(ctx->seen, id, NULL))
+		return 0;
+
+	ids = libbpf_reallocarray(ctx->pending_ids, ctx->pending_cnt + 1,
+				  sizeof(*ids));
+	if (!ids) {
+		p_err("mem alloc failed");
+		return -1;
+	}
+	ctx->pending_ids = ids;

This isn't a bug, but since libbpf_internal.h is already being pulled in here, would libbpf_ensure_mem() (with a cap field in struct map_dump_ctx) be a better fit than reallocating by one element per discovered inner map?

The pending-id array grows by exactly one element for every newly discovered inner map, so a map-of-maps with N distinct inner maps performs N reallocations. libbpf_ensure_mem() is designed for this pattern and grows the buffer geometrically, keeping the growth amortised.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34560347455

#3 lore raw

PATCH v5 2/2 selftests/bpf: Cover recursive bpftool map dumps

In reply to Tianyi Chen

From: Tianyi Chen <[email protected]>

Exercise recursive map dumping for array-of-maps and hash-of-maps, including shared inner maps, empty outer and inner maps, BTF-formatted values and multiple selected roots.

Check complete JSON documents against the existing nonrecursive entry representations, and check plain headers and typed values. Verify both short and long options, root ordering, deduplication and unchanged default output.

Cover ordinary and nested perf event arrays to ensure unsupported element lookups retain their error markers without failing the dump. Dump 64 distinct inner maps under RLIMIT_NOFILE=32 in both JSON and plain modes to catch holding every discovered inner-map FD open.

Open 32 extra descriptors before forking, close inherited descriptors in the child before imposing the limit, and verify the parent's file descriptors and outer map remain usable.

Assisted-by LLM
---
 .../bpf/prog_tests/bpftool_map_dump.c         | 474 ++++++++++++++++++
 1 file changed, 474 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/bpftool_map_dump.c
tools/testing/selftests/bpf/prog_tests/bpftool_map_dump.c new file +474 -0
new file mode 100644
index 00000000000..1fb877fc68c
@@ -0,0 +1,474 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <test_progs.h>
+#include <bpftool_helpers.h>
+#include <bpf/btf.h>
+#include <sys/resource.h>
+#include <dirent.h>
+
+#define OUTPUT_SIZE 8192
+
+static bool dump_map(__u32 id, const char *options, char *output)
+{
+	char command[MAX_BPFTOOL_CMD_LEN];
+
+	snprintf(command, sizeof(command), "%s map dump id %u", options, id);
+	memset(output, 0, OUTPUT_SIZE);
+	if (!ASSERT_OK(get_bpftool_command_output(command, output, OUTPUT_SIZE - 1),
+		       "dump_map"))
+		return false;
+	/* The helper leaves the trailing newline in place. */
+	output[strcspn(output, "\n")] = '\0';
+	return true;
+}
+
+static __u32 map_id(int fd)
+{
+	struct bpf_map_info info = {};
+	__u32 len = sizeof(info);
+
+	if (!ASSERT_OK(bpf_map_get_info_by_fd(fd, &info, &len), "map_info"))
+		return 0;
+	return info.id;
+}
+
+static int count_token(const char *output, const char *token)
+{
+	int count = 0;
+
+	while ((output = strstr(output, token))) {
+		count++;
+		output += strlen(token);
+	}
+	return count;
+}
+
+static void check_plain(__u32 root_id, __u32 inner_id, const char *type,
+			int entries, bool typed)
+{
+	char command[MAX_BPFTOOL_CMD_LEN], header[128];
+	char output[OUTPUT_SIZE] = {};
+	const char *root, *inner;
+
+	snprintf(command, sizeof(command), "--recursive map dump id %u", root_id);
+	if (!ASSERT_OK(get_bpftool_command_output(command, output, sizeof(output) - 1),
+		       "plain_dump"))
+		return;
+	snprintf(header, sizeof(header), "%u: %s  name dump_outer  ", root_id, type);
+	root = strstr(output, header);
+	if (!ASSERT_OK_PTR(root, "plain_root_header"))
+		return;
+	ASSERT_EQ(root - output, 0, "plain_root_first");
+	ASSERT_EQ(count_token(output, "inner_map_id:"), entries, "plain_references");
+	if (entries) {
+		snprintf(header, sizeof(header), "%u: hash  name dump_inner  ", inner_id);
+		inner = strstr(output, header);
+		if (ASSERT_OK_PTR(inner, "plain_inner_header"))
+			ASSERT_GT(inner - root, 0, "plain_inner_after_root");
+		ASSERT_EQ(count_token(output, header), 1, "plain_inner_once");
+	}
+	ASSERT_EQ(count_token(output, "Found "), entries && !typed ? 2 : 1,
+		  "plain_map_count");
+	if (typed) {
+		ASSERT_HAS_SUBSTR(output, "\"key\": 0", "plain_btf_key");
+		ASSERT_HAS_SUBSTR(output, "\"value\": 16843009", "plain_btf_value");
+	}
+}
+
+static void test_outer(enum bpf_map_type type, int entries, bool empty_inner,
+		       bool typed)
+{
+	LIBBPF_OPTS(bpf_map_create_opts, opts);
+	LIBBPF_OPTS(bpf_map_create_opts, inner_opts);
+	struct btf *btf = NULL;
+	char outer[OUTPUT_SIZE], inner[OUTPUT_SIZE], output[OUTPUT_SIZE];
+	char expected[OUTPUT_SIZE * 3], reference[64];
+	const char *type_name = libbpf_bpf_map_type_str(type);
+	int inner_fd = -1, outer_fd = -1;
+	__u32 root_id, inner_id, key, value = 0x01010101;
+
+	if (typed) {
+		btf = btf__new_empty();
+		if (!ASSERT_OK_PTR(btf, "create_btf") ||
+		    !ASSERT_EQ(btf__add_int(btf, "unsigned int", 4, 0), 1, "btf_int") ||
+		    !ASSERT_OK(btf__load_into_kernel(btf), "load_btf"))
+			goto out;
+		inner_opts.btf_fd = btf__fd(btf);
+		inner_opts.btf_key_type_id = 1;
+		inner_opts.btf_value_type_id = 1;
+	}
+	inner_fd = bpf_map_create(BPF_MAP_TYPE_HASH, "dump_inner", sizeof(key),
+				  sizeof(value), 2, &inner_opts);
+	if (!ASSERT_OK_FD(inner_fd, "create_inner"))
+		goto out;
+	key = 0;
+	if (!empty_inner &&
+	    !ASSERT_OK(bpf_map_update_elem(inner_fd, &key, &value, BPF_ANY),
+		       "populate_inner"))
+		goto out;
+	opts.inner_map_fd = inner_fd;
+	outer_fd = bpf_map_create(type, "dump_outer", sizeof(key), sizeof(__u32),
+				  3, &opts);
+	if (!ASSERT_OK_FD(outer_fd, "create_outer"))
+		goto out;
+	/* The unused third array slot also exercises failed lookups. */
+	for (key = 0; key < entries; key++)
+		if (!ASSERT_OK(bpf_map_update_elem(outer_fd, &key, &inner_fd, BPF_ANY),
+			       "populate_outer"))
+			goto out;
+	root_id = map_id(outer_fd);
+	inner_id = map_id(inner_fd);
+	if (!root_id || !inner_id || !dump_map(root_id, "-j", outer) ||
+	    !dump_map(inner_id, "-j", inner))
+		goto out;
+
+	ASSERT_EQ(outer[0], '[', "default_array");
+	ASSERT_EQ(count_token(outer, "\"elements\":"), 0, "default_no_wrapper");
+	ASSERT_EQ(count_token(outer, "\"id\":"), 0, "default_no_header");
+	snprintf(reference, sizeof(reference), "\"inner_map_id\":%u", inner_id);
+	ASSERT_EQ(count_token(outer, reference), entries, "default_references");
+	if (!entries)
+		ASSERT_STREQ(outer, "[]", "empty_outer_default");
+	if (empty_inner)
+		ASSERT_STREQ(inner, "[]", "empty_inner_default");
+	else if (typed)
+		ASSERT_HAS_SUBSTR(inner, "\"formatted\":{\"key\":0,\"value\":16843009}",
+				  "typed_inner");
+	else
+		ASSERT_STREQ(inner,
+			     "[{\"key\":[\"0x00\",\"0x00\",\"0x00\",\"0x00\"],"
+			     "\"value\":[\"0x01\",\"0x01\",\"0x01\",\"0x01\"]}]",
+			     "ordinary_default");
+
+	/*
+	 * Compare the complete JSON document: a flat array with the root first,
+	 * one copy of the shared inner map, and unchanged entry representations.
+	 */
+	if (entries)
+		snprintf(expected, sizeof(expected),
+			 "[{\"id\":%u,\"type\":\"%s\",\"name\":\"dump_outer\","
+			 "\"flags\":0,\"elements\":%s},{\"id\":%u,\"type\":\"hash\","
+			 "\"name\":\"dump_inner\",\"flags\":0,\"elements\":%s}]",
+			 root_id, type_name, outer, inner_id, inner);
+	else
+		snprintf(expected, sizeof(expected),
+			 "[{\"id\":%u,\"type\":\"%s\",\"name\":\"dump_outer\","
+			 "\"flags\":0,\"elements\":[]}]", root_id, type_name);
+	if (dump_map(root_id, "-j -r", output))
+		ASSERT_STREQ(output, expected, "recursive_json");
+	if (dump_map(root_id, "--json --recursive", output))
+		ASSERT_STREQ(output, expected, "recursive_long_options");
+	check_plain(root_id, inner_id, type_name, entries, typed);
+
+	/* Recursion on an ordinary map still emits a single map object. */
+	snprintf(expected, sizeof(expected),
+		 "[{\"id\":%u,\"type\":\"hash\",\"name\":\"dump_inner\","
+		 "\"flags\":0,\"elements\":%s}]", inner_id, inner);
+	if (dump_map(inner_id, "-j -r", output))
+		ASSERT_STREQ(output, expected, "ordinary_recursive");
+out:
+	if (outer_fd >= 0)
+		close(outer_fd);
+	if (inner_fd >= 0)
+		close(inner_fd);
+	btf__free(btf);
+}
+
+static void test_multiple_roots(void)
+{
+	LIBBPF_OPTS(bpf_map_create_opts, opts);
+	char command[MAX_BPFTOOL_CMD_LEN], name[BPF_OBJ_NAME_LEN];
+	char output[OUTPUT_SIZE] = {}, expected[OUTPUT_SIZE * 4], elements[OUTPUT_SIZE];
+	static const char * const types[] = { "hash", "array_of_maps", "hash_of_maps", "hash" };
+	int fds[] = { -1, -1, -1, -1 };
+	__u32 ids[4], key;
+	int i, len = 0;
+
+	/*
+	 * Select the first inner map and both outers as roots. The other inner
+	 * map must be appended after all three roots, even though it is found
+	 * while dumping the first outer. A process-specific name avoids other
+	 * tests' maps joining the selection.
+	 */
+	snprintf(name, sizeof(name), "dump_%u", getpid());
+	fds[0] = bpf_map_create(BPF_MAP_TYPE_HASH, name, 4, 4, 1, NULL);
+	if (!ASSERT_OK_FD(fds[0], "create_selected_inner"))
+		goto out;
+	fds[3] = bpf_map_create(BPF_MAP_TYPE_HASH, "dump_discovered", 4, 4, 1, NULL);
+	if (!ASSERT_OK_FD(fds[3], "create_discovered_inner"))
+		goto out;
+	opts.inner_map_fd = fds[0];
+	fds[1] = bpf_map_create(BPF_MAP_TYPE_ARRAY_OF_MAPS, name, 4, 4, 2, &opts);
+	if (!ASSERT_OK_FD(fds[1], "create_array_root"))
+		goto out;
+	fds[2] = bpf_map_create(BPF_MAP_TYPE_HASH_OF_MAPS, name, 4, 4, 2, &opts);
+	if (!ASSERT_OK_FD(fds[2], "create_hash_root"))
+		goto out;
+	for (i = 1; i <= 2; i++) {
+		key = 0;
+		if (!ASSERT_OK(bpf_map_update_elem(fds[i], &key, &fds[0], BPF_ANY),
+			       "reference_selected_inner"))
+			goto out;
+		key = 1;
+		if (!ASSERT_OK(bpf_map_update_elem(fds[i], &key, &fds[3], BPF_ANY),
+			       "reference_discovered_inner"))
+			goto out;
+	}
+	for (i = 0; i < ARRAY_SIZE(fds); i++) {
+		ids[i] = map_id(fds[i]);
+		if (!ids[i] || !dump_map(ids[i], "-j", elements))
+			goto out;
+		len += snprintf(expected + len, sizeof(expected) - len,
+				"%s{\"id\":%u,\"type\":\"%s\",\"name\":\"%s\","
+				"\"flags\":0,\"elements\":%s}%s",
+				i ? "," : "[", ids[i], types[i],
+				i == 3 ? "dump_discovered" : name, elements, i == 3 ? "]" : "");
+	}
+	snprintf(command, sizeof(command), "-j -r map dump name %s", name);
+	if (ASSERT_OK(get_bpftool_command_output(command, output, sizeof(output) - 1),
+		      "dump_multiple_roots")) {
+		output[strcspn(output, "\n")] = '\0';
+		ASSERT_STREQ(output, expected, "roots_first_and_seed_dedup");
+	}
+out:
+	for (i = 0; i < ARRAY_SIZE(fds); i++)
+		if (fds[i] >= 0)
+			close(fds[i]);
+}
+
+static void test_unreadable(bool outer)
+{
+	LIBBPF_OPTS(bpf_map_create_opts, opts);
+	char elements[OUTPUT_SIZE], output[OUTPUT_SIZE], root[OUTPUT_SIZE];
+	char expected[OUTPUT_SIZE * 3], plain[OUTPUT_SIZE] = {};
+	char command[MAX_BPFTOOL_CMD_LEN];
+	int inner_fd = -1, outer_fd = -1, lookup_errno;
+	__u32 inner_id, root_id, key = 0, value;
+
+	/*
+	 * Every key is enumerable, but PERF_EVENT_ARRAY lookup returns
+	 * ENOTSUPP (the kernel-internal errno). Check both entries so an
+	 * early exit on the first lookup failure cannot pass.
+	 */
+	inner_fd = bpf_map_create(BPF_MAP_TYPE_PERF_EVENT_ARRAY, "dump_unreadable",
+				  sizeof(key), sizeof(value), 2, NULL);
+	if (!ASSERT_OK_FD(inner_fd, "create_unreadable"))
+		goto out;
+	if (!ASSERT_LT(bpf_map_lookup_elem(inner_fd, &key, &value), 0,
+		       "unreadable_lookup"))
+		goto out;
+	lookup_errno = errno;
+	if (!ASSERT_NEQ(lookup_errno, ENOENT, "unreadable_not_missing"))
+		goto out;
+	inner_id = map_id(inner_fd);
+	if (!inner_id || !dump_map(inner_id, "-j", elements))
+		goto out;
+	ASSERT_EQ(count_token(elements, "\"error\":"), 2, "default_json_errors");
+	snprintf(command, sizeof(command), "map dump id %u", inner_id);
+	if (!ASSERT_OK(get_bpftool_command_output(command, plain, sizeof(plain) - 1),
+		       "default_plain_unreadable"))
+		goto out;
+	ASSERT_EQ(count_token(plain, strerror(lookup_errno)), 2, "default_plain_errors");
+	ASSERT_HAS_SUBSTR(plain, "Found 0 elements", "default_plain_count");
+
+	root_id = inner_id;
+	if (outer) {
+		opts.inner_map_fd = inner_fd;
+		outer_fd = bpf_map_create(BPF_MAP_TYPE_ARRAY_OF_MAPS, "dump_outer",
+					  sizeof(key), sizeof(value), 1, &opts);
+		if (!ASSERT_OK_FD(outer_fd, "create_outer") ||
+		    !ASSERT_OK(bpf_map_update_elem(outer_fd, &key, &inner_fd, BPF_ANY),
+			       "populate_outer"))
+			goto out;
+		root_id = map_id(outer_fd);
+		if (!root_id || !dump_map(root_id, "-j", root))
+			goto out;
+		snprintf(expected, sizeof(expected),
+			 "[{\"id\":%u,\"type\":\"array_of_maps\",\"name\":\"dump_outer\","
+			 "\"flags\":0,\"elements\":%s},{\"id\":%u,"
+			 "\"type\":\"perf_event_array\",\"name\":\"dump_unreadable\","
+			 "\"flags\":0,\"elements\":%s}]", root_id, root, inner_id, elements);
+	} else {
+		snprintf(expected, sizeof(expected),
+			 "[{\"id\":%u,\"type\":\"perf_event_array\","
+			 "\"name\":\"dump_unreadable\",\"flags\":0,\"elements\":%s}]",
+			 inner_id, elements);
+	}
+	if (dump_map(root_id, "-j -r", output))
+		ASSERT_STREQ(output, expected, "recursive_unreadable_json");
+	memset(output, 0, sizeof(output));
+	snprintf(command, sizeof(command), "-r map dump id %u", root_id);
+	if (ASSERT_OK(get_bpftool_command_output(command, output, sizeof(output) - 1),
+		      "recursive_unreadable_plain")) {
+		ASSERT_HAS_SUBSTR(output, plain, "recursive_plain_preserves_errors");
+		ASSERT_EQ(count_token(output, strerror(lookup_errno)), 2,
+			  "recursive_plain_errors");
+		ASSERT_EQ(count_token(output, "Found "), outer ? 2 : 1,
+			  "recursive_plain_maps");
+	}
+out:
+	if (outer_fd >= 0)
+		close(outer_fd);
+	if (inner_fd >= 0)
+		close(inner_fd);
+}
+
+static void test_many_inner_maps(bool json)
+{
+	enum {
+		DUMP_OK,
+		DUMP_ERR_RLIMIT,
+		DUMP_ERR_COMMAND,
+		DUMP_ERR_COUNTS,
+		DUMP_ERR_IDS,
+		DUMP_ERR_JSON,
+		DUMP_ERR_FDS,
+	};
+	LIBBPF_OPTS(bpf_map_create_opts, opts);
+	const struct rlimit limit = { .rlim_cur = 32, .rlim_max = 32 };
+	char command[MAX_BPFTOOL_CMD_LEN], token[64];
+	__u32 ids[64], root_id, key;
+	int inner_fd = -1, outer_fd = -1, status;
+	int inherited_fds[32], nr_inherited = 0, i;
+	char *output = NULL;
+	pid_t pid;
+
+	inner_fd = bpf_map_create(BPF_MAP_TYPE_HASH, "dump_inner", 4, 4, 1, NULL);
+	if (!ASSERT_OK_FD(inner_fd, "create_template"))
+		goto out;
+	opts.inner_map_fd = inner_fd;
+	outer_fd = bpf_map_create(BPF_MAP_TYPE_ARRAY_OF_MAPS, "dump_outer", 4, 4,
+				  ARRAY_SIZE(ids), &opts);
+	close(inner_fd);
+	inner_fd = -1;
+	if (!ASSERT_OK_FD(outer_fd, "create_outer"))
+		goto out;
+	for (key = 0; key < ARRAY_SIZE(ids); key++) {
+		inner_fd = bpf_map_create(BPF_MAP_TYPE_HASH, "dump_inner", 4, 4, 1, NULL);
+		if (!ASSERT_OK_FD(inner_fd, "create_inner") ||
+		    !ASSERT_OK(bpf_map_update_elem(outer_fd, &key, &inner_fd, BPF_ANY),
+			       "populate_outer"))
+			goto out;
+		ids[key] = map_id(inner_fd);
+		if (!ids[key])
+			goto out;
+		/* The outer map keeps each distinct inner map alive. */
+		close(inner_fd);
+		inner_fd = -1;
+	}
+	root_id = map_id(outer_fd);
+	output = calloc(1, 65536);
+	if (!root_id || !ASSERT_OK_PTR(output, "allocate_output"))
+		goto out;
+
+	/* Fill the low FD slots to exercise inherited descriptor cleanup. */
+	for (i = 0; i < ARRAY_SIZE(inherited_fds); i++) {
+		int fd = open("/dev/null", O_RDONLY);
+
+		if (!ASSERT_OK_FD(fd, "open_inherited_fd"))
+			goto out;
+		inherited_fds[nr_inherited++] = fd;
+	}
+
+	/*
+	 * Create all fixtures before lowering the limit, and keep the test
+	 * runner's limit unchanged. Retaining every discovered FD would exceed
+	 * this limit before the recursive dump could visit all inner maps.
+	 */
+	pid = fork();
+	if (!ASSERT_GE(pid, 0, "fork"))
+		goto out;
+	if (!pid) {
+		struct dirent *entry;
+		DIR *dir;
+
+		/* Reserve a slot for the directory even if the parent is full. */
+		close(inherited_fds[nr_inherited - 1]);
+		dir = opendir("/proc/self/fd");
+		if (!dir)
+			_exit(DUMP_ERR_FDS);
+		/* The parent keeps the outer map and its inner maps alive. */
+		for (;;) {
+			char *end;
+			long fd;
+
+			errno = 0;
+			entry = readdir(dir);
+			if (!entry) {
+				if (errno)
+					_exit(DUMP_ERR_FDS);
+				break;
+			}
+			fd = strtol(entry->d_name, &end, 10);
+			if (*end || fd < 3 || fd == dirfd(dir))
+				continue;
+			close(fd);
+		}
+		if (closedir(dir))
+			_exit(DUMP_ERR_FDS);
+		if (setrlimit(RLIMIT_NOFILE, &limit))
+			_exit(DUMP_ERR_RLIMIT);
+		snprintf(command, sizeof(command), "%s -r map dump id %u",
+			 json ? "-j" : "", root_id);
+		if (get_bpftool_command_output(command, output, 65535))
+			_exit(DUMP_ERR_COMMAND);
+		if (count_token(output, json ? "\"id\":" : "Found ") != ARRAY_SIZE(ids) + 1 ||
+		    count_token(output, json ? "\"inner_map_id\":" : "inner_map_id:") !=
+		    ARRAY_SIZE(ids))
+			_exit(DUMP_ERR_COUNTS);
+		for (key = 0; key < ARRAY_SIZE(ids); key++) {
+			if (json)
+				snprintf(token, sizeof(token), "\"id\":%u,", ids[key]);
+			else
+				snprintf(token, sizeof(token), "\n%u: hash  name dump_inner  ",
+					 ids[key]);
+			if (count_token(output, token) != 1)
+				_exit(DUMP_ERR_IDS);
+		}
+		if (json && (output[0] != '[' ||
+			     strcmp(output + strlen(output) - 2, "]\n")))
+			_exit(DUMP_ERR_JSON);
+		_exit(DUMP_OK);
+	}
+	if (ASSERT_EQ(waitpid(pid, &status, 0), pid, "waitpid") &&
+	    ASSERT_TRUE(WIFEXITED(status), "child_exited"))
+		ASSERT_EQ(WEXITSTATUS(status), DUMP_OK, "dump_with_low_fd_limit");
+	for (i = 0; i < nr_inherited; i++)
+		ASSERT_GE(fcntl(inherited_fds[i], F_GETFD), 0, "parent_fd_preserved");
+	ASSERT_EQ(map_id(outer_fd), root_id, "parent_outer_preserved");
+out:
+	while (nr_inherited)
+		close(inherited_fds[--nr_inherited]);
+	free(output);
+	if (outer_fd >= 0)
+		close(outer_fd);
+	if (inner_fd >= 0)
+		close(inner_fd);
+}
+
+void test_bpftool_map_dump(void)
+{
+	if (test__start_subtest("unreadable_ordinary"))
+		test_unreadable(false);
+	if (test__start_subtest("unreadable_inner"))
+		test_unreadable(true);
+	if (test__start_subtest("many_inner_maps_json"))
+		test_many_inner_maps(true);
+	if (test__start_subtest("many_inner_maps_plain"))
+		test_many_inner_maps(false);
+	if (test__start_subtest("multiple_roots"))
+		test_multiple_roots();
+	if (test__start_subtest("array_of_maps"))
+		test_outer(BPF_MAP_TYPE_ARRAY_OF_MAPS, 1, false, false);
+	if (test__start_subtest("hash_of_maps"))
+		test_outer(BPF_MAP_TYPE_HASH_OF_MAPS, 1, false, false);
+	if (test__start_subtest("shared_inner"))
+		test_outer(BPF_MAP_TYPE_ARRAY_OF_MAPS, 2, false, false);
+	if (test__start_subtest("empty_array_of_maps"))
+		test_outer(BPF_MAP_TYPE_ARRAY_OF_MAPS, 0, false, false);
+	if (test__start_subtest("empty_hash_of_maps"))
+		test_outer(BPF_MAP_TYPE_HASH_OF_MAPS, 0, false, false);
+	if (test__start_subtest("btf_inner"))
+		test_outer(BPF_MAP_TYPE_HASH_OF_MAPS, 1, false, true);
+	if (test__start_subtest("empty_inner"))
+		test_outer(BPF_MAP_TYPE_HASH_OF_MAPS, 1, true, false);
+}
Signature · 1 line
2.55.0
#4 lore raw

Re: [PATCH bpf-next v5 2/2] selftests/bpf: Cover recursive bpftool map dumps

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Dangerous accumulation of `snprintf` return values without bounds checking can lead to a stack…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Dangerous accumulation of snprintf return values without bounds checking can lead to a stack buffer overflow. --

commit 9bcae338c80433d98472777c175bac68e5e0d8b5
Author: Tianyi Chen <[email protected]>

selftests/bpf: Cover recursive bpftool map dumps

This commit adds selftests to exercise recursive map dumping for array-of-maps and hash-of-maps. It verifies complete JSON documents against nonrecursive entry representations, checks both short and long options, and tests file descriptor retention under limits.

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/bpftool_map_dump.c new file +0 -0
new file mode 100644

[ ... ]

Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/bpftool_map_dump.c new file +5 -0
+static void test_multiple_roots(void)
+{
+	LIBBPF_OPTS(bpf_map_create_opts, opts);
+	char command[MAX_BPFTOOL_CMD_LEN], name[BPF_OBJ_NAME_LEN];
+	char output[OUTPUT_SIZE] = {}, expected[OUTPUT_SIZE * 4], elements[OUTPUT_SIZE];

[ ... ]

tools/testing/selftests/bpf/prog_tests/bpftool_map_dump.c line 224

Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/bpftool_map_dump.c new file +9 -0
+	for (i = 0; i < ARRAY_SIZE(fds); i++) {
+		ids[i] = map_id(fds[i]);
+		if (!ids[i] || !dump_map(ids[i], "-j", elements))
+			goto out;
+		len += snprintf(expected + len, sizeof(expected) - len,
+				"%s{\"id\":%u,\"type\":\"%s\",\"name\":\"%s\","
+				"\"flags\":0,\"elements\":%s}%s",
+				i ? "," : "[", ids[i], types[i],
+				i == 3 ? "dump_discovered" : name, elements, i == 3 ? "]" : "");

[Severity: Medium]
Could this accumulation of the snprintf return value cause a stack buffer overflow?

snprintf returns the number of bytes that would have been written, not the number of bytes actually written. If the accumulated length exceeds sizeof(expected), the subsequent loop iteration passes sizeof(expected) - len as the buffer size, which underflows to a large unsigned integer since the sizeof operator yields a size_t.

Since the elements buffer can hold up to 8191 bytes, over 4 iterations this can mathematically exceed the 32768 bytes of expected when combined with the JSON formatting overhead (for instance, if bpftool produces unexpectedly long output or errors).

Does this code need a bounds check before adding to len to prevent writing past the end of the expected stack buffer?

Quoted 1 line
tools/testing/selftests/bpf/prog_tests/bpftool_map_dump.c new file +1 -0
+	}

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#5 lore raw

Re: [PATCH bpf-next v5 2/2] selftests/bpf: Cover recursive bpftool map dumps

The bounds check is useful, and v6 now checks for a negative snprintf() return and truncation before advancing len.

I have not established the reported overflow in the v5 fixture. The loop has exactly four iterations. Even three maximal elements strings and their JSON wrappers fit in the 32768-byte expected buffer. Only the fourth append could truncate, and there is no following append using that accumulated offset. The actual fixed maps are much smaller.

The explicit guard makes this safe to extend and reports truncation directly as a test failure. All 12 recursive-dump subtests passed, including the multiple-root case.

v6: https://lore.kernel.org/r/[email protected]

Thanks,
Tianyi

#6 lore raw

Re: [PATCH bpf-next v5 2/2] selftests/bpf: Cover recursive bpftool map dumps

In reply to Tianyi Chen
This isn't a bug, but could the changelog be trimmed to focus on the coverage added and the reason RLIMIT_NOFILE=32 / 64 inner maps is the interesting case?
Quoted 9 lines

Exercise recursive map dumping for array-of-maps and hash-of-maps, including shared inner maps, empty outer and inner maps, BTF-formatted values and multiple selected roots.

Check complete JSON documents against the existing nonrecursive entry representations, and check plain headers and typed values. Verify both short and long options, root ordering, deduplication and unchanged default output.

Quoted 8 lines

Cover ordinary and nested perf event arrays to ensure unsupported element lookups retain their error markers without failing the dump. Dump 64 distinct inner maps under RLIMIT_NOFILE=32 in both JSON and plain modes to catch holding every discovered inner-map FD open.

Open 32 extra descriptors before forking, close inherited descriptors in the child before imposing the limit, and verify the parent's file descriptors and outer map remain usable.

This isn't a bug, but could the changelog be trimmed to focus on the coverage added and the reason RLIMIT_NOFILE=32 / 64 inner maps is the interesting case?

The last paragraph in particular restates the code almost line for line (opening 32 descriptors, closing inherited descriptors in the child, lowering the limit, re-checking the parent), which a reader can get from the diff. The load-bearing part of this patch is the one non-obvious idea, i.e. that RLIMIT_NOFILE=32 with 64 inner maps is what makes the FD-per- inner-map regression detectable.

Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/bpftool_map_dump.c new file +0 -0
new file mode 100644
index 000000000000..1fb877fc68c8

[ ... ]

Quoted 11 lines
tools/testing/selftests/bpf/prog_tests/bpftool_map_dump.c new file +11 -0
+static void test_outer(enum bpf_map_type type, int entries, bool empty_inner,
+		       bool typed)
+{
+	LIBBPF_OPTS(bpf_map_create_opts, opts);
+	LIBBPF_OPTS(bpf_map_create_opts, inner_opts);
+	struct btf *btf = NULL;
+	char outer[OUTPUT_SIZE], inner[OUTPUT_SIZE], output[OUTPUT_SIZE];
+	char expected[OUTPUT_SIZE * 3], reference[64];
+	const char *type_name = libbpf_bpf_map_type_str(type);
+	int inner_fd = -1, outer_fd = -1;
+	__u32 root_id, inner_id, key, value = 0x01010101;

[ ... ]

tools/testing/selftests/bpf/prog_tests/bpftool_map_dump.c line 117

Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/bpftool_map_dump.c new file +9 -0
+	outer_fd = bpf_map_create(type, "dump_outer", sizeof(key), sizeof(__u32),
+				  3, &opts);
+	if (!ASSERT_OK_FD(outer_fd, "create_outer"))
+		goto out;
+	/* The unused third array slot also exercises failed lookups. */
+	for (key = 0; key < entries; key++)
+		if (!ASSERT_OK(bpf_map_update_elem(outer_fd, &key, &inner_fd, BPF_ANY),
+			       "populate_outer"))
+			goto out;

This isn't a bug, but would it be clearer to say the outer map is over- sized so the array_of_maps cases also walk unfilled slots, rather than naming 'the unused third array slot'?

The comment describes one specific configuration rather than what the loop does across the seven call sites. test_outer is invoked with BPF_MAP_TYPE_HASH_OF_MAPS in four of the seven subtests, where a hash_of_maps enumerates only the keys that were inserted, so there is no third slot and no failed lookup at all; and for entries == 0 or 1 (four more subtests) the number of unfilled array slots is three or two, not one.

[ ... ]

Quoted 19 lines
tools/testing/selftests/bpf/prog_tests/bpftool_map_dump.c new file +19 -0
+static void test_many_inner_maps(bool json)
+{
+	enum {
+		DUMP_OK,
+		DUMP_ERR_RLIMIT,
+		DUMP_ERR_COMMAND,
+		DUMP_ERR_COUNTS,
+		DUMP_ERR_IDS,
+		DUMP_ERR_JSON,
+		DUMP_ERR_FDS,
+	};
+	LIBBPF_OPTS(bpf_map_create_opts, opts);
+	const struct rlimit limit = { .rlim_cur = 32, .rlim_max = 32 };
+	char command[MAX_BPFTOOL_CMD_LEN], token[64];
+	__u32 ids[64], root_id, key;
+	int inner_fd = -1, outer_fd = -1, status;
+	int inherited_fds[32], nr_inherited = 0, i;
+	char *output = NULL;
+	pid_t pid;

[ ... ]

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/bpftool_map_dump.c new file +4 -0
+	root_id = map_id(outer_fd);
+	output = calloc(1, 65536);
+	if (!root_id || !ASSERT_OK_PTR(output, "allocate_output"))
+		goto out;

[ ... ]

tools/testing/selftests/bpf/prog_tests/bpftool_map_dump.c line 413

Quoted 4 lines
tools/testing/selftests/bpf/prog_tests/bpftool_map_dump.c new file +4 -0
+		snprintf(command, sizeof(command), "%s -r map dump id %u",
+			 json ? "-j" : "", root_id);
+		if (get_bpftool_command_output(command, output, 65535))
+			_exit(DUMP_ERR_COMMAND);

This isn't a bug, but would a macro alongside OUTPUT_SIZE (and passing size - 1 to the helper, as the other call sites do) keep these two sizes from drifting apart?

The output buffer size is spelled out as two related magic numbers in two places (calloc(1, 65536) and get_bpftool_command_output(..., 65535)), which have to be kept in the N / N-1 relationship by hand. Every other buffer in the file goes through the OUTPUT_SIZE macro and passes sizeof(buf) - 1, e.g. dump_map()'s get_bpftool_command_output(command, output, OUTPUT_SIZE - 1) and check_plain()'s sizeof(output) - 1, so this is the only place where the size is not derived from a single definition.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/34560347455