Skip to contentSkip to conversation

libbpf: Support uniform BTF-defined key/value specification across all BPF maps

PATCH In review Hengqi Chen 2 participants 4 messages Sep 30, 2021 – Oct 1, 2021

Subsystems (5)
Kselftest BPF BPF selftests libbpf +1 more subsystems
lore
Share & tools
lore
#0 lore raw

cover libbpf: Support uniform BTF-defined key/value specification across all BPF maps

Currently a bunch of (usually pretty specialized) BPF maps do not support specifying BTF types for they key and value. For such maps, specifying their definition like this:

  struct {
      __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
      __type(key, int);
      __type(value, int);
  } my_perf_buf SEC(".maps");

Would actually produce warnings about retrying BPF map creation without BTF. Users are forced to know such nuances and use __uint(key_size, 4) instead. This is non-uniform, annoying, and inconvenient.

This patch set teaches libbpf to recognize those specialized maps and removes BTF type IDs when creating BPF map. Also, update existing BPF selftests to exericse this change.

Hengqi Chen (2):
libbpf: Support uniform BTF-defined key/value specification across all
BPF maps
selftests/bpf: Use BTF-defined key/value for map definitions

 tools/lib/bpf/libbpf.c                        | 24 +++++++++++++++++++
 tools/testing/selftests/bpf/progs/kfree_skb.c |  4 ++--
 .../selftests/bpf/progs/perf_event_stackmap.c |  4 ++--
 .../bpf/progs/sockmap_verdict_prog.c          | 12 +++++-----
 .../selftests/bpf/progs/test_btf_map_in_map.c | 14 +++++------
 .../selftests/bpf/progs/test_map_in_map.c     | 10 ++++----
 .../bpf/progs/test_map_in_map_invalid.c       |  2 +-
 .../bpf/progs/test_pe_preserve_elems.c        |  8 +++----
 .../selftests/bpf/progs/test_perf_buffer.c    |  4 ++--
 .../bpf/progs/test_select_reuseport_kern.c    |  4 ++--
 .../bpf/progs/test_stacktrace_build_id.c      |  4 ++--
 .../selftests/bpf/progs/test_stacktrace_map.c |  4 ++--
 .../selftests/bpf/progs/test_tcpnotify_kern.c |  4 ++--
 .../selftests/bpf/progs/test_xdp_bpf2bpf.c    |  4 ++--
 14 files changed, 62 insertions(+), 40 deletions(-)

--
2.30.2

#1 lore raw

PATCH 1/2 libbpf: Support uniform BTF-defined key/value specification across all BPF maps

A bunch of BPF maps do not support specifying BTF types for key and value. This is non-uniform and inconvenient[0]. Currently, libbpf uses a retry logic which removes BTF type IDs when BPF map creation failed. Instead of retrying, this commit recognizes those specialized maps and removes BTF type IDs when creating BPF map.

[0] Closes: https://github.com/libbpf/libbpf/issues/355

---
 tools/lib/bpf/libbpf.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
tools/lib/bpf/libbpf.c +24 -0
index ef5db34bf913..b96eb3a64cca 100644
@@ -4643,6 +4643,30 @@ static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map, b
 			create_attr.inner_map_fd = map->inner_map_fd;
 	}
 
+	switch (def->type) {
+	case BPF_MAP_TYPE_PERF_EVENT_ARRAY:
+	case BPF_MAP_TYPE_CGROUP_ARRAY:
+	case BPF_MAP_TYPE_STACK_TRACE:
+	case BPF_MAP_TYPE_ARRAY_OF_MAPS:
+	case BPF_MAP_TYPE_HASH_OF_MAPS:
+	case BPF_MAP_TYPE_DEVMAP:
+	case BPF_MAP_TYPE_DEVMAP_HASH:
+	case BPF_MAP_TYPE_CPUMAP:
+	case BPF_MAP_TYPE_XSKMAP:
+	case BPF_MAP_TYPE_SOCKMAP:
+	case BPF_MAP_TYPE_SOCKHASH:
+	case BPF_MAP_TYPE_QUEUE:
+	case BPF_MAP_TYPE_STACK:
+	case BPF_MAP_TYPE_RINGBUF:
+		create_attr.btf_fd = 0;
+		create_attr.btf_key_type_id = 0;
+		create_attr.btf_value_type_id = 0;
+		map->btf_key_type_id = 0;
+		map->btf_value_type_id = 0;
+	default:
+		break;
+	}
+
 	if (obj->gen_loader) {
 		bpf_gen__map_create(obj->gen_loader, &create_attr, is_inner ? -1 : map - obj->maps);
 		/* Pretend to have valid FD to pass various fd >= 0 checks.

--
2.30.2

#2 lore raw

PATCH 2/2 selftests/bpf: Use BTF-defined key/value for map definitions

In reply to Hengqi Chen

Change map definitions in BPF selftests to use BTF-defined key/value types. This unifies the map definitions and ensures libbpf won't emit warning about retrying map creation.

---
 tools/testing/selftests/bpf/progs/kfree_skb.c      |  4 ++--
 .../selftests/bpf/progs/perf_event_stackmap.c      |  4 ++--
 .../selftests/bpf/progs/sockmap_verdict_prog.c     | 12 ++++++------
 .../selftests/bpf/progs/test_btf_map_in_map.c      | 14 +++++++-------
 .../testing/selftests/bpf/progs/test_map_in_map.c  | 10 ++++------
 .../selftests/bpf/progs/test_map_in_map_invalid.c  |  2 +-
 .../selftests/bpf/progs/test_pe_preserve_elems.c   |  8 ++++----
 .../testing/selftests/bpf/progs/test_perf_buffer.c |  4 ++--
 .../bpf/progs/test_select_reuseport_kern.c         |  4 ++--
 .../selftests/bpf/progs/test_stacktrace_build_id.c |  4 ++--
 .../selftests/bpf/progs/test_stacktrace_map.c      |  4 ++--
 .../selftests/bpf/progs/test_tcpnotify_kern.c      |  4 ++--
 .../testing/selftests/bpf/progs/test_xdp_bpf2bpf.c |  4 ++--
 13 files changed, 38 insertions(+), 40 deletions(-)
tools/testing/selftests/bpf/progs/kfree_skb.c +2 -2
index 55e283050cab..7236da72ce80 100644
@@ -9,8 +9,8 @@
 char _license[] SEC("license") = "GPL";
 struct {
 	__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
-	__uint(key_size, sizeof(int));
-	__uint(value_size, sizeof(int));
+	__type(key, int);
+	__type(value, int);
 } perf_buf_map SEC(".maps");
 
 #define _(P) (__builtin_preserve_access_index(P))
tools/testing/selftests/bpf/progs/perf_event_stackmap.c +2 -2
index 25467d13c356..b3fcb5274ee0 100644
@@ -11,8 +11,8 @@ typedef __u64 stack_trace_t[PERF_MAX_STACK_DEPTH];
 struct {
 	__uint(type, BPF_MAP_TYPE_STACK_TRACE);
 	__uint(max_entries, 16384);
-	__uint(key_size, sizeof(__u32));
-	__uint(value_size, sizeof(stack_trace_t));
+	__type(key, __u32);
+	__type(value, stack_trace_t);
 } stackmap SEC(".maps");
 
 struct {
tools/testing/selftests/bpf/progs/sockmap_verdict_prog.c +6 -6
index 4797dc985064..73872c535cbb 100644
@@ -7,22 +7,22 @@ int _version SEC("version") = 1;
 struct {
 	__uint(type, BPF_MAP_TYPE_SOCKMAP);
 	__uint(max_entries, 20);
-	__uint(key_size, sizeof(int));
-	__uint(value_size, sizeof(int));
+	__type(key, int);
+	__type(value, int);
 } sock_map_rx SEC(".maps");
 
 struct {
 	__uint(type, BPF_MAP_TYPE_SOCKMAP);
 	__uint(max_entries, 20);
-	__uint(key_size, sizeof(int));
-	__uint(value_size, sizeof(int));
+	__type(key, int);
+	__type(value, int);
 } sock_map_tx SEC(".maps");
 
 struct {
 	__uint(type, BPF_MAP_TYPE_SOCKMAP);
 	__uint(max_entries, 20);
-	__uint(key_size, sizeof(int));
-	__uint(value_size, sizeof(int));
+	__type(key, int);
+	__type(value, int);
 } sock_map_msg SEC(".maps");
 
 struct {
tools/testing/selftests/bpf/progs/test_btf_map_in_map.c +7 -7
index c1e0c8c7c55f..c218cf8989a9 100644
@@ -21,8 +21,8 @@ struct inner_map_sz2 {
 struct outer_arr {
 	__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
 	__uint(max_entries, 3);
-	__uint(key_size, sizeof(int));
-	__uint(value_size, sizeof(int));
+	__type(key, int);
+	__type(value, int);
 	/* it's possible to use anonymous struct as inner map definition here */
 	__array(values, struct {
 		__uint(type, BPF_MAP_TYPE_ARRAY);
@@ -61,8 +61,8 @@ struct inner_map_sz4 {
 struct outer_arr_dyn {
 	__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
 	__uint(max_entries, 3);
-	__uint(key_size, sizeof(int));
-	__uint(value_size, sizeof(int));
+	__type(key, int);
+	__type(value, int);
 	__array(values, struct {
 		__uint(type, BPF_MAP_TYPE_ARRAY);
 		__uint(map_flags, BPF_F_INNER_MAP);
@@ -81,7 +81,7 @@ struct outer_arr_dyn {
 struct outer_hash {
 	__uint(type, BPF_MAP_TYPE_HASH_OF_MAPS);
 	__uint(max_entries, 5);
-	__uint(key_size, sizeof(int));
+	__type(key, int);
 	/* Here everything works flawlessly due to reuse of struct inner_map
 	 * and compiler will complain at the attempt to use non-inner_map
 	 * references below. This is great experience.
@@ -111,8 +111,8 @@ struct sockarr_sz2 {
 struct outer_sockarr_sz1 {
 	__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
 	__uint(max_entries, 1);
-	__uint(key_size, sizeof(int));
-	__uint(value_size, sizeof(int));
+	__type(key, int);
+	__type(value, int);
 	__array(values, struct sockarr_sz1);
 } outer_sockarr SEC(".maps") = {
 	.values = { (void *)&sockarr_sz1 },
tools/testing/selftests/bpf/progs/test_map_in_map.c +4 -6
index 1cfeb940cf9f..68a984695184 100644
@@ -9,18 +9,16 @@ struct {
 	__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
 	__uint(max_entries, 1);
 	__uint(map_flags, 0);
-	__uint(key_size, sizeof(__u32));
-	/* must be sizeof(__u32) for map in map */
-	__uint(value_size, sizeof(__u32));
+	__type(key, __u32);
+	__type(value, __u32);
 } mim_array SEC(".maps");
 
 struct {
 	__uint(type, BPF_MAP_TYPE_HASH_OF_MAPS);
 	__uint(max_entries, 1);
 	__uint(map_flags, 0);
-	__uint(key_size, sizeof(int));
-	/* must be sizeof(__u32) for map in map */
-	__uint(value_size, sizeof(__u32));
+	__type(key, int);
+	__type(value, __u32);
 } mim_hash SEC(".maps");
 
 SEC("xdp_mimtest")
tools/testing/selftests/bpf/progs/test_map_in_map_invalid.c +1 -1
index 703c08e06442..9c7d75cf0bd6 100644
@@ -13,7 +13,7 @@ struct inner {
 struct {
 	__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
 	__uint(max_entries, 0); /* This will make map creation to fail */
-	__uint(key_size, sizeof(__u32));
+	__type(key, __u32);
 	__array(values, struct inner);
 } mim SEC(".maps");
 
tools/testing/selftests/bpf/progs/test_pe_preserve_elems.c +4 -4
index fb22de7c365d..1249a945699f 100644
@@ -7,15 +7,15 @@
 struct {
 	__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
 	__uint(max_entries, 1);
-	__uint(key_size, sizeof(int));
-	__uint(value_size, sizeof(int));
+	__type(key, int);
+	__type(value, int);
 } array_1 SEC(".maps");
 
 struct {
 	__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
 	__uint(max_entries, 1);
-	__uint(key_size, sizeof(int));
-	__uint(value_size, sizeof(int));
+	__type(key, int);
+	__type(value, int);
 	__uint(map_flags, BPF_F_PRESERVE_ELEMS);
 } array_2 SEC(".maps");
 
tools/testing/selftests/bpf/progs/test_perf_buffer.c +2 -2
index 8207a2dc2f9d..d37ce29fd393 100644
@@ -8,8 +8,8 @@
 
 struct {
 	__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
-	__uint(key_size, sizeof(int));
-	__uint(value_size, sizeof(int));
+	__type(key, int);
+	__type(value, int);
 } perf_buf_map SEC(".maps");
 
 SEC("tp/raw_syscalls/sys_enter")
tools/testing/selftests/bpf/progs/test_select_reuseport_kern.c +2 -2
index 26e77dcc7e91..0f9bc258225e 100644
@@ -24,8 +24,8 @@ int _version SEC("version") = 1;
 struct {
 	__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
 	__uint(max_entries, 1);
-	__uint(key_size, sizeof(__u32));
-	__uint(value_size, sizeof(__u32));
+	__type(key, __u32);
+	__type(value, __u32);
 } outer_map SEC(".maps");
 
 struct {
tools/testing/selftests/bpf/progs/test_stacktrace_build_id.c +2 -2
index 0cf0134631b4..7449fdb1763b 100644
@@ -28,8 +28,8 @@ struct {
 	__uint(type, BPF_MAP_TYPE_STACK_TRACE);
 	__uint(max_entries, 128);
 	__uint(map_flags, BPF_F_STACK_BUILD_ID);
-	__uint(key_size, sizeof(__u32));
-	__uint(value_size, sizeof(stack_trace_t));
+	__type(key, __u32);
+	__type(value, stack_trace_t);
 } stackmap SEC(".maps");
 
 struct {
tools/testing/selftests/bpf/progs/test_stacktrace_map.c +2 -2
index 00ed48672620..a8233e7f173b 100644
@@ -27,8 +27,8 @@ typedef __u64 stack_trace_t[PERF_MAX_STACK_DEPTH];
 struct {
 	__uint(type, BPF_MAP_TYPE_STACK_TRACE);
 	__uint(max_entries, 16384);
-	__uint(key_size, sizeof(__u32));
-	__uint(value_size, sizeof(stack_trace_t));
+	__type(key, __u32);
+	__type(value, stack_trace_t);
 } stackmap SEC(".maps");
 
 struct {
tools/testing/selftests/bpf/progs/test_tcpnotify_kern.c +2 -2
index ac63410bb541..24e9344994ef 100644
@@ -24,8 +24,8 @@ struct {
 struct {
 	__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
 	__uint(max_entries, 2);
-	__uint(key_size, sizeof(int));
-	__uint(value_size, sizeof(__u32));
+	__type(key, int);
+	__type(value, __u32);
 } perf_event_map SEC(".maps");
 
 int _version SEC("version") = 1;
tools/testing/selftests/bpf/progs/test_xdp_bpf2bpf.c +2 -2
index a038e827f850..58cf4345f5cc 100644
@@ -36,8 +36,8 @@ struct meta {
 
 struct {
 	__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
-	__uint(key_size, sizeof(int));
-	__uint(value_size, sizeof(int));
+	__type(key, int);
+	__type(value, int);
 } perf_buf_map SEC(".maps");
 
 __u64 test_result_fentry = 0;

--
2.30.2

#3 lore raw
In reply to Hengqi Chen
Quoted 40 lines · Hengqi Chen

Currently a bunch of (usually pretty specialized) BPF maps do not support specifying BTF types for they key and value. For such maps, specifying their definition like this:

  struct {
      __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
      __type(key, int);
      __type(value, int);
  } my_perf_buf SEC(".maps");

Would actually produce warnings about retrying BPF map creation without BTF. Users are forced to know such nuances and use __uint(key_size, 4) instead. This is non-uniform, annoying, and inconvenient.

This patch set teaches libbpf to recognize those specialized maps and removes BTF type IDs when creating BPF map. Also, update existing BPF selftests to exericse this change.

Hengqi Chen (2):
libbpf: Support uniform BTF-defined key/value specification across all
BPF maps
selftests/bpf: Use BTF-defined key/value for map definitions

 tools/lib/bpf/libbpf.c                        | 24 +++++++++++++++++++
 tools/testing/selftests/bpf/progs/kfree_skb.c |  4 ++--
 .../selftests/bpf/progs/perf_event_stackmap.c |  4 ++--
 .../bpf/progs/sockmap_verdict_prog.c          | 12 +++++-----
 .../selftests/bpf/progs/test_btf_map_in_map.c | 14 +++++------
 .../selftests/bpf/progs/test_map_in_map.c     | 10 ++++----
 .../bpf/progs/test_map_in_map_invalid.c       |  2 +-
 .../bpf/progs/test_pe_preserve_elems.c        |  8 +++----
 .../selftests/bpf/progs/test_perf_buffer.c    |  4 ++--
 .../bpf/progs/test_select_reuseport_kern.c    |  4 ++--
 .../bpf/progs/test_stacktrace_build_id.c      |  4 ++--
 .../selftests/bpf/progs/test_stacktrace_map.c |  4 ++--
 .../selftests/bpf/progs/test_tcpnotify_kern.c |  4 ++--
 .../selftests/bpf/progs/test_xdp_bpf2bpf.c    |  4 ++--
 14 files changed, 62 insertions(+), 40 deletions(-)
Quoted 2 lines

--
2.30.2

Looks good. Applied to bpf-next, thanks.