Skip to contentSkip to conversation

selftests/bpf: Add benchmark for local_storage get

PATCH Superseded Dave Marchevsky 3 participants 4 messages May 8, 2022 – May 13, 2022

Subsystems (3)

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 selftests/bpf: Add benchmark for local_storage get

Add a benchmarks to demonstrate the performance cliff for local_storage get as the number of local_storage maps increases beyond current local_storage implementation's cache size.

"sequential get" and "interleaved get" benchmarks are added, both of which do many bpf_task_storage_get calls on a set of {10, 100, 1000} task local_storage maps, while considering a single specific map to be 'important' and counting task_storage_gets to the important map separately in addition to normal 'hits' count of all gets. Goal here is to mimic scenario where a particular program using one map - the important one - is running on a system where many other local_storage maps exist and are accessed often.

While "sequential get" benchmark does bpf_task_storage_get for map 0, 1, ..., {9, 99, 999} in order, "interleaved" benchmark interleaves 4 bpf_task_storage_gets for the important map for every 10 map gets. This is meant to highlight performance differences when important map is accessed far more frequently than non-important maps.

Addition of this benchmark is inspired by conversation with Alexei in a previous patchset's thread [0], which highlighted the need for such a benchmark to motivate and validate improvements to local_storage implementation. My approach in that series focused on improving performance for explicitly-marked 'important' maps and was rejected with feedback to make more generally-applicable improvements while avoiding explicitly marking maps as important. Thus the benchmark reports both general and important-map-focused metrics, so effect of future work on both is clear.

Regarding the benchmark results. On a powerful system (Skylake, 20 cores, 256gb ram):

Local Storage
=============

        num_maps: 10

local_storage cache sequential get: hits throughput: 20.013 ± 0.818 M ops/s, hits latency: 49.967 ns/op, important_hits throughput: 2.001 ± 0.082 M ops/s local_storage cache interleaved get: hits throughput: 23.149 ± 0.342 M ops/s, hits latency: 43.198 ns/op, important_hits throughput: 8.268 ± 0.122 M ops/s

        num_maps: 100

local_storage cache sequential get: hits throughput: 6.149 ± 0.220 M ops/s, hits latency: 162.630 ns/op, important_hits throughput: 0.061 ± 0.002 M ops/s local_storage cache interleaved get: hits throughput: 7.659 ± 0.177 M ops/s, hits latency: 130.565 ns/op, important_hits throughput: 2.243 ± 0.052 M ops/s

        num_maps: 1000

local_storage cache sequential get: hits throughput: 0.917 ± 0.029 M ops/s, hits latency: 1090.711 ns/op, important_hits throughput: 0.002 ± 0.000 M ops/s local_storage cache interleaved get: hits throughput: 1.121 ± 0.016 M ops/s, hits latency: 892.299 ns/op, important_hits throughput: 0.322 ± 0.005 M ops/s

Looking at the "sequential get" results, it's clear that as the number of task local_storage maps grows beyond the current cache size (16), there's a significant reduction in hits throughput. Note that current local_storage implementation assigns a cache_idx to maps as they are created. Since "sequential get" is creating maps 0..n in order and then doing bpf_task_storage_get calls in the same order, the benchmark is effectively ensuring that a map will not be in cache when the program tries to access it.

For "interleaved get" results, important-map hits throughput is greatly increased as the important map is more likely to be in cache by virtue of being accessed far more frequently. Throughput still reduces as # maps increases, though.

Note that 100- and 1000-map variants of the benchmarks need to split task_storage_get calls across multiple programs to work around the verifier's MAX_USED_MAPS limitation. To rule out the effect of addtional overhead of this splitting on the results I created a version of local_storage_bench__get100 which split the 100 gets across 8 progs instead of 2. The __get100 results were not affected.

When running the benchmarks it may be necessary to bump 'open files' ulimit for a successful run.

[0]: https://lore.kernel.org/all/[email protected]

---
 tools/testing/selftests/bpf/Makefile          |  10 +-
 tools/testing/selftests/bpf/bench.c           |  55 ++++
 tools/testing/selftests/bpf/bench.h           |   5 +
 .../bpf/benchs/bench_local_storage.c          | 264 ++++++++++++++++++
 .../bpf/benchs/run_bench_local_storage.sh     |  16 ++
 .../selftests/bpf/benchs/run_common.sh        |  17 ++
 .../progs/local_storage_bench__create_map.c   |  12 +
 .../bpf/progs/local_storage_bench__get10.c    |  24 ++
 .../bpf/progs/local_storage_bench__get100.c   |  42 +++
 .../bpf/progs/local_storage_bench__get1000.c  |  56 ++++
 ...local_storage_bench__get1000_interleaved.c |  56 ++++
 .../local_storage_bench__get100_interleaved.c |  42 +++
 .../local_storage_bench__get10_interleaved.c  |  24 ++
 .../bpf/progs/local_storage_bench_helpers.h   |  85 ++++++
 14 files changed, 707 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_local_storage.c
 create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_local_storage.sh
 create mode 100644 tools/testing/selftests/bpf/progs/local_storage_bench__create_map.c
 create mode 100644 tools/testing/selftests/bpf/progs/local_storage_bench__get10.c
 create mode 100644 tools/testing/selftests/bpf/progs/local_storage_bench__get100.c
 create mode 100644 tools/testing/selftests/bpf/progs/local_storage_bench__get1000.c
 create mode 100644 tools/testing/selftests/bpf/progs/local_storage_bench__get1000_interleaved.c
 create mode 100644 tools/testing/selftests/bpf/progs/local_storage_bench__get100_interleaved.c
 create mode 100644 tools/testing/selftests/bpf/progs/local_storage_bench__get10_interleaved.c
 create mode 100644 tools/testing/selftests/bpf/progs/local_storage_bench_helpers.h
tools/testing/selftests/bpf/Makefile +9 -1
index bafdc5373a13..00b62ab5c932 100644
@@ -558,6 +558,13 @@ $(OUTPUT)/bench_ringbufs.o: $(OUTPUT)/ringbuf_bench.skel.h \
 $(OUTPUT)/bench_bloom_filter_map.o: $(OUTPUT)/bloom_filter_bench.skel.h
 $(OUTPUT)/bench_bpf_loop.o: $(OUTPUT)/bpf_loop_bench.skel.h
 $(OUTPUT)/bench_strncmp.o: $(OUTPUT)/strncmp_bench.skel.h
+$(OUTPUT)/bench_local_storage.o: $(OUTPUT)/local_storage_bench__create_map.skel.h \
+				  $(OUTPUT)/local_storage_bench__get10.skel.h \
+				  $(OUTPUT)/local_storage_bench__get100.skel.h \
+				  $(OUTPUT)/local_storage_bench__get1000.skel.h \
+				  $(OUTPUT)/local_storage_bench__get10_interleaved.skel.h \
+				  $(OUTPUT)/local_storage_bench__get100_interleaved.skel.h \
+				  $(OUTPUT)/local_storage_bench__get1000_interleaved.skel.h
 $(OUTPUT)/bench.o: bench.h testing_helpers.h $(BPFOBJ)
 $(OUTPUT)/bench: LDLIBS += -lm
 $(OUTPUT)/bench: $(OUTPUT)/bench.o \
@@ -569,7 +576,8 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \
 		 $(OUTPUT)/bench_ringbufs.o \
 		 $(OUTPUT)/bench_bloom_filter_map.o \
 		 $(OUTPUT)/bench_bpf_loop.o \
-		 $(OUTPUT)/bench_strncmp.o
+		 $(OUTPUT)/bench_strncmp.o \
+		 $(OUTPUT)/bench_local_storage.o
 	$(call msg,BINARY,,$@)
 	$(Q)$(CC) $(CFLAGS) $(LDFLAGS) $(filter %.a %.o,$^) $(LDLIBS) -o $@
 
tools/testing/selftests/bpf/bench.c +55 -0

90 lines not on this page · raw message

tools/testing/selftests/bpf/bench.h +5 -0
index fb3e213df3dc..0a137eedc959 100644
@@ -34,12 +34,14 @@ struct bench_res {
 	long hits;
 	long drops;
 	long false_hits;
+	long important_hits;
 };
 
 struct bench {
 	const char *name;
 	void (*validate)(void);
 	void (*setup)(void);
+	void (*teardown)(void);
 	void *(*producer_thread)(void *ctx);
 	void *(*consumer_thread)(void *ctx);
 	void (*measure)(struct bench_res* res);
@@ -61,6 +63,9 @@ void false_hits_report_progress(int iter, struct bench_res *res, long delta_ns);
 void false_hits_report_final(struct bench_res res[], int res_cnt);
 void ops_report_progress(int iter, struct bench_res *res, long delta_ns);
 void ops_report_final(struct bench_res res[], int res_cnt);
+void local_storage_report_progress(int iter, struct bench_res *res,
+				   long delta_ns);
+void local_storage_report_final(struct bench_res res[], int res_cnt);
 
 static inline __u64 get_time_ns(void)
 {
tools/testing/selftests/bpf/benchs/bench_local_storage.c new file +264 -0

264 lines not on this page · raw message

tools/testing/selftests/bpf/benchs/run_bench_local_storage.sh new file +16 -0
new file mode 100755
index 000000000000..a6f6f704cb7b
@@ -0,0 +1,16 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+source ./benchs/run_common.sh
+
+set -eufo pipefail
+
+header "Local Storage"
+for i in 10 100 1000; do
+subtitle "num_maps: $i"
+	summarize_local_storage "local_storage cache sequential  get: "\
+		"$(./bench --nr_maps $i local-storage-cache-seq-get)"
+	summarize_local_storage "local_storage cache interleaved get: "\
+		"$(./bench --nr_maps $i local-storage-cache-interleaved-get)"
+	printf "\n"
+done
tools/testing/selftests/bpf/benchs/run_common.sh +17 -0
index 6c5e6023a69f..d9f40af82006 100644
@@ -41,6 +41,16 @@ function ops()
 	echo "$*" | sed -E "s/.*latency\s+([0-9]+\.[0-9]+\sns\/op).*/\1/"
 }
 
+function local_storage()
+{
+	echo -n "hits throughput: "
+	echo -n "$*" | sed -E "s/.* hits throughput\s+([0-9]+\.[0-9]+ ± [0-9]+\.[0-9]+\sM\sops\/s).*/\1/"
+	echo -n -e ", hits latency: "
+	echo -n "$*" | sed -E "s/.* hits latency\s+([0-9]+\.[0-9]+\sns\/op).*/\1/"
+	echo -n ", important_hits throughput: "
+	echo "$*" | sed -E "s/.*important_hits throughput\s+([0-9]+\.[0-9]+ ± [0-9]+\.[0-9]+\sM\sops\/s).*/\1/"
+}
+
 function total()
 {
 	echo "$*" | sed -E "s/.*total operations\s+([0-9]+\.[0-9]+ ± [0-9]+\.[0-9]+M\/s).*/\1/"
@@ -67,6 +77,13 @@ function summarize_ops()
 	printf "%-20s %s\n" "$bench" "$(ops $summary)"
 }
 
+function summarize_local_storage()
+{
+	bench="$1"
+	summary=$(echo $2 | tail -n1)
+	printf "%-20s %s\n" "$bench" "$(local_storage $summary)"
+}
+
 function summarize_total()
 {
 	bench="$1"
tools/testing/selftests/bpf/progs/local_storage_bench__create_map.c new file +12 -0
new file mode 100644
index 000000000000..786465a79e9e
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+
+struct { \
+	__uint(type, BPF_MAP_TYPE_TASK_STORAGE); \
+	__uint(map_flags, BPF_F_NO_PREALLOC); \
+	__type(key, int); \
+	__type(value, __u32); \
+} map_to_pin SEC(".maps");
tools/testing/selftests/bpf/progs/local_storage_bench__get10.c new file +24 -0
new file mode 100644
index 000000000000..75f50f961654
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "local_storage_bench_helpers.h"
+
+long important_hits = 0;
+long hits = 0;
+
+/* Create maps local_storage_bench_pinned_{0000, .., 0009} */
+PINNED_MAP10(0, 0, 0);
+
+SEC("fentry/" SYS_PREFIX "sys_getpgid")
+int do_map_get(void *ctx)
+{
+	struct task_struct *task = bpf_get_current_task_btf();
+
+	TASK_STORAGE_GET10(0, 0, 0);
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/local_storage_bench__get100.c new file +42 -0

42 lines not on this page · raw message

tools/testing/selftests/bpf/progs/local_storage_bench__get1000.c new file +56 -0

56 lines not on this page · raw message

tools/testing/selftests/bpf/progs/local_storage_bench__get1000_interleaved.c new file +56 -0

56 lines not on this page · raw message

tools/testing/selftests/bpf/progs/local_storage_bench__get100_interleaved.c new file +42 -0

42 lines not on this page · raw message

tools/testing/selftests/bpf/progs/local_storage_bench__get10_interleaved.c new file +24 -0
new file mode 100644
index 000000000000..26af26c29738
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "local_storage_bench_helpers.h"
+
+long important_hits = 0;
+long hits = 0;
+
+/* Create maps local_storage_bench_pinned_{0000, .., 0009} */
+PINNED_MAP10(0, 0, 0);
+
+SEC("fentry/" SYS_PREFIX "sys_getpgid")
+int do_map_get(void *ctx)
+{
+	struct task_struct *task = bpf_get_current_task_btf();
+
+	TASK_STORAGE_GET10_INTERLEAVED(0, 0, 0);
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/local_storage_bench_helpers.h new file +85 -0

85 lines not on this page · raw message

Signature · 1 line
2.30.2
#1 lore raw
Quoted 38 lines · Dave Marchevsky

Add a benchmarks to demonstrate the performance cliff for local_storage get as the number of local_storage maps increases beyond current local_storage implementation's cache size.

"sequential get" and "interleaved get" benchmarks are added, both of which do many bpf_task_storage_get calls on a set of {10, 100, 1000} task local_storage maps, while considering a single specific map to be 'important' and counting task_storage_gets to the important map separately in addition to normal 'hits' count of all gets. Goal here is to mimic scenario where a particular program using one map - the important one - is running on a system where many other local_storage maps exist and are accessed often.

While "sequential get" benchmark does bpf_task_storage_get for map 0, 1, ..., {9, 99, 999} in order, "interleaved" benchmark interleaves 4 bpf_task_storage_gets for the important map for every 10 map gets. This is meant to highlight performance differences when important map is accessed far more frequently than non-important maps.

Addition of this benchmark is inspired by conversation with Alexei in a previous patchset's thread [0], which highlighted the need for such a benchmark to motivate and validate improvements to local_storage implementation. My approach in that series focused on improving performance for explicitly-marked 'important' maps and was rejected with feedback to make more generally-applicable improvements while avoiding explicitly marking maps as important. Thus the benchmark reports both general and important-map-focused metrics, so effect of future work on both is clear.

Regarding the benchmark results. On a powerful system (Skylake, 20 cores, 256gb ram):

Local Storage
=============

        num_maps: 10

local_storage cache sequential get: hits throughput: 20.013 ± 0.818 M ops/s, hits latency: 49.967 ns/op, important_hits throughput: 2.001 ± 0.082 M ops/s local_storage cache interleaved get: hits throughput: 23.149 ± 0.342 M ops/s, hits latency: 43.198 ns/op, important_hits throughput: 8.268 ± 0.122 M ops/s

Quoted 7 lines
        num_maps: 100

local_storage cache sequential get: hits throughput: 6.149 ± 0.220 M ops/s, hits latency: 162.630 ns/op, important_hits throughput: 0.061 ± 0.002 M ops/s local_storage cache interleaved get: hits throughput: 7.659 ± 0.177 M ops/s, hits latency: 130.565 ns/op, important_hits throughput: 2.243 ± 0.052 M ops/s

        num_maps: 1000

local_storage cache sequential get: hits throughput: 0.917 ± 0.029 M ops/s, hits latency: 1090.711 ns/op, important_hits throughput: 0.002 ± 0.000 M ops/s local_storage cache interleaved get: hits throughput: 1.121 ± 0.016 M ops/s, hits latency: 892.299 ns/op, important_hits throughput: 0.322 ± 0.005 M ops/s

Thanks for crafting a benchmark. It certainly helps to understand the cliff. Is there a way to make it more configurable?
10,100,1000 are hard coded and not easy to change.
In particular I'm interested in the numbers:
1, 16, 17, 32, 100.
If my understanding of implementation is correct 1 and 16 should have pretty much the same performance.
17 should see the cliff which should linearly increase in 32 and in 100. Between just two points 100 and 1000 there is no easy way
to compute the linear degradation.
Also could you add a hash map with key=tid.
It would be interesting to compare local storage with hash map.
iirc when local storage was introduced it was 2.5 times faster than large hashmap. Since then both have changed.

#2 lore raw
Quoted 47 lines · Alexei Starovoitov
Quoted 38 lines · Dave Marchevsky

Add a benchmarks to demonstrate the performance cliff for local_storage get as the number of local_storage maps increases beyond current local_storage implementation's cache size.

"sequential get" and "interleaved get" benchmarks are added, both of which do many bpf_task_storage_get calls on a set of {10, 100, 1000} task local_storage maps, while considering a single specific map to be 'important' and counting task_storage_gets to the important map separately in addition to normal 'hits' count of all gets. Goal here is to mimic scenario where a particular program using one map - the important one - is running on a system where many other local_storage maps exist and are accessed often.

While "sequential get" benchmark does bpf_task_storage_get for map 0, 1, ..., {9, 99, 999} in order, "interleaved" benchmark interleaves 4 bpf_task_storage_gets for the important map for every 10 map gets. This is meant to highlight performance differences when important map is accessed far more frequently than non-important maps.

Addition of this benchmark is inspired by conversation with Alexei in a previous patchset's thread [0], which highlighted the need for such a benchmark to motivate and validate improvements to local_storage implementation. My approach in that series focused on improving performance for explicitly-marked 'important' maps and was rejected with feedback to make more generally-applicable improvements while avoiding explicitly marking maps as important. Thus the benchmark reports both general and important-map-focused metrics, so effect of future work on both is clear.

Regarding the benchmark results. On a powerful system (Skylake, 20 cores, 256gb ram):

Local Storage
=============

        num_maps: 10

local_storage cache sequential get: hits throughput: 20.013 ± 0.818 M ops/s, hits latency: 49.967 ns/op, important_hits throughput: 2.001 ± 0.082 M ops/s local_storage cache interleaved get: hits throughput: 23.149 ± 0.342 M ops/s, hits latency: 43.198 ns/op, important_hits throughput: 8.268 ± 0.122 M ops/s

Quoted 7 lines
        num_maps: 100

local_storage cache sequential get: hits throughput: 6.149 ± 0.220 M ops/s, hits latency: 162.630 ns/op, important_hits throughput: 0.061 ± 0.002 M ops/s local_storage cache interleaved get: hits throughput: 7.659 ± 0.177 M ops/s, hits latency: 130.565 ns/op, important_hits throughput: 2.243 ± 0.052 M ops/s

        num_maps: 1000

local_storage cache sequential get: hits throughput: 0.917 ± 0.029 M ops/s, hits latency: 1090.711 ns/op, important_hits throughput: 0.002 ± 0.000 M ops/s local_storage cache interleaved get: hits throughput: 1.121 ± 0.016 M ops/s, hits latency: 892.299 ns/op, important_hits throughput: 0.322 ± 0.005 M ops/s

Quoted 10 lines

Thanks for crafting a benchmark. It certainly helps to understand the cliff. Is there a way to make it more configurable?
10,100,1000 are hard coded and not easy to change.
In particular I'm interested in the numbers:
1, 16, 17, 32, 100.
If my understanding of implementation is correct 1 and 16 should have pretty much the same performance.
17 should see the cliff which should linearly increase in 32 and in 100. Between just two points 100 and 1000 there is no easy way
to compute the linear degradation.

Agreed that being able to choose an arbitrary number of local_storage maps for the benchmark would be ideal. I tried to do this in an earlier iteration of the patch, but abandoned it as it would require a bcc-style approach, something like generating .c bpf program with python at runtime, then running through libbpf loader.

The easiest path forward is probably just generating bytecode and using raw API. Will give that a shot, hopefully the test prog is still comprehensible in that form.

Quoted 4 lines

Also could you add a hash map with key=tid.
It would be interesting to compare local storage with hash map.
iirc when local storage was introduced it was 2.5 times faster than large hashmap. Since then both have changed.

Will do.

#3 lore raw
Quoted 66 lines · Dave Marchevsky
Quoted 47 lines · Alexei Starovoitov
Quoted 38 lines · Dave Marchevsky

Add a benchmarks to demonstrate the performance cliff for local_storage get as the number of local_storage maps increases beyond current local_storage implementation's cache size.

"sequential get" and "interleaved get" benchmarks are added, both of which do many bpf_task_storage_get calls on a set of {10, 100, 1000} task local_storage maps, while considering a single specific map to be 'important' and counting task_storage_gets to the important map separately in addition to normal 'hits' count of all gets. Goal here is to mimic scenario where a particular program using one map - the important one - is running on a system where many other local_storage maps exist and are accessed often.

While "sequential get" benchmark does bpf_task_storage_get for map 0, 1, ..., {9, 99, 999} in order, "interleaved" benchmark interleaves 4 bpf_task_storage_gets for the important map for every 10 map gets. This is meant to highlight performance differences when important map is accessed far more frequently than non-important maps.

Addition of this benchmark is inspired by conversation with Alexei in a previous patchset's thread [0], which highlighted the need for such a benchmark to motivate and validate improvements to local_storage implementation. My approach in that series focused on improving performance for explicitly-marked 'important' maps and was rejected with feedback to make more generally-applicable improvements while avoiding explicitly marking maps as important. Thus the benchmark reports both general and important-map-focused metrics, so effect of future work on both is clear.

Regarding the benchmark results. On a powerful system (Skylake, 20 cores, 256gb ram):

Local Storage
=============

        num_maps: 10

local_storage cache sequential get: hits throughput: 20.013 ± 0.818 M ops/s, hits latency: 49.967 ns/op, important_hits throughput: 2.001 ± 0.082 M ops/s local_storage cache interleaved get: hits throughput: 23.149 ± 0.342 M ops/s, hits latency: 43.198 ns/op, important_hits throughput: 8.268 ± 0.122 M ops/s

Quoted 7 lines
        num_maps: 100

local_storage cache sequential get: hits throughput: 6.149 ± 0.220 M ops/s, hits latency: 162.630 ns/op, important_hits throughput: 0.061 ± 0.002 M ops/s local_storage cache interleaved get: hits throughput: 7.659 ± 0.177 M ops/s, hits latency: 130.565 ns/op, important_hits throughput: 2.243 ± 0.052 M ops/s

        num_maps: 1000

local_storage cache sequential get: hits throughput: 0.917 ± 0.029 M ops/s, hits latency: 1090.711 ns/op, important_hits throughput: 0.002 ± 0.000 M ops/s local_storage cache interleaved get: hits throughput: 1.121 ± 0.016 M ops/s, hits latency: 892.299 ns/op, important_hits throughput: 0.322 ± 0.005 M ops/s

Quoted 10 lines

Thanks for crafting a benchmark. It certainly helps to understand the cliff. Is there a way to make it more configurable?
10,100,1000 are hard coded and not easy to change.
In particular I'm interested in the numbers:
1, 16, 17, 32, 100.
If my understanding of implementation is correct 1 and 16 should have pretty much the same performance.
17 should see the cliff which should linearly increase in 32 and in 100. Between just two points 100 and 1000 there is no easy way
to compute the linear degradation.

Agreed that being able to choose an arbitrary number of local_storage maps for the benchmark would be ideal. I tried to do this in an earlier iteration of the patch, but abandoned it as it would require a bcc-style approach, something like generating .c bpf program with python at runtime, then running through libbpf loader.

Quoted 3 lines

The easiest path forward is probably just generating bytecode and using raw API. Will give that a shot, hopefully the test prog is still comprehensible in that form.

Can this be avoided by using ARRAY_OF_MAPS and doing bounded loops? I see that we have inlining for ARRAY_OF_MAPS, just like for ARRAY, so it shouldn't hurt performance (not significantly at least).

Quoted 7 lines
Quoted 5 lines

Also could you add a hash map with key=tid.
It would be interesting to compare local storage with hash map.
iirc when local storage was introduced it was 2.5 times faster than large hashmap. Since then both have changed.

Will do.