bpf: Populate mmap-able array maps lazily
Subsystems (4)
Song Liu
reviewercover bpf: Populate mmap-able array maps lazily
An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory vmalloc'ed up front at map creation time. array_map_mmap() then wires up the whole mapping eagerly via remap_vmalloc_range(), which calls vm_insert_page() for every page of the map. This makes every mmap() cost O(number of pages): an 8MiB map inserts 2048 PTEs on each mmap() and tears them all down again on munmap(), even when user space only touches a few pages (or none at all).
This series makes the mapping populate lazily instead, the same way the arena map already does, turning mmap()/munmap() of an mmap-able array from O(map size) into O(1). The pages are inserted on demand by a new map_mmap_fault() callback, dispatched from the shared bpf_map_default_vmops so that the existing VMA open/close accounting (VM_MAYWRITE write-active tracking, freeze handling) stays centralized.
Callers that want the pages populated up front can still request that explicitly with MAP_POPULATE, matching regular file semantics. Kernel-side access to the map (via the vmalloc address) is unaffected.
Patch 1 implements the lazy faulting. Patch 2 adds an "arraymap-mmap" benchmark that mmap()s/munmap()s an mmap-able array map from a configurable number of threads, useful to observe how the cost scales with the mapping size.
Benchmark results:
Before the patch
nr_threads: 1, map_size: 1048576
arraymap-mmap: throughput: 0.034 +/- 0.000 M ops/s, latency: 29134.211 ns/op
nr_threads: 1, map_size: 8388608
arraymap-mmap: throughput: 0.005 +/- 0.000 M ops/s, latency: 220365.807 ns/op
nr_threads: 1, map_size: 67108864
arraymap-mmap: throughput: 0.001 +/- 0.000 M ops/s, latency: 1749642.119 ns/op
After the patch
nr_threads: 1, map_size: 1048576
arraymap-mmap: throughput: 0.819 +/- 0.002 M ops/s, latency: 1221.476 ns/op
nr_threads: 1, map_size: 8388608
arraymap-mmap: throughput: 0.808 +/- 0.005 M ops/s, latency: 1236.975 ns/op
nr_threads: 1, map_size: 67108864arraymap-mmap: throughput: 0.738 +/- 0.003 M ops/s, latency: 1355.100 ns/op
Song Liu (2):
bpf: Populate mmap-able array map memory lazily selftests/bpf: Add mmap/munmap benchmark for array maps
include/linux/bpf.h | 1 + kernel/bpf/arraymap.c | 40 ++++- kernel/bpf/syscall.c | 17 ++ tools/testing/selftests/bpf/Makefile | 1 + tools/testing/selftests/bpf/bench.c | 4 + .../bpf/benchs/bench_arraymap_mmap.c | 146 ++++++++++++++++++ .../bpf/benchs/run_bench_arraymap_mmap.sh | 15 ++ 7 files changed, 221 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_arraymap_mmap.sh
--
2.53.0-Meta
Song Liu
reviewerPATCH 1/2 bpf: Populate mmap-able array map memory lazily
An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory vmalloc'ed up front at map creation time. array_map_mmap() then wired up the whole mapping eagerly via remap_vmalloc_range(), which calls vm_insert_page() for every page of the map. For large maps this makes every mmap() O(number of pages): an 8MiB map inserts 2048 PTEs per mmap() and tears them all down again on munmap(), even when user space only touches a few pages (or none at all).
Populate the mapping lazily instead, the same way the arena map already does. array_map_mmap() now only performs the bounds check and returns, leaving the PTEs unpopulated; the pages are inserted on demand by a new array_map_mmap_fault() handler. Because the memory is already resident, the fault handler simply resolves the vmalloc page and hands it to the fault path.
To keep the existing VMA open/close accounting (VM_MAYWRITE write-active tracking, freeze handling) centralized, the fault handler is reached through a new optional ->map_mmap_fault callback dispatched from the shared bpf_map_default_vmops, rather than each map installing its own vm_operations_struct.
This turns mmap()/munmap() of an mmap-able array from O(map size) into O(1). Callers that want the pages populated up front can still request that explicitly with MAP_POPULATE, matching regular file semantics. Kernel-side access to the map (via the vmalloc address) is unaffected.
--- include/linux/bpf.h | 1 + kernel/bpf/arraymap.c | 40 +++++++++++++++++++++++++++++++++++++--- kernel/bpf/syscall.c | 17 +++++++++++++++++ 3 files changed, 55 insertions(+), 3 deletions(-)
include/linux/bpf.h +1 -0
@@ -145,6 +145,7 @@ struct bpf_map_ops {int (*map_direct_value_meta)(const struct bpf_map *map,u64 imm,u32 *off);int (*map_mmap)(struct bpf_map *map,struct vm_area_struct *vma); +vm_fault_t (*map_mmap_fault)(struct bpf_map *map,struct vm_fault *vmf);__poll_t (*map_poll)(struct bpf_map *map,struct file *filp,struct poll_table_struct *pts); unsigned long (*map_get_unmapped_area)(struct file *filep, unsigned long addr,
kernel/bpf/arraymap.c +37 -3
@@ -576,7 +576,6 @@ static int array_map_check_btf(struct bpf_map *map,static int array_map_mmap (struct bpf_map *map,struct vm_area_struct *vma) { struct bpf_array *array = container_of(map, struct bpf_array, map); -pgoff_t pgoff =PAGE_ALIGN (sizeof (*array)) >>PAGE_SHIFT ;if (!(map->map_flags &BPF_F_MMAPABLE ))return -EINVAL ;
@@ -585,8 +584,42 @@ static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma) PAGE_ALIGN((u64)array->map.max_entries * array->elem_size)) return -EINVAL; -return remap_vmalloc_range (vma,array_map_vmalloc_addr (array), - vma->vm_pgoff + pgoff); +/* +* The backing memory is vmalloc'ed up front, so instead of wiring up +* every PTE here we let array_map_mmap_fault() insert pages on demand. +* remap_vmalloc_range_partial() used to set these flags for us; keep +* them to preserve behavior (no VMA expansion, excluded from coredump). +*/ +vm_flags_set (vma,VM_DONTEXPAND |VM_DONTDUMP ); + +return 0 ; +} + +static vm_fault_t array_map_mmap_fault (struct bpf_map *map, +struct vm_fault *vmf) +{ + struct bpf_array *array = container_of(map, struct bpf_array, map); +pgoff_t pgoff =PAGE_ALIGN (sizeof (*array)) >>PAGE_SHIFT ; +struct page *page; +void *kaddr; + +/* +* vmf->pgoff is the page offset within the mapping (vma->vm_pgoff plus +* the offset of the faulting address). The data pages start after the +* bpf_array header, hence the extra pgoff. This mirrors the offset that +* remap_vmalloc_range(vma, ..., vma->vm_pgoff + pgoff) used to apply. +*/ + kaddr =array_map_vmalloc_addr (array) + + ((u64)(vmf->pgoff + pgoff) <<PAGE_SHIFT ); + + page =vmalloc_to_page (kaddr); +if (!page) +return VM_FAULT_SIGBUS ; + +get_page (page); + vmf->page = page; + +return 0 ; } static bool array_map_meta_equal(const struct bpf_map *meta0,
@@ -812,6 +845,7 @@ const struct bpf_map_ops array_map_ops = { .map_direct_value_addr = array_map_direct_value_addr, .map_direct_value_meta = array_map_direct_value_meta, .map_mmap = array_map_mmap, + .map_mmap_fault = array_map_mmap_fault, .map_seq_show_elem = array_map_seq_show_elem, .map_check_btf = array_map_check_btf, .map_lookup_batch = generic_map_lookup_batch,
kernel/bpf/syscall.c +17 -0
@@ -1077,9 +1077,26 @@ static void bpf_map_mmap_close(struct vm_area_struct *vma)bpf_map_write_active_dec (map); } +/* +* Called for maps that populate their memory-mapped region lazily, i.e. that +* return without wiring up any PTEs from their ->map_mmap callback. Maps that +* populate the whole VMA eagerly (e.g. via remap_vmalloc_range()) never reach +* here. +*/ +static vm_fault_t bpf_map_mmap_fault (struct vm_fault *vmf) +{ +struct bpf_map *map = vmf->vma->vm_private_data; + +if (!map->ops->map_mmap_fault) +return VM_FAULT_SIGBUS ; + +return map->ops->map_mmap_fault (map, vmf); +} +static const struct vm_operations_struct bpf_map_default_vmops = { .open = bpf_map_mmap_open, .close = bpf_map_mmap_close, + .fault = bpf_map_mmap_fault, };static int bpf_map_mmap (struct file *filp,struct vm_area_struct *vma)
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next 1/2] bpf: Populate mmap-able array map memory lazily
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] An integer overflow in `array_map_mmap` on 32-bit architectures allows bypassing the VMA…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] An integer overflow in array_map_mmap on 32-bit architectures allows bypassing the VMA bounds check. Combined with pointer arithmetic truncation in the new array_map_mmap_fault handler, an attacker can map the struct bpf_array header into user space, leading to Local Privilege Escalation (LPE) by overwriting the BPF map's operations pointer. --
--- Patch [1]: [PATCH bpf-next 1/2] bpf: Populate mmap-able array map memory lazily --- commit 07e8d5ac0a564f432fdda75a2eb43f2790e3860c
Author: Song Liu <[email protected]>
bpf: Populate mmap-able array map memory lazily
This commit modifies mmap-able BPF array maps to populate memory lazily rather than upfront at map creation time. It adds a new map_mmap_fault callback to handle inserting pages on demand, reducing the mmap and munmap complexity to O(1) for large maps.
Quoted 12 lines
kernel/bpf/arraymap.c +0
-1@@ -576,7 +576,6 @@ static int array_map_check_btf(struct bpf_map *map,static int array_map_mmap (struct bpf_map *map,struct vm_area_struct *vma) { struct bpf_array *array = container_of(map, struct bpf_array, map); -pgoff_t pgoff =PAGE_ALIGN (sizeof (*array)) >>PAGE_SHIFT ;if (!(map->map_flags &BPF_F_MMAPABLE ))return -EINVAL ;
kernel/bpf/arraymap.c static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma) · line 585
Quoted 2 lines
kernel/bpf/arraymap.c +0
-0@@ -585,8 +584,42 @@ static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma) PAGE_ALIGN((u64)array->map.max_entries * array->elem_size))return -EINVAL ;
[Severity: Critical]
Does this bounds check in array_map_mmap overflow on 32-bit architectures?
Since the left-hand side calculation uses 32-bit unsigned long arithmetic, could an excessive page offset overflow the result to zero, thereby bypassing the check entirely?
Quoted 19 lines
kernel/bpf/arraymap.c +17
-2-return remap_vmalloc_range (vma,array_map_vmalloc_addr (array), - vma->vm_pgoff + pgoff); +/* +* The backing memory is vmalloc'ed up front, so instead of wiring up +* every PTE here we let array_map_mmap_fault() insert pages on demand. +* remap_vmalloc_range_partial() used to set these flags for us; keep +* them to preserve behavior (no VMA expansion, excluded from coredump). +*/ +vm_flags_set (vma,VM_DONTEXPAND |VM_DONTDUMP ); + +return 0 ; +} + +static vm_fault_t array_map_mmap_fault (struct bpf_map *map, +struct vm_fault *vmf) +{ + struct bpf_array *array = container_of(map, struct bpf_array, map); +pgoff_t pgoff =PAGE_ALIGN (sizeof (*array)) >>PAGE_SHIFT ; +struct page *page;
kernel/bpf/arraymap.c static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma) · line 613
Quoted 10 lines
kernel/bpf/arraymap.c +10
-0+void *kaddr; + +/* +* vmf->pgoff is the page offset within the mapping (vma->vm_pgoff plus +* the offset of the faulting address). The data pages start after the +* bpf_array header, hence the extra pgoff. This mirrors the offset that +* remap_vmalloc_range(vma, ..., vma->vm_pgoff + pgoff) used to apply. +*/ + kaddr =array_map_vmalloc_addr (array) + + ((u64)(vmf->pgoff + pgoff) <<PAGE_SHIFT );
[Severity: Critical]
Can this pointer arithmetic lead to truncation when calculating kaddr?
On 32-bit systems, array_map_vmalloc_addr returns a 32-bit pointer. If the 64-bit offset calculation is added to this pointer, does it truncate the upper bits? If an overflowed pgoff was passed from the mmap bounds check, could this truncation cause the mapping to wrap around and incorrectly expose the internal bpf_array header memory to user space?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Song Liu
reviewerPATCH 2/2 selftests/bpf: Add mmap/munmap benchmark for array maps
In reply to Song LiuAdd a benchmark that repeatedly mmap()s and munmap()s an mmap-able BPF array map (BPF_F_MMAPABLE) from a configurable number of threads. Both the number of worker threads and the size of the map are tunable via --nr-threads and --map-size, defaulting to 16 threads and 8MiB.
The benchmark is useful to observe how the cost of mmap()/munmap() scales with the mapping size, e.g. eager vs lazy page-table population.
--- tools/testing/selftests/bpf/Makefile | 1 + tools/testing/selftests/bpf/bench.c | 4 + .../bpf/benchs/bench_arraymap_mmap.c | 146 ++++++++++++++++++ .../bpf/benchs/run_bench_arraymap_mmap.sh | 15 ++ 4 files changed, 166 insertions(+) create mode 100644 tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_arraymap_mmap.sh
tools/testing/selftests/bpf/Makefile +1 -0
@@ -1003,6 +1003,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \ $(OUTPUT)/bench_bpf_timing.o \ $(OUTPUT)/bench_bpf_nop.o \ $(OUTPUT)/bench_xdp_lb.o \ + $(OUTPUT)/bench_arraymap_mmap.o \ $(OUTPUT)/usdt_1.o \ $(OUTPUT)/usdt_2.o \ #
tools/testing/selftests/bpf/bench.c +4 -0
@@ -287,6 +287,7 @@ extern struct argp bench_crypto_argp;extern struct argp bench_sockmap_argp;extern struct argp bench_lpm_trie_map_argp;extern struct argp bench_xdp_lb_argp; +extern struct argp bench_arraymap_mmap_argp;static const struct argp_child bench_parsers[] = { { &bench_ringbufs_argp,0 ,"Ring buffers benchmark" ,0 },
@@ -304,6 +305,7 @@ static const struct argp_child bench_parsers[] = { { &bench_sockmap_argp, 0, "bpf sockmap benchmark", 0 }, { &bench_lpm_trie_map_argp, 0, "LPM trie map benchmark", 0 }, { &bench_xdp_lb_argp, 0, "XDP load-balancer benchmark", 0 }, + { &bench_arraymap_mmap_argp, 0, "arraymap-mmap benchmark", 0 }, {}, };
@@ -582,6 +584,7 @@ extern const struct bench bench_lpm_trie_delete;extern const struct bench bench_lpm_trie_free;extern const struct bench bench_bpf_nop;extern const struct bench bench_xdp_lb; +extern const struct bench bench_arraymap_mmap;static const struct bench *benchs[] = { &bench_count_global,
@@ -665,6 +668,7 @@ static const struct bench *benchs[] = { &bench_lpm_trie_free, &bench_bpf_nop, &bench_xdp_lb, + &bench_arraymap_mmap, };static void find_benchmark (void )
tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c new file +146 -0
146 lines not on this page · raw message
tools/testing/selftests/bpf/benchs/run_bench_arraymap_mmap.sh new file +15 -0
@@ -0,0 +1,15 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +source ./benchs/run_common.sh + +set -eufo pipefail + +for t in 1 4 8 16; do +for s in 1048576 8388608 67108864; do +subtitle "nr_threads: $t, map_size: $s" + summarize_ops "arraymap-mmap: " \ + "$($RUN_BENCH --nr-threads $t --map-size $s arraymap-mmap)" + printf "\n" +done +done
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next 2/2] selftests/bpf: Add mmap/munmap benchmark for array maps
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Mutating `env.producer_cnt` inside `validate()` causes a heap buffer overflow in the core…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Mutating env.producer_cnt inside validate() causes a heap buffer overflow in the core benchmark runner framework. --
--- Patch [2]: [PATCH bpf-next 2/2] selftests/bpf: Add mmap/munmap benchmark for array maps --- commit 39f3ece9fcc36c46f5da11156d115c4340de663c
Author: Song Liu <[email protected]>
selftests/bpf: Add mmap/munmap benchmark for array maps
Add a benchmark that repeatedly mmap()s and munmap()s an mmap-able BPF array map (BPF_F_MMAPABLE) from a configurable number of threads. Both the number of worker threads and the size of the map are tunable via --nr-threads and --map-size, defaulting to 16 threads and 8MiB.
Quoted 6 lines
tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c new file +0
-0@@ -0,0 +1,146 @@
[ ... ]
Quoted 4 lines
tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c new file +4
-0+static void validate (void ) +{ +if (env.consumer_cnt !=0 ) { +fprintf (stderr,"benchmark doesn't support consumer!\n" );
tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c line 78
Quoted 10 lines
tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c new file +10
-0+exit (1 ); + } + +/* +* The number of worker threads is controlled by --nr-threads and +* drives the framework's producer machinery, so per-producer stats +* are reported correctly. +*/ + env.producer_cnt = args.nr_threads; +}
[Severity: Medium]
Does mutating env.producer_cnt inside validate() cause a heap buffer overflow?
The state.producers array is allocated in setup_benchmark() based on the initial env.producer_cnt:
tools/testing/selftests/bpf/bench.c:setup_benchmark() {
...
state.producers = calloc(env.producer_cnt, sizeof(*state.producers));
...
}When this validate callback overwrites env.producer_cnt with args.nr_threads (which defaults to 16), the core framework later iterates over the new count when spawning threads:
tools/testing/selftests/bpf/bench.c:setup_benchmark() {
...
for (i = 0; i < env.producer_cnt; i++) {
...
err = pthread_create(&state.producers[i], NULL,
bench->producer_thread, (void *)(long)i);
...
}This will write out of bounds since the array was originally sized for the unmodified env.producer_cnt.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Re: [PATCH bpf-next 2/2] selftests/bpf: Add mmap/munmap benchmark for array maps
In reply to Song LiuQuoted 155 lines · Song Liu
Add a benchmark that repeatedly mmap()s and munmap()s an mmap-able BPF array map (
BPF_F_MMAPABLE) from a configurable number of threads. Both the number of worker threads and the size of the map are tunable via --nr-threads and --map-size, defaulting to 16 threads and 8MiB.The benchmark is useful to observe how the cost of mmap()/munmap() scales with the mapping size, e.g. eager vs lazy page-table population.
Signed-off-by Song Liu <[email protected]>Assisted-by Claude:claude-opus-4-8--- tools/testing/selftests/bpf/Makefile | 1 + tools/testing/selftests/bpf/bench.c | 4 + .../bpf/benchs/bench_arraymap_mmap.c | 146 ++++++++++++++++++ .../bpf/benchs/run_bench_arraymap_mmap.sh | 15 ++ 4 files changed, 166 insertions(+) create mode 100644 tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_arraymap_mmap.shtools/testing/selftests/bpf/Makefile +1
-0@@ -1003,6 +1003,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \ $(OUTPUT)/bench_bpf_timing.o \ $(OUTPUT)/bench_bpf_nop.o \ $(OUTPUT)/bench_xdp_lb.o \ + $(OUTPUT)/bench_arraymap_mmap.o \ $(OUTPUT)/usdt_1.o \ $(OUTPUT)/usdt_2.o \ #tools/testing/selftests/bpf/bench.c +4
-0@@ -287,6 +287,7 @@ extern struct argp bench_crypto_argp;extern struct argp bench_sockmap_argp;extern struct argp bench_lpm_trie_map_argp;extern struct argp bench_xdp_lb_argp; +extern struct argp bench_arraymap_mmap_argp;static const struct argp_child bench_parsers[] = { { &bench_ringbufs_argp,0 ,"Ring buffers benchmark" ,0 },@@ -304,6 +305,7 @@ static const struct argp_child bench_parsers[] = { { &bench_sockmap_argp, 0, "bpf sockmap benchmark", 0 }, { &bench_lpm_trie_map_argp, 0, "LPM trie map benchmark", 0 }, { &bench_xdp_lb_argp, 0, "XDP load-balancer benchmark", 0 }, + { &bench_arraymap_mmap_argp, 0, "arraymap-mmap benchmark", 0 }, {}, };@@ -582,6 +584,7 @@ extern const struct bench bench_lpm_trie_delete;extern const struct bench bench_lpm_trie_free;extern const struct bench bench_bpf_nop;extern const struct bench bench_xdp_lb; +extern const struct bench bench_arraymap_mmap;static const struct bench *benchs[] = { &bench_count_global,@@ -665,6 +668,7 @@ static const struct bench *benchs[] = { &bench_lpm_trie_free, &bench_bpf_nop, &bench_xdp_lb, + &bench_arraymap_mmap, };static void find_benchmark (void )tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c new file +82
-0@@ -0,0 +1,146 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. */ +#include <argp.h> +#include <sys/mman.h> +#include "bench.h" + +/* Benchmark mmap()/munmap() of an mmap-able BPF array map from N threads. */ + +static struct ctx { +int map_fd; +size_t mmap_sz; +} ctx; + +static struct { +__u32 nr_threads; +__u64 map_size; +} args = { + .nr_threads =16 , + .map_size =8 *1024 *1024 ,/* 8 MiB */ +}; + +enum { +ARG_NR_THREADS =5000 , +ARG_MAP_SIZE =5001 , +}; + +static const struct argp_option opts[] = { + {"nr-threads" ,ARG_NR_THREADS ,"NR_THREADS" ,0 , +"Number of threads that mmap/munmap the map (default 16)" }, + {"map-size" ,ARG_MAP_SIZE ,"BYTES" ,0 , +"Size of the BPF array map in bytes (default 8388608)" }, + {}, +}; + +static error_t parse_arg (int key,char *arg,struct argp_state *state) +{ +switch (key) { +case ARG_NR_THREADS : + args.nr_threads =strtoul (arg,NULL ,0 ); +if (args.nr_threads ==0 ) { +fprintf (stderr,"Invalid nr-threads: %s\n" , arg); +argp_usage (state); + } +break ; +case ARG_MAP_SIZE : + args.map_size =strtoull (arg,NULL ,0 ); +if (args.map_size ==0 ) { +fprintf (stderr,"Invalid map-size: %s\n" , arg); +argp_usage (state); + } +break ; +default : +return ARGP_ERR_UNKNOWN ; + } + +return 0 ; +} + +/* exported into benchmark runner */ +const struct argp bench_arraymap_mmap_argp = { + .options = opts, + .parser = parse_arg, +}; + +static void validate (void ) +{ +if (env.consumer_cnt !=0 ) { +fprintf (stderr,"benchmark doesn't support consumer!\n" ); +exit (1 ); + } + +/* +* The number of worker threads is controlled by --nr-threads and +* drives the framework's producer machinery, so per-producer stats +* are reported correctly. +*/ + env.producer_cnt = args.nr_threads; +} + +static long hits; + +static void *producer (void *input)
tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c line 92
Quoted 10 lines
tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c new file +10
-0+{ +while (true) { +void *addr; + + addr =mmap (NULL , ctx.mmap_sz,PROT_READ |PROT_WRITE , +MAP_SHARED , ctx.map_fd,0 ); +if (addr ==MAP_FAILED ) { +fprintf (stderr,"mmap failed: %d\n" , -errno); +exit (1 ); + }
hi,
so now pages are populated only when accessed, benchmark shows this nicely:
before:
nr_threads: 1, map_size: 1048576 arraymap-mmap: throughput: 0.006 ± 0.000 M ops/s, latency: 172703.437 ns/op
after:
nr_threads: 1, map_size: 1048576 arraymap-mmap: throughput: 0.082 ± 0.002 M ops/s, latency: 12212.778 ns/op
would it make sense to add some map access in here (each page I guess) enabled by new option and measure the impact for accessed map ? I assume latency should be equal in such case
jirka
Quoted 79 lines
tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c new file +54
-0+if (munmap (addr, ctx.mmap_sz)) { +fprintf (stderr,"munmap failed: %d\n" , -errno); +exit (1 ); + } +atomic_inc (&hits); + } + +return NULL ; +} + +static void measure (struct bench_res *res) +{ + res->hits =atomic_swap (&hits,0 ); +} + +static void setup (void ) +{ + LIBBPF_OPTS(bpf_map_create_opts, opts, .map_flags = BPF_F_MMAPABLE); +long page_sz =sysconf (_SC_PAGESIZE); +__u32 value_size =sizeof (__u64); +__u32 max_entries; + +setup_libbpf (); + +/* +* mmap-able array maps round the value size up to 8 bytes and expose +* PAGE_ALIGN(max_entries * elem_size) bytes to user space. +*/ + max_entries = (args.map_size + value_size -1 ) / value_size; + ctx.mmap_sz = ((__u64 )max_entries * value_size + page_sz -1 ) & + ~(page_sz -1 ); + + ctx.map_fd =bpf_map_create (BPF_MAP_TYPE_ARRAY ,"mmap_array" , +sizeof (__u32), value_size, max_entries, + &opts); +if (ctx.map_fd <0 ) { +fprintf (stderr,"failed to create map: %d\n" , -errno); +exit (1 ); + } + +printf ("array map: %u entries, mmap size %zu bytes, %u threads\n" , + max_entries, ctx.mmap_sz, args.nr_threads); +} + +const struct bench bench_arraymap_mmap = { + .name ="arraymap-mmap" , + .argp = &bench_arraymap_mmap_argp, + .validate = validate, + .setup = setup, + .producer_thread = producer, + .measure = measure, + .report_progress = ops_report_progress, + .report_final = ops_report_final, +};tools/testing/selftests/bpf/benchs/run_bench_arraymap_mmap.sh new file +15
-0@@ -0,0 +1,15 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +source ./benchs/run_common.sh + +set -eufo pipefail + +for t in 1 4 8 16; do +for s in 1048576 8388608 67108864; do +subtitle "nr_threads: $t, map_size: $s" + summarize_ops "arraymap-mmap: " \ + "$($RUN_BENCH --nr-threads $t --map-size $s arraymap-mmap)" + printf "\n" +done +doneSignature · 1 line
2.53.0-Meta
Song Liu
reviewerRe: [PATCH bpf-next 2/2] selftests/bpf: Add mmap/munmap benchmark for array maps
On Thu, Jul 23, 2026 at 11:46 AM Jiri Olsa <[email protected]> wrote:
[...]
Quoted 20 lines
tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c line 92
Quoted 11 lines
tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c new file +11
-0+static void *producer (void *input) +{ +while (true) { +void *addr; + + addr =mmap (NULL , ctx.mmap_sz,PROT_READ |PROT_WRITE , +MAP_SHARED , ctx.map_fd,0 ); +if (addr ==MAP_FAILED ) { +fprintf (stderr,"mmap failed: %d\n" , -errno); +exit (1 ); + }hi,
so now pages are populated only when accessed, benchmark shows this nicely:before:
nr_threads: 1, map_size: 1048576 arraymap-mmap: throughput: 0.006 ą 0.000 M ops/s, latency: 172703.437 ns/op
Quoted 8 lines
after:
nr_threads: 1, map_size: 1048576 arraymap-mmap: throughput: 0.082 ą 0.002 M ops/s, latency: 12212.778 ns/opwould it make sense to add some map access in here (each page I guess) enabled by new option and measure the impact for accessed map ? I assume latency should be equal in such case.
If we add a read per page, which is the worst case:
before:
nr_threads: 1, map_size: 1048576
arraymap-mmap: throughput: 0.032 ± 0.000 M ops/s, latency: 31248.846 ns/op
nr_threads: 1, map_size: 8388608
arraymap-mmap: throughput: 0.003 ± 0.000 M ops/s, latency:
372565.622 ns/op
nr_threads: 1, map_size: 67108864
arraymap-mmap: throughput: 0.000 ± 0.000 M ops/s, latency:
2061083.005 ns/op
after:
nr_threads: 1, map_size: 1048576
arraymap-mmap: throughput: 0.006 ± 0.000 M ops/s, latency:
170794.193 ns/op
nr_threads: 1, map_size: 8388608
arraymap-mmap: throughput: 0.001 ± 0.000 M ops/s, latency:
1346389.229 ns/op
nr_threads: 1, map_size: 67108864
arraymap-mmap: throughput: 0.000 ± 0.000 M ops/s, latency:
10700389.105 ns/op
So filling all the pages in the mmap is faster than faulting one page at a time in many page faults. IOW, there is a performance regression in this case.
Thanks,
Song
Re: [PATCH bpf-next 2/2] selftests/bpf: Add mmap/munmap benchmark for array maps
Quoted 54 lines · Song Liu
On Thu, Jul 23, 2026 at 11:46 AM Jiri Olsa <[email protected]> wrote:
[...]Quoted 20 lines
tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c line 92
Quoted 11 lines
tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c new file +11
-0+static void *producer (void *input) +{ +while (true) { +void *addr; + + addr =mmap (NULL , ctx.mmap_sz,PROT_READ |PROT_WRITE , +MAP_SHARED , ctx.map_fd,0 ); +if (addr ==MAP_FAILED ) { +fprintf (stderr,"mmap failed: %d\n" , -errno); +exit (1 ); + }hi,
so now pages are populated only when accessed, benchmark shows this nicely:before:
nr_threads: 1, map_size: 1048576 arraymap-mmap: throughput: 0.006 ą 0.000 M ops/s, latency: 172703.437 ns/opQuoted 8 lines
after:
nr_threads: 1, map_size: 1048576 arraymap-mmap: throughput: 0.082 ą 0.002 M ops/s, latency: 12212.778 ns/opwould it make sense to add some map access in here (each page I guess) enabled by new option and measure the impact for accessed map ? I assume latency should be equal in such case.
If we add a read per page, which is the worst case:
before:
nr_threads: 1, map_size: 1048576arraymap-mmap: throughput: 0.032 ± 0.000 M ops/s, latency: 31248.846 ns/op
nr_threads: 1, map_size: 8388608arraymap-mmap: throughput: 0.003 ± 0.000 M ops/s, latency:
372565.622 ns/opnr_threads: 1, map_size: 67108864arraymap-mmap: throughput: 0.000 ± 0.000 M ops/s, latency:
2061083.005 ns/opafter:
nr_threads: 1, map_size: 1048576arraymap-mmap: throughput: 0.006 ± 0.000 M ops/s, latency:
170794.193 ns/opnr_threads: 1, map_size: 8388608arraymap-mmap: throughput: 0.001 ± 0.000 M ops/s, latency:
1346389.229 ns/op
Quoted 6 lines
nr_threads: 1, map_size: 67108864arraymap-mmap: throughput: 0.000 ± 0.000 M ops/s, latency:
10700389.105 ns/opSo filling all the pages in the mmap is faster than faulting one page at a time in many page faults. IOW, there is a performance regression in this case.
would it make sense to add new map flag for that?
jirka
Song Liu
reviewerRe: [PATCH bpf-next 2/2] selftests/bpf: Add mmap/munmap benchmark for array maps
On Fri, Jul 24, 2026 at 1:54 AM Jiri Olsa <[email protected]> wrote:
[...]
Quoted 23 lines
Quoted 23 lines
If we add a read per page, which is the worst case:
before:
nr_threads: 1, map_size: 1048576arraymap-mmap: throughput: 0.032 ± 0.000 M ops/s, latency: 31248.846 ns/op
nr_threads: 1, map_size: 8388608arraymap-mmap: throughput: 0.003 ± 0.000 M ops/s, latency:
372565.622 ns/opnr_threads: 1, map_size: 67108864arraymap-mmap: throughput: 0.000 ± 0.000 M ops/s, latency:
2061083.005 ns/opafter:
nr_threads: 1, map_size: 1048576arraymap-mmap: throughput: 0.006 ± 0.000 M ops/s, latency:
170794.193 ns/opnr_threads: 1, map_size: 8388608arraymap-mmap: throughput: 0.001 ± 0.000 M ops/s, latency:
1346389.229 ns/op
Quoted 8 lines
Quoted 6 lines
nr_threads: 1, map_size: 67108864arraymap-mmap: throughput: 0.000 ± 0.000 M ops/s, latency:
10700389.105 ns/opSo filling all the pages in the mmap is faster than faulting one page at a time in many page faults. IOW, there is a performance regression in this case.
would it make sense to add new map flag for that?
If possible, I would avoid adding a new flag for this. I think the regression is on a very special corner case. BPF arena has similar behavior (fill in page table on fault). So it should be OK to have array map do the same. WDYT?
Thanks,
Song
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next 2/2] selftests/bpf: Add mmap/munmap benchmark for array maps
Quoted 32 lines · Song Liu
On Fri, Jul 24, 2026 at 1:54 AM Jiri Olsa <[email protected]> wrote:
[...]Quoted 30 lines
Quoted 29 lines
If we add a read per page, which is the worst case:
before:
nr_threads: 1, map_size: 1048576arraymap-mmap: throughput: 0.032 ± 0.000 M ops/s, latency: 31248.846 ns/op
nr_threads: 1, map_size: 8388608arraymap-mmap: throughput: 0.003 ± 0.000 M ops/s, latency:
372565.622 ns/opnr_threads: 1, map_size: 67108864arraymap-mmap: throughput: 0.000 ± 0.000 M ops/s, latency:
2061083.005 ns/opafter:
nr_threads: 1, map_size: 1048576arraymap-mmap: throughput: 0.006 ± 0.000 M ops/s, latency:
170794.193 ns/opnr_threads: 1, map_size: 8388608arraymap-mmap: throughput: 0.001 ± 0.000 M ops/s, latency:
1346389.229 ns/opnr_threads: 1, map_size: 67108864arraymap-mmap: throughput: 0.000 ± 0.000 M ops/s, latency:
10700389.105 ns/opSo filling all the pages in the mmap is faster than faulting one page at a time in many page faults. IOW, there is a performance regression in this case.
Quoted 7 lines
Quoted 1 line
would it make sense to add new map flag for that?
If possible, I would avoid adding a new flag for this. I think the regression is on a very special corner case. BPF arena has similar behavior (fill in page table on fault). So it should be OK to have array map do the same. WDYT?
I think it would make sense to compare cumulative cost of mmap+read-every-page. I am sure it would also be lower before this change, due to the added context switches for fault on every page access that would be amortized before this, but may give a better picture about the difference.
I do feel like it's a corner case realistically, and after the first fault the page remains mapped (and users who care about pages being present can go the route of passing MAP_POPULATE). I am assuming passing that before was a noop, so it can be added to code in a backwards compatible fashion if someone really depends on this?
Quoted 2 lines
Thanks,
Song
Song Liu
reviewerRe: [PATCH bpf-next 2/2] selftests/bpf: Add mmap/munmap benchmark for array maps
Quoted 47 lines · Kumar Kartikeya Dwivedi
Quoted 32 lines · Song Liu
On Fri, Jul 24, 2026 at 1:54 AM Jiri Olsa <[email protected]> wrote:
[...]Quoted 30 lines
Quoted 29 lines
If we add a read per page, which is the worst case:
before:
nr_threads: 1, map_size: 1048576arraymap-mmap: throughput: 0.032 ± 0.000 M ops/s, latency: 31248.846 ns/op
nr_threads: 1, map_size: 8388608arraymap-mmap: throughput: 0.003 ± 0.000 M ops/s, latency:
372565.622 ns/opnr_threads: 1, map_size: 67108864arraymap-mmap: throughput: 0.000 ± 0.000 M ops/s, latency:
2061083.005 ns/opafter:
nr_threads: 1, map_size: 1048576arraymap-mmap: throughput: 0.006 ± 0.000 M ops/s, latency:
170794.193 ns/opnr_threads: 1, map_size: 8388608arraymap-mmap: throughput: 0.001 ± 0.000 M ops/s, latency:
1346389.229 ns/opnr_threads: 1, map_size: 67108864arraymap-mmap: throughput: 0.000 ± 0.000 M ops/s, latency:
10700389.105 ns/opSo filling all the pages in the mmap is faster than faulting one page at a time in many page faults. IOW, there is a performance regression in this case.
Quoted 7 lines
Quoted 1 line
would it make sense to add new map flag for that?
If possible, I would avoid adding a new flag for this. I think the regression is on a very special corner case. BPF arena has similar behavior (fill in page table on fault). So it should be OK to have array map do the same. WDYT?
I think it would make sense to compare cumulative cost of mmap+read-every-page. I am sure it would also be lower before this change, due to the added context switches for fault on every page access that would be amortized before this, but may give a better picture about the difference.
Quoted 5 lines
I do feel like it's a corner case realistically, and after the first fault the page remains mapped (and users who care about pages being present can go the route of passing
MAP_POPULATE). I am assuming passing that before was a noop, so it can be added to code in a backwards compatible fashion if someone really depends on this?
Given there is MAP_POPULATE flag, yes, the user can preserve the old behavior with this flag. While this requires users to make changes for this corner case, I think it is still cleaner than adding a BPF map flag for this. If there are no strong objections, I will keep this behavior in v3.
Thanks,
Song
Song Liu
reviewerRe: [PATCH bpf-next 2/2] selftests/bpf: Add mmap/munmap benchmark for array maps
On Mon, Jul 27, 2026 at 5:19 PM Song Liu <[email protected]> wrote:
[...]
Quoted 11 lines
Quoted 10 lines
I think it would make sense to compare cumulative cost of mmap+read-every-page. I am sure it would also be lower before this change, due to the added context switches for fault on every page access that would be amortized before this, but may give a better picture about the difference.
I do feel like it's a corner case realistically, and after the first fault the page remains mapped (and users who care about pages being present can go the route of passing
MAP_POPULATE). I am assuming passing that before was a noop, so it can be added to code in a backwards compatible fashion if someone really depends on this?
Quoted 4 lines
Given there is
MAP_POPULATEflag, yes, the user can preserve the old behavior with this flag. While this requires users to make changes for this corner case, I think it is still cleaner than adding a BPF map flag for this. If there are no strong objections, I will keep this behavior in v3.
This turns out to be a little more complicated:
1. Bad news: MAP_POPULATE is not a noop for old kernels. The pre fault mechanism still adds some overhead (for almost no gain). For 8MiB mmap, MAP_POPULATE addes about 80us (raw data below).
2. Good news: We can further improve MAP_POPULATE with a batch pre fault (vm_operations_struct->map_pages). With batch pre fault, mmap then accessing 8MiB is about 34us slower than the best option before the set:
(before: no MAP_POPULATE => access all pages: 252us;
after: MAP_POPULATE => access all pages: 286us).
Note that, all the above are worst cases: we need to access all pages and fill the page table one way or another. If we don't need to access all pages, the change saves a lot (220us => 1.2us) for the mmap.
Overall, I think this is a net win for most use cases.
Thanks,
Song
Raw data:
All of the following are for 8MiB, access times are in microseconds (us).
Before:
no MAP_POPULATE (do not access pages) 220us
no MAP_POPULATE => access all pages: 252usMAP_POPULATE (do not access pages) 304usMAP_POPULATE => access all pages: 334us
After:
no MAP_POPULATE (do not access pages): 1.2us
no MAP_POPULATE => access all pages: 543usMAP_POPULATE (do not access pages): 262usMAP_POPULATE => access all pages: 286us
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next 2/2] selftests/bpf: Add mmap/munmap benchmark for array maps
Quoted 24 lines · Song Liu
On Mon, Jul 27, 2026 at 5:19 PM Song Liu <[email protected]> wrote:
[...]Quoted 11 lines
Quoted 10 lines
I think it would make sense to compare cumulative cost of mmap+read-every-page. I am sure it would also be lower before this change, due to the added context switches for fault on every page access that would be amortized before this, but may give a better picture about the difference.
I do feel like it's a corner case realistically, and after the first fault the page remains mapped (and users who care about pages being present can go the route of passing
MAP_POPULATE). I am assuming passing that before was a noop, so it can be added to code in a backwards compatible fashion if someone really depends on this?Quoted 4 lines
Given there is
MAP_POPULATEflag, yes, the user can preserve the old behavior with this flag. While this requires users to make changes for this corner case, I think it is still cleaner than adding a BPF map flag for this. If there are no strong objections, I will keep this behavior in v3.This turns out to be a little more complicated:
1. Bad news:
MAP_POPULATEis not a noop for old kernels. The pre fault mechanism still adds some overhead (for almost no gain). For 8MiB mmap,MAP_POPULATEaddes about 80us (raw data below).
Quoted 6 lines
2. Good news: We can further improve
MAP_POPULATEwith a batch pre fault (vm_operations_struct->map_pages). With batch pre fault, mmap then accessing 8MiB is about 34us slower than the best option before the set:
(before: noMAP_POPULATE=> access all pages: 252us;
after:MAP_POPULATE=> access all pages: 286us).
Interesting, though I figure we can let it be, since repeated mmap()+MAP_POPULATE on very big array maps should not be a common occurence. We can always revisit in the future though.
Quoted 7 lines
Note that, all the above are worst cases: we need to access all pages and fill the page table one way or another. If we don't need to access all pages, the change saves a lot (220us => 1.2us) for the mmap.
Overall, I think this is a net win for most use cases.
Thanks for the analysis and data, makes a lot of sense, and agreed about going forward with the set.
Quoted 18 lines
Thanks,
SongRaw data:
All of the following are for 8MiB, access times are in microseconds (us).
Before:
noMAP_POPULATE(do not access pages) 220us
noMAP_POPULATE=> access all pages: 252usMAP_POPULATE(do not access pages) 304usMAP_POPULATE=> access all pages: 334usAfter:
noMAP_POPULATE(do not access pages): 1.2us
noMAP_POPULATE=> access all pages: 543usMAP_POPULATE(do not access pages): 262usMAP_POPULATE=> access all pages: 286us
Re: [PATCH bpf-next 2/2] selftests/bpf: Add mmap/munmap benchmark for array maps
In reply to Song LiuQuoted 31 lines · Song Liu
On Mon, Jul 27, 2026 at 5:19 PM Song Liu <[email protected]> wrote:
[...]Quoted 11 lines
Quoted 10 lines
I think it would make sense to compare cumulative cost of mmap+read-every-page. I am sure it would also be lower before this change, due to the added context switches for fault on every page access that would be amortized before this, but may give a better picture about the difference.
I do feel like it's a corner case realistically, and after the first fault the page remains mapped (and users who care about pages being present can go the route of passing
MAP_POPULATE). I am assuming passing that before was a noop, so it can be added to code in a backwards compatible fashion if someone really depends on this?Quoted 4 lines
Given there is
MAP_POPULATEflag, yes, the user can preserve the old behavior with this flag. While this requires users to make changes for this corner case, I think it is still cleaner than adding a BPF map flag for this. If there are no strong objections, I will keep this behavior in v3.This turns out to be a little more complicated:
1. Bad news:
MAP_POPULATEis not a noop for old kernels. The pre fault mechanism still adds some overhead (for almost no gain). For 8MiB mmap,MAP_POPULATEaddes about 80us (raw data below).2. Good news: We can further improve
MAP_POPULATEwith a batch pre fault (vm_operations_struct->map_pages). With batch pre fault, mmap then accessing 8MiB is about 34us slower than the best option before the set:
(before: noMAP_POPULATE=> access all pages: 252us;
after:MAP_POPULATE=> access all pages: 286us).
Quoted 5 lines
Note that, all the above are worst cases: we need to access all pages and fill the page table one way or another. If we don't need to access all pages, the change saves a lot (220us => 1.2us) for the mmap.
Overall, I think this is a net win for most use cases.
sounds good, thanks for all the details
jirka
Quoted 19 lines
Thanks,
SongRaw data:
All of the following are for 8MiB, access times are in microseconds (us).
Before:
noMAP_POPULATE(do not access pages) 220us
noMAP_POPULATE=> access all pages: 252usMAP_POPULATE(do not access pages) 304usMAP_POPULATE=> access all pages: 334usAfter:
noMAP_POPULATE(do not access pages): 1.2us
noMAP_POPULATE=> access all pages: 543usMAP_POPULATE(do not access pages): 262usMAP_POPULATE=> access all pages: 286us