Skip to contentSkip to conversation

bpf: Populate mmap-able array maps lazily

PATCH Superseded Song Liu 2 participants 7 messages Jul 22 – Jul 28

Subsystems (4)

v2 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Song Liu

author reviewer
#0 lore raw

v2 cover bpf: Populate mmap-able array maps lazily

An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory vmalloc'ed up front at map creation time. array_map_mmap() then wires up the whole mapping eagerly via remap_vmalloc_range(), which calls vm_insert_page() for every page of the map. This makes every mmap() cost O(number of pages): an 8MiB map inserts 2048 PTEs on each mmap() and tears them all down again on munmap(), even when user space only touches a few pages (or none at all).

This series makes the mapping populate lazily instead, the same way the arena map already does, turning mmap()/munmap() of an mmap-able array from O(map size) into O(1). The pages are inserted on demand by a new map_mmap_fault() callback, dispatched from the shared bpf_map_default_vmops so that the existing VMA open/close accounting (VM_MAYWRITE write-active tracking, freeze handling) stays centralized.

Callers that want the pages populated up front can still request that explicitly with MAP_POPULATE, matching regular file semantics. Kernel-side access to the map (via the vmalloc address) is unaffected.

Patch 1 implements the lazy faulting. Patch 2 adds an "arraymap-mmap" benchmark that mmap()s/munmap()s an mmap-able array map, useful to observe how the cost scales with the mapping size.

Changes in v2:
- Compute the mapping offset with 64-bit arithmetic in array_map_mmap() and the fault handler so that a large vm_pgoff cannot overflow the bounds check on 32-bit architectures. remap_vmalloc_range(), which is no longer used, used to guard against this.
- Benchmark: use the standard -p/--producers option for the thread count instead of a custom --nr-threads option. The v1 approach set env.producer_cnt from validate(), which runs after the framework has already sized the producers array, causing an out-of-bounds write.
(Both issues were pointed out by the Sashiko AI review.)

v1: https://lore.kernel.org/bpf/[email protected]/

Benchmark results (single producer thread, ./bench -p1):

Before the patch
nr_threads: 1, map_size: 1048576
arraymap-mmap: throughput: 0.034 +/- 0.000 M ops/s, latency: 29187.164 ns/op

nr_threads: 1, map_size: 8388608
arraymap-mmap: throughput: 0.005 +/- 0.000 M ops/s, latency: 220910.150 ns/op

nr_threads: 1, map_size: 67108864
arraymap-mmap: throughput: 0.001 +/- 0.000 M ops/s, latency: 1765083.440 ns/op

After the patch
nr_threads: 1, map_size: 1048576
arraymap-mmap: throughput: 0.838 +/- 0.001 M ops/s, latency: 1193.553 ns/op

nr_threads: 1, map_size: 8388608
arraymap-mmap: throughput: 0.811 +/- 0.002 M ops/s, latency: 1232.728 ns/op

nr_threads: 1, map_size: 67108864
arraymap-mmap: throughput: 0.747 +/- 0.004 M ops/s, latency: 1339.051 ns/op

Song Liu (2):
bpf: Populate mmap-able array map memory lazily selftests/bpf: Add mmap/munmap benchmark for array maps

 include/linux/bpf.h                           |   1 +
 kernel/bpf/arraymap.c                         |  43 +++++-
 kernel/bpf/syscall.c                          |  17 +++
 tools/testing/selftests/bpf/Makefile          |   1 +
 tools/testing/selftests/bpf/bench.c           |   4 +
 .../bpf/benchs/bench_arraymap_mmap.c          | 127 ++++++++++++++++++
 .../bpf/benchs/run_bench_arraymap_mmap.sh     |  15 +++
 7 files changed, 204 insertions(+), 4 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c
 create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_arraymap_mmap.sh

--
2.53.0-Meta

author reviewer
#1 lore raw

PATCH v2 1/2 bpf: Populate mmap-able array map memory lazily

An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory vmalloc'ed up front at map creation time. array_map_mmap() then wired up the whole mapping eagerly via remap_vmalloc_range(), which calls vm_insert_page() for every page of the map. For large maps this makes every mmap() O(number of pages): an 8MiB map inserts 2048 PTEs per mmap() and tears them all down again on munmap(), even when user space only touches a few pages (or none at all).

Populate the mapping lazily instead, the same way the arena map already does. array_map_mmap() now only performs the bounds check and returns, leaving the PTEs unpopulated; the pages are inserted on demand by a new array_map_mmap_fault() handler. Because the memory is already resident, the fault handler simply resolves the vmalloc page and hands it to the fault path.

To keep the existing VMA open/close accounting (VM_MAYWRITE write-active tracking, freeze handling) centralized, the fault handler is reached through a new optional ->map_mmap_fault callback dispatched from the shared bpf_map_default_vmops, rather than each map installing its own vm_operations_struct.

remap_vmalloc_range() previously guarded against offset overflow; since it is no longer used, array_map_mmap() and the fault handler compute the mapping offset with 64-bit arithmetic so that a large vm_pgoff cannot overflow the bounds check on 32-bit architectures.

This turns mmap()/munmap() of an mmap-able array from O(map size) into O(1). Callers that want the pages populated up front can still request that explicitly with MAP_POPULATE, matching regular file semantics. Kernel-side access to the map (via the vmalloc address) is unaffected.

Assisted-by Claude:claude-opus-4-8
---
 include/linux/bpf.h   |  1 +
 kernel/bpf/arraymap.c | 43 +++++++++++++++++++++++++++++++++++++++----
 kernel/bpf/syscall.c  | 17 +++++++++++++++++
 3 files changed, 57 insertions(+), 4 deletions(-)
include/linux/bpf.h +1 -0
index e066f44a9c05..b536f3c9e103 100644
@@ -145,6 +145,7 @@ struct bpf_map_ops {
 	int (*map_direct_value_meta)(const struct bpf_map *map,
 				     u64 imm, u32 *off);
 	int (*map_mmap)(struct bpf_map *map, struct vm_area_struct *vma);
+	vm_fault_t (*map_mmap_fault)(struct bpf_map *map, struct vm_fault *vmf);
 	__poll_t (*map_poll)(struct bpf_map *map, struct file *filp,
 			     struct poll_table_struct *pts);
 	unsigned long (*map_get_unmapped_area)(struct file *filep, unsigned long addr,
kernel/bpf/arraymap.c +39 -4

62 lines not on this page · raw message

kernel/bpf/syscall.c +17 -0
index 0ff9e3aa293d..fb0e1f55c487 100644
@@ -1077,9 +1077,26 @@ static void bpf_map_mmap_close(struct vm_area_struct *vma)
 		bpf_map_write_active_dec(map);
 }
 
+/*
+ * Called for maps that populate their memory-mapped region lazily, i.e. that
+ * return without wiring up any PTEs from their ->map_mmap callback. Maps that
+ * populate the whole VMA eagerly (e.g. via remap_vmalloc_range()) never reach
+ * here.
+ */
+static vm_fault_t bpf_map_mmap_fault(struct vm_fault *vmf)
+{
+	struct bpf_map *map = vmf->vma->vm_private_data;
+
+	if (!map->ops->map_mmap_fault)
+		return VM_FAULT_SIGBUS;
+
+	return map->ops->map_mmap_fault(map, vmf);
+}
+
 static const struct vm_operations_struct bpf_map_default_vmops = {
 	.open		= bpf_map_mmap_open,
 	.close		= bpf_map_mmap_close,
+	.fault		= bpf_map_mmap_fault,
 };
 
 static int bpf_map_mmap(struct file *filp, struct vm_area_struct *vma)
Signature · 1 line
2.53.0-Meta
#2 lore raw

Re: [PATCH v2 bpf-next 1/2] bpf: Populate mmap-able array map memory lazily

Quoted 91 lines · Song Liu

An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory vmalloc'ed up front at map creation time. array_map_mmap() then wired up the whole mapping eagerly via remap_vmalloc_range(), which calls vm_insert_page() for every page of the map. For large maps this makes every mmap() O(number of pages): an 8MiB map inserts 2048 PTEs per mmap() and tears them all down again on munmap(), even when user space only touches a few pages (or none at all).

Populate the mapping lazily instead, the same way the arena map already does. array_map_mmap() now only performs the bounds check and returns, leaving the PTEs unpopulated; the pages are inserted on demand by a new array_map_mmap_fault() handler. Because the memory is already resident, the fault handler simply resolves the vmalloc page and hands it to the fault path.

To keep the existing VMA open/close accounting (VM_MAYWRITE write-active tracking, freeze handling) centralized, the fault handler is reached through a new optional ->map_mmap_fault callback dispatched from the shared bpf_map_default_vmops, rather than each map installing its own vm_operations_struct.

remap_vmalloc_range() previously guarded against offset overflow; since it is no longer used, array_map_mmap() and the fault handler compute the mapping offset with 64-bit arithmetic so that a large vm_pgoff cannot overflow the bounds check on 32-bit architectures.

This turns mmap()/munmap() of an mmap-able array from O(map size) into O(1). Callers that want the pages populated up front can still request that explicitly with MAP_POPULATE, matching regular file semantics. Kernel-side access to the map (via the vmalloc address) is unaffected.

Assisted-by Claude:claude-opus-4-8
---
 include/linux/bpf.h   |  1 +
 kernel/bpf/arraymap.c | 43 +++++++++++++++++++++++++++++++++++++++----
 kernel/bpf/syscall.c  | 17 +++++++++++++++++
 3 files changed, 57 insertions(+), 4 deletions(-)
include/linux/bpf.h +1 -0
index e066f44a9c05..b536f3c9e103 100644
@@ -145,6 +145,7 @@ struct bpf_map_ops {
        int (*map_direct_value_meta)(const struct bpf_map *map,
                                     u64 imm, u32 *off);
        int (*map_mmap)(struct bpf_map *map, struct vm_area_struct *vma);
+       vm_fault_t (*map_mmap_fault)(struct bpf_map *map, struct vm_fault *vmf);
        __poll_t (*map_poll)(struct bpf_map *map, struct file *filp,
                             struct poll_table_struct *pts);
        unsigned long (*map_get_unmapped_area)(struct file *filep, unsigned long addr,
kernel/bpf/arraymap.c +20 -4
index 248b4818178c..59fda56062a9 100644
@@ -576,17 +576,51 @@ static int array_map_check_btf(struct bpf_map *map,
 static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma)
 {
        struct bpf_array *array = container_of(map, struct bpf_array, map);
-       pgoff_t pgoff = PAGE_ALIGN(sizeof(*array)) >> PAGE_SHIFT;
 
        if (!(map->map_flags & BPF_F_MMAPABLE))
                return -EINVAL;
 
-       if (vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
+       /* use u64 math so the offset cannot overflow on 32-bit archs */
+       if ((u64)vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
            PAGE_ALIGN((u64)array->map.max_entries * array->elem_size))
                return -EINVAL;
 
-       return remap_vmalloc_range(vma, array_map_vmalloc_addr(array),
-                                  vma->vm_pgoff + pgoff);
+       /*
+        * The backing memory is vmalloc'ed up front, so instead of wiring up
+        * every PTE here we let array_map_mmap_fault() insert pages on demand.
+        * remap_vmalloc_range_partial() used to set these flags for us; keep
+        * them to preserve behavior (no VMA expansion, excluded from coredump).
+        */
+       vm_flags_set(vma, VM_DONTEXPAND | VM_DONTDUMP);
+
+       return 0;
+}
+
+static vm_fault_t array_map_mmap_fault(struct bpf_map *map,
+                                      struct vm_fault *vmf)
+{
+       struct bpf_array *array = container_of(map, struct bpf_array, map);
+       pgoff_t pgoff = PAGE_ALIGN(sizeof(*array)) >> PAGE_SHIFT;
+       struct page *page;
+       void *kaddr;

kernel/bpf/arraymap.c static int array_map_check_btf(struct bpf_map *map, · line 615

Quoted 10 lines
kernel/bpf/arraymap.c +10 -0
+
+       /*
+        * vmf->pgoff is the page offset within the mapping (vma->vm_pgoff plus
+        * the offset of the faulting address). The data pages start after the
+        * bpf_array header, hence the extra pgoff. This mirrors the offset that
+        * remap_vmalloc_range(vma, ..., vma->vm_pgoff + pgoff) used to apply.
+        */
+       kaddr = array_map_vmalloc_addr(array) +
+               (((u64)vmf->pgoff + pgoff) << PAGE_SHIFT);
+

remap_vmalloc_range() uses careful check_shl_overflow() and check_add_overflow() checks, while we just (u64) cast and go... should we be more careful and avoid some weird edge cases. Not sure if that's possible, but maybe better safe than sorry?

Quoted 53 lines
kernel/bpf/arraymap.c +9 -0
+       page = vmalloc_to_page(kaddr);
+       if (!page)
+               return VM_FAULT_SIGBUS;
+
+       get_page(page);
+       vmf->page = page;
+
+       return 0;
 }
 
 static bool array_map_meta_equal(const struct bpf_map *meta0,
@@ -812,6 +846,7 @@ const struct bpf_map_ops array_map_ops = {
        .map_direct_value_addr = array_map_direct_value_addr,
        .map_direct_value_meta = array_map_direct_value_meta,
        .map_mmap = array_map_mmap,
+       .map_mmap_fault = array_map_mmap_fault,
        .map_seq_show_elem = array_map_seq_show_elem,
        .map_check_btf = array_map_check_btf,
        .map_lookup_batch = generic_map_lookup_batch,
kernel/bpf/syscall.c +17 -1
index 0ff9e3aa293d..fb0e1f55c487 100644
@@ -1077,9 +1077,26 @@ static void bpf_map_mmap_close(struct vm_area_struct *vma)
                bpf_map_write_active_dec(map);
 }
 
+/*
+ * Called for maps that populate their memory-mapped region lazily, i.e. that
+ * return without wiring up any PTEs from their ->map_mmap callback. Maps that
+ * populate the whole VMA eagerly (e.g. via remap_vmalloc_range()) never reach
+ * here.
+ */
+static vm_fault_t bpf_map_mmap_fault(struct vm_fault *vmf)
+{
+       struct bpf_map *map = vmf->vma->vm_private_data;
+
+       if (!map->ops->map_mmap_fault)
+               return VM_FAULT_SIGBUS;
+
+       return map->ops->map_mmap_fault(map, vmf);
+}
+
 static const struct vm_operations_struct bpf_map_default_vmops = {
        .open           = bpf_map_mmap_open,
        .close          = bpf_map_mmap_close,
+       .fault          = bpf_map_mmap_fault,
 };
 
 static int bpf_map_mmap(struct file *filp, struct vm_area_struct *vma)
--

2.53.0-Meta

author reviewer
#3 lore raw

Re: [PATCH v2 bpf-next 1/2] bpf: Populate mmap-able array map memory lazily

Quoted 100 lines · Andrii Nakryiko
Quoted 98 lines · Song Liu

An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory vmalloc'ed up front at map creation time. array_map_mmap() then wired up the whole mapping eagerly via remap_vmalloc_range(), which calls vm_insert_page() for every page of the map. For large maps this makes every mmap() O(number of pages): an 8MiB map inserts 2048 PTEs per mmap() and tears them all down again on munmap(), even when user space only touches a few pages (or none at all).

Populate the mapping lazily instead, the same way the arena map already does. array_map_mmap() now only performs the bounds check and returns, leaving the PTEs unpopulated; the pages are inserted on demand by a new array_map_mmap_fault() handler. Because the memory is already resident, the fault handler simply resolves the vmalloc page and hands it to the fault path.

To keep the existing VMA open/close accounting (VM_MAYWRITE write-active tracking, freeze handling) centralized, the fault handler is reached through a new optional ->map_mmap_fault callback dispatched from the shared bpf_map_default_vmops, rather than each map installing its own vm_operations_struct.

remap_vmalloc_range() previously guarded against offset overflow; since it is no longer used, array_map_mmap() and the fault handler compute the mapping offset with 64-bit arithmetic so that a large vm_pgoff cannot overflow the bounds check on 32-bit architectures.

This turns mmap()/munmap() of an mmap-able array from O(map size) into O(1). Callers that want the pages populated up front can still request that explicitly with MAP_POPULATE, matching regular file semantics. Kernel-side access to the map (via the vmalloc address) is unaffected.

Assisted-by Claude:claude-opus-4-8
---
 include/linux/bpf.h   |  1 +
 kernel/bpf/arraymap.c | 43 +++++++++++++++++++++++++++++++++++++++----
 kernel/bpf/syscall.c  | 17 +++++++++++++++++
 3 files changed, 57 insertions(+), 4 deletions(-)
include/linux/bpf.h +1 -0
index e066f44a9c05..b536f3c9e103 100644
@@ -145,6 +145,7 @@ struct bpf_map_ops {
        int (*map_direct_value_meta)(const struct bpf_map *map,
                                     u64 imm, u32 *off);
        int (*map_mmap)(struct bpf_map *map, struct vm_area_struct *vma);
+       vm_fault_t (*map_mmap_fault)(struct bpf_map *map, struct vm_fault *vmf);
        __poll_t (*map_poll)(struct bpf_map *map, struct file *filp,
                             struct poll_table_struct *pts);
        unsigned long (*map_get_unmapped_area)(struct file *filep, unsigned long addr,
kernel/bpf/arraymap.c +27 -4
index 248b4818178c..59fda56062a9 100644
@@ -576,17 +576,51 @@ static int array_map_check_btf(struct bpf_map *map,
 static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma)
 {
        struct bpf_array *array = container_of(map, struct bpf_array, map);
-       pgoff_t pgoff = PAGE_ALIGN(sizeof(*array)) >> PAGE_SHIFT;
 
        if (!(map->map_flags & BPF_F_MMAPABLE))
                return -EINVAL;
 
-       if (vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
+       /* use u64 math so the offset cannot overflow on 32-bit archs */
+       if ((u64)vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
            PAGE_ALIGN((u64)array->map.max_entries * array->elem_size))
                return -EINVAL;
 
-       return remap_vmalloc_range(vma, array_map_vmalloc_addr(array),
-                                  vma->vm_pgoff + pgoff);
+       /*
+        * The backing memory is vmalloc'ed up front, so instead of wiring up
+        * every PTE here we let array_map_mmap_fault() insert pages on demand.
+        * remap_vmalloc_range_partial() used to set these flags for us; keep
+        * them to preserve behavior (no VMA expansion, excluded from coredump).
+        */
+       vm_flags_set(vma, VM_DONTEXPAND | VM_DONTDUMP);
+
+       return 0;
+}
+
+static vm_fault_t array_map_mmap_fault(struct bpf_map *map,
+                                      struct vm_fault *vmf)
+{
+       struct bpf_array *array = container_of(map, struct bpf_array, map);
+       pgoff_t pgoff = PAGE_ALIGN(sizeof(*array)) >> PAGE_SHIFT;
+       struct page *page;
+       void *kaddr;
+
+       /*
+        * vmf->pgoff is the page offset within the mapping (vma->vm_pgoff plus
+        * the offset of the faulting address). The data pages start after the
+        * bpf_array header, hence the extra pgoff. This mirrors the offset that
+        * remap_vmalloc_range(vma, ..., vma->vm_pgoff + pgoff) used to apply.
+        */
Quoted 8 lines

kernel/bpf/arraymap.c static int array_map_check_btf(struct bpf_map *map, · line 615

Quoted 3 lines
kernel/bpf/arraymap.c +3 -0
+       kaddr = array_map_vmalloc_addr(array) +
+               (((u64)vmf->pgoff + pgoff) << PAGE_SHIFT);
+

remap_vmalloc_range() uses careful check_shl_overflow() and check_add_overflow() checks, while we just (u64) cast and go... should we be more careful and avoid some weird edge cases. Not sure if that's possible, but maybe better safe than sorry?

Good catch! Let me add the check in v3.

Thanks,
Song

author reviewer
#4 lore raw

PATCH v2 2/2 selftests/bpf: Add mmap/munmap benchmark for array maps

In reply to Song Liu

Add a benchmark that repeatedly mmap()s and munmap()s an mmap-able BPF array map (BPF_F_MMAPABLE). The number of worker threads is controlled by the standard -p/--producers option, and the size of the map by --map-size (default 8MiB).

The benchmark is useful to observe how the cost of mmap()/munmap() scales with the mapping size, e.g. eager vs lazy page-table population.

Assisted-by Claude:claude-opus-4-8
---
 tools/testing/selftests/bpf/Makefile          |   1 +
 tools/testing/selftests/bpf/bench.c           |   4 +
 .../bpf/benchs/bench_arraymap_mmap.c          | 127 ++++++++++++++++++
 .../bpf/benchs/run_bench_arraymap_mmap.sh     |  15 +++
 4 files changed, 147 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c
 create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_arraymap_mmap.sh
tools/testing/selftests/bpf/Makefile +1 -0
index 55d394438705..b6593fb1135b 100644
@@ -1003,6 +1003,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \
 		 $(OUTPUT)/bench_bpf_timing.o \
 		 $(OUTPUT)/bench_bpf_nop.o \
 		 $(OUTPUT)/bench_xdp_lb.o \
+		 $(OUTPUT)/bench_arraymap_mmap.o \
 		 $(OUTPUT)/usdt_1.o \
 		 $(OUTPUT)/usdt_2.o \
 		 #
tools/testing/selftests/bpf/bench.c +4 -0
index 3d9d2cd7764b..192d5fcb7a74 100644
@@ -287,6 +287,7 @@ extern struct argp bench_crypto_argp;
 extern struct argp bench_sockmap_argp;
 extern struct argp bench_lpm_trie_map_argp;
 extern struct argp bench_xdp_lb_argp;
+extern struct argp bench_arraymap_mmap_argp;
 
 static const struct argp_child bench_parsers[] = {
 	{ &bench_ringbufs_argp, 0, "Ring buffers benchmark", 0 },
@@ -304,6 +305,7 @@ static const struct argp_child bench_parsers[] = {
 	{ &bench_sockmap_argp, 0, "bpf sockmap benchmark", 0 },
 	{ &bench_lpm_trie_map_argp, 0, "LPM trie map benchmark", 0 },
 	{ &bench_xdp_lb_argp, 0, "XDP load-balancer benchmark", 0 },
+	{ &bench_arraymap_mmap_argp, 0, "arraymap-mmap benchmark", 0 },
 	{},
 };
 
@@ -582,6 +584,7 @@ extern const struct bench bench_lpm_trie_delete;
 extern const struct bench bench_lpm_trie_free;
 extern const struct bench bench_bpf_nop;
 extern const struct bench bench_xdp_lb;
+extern const struct bench bench_arraymap_mmap;
 
 static const struct bench *benchs[] = {
 	&bench_count_global,
@@ -665,6 +668,7 @@ static const struct bench *benchs[] = {
 	&bench_lpm_trie_free,
 	&bench_bpf_nop,
 	&bench_xdp_lb,
+	&bench_arraymap_mmap,
 };
 
 static void find_benchmark(void)
tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c new file +127 -0

127 lines not on this page · raw message

tools/testing/selftests/bpf/benchs/run_bench_arraymap_mmap.sh new file +15 -0
new file mode 100755
index 000000000000..22c569311137
@@ -0,0 +1,15 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+source ./benchs/run_common.sh
+
+set -eufo pipefail
+
+for t in 1 4 8 16; do
+for s in 1048576 8388608 67108864; do
+subtitle "nr_threads: $t, map_size: $s"
+	summarize_ops "arraymap-mmap: " \
+	    "$($RUN_BENCH -p $t --map-size $s arraymap-mmap)"
+	printf "\n"
+done
+done
Signature · 1 line
2.53.0-Meta
#5 lore raw
In reply to Song Liu
Quoted 63 lines · Song Liu

An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory vmalloc'ed up front at map creation time. array_map_mmap() then wires up the whole mapping eagerly via remap_vmalloc_range(), which calls vm_insert_page() for every page of the map. This makes every mmap() cost O(number of pages): an 8MiB map inserts 2048 PTEs on each mmap() and tears them all down again on munmap(), even when user space only touches a few pages (or none at all).

This series makes the mapping populate lazily instead, the same way the arena map already does, turning mmap()/munmap() of an mmap-able array from O(map size) into O(1). The pages are inserted on demand by a new map_mmap_fault() callback, dispatched from the shared bpf_map_default_vmops so that the existing VMA open/close accounting (VM_MAYWRITE write-active tracking, freeze handling) stays centralized.

Callers that want the pages populated up front can still request that explicitly with MAP_POPULATE, matching regular file semantics. Kernel-side access to the map (via the vmalloc address) is unaffected.

Patch 1 implements the lazy faulting. Patch 2 adds an "arraymap-mmap" benchmark that mmap()s/munmap()s an mmap-able array map, useful to observe how the cost scales with the mapping size.

Changes in v2:
- Compute the mapping offset with 64-bit arithmetic in array_map_mmap() and the fault handler so that a large vm_pgoff cannot overflow the bounds check on 32-bit architectures. remap_vmalloc_range(), which is no longer used, used to guard against this.
- Benchmark: use the standard -p/--producers option for the thread count instead of a custom --nr-threads option. The v1 approach set env.producer_cnt from validate(), which runs after the framework has already sized the producers array, causing an out-of-bounds write.
(Both issues were pointed out by the Sashiko AI review.)

v1: https://lore.kernel.org/bpf/[email protected]/

Benchmark results (single producer thread, ./bench -p1):

Before the patch
nr_threads: 1, map_size: 1048576
arraymap-mmap: throughput: 0.034 +/- 0.000 M ops/s, latency: 29187.164 ns/op

nr_threads: 1, map_size: 8388608
arraymap-mmap: throughput: 0.005 +/- 0.000 M ops/s, latency: 220910.150 ns/op

nr_threads: 1, map_size: 67108864
arraymap-mmap: throughput: 0.001 +/- 0.000 M ops/s, latency: 1765083.440 ns/op

After the patch
nr_threads: 1, map_size: 1048576
arraymap-mmap: throughput: 0.838 +/- 0.001 M ops/s, latency: 1193.553 ns/op

nr_threads: 1, map_size: 8388608
arraymap-mmap: throughput: 0.811 +/- 0.002 M ops/s, latency: 1232.728 ns/op

nr_threads: 1, map_size: 67108864
arraymap-mmap: throughput: 0.747 +/- 0.004 M ops/s, latency: 1339.051 ns/op

Song Liu (2):
bpf: Populate mmap-able array map memory lazily selftests/bpf: Add mmap/munmap benchmark for array maps

Quoted 10 lines
 include/linux/bpf.h                           |   1 +
 kernel/bpf/arraymap.c                         |  43 +++++-
 kernel/bpf/syscall.c                          |  17 +++
 tools/testing/selftests/bpf/Makefile          |   1 +
 tools/testing/selftests/bpf/bench.c           |   4 +
 .../bpf/benchs/bench_arraymap_mmap.c          | 127 ++++++++++++++++++
 .../bpf/benchs/run_bench_arraymap_mmap.sh     |  15 +++
 7 files changed, 204 insertions(+), 4 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c
 create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_arraymap_mmap.sh

do we really need these benchmarks? yes, it's obvious that not doing anything per-page will be faster :) (I mean, it's good to see that it's 30us for 1MB array, but I don't think we should carry benchmark in the kernel tree)

Quoted 3 lines

--
2.53.0-Meta

author reviewer
#6 lore raw
Quoted 76 lines · Andrii Nakryiko
Quoted 63 lines · Song Liu

An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory vmalloc'ed up front at map creation time. array_map_mmap() then wires up the whole mapping eagerly via remap_vmalloc_range(), which calls vm_insert_page() for every page of the map. This makes every mmap() cost O(number of pages): an 8MiB map inserts 2048 PTEs on each mmap() and tears them all down again on munmap(), even when user space only touches a few pages (or none at all).

This series makes the mapping populate lazily instead, the same way the arena map already does, turning mmap()/munmap() of an mmap-able array from O(map size) into O(1). The pages are inserted on demand by a new map_mmap_fault() callback, dispatched from the shared bpf_map_default_vmops so that the existing VMA open/close accounting (VM_MAYWRITE write-active tracking, freeze handling) stays centralized.

Callers that want the pages populated up front can still request that explicitly with MAP_POPULATE, matching regular file semantics. Kernel-side access to the map (via the vmalloc address) is unaffected.

Patch 1 implements the lazy faulting. Patch 2 adds an "arraymap-mmap" benchmark that mmap()s/munmap()s an mmap-able array map, useful to observe how the cost scales with the mapping size.

Changes in v2:
- Compute the mapping offset with 64-bit arithmetic in array_map_mmap() and the fault handler so that a large vm_pgoff cannot overflow the bounds check on 32-bit architectures. remap_vmalloc_range(), which is no longer used, used to guard against this.
- Benchmark: use the standard -p/--producers option for the thread count instead of a custom --nr-threads option. The v1 approach set env.producer_cnt from validate(), which runs after the framework has already sized the producers array, causing an out-of-bounds write.
(Both issues were pointed out by the Sashiko AI review.)

v1: https://lore.kernel.org/bpf/[email protected]/

Benchmark results (single producer thread, ./bench -p1):

Before the patch
nr_threads: 1, map_size: 1048576
arraymap-mmap: throughput: 0.034 +/- 0.000 M ops/s, latency: 29187.164 ns/op

nr_threads: 1, map_size: 8388608
arraymap-mmap: throughput: 0.005 +/- 0.000 M ops/s, latency: 220910.150 ns/op

nr_threads: 1, map_size: 67108864
arraymap-mmap: throughput: 0.001 +/- 0.000 M ops/s, latency: 1765083.440 ns/op

After the patch
nr_threads: 1, map_size: 1048576
arraymap-mmap: throughput: 0.838 +/- 0.001 M ops/s, latency: 1193.553 ns/op

nr_threads: 1, map_size: 8388608
arraymap-mmap: throughput: 0.811 +/- 0.002 M ops/s, latency: 1232.728 ns/op

nr_threads: 1, map_size: 67108864
arraymap-mmap: throughput: 0.747 +/- 0.004 M ops/s, latency: 1339.051 ns/op

Song Liu (2):
bpf: Populate mmap-able array map memory lazily selftests/bpf: Add mmap/munmap benchmark for array maps

Quoted 10 lines
 include/linux/bpf.h                           |   1 +
 kernel/bpf/arraymap.c                         |  43 +++++-
 kernel/bpf/syscall.c                          |  17 +++
 tools/testing/selftests/bpf/Makefile          |   1 +
 tools/testing/selftests/bpf/bench.c           |   4 +
 .../bpf/benchs/bench_arraymap_mmap.c          | 127 ++++++++++++++++++
 .../bpf/benchs/run_bench_arraymap_mmap.sh     |  15 +++
 7 files changed, 204 insertions(+), 4 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_arraymap_mmap.c
 create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_arraymap_mmap.sh
Quoted 4 lines

do we really need these benchmarks? yes, it's obvious that not doing anything per-page will be faster :) (I mean, it's good to see that it's 30us for 1MB array, but I don't think we should carry benchmark in the kernel tree)

I am OK removing this in v3. AI wrote most of it, so it is not a lot of wasted effort.

Thanks,
Song