Skip to contentSkip to conversation

bpf: Populate mmap-able array map memory lazily

PATCH Superseded Song Liu 2 participants 2 messages Aug 12 – Aug 13

Subsystems (2)

v5 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Song Liu

author reviewer
#0 lore raw

PATCH v5 1/1 bpf: Populate mmap-able array map memory lazily

An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory vmalloc'ed up front at map creation time. array_map_mmap() then wired up the whole mapping eagerly via remap_vmalloc_range(), which calls vm_insert_page() for every page of the map. For large maps this makes every mmap() O(number of pages): an 8MiB map inserts 2048 PTEs per mmap() and tears them all down again on munmap(), even when user space only touches a few pages (or none at all).

Populate the mapping lazily instead, the same way the arena map already does. array_map_mmap() now only performs the bounds check and returns, leaving the PTEs unpopulated; pages are inserted on demand by a new array_map_mmap_fault() handler. Because the memory is already resident, the fault handler simply resolves the vmalloc page and hands it to the fault path. This turns mmap()/munmap() into an O(1) operation.

The handler is reached through a new optional ->map_mmap_fault callback dispatched from the shared bpf_map_default_vmops, so the existing VMA open/close accounting (VM_MAYWRITE write-active tracking, freeze handling) stays centralized rather than each map installing its own vm_operations_struct.

Lazy population requires page faults, so on !CONFIG_MMU array_map_mmap() keeps populating the mapping eagerly via remap_vmalloc_range().

Callers that want the pages populated up front can still request that explicitly with MAP_POPULATE. Kernel-side access to the map (via the vmalloc address) is unaffected.

Time for one mmap()+munmap() of an 8MiB mmap-able array map:

before after

  no MAP_POPULATE, no access            226us     1.1us
  no MAP_POPULATE, access all pages     236us    1341us
  MAP_POPULATE, no access               312us     493us
  MAP_POPULATE, access all pages        318us     519us

Mapping without touching the data, which is what this change targets, gets ~160x cheaper. Faulting in the whole mapping one page at a time is more expensive than the eager remap_vmalloc_range() loop, so users that do touch every page should ask for MAP_POPULATE. Note that MAP_POPULATE is not free before this change either: it adds ~85us (226us => 312us) for no benefit, as the mapping is already fully populated.

Assisted-by Claude:claude-opus-4-8

---
Changes in v5:
- Drop the ->map_pages (fault-around) handler, it pulls in too much mm internal API for the gain. (Andrii)
- Drop the fault path overflow and bounds checks, and the verbose comments; VM_DONTEXPAND plus the mmap() time check already bound the faulting offset. (Andrii)
- No cover letter for a single patch. (Andrii)
v4: https://lore.kernel.org/bpf/[email protected]/

Changes in v4:
- Flush the D-cache before exposing a page at a new user address, as the eager vm_insert_page() path did. (Sashiko AI review)
- Fix the build on !CONFIG_MMU: keep populating the mapping eagerly there, as there are no page faults. (kernel test robot)
v3: https://lore.kernel.org/bpf/[email protected]/

Changes in v3:
- Add a ->map_pages (fault-around) handler so mmap(MAP_POPULATE) and linear access populate PTEs in batches instead of one fault per page.
- Harden the fault path with check_shl_overflow() and explicit bounds checks instead of a plain (u64) cast. (Andrii)
- Drop selftests (2/2 in v2). (Andrii)
v2: https://lore.kernel.org/bpf/[email protected]/

Changes in v2:
- Use 64-bit arithmetic for the mmap offset and bounds check to avoid a potential overflow on 32-bit architectures.
v1: https://lore.kernel.org/bpf/[email protected]/

---
 include/linux/bpf.h   |  1 +
 kernel/bpf/arraymap.c | 37 +++++++++++++++++++++++++++++++++----
 kernel/bpf/syscall.c  | 12 ++++++++++++
 3 files changed, 46 insertions(+), 4 deletions(-)
include/linux/bpf.h +1 -0
index b4a10c9878cf..4a50324e7818 100644
@@ -145,6 +145,7 @@ struct bpf_map_ops {
 	int (*map_direct_value_meta)(const struct bpf_map *map,
 				     u64 imm, u32 *off);
 	int (*map_mmap)(struct bpf_map *map, struct vm_area_struct *vma);
+	vm_fault_t (*map_mmap_fault)(struct bpf_map *map, struct vm_fault *vmf);
 	__poll_t (*map_poll)(struct bpf_map *map, struct file *filp,
 			     struct poll_table_struct *pts);
 	unsigned long (*map_get_unmapped_area)(struct file *filep, unsigned long addr,
kernel/bpf/arraymap.c +33 -4
index 248b4818178c..a1767ef2b473 100644
@@ -576,17 +576,45 @@ static int array_map_check_btf(struct bpf_map *map,
 static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma)
 {
 	struct bpf_array *array = container_of(map, struct bpf_array, map);
-	pgoff_t pgoff = PAGE_ALIGN(sizeof(*array)) >> PAGE_SHIFT;
 
 	if (!(map->map_flags & BPF_F_MMAPABLE))
 		return -EINVAL;
 
-	if (vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
+	/* use u64 math so the offset cannot overflow on 32-bit archs */
+	if ((u64)vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
 	    PAGE_ALIGN((u64)array->map.max_entries * array->elem_size))
 		return -EINVAL;
 
-	return remap_vmalloc_range(vma, array_map_vmalloc_addr(array),
-				   vma->vm_pgoff + pgoff);
+	if (!IS_ENABLED(CONFIG_MMU)) {
+		pgoff_t pgoff = PAGE_ALIGN(sizeof(*array)) >> PAGE_SHIFT;
+
+		/* no page faults without an MMU, populate the mapping now */
+		return remap_vmalloc_range(vma, array_map_vmalloc_addr(array),
+					   vma->vm_pgoff + pgoff);
+	}
+
+	/* pages are faulted in on demand by array_map_mmap_fault() */
+	vm_flags_set(vma, VM_DONTEXPAND | VM_DONTDUMP);
+
+	return 0;
+}
+
+static vm_fault_t array_map_mmap_fault(struct bpf_map *map,
+				       struct vm_fault *vmf)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+	struct page *page;
+
+	page = vmalloc_to_page(array->value + ((u64)vmf->pgoff << PAGE_SHIFT));
+	if (!page)
+		return VM_FAULT_SIGBUS;
+
+	/* the eager remap_vmalloc_range() flushed via vm_insert_page() */
+	flush_dcache_folio(page_folio(page));
+	get_page(page);
+	vmf->page = page;
+
+	return 0;
 }
 
 static bool array_map_meta_equal(const struct bpf_map *meta0,
@@ -812,6 +840,7 @@ const struct bpf_map_ops array_map_ops = {
 	.map_direct_value_addr = array_map_direct_value_addr,
 	.map_direct_value_meta = array_map_direct_value_meta,
 	.map_mmap = array_map_mmap,
+	.map_mmap_fault = array_map_mmap_fault,
 	.map_seq_show_elem = array_map_seq_show_elem,
 	.map_check_btf = array_map_check_btf,
 	.map_lookup_batch = generic_map_lookup_batch,
kernel/bpf/syscall.c +12 -0
index 8d111da88655..ee0b01373a4b 100644
@@ -1077,9 +1077,21 @@ static void bpf_map_mmap_close(struct vm_area_struct *vma)
 		bpf_map_write_active_dec(map);
 }
 
+/* Called for maps that populate their memory-mapped region lazily. */
+static vm_fault_t bpf_map_mmap_fault(struct vm_fault *vmf)
+{
+	struct bpf_map *map = vmf->vma->vm_private_data;
+
+	if (!map->ops->map_mmap_fault)
+		return VM_FAULT_SIGBUS;
+
+	return map->ops->map_mmap_fault(map, vmf);
+}
+
 static const struct vm_operations_struct bpf_map_default_vmops = {
 	.open		= bpf_map_mmap_open,
 	.close		= bpf_map_mmap_close,
+	.fault		= bpf_map_mmap_fault,
 };
 
 static int bpf_map_mmap(struct file *filp, struct vm_area_struct *vma)

base-commit: 3a59f11e0f989bdd637c87151992605a6559a7cb

Signature · 1 line
2.53.0-Meta
#1 lore raw
This isn't a bug, but can the !CONFIG_MMU branch actually be reached?
Quoted 30 lines
include/linux/bpf.h +1 -0
index b4a10c9878cf8..4a50324e78189 100644
@@ -145,6 +145,7 @@ struct bpf_map_ops {
 	int (*map_direct_value_meta)(const struct bpf_map *map,
 				     u64 imm, u32 *off);
 	int (*map_mmap)(struct bpf_map *map, struct vm_area_struct *vma);
+	vm_fault_t (*map_mmap_fault)(struct bpf_map *map, struct vm_fault *vmf);
 	__poll_t (*map_poll)(struct bpf_map *map, struct file *filp,
 			     struct poll_table_struct *pts);
 	unsigned long (*map_get_unmapped_area)(struct file *filep, unsigned long addr,
kernel/bpf/arraymap.c +2 -2
index 248b4818178cd..a1767ef2b473f 100644
@@ -576,17 +576,45 @@ static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma)
 static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma)
 {
 	struct bpf_array *array = container_of(map, struct bpf_array, map);
-	pgoff_t pgoff = PAGE_ALIGN(sizeof(*array)) >> PAGE_SHIFT;
 
 	if (!(map->map_flags & BPF_F_MMAPABLE))
 		return -EINVAL;
 
-	if (vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
+	/* use u64 math so the offset cannot overflow on 32-bit archs */
+	if ((u64)vma->vm_pgoff * PAGE_SIZE + (vma->vm_end - vma->vm_start) >
 	    PAGE_ALIGN((u64)array->map.max_entries * array->elem_size))
 		return -EINVAL;

kernel/bpf/arraymap.c static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma) · line 594

Quoted 10 lines
kernel/bpf/arraymap.c +7 -2
 
-	return remap_vmalloc_range(vma, array_map_vmalloc_addr(array),
-				   vma->vm_pgoff + pgoff);
+	if (!IS_ENABLED(CONFIG_MMU)) {
+		pgoff_t pgoff = PAGE_ALIGN(sizeof(*array)) >> PAGE_SHIFT;
+
+		/* no page faults without an MMU, populate the mapping now */
+		return remap_vmalloc_range(vma, array_map_vmalloc_addr(array),
+					   vma->vm_pgoff + pgoff);
+	}

This isn't a bug, but can the !CONFIG_MMU branch actually be reached?

On nommu, bpf_map_new_fd() uses anon_inode_getfd(), which creates an inode with no S_IFMT type (alloc_anon_inode() at fs/libfs.c sets i_mode to just S_IRUSR|S_IWUSR). Since bpf_map_fops has no ->mmap_capabilities, nommu's validate_mmap_request() falls into the default case and returns -EINVAL before ->mmap() runs.

So it looks like the eager path preserves an -EINVAL that nommu already produces one layer up, rather than providing a working fallback.

Quoted 6 lines
kernel/bpf/arraymap.c +6 -0
+
+	/* pages are faulted in on demand by array_map_mmap_fault() */
+	vm_flags_set(vma, VM_DONTEXPAND | VM_DONTDUMP);
+
+	return 0;
+}

[ ... ]

Quoted 16 lines
kernel/bpf/syscall.c +8 -0
index 8d111da886553..ee0b01373a4b6 100644
@@ -1077,9 +1077,21 @@ static void bpf_map_mmap_close(struct vm_area_struct *vma)
 		bpf_map_write_active_dec(map);
 }
 
+/* Called for maps that populate their memory-mapped region lazily. */
+static vm_fault_t bpf_map_mmap_fault(struct vm_fault *vmf)
+{
+	struct bpf_map *map = vmf->vma->vm_private_data;
+
+	if (!map->ops->map_mmap_fault)
+		return VM_FAULT_SIGBUS;
+

kernel/bpf/syscall.c static void bpf_map_mmap_close(struct vm_area_struct *vma) · line 1097

Quoted 10 lines
kernel/bpf/syscall.c +4 -0
+	return map->ops->map_mmap_fault(map, vmf);
+}
+
 static const struct vm_operations_struct bpf_map_default_vmops = {
 	.open		= bpf_map_mmap_open,
 	.close		= bpf_map_mmap_close,
+	.fault		= bpf_map_mmap_fault,
 };
 
 static int bpf_map_mmap(struct file *filp, struct vm_area_struct *vma)

The commit message states:

Quoted 7 lines

leaving the PTEs unpopulated; pages are inserted on demand by a new array_map_mmap_fault() handler. Because the memory is already resident, the fault handler simply resolves the vmalloc page and hands it to the fault path. This turns mmap()/munmap() into an O(1) operation.

Lazy population requires page faults, so on !CONFIG_MMU array_map_mmap() keeps populating the mapping eagerly via remap_vmalloc_range().

This isn't a bug, but could the O(1) claim be scoped to mmap() only? munmap() still zaps every PTE that was faulted in, which is why the benchmark table shows the access-all case going from 236us to 1341us.

Also, could the !CONFIG_MMU paragraph clarify what the branch is actually for? As noted above, nommu cannot mmap() a BPF map fd in the first place, since validate_mmap_request() rejects it before ->mmap() runs.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31650392779