bpf: Populate mmap-able array maps lazily
Song Liu
reviewerv3 cover bpf: Populate mmap-able array maps lazily
An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory vmalloc'ed up front at map creation time. array_map_mmap() then wires up the whole mapping eagerly via remap_vmalloc_range(), which calls vm_insert_page() for every page of the map. This makes every mmap() cost O(number of pages): an 8MiB map inserts 2048 PTEs on each mmap() and tears them all down again on munmap(), even when user space only touches a few pages (or none at all).
Populate the mapping lazily instead, the same way the arena map already does. array_map_mmap() only performs the bounds check and returns, and the pages are inserted on demand by a new ->map_mmap_fault handler. This turns mmap()/munmap() of an mmap-able array from O(map size) into O(1).
To keep populating a large mapping cheap, a ->map_pages (fault-around) handler batch-installs PTEs for the whole fault-around window under a single page-table lock, so that mmap(MAP_POPULATE) and linear access patterns do not take one full fault per page. Both handlers are reached through new optional callbacks dispatched from the shared bpf_map_default_vmops, keeping the existing VMA open/close accounting (VM_MAYWRITE write-active tracking, freeze handling) centralized.
Callers that want the pages populated up front can still request that explicitly with MAP_POPULATE. Kernel-side access to the map (via the vmalloc address) is unaffected.
This posting contains the kernel change only; the arraymap-mmap benchmark used to gather the numbers below will be sent separately.
Changes in v3:
- Add a ->map_pages (fault-around) handler so mmap(MAP_POPULATE) and linear access populate PTEs in batches instead of one fault per page.
- Harden the fault path with check_shl_overflow() and explicit bounds checks instead of a plain (u64) cast. (Andrii)
- Drop selftests (2/2 in v2). (Andrii)
v2: https://lore.kernel.org/bpf/[email protected]/
Changes in v2:
- Use 64-bit arithmetic for the mmap offset/bounds check to avoid a potential overflow on 32-bit architectures.
(Both v2 and v3 hardening were prompted by the Sashiko AI review.)
v1: https://lore.kernel.org/bpf/[email protected]/
Benchmark results:
For 8MiB, access times are microseconds (us).
Before:
no MAP_POPULATE (do not access pages) 220us
no MAP_POPULATE => access all pages: 252usMAP_POPULATE (do not access pages) 304usMAP_POPULATE => access all pages: 334us
After:
no MAP_POPULATE (do not access pages): 1.2us
no MAP_POPULATE => access all pages: 543usMAP_POPULATE (do not access pages): 262usMAP_POPULATE => access all pages: 286us
Key take aways from these tests:
MAP_POPULATE is not a noop for old kernels. The pre fault mechanism still adds some overhead (for almost no gain). For 8MiB mmap, MAP_POPULATE addes about 80us.
Compared against v2, we further improved MAP_POPULATE with a batch pre fault (vm_operations_struct->map_pages). With batch pre fault, mmap then accessing 8MiB is about 34us slower than the best option before the set:
before: no MAP_POPULATE => access all pages: 252us;
after: MAP_POPULATE => access all pages: 286us.
Note that, all the above are worst cases: we need to access all pages and fill the page table one way or another. If we don't need to access all pages, the change saves a lot (220us => 1.2us) for the mmap.
Song Liu (1):
bpf: Populate mmap-able array map memory lazily
include/linux/bpf.h | 3 ++ kernel/bpf/arraymap.c | 112 ++++++++++++++++++++++++++++++++++++++++-- kernel/bpf/syscall.c | 30 +++++++++++ 3 files changed, 141 insertions(+), 4 deletions(-)
base-commit: fdec474c65fd35d5a6e1497ed50a9f98c07192f0 --
2.53.0-Meta
Song Liu
reviewerPATCH v3 1/1 bpf: Populate mmap-able array map memory lazily
An mmap-able BPF array map (BPF_F_MMAPABLE) has its backing memory vmalloc'ed up front at map creation time. array_map_mmap() then wired up the whole mapping eagerly via remap_vmalloc_range(), which calls vm_insert_page() for every page of the map. For large maps this makes every mmap() O(number of pages): an 8MiB map inserts 2048 PTEs per mmap() and tears them all down again on munmap(), even when user space only touches a few pages (or none at all).
Populate the mapping lazily instead, the same way the arena map already does. array_map_mmap() now only performs the bounds check and returns, leaving the PTEs unpopulated; pages are inserted on demand by a new array_map_mmap_fault() handler. Because the memory is already resident, the fault handler simply resolves the vmalloc page and hands it to the fault path. This turns mmap()/munmap() into an O(1) operation.
To keep populating a large mapping cheap, also provide an array_map_mmap_pages() (->map_pages) handler that batch-installs PTEs for the whole fault-around window under a single page-table lock. This lets mmap(MAP_POPULATE) and linear access patterns avoid taking one full fault per page.
Both handlers are reached through new optional ->map_mmap_fault and ->map_mmap_pages callbacks dispatched from the shared bpf_map_default_vmops, so the existing VMA open/close accounting (VM_MAYWRITE write-active tracking, freeze handling) stays centralized rather than each map installing its own vm_operations_struct.
remap_vmalloc_range() previously guarded against offset overflow; since it is no longer used, array_map_mmap() and the fault handlers compute the mapping offset with 64-bit arithmetic and check_shl_overflow() so that a large vm_pgoff cannot overflow the bounds check on 32-bit architectures.
Callers that want the pages populated up front can still request that explicitly with MAP_POPULATE. Kernel-side access to the map (via the vmalloc address) is unaffected.
--- include/linux/bpf.h | 3 ++ kernel/bpf/arraymap.c | 112 ++++++++++++++++++++++++++++++++++++++++-- kernel/bpf/syscall.c | 30 +++++++++++ 3 files changed, 141 insertions(+), 4 deletions(-)
include/linux/bpf.h +3 -0
@@ -145,6 +145,9 @@ struct bpf_map_ops {int (*map_direct_value_meta)(const struct bpf_map *map,u64 imm,u32 *off);int (*map_mmap)(struct bpf_map *map,struct vm_area_struct *vma); +vm_fault_t (*map_mmap_fault)(struct bpf_map *map,struct vm_fault *vmf); +vm_fault_t (*map_mmap_pages)(struct bpf_map *map,struct vm_fault *vmf, +pgoff_t start_pgoff,pgoff_t end_pgoff);__poll_t (*map_poll)(struct bpf_map *map,struct file *filp,struct poll_table_struct *pts); unsigned long (*map_get_unmapped_area)(struct file *filep, unsigned long addr,
kernel/bpf/arraymap.c +108 -4
137 lines not on this page · raw message
kernel/bpf/syscall.c +30 -0
@@ -1077,9 +1077,39 @@ static void bpf_map_mmap_close(struct vm_area_struct *vma)bpf_map_write_active_dec (map); } +/* +* Called for maps that populate their memory-mapped region lazily, i.e. that +* return without wiring up any PTEs from their ->map_mmap callback. Maps that +* populate the whole VMA eagerly (e.g. via remap_vmalloc_range()) never reach +* here. +*/ +static vm_fault_t bpf_map_mmap_fault (struct vm_fault *vmf) +{ +struct bpf_map *map = vmf->vma->vm_private_data; + +if (!map->ops->map_mmap_fault) +return VM_FAULT_SIGBUS ; + +return map->ops->map_mmap_fault (map, vmf); +} + +/* Fault-around: batch-populate PTEs around the faulting address. */ +static vm_fault_t bpf_map_mmap_pages (struct vm_fault *vmf, +pgoff_t start_pgoff,pgoff_t end_pgoff) +{ +struct bpf_map *map = vmf->vma->vm_private_data; + +if (!map->ops->map_mmap_pages) +return 0 ; + +return map->ops->map_mmap_pages (map, vmf, start_pgoff, end_pgoff); +} +static const struct vm_operations_struct bpf_map_default_vmops = { .open = bpf_map_mmap_open, .close = bpf_map_mmap_close, + .fault = bpf_map_mmap_fault, + .map_pages = bpf_map_mmap_pages, };static int bpf_map_mmap (struct file *filp,struct vm_area_struct *vma)
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH v3 bpf-next 1/1] bpf: Populate mmap-able array map memory lazily
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Missing D-cache flush when lazily populating userspace mappings of vmalloc'ed memory, causing…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Missing D-cache flush when lazily populating userspace mappings of vmalloc'ed memory, causing potential stale data disclosure. --
--- Patch [1]: [PATCH v3 bpf-next 1/1] bpf: Populate mmap-able array map memory lazily --- commit 9d0c286aed96b18164a50c44ca2ef60d558d9222
Author: Song Liu <[email protected]>
bpf: Populate mmap-able array map memory lazily
This commit changes BPF array maps with BPF_F_MMAPABLE to map memory lazily. It replaces the eager remap_vmalloc_range() call with custom fault handlers that populate PTEs on demand to improve mmap performance for large maps.
Quoted 4 lines
kernel/bpf/arraymap.c +0
-0
[ ... ]
Quoted 4 lines
kernel/bpf/arraymap.c +3
-0@@ -576,17 +577,118 @@ static int array_map_check_btf(struct bpf_map *map, +static vm_fault_t array_map_mmap_fault (struct bpf_map *map, +struct vm_fault *vmf) +{
kernel/bpf/arraymap.c static int array_map_check_btf(struct bpf_map *map, · line 589
Quoted 10 lines
kernel/bpf/arraymap.c +10
-0+ struct bpf_array *array = container_of(map, struct bpf_array, map); +struct page *page; + + page =array_map_fault_page (array, vmf->pgoff); +if (!page) +return VM_FAULT_SIGBUS ; + +get_page (page); + vmf->page = page; +
[Severity: High]
Does this fault handler need to explicitly flush the D-cache for the resolved vmalloc page?
Previously, the eager mapping used remap_vmalloc_range_partial() which calls vm_insert_page(), and vm_insert_page() explicitly calls flush_dcache_folio(). By returning the page directly to the fault handler without a D-cache flush, can architectures with aliasing D-caches expose uninitialized physical memory to userspace, bypassing the kernel's cached writes?
Quoted 2 lines
kernel/bpf/arraymap.c +2
-0+return 0 ; +}
[ ... ]
Quoted 3 lines
kernel/bpf/arraymap.c +3
-0+static vm_fault_t array_map_mmap_pages (struct bpf_map *map,struct vm_fault *vmf, +pgoff_t start_pgoff,pgoff_t end_pgoff) +{
[ ... ]
Quoted 4 lines
kernel/bpf/arraymap.c +4
-0+for (pgoff = start_pgoff; pgoff <= end_pgoff; + pgoff++, vmf->pte++, addr +=PAGE_SIZE ) { +struct folio *folio; +struct page *page;
kernel/bpf/arraymap.c static int array_map_check_btf(struct bpf_map *map, · line 681
Quoted 10 lines
kernel/bpf/arraymap.c +10
-0+ +if (!pte_none (ptep_get (vmf->pte))) +continue ; + + page =array_map_fault_page (array, pgoff); +if (!page) +continue ; + + folio =page_folio (page); +folio_get (folio);
[Severity: High]
Similarly, in this fault-around handler, is a flush_dcache_folio() call needed before installing the PTE?
Calling set_pte_range() directly bypasses the D-cache flush that vm_insert_page() would have provided, which might result in userspace reading stale physical RAM contents on architectures with VIVT or VIPT aliasing D-caches.
Quoted 5 lines
kernel/bpf/arraymap.c +5
-0+set_pte_range (vmf, folio, page,1 , addr); + rss++; +if (addr == vmf->address) + ret =VM_FAULT_NOPAGE ; + }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
kernel test robot
build botRe: [PATCH v3 bpf-next 1/1] bpf: Populate mmap-able array map memory lazily
In reply to Song Liukernel test robot noticed the following build errors:
Hi Song,
kernel test robot noticed the following build errors:
[auto build test ERROR on fdec474c65fd35d5a6e1497ed50a9f98c07192f0]
url: https://github.com/intel-lab-lkp/linux/commits/Song-Liu/bpf-Populate-mmap-able-array-map-memory-lazily/20260729-093241 base: fdec474c65fd35d5a6e1497ed50a9f98c07192f0 patch link: https://lore.kernel.org/r/20260729001033.3433328-2-song%40kernel.org patch subject: [PATCH v3 bpf-next 1/1] bpf: Populate mmap-able array map memory lazily config: arm-randconfig-002-20260729 (https://download.01.org/0day-ci/archive/20260729/[email protected]/config) compiler: arm-linux-gnueabi-gcc (GCC) 8.5.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260729/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
kernel/bpf/arraymap.c: In function 'array_map_mmap_pages':
>> kernel/bpf/arraymap.c:659:6: error: implicit declaration of function 'pmd_none'; did you mean 'p4d_none'? [-Werror=implicit-function-declaration]
if (pmd_none(*vmf->pmd))
^~~~~~~~
p4d_none
>> kernel/bpf/arraymap.c:673:8: error: implicit declaration of function 'pte_none'; did you mean 'p4d_none'? [-Werror=implicit-function-declaration]
if (!pte_none(ptep_get(vmf->pte)))
^~~~~~~~
p4d_none
>> kernel/bpf/arraymap.c:673:17: error: implicit declaration of function 'ptep_get'; did you mean 'btf_get'? [-Werror=implicit-function-declaration]
if (!pte_none(ptep_get(vmf->pte)))
^~~~~~~~
btf_get
>> kernel/bpf/arraymap.c:682:3: error: implicit declaration of function 'set_pte_range'; did you mean 'free_pgd_range'? [-Werror=implicit-function-declaration]
set_pte_range(vmf, folio, page, 1, addr);
^~~~~~~~~~~~~
free_pgd_range
In file included from include/linux/kallsyms.h:13,
from include/linux/bpf.h:23,
from kernel/bpf/arraymap.c:5:
>> include/linux/mm.h:3848:2: error: implicit declaration of function 'pte_unmap'; did you mean 'memunmap'? [-Werror=implicit-function-declaration]
pte_unmap(pte); \
^~~~~~~~~
kernel/bpf/arraymap.c:689:2: note: in expansion of macro 'pte_unmap_unlock'
pte_unmap_unlock(start_pte, vmf->ptl);
^~~~~~~~~~~~~~~~cc1: some warnings being treated as errors
vim +659 kernel/bpf/arraymap.c
637
638 /*
639 * Fault-around handler: install PTEs for the whole [start_pgoff, end_pgoff]
640 * window under a single page-table lock, so that populating a large mapping
641 * (e.g. mmap(MAP_POPULATE) or a linear access pattern) does not take one full
642 * fault per page.
643 */
644 static vm_fault_t array_map_mmap_pages(struct bpf_map *map, struct vm_fault *vmf,
645 pgoff_t start_pgoff, pgoff_t end_pgoff)
646 {
647 struct bpf_array *array = container_of(map, struct bpf_array, map);
648 struct vm_area_struct *vma = vmf->vma;
649 unsigned long addr, rss = 0;
650 vm_fault_t ret = 0;
651 pte_t *start_pte;
652 pgoff_t pgoff;
653
654 /*
655 * The PTE table for this PMD must already exist; installing it needs
656 * mm-internal helpers. If it is missing, let the regular ->fault path
657 * install it and rely on the next fault-around to batch the rest.
658 */
> 659 if (pmd_none(*vmf->pmd))
660 return 0;
661
662 addr = vma->vm_start + ((start_pgoff - vma->vm_pgoff) << PAGE_SHIFT);
663 start_pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl);
664 if (!start_pte)
665 return 0;
666 vmf->pte = start_pte;
667
668 for (pgoff = start_pgoff; pgoff <= end_pgoff;
669 pgoff++, vmf->pte++, addr += PAGE_SIZE) {
670 struct folio *folio;
671 struct page *page;
672
> 673 if (!pte_none(ptep_get(vmf->pte)))
674 continue;
675
676 page = array_map_fault_page(array, pgoff);
677 if (!page)
678 continue;
679
680 folio = page_folio(page);
681 folio_get(folio);
> 682 set_pte_range(vmf, folio, page, 1, addr); 683 rss++;
684 if (addr == vmf->address)
685 ret = VM_FAULT_NOPAGE;
686 }
687
688 add_mm_counter(vma->vm_mm, MM_FILEPAGES, rss);
689 pte_unmap_unlock(start_pte, vmf->ptl);
690
691 return ret;
692 }
693
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki