mm: improve large folio readahead and alignment for exec memory

The mmap_miss counter in do_sync_mmap_readahead() tracks whether readahead is useful for mmap'd file access. It is incremented by 1 on every page cache miss in do_sync_mmap_readahead(), and decremented in two places:

- filemap_map_pages(): decremented by N for each of N pages successfully mapped via fault-around (pages found already in cache, evidence readahead was useful). Only pages not in the workingset count as hits.

- do_async_mmap_readahead(): decremented by 1 when a page with PG_readahead is found in cache.

When the counter exceeds MMAP_LOTSAMISS (100), all readahead is disabled, including the targeted VM_EXEC readahead [1] that requests large folio orders for contpte mapping.

On arm64 with 64K base pages, both decrement paths are inactive:

1. filemap_map_pages() is never called because fault_around_pages (65536 >> PAGE_SHIFT = 1) disables should_fault_around(), which requires fault_around_pages > 1. With only 1 page in the fault-around window, there is nothing "around" to map.

2. do_async_mmap_readahead() never fires for exec mappings because exec readahead sets async_size = 0, so no PG_readahead markers are placed.

With no decrements, mmap_miss monotonically increases past MMAP_LOTSAMISS after 100 page faults, disabling all subsequent exec readahead.

Fix this by excluding VM_EXEC VMAs from the mmap_miss logic, similar to how VM_SEQ_READ is already excluded. The exec readahead path is targeted (one folio at the fault location, async_size=0), not speculative prefetch, so the mmap_miss heuristic designed to throttle wasteful speculative readahead should not apply to it.

[1] https://lore.kernel.org/all/[email protected]/

Signed-off-by Usama Arif <[email protected]>

---
 mm/filemap.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

mm/filemap.c +1 -1

index 6cd7974d4adab..7d89c6b384cc4 100644
@@ -3331,7 +3331,7 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
 		}
 	}
 
-	if (!(vm_flags & VM_SEQ_READ)) {
+	if (!(vm_flags & (VM_SEQ_READ | VM_EXEC))) {
 		/* Avoid banging the cache line if not needed */
 		mmap_miss = READ_ONCE(ra->mmap_miss);
 		if (mmap_miss < MMAP_LOTSAMISS * 10)

Signature · 1 line

2.52.0

+ * allocations when memory is tight.
+ */

mm/filemap.c +16 -0

+static unsigned int preferred_exec_order(struct vm_area_struct *vma)
+{
+	int order;
+	unsigned long vma_len = vma_pages(vma);
+	struct zone *zone;
+	gfp_t gfp;
+
+	if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE))
+		return 0;
+
+	/* Cap at min(PMD_ORDER, 2M) */
+	order = min(HPAGE_PMD_ORDER, ilog2(SZ_2M >> PAGE_SHIFT));
+
+	/* Don't request folios larger than the VMA */
+	order = min(order, ilog2(vma_len));
+

Replace the arch-specific exec_folio_order() hook with a generic preferred_exec_order() that dynamically computes the readahead folio order for executable memory. It targets min(PMD_ORDER, 2M) as the maximum, which optimally gives the right answer for contpte (arm64), PMD mapping (x86, arm64 4K), and architectures with smaller PMDs (s390 1M). It adapts at runtime based on:

- VMA size: caps the order so folios fit within the mapping
- Memory pressure: steps down the order when the local node's free memory is below the high watermark for the requested order

This avoids over-allocating on memory-constrained systems while still requesting the optimal order when memory is plentiful.

Since exec_folio_order() is no longer needed, remove the arm64 definition and the generic default from pgtable.h.

Signed-off-by Usama Arif <[email protected]>

mm/filemap.c +5 -0

+static unsigned int preferred_exec_order(struct vm_area_struct *vma)
+{
+	int order;
+	unsigned long vma_len = vma_pages(vma);
+	struct zone *zone;

Quoted 18 lines · Usama Arif

Replace the arch-specific exec_folio_order() hook with a generic preferred_exec_order() that dynamically computes the readahead folio order for executable memory. It targets min(PMD_ORDER, 2M) as the maximum, which optimally gives the right answer for contpte (arm64), PMD mapping (x86, arm64 4K), and architectures with smaller PMDs (s390 1M). It adapts at runtime based on:

- VMA size: caps the order so folios fit within the mapping
- Memory pressure: steps down the order when the local node's free memory is below the high watermark for the requested order

This avoids over-allocating on memory-constrained systems while still requesting the optimal order when memory is plentiful.

Since exec_folio_order() is no longer needed, remove the arm64 definition and the generic default from pgtable.h.

Signed-off-by Usama Arif <[email protected]>

...

Quoted 5 lines

mm/filemap.c +5 -0

+static unsigned int preferred_exec_order(struct vm_area_struct *vma)
+{
+	int order;
+	unsigned long vma_len = vma_pages(vma);
+	struct zone *zone;

mm/filemap.c static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio, · line 3323

Quoted 10 lines

mm/filemap.c +10 -0

+	gfp_t gfp;
+
+	if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE))
+		return 0;
+
+	/* Cap at min(PMD_ORDER, 2M) */
+	order = min(HPAGE_PMD_ORDER, ilog2(SZ_2M >> PAGE_SHIFT));
+
+	/* Don't request folios larger than the VMA */
+	order = min(order, ilog2(vma_len));

Hum, as far as I'm checking page_cache_ra_order() used in do_sync_mmap_readahead(), ra->order is the preferred order but it will be trimmed down to fit both within the file and within ra->size. And ra->size is set for the readahead to fit within the vma so I don't think any order trimming based on vma length is needed in this place?

mm/filemap.c static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio, · line 3334

Quoted 11 lines

mm/filemap.c +10 -0

+	/* Step down under memory pressure */
+	gfp = mapping_gfp_mask(vma->vm_file->f_mapping);
+	zone = first_zones_zonelist(node_zonelist(numa_node_id(), gfp),
+				    gfp_zone(gfp), NULL)->zone;
+	if (zone) {
+		while (order > 0 &&
+		       !zone_watermark_ok(zone, order,
+					  high_wmark_pages(zone), 0, 0))
+			order--;
+	}

It looks wrong for this logic to be here. Trimming order based on memory pressure makes sense (and we've already got reports that on memory limited devices large order folios in the page cache have too big memory overhead so we'll likely need to handle that for page cache allocations in general) but IMHO it belongs to page_cache_ra_order() or some other common place like that.

Honza

Quoted 18 lines · Jan Kara

mm/filemap.c static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio, · line 3334

Quoted 10 lines

mm/filemap.c +10 -0

+	/* Step down under memory pressure */
+	gfp = mapping_gfp_mask(vma->vm_file->f_mapping);
+	zone = first_zones_zonelist(node_zonelist(numa_node_id(), gfp),
+				    gfp_zone(gfp), NULL)->zone;
+	if (zone) {
+		while (order > 0 &&
+		       !zone_watermark_ok(zone, order,
+					  high_wmark_pages(zone), 0, 0))
+			order--;
+	}

It looks wrong for this logic to be here. Trimming order based on memory pressure makes sense (and we've already got reports that on memory limited devices large order folios in the page cache have too big memory overhead so we'll likely need to handle that for page cache allocations in general) but IMHO it belongs to page_cache_ra_order() or some other common place like that.

Quoted 1 line

Honza

So I have been thinking about this. readahead_gfp_mask() already sets __GFP_NORETRY, so we wont try aggressive reclaim/compaction to satisfy the allocation. page_cache_ra_order() falls through to the fallback path faulting in order 0 page when allocation is not satsified.

So the allocator already naturally steps down under memory pressure, the explicit zone_watermark_ok() loop might be redundant?

What are your thoughts on just setting
ra->order = min(HPAGE_PMD_ORDER, ilog2(SZ_2M >> PAGE_SHIFT))? We can do the higher orlder allocation with gfp &= ~__GFP_RECLAIM for the VM_EXEC case.

For PIE binaries (ET_DYN), the load address is randomized at PAGE_SIZE granularity via arch_mmap_rnd(). On arm64 with 64K base pages, this means the binary is 64K-aligned, but contpte mapping requires 2M (CONT_PTE_SIZE) alignment.

Without proper virtual address alignment, readahead patches that allocate 2M folios with 2M-aligned file offsets and physical addresses cannot benefit from contpte mapping, as the contpte fold check in contpte_set_ptes() requires the virtual address to be CONT_PTE_SIZE- aligned.

Fix this by extending maximum_alignment() to consider the maximum folio size supported by the page cache (via mapping_max_folio_size()). For each PT_LOAD segment, the alignment is bumped to the largest power-of-2 that fits within the segment size, capped by the max folio size the filesystem will allocate, if:

- Both p_vaddr and p_offset are aligned to that size
- The segment is large enough (p_filesz >= size)

This ensures load_bias is folio-aligned so that file-offset-aligned folios map to properly aligned virtual addresses, enabling hardware PTE coalescing (e.g. arm64 contpte) and PMD mappings for large folios.

The segment size check avoids reducing ASLR entropy for small binaries that cannot benefit from large folio alignment.

Signed-off-by Usama Arif <[email protected]>

fs/binfmt_elf.c +16 -0

+
+			/*
+			 * Try to align the binary to the largest folio
+			 * size that the page cache supports, so the
+			 * hardware can coalesce PTEs (e.g. arm64
+			 * contpte) or use PMD mappings for large folios.
+			 *
+			 * Use the largest power-of-2 that fits within
+			 * the segment size, capped by what the page
+			 * cache will allocate. Only align when the
+			 * segment's virtual address and file offset are
+			 * already aligned to the folio size, as
+			 * misalignment would prevent coalescing anyway.
+			 *
+			 * The segment size check avoids reducing ASLR
+			 * entropy for small binaries that cannot

Quoted 8 lines · Usama Arif

fs/binfmt_elf.c +3 -1

-static unsigned long maximum_alignment(struct elf_phdr *cmds, int nr)
+static unsigned long maximum_alignment(struct elf_phdr *cmds, int nr,
+				       struct file *filp)
 {
 	unsigned long alignment = 0;
+	unsigned long max_folio_size = PAGE_SIZE;
 	int i;

fs/binfmt_elf.c static int elf_read(struct file *file, void *buf, size_t len, loff_t pos) · line 500

Quoted 2 lines

fs/binfmt_elf.c +2 -0

+	if (filp && filp->f_mapping)
+		max_folio_size = mapping_max_folio_size(filp->f_mapping);

Under what circumstances can bprm->file be NULL?

Also we tend to prefer the name "file" rather than "filp" for new code (yes, there's a lot of old code out there).

Quoted 16 lines

fs/binfmt_elf.c +16 -0

+
+			/*
+			 * Try to align the binary to the largest folio
+			 * size that the page cache supports, so the
+			 * hardware can coalesce PTEs (e.g. arm64
+			 * contpte) or use PMD mappings for large folios.
+			 *
+			 * Use the largest power-of-2 that fits within
+			 * the segment size, capped by what the page
+			 * cache will allocate. Only align when the
+			 * segment's virtual address and file offset are
+			 * already aligned to the folio size, as
+			 * misalignment would prevent coalescing anyway.
+			 *
+			 * The segment size check avoids reducing ASLR
+			 * entropy for small binaries that cannot

fs/binfmt_elf.c static int elf_read(struct file *file, void *buf, size_t len, loff_t pos) · line 536

Quoted 10 lines

fs/binfmt_elf.c +10 -0

+			 * benefit.
+			 */
+			if (!cmds[i].p_filesz)
+				continue;
+			size = rounddown_pow_of_two(cmds[i].p_filesz);
+			size = min(size, max_folio_size);
+			if (size > PAGE_SIZE &&
+			    IS_ALIGNED(cmds[i].p_vaddr, size) &&
+			    IS_ALIGNED(cmds[i].p_offset, size))
+				alignment = max(alignment, size);

Can this not all be factored out into a different function? Also, I think it was done a bit better here:
https://lore.kernel.org/linux-fsdevel/[email protected]/

fs/binfmt_elf.c +1 -0

index 8e89cc5b28200..042af81766fcd 100644
@@ -49,6 +49,7 @@
 #include <uapi/linux/rseq.h>
 #include <asm/param.h>
 #include <asm/page.h>
+#include <linux/pagemap.h>
 
 #ifndef ELF_COMPAT
 #define ELF_COMPAT 0

@@ -488,19 +489,51 @@ static int elf_read(struct file *file, void *buf, size_t len, loff_t pos)
        return 0;
 }
 

fs/binfmt_elf.c +18 -0

+
        for (i = 0; i < nr; i++) {
                if (cmds[i].p_type == PT_LOAD) {
                        unsigned long p_align = cmds[i].p_align;
+                       unsigned long size;
 
                        /* skip non-power of two alignments as invalid */
                        if (!is_power_of_2(p_align))
                                continue;
                        alignment = max(alignment, p_align);
+
+                       /*
+                        * Try to align the binary to the largest folio
+                        * size that the page cache supports, so the
+                        * hardware can coalesce PTEs (e.g. arm64
+                        * contpte) or use PMD mappings for large folios.
+                        *
+                        * Use the largest power-of-2 that fits within
+                        * the segment size, capped by what the page
+                        * cache will allocate. Only align when the
+                        * segment's virtual address and file offset are
+                        * already aligned to the folio size, as
+                        * misalignment would prevent coalescing anyway.
+                        *
+                        * The segment size check avoids reducing ASLR
+                        * entropy for small binaries that cannot

fs/binfmt_elf.c +2 -2

                }
        }
 

@@ -1104,7 +1137,8 @@ static int load_elf_binary(struct linux_binprm *bprm)
                        }
 
                        /* Calculate any requested alignment. */
-                       alignment = maximum_alignment(elf_phdata, elf_ex->e_phnum);
+                       alignment = maximum_alignment(elf_phdata, elf_ex->e_phnum,
+                                                     bprm->file);
 
                        /**
                         * DOC: PIE handling
--

2.52.0

Quoted 1 line

From experiments (with 16K base pages), mapping_max_folio_size() appears to

depend on the filesystem. It returns 8M on ext4, while on btrfs it always falls back to PAGE_SIZE (it seems CONFIG_BTRFS_EXPERIMENTAL=y may change this). This looks overly conservative and ends up missing practical optimization opportunities.

Hi Usama,

Quoted 24 lines · Usama Arif

fs/binfmt_elf.c +4 -1

index 8e89cc5b28200..042af81766fcd 100644
@@ -49,6 +49,7 @@
 #include <uapi/linux/rseq.h>
 #include <asm/param.h>
 #include <asm/page.h>
+#include <linux/pagemap.h>
 
 #ifndef ELF_COMPAT
 #define ELF_COMPAT 0

@@ -488,19 +489,51 @@ static int elf_read(struct file *file, void *buf, size_t len, loff_t pos)
        return 0;
 }
 
-static unsigned long maximum_alignment(struct elf_phdr *cmds, int nr)
+static unsigned long maximum_alignment(struct elf_phdr *cmds, int nr,
+                                      struct file *filp)
 {
        unsigned long alignment = 0;
+       unsigned long max_folio_size = PAGE_SIZE;
        int i;
 

fs/binfmt_elf.c static int elf_read(struct file *file, void *buf, size_t len, loff_t pos) · line 500

Quoted 2 lines

fs/binfmt_elf.c +2 -0

+       if (filp && filp->f_mapping)
+               max_folio_size = mapping_max_folio_size(filp->f_mapping);

From experiments (with 16K base pages), mapping_max_folio_size() appears to depend on the filesystem. It returns 8M on ext4, while on btrfs it always falls back to PAGE_SIZE (it seems CONFIG_BTRFS_EXPERIMENTAL=y may change this). This looks overly conservative and ends up missing practical optimization opportunities.

Quoted 27 lines

fs/binfmt_elf.c +18 -0

+
        for (i = 0; i < nr; i++) {
                if (cmds[i].p_type == PT_LOAD) {
                        unsigned long p_align = cmds[i].p_align;
+                       unsigned long size;
 
                        /* skip non-power of two alignments as invalid */
                        if (!is_power_of_2(p_align))
                                continue;
                        alignment = max(alignment, p_align);
+
+                       /*
+                        * Try to align the binary to the largest folio
+                        * size that the page cache supports, so the
+                        * hardware can coalesce PTEs (e.g. arm64
+                        * contpte) or use PMD mappings for large folios.
+                        *
+                        * Use the largest power-of-2 that fits within
+                        * the segment size, capped by what the page
+                        * cache will allocate. Only align when the
+                        * segment's virtual address and file offset are
+                        * already aligned to the folio size, as
+                        * misalignment would prevent coalescing anyway.
+                        *
+                        * The segment size check avoids reducing ASLR
+                        * entropy for small binaries that cannot

fs/binfmt_elf.c static int elf_read(struct file *file, void *buf, size_t len, loff_t pos) · line 536

Quoted 10 lines

fs/binfmt_elf.c +10 -0

+                        * benefit.
+                        */
+                       if (!cmds[i].p_filesz)
+                               continue;
+                       size = rounddown_pow_of_two(cmds[i].p_filesz);
+                       size = min(size, max_folio_size);
+                       if (size > PAGE_SIZE &&
+                           IS_ALIGNED(cmds[i].p_vaddr, size) &&
+                           IS_ALIGNED(cmds[i].p_offset, size))
+                               alignment = max(alignment, size);

In my patch [1], by aligning eligible segments to PMD_SIZE, THP can quickly collapse them into large mappings with minimal warmup. That doesn’t happen with the current behavior. I think allowing a reasonably sized PMD (say <= 32M) is worth considering. All we really need here is to ensure virtual address alignment. The rest can be left to THP under always, which can decide whether to collapse or not based on memory pressure and other factors.

[1] https://lore.kernel.org/linux-fsdevel/[email protected]

Quoted 14 lines

fs/binfmt_elf.c +2 -2

                }
        }
 

@@ -1104,7 +1137,8 @@ static int load_elf_binary(struct linux_binprm *bprm)
                        }
 
                        /* Calculate any requested alignment. */
-                       alignment = maximum_alignment(elf_phdata, elf_ex->e_phnum);
+                       alignment = maximum_alignment(elf_phdata, elf_ex->e_phnum,
+                                                     bprm->file);
 
                        /**
                         * DOC: PIE handling
--

Quoted 2 lines

2.52.0

Thanks,
Rui

mapping_max_folio_size() reflects what the page cache will actually allocate for a given filesystem, since readahead caps folio allocation at mapping_max_folio_order() (in page_cache_ra_order()). If btrfs reports PAGE_SIZE, readahead won't allocate large folios for it, so there are no large folios to coalesce PTEs for, aligning the binary beyond that would only reduce ASLR entropy for no benefit.

I don't think we should over-align binaries on filesystems that can't take advantage of it.

Quoted 9 lines

mapping_max_folio_size() reflects what the page cache will actually allocate for a given filesystem, since readahead caps folio allocation at mapping_max_folio_order() (in page_cache_ra_order()). If btrfs reports PAGE_SIZE, readahead won't allocate large folios for it, so there are no large folios to coalesce PTEs for, aligning the binary beyond that would only reduce ASLR entropy for no benefit.

I don't think we should over-align binaries on filesystems that can't take advantage of it.

Ah, it looks like this might be overlooking another path that can create huge page mappings for read-only code segments: even when the filesystem (e.g. btrfs without experimental) didn't support large folios, READ_ONLY_THP_FOR_FS still allowed read-only file-backed code segments to be collapsed into huge page mappings via khugepaged.

As Wilcox pointed out, it may take quite some time for many filesystems to gain full large folio support? So what I'm trying to clarify is that using mapping_max_folio_size() on this path is not favorable for khugepaged-based optimizations.

Quoted 7 lines · WANG Rui

Quoted 7 lines

mapping_max_folio_size() reflects what the page cache will actually allocate for a given filesystem, since readahead caps folio allocation at mapping_max_folio_order() (in page_cache_ra_order()). If btrfs reports PAGE_SIZE, readahead won't allocate large folios for it, so there are no large folios to coalesce PTEs for, aligning the binary beyond that would only reduce ASLR entropy for no benefit.

Quoted 8 lines

Quoted 2 lines

I don't think we should over-align binaries on filesystems that can't take advantage of it.

Ah, it looks like this might be overlooking another path that can create huge page mappings for read-only code segments: even when the filesystem (e.g. btrfs without experimental) didn't support large folios, READ_ONLY_THP_FOR_FS still allowed read-only file-backed code segments to be collapsed into huge page mappings via khugepaged.

Quoted 5 lines

As Wilcox pointed out, it may take quite some time for many filesystems to gain full large folio support? So what I'm trying to clarify is that using mapping_max_folio_size() on this path is not favorable for khugepaged-based optimizations.

Nono, that's not what I'm pointing out! btrfs is simply not putting in the effort to support large folios, and that needs to change. READ_ONLY_THP_FOR_FS unnecessaily burdens the rest of the kernel. It was a great hack for its time and paved the path for a lot of what we have today, but it's time to remove it.

thp_get_unmapped_area() is the get_unmapped_area callback for filesystems like ext4, xfs, and btrfs. It attempts to align the virtual address for PMD_SIZE THP mappings, but on arm64 with 64K base pages PMD_SIZE is 512M, which is too large for typical shared library mappings, so the alignment always fails and falls back to PAGE_SIZE.

This means shared libraries loaded by ld.so via mmap() get 64K-aligned virtual addresses, preventing contpte mapping even when 2M folios are allocated with properly aligned file offsets and physical addresses.

Add a fallback in thp_get_unmapped_area_vmflags() that uses the filesystem's mapping_max_folio_size() to determine alignment, capped to the mapping length via rounddown_pow_of_two(len). This aligns mappings to the largest folio the page cache will actually allocate, without over-aligning small mappings.

The fallback is naturally a no-op for filesystems that don't support large folios and skips the retry when the alignment would equal PMD_SIZE (already attempted above).

Signed-off-by Usama Arif <[email protected]>

---
 mm/huge_memory.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

mm/huge_memory.c +0 -0

index 8e2746ea74adf..4005084c9c65b 100644
@@ -1242,6 +1242,20 @@ unsigned long thp_get_unmapped_area_vmflags(struct file *filp, unsigned long add
 	if (ret)
 		return ret;