mm: improve large folio readahead for exec memory

The mmap_miss heuristic is intended to stop speculative mmap readahead when a file looks like a random-access workload. That does not fit the VM_EXEC path very well.

VM_EXEC readahead is already constrained differently from ordinary mmap
read-around: it is bounded by the VMA, uses exec_folio_order() to choose an order useful for executable mappings, and sets async_size to 0 so it does not create follow-on readahead. When VM_HUGEPAGE is also present, the larger readahead is an explicit userspace opt-in.

The mmap_miss counter is decremented from cache-hit paths in do_async_mmap_readahead() and filemap_map_pages(). Those paths are not always enough to balance the synchronous miss increments for executable mappings. In particular, when fault-around is effectively disabled, such as configurations where fault_around_pages is 1, filemap_map_pages() is not reached from the fault path. The counter can then become a stale throttle for VM_EXEC mappings and suppress the readahead behavior that the executable-specific path is trying to provide.

Skip both mmap_miss increments and decrements for VM_EXEC mappings, matching the existing VM_SEQ_READ treatment and keeping the counter accounting symmetric.

Signed-off-by Usama Arif <[email protected]>

Reviewed-by Jan Kara <[email protected]>

Reviewed-by Kiryl Shutsemau (Meta) <[email protected]>

---
 mm/filemap.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

mm/filemap.c +7 -7

index cca20e350c95..a16b33e0fc71 100644
@@ -3339,7 +3339,7 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
 		}
 	}
 
-	if (!(vm_flags & VM_SEQ_READ)) {
+	if (!(vm_flags & (VM_SEQ_READ | VM_EXEC))) {
 		/* Avoid banging the cache line if not needed */
 		mmap_miss = READ_ONCE(ra->mmap_miss);
 		if (mmap_miss < MMAP_LOTSAMISS * 10)

@@ -3434,12 +3434,12 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf,
 	 * times for a single folio and break the balance with mmap_miss
 	 * increase in do_sync_mmap_readahead().
 	 *
-	 * VM_SEQ_READ mappings skip the mmap_miss increment in
+	 * VM_SEQ_READ and VM_EXEC mappings skip the mmap_miss increment in
 	 * do_sync_mmap_readahead(), so skip the decrement here as well to
 	 * keep the counter symmetric.
 	 */
 	if (likely(!folio_test_locked(folio)) &&
-	    !(vmf->vma->vm_flags & VM_SEQ_READ)) {
+	    !(vmf->vma->vm_flags & (VM_SEQ_READ | VM_EXEC))) {
 		mmap_miss = READ_ONCE(ra->mmap_miss);
 		if (mmap_miss)
 			WRITE_ONCE(ra->mmap_miss, --mmap_miss);

@@ -3941,14 +3941,14 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,
 		 * Don't decrease mmap_miss in this scenario to make sure
 		 * we can stop read-ahead.
 		 *
-		 * VM_SEQ_READ mappings skip the mmap_miss increment in
-		 * do_sync_mmap_readahead(), so skip the decrement here as
-		 * well to keep the counter symmetric.
+		 * VM_SEQ_READ and VM_EXEC mappings skip the mmap_miss
+		 * increment in do_sync_mmap_readahead(), so skip the
+		 * decrement here as well to keep the counter symmetric.
 		 */
 		if ((map_ret & VM_FAULT_NOPAGE) &&
 		    !(vmf->flags & FAULT_FLAG_TRIED) &&
 		    !folio_test_workingset(folio) &&
-		    !(vma->vm_flags & VM_SEQ_READ)) {
+		    !(vma->vm_flags & (VM_SEQ_READ | VM_EXEC))) {
 			unsigned short mmap_miss;
 
 			mmap_miss = READ_ONCE(file->f_ra.mmap_miss);

Signature · 1 line

2.52.0

The force_thp_readahead path in do_sync_mmap_readahead() is gated on HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER and always requests HPAGE_PMD_ORDER / HPAGE_PMD_NR. On configurations where HPAGE_PMD_ORDER exceeds MAX_PAGECACHE_ORDER, notably arm64 with a 64K base page size, VM_HUGEPAGE mappings cannot use this path and fall back to the non-forced mmap readahead path even when the mapping supports useful large folios.

Keep the existing PMD-sized behavior when HPAGE_PMD_ORDER fits in the page cache. When it does not, enable forced readahead for mappings that support large folios and request an order capped by both mapping_max_folio_order(mapping) and 2MB.

2MB is chosen as the cap because it matches the PMD size on x86_64 and on arm64 with 4K or 16K base pages, so the size/memory-pressure

tradeoff for folios of that size is already well understood. On arm64 with a 64K base page size, 2MB is also the contiguous-PTE (contpte) block size, so the resulting folios coalesce into a single TLB entry and reduce TLB pressure on the readahead path.

The final allocation order may still be clamped by page_cache_ra_order() to the mapping and request geometry, but this gives VM_HUGEPAGE mappings on such configurations a large-folio readahead request instead of dropping back to base-page readahead.

Signed-off-by Usama Arif <[email protected]>

---
 mm/filemap.c | 27 +++++++++++++++++++--------
 1 file changed, 19 insertions(+), 8 deletions(-)

mm/filemap.c +1 -2

index a16b33e0fc71..bfb891d9da1f 100644
@@ -3312,14 +3312,23 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
 	struct file *fpin = NULL;
 	vm_flags_t vm_flags = vmf->vma->vm_flags;
 	bool force_thp_readahead = false;
+	unsigned int thp_order = 0;
 	unsigned short mmap_miss;
 
 	ractl._max_index = vmf->vma->vm_pgoff + vma_pages(vmf->vma) - 1;
 
 	/* Use the readahead code, even if readahead is disabled */
-	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
-	    (vm_flags & VM_HUGEPAGE) && HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER)

mm/filemap.c +2 -0

+		}
+	}
 
 	if (!force_thp_readahead) {
 		/*

@@ -3354,17 +3363,19 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
 	}
 
 	if (force_thp_readahead) {

Quoted 12 lines · Usama Arif

The force_thp_readahead path in do_sync_mmap_readahead() is gated on HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER and always requests HPAGE_PMD_ORDER / HPAGE_PMD_NR. On configurations where HPAGE_PMD_ORDER exceeds MAX_PAGECACHE_ORDER, notably arm64 with a 64K base page size, VM_HUGEPAGE mappings cannot use this path and fall back to the non-forced mmap readahead path even when the mapping supports useful large folios.

Keep the existing PMD-sized behavior when HPAGE_PMD_ORDER fits in the page cache. When it does not, enable forced readahead for mappings that support large folios and request an order capped by both mapping_max_folio_order(mapping) and 2MB.

Quoted 2 lines

2MB is chosen as the cap because it matches the PMD size on x86_64 and on arm64 with 4K or 16K base pages, so the size/memory-pressure

16K base page size's PMDs should be 32M, no? Am I misunderstanding what you mean here?

Quoted 31 lines

tradeoff for folios of that size is already well understood. On arm64 with a 64K base page size, 2MB is also the contiguous-PTE (contpte) block size, so the resulting folios coalesce into a single TLB entry and reduce TLB pressure on the readahead path.

The final allocation order may still be clamped by page_cache_ra_order() to the mapping and request geometry, but this gives VM_HUGEPAGE mappings on such configurations a large-folio readahead request instead of dropping back to base-page readahead.

Signed-off-by Usama Arif <[email protected]>

---
 mm/filemap.c | 27 +++++++++++++++++++--------
 1 file changed, 19 insertions(+), 8 deletions(-)

mm/filemap.c +1 -2

index a16b33e0fc71..bfb891d9da1f 100644
@@ -3312,14 +3312,23 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
 	struct file *fpin = NULL;
 	vm_flags_t vm_flags = vmf->vma->vm_flags;
 	bool force_thp_readahead = false;
+	unsigned int thp_order = 0;
 	unsigned short mmap_miss;
 
 	ractl._max_index = vmf->vma->vm_pgoff + vma_pages(vmf->vma) - 1;
 
 	/* Use the readahead code, even if readahead is disabled */
-	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
-	    (vm_flags & VM_HUGEPAGE) && HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER)

mm/filemap.c static struct file *do_sync_mmap_readahead(struct vm_fault *vmf) · line 3329

Quoted 10 lines

mm/filemap.c +9 -1

-		force_thp_readahead = true;
+	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (vm_flags & VM_HUGEPAGE)) {
+		if (HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER) {
+			force_thp_readahead = true;
+			thp_order = HPAGE_PMD_ORDER;
+		} else if (mapping_large_folio_support(mapping)) {
+			force_thp_readahead = true;
+			thp_order = min_t(unsigned int,
+					  mapping_max_folio_order(mapping),
+					  get_order(SZ_2M));

This looks somewhat arbitrary to me (as does the old logic). It seems more natural (correct?) to do this:

	if (THP_ENABLED && (vm_flags & VM_HUGEPAGE)) {
		if (mapping_large_folio_support(mapping)) {
			force_thp_readahead = true;
			/*
			 * Cap THP readahead to either the max folio order the
			 * mapping supports, or the max order the page cache
			 * supports (useless to try more), or the hugepage PMD
			 * order (CPU can't benefit from larger).
			 */
			thp_order = min3(mapping_max_folio_order(mapping),
					MAX_PAGECACHE_ORDER, HPAGE_PMD_ORDER);
		}
	}

Quoted 9 lines

mm/filemap.c +2 -0

+		}
+	}
 
 	if (!force_thp_readahead) {
 		/*

@@ -3354,17 +3363,19 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
 	}
 
 	if (force_thp_readahead) {

mm/filemap.c static struct file *do_sync_mmap_readahead(struct vm_fault *vmf) · line 3372

Quoted 10 lines

mm/filemap.c +5 -3

+		unsigned long folio_nr_pages = 1UL << thp_order;
+
 		fpin = maybe_unlock_mmap_for_io(vmf, fpin);
-		ractl._index &= ~((unsigned long)HPAGE_PMD_NR - 1);
-		ra->size = HPAGE_PMD_NR;
+		ractl._index &= ~(folio_nr_pages - 1);
+		ra->size = folio_nr_pages;
 		/*
-		 * Fetch two PMD folios, so we get the chance to actually
+		 * Fetch two folios so we get the chance to actually

^^ large folios?

mm/filemap.c static struct file *do_sync_mmap_readahead(struct vm_fault *vmf) · line 3378

Quoted 8 lines

mm/filemap.c +2 -2

 		 * readahead, unless we've been told not to.
 		 */
 		if (!(vm_flags & VM_RAND_READ))
 			ra->size *= 2;
-		ra->async_size = HPAGE_PMD_NR;
-		ra->order = HPAGE_PMD_ORDER;
+		ra->async_size = folio_nr_pages;
+		ra->order = thp_order;

This part LGTM.

Quoted 13 lines · Pedro Falcato

Quoted 12 lines · Usama Arif

The force_thp_readahead path in do_sync_mmap_readahead() is gated on HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER and always requests HPAGE_PMD_ORDER / HPAGE_PMD_NR. On configurations where HPAGE_PMD_ORDER exceeds MAX_PAGECACHE_ORDER, notably arm64 with a 64K base page size, VM_HUGEPAGE mappings cannot use this path and fall back to the non-forced mmap readahead path even when the mapping supports useful large folios.

Keep the existing PMD-sized behavior when HPAGE_PMD_ORDER fits in the page cache. When it does not, enable forced readahead for mappings that support large folios and request an order capped by both mapping_max_folio_order(mapping) and 2MB.

Quoted 6 lines

Quoted 2 lines

2MB is chosen as the cap because it matches the PMD size on x86_64 and on arm64 with 4K or 16K base pages, so the size/memory-pressure

16K base page size's PMDs should be 32M, no? Am I misunderstanding what you mean here?

Yes, should have just said 4K. Messed up when rewriting the commit message. Andrew Thanks for merging in mm-new, would it be possible to remove "or 16K" from the commit message. I can send a new revision as well if that is easier and preferred.

Quoted 45 lines

Quoted 31 lines

tradeoff for folios of that size is already well understood. On arm64 with a 64K base page size, 2MB is also the contiguous-PTE (contpte) block size, so the resulting folios coalesce into a single TLB entry and reduce TLB pressure on the readahead path.

The final allocation order may still be clamped by page_cache_ra_order() to the mapping and request geometry, but this gives VM_HUGEPAGE mappings on such configurations a large-folio readahead request instead of dropping back to base-page readahead.

Signed-off-by Usama Arif <[email protected]>

---
 mm/filemap.c | 27 +++++++++++++++++++--------
 1 file changed, 19 insertions(+), 8 deletions(-)

mm/filemap.c +1 -2

index a16b33e0fc71..bfb891d9da1f 100644
@@ -3312,14 +3312,23 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
 	struct file *fpin = NULL;
 	vm_flags_t vm_flags = vmf->vma->vm_flags;
 	bool force_thp_readahead = false;
+	unsigned int thp_order = 0;
 	unsigned short mmap_miss;
 
 	ractl._max_index = vmf->vma->vm_pgoff + vma_pages(vmf->vma) - 1;
 
 	/* Use the readahead code, even if readahead is disabled */
-	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
-	    (vm_flags & VM_HUGEPAGE) && HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER)

mm/filemap.c static struct file *do_sync_mmap_readahead(struct vm_fault *vmf) · line 3329

Quoted 10 lines

mm/filemap.c +9 -1

-		force_thp_readahead = true;
+	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (vm_flags & VM_HUGEPAGE)) {
+		if (HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER) {
+			force_thp_readahead = true;
+			thp_order = HPAGE_PMD_ORDER;
+		} else if (mapping_large_folio_support(mapping)) {
+			force_thp_readahead = true;
+			thp_order = min_t(unsigned int,
+					  mapping_max_folio_order(mapping),
+					  get_order(SZ_2M));

This looks somewhat arbitrary to me (as does the old logic). It seems more natural (correct?) to do this:

Quoted 14 lines

	if (THP_ENABLED && (vm_flags & VM_HUGEPAGE)) {
		if (mapping_large_folio_support(mapping)) {
			force_thp_readahead = true;
			/*
			 * Cap THP readahead to either the max folio order the
			 * mapping supports, or the max order the page cache
			 * supports (useless to try more), or the hugepage PMD
			 * order (CPU can't benefit from larger).
			 */
			thp_order = min3(mapping_max_folio_order(mapping),
					MAX_PAGECACHE_ORDER, HPAGE_PMD_ORDER);
		}
	}

So this can result in extreme memory pressure. For example, on 64K base page size, with HPAGE_PMD_ORDER = 13 (512M) and MAX_PAGECACHE_ORDER = 11 (128M), this will evaluate THP order to 128M. I think Jan mentioned in previous versions that there are already reports of high memory pressure due to large file folios, and we are seeing this in our fleet as well with xfs.

I chose 2M cap as that is what most (not all) architectures which currently support this hugepage code flow are currently used to, and on archs + base page size combination which just evaluate this code to base page, for example 64K base page size, it will provide a performance uplift without causing excessive memory pressure.

In pagemap.h, we have

#define PREFERRED_MAX_PAGECACHE_ORDER HPAGE_PMD_ORDER
#define MAX_PAGECACHE_ORDER min(MAX_XAS_ORDER, PREFERRED_MAX_PAGECACHE_ORDER)

So MAX_PAGECACHE_ORDER <= HPAGE_PMD_ORDER is always true.

At filesystem/inode setup: mapping_set_folio_order_range()

	if (max > MAX_PAGECACHE_ORDER)
		max = MAX_PAGECACHE_ORDER;

so mapping_max_folio_order() <= MAX_PAGECACHE_ORDER is also always true.

which means mapping_max_folio_order(mapping) <= MAX_PAGECACHE_ORDER <= HPAGE_PMD_ORDER is always true, and you dont need the min3(..) in your diff.

Now the question is if then why not just do:

	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (vm_flags & VM_HUGEPAGE)) {
		if (mapping_large_folio_support(mapping)) {
			force_thp_readahead = true;
			thp_order = min_t(unsigned int,
					  mapping_max_folio_order(mapping),
					  get_order(SZ_2M));
		}
	}

This is because this will regress the 16K ARM case where we already got 32M folios. Someone might upgrade the kernel and start getting 2M folios now.

So IMHO the current code is correct, both in maintaining what the existing behaviour has been, but also trying to provide large folios to architectures that currently dont get them but in a controlled way.

Later on, when we think or evaluate that 2M was too small a cap, it can be raised. I think its very easy to raise a cap, but much more difficult to make the cap smaller, as we might get reports on kernel upgrades that someone was getting a certain size of folio but now its much smaller.

Quoted 94 lines · Usama Arif

Quoted 13 lines · Pedro Falcato

Quoted 12 lines · Usama Arif

The force_thp_readahead path in do_sync_mmap_readahead() is gated on HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER and always requests HPAGE_PMD_ORDER / HPAGE_PMD_NR. On configurations where HPAGE_PMD_ORDER exceeds MAX_PAGECACHE_ORDER, notably arm64 with a 64K base page size, VM_HUGEPAGE mappings cannot use this path and fall back to the non-forced mmap readahead path even when the mapping supports useful large folios.

Keep the existing PMD-sized behavior when HPAGE_PMD_ORDER fits in the page cache. When it does not, enable forced readahead for mappings that support large folios and request an order capped by both mapping_max_folio_order(mapping) and 2MB.

Quoted 6 lines

Quoted 2 lines

2MB is chosen as the cap because it matches the PMD size on x86_64 and on arm64 with 4K or 16K base pages, so the size/memory-pressure

16K base page size's PMDs should be 32M, no? Am I misunderstanding what you mean here?

Yes, should have just said 4K. Messed up when rewriting the commit message. Andrew Thanks for merging in mm-new, would it be possible to remove "or 16K" from the commit message. I can send a new revision as well if that is easier and preferred.

Quoted 45 lines

Quoted 31 lines

tradeoff for folios of that size is already well understood. On arm64 with a 64K base page size, 2MB is also the contiguous-PTE (contpte) block size, so the resulting folios coalesce into a single TLB entry and reduce TLB pressure on the readahead path.

The final allocation order may still be clamped by page_cache_ra_order() to the mapping and request geometry, but this gives VM_HUGEPAGE mappings on such configurations a large-folio readahead request instead of dropping back to base-page readahead.

Signed-off-by Usama Arif <[email protected]>

---
 mm/filemap.c | 27 +++++++++++++++++++--------
 1 file changed, 19 insertions(+), 8 deletions(-)

mm/filemap.c +1 -2

index a16b33e0fc71..bfb891d9da1f 100644
@@ -3312,14 +3312,23 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
 	struct file *fpin = NULL;
 	vm_flags_t vm_flags = vmf->vma->vm_flags;
 	bool force_thp_readahead = false;
+	unsigned int thp_order = 0;
 	unsigned short mmap_miss;
 
 	ractl._max_index = vmf->vma->vm_pgoff + vma_pages(vmf->vma) - 1;
 
 	/* Use the readahead code, even if readahead is disabled */
-	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
-	    (vm_flags & VM_HUGEPAGE) && HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER)

mm/filemap.c static struct file *do_sync_mmap_readahead(struct vm_fault *vmf) · line 3329

Quoted 10 lines

mm/filemap.c +9 -1

-		force_thp_readahead = true;
+	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (vm_flags & VM_HUGEPAGE)) {
+		if (HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER) {
+			force_thp_readahead = true;
+			thp_order = HPAGE_PMD_ORDER;
+		} else if (mapping_large_folio_support(mapping)) {
+			force_thp_readahead = true;
+			thp_order = min_t(unsigned int,
+					  mapping_max_folio_order(mapping),
+					  get_order(SZ_2M));

This looks somewhat arbitrary to me (as does the old logic). It seems more natural (correct?) to do this:

Quoted 14 lines

	if (THP_ENABLED && (vm_flags & VM_HUGEPAGE)) {
		if (mapping_large_folio_support(mapping)) {
			force_thp_readahead = true;
			/*
			 * Cap THP readahead to either the max folio order the
			 * mapping supports, or the max order the page cache
			 * supports (useless to try more), or the hugepage PMD
			 * order (CPU can't benefit from larger).
			 */
			thp_order = min3(mapping_max_folio_order(mapping),
					MAX_PAGECACHE_ORDER, HPAGE_PMD_ORDER);
		}
	}

So this can result in extreme memory pressure. For example, on 64K base page size, with HPAGE_PMD_ORDER = 13 (512M) and MAX_PAGECACHE_ORDER = 11 (128M), this will evaluate THP order to 128M. I think Jan mentioned in previous versions that there are already reports of high memory pressure due to large file folios, and we are seeing this in our fleet as well with xfs.

Quoted 4 lines

I chose 2M cap as that is what most (not all) architectures which currently support this hugepage code flow are currently used to, and on archs + base page size combination which just evaluate this code to base page, for example 64K base page size, it will provide a performance uplift without causing excessive memory pressure.

For what it's worth, memory pressure is still a problem even on 2MB folios. But yes, I understand.

For what it's worth, I don't think it's such a big issue in this case - you're asking for THPs, and you're getting them - though maybe not efficiently because these aren't at the PMD level. But this is also true for e.g x86 where an an order-3 folio or order-7 folio are completely useless for the hardware, since there is no mTHP to speak of (and, with that, you also lose PG_active and PG_reference precision as you only keep one of each for each folio, amongst other tradeoffs).

Quoted 6 lines

In pagemap.h, we have

#define PREFERRED_MAX_PAGECACHE_ORDER HPAGE_PMD_ORDER
#define MAX_PAGECACHE_ORDER min(MAX_XAS_ORDER, PREFERRED_MAX_PAGECACHE_ORDER)

Quoted 8 lines

So MAX_PAGECACHE_ORDER <= HPAGE_PMD_ORDER is always true.

At filesystem/inode setup: mapping_set_folio_order_range()

	if (max > MAX_PAGECACHE_ORDER)
		max = MAX_PAGECACHE_ORDER;

so mapping_max_folio_order() <= MAX_PAGECACHE_ORDER is also always true.

Right, good point. (well, I suppose it will not be true in the future as someone tries to harness PMD-level cont bits)

Quoted 16 lines

which means mapping_max_folio_order(mapping) <= MAX_PAGECACHE_ORDER <= HPAGE_PMD_ORDER is always true, and you dont need the min3(..) in your diff.

Now the question is if then why not just do:

	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (vm_flags & VM_HUGEPAGE)) {
		if (mapping_large_folio_support(mapping)) {
			force_thp_readahead = true;
			thp_order = min_t(unsigned int,
					  mapping_max_folio_order(mapping),
					  get_order(SZ_2M));
		}
	}

Quoted 2 lines

This is because this will regress the 16K ARM case where we already got 32M folios. Someone might upgrade the kernel and start getting 2M folios now.

So maybe limit to 32MB? It's still arbitrary but at least you get simpler logic. If the architecture does not support 32MiB folios, it will clamp the maximum folio order to HPAGE_PMD_ORDER, and you get the same result.

Does this sound correct?

Quoted 9 lines

So IMHO the current code is correct, both in maintaining what the existing behaviour has been, but also trying to provide large folios to architectures that currently dont get them but in a controlled way.

Later on, when we think or evaluate that 2M was too small a cap, it can be raised. I think its very easy to raise a cap, but much more difficult to make the cap smaller, as we might get reports on kernel upgrades that someone was getting a certain size of folio but now its much smaller.

I don't think this is a valid argument. You can also have reports that now folios are too large and they're regressing workloads because of thrashing/ compaction/possibly TLB thrashing. And indeed we have several reports of that
(https://lore.kernel.org/linux-fsdevel/[email protected]/
comes to mind, plus all of the anon THP regressions over the years, including in $current_year).

Bottom line is that changing things will always affect someone :) Particularly since the logic we have is not too careful at deciding what should or should not be a THP (both in anon and file cases). And if (once?) we make it smarter, it will surely also regress someone!

Quoted 24 lines · Pedro Falcato

Quoted 16 lines · Usama Arif

which means mapping_max_folio_order(mapping) <= MAX_PAGECACHE_ORDER <= HPAGE_PMD_ORDER is always true, and you dont need the min3(..) in your diff.

Now the question is if then why not just do:

	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (vm_flags & VM_HUGEPAGE)) {
		if (mapping_large_folio_support(mapping)) {
			force_thp_readahead = true;
			thp_order = min_t(unsigned int,
					  mapping_max_folio_order(mapping),
					  get_order(SZ_2M));
		}
	}

Quoted 2 lines

This is because this will regress the 16K ARM case where we already got 32M folios. Someone might upgrade the kernel and start getting 2M folios now.

So maybe limit to 32MB? It's still arbitrary but at least you get simpler logic. If the architecture does not support 32MiB folios, it will clamp the maximum folio order to HPAGE_PMD_ORDER, and you get the same result.

Quoted 2 lines

Does this sound correct?

Yes, so if we replace it with SZ_32M, it sounds correct. I just think the 32M size is too large. But as you pointed out, even 2M can be too large...

Quoted 4 lines

Bottom line is that changing things will always affect someone :) Particularly since the logic we have is not too careful at deciding what should or should not be a THP (both in anon and file cases). And if (once?) we make it smarter, it will surely also regress someone!

Yes completely agree on this as well.

So personally I do have a preference of keeping the cap at 2M atleast initially while we currently try and solve the issues we see with 2M alone. As we are already seeing reports of thrashing and compaction with just 2M, I dont think the logic in this patch with just an if else is that complicated.

Matthew, Jan, do you have any thoughts or strong preferences on cap size?

Quoted 25 lines · Usama Arif

Quoted 24 lines · Pedro Falcato

Quoted 16 lines · Usama Arif

which means mapping_max_folio_order(mapping) <= MAX_PAGECACHE_ORDER <= HPAGE_PMD_ORDER is always true, and you dont need the min3(..) in your diff.

Now the question is if then why not just do:

	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (vm_flags & VM_HUGEPAGE)) {
		if (mapping_large_folio_support(mapping)) {
			force_thp_readahead = true;
			thp_order = min_t(unsigned int,
					  mapping_max_folio_order(mapping),
					  get_order(SZ_2M));
		}
	}

Quoted 2 lines

This is because this will regress the 16K ARM case where we already got 32M folios. Someone might upgrade the kernel and start getting 2M folios now.

So maybe limit to 32MB? It's still arbitrary but at least you get simpler logic. If the architecture does not support 32MiB folios, it will clamp the maximum folio order to HPAGE_PMD_ORDER, and you get the same result.

Quoted 5 lines

Quoted 2 lines

Does this sound correct?

Yes, so if we replace it with SZ_32M, it sounds correct. I just think the 32M size is too large. But as you pointed out, even 2M can be too large...

So AFAIU the practical discussion is about two options:

1) limiting at 2MB with a slighly more complicated logic to keep mapping at PMD order for 16k pagesize on ARM but use 2MB pages for 64k pagesize on ARM

or

2) limit at 32MB with simple logic which results in larger (32MB) folios with 16k and 64k pagesize on ARM and thus larger memory overhead.

I'd like to maybe offer option 3): limit at 2MB with simple logic. This will reduce folio size on 16k pagesize ARM compared to 1) but do we really care? I.e., is there big enough practical performance impact with conpte and other tricks ARM is playing?

Quoted 12 lines

Quoted 4 lines

Bottom line is that changing things will always affect someone :) Particularly since the logic we have is not too careful at deciding what should or should not be a THP (both in anon and file cases). And if (once?) we make it smarter, it will surely also regress someone!

Yes completely agree on this as well.

So personally I do have a preference of keeping the cap at 2M atleast initially while we currently try and solve the issues we see with 2M alone. As we are already seeing reports of thrashing and compaction with just 2M, I dont think the logic in this patch with just an if else is that complicated.

Quoted 1 line

Matthew, Jan, do you have any thoughts or strong preferences on cap size?

Frankly, no strong opinion. I'd think 3) is worth trying for its simplicity and seeing whether somebody complains, otherwise I can live with both 1) and 2).

Honza

Quoted 25 lines · Usama Arif

Quoted 24 lines · Pedro Falcato

Quoted 16 lines · Usama Arif

which means mapping_max_folio_order(mapping) <= MAX_PAGECACHE_ORDER <= HPAGE_PMD_ORDER is always true, and you dont need the min3(..) in your diff.

Now the question is if then why not just do:

	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (vm_flags & VM_HUGEPAGE)) {
		if (mapping_large_folio_support(mapping)) {
			force_thp_readahead = true;
			thp_order = min_t(unsigned int,
					  mapping_max_folio_order(mapping),
					  get_order(SZ_2M));
		}
	}

Quoted 2 lines

This is because this will regress the 16K ARM case where we already got 32M folios. Someone might upgrade the kernel and start getting 2M folios now.

So maybe limit to 32MB? It's still arbitrary but at least you get simpler logic. If the architecture does not support 32MiB folios, it will clamp the maximum folio order to HPAGE_PMD_ORDER, and you get the same result.

Quoted 5 lines

Quoted 2 lines

Does this sound correct?

Yes, so if we replace it with SZ_32M, it sounds correct. I just think the 32M size is too large. But as you pointed out, even 2M can be too large...

So AFAIU the practical discussion is about two options:

1) limiting at 2MB with a slighly more complicated logic to keep mapping at PMD order for 16k pagesize on ARM but use 2MB pages for 64k pagesize on ARM

or

2) limit at 32MB with simple logic which results in larger (32MB) folios with 16k and 64k pagesize on ARM and thus larger memory overhead.

I'd like to maybe offer option 3): limit at 2MB with simple logic. This will reduce folio size on 16k pagesize ARM compared to 1) but do we really care? I.e., is there big enough practical performance impact with conpte and other tricks ARM is playing?

Quoted 39 lines · Jan Kara

Quoted 25 lines · Usama Arif

Quoted 24 lines · Pedro Falcato

Quoted 16 lines · Usama Arif

which means mapping_max_folio_order(mapping) <= MAX_PAGECACHE_ORDER <= HPAGE_PMD_ORDER is always true, and you dont need the min3(..) in your diff.

Now the question is if then why not just do:

	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (vm_flags & VM_HUGEPAGE)) {
		if (mapping_large_folio_support(mapping)) {
			force_thp_readahead = true;
			thp_order = min_t(unsigned int,
					  mapping_max_folio_order(mapping),
					  get_order(SZ_2M));
		}
	}

Quoted 2 lines

This is because this will regress the 16K ARM case where we already got 32M folios. Someone might upgrade the kernel and start getting 2M folios now.

So maybe limit to 32MB? It's still arbitrary but at least you get simpler logic. If the architecture does not support 32MiB folios, it will clamp the maximum folio order to HPAGE_PMD_ORDER, and you get the same result.

Quoted 5 lines

Quoted 2 lines

Does this sound correct?

Yes, so if we replace it with SZ_32M, it sounds correct. I just think the 32M size is too large. But as you pointed out, even 2M can be too large...

So AFAIU the practical discussion is about two options:

1) limiting at 2MB with a slighly more complicated logic to keep mapping at PMD order for 16k pagesize on ARM but use 2MB pages for 64k pagesize on ARM

or

Quoted 8 lines

2) limit at 32MB with simple logic which results in larger (32MB) folios with 16k and 64k pagesize on ARM and thus larger memory overhead.

I'd like to maybe offer option 3): limit at 2MB with simple logic. This will reduce folio size on 16k pagesize ARM compared to 1) but do we really care? I.e., is there big enough practical performance impact with conpte and other tricks ARM is playing?

arm64 16K contpte tops out at 256KB TLB entries. It's quite a lot smaller than a PMD entry. Also, something that was discussed at LSFMM was its effectiveness. Apparently, most of the gains seem to sit on actually having a larger page size (perhaps Dev/Ryan can comment; sadly the slides were not posted anywhere on the ML, so I don't have numbers).

To me, the question is quite clear: do we trust users that say "please give me hugepages" enough to unconditionally give them hugepages? I would assume the answer lies somewhere between "yes" and "no", but 32MB I would say is not particularly excessive. 512MB is... much worse.

Quoted 55 lines · Pedro Falcato

Quoted 39 lines · Jan Kara

Quoted 25 lines · Usama Arif

Quoted 24 lines · Pedro Falcato

Quoted 16 lines · Usama Arif

which means mapping_max_folio_order(mapping) <= MAX_PAGECACHE_ORDER <= HPAGE_PMD_ORDER is always true, and you dont need the min3(..) in your diff.

Now the question is if then why not just do:

	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (vm_flags & VM_HUGEPAGE)) {
		if (mapping_large_folio_support(mapping)) {
			force_thp_readahead = true;
			thp_order = min_t(unsigned int,
					  mapping_max_folio_order(mapping),
					  get_order(SZ_2M));
		}
	}

Quoted 2 lines

This is because this will regress the 16K ARM case where we already got 32M folios. Someone might upgrade the kernel and start getting 2M folios now.

So maybe limit to 32MB? It's still arbitrary but at least you get simpler logic. If the architecture does not support 32MiB folios, it will clamp the maximum folio order to HPAGE_PMD_ORDER, and you get the same result.

Quoted 5 lines

Quoted 2 lines

Does this sound correct?

Yes, so if we replace it with SZ_32M, it sounds correct. I just think the 32M size is too large. But as you pointed out, even 2M can be too large...

So AFAIU the practical discussion is about two options:

1) limiting at 2MB with a slighly more complicated logic to keep mapping at PMD order for 16k pagesize on ARM but use 2MB pages for 64k pagesize on ARM

or

Quoted 8 lines

2) limit at 32MB with simple logic which results in larger (32MB) folios with 16k and 64k pagesize on ARM and thus larger memory overhead.

I'd like to maybe offer option 3): limit at 2MB with simple logic. This will reduce folio size on 16k pagesize ARM compared to 1) but do we really care? I.e., is there big enough practical performance impact with conpte and other tricks ARM is playing?

arm64 16K contpte tops out at 256KB TLB entries. It's quite a lot smaller than a PMD entry. Also, something that was discussed at LSFMM was its effectiveness. Apparently, most of the gains seem to sit on actually having a larger page size (perhaps Dev/Ryan can comment; sadly the slides were not posted anywhere on the ML, so I don't have numbers).

Quoted 5 lines

To me, the question is quite clear: do we trust users that say "please give me hugepages" enough to unconditionally give them hugepages? I would assume the answer lies somewhere between "yes" and "no", but 32MB I would say is not particularly excessive. 512MB is... much worse.

I think the other question also is, if the userspace asks for hugepages, is it asking for the biggest possible one? I think the answer is yes on 4K base page size when largest is 2M, but maybe not the case for 16K and 64K.

/sys/kernel/mm/transparent_hugepage/hugepages-* is supposed to be used for anon only, but maybe in the future we could use that to determine the size of THP to give to the user for file over here? For e.g. over here we could have used it to determine what the biggest size is that has madvise (or always) set and used it over here. Its probably a much bigger discussion.

Quoted 103 lines · Pedro Falcato

Quoted 94 lines · Usama Arif

Quoted 13 lines · Pedro Falcato

Quoted 12 lines · Usama Arif

The force_thp_readahead path in do_sync_mmap_readahead() is gated on HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER and always requests HPAGE_PMD_ORDER / HPAGE_PMD_NR. On configurations where HPAGE_PMD_ORDER exceeds MAX_PAGECACHE_ORDER, notably arm64 with a 64K base page size, VM_HUGEPAGE mappings cannot use this path and fall back to the non-forced mmap readahead path even when the mapping supports useful large folios.

Keep the existing PMD-sized behavior when HPAGE_PMD_ORDER fits in the page cache. When it does not, enable forced readahead for mappings that support large folios and request an order capped by both mapping_max_folio_order(mapping) and 2MB.

Quoted 6 lines

Quoted 2 lines

2MB is chosen as the cap because it matches the PMD size on x86_64 and on arm64 with 4K or 16K base pages, so the size/memory-pressure

16K base page size's PMDs should be 32M, no? Am I misunderstanding what you mean here?

Yes, should have just said 4K. Messed up when rewriting the commit message. Andrew Thanks for merging in mm-new, would it be possible to remove "or 16K" from the commit message. I can send a new revision as well if that is easier and preferred.

Quoted 45 lines

Quoted 31 lines

tradeoff for folios of that size is already well understood. On arm64 with a 64K base page size, 2MB is also the contiguous-PTE (contpte) block size, so the resulting folios coalesce into a single TLB entry and reduce TLB pressure on the readahead path.

The final allocation order may still be clamped by page_cache_ra_order() to the mapping and request geometry, but this gives VM_HUGEPAGE mappings on such configurations a large-folio readahead request instead of dropping back to base-page readahead.

Signed-off-by Usama Arif <[email protected]>

---
 mm/filemap.c | 27 +++++++++++++++++++--------
 1 file changed, 19 insertions(+), 8 deletions(-)

mm/filemap.c +1 -2

index a16b33e0fc71..bfb891d9da1f 100644
@@ -3312,14 +3312,23 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
 	struct file *fpin = NULL;
 	vm_flags_t vm_flags = vmf->vma->vm_flags;
 	bool force_thp_readahead = false;
+	unsigned int thp_order = 0;
 	unsigned short mmap_miss;
 
 	ractl._max_index = vmf->vma->vm_pgoff + vma_pages(vmf->vma) - 1;
 
 	/* Use the readahead code, even if readahead is disabled */
-	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
-	    (vm_flags & VM_HUGEPAGE) && HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER)

mm/filemap.c static struct file *do_sync_mmap_readahead(struct vm_fault *vmf) · line 3329

Quoted 10 lines

mm/filemap.c +9 -1

-		force_thp_readahead = true;
+	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (vm_flags & VM_HUGEPAGE)) {
+		if (HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER) {
+			force_thp_readahead = true;
+			thp_order = HPAGE_PMD_ORDER;
+		} else if (mapping_large_folio_support(mapping)) {
+			force_thp_readahead = true;
+			thp_order = min_t(unsigned int,
+					  mapping_max_folio_order(mapping),
+					  get_order(SZ_2M));

This looks somewhat arbitrary to me (as does the old logic). It seems more natural (correct?) to do this:

Quoted 14 lines

	if (THP_ENABLED && (vm_flags & VM_HUGEPAGE)) {
		if (mapping_large_folio_support(mapping)) {
			force_thp_readahead = true;
			/*
			 * Cap THP readahead to either the max folio order the
			 * mapping supports, or the max order the page cache
			 * supports (useless to try more), or the hugepage PMD
			 * order (CPU can't benefit from larger).
			 */
			thp_order = min3(mapping_max_folio_order(mapping),
					MAX_PAGECACHE_ORDER, HPAGE_PMD_ORDER);
		}
	}

So this can result in extreme memory pressure. For example, on 64K base page size, with HPAGE_PMD_ORDER = 13 (512M) and MAX_PAGECACHE_ORDER = 11 (128M), this will evaluate THP order to 128M. I think Jan mentioned in previous versions that there are already reports of high memory pressure due to large file folios, and we are seeing this in our fleet as well with xfs.

Quoted 4 lines

I chose 2M cap as that is what most (not all) architectures which currently support this hugepage code flow are currently used to, and on archs + base page size combination which just evaluate this code to base page, for example 64K base page size, it will provide a performance uplift without causing excessive memory pressure.

For what it's worth, memory pressure is still a problem even on 2MB folios. But yes, I understand.

Quoted 8 lines

For what it's worth, I don't think it's such a big issue in this case - you're asking for THPs, and you're getting them - though maybe not efficiently because these aren't at the PMD level. But this is also true for e.g x86 where an an order-3 folio or order-7 folio are completely useless for the hardware, since there is no mTHP to speak of (and, with that, you also lose PG_active and PG_reference precision as you only keep one of each for each folio, amongst other tradeoffs).

So there is hardware TLB coalescing on almost every new AMD CPU, I believe it is at 32kB, and we would benefit from that. On top of that, we would reduce pagefaults.

The force_thp_readahead path in do_sync_mmap_readahead() is gated on HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER and always requests HPAGE_PMD_ORDER / HPAGE_PMD_NR. On configurations where HPAGE_PMD_ORDER exceeds MAX_PAGECACHE_ORDER, notably arm64 with a 64K base page size, VM_HUGEPAGE mappings cannot use this path and fall back to the non-forced mmap readahead path even when the mapping supports useful large folios.

Keep the existing PMD-sized behavior when HPAGE_PMD_ORDER fits in the page cache. When it does not, enable forced readahead for mappings that support large folios and request an order capped by both mapping_max_folio_order(mapping) and 2MB.

2MB is chosen as the cap because it matches the PMD size on x86_64 and on arm64 with 4K or 16K base pages, so the size/memory-pressure tradeoff for folios of that size is already well understood. On arm64 with a 64K base page size, 2MB is also the contiguous-PTE (contpte) block size, so the resulting folios coalesce into a single TLB entry and reduce TLB pressure on the readahead path.

The final allocation order may still be clamped by page_cache_ra_order() to the mapping and request geometry, but this gives VM_HUGEPAGE mappings on such configurations a large-folio readahead request instead of dropping back to base-page readahead.

Signed-off-by Usama Arif <[email protected]>

---
 mm/filemap.c | 27 +++++++++++++++++++--------
 1 file changed, 19 insertions(+), 8 deletions(-)

mm/filemap.c +3 -3

index a16b33e0fc71..bfb891d9da1f 100644
@@ -3312,14 +3312,23 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
 	struct file *fpin = NULL;
 	vm_flags_t vm_flags = vmf->vma->vm_flags;
 	bool force_thp_readahead = false;
+	unsigned int thp_order = 0;
 	unsigned short mmap_miss;
 
 	ractl._max_index = vmf->vma->vm_pgoff + vma_pages(vmf->vma) - 1;
 
 	/* Use the readahead code, even if readahead is disabled */
-	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
-	    (vm_flags & VM_HUGEPAGE) && HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER)
-		force_thp_readahead = true;
+	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (vm_flags & VM_HUGEPAGE)) {
+		if (HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER) {

Two checks in do_sync_mmap_readahead() limit large-folio readahead:

1. The mmap_miss heuristic is meant to throttle wasteful speculative readahead. It is currently also applied to the VM_EXEC readahead path, which is targeted rather than speculative. Once mmap_miss exceeds MMAP_LOTSAMISS, exec readahead - including the large-folio order requested by exec_folio_order() - is disabled. On configurations where the mmap_miss decrement paths are not active (see patch 1) the counter only grows, so exec readahead is permanently disabled after the first 100 faults.

2. The force_thp_readahead path is gated only on HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER and always drives the readahead at HPAGE_PMD_ORDER. Configurations where HPAGE_PMD_ORDER exceeds MAX_PAGECACHE_ORDER never reach this path, even when the mapping itself supports usefully large folios well below the cap.

Both issues are most visible on arm64 with a 64K base page size, where HPAGE_PMD_ORDER is 13 (512MB) -- above MAX_PAGECACHE_ORDER
(11) -- and where fault_around_pages collapses to 1 disabling should_fault_around() (one of the two mmap_miss decrement sites). However the fixes are architecture-agnostic: patch 1 reflects the nature of VM_EXEC readahead regardless of base page size, and patch 2 generalises the gate so any mapping advertising a usefully large maximum folio order can benefit.