Skip to contentSkip to conversation

mm: readahead: improve mmap_miss heuristic for concurrent faults

PATCH In review Roman Gushchin 4 participants 6 messages Aug 15, 2025 – Aug 25, 2025

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm: readahead: improve mmap_miss heuristic for concurrent faults

If two or more threads of an application faulting on the same folio, the mmap_miss counter can be decreased multiple times. It breaks the mmap_miss heuristic and keeps the readahead enabled even under extreme levels of memory pressure.

It happens often if file folios backing a multi-threaded application are getting evicted and re-faulted.

Fix it by skipping decreasing mmap_miss if the folio is locked.

This change was evaluated on several hundred thousands hosts in Google's production over a couple of weeks. The number of containers being stuck in a vicious reclaim cycle for a long time was reduced several fold (~10-20x), as well as the overall fleet-wide cpu time spent in direct memory reclaim was meaningfully reduced. No regressions were observed.

Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Jan Kara <[email protected]>
---
 mm/filemap.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
mm/filemap.c +11 -3
index c21e98657e0b..983ba1019674 100644
@@ -3324,9 +3324,17 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf,
 	if (vmf->vma->vm_flags & VM_RAND_READ || !ra->ra_pages)
 		return fpin;
 
-	mmap_miss = READ_ONCE(ra->mmap_miss);
-	if (mmap_miss)
-		WRITE_ONCE(ra->mmap_miss, --mmap_miss);
+	/*
+	 * If the folio is locked, we're likely racing against another fault.
+	 * Don't touch the mmap_miss counter to avoid decreasing it multiple
+	 * times for a single folio and break the balance with mmap_miss
+	 * increase in do_sync_mmap_readahead().
+	 */
+	if (likely(!folio_test_locked(folio))) {
+		mmap_miss = READ_ONCE(ra->mmap_miss);
+		if (mmap_miss)
+			WRITE_ONCE(ra->mmap_miss, --mmap_miss);
+	}
 
 	if (folio_test_readahead(folio)) {
 		fpin = maybe_unlock_mmap_for_io(vmf, fpin);
Signature · 1 line
2.50.1
#1 lore raw
Quoted 37 lines · Roman Gushchin

If two or more threads of an application faulting on the same folio, the mmap_miss counter can be decreased multiple times. It breaks the mmap_miss heuristic and keeps the readahead enabled even under extreme levels of memory pressure.

It happens often if file folios backing a multi-threaded application are getting evicted and re-faulted.

Fix it by skipping decreasing mmap_miss if the folio is locked.

This change was evaluated on several hundred thousands hosts in Google's production over a couple of weeks. The number of containers being stuck in a vicious reclaim cycle for a long time was reduced several fold (~10-20x), as well as the overall fleet-wide cpu time spent in direct memory reclaim was meaningfully reduced. No regressions were observed.

Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Jan Kara <[email protected]>
---
  mm/filemap.c | 14 +++++++++++---
  1 file changed, 11 insertions(+), 3 deletions(-)
mm/filemap.c +1 -3
index c21e98657e0b..983ba1019674 100644
@@ -3324,9 +3324,17 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf,
  	if (vmf->vma->vm_flags & VM_RAND_READ || !ra->ra_pages)
  		return fpin;
 
-	mmap_miss = READ_ONCE(ra->mmap_miss);
-	if (mmap_miss)
-		WRITE_ONCE(ra->mmap_miss, --mmap_miss);
+	/*

mm/filemap.c static struct file *do_async_mmap_readahead(struct vm_fault *vmf, · line 3337

Quoted 10 lines
mm/filemap.c +10 -0
+	 * If the folio is locked, we're likely racing against another fault.
+	 * Don't touch the mmap_miss counter to avoid decreasing it multiple
+	 * times for a single folio and break the balance with mmap_miss
+	 * increase in do_sync_mmap_readahead().
+	 */
+	if (likely(!folio_test_locked(folio))) {
+		mmap_miss = READ_ONCE(ra->mmap_miss);
+		if (mmap_miss)
+			WRITE_ONCE(ra->mmap_miss, --mmap_miss);
+	}

Makes sense to me, bud I am no readahead expert.

Signature · 3 lines
Cheers

David / dhildenb
maintainer Reviewed-by on v1 1/1
#2 lore raw
In reply to Roman Gushchin
Quoted 17 lines · Roman Gushchin

If two or more threads of an application faulting on the same folio, the mmap_miss counter can be decreased multiple times. It breaks the mmap_miss heuristic and keeps the readahead enabled even under extreme levels of memory pressure.

It happens often if file folios backing a multi-threaded application are getting evicted and re-faulted.

Fix it by skipping decreasing mmap_miss if the folio is locked.

This change was evaluated on several hundred thousands hosts in Google's production over a couple of weeks. The number of containers being stuck in a vicious reclaim cycle for a long time was reduced several fold (~10-20x), as well as the overall fleet-wide cpu time spent in direct memory reclaim was meaningfully reduced. No regressions were observed.

Quoted 4 lines
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Jan Kara <[email protected]>

Looks good! Feel free to add:

Honza

Quoted 32 lines
---
 mm/filemap.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
mm/filemap.c +11 -3
index c21e98657e0b..983ba1019674 100644
@@ -3324,9 +3324,17 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf,
 	if (vmf->vma->vm_flags & VM_RAND_READ || !ra->ra_pages)
 		return fpin;
 
-	mmap_miss = READ_ONCE(ra->mmap_miss);
-	if (mmap_miss)
-		WRITE_ONCE(ra->mmap_miss, --mmap_miss);
+	/*
+	 * If the folio is locked, we're likely racing against another fault.
+	 * Don't touch the mmap_miss counter to avoid decreasing it multiple
+	 * times for a single folio and break the balance with mmap_miss
+	 * increase in do_sync_mmap_readahead().
+	 */
+	if (likely(!folio_test_locked(folio))) {
+		mmap_miss = READ_ONCE(ra->mmap_miss);
+		if (mmap_miss)
+			WRITE_ONCE(ra->mmap_miss, --mmap_miss);
+	}
 
 	if (folio_test_readahead(folio)) {
 		fpin = maybe_unlock_mmap_for_io(vmf, fpin);
Signature · 1 line
2.50.1
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#3 lore raw
Quoted 18 lines · Jan Kara
Quoted 17 lines · Roman Gushchin

If two or more threads of an application faulting on the same folio, the mmap_miss counter can be decreased multiple times. It breaks the mmap_miss heuristic and keeps the readahead enabled even under extreme levels of memory pressure.

It happens often if file folios backing a multi-threaded application are getting evicted and re-faulted.

Fix it by skipping decreasing mmap_miss if the folio is locked.

This change was evaluated on several hundred thousands hosts in Google's production over a couple of weeks. The number of containers being stuck in a vicious reclaim cycle for a long time was reduced several fold (~10-20x), as well as the overall fleet-wide cpu time spent in direct memory reclaim was meaningfully reduced. No regressions were observed.

Quoted 8 lines
Quoted 4 lines
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Jan Kara <[email protected]>

Looks good! Feel free to add:

Thank you!

#4 lore raw
In reply to Roman Gushchin
Quoted 37 lines · Roman Gushchin

If two or more threads of an application faulting on the same folio, the mmap_miss counter can be decreased multiple times. It breaks the mmap_miss heuristic and keeps the readahead enabled even under extreme levels of memory pressure.

It happens often if file folios backing a multi-threaded application are getting evicted and re-faulted.

Fix it by skipping decreasing mmap_miss if the folio is locked.

This change was evaluated on several hundred thousands hosts in Google's production over a couple of weeks. The number of containers being stuck in a vicious reclaim cycle for a long time was reduced several fold (~10-20x), as well as the overall fleet-wide cpu time spent in direct memory reclaim was meaningfully reduced. No regressions were observed.

Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Jan Kara <[email protected]>
---
 mm/filemap.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
mm/filemap.c +1 -3
index c21e98657e0b..983ba1019674 100644
@@ -3324,9 +3324,17 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf,
 	if (vmf->vma->vm_flags & VM_RAND_READ || !ra->ra_pages)
 		return fpin;
 
-	mmap_miss = READ_ONCE(ra->mmap_miss);
-	if (mmap_miss)
-		WRITE_ONCE(ra->mmap_miss, --mmap_miss);
+	/*

mm/filemap.c static struct file *do_async_mmap_readahead(struct vm_fault *vmf, · line 3337

Quoted 10 lines
mm/filemap.c +10 -0
+	 * If the folio is locked, we're likely racing against another fault.
+	 * Don't touch the mmap_miss counter to avoid decreasing it multiple
+	 * times for a single folio and break the balance with mmap_miss
+	 * increase in do_sync_mmap_readahead().
+	 */
+	if (likely(!folio_test_locked(folio))) {
+		mmap_miss = READ_ONCE(ra->mmap_miss);
+		if (mmap_miss)
+			WRITE_ONCE(ra->mmap_miss, --mmap_miss);
+	}

I'm not an mm person.

The comment implies the change fixes the race, but it is not at all clear to me how.

Does it merely make it significantly less likely?

#5 lore raw
Quoted 51 lines · Mateusz Guzik
Quoted 37 lines · Roman Gushchin

If two or more threads of an application faulting on the same folio, the mmap_miss counter can be decreased multiple times. It breaks the mmap_miss heuristic and keeps the readahead enabled even under extreme levels of memory pressure.

It happens often if file folios backing a multi-threaded application are getting evicted and re-faulted.

Fix it by skipping decreasing mmap_miss if the folio is locked.

This change was evaluated on several hundred thousands hosts in Google's production over a couple of weeks. The number of containers being stuck in a vicious reclaim cycle for a long time was reduced several fold (~10-20x), as well as the overall fleet-wide cpu time spent in direct memory reclaim was meaningfully reduced. No regressions were observed.

Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Jan Kara <[email protected]>
---
 mm/filemap.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
mm/filemap.c +1 -3
index c21e98657e0b..983ba1019674 100644
@@ -3324,9 +3324,17 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf,
 	if (vmf->vma->vm_flags & VM_RAND_READ || !ra->ra_pages)
 		return fpin;
 
-	mmap_miss = READ_ONCE(ra->mmap_miss);
-	if (mmap_miss)
-		WRITE_ONCE(ra->mmap_miss, --mmap_miss);
+	/*

mm/filemap.c static struct file *do_async_mmap_readahead(struct vm_fault *vmf, · line 3337

Quoted 10 lines
mm/filemap.c +10 -0
+	 * If the folio is locked, we're likely racing against another fault.
+	 * Don't touch the mmap_miss counter to avoid decreasing it multiple
+	 * times for a single folio and break the balance with mmap_miss
+	 * increase in do_sync_mmap_readahead().
+	 */
+	if (likely(!folio_test_locked(folio))) {
+		mmap_miss = READ_ONCE(ra->mmap_miss);
+		if (mmap_miss)
+			WRITE_ONCE(ra->mmap_miss, --mmap_miss);
+	}

I'm not an mm person.

Quoted 4 lines

The comment implies the change fixes the race, but it is not at all clear to me how.

Does it merely make it significantly less likely?

It's not fixing any race, it's fixing the imbalance in the upward and downward pressure on the mmap_miss variable. This improves the readahead behavior under very special circumstances: a multi-threaded application under very heavy memory pressure. There should be no visible difference in behavior in other cases.

Thanks!