mm: readahead: improve mmap_miss heuristic for concurrent faults
Subsystems (2)
PATCH 1/1 mm: readahead: improve mmap_miss heuristic for concurrent faults
If two or more threads of an application faulting on the same folio, the mmap_miss counter can be decreased multiple times. It breaks the mmap_miss heuristic and keeps the readahead enabled even under extreme levels of memory pressure.
It happens often if file folios backing a multi-threaded application are getting evicted and re-faulted.
Fix it by skipping decreasing mmap_miss if the folio is locked.
This change was evaluated on several hundred thousands hosts in Google's production over a couple of weeks. The number of containers being stuck in a vicious reclaim cycle for a long time was reduced several fold (~10-20x), as well as the overall fleet-wide cpu time spent in direct memory reclaim was meaningfully reduced. No regressions were observed.
--- mm/filemap.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
mm/filemap.c +11 -3
@@ -3324,9 +3324,17 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf,if (vmf->vma->vm_flags &VM_RAND_READ || !ra->ra_pages)return fpin; - mmap_miss =READ_ONCE (ra->mmap_miss); -if (mmap_miss) -WRITE_ONCE (ra->mmap_miss, --mmap_miss); +/* +* If the folio is locked, we're likely racing against another fault. +* Don't touch the mmap_miss counter to avoid decreasing it multiple +* times for a single folio and break the balance with mmap_miss +* increase in do_sync_mmap_readahead(). +*/ +if (likely (!folio_test_locked (folio))) { + mmap_miss =READ_ONCE (ra->mmap_miss); +if (mmap_miss) +WRITE_ONCE (ra->mmap_miss, --mmap_miss); + }if (folio_test_readahead (folio)) { fpin =maybe_unlock_mmap_for_io (vmf, fpin);
Signature · 1 line
2.50.1
Quoted 37 lines · Roman Gushchin
If two or more threads of an application faulting on the same folio, the
mmap_misscounter can be decreased multiple times. It breaks themmap_missheuristic and keeps the readahead enabled even under extreme levels of memory pressure.It happens often if file folios backing a multi-threaded application are getting evicted and re-faulted.
Fix it by skipping decreasing
mmap_missif the folio is locked.This change was evaluated on several hundred thousands hosts in Google's production over a couple of weeks. The number of containers being stuck in a vicious reclaim cycle for a long time was reduced several fold (~10-20x), as well as the overall fleet-wide cpu time spent in direct memory reclaim was meaningfully reduced. No regressions were observed.
Signed-off-by Roman Gushchin <[email protected]>Cc Matthew Wilcox (Oracle) <[email protected]>Cc Jan Kara <[email protected]>--- mm/filemap.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)mm/filemap.c +1
-3@@ -3324,9 +3324,17 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf,if (vmf->vma->vm_flags &VM_RAND_READ || !ra->ra_pages)return fpin; - mmap_miss =READ_ONCE (ra->mmap_miss); -if (mmap_miss) -WRITE_ONCE (ra->mmap_miss, --mmap_miss); +/*
mm/filemap.c static struct file *do_async_mmap_readahead(struct vm_fault *vmf, · line 3337
Quoted 10 lines
mm/filemap.c +10
-0+* If the folio is locked, we're likely racing against another fault. +* Don't touch the mmap_miss counter to avoid decreasing it multiple +* times for a single folio and break the balance with mmap_miss +* increase in do_sync_mmap_readahead(). +*/ +if (likely (!folio_test_locked (folio))) { + mmap_miss =READ_ONCE (ra->mmap_miss); +if (mmap_miss) +WRITE_ONCE (ra->mmap_miss, --mmap_miss); + }
Makes sense to me, bud I am no readahead expert.
Signature · 3 lines
Cheers David / dhildenb
Quoted 17 lines · Roman Gushchin
If two or more threads of an application faulting on the same folio, the
mmap_misscounter can be decreased multiple times. It breaks themmap_missheuristic and keeps the readahead enabled even under extreme levels of memory pressure.It happens often if file folios backing a multi-threaded application are getting evicted and re-faulted.
Fix it by skipping decreasing
mmap_missif the folio is locked.This change was evaluated on several hundred thousands hosts in Google's production over a couple of weeks. The number of containers being stuck in a vicious reclaim cycle for a long time was reduced several fold (~10-20x), as well as the overall fleet-wide cpu time spent in direct memory reclaim was meaningfully reduced. No regressions were observed.
Quoted 4 lines
Signed-off-by Roman Gushchin <[email protected]>Cc Matthew Wilcox (Oracle) <[email protected]>Cc Jan Kara <[email protected]>
Looks good! Feel free to add:
Honza
Quoted 32 lines
--- mm/filemap.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)mm/filemap.c +11
-3@@ -3324,9 +3324,17 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf,if (vmf->vma->vm_flags &VM_RAND_READ || !ra->ra_pages)return fpin; - mmap_miss =READ_ONCE (ra->mmap_miss); -if (mmap_miss) -WRITE_ONCE (ra->mmap_miss, --mmap_miss); +/* +* If the folio is locked, we're likely racing against another fault. +* Don't touch the mmap_miss counter to avoid decreasing it multiple +* times for a single folio and break the balance with mmap_miss +* increase in do_sync_mmap_readahead(). +*/ +if (likely (!folio_test_locked (folio))) { + mmap_miss =READ_ONCE (ra->mmap_miss); +if (mmap_miss) +WRITE_ONCE (ra->mmap_miss, --mmap_miss); + }if (folio_test_readahead (folio)) { fpin =maybe_unlock_mmap_for_io (vmf, fpin);Signature · 1 line
2.50.1
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 18 lines · Jan Kara
Quoted 17 lines · Roman Gushchin
If two or more threads of an application faulting on the same folio, the
mmap_misscounter can be decreased multiple times. It breaks themmap_missheuristic and keeps the readahead enabled even under extreme levels of memory pressure.It happens often if file folios backing a multi-threaded application are getting evicted and re-faulted.
Fix it by skipping decreasing
mmap_missif the folio is locked.This change was evaluated on several hundred thousands hosts in Google's production over a couple of weeks. The number of containers being stuck in a vicious reclaim cycle for a long time was reduced several fold (~10-20x), as well as the overall fleet-wide cpu time spent in direct memory reclaim was meaningfully reduced. No regressions were observed.
Quoted 8 lines
Quoted 4 lines
Signed-off-by Roman Gushchin <[email protected]>Cc Matthew Wilcox (Oracle) <[email protected]>Cc Jan Kara <[email protected]>Looks good! Feel free to add:
Reviewed-by Jan Kara <[email protected]>
Thank you!
Quoted 37 lines · Roman Gushchin
If two or more threads of an application faulting on the same folio, the
mmap_misscounter can be decreased multiple times. It breaks themmap_missheuristic and keeps the readahead enabled even under extreme levels of memory pressure.It happens often if file folios backing a multi-threaded application are getting evicted and re-faulted.
Fix it by skipping decreasing
mmap_missif the folio is locked.This change was evaluated on several hundred thousands hosts in Google's production over a couple of weeks. The number of containers being stuck in a vicious reclaim cycle for a long time was reduced several fold (~10-20x), as well as the overall fleet-wide cpu time spent in direct memory reclaim was meaningfully reduced. No regressions were observed.
Signed-off-by Roman Gushchin <[email protected]>Cc Matthew Wilcox (Oracle) <[email protected]>Cc Jan Kara <[email protected]>--- mm/filemap.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)mm/filemap.c +1
-3@@ -3324,9 +3324,17 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf,if (vmf->vma->vm_flags &VM_RAND_READ || !ra->ra_pages)return fpin; - mmap_miss =READ_ONCE (ra->mmap_miss); -if (mmap_miss) -WRITE_ONCE (ra->mmap_miss, --mmap_miss); +/*
mm/filemap.c static struct file *do_async_mmap_readahead(struct vm_fault *vmf, · line 3337
Quoted 10 lines
mm/filemap.c +10
-0+* If the folio is locked, we're likely racing against another fault. +* Don't touch the mmap_miss counter to avoid decreasing it multiple +* times for a single folio and break the balance with mmap_miss +* increase in do_sync_mmap_readahead(). +*/ +if (likely (!folio_test_locked (folio))) { + mmap_miss =READ_ONCE (ra->mmap_miss); +if (mmap_miss) +WRITE_ONCE (ra->mmap_miss, --mmap_miss); + }
I'm not an mm person.
The comment implies the change fixes the race, but it is not at all clear to me how.
Does it merely make it significantly less likely?
Quoted 51 lines · Mateusz Guzik
Quoted 37 lines · Roman Gushchin
If two or more threads of an application faulting on the same folio, the
mmap_misscounter can be decreased multiple times. It breaks themmap_missheuristic and keeps the readahead enabled even under extreme levels of memory pressure.It happens often if file folios backing a multi-threaded application are getting evicted and re-faulted.
Fix it by skipping decreasing
mmap_missif the folio is locked.This change was evaluated on several hundred thousands hosts in Google's production over a couple of weeks. The number of containers being stuck in a vicious reclaim cycle for a long time was reduced several fold (~10-20x), as well as the overall fleet-wide cpu time spent in direct memory reclaim was meaningfully reduced. No regressions were observed.
Signed-off-by Roman Gushchin <[email protected]>Cc Matthew Wilcox (Oracle) <[email protected]>Cc Jan Kara <[email protected]>--- mm/filemap.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)mm/filemap.c +1
-3@@ -3324,9 +3324,17 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf,if (vmf->vma->vm_flags &VM_RAND_READ || !ra->ra_pages)return fpin; - mmap_miss =READ_ONCE (ra->mmap_miss); -if (mmap_miss) -WRITE_ONCE (ra->mmap_miss, --mmap_miss); +/* mm/filemap.c static struct file *do_async_mmap_readahead(struct vm_fault *vmf, · line 3337
Quoted 10 lines
mm/filemap.c +10
-0+* If the folio is locked, we're likely racing against another fault. +* Don't touch the mmap_miss counter to avoid decreasing it multiple +* times for a single folio and break the balance with mmap_miss +* increase in do_sync_mmap_readahead(). +*/ +if (likely (!folio_test_locked (folio))) { + mmap_miss =READ_ONCE (ra->mmap_miss); +if (mmap_miss) +WRITE_ONCE (ra->mmap_miss, --mmap_miss); + }I'm not an mm person.
Quoted 4 lines
The comment implies the change fixes the race, but it is not at all clear to me how.
Does it merely make it significantly less likely?
It's not fixing any race, it's fixing the imbalance in the upward and downward pressure on the mmap_miss variable. This improves the readahead behavior under very special circumstances: a multi-threaded application under very heavy memory pressure. There should be no visible difference in behavior in other cases.
Thanks!