mm/memory: align alloc_swap_folio() logic with alloc_anon_folio()
Subsystems (2)
cover mm/memory: align alloc_swap_folio() logic with alloc_anon_folio()
This series cleans up the logic used when swapping in large folios to bring it in line with the established patterns in alloc_anon_folio().
Context:
When swapping in a large folio, the kernel performs a process similar to alloc_anon_folio(): it scans the PTE range to determine the highest suitable order for a large folio and then attempts allocation. However, two key discrepancies currently exist in alloc_swap_folio():
1. Improper Fallback: It currently falls back to an order-0 folio if pte_offset_map_lock() returns NULL.
2. Unnecessary Locking: It locks the page table during the initial scanning phase.
Improvements:
Based on an analysis of the alloc_anon_folio() implementation, this series applies the following corrections to the swap-in path:
1. Bail out on NULL PTE: If pte_offset_map_lock() returns NULL, it indicates the page table has changed under us. In this scenario, the operation is likely to fail later anyway; continuing the process is inefficient and unnecessary.
2. Lockless Initial Scan: The initial scan only reads PTEs to determine the appropriate folio order. Since no modifications are made at this stage, and the table is re-verified and locked before the actual write, the initial lock is redundant. Removing it reduces lock contention and system load.
Wei Yang (2):
mm: bail out do_swap_page() when no PTE table exist
mm: avoid unnecessary PTE table lock during initial swap folio scan
mm/memory.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
Signature · 1 line
2.34.1
PATCH 1/2 mm: bail out do_swap_page() when no PTE table exist
The alloc_swap_folio() function scans the PTE table to determine the potential size (order) of the folio content to be swapped in.
Currently, if the call to pte_offset_map_lock() returns NULL, it indicates that no PTE table exists for that range. Despite this, the code proceeds to allocate an order-0 folio and continues the swap-in process. This is unnecessary if the required table is absent.
This commit modifies the logic to immediately bail out of the swap-in process when the PTE table is missing (i.e., pte_offset_map_lock() returns NULL). This ensures we do not attempt to continue swapping when the page table structure is incomplete or changed, preventing unnecessary work.
--- mm/memory.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
mm/memory.c +3 -1
@@ -4566,7 +4566,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf) pte =pte_offset_map_lock (vmf->vma->vm_mm, vmf->pmd, vmf->address &PMD_MASK , &ptl);if (unlikely (!pte)) - goto fallback; +return ERR_PTR (-EAGAIN );/* * For do_swap_page, find the highest order where the aligned range is
@@ -4709,6 +4709,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) __swap_count(entry) == 1) {/* skip swapcache */ folio =alloc_swap_folio (vmf); +if (IS_ERR (folio)) + goto out;if (folio) {__folio_set_locked (folio);__folio_set_swapbacked (folio);
Signature · 1 line
2.34.1
David Hildenbrand (Red Hat)
maintainerRe: [PATCH 1/2] mm: bail out do_swap_page() when no PTE table exist
Quoted 35 lines · Wei Yang
The
alloc_swap_folio()function scans the PTE table to determine the potential size (order) of the folio content to be swapped in.Currently, if the call to
pte_offset_map_lock()returns NULL, it indicates that no PTE table exists for that range. Despite this, the code proceeds to allocate an order-0 folio and continues the swap-in process. This is unnecessary if the required table is absent.This commit modifies the logic to immediately bail out of the swap-in process when the PTE table is missing (i.e.,
pte_offset_map_lock()returns NULL). This ensures we do not attempt to continue swapping when the page table structure is incomplete or changed, preventing unnecessary work.Signed-off-by Wei Yang <[email protected]>Cc Chuanhua Han <[email protected]>Cc Barry Song <[email protected]>--- mm/memory.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/memory.c +1
-1@@ -4566,7 +4566,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf) pte =pte_offset_map_lock (vmf->vma->vm_mm, vmf->pmd, vmf->address &PMD_MASK , &ptl);if (unlikely (!pte)) - goto fallback; +return ERR_PTR (-EAGAIN );/* * For do_swap_page, find the highest order where the aligned range is
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4716
Quoted 8 lines
mm/memory.c +2
-0@@ -4709,6 +4709,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) __swap_count(entry) == 1) {/* skip swapcache */ folio =alloc_swap_folio (vmf); +if (IS_ERR (folio)) + goto out;if (folio) {__folio_set_locked (folio);__folio_set_swapbacked (folio);
How would we be able to even trigger this?
Trigger a swap fault with concurrent MADV_DONTNEED and concurrent page table reclaim.
Is that really something we should be worrying about?
Signature · 3 lines
Cheers David
Re: [PATCH 1/2] mm: bail out do_swap_page() when no PTE table exist
Quoted 37 lines · David Hildenbrand (Red Hat)
Quoted 36 lines · Wei Yang
The
alloc_swap_folio()function scans the PTE table to determine the potential size (order) of the folio content to be swapped in.Currently, if the call to
pte_offset_map_lock()returns NULL, it indicates that no PTE table exists for that range. Despite this, the code proceeds to allocate an order-0 folio and continues the swap-in process. This is unnecessary if the required table is absent.This commit modifies the logic to immediately bail out of the swap-in process when the PTE table is missing (i.e.,
pte_offset_map_lock()returns NULL). This ensures we do not attempt to continue swapping when the page table structure is incomplete or changed, preventing unnecessary work.Signed-off-by Wei Yang <[email protected]>Cc Chuanhua Han <[email protected]>Cc Barry Song <[email protected]>--- mm/memory.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/memory.c +1
-1@@ -4566,7 +4566,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf) pte =pte_offset_map_lock (vmf->vma->vm_mm, vmf->pmd, vmf->address &PMD_MASK , &ptl);if (unlikely (!pte)) - goto fallback; +return ERR_PTR (-EAGAIN );/* * For do_swap_page, find the highest order where the aligned range is @@ -4709,6 +4709,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) __swap_count(entry) == 1) {/* skip swapcache */
Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4716
Quoted 6 lines
mm/memory.c +2
-0folio =alloc_swap_folio (vmf); +if (IS_ERR (folio)) + goto out;if (folio) {__folio_set_locked (folio);__folio_set_swapbacked (folio);How would we be able to even trigger this?
To be honest, I haven't thought about it. Thanks for the question.
Quoted 4 lines
Trigger a swap fault with concurrent
MADV_DONTNEEDand concurrent page table reclaim.
Let me try to catch up with you.
swap fault is triggered because user is access this range. MADV_DONTNEED is also triggered by user and means this range is not necessary.
So, we don't expect user will do these two contrary behavior at the same time. This is your point, right?
Quoted 2 lines
Is that really something we should be worrying about?
Now I question myself why alloc_anon_folio() need to bail out like this.
page fault vs MADV_DONTNEED is not in concern. So page fault vs khugepaged is the case? But I see khugepaged would mmap_write_lock(), which prevent page fault in my mind, before collapsing pmd.
I am curious alloc_anon_folio() tries to handle which case here.
Quoted 4 lines
Signature · 3 lines
Cheers David
Signature · 2 lines
Wei Yang Help you, Help me
David Hildenbrand (Red Hat)
maintainerRe: [PATCH 1/2] mm: bail out do_swap_page() when no PTE table exist
Quoted 47 lines · Wei Yang
Quoted 37 lines · David Hildenbrand (Red Hat)
Quoted 36 lines · Wei Yang
The
alloc_swap_folio()function scans the PTE table to determine the potential size (order) of the folio content to be swapped in.Currently, if the call to
pte_offset_map_lock()returns NULL, it indicates that no PTE table exists for that range. Despite this, the code proceeds to allocate an order-0 folio and continues the swap-in process. This is unnecessary if the required table is absent.This commit modifies the logic to immediately bail out of the swap-in process when the PTE table is missing (i.e.,
pte_offset_map_lock()returns NULL). This ensures we do not attempt to continue swapping when the page table structure is incomplete or changed, preventing unnecessary work.Signed-off-by Wei Yang <[email protected]>Cc Chuanhua Han <[email protected]>Cc Barry Song <[email protected]>--- mm/memory.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)mm/memory.c +1
-1@@ -4566,7 +4566,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf) pte =pte_offset_map_lock (vmf->vma->vm_mm, vmf->pmd, vmf->address &PMD_MASK , &ptl);if (unlikely (!pte)) - goto fallback; +return ERR_PTR (-EAGAIN );/* * For do_swap_page, find the highest order where the aligned range is @@ -4709,6 +4709,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) __swap_count(entry) == 1) {/* skip swapcache */ Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4716
Quoted 6 lines
mm/memory.c +2
-0folio =alloc_swap_folio (vmf); +if (IS_ERR (folio)) + goto out;if (folio) {__folio_set_locked (folio);__folio_set_swapbacked (folio);How would we be able to even trigger this?
Quoted 1 line
To be honest, I haven't thought about it. Thanks for the question.
Always ask yourself that question when trying to optimize something :)
Optimizing out unnecessary work in something that doesn't happen all the frequently is not particularly helpful.
Quoted 9 lines
Quoted 5 lines
Trigger a swap fault with concurrent
MADV_DONTNEEDand concurrent page table reclaim.Let me try to catch up with you.
swap fault is triggered because user is access this range.
And there was a page table with a swap entry.
Quoted 1 line
MADV_DONTNEEDis also triggered by user and means this range is not necessary.
Right, another thread could be zapping that range.
Quoted 3 lines
So, we don't expect user will do these two contrary behavior at the same time. This is your point, right?
They could. And it's valid. It just likely doesn't make a lot of sense :)
Quoted 2 lines
Quoted 2 lines
Is that really something we should be worrying about?
But for the page table to vanish you'd actually need page table reclaim (as triggered by MADV_DONTNEED) to zap the whole page table.
Quoted 3 lines
Quoted 1 line
Now I question myself why
alloc_anon_folio()need to bail out like this.
Your patch adds more complication by making it valid for alloc_swap_folio() to return an error pointer. I don't like that when there is no evidence that we frequently trigger that.
Also, there is no real difference between not finding a page table (reclaimed) or if the pte changed (as handled by can_swapin_thp()).
In fact, the latter (PTE changed) is even much more likely than a reclaimed page table.
Signature · 3 lines
Cheers David
PATCH 2/2 mm: avoid unnecessary PTE table lock during initial swap folio scan
In reply to Wei YangThe alloc_swap_folio() function performs an initial scan of the PTE table solely to determine the potential size (order) of the folio content that needs to be swapped in.
Locking the PTE table during this initial read is unnecessary for two reasons:
* We are not writing to the PTE table at this stage.
* The code will re-check and lock the table again immediately before any actual modification is attempted.
This commit refactors the initial scan to map the PTE table without acquiring the lock. This reduces contention and overhead, improving performance of the swap-in path.
--- mm/memory.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
mm/memory.c +2 -4
@@ -4529,7 +4529,6 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)struct folio *folio;unsigned long addr;softleaf_t entry; -spinlock_t *ptl;pte_t *pte;gfp_t gfp;int order;
@@ -4563,8 +4562,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)if (!orders) goto fallback; - pte =pte_offset_map_lock (vmf->vma->vm_mm, vmf->pmd, - vmf->address &PMD_MASK , &ptl); + pte =pte_offset_map (vmf->pmd, vmf->address &PMD_MASK );if (unlikely (!pte))return ERR_PTR (-EAGAIN );
@@ -4580,7 +4578,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf) order =next_order (&orders, order); } -pte_unmap_unlock (pte, ptl); +pte_unmap (pte);/* Try allocating the highest of the remaining orders. */ gfp =vma_thp_gfp_mask (vma);
Signature · 1 line
2.34.1
David Hildenbrand (Red Hat)
maintainerRe: [PATCH 2/2] mm: avoid unnecessary PTE table lock during initial swap folio scan
Quoted 36 lines · Wei Yang
The
alloc_swap_folio()function performs an initial scan of the PTE table solely to determine the potential size (order) of the folio content that needs to be swapped in.Locking the PTE table during this initial read is unnecessary for two reasons:
* We are not writing to the PTE table at this stage.
* The code will re-check and lock the table again immediately before any actual modification is attempted.
This commit refactors the initial scan to map the PTE table without acquiring the lock. This reduces contention and overhead, improving performance of the swap-in path.
Signed-off-by Wei Yang <[email protected]>Cc Chuanhua Han <[email protected]>Cc Barry Song <[email protected]>--- mm/memory.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)mm/memory.c +0
-1@@ -4529,7 +4529,6 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)struct folio *folio;unsigned long addr;softleaf_t entry; -spinlock_t *ptl;pte_t *pte;gfp_t gfp;int order;
mm/memory.c static struct folio *alloc_swap_folio(struct vm_fault *vmf) · line 4566
Quoted 7 lines
mm/memory.c +1
-2@@ -4563,8 +4562,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)if (!orders) goto fallback; - pte =pte_offset_map_lock (vmf->vma->vm_mm, vmf->pmd, - vmf->address &PMD_MASK , &ptl); + pte =pte_offset_map (vmf->pmd, vmf->address &PMD_MASK );if (unlikely (!pte))
Can can_swapin_thp() deal with concurrent unmap and possible freeing of pages+swap?
We have some code that depends on swap entries stabilizing the swap device etc; the moment you allow for that concurrently to go away you open a can of worns.
Signature · 3 lines
Cheers David
Re: [PATCH 2/2] mm: avoid unnecessary PTE table lock during initial swap folio scan
Quoted 47 lines · David Hildenbrand (Red Hat)
Quoted 36 lines · Wei Yang
The
alloc_swap_folio()function performs an initial scan of the PTE table solely to determine the potential size (order) of the folio content that needs to be swapped in.Locking the PTE table during this initial read is unnecessary for two reasons:
* We are not writing to the PTE table at this stage.
* The code will re-check and lock the table again immediately before any actual modification is attempted.
This commit refactors the initial scan to map the PTE table without acquiring the lock. This reduces contention and overhead, improving performance of the swap-in path.
Signed-off-by Wei Yang <[email protected]>Cc Chuanhua Han <[email protected]>Cc Barry Song <[email protected]>--- mm/memory.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)mm/memory.c +0
-1@@ -4529,7 +4529,6 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)struct folio *folio;unsigned long addr;softleaf_t entry; -spinlock_t *ptl;pte_t *pte;gfp_t gfp;int order;mm/memory.c static struct folio *alloc_swap_folio(struct vm_fault *vmf) · line 4565
Quoted 6 lines
mm/memory.c +1
-2@@ -4563,8 +4562,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)if (!orders) goto fallback; - pte =pte_offset_map_lock (vmf->vma->vm_mm, vmf->pmd, - vmf->address &PMD_MASK , &ptl); + pte =pte_offset_map (vmf->pmd, vmf->address &PMD_MASK );if (unlikely (!pte))Can
can_swapin_thp()deal with concurrent unmap and possible freeing of pages+swap?
Quoted 4 lines
We have some code that depends on swap entries stabilizing the swap device etc; the moment you allow for that concurrently to go away you open a can of worns.
Sorry I don't follow you.
You mean some swap entry would be unmapped and cleared?
This is the first scan of swap entries, and there is another swap_entry_batch() before real work with ptl locked to make sure the range is still valid for swap in.
If someone change the pte at this moment, it will be detected after ptl locked by swap_entry_batch().
Maybe I misunderstand you. Would you mind giving me more hint?
Quoted 4 lines
Signature · 3 lines
Cheers David
Signature · 2 lines
Wei Yang Help you, Help me
David Hildenbrand (Red Hat)
maintainerRe: [PATCH 2/2] mm: avoid unnecessary PTE table lock during initial swap folio scan
Quoted 48 lines · Wei Yang
Quoted 47 lines · David Hildenbrand (Red Hat)
Quoted 36 lines · Wei Yang
The
alloc_swap_folio()function performs an initial scan of the PTE table solely to determine the potential size (order) of the folio content that needs to be swapped in.Locking the PTE table during this initial read is unnecessary for two reasons:
* We are not writing to the PTE table at this stage.
* The code will re-check and lock the table again immediately before any actual modification is attempted.
This commit refactors the initial scan to map the PTE table without acquiring the lock. This reduces contention and overhead, improving performance of the swap-in path.
Signed-off-by Wei Yang <[email protected]>Cc Chuanhua Han <[email protected]>Cc Barry Song <[email protected]>--- mm/memory.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)mm/memory.c +0
-1@@ -4529,7 +4529,6 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)struct folio *folio;unsigned long addr;softleaf_t entry; -spinlock_t *ptl;pte_t *pte;gfp_t gfp;int order;mm/memory.c static struct folio *alloc_swap_folio(struct vm_fault *vmf) · line 4565
Quoted 6 lines
mm/memory.c +1
-2@@ -4563,8 +4562,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)if (!orders) goto fallback; - pte =pte_offset_map_lock (vmf->vma->vm_mm, vmf->pmd, - vmf->address &PMD_MASK , &ptl); + pte =pte_offset_map (vmf->pmd, vmf->address &PMD_MASK );if (unlikely (!pte))Can
can_swapin_thp()deal with concurrent unmap and possible freeing of pages+swap?
Quoted 8 lines
Quoted 4 lines
We have some code that depends on swap entries stabilizing the swap device etc; the moment you allow for that concurrently to go away you open a can of worns.
Sorry I don't follow you.
You mean some swap entry would be unmapped and cleared?
We could concurrently be zapping the page table. That means, after we read a swap-PTE, we could be concurrently freeing the swap entry from a different thread.
So the moment you depend on something that goes from PTE to something in the swap subsystem you might be in trouble.
swap_pte_batch() does things like lookup_swap_cgroup_id(), and can_swapin_thp() does things like swap_zeromap_batch() and non_swapcache_batch().
I don't know what happens if we can have concurrent zap+freeing of swap entries there, and if we could trigger some undefined behavior.
Therefore, we have to a bit more careful here.
Because I assume this is the first time that we walk swap entries without the PTE lock held?
Signature · 3 lines
Cheers David