mm: folio conversion and refactoring in do_swap_page()
Subsystems (2)
v2 cover mm: folio conversion and refactoring in do_swap_page()
From: Hongfu Li <[email protected]>
Convert the device-private migration path in do_swap_page() to use folio APIs, then extract all non-swap entry handling into a new do_non_swap_page() helper to improve readability and reduce the size of the main swap fault function.
Hongfu Li (2):
mm: Use a folio in the softleaf_is_device_private path
mm: Extract non-swap page handling to do_non_swap_page()
mm/memory.c | 119 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 69 insertions(+), 50 deletions(-)
---
v2:
- Drop fault_folio, reuse the existing folio variable instead
- Refresh commit messages
- Update comment to use "folio" instead of "page" in the
device_private path
- Add Acked-by: David Hildenbrand
- Add patch 2/2 to extract do_non_swap_page() from do_swap_page()
- Link to v1: https://lore.kernel.org/all/[email protected]/
Signature · 1 line
2.54.0
PATCH v2 1/2 mm: Use a folio in the softleaf_is_device_private path
From: Hongfu Li <[email protected]>
Use the folio APIs in the device_private migration path of do_swap_page(), replacing four calls to compound_head() with one.
--- mm/memory.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
mm/memory.c +6 -5
@@ -4871,18 +4871,19 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto unlock;/* -* Get a page reference while we know the page can't be +* Get a folio reference while we know the folio can't be * freed. */ -if (trylock_page (vmf->page)) { + folio =page_folio (vmf->page); +if (folio_trylock (folio)) {struct dev_pagemap *pgmap; -get_page (vmf->page); +folio_get (folio);pte_unmap_unlock (vmf->pte, vmf->ptl); pgmap =page_pgmap (vmf->page); ret = pgmap->ops->migrate_to_ram (vmf); -unlock_page (vmf->page); -put_page (vmf->page); +folio_unlock (folio); +folio_put (folio); }else {pte_unmap (vmf->pte);softleaf_entry_wait_on_locked (entry, vmf->ptl);
Signature · 1 line
2.54.0
Re: [PATCH v2 1/2] mm: Use a folio in the softleaf_is_device_private path
Quoted 7 lines · Hongfu Li
From: Hongfu Li <[email protected]>
Use the folio APIs in the
device_privatemigration path ofdo_swap_page(), replacing four calls tocompound_head()with one.Signed-off-by Hongfu Li <[email protected]>
With changes below applied LGTM, so:
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4872
Quoted 11 lines
--- mm/memory.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)mm/memory.c +0
-0@@ -4871,18 +4871,19 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto unlock;
Up here we have:
vmf->page = softleaf_to_page(entry); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte)))
Since we're canonicalising to folio why not actually set folio using softleaf_to_folio()?
So
+ folio = softleaf_to_folio(entry);
- vmf->page = softleaf_to_page(entry);
+ vmf->page = &folio->page;
This would also make more sense in your 2/2. But more on that in a sec.
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4874
Quoted 3 lines
mm/memory.c +1
-1/* -* Get a page reference while we know the page can't be +* Get a folio reference while we know the folio can't be
NIT: be -> next line for 80 chars.
Quoted 20 lines
mm/memory.c +5
-5* freed. */ -if (trylock_page (vmf->page)) { + folio =page_folio (vmf->page); +if (folio_trylock (folio)) {struct dev_pagemap *pgmap; -get_page (vmf->page); +folio_get (folio);pte_unmap_unlock (vmf->pte, vmf->ptl); pgmap =page_pgmap (vmf->page); ret = pgmap->ops->migrate_to_ram (vmf); -unlock_page (vmf->page); -put_page (vmf->page); +folio_unlock (folio); +folio_put (folio); }else {pte_unmap (vmf->pte);softleaf_entry_wait_on_locked (entry, vmf->ptl); --
Quoted 2 lines
2.54.0
Thanks, Lorenzo
Re: [PATCH v2 1/2] mm: Use a folio in the softleaf_is_device_private path
Quoted 29 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4872
Quoted 10 lines
mm/memory.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)mm/memory.c +0
-0@@ -4871,18 +4871,19 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto unlock;Up here we have:
vmf->page = softleaf_to_page(entry); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte)))Since we're canonicalising to folio why not actually set folio using
softleaf_to_folio()?So
(quoted diff) +2
-1+ folio = softleaf_to_folio(entry); - vmf->page = softleaf_to_page(entry); + vmf->page = &folio->page;
Quoted 1 line
This would also make more sense in your 2/2. But more on that in a sec.
Thanks, I've thought through your suggestion carefully.
Could entry reference a tail page of a folio here?
If so, softleaf_to_folio() retrieves the folio containing this page, and vmf->page = &folio->page would point to the folio's head page, which may not be correct.
Best regards,
Hongfu
Lorenzo Stoakes (ARM)
reviewerRe: [PATCH v2 1/2] mm: Use a folio in the softleaf_is_device_private path
Quoted 29 lines · Hongfu Li
Quoted 29 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4872
Quoted 10 lines
mm/memory.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)mm/memory.c +0
-0@@ -4871,18 +4871,19 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto unlock;Up here we have:
vmf->page = softleaf_to_page(entry); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte)))Since we're canonicalising to folio why not actually set folio using
softleaf_to_folio()?So
(quoted diff) +2
-1+ folio = softleaf_to_folio(entry); - vmf->page = softleaf_to_page(entry); + vmf->page = &folio->page;
Quoted 8 lines
Quoted 1 line
This would also make more sense in your 2/2. But more on that in a sec.
Thanks, I've thought through your suggestion carefully.
Could entry reference a tail page of a folio here?
If so,softleaf_to_folio()retrieves the folio containing this page, andvmf->page=&folio->pagewould point to the folio's head page, which may not be correct.
Yeah that's true! My bad, disregard that bit then :)
Quoted 3 lines
Best regards,
Hongfu
--
Cheers, Lorenzo
PATCH v2 2/2 mm: Extract non-swap page handling to do_non_swap_page()
In reply to Hongfu LiFrom: Hongfu Li <[email protected]>
Factor non-swap entry logic out of bloated do_swap_page() to simplify the main swap fault flow and improve readability.
The new do_non_swap_page() handles migration, device private/exclusive, hwpoison, marker and bad PTE entries.
--- mm/memory.c | 120 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 69 insertions(+), 51 deletions(-)
mm/memory.c +69 -51
@@ -4815,6 +4815,74 @@ static void check_swap_exclusive(struct folio *folio, swp_entry_t entry, }while (--nr_pages); } +static vm_fault_t do_non_swap_page (struct vm_fault *vmf) +{ +struct vm_area_struct *vma = vmf->vma; +struct folio *folio; +softleaf_t entry; +vm_fault_t ret =0 ; + + entry =softleaf_from_pte (vmf->orig_pte); +if (softleaf_is_migration (entry)) { +migration_entry_wait (vma->vm_mm, vmf->pmd, + vmf->address); + }else if (softleaf_is_device_exclusive (entry)) { + vmf->page =softleaf_to_page (entry); + ret =remove_device_exclusive_entry (vmf); + }else if (softleaf_is_device_private (entry)) { + +if (vmf->flags &FAULT_FLAG_VMA_LOCK ) { +/* +* migrate_to_ram is not yet ready to operate +* under VMA lock. +*/ +vma_end_read (vma); +return VM_FAULT_RETRY ; + } + + vmf->page =softleaf_to_page (entry); + vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); +if (unlikely (!vmf->pte || + !pte_same (ptep_get (vmf->pte), + vmf->orig_pte))) + goto unlock; + +/* +* Get a folio reference while we know the folio can't be +* freed. +*/ + folio =page_folio (vmf->page); +if (folio_trylock (folio)) { +struct dev_pagemap *pgmap; + +folio_get (folio); +pte_unmap_unlock (vmf->pte, vmf->ptl); + pgmap =page_pgmap (vmf->page); + ret = pgmap->ops->migrate_to_ram (vmf); +folio_unlock (folio); +folio_put (folio); + }else { +pte_unmap (vmf->pte); +softleaf_entry_wait_on_locked (entry, vmf->ptl); + } + }else if (softleaf_is_hwpoison (entry)) { + ret =VM_FAULT_HWPOISON ; + }else if (softleaf_is_marker (entry)) { + ret =handle_pte_marker (vmf); + }else { +print_bad_pte (vma, vmf->address, vmf->orig_pte,NULL ); + ret =VM_FAULT_SIGBUS ; + } + +return ret; + +unlock: +if (vmf->pte) +pte_unmap_unlock (vmf->pte, vmf->ptl); +return ret; +} +/* * We enter with either the VMA lock or the mmap_lock held (see * FAULT_FLAG_VMA_LOCK), and pte mapped but not yet locked.
@@ -4845,57 +4913,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) entry =softleaf_from_pte (vmf->orig_pte);if (unlikely (!softleaf_is_swap (entry))) { -if (softleaf_is_migration (entry)) { -migration_entry_wait (vma->vm_mm, vmf->pmd, - vmf->address); - }else if (softleaf_is_device_exclusive (entry)) { - vmf->page =softleaf_to_page (entry); - ret =remove_device_exclusive_entry (vmf); - }else if (softleaf_is_device_private (entry)) { -if (vmf->flags &FAULT_FLAG_VMA_LOCK ) { -/* -* migrate_to_ram is not yet ready to operate -* under VMA lock. -*/ -vma_end_read (vma); - ret =VM_FAULT_RETRY ; - goto out; - } - - vmf->page =softleaf_to_page (entry); - vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); -if (unlikely (!vmf->pte || - !pte_same (ptep_get (vmf->pte), - vmf->orig_pte))) - goto unlock; - -/* -* Get a folio reference while we know the folio can't be -* freed. -*/ - folio =page_folio (vmf->page); -if (folio_trylock (folio)) { -struct dev_pagemap *pgmap; - -folio_get (folio); -pte_unmap_unlock (vmf->pte, vmf->ptl); - pgmap =page_pgmap (vmf->page); - ret = pgmap->ops->migrate_to_ram (vmf); -folio_unlock (folio); -folio_put (folio); - }else { -pte_unmap (vmf->pte); -softleaf_entry_wait_on_locked (entry, vmf->ptl); - } - }else if (softleaf_is_hwpoison (entry)) { - ret =VM_FAULT_HWPOISON ; - }else if (softleaf_is_marker (entry)) { - ret =handle_pte_marker (vmf); - }else { -print_bad_pte (vma, vmf->address, vmf->orig_pte,NULL ); - ret =VM_FAULT_SIGBUS ; - } + ret =do_non_swap_page (vmf); goto out; }
Signature · 1 line
2.54.0
David Hildenbrand (Arm)
maintainerRe: [PATCH v2 2/2] mm: Extract non-swap page handling to do_non_swap_page()
Quoted 18 lines · Hongfu Li
From: Hongfu Li <[email protected]>
Factor non-swap entry logic out of bloated
do_swap_page()to simplify the main swap fault flow and improve readability.The new
do_non_swap_page()handles migration, device private/exclusive, hwpoison, marker and bad PTE entries.Signed-off-by Hongfu Li <[email protected]>--- mm/memory.c | 120 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 69 insertions(+), 51 deletions(-)mm/memory.c +0
-0
mm/memory.c static void check_swap_exclusive(struct folio *folio, swp_entry_t entry, · line 4818
Quoted 4 lines
mm/memory.c +1
-0@@ -4815,6 +4815,74 @@ static void check_swap_exclusive(struct folio *folio, swp_entry_t entry, }while (--nr_pages); } +static vm_fault_t do_non_swap_page (struct vm_fault *vmf)
We're handling softleaf entries that are not swap entries.
do_nonswap_softleaf() or sth like that would be better than talking about "page" if there is not even a page.
But actually it should be the other way around:
do_softleaf_entry() [currently do_swap_page]
... non-swap entries
do_swap_entry()
Or sth like that. Then, maybe in the future, we can just move do_swap_entry() out of memory.c
So we're factoring the wrong thing out in this patch.
Signature · 3 lines
Cheers, David
Re: [PATCH v2 2/2] mm: Extract non-swap page handling to do_non_swap_page()
Quoted 20 lines
mm/memory.c static void check_swap_exclusive(struct folio *folio, swp_entry_t entry, · line 4818
Quoted 12 lines
mm/memory.c | 120 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 69 insertions(+), 51 deletions(-)mm/memory.c +1
-0@@ -4815,6 +4815,74 @@ static void check_swap_exclusive(struct folio *folio, swp_entry_t entry, }while (--nr_pages); } +static vm_fault_t do_non_swap_page (struct vm_fault *vmf)We're handling softleaf entries that are not swap entries.
do_nonswap_softleaf()or sth like that would be better than talking about "page" if there is not even a page.But actually it should be the other way around:
Quoted 8 lines
do_softleaf_entry()[currentlydo_swap_page]
... non-swap entries
do_swap_entry()Or sth like that. Then, maybe in the future, we can just move
do_swap_entry()out ofmemory.cSo we're factoring the wrong thing out in this patch.
Thank you very much for the detailed review, your analysis makes perfect sense to me.
I'm glad to send out v3 shortly for your further review, with do_non_swap_page renamed to do_nonswap_softleaf.
Best regards,
Hongfu
David Hildenbrand (Arm)
maintainerRe: [PATCH v2 2/2] mm: Extract non-swap page handling to do_non_swap_page()
Quoted 27 lines · Hongfu Li
Quoted 27 lines
mm/memory.c static void check_swap_exclusive(struct folio *folio, swp_entry_t entry, · line 4818
Quoted 12 lines
mm/memory.c | 120 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 69 insertions(+), 51 deletions(-)mm/memory.c +1
-0@@ -4815,6 +4815,74 @@ static void check_swap_exclusive(struct folio *folio, swp_entry_t entry, }while (--nr_pages); } +static vm_fault_t do_non_swap_page (struct vm_fault *vmf)We're handling softleaf entries that are not swap entries.
do_nonswap_softleaf()or sth like that would be better than talking about "page" if there is not even a page.But actually it should be the other way around:
do_softleaf_entry()[currentlydo_swap_page]
... non-swap entries
do_swap_entry()Or sth like that. Then, maybe in the future, we can just move
do_swap_entry()out ofmemory.c
Quoted 6 lines
Quoted 1 line
So we're factoring the wrong thing out in this patch.
Thank you very much for the detailed review, your analysis makes perfect sense to me.
I'm glad to send out v3 shortly for your further review, with
do_non_swap_pagerenamed todo_nonswap_softleaf.
Read again, that's not what I meant.
Signature · 3 lines
Cheers, David
Lorenzo Stoakes (ARM)
reviewerRe: [PATCH v2 2/2] mm: Extract non-swap page handling to do_non_swap_page()
Quoted 28 lines · David Hildenbrand (Arm)
Quoted 27 lines · Hongfu Li
Quoted 27 lines
mm/memory.c static void check_swap_exclusive(struct folio *folio, swp_entry_t entry, · line 4818
Quoted 12 lines
mm/memory.c | 120 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 69 insertions(+), 51 deletions(-)mm/memory.c +1
-0@@ -4815,6 +4815,74 @@ static void check_swap_exclusive(struct folio *folio, swp_entry_t entry, }while (--nr_pages); } +static vm_fault_t do_non_swap_page (struct vm_fault *vmf)We're handling softleaf entries that are not swap entries.
do_nonswap_softleaf()or sth like that would be better than talking about "page" if there is not even a page.But actually it should be the other way around:
do_softleaf_entry()[currentlydo_swap_page]
... non-swap entries
do_swap_entry()Or sth like that. Then, maybe in the future, we can just move
do_swap_entry()out ofmemory.c
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
So we're factoring the wrong thing out in this patch.
Thank you very much for the detailed review, your analysis makes perfect sense to me.
I'm glad to send out v3 shortly for your further review, with
do_non_swap_pagerenamed todo_nonswap_softleaf.Read again, that's not what I meant.
Yeah please do NOT introduce 'nonswap' we already had fun with the nonswap swap entry before :)
Quoting David:
do_softleaf_entry() [currently do_swap_page]
... non-swap entries do_swap_entry()
Also, swap entries _are_ softleaf entries.
The whole point of the softleaf change/renaming was to differentiate page table entries that are purely there for mapping stuff from the hardware's point of view (non-present because nothing is mapped there, present if mapping something) vs. something non-present because the entry is software-defined (hence SOFT-leaf).
Swap entries are software-defined and thus fall into category.
You can see all of the softleaf types trivially in include/linux/leafops.h in enum softleaf_type:
enum softleaf_type {
/* Fundamental types. */
SOFTLEAF_NONE,
SOFTLEAF_SWAP,
/* Migration types. */
SOFTLEAF_MIGRATION_READ,
SOFTLEAF_MIGRATION_READ_EXCLUSIVE,
SOFTLEAF_MIGRATION_WRITE,
/* Device types. */
SOFTLEAF_DEVICE_PRIVATE_READ,
SOFTLEAF_DEVICE_PRIVATE_WRITE,
SOFTLEAF_DEVICE_EXCLUSIVE,
/* H/W posion types. */
SOFTLEAF_HWPOISON,
/* Marker types. */
SOFTLEAF_MARKER,
};Quoted 5 lines
--
Cheers,David
Cheers, Lorenzo
P.S. I do remember suggesting that you _slow down_ :) we are into the quiet period now so let's maybe reduce the pace of series + respins, thanks.
Re: [PATCH v2 2/2] mm: Extract non-swap page handling to do_non_swap_page()
In reply to David Hildenbrand (Arm)Quoted 11 lines
Quoted 9 lines
Quoted 4 lines
Or sth like that. Then, maybe in the future, we can just move
do_swap_entry()out ofmemory.cSo we're factoring the wrong thing out in this patch.
Thank you very much for the detailed review, your analysis makes perfect sense to me.
I'm glad to send out v3 shortly for your further review, with
do_non_swap_pagerenamed todo_nonswap_softleaf.Read again, that's not what I meant.
Sorry, I completely misunderstood your intention.
I mistakenly sent v3 without checking your follow-up email, which was my oversight.
I'll drop the second patch entirely in v4, and carefully go through all comments before resending next time.
Thank you for your patience.
Best regards,
Hongfu
Re: [PATCH v2 2/2] mm: Extract non-swap page handling to do_non_swap_page()
In reply to David Hildenbrand (Arm)Quoted 7 lines · David Hildenbrand (Arm)
We're handling softleaf entries that are not swap entries.
do_nonswap_softleaf()or sth like that would be better than talking about "page" if there is not even a page.But actually it should be the other way around:
Quoted 8 lines
do_softleaf_entry()[currentlydo_swap_page]
... non-swap entries
do_swap_entry()Or sth like that. Then, maybe in the future, we can just move
do_swap_entry()out ofmemory.cSo we're factoring the wrong thing out in this patch.
Yes. From the code structure it also is really obvious that the block handling the softleaf_is_device_private() should be a separate helper instead of a giant code block sitting in the middle of an if/else chain where every other case has a well-defined helper.
Lorenzo Stoakes (ARM)
reviewerRe: [PATCH v2 2/2] mm: Extract non-swap page handling to do_non_swap_page()
In reply to Hongfu LiQuoted 9 lines · Hongfu Li
From: Hongfu Li <[email protected]>
Factor non-swap entry logic out of bloated
do_swap_page()to simplify the main swap fault flow and improve readability.The new
do_non_swap_page()handles migration, device private/exclusive, hwpoison, marker and bad PTE entries.Signed-off-by Hongfu Li <[email protected]>
We've already gone over this elsewhere in the thread, we don't want a do_non_swap_page() (HELL no).
But in general I want to finish what I started with softleaves, which still has some work to do with swap entries so maybe could you just drop this patch so I can follow up with that later?
That way you can do a respin of just the 1/2 as a v3.
Thanks, Lorenzo
Quoted 145 lines
--- mm/memory.c | 120 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 69 insertions(+), 51 deletions(-)mm/memory.c +69
-52@@ -4815,6 +4815,74 @@ static void check_swap_exclusive(struct folio *folio, swp_entry_t entry, }while (--nr_pages); } +static vm_fault_t do_non_swap_page (struct vm_fault *vmf) +{ +struct vm_area_struct *vma = vmf->vma; +struct folio *folio; +softleaf_t entry; +vm_fault_t ret =0 ; + + entry =softleaf_from_pte (vmf->orig_pte); +if (softleaf_is_migration (entry)) { +migration_entry_wait (vma->vm_mm, vmf->pmd, + vmf->address); + }else if (softleaf_is_device_exclusive (entry)) { + vmf->page =softleaf_to_page (entry); + ret =remove_device_exclusive_entry (vmf); + }else if (softleaf_is_device_private (entry)) { + +if (vmf->flags &FAULT_FLAG_VMA_LOCK ) { +/* +* migrate_to_ram is not yet ready to operate +* under VMA lock. +*/ +vma_end_read (vma); +return VM_FAULT_RETRY ; + } + + vmf->page =softleaf_to_page (entry); + vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); +if (unlikely (!vmf->pte || + !pte_same (ptep_get (vmf->pte), + vmf->orig_pte))) + goto unlock; + +/* +* Get a folio reference while we know the folio can't be +* freed. +*/ + folio =page_folio (vmf->page); +if (folio_trylock (folio)) { +struct dev_pagemap *pgmap; + +folio_get (folio); +pte_unmap_unlock (vmf->pte, vmf->ptl); + pgmap =page_pgmap (vmf->page); + ret = pgmap->ops->migrate_to_ram (vmf); +folio_unlock (folio); +folio_put (folio); + }else { +pte_unmap (vmf->pte); +softleaf_entry_wait_on_locked (entry, vmf->ptl); + } + }else if (softleaf_is_hwpoison (entry)) { + ret =VM_FAULT_HWPOISON ; + }else if (softleaf_is_marker (entry)) { + ret =handle_pte_marker (vmf); + }else { +print_bad_pte (vma, vmf->address, vmf->orig_pte,NULL ); + ret =VM_FAULT_SIGBUS ; + } + +return ret; + +unlock: +if (vmf->pte) +pte_unmap_unlock (vmf->pte, vmf->ptl); +return ret; +} +/* * We enter with either the VMA lock or the mmap_lock held (see * FAULT_FLAG_VMA_LOCK), and pte mapped but not yet locked. @@ -4845,57 +4913,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) entry =softleaf_from_pte (vmf->orig_pte);if (unlikely (!softleaf_is_swap (entry))) { -if (softleaf_is_migration (entry)) { -migration_entry_wait (vma->vm_mm, vmf->pmd, - vmf->address); - }else if (softleaf_is_device_exclusive (entry)) { - vmf->page =softleaf_to_page (entry); - ret =remove_device_exclusive_entry (vmf); - }else if (softleaf_is_device_private (entry)) { -if (vmf->flags &FAULT_FLAG_VMA_LOCK ) { -/* -* migrate_to_ram is not yet ready to operate -* under VMA lock. -*/ -vma_end_read (vma); - ret =VM_FAULT_RETRY ; - goto out; - } - - vmf->page =softleaf_to_page (entry); - vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); -if (unlikely (!vmf->pte || - !pte_same (ptep_get (vmf->pte), - vmf->orig_pte))) - goto unlock; - -/* -* Get a folio reference while we know the folio can't be -* freed. -*/ - folio =page_folio (vmf->page); -if (folio_trylock (folio)) { -struct dev_pagemap *pgmap; - -folio_get (folio); -pte_unmap_unlock (vmf->pte, vmf->ptl); - pgmap =page_pgmap (vmf->page); - ret = pgmap->ops->migrate_to_ram (vmf); -folio_unlock (folio); -folio_put (folio); - }else { -pte_unmap (vmf->pte); -softleaf_entry_wait_on_locked (entry, vmf->ptl); - } - }else if (softleaf_is_hwpoison (entry)) { - ret =VM_FAULT_HWPOISON ; - }else if (softleaf_is_marker (entry)) { - ret =handle_pte_marker (vmf); - }else { -print_bad_pte (vma, vmf->address, vmf->orig_pte,NULL ); - ret =VM_FAULT_SIGBUS ; - } + ret =do_non_swap_page (vmf); goto out; } --2.54.0