Skip to contentSkip to conversation

mm: vm_normal_page*() + CoW PFNMAP improvements

RFC PATCH In review David Hildenbrand 8 participants 65 messages Jun 17, 2025 – Jul 14, 2025

Subsystems (10)
lore
Share & tools
lore
#0 lore raw

cover mm: vm_normal_page*() + CoW PFNMAP improvements

RFC because it's based on mm-new where some things might still change around the devmap removal stuff.

While removing support for CoW PFNMAPs is a noble goal, I am not even sure if we can remove said support for e.g., /dev/mem that easily.

In the end, Cow PFNMAPs are pretty simple: everything is "special" except CoW'ed anon folios, that are "normal".

The only complication is: how to identify such pages without pte_special(). Because with pte_special(), it's easy.

Well, of course, one day all architectures might support pte_special() ... either because we added support for pte_special() or removed support for ... these architectures from Linux.

No need to wait for that day. Let's do some cleanups around vm_normal_page()/vm_normal_page_pmd() and handling of the huge zero folio, and remove the "horrible special case to handle copy-on-write behaviour" that does questionable things in remap_pfn_range() with a VMA, simply by

... looking for anonymous folios in CoW PFNMAPs to identify anonymous folios? I know, sounds crazy ;)

Of course, we add sanity checks that nobody dares installing PFNMAPs of anonymous folios in these configs, that could trick us in assuming that these are due to CoW.

We only have to do that without support for pte_special(); for architectures that support pte_special(), nothing changes.

In the same process, add vm_normal_page_pud(), which is easy after the cleanups, and improve our documentation. Well, and clarify that "vm_ops->find_special_page" XEN thingy.

Briefly tested on UML (making sure vm_normal_page() still works as expected without pte_special() support) and on x86-64 with a bunch of tests (including ndctl, and cow.c which tests the huge zero folio).

Cc Andrew Morton <[email protected]>
Cc Juergen Gross <[email protected]>
Cc Stefano Stabellini <[email protected]>
Cc Oleksandr Tyshchenko <[email protected]>
Cc Dan Williams <[email protected]>
Cc Alistair Popple <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Jan Kara <[email protected]>
Cc Alexander Viro <[email protected]>
Cc Christian Brauner <[email protected]>
Cc Zi Yan <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Lorenzo Stoakes <[email protected]>
Cc "Liam R. Howlett" <[email protected]>
Cc Nico Pache <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc Dev Jain <[email protected]>
Cc Barry Song <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Jann Horn <[email protected]>
Cc Pedro Falcato <[email protected]>

David Hildenbrand (14):
mm/memory: drop highest_memmap_pfn sanity check in vm_normal_page()
mm: drop highest_memmap_pfn
mm: compare pfns only if the entry is present when inserting
pfns/pages
mm/huge_memory: move more common code into insert_pmd() mm/huge_memory: move more common code into insert_pud() mm/huge_memory: support huge zero folio in vmf_insert_folio_pmd() fs/dax: use vmf_insert_folio_pmd() to insert the huge zero folio mm/huge_memory: mark PMD mappings of the huge zero folio special mm/memory: introduce is_huge_zero_pfn() and use it in
vm_normal_page_pmd()
mm/memory: factor out common code from vm_normal_page_*()
mm: remove "horrible special case to handle copy-on-write behaviour"
mm: drop addr parameter from vm_normal_*_pmd()
mm: introduce and use vm_normal_page_pud()
mm: rename vm_ops->find_special_page() to vm_ops->find_normal_page()

 drivers/xen/Kconfig              |   1 +
 drivers/xen/gntdev.c             |   5 +-
 fs/dax.c                         |  47 ++----
 fs/proc/task_mmu.c               |   6 +-
 include/linux/huge_mm.h          |  12 +-
 include/linux/mm.h               |  41 +++--
 mm/Kconfig                       |   2 +
 mm/huge_memory.c                 | 135 +++++++---------
 mm/internal.h                    |   2 -
 mm/memory.c                      | 261 ++++++++++++++++---------------
 mm/mm_init.c                     |   3 -
 mm/nommu.c                       |   1 -
 mm/pagewalk.c                    |  22 +--
 tools/testing/vma/vma_internal.h |  18 ++-
 14 files changed, 281 insertions(+), 275 deletions(-)

base-commit: 2f24ee10fe6d1959d674a4298d63b66f54508a68

Signature · 1 line
2.49.0
#1 lore raw

RFC PATCH 1/14 mm/memory: drop highest_memmap_pfn sanity check in vm_normal_page()

In 2009, we converted a VM_BUG_ON(!pfn_valid(pfn)) to the current highest_memmap_pfn sanity check in commit 22b31eec63e5 ("badpage:
vm_normal_page use print_bad_pte"), because highest_memmap_pfn was readily available.

Nowadays, this is the last remaining highest_memmap_pfn user, and this sanity check is not really triggering ... frequently.

Let's convert it to VM_WARN_ON_ONCE(!pfn_valid(pfn)), so we can simplify and get rid of highest_memmap_pfn. Checking for pfn_to_online_page() might be even better, but it would not handle ZONE_DEVICE properly.

Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...

What might be better in the future is having a runtime option like page-table-check to enable such checks dynamically on-demand. Something for the future.

---
 mm/memory.c | 15 +++------------
 1 file changed, 3 insertions(+), 12 deletions(-)
mm/memory.c +3 -12
index 0163d127cece9..188b84ebf479a 100644
@@ -590,7 +590,7 @@ struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr,
 
 	if (IS_ENABLED(CONFIG_ARCH_HAS_PTE_SPECIAL)) {
 		if (likely(!pte_special(pte)))
-			goto check_pfn;
+			goto out;
 		if (vma->vm_ops && vma->vm_ops->find_special_page)
 			return vma->vm_ops->find_special_page(vma, addr);
 		if (vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP))
@@ -608,9 +608,6 @@ struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr,
 		if (vma->vm_flags & VM_MIXEDMAP) {
 			if (!pfn_valid(pfn))
 				return NULL;
-			if (is_zero_pfn(pfn))
-				return NULL;
-			goto out;
 		} else {
 			unsigned long off;
 			off = (addr - vma->vm_start) >> PAGE_SHIFT;
@@ -624,17 +621,12 @@ struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr,
 	if (is_zero_pfn(pfn))
 		return NULL;
 
-check_pfn:
-	if (unlikely(pfn > highest_memmap_pfn)) {
-		print_bad_pte(vma, addr, pte, NULL);
-		return NULL;
-	}
-
 	/*
 	 * NOTE! We still have PageReserved() pages in the page tables.
 	 * eg. VDSO mappings can cause them to exist.
 	 */
 out:
+	VM_WARN_ON_ONCE(!pfn_valid(pfn));
 	VM_WARN_ON_ONCE(is_zero_pfn(pfn));
 	return pfn_to_page(pfn);
 }
@@ -676,14 +668,13 @@ struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr,
 
 	if (is_huge_zero_pmd(pmd))
 		return NULL;
-	if (unlikely(pfn > highest_memmap_pfn))
-		return NULL;
 
 	/*
 	 * NOTE! We still have PageReserved() pages in the page tables.
 	 * eg. VDSO mappings can cause them to exist.
 	 */
 out:
+	VM_WARN_ON_ONCE(!pfn_valid(pfn));
 	return pfn_to_page(pfn);
 }
 
Signature · 1 line
2.49.0
#2 lore raw

Re: [PATCH RFC 01/14] mm/memory: drop highest_memmap_pfn sanity check in vm_normal_page()

Quoted 13 lines · David Hildenbrand

In 2009, we converted a VM_BUG_ON(!pfn_valid(pfn)) to the current highest_memmap_pfn sanity check in commit 22b31eec63e5 ("badpage:
vm_normal_page use print_bad_pte"), because highest_memmap_pfn was readily available.

Nowadays, this is the last remaining highest_memmap_pfn user, and this sanity check is not really triggering ... frequently.

Let's convert it to VM_WARN_ON_ONCE(!pfn_valid(pfn)), so we can simplify and get rid of highest_memmap_pfn. Checking for pfn_to_online_page() might be even better, but it would not handle ZONE_DEVICE properly.

Quoted 8 lines

Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...

What might be better in the future is having a runtime option like page-table-check to enable such checks dynamically on-demand. Something for the future.

I'm confused, I'm missing something here.
Before this change we would return NULL if e.g: pfn > highest_memmap_pfn, but now we just print the warning and call pfn_to_page() anyway. AFAIK, pfn_to_page() doesn't return NULL?

Signature · 2 lines
Oscar Salvador
SUSE Labs
#3 lore raw

Re: [PATCH RFC 01/14] mm/memory: drop highest_memmap_pfn sanity check in vm_normal_page()

Quoted 14 lines · Oscar Salvador
Quoted 13 lines · David Hildenbrand

In 2009, we converted a VM_BUG_ON(!pfn_valid(pfn)) to the current highest_memmap_pfn sanity check in commit 22b31eec63e5 ("badpage:
vm_normal_page use print_bad_pte"), because highest_memmap_pfn was readily available.

Nowadays, this is the last remaining highest_memmap_pfn user, and this sanity check is not really triggering ... frequently.

Let's convert it to VM_WARN_ON_ONCE(!pfn_valid(pfn)), so we can simplify and get rid of highest_memmap_pfn. Checking for pfn_to_online_page() might be even better, but it would not handle ZONE_DEVICE properly.

Quoted 9 lines
Quoted 8 lines

Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...

What might be better in the future is having a runtime option like page-table-check to enable such checks dynamically on-demand. Something for the future.

Hi Oscar,

Quoted 4 lines

I'm confused, I'm missing something here.
Before this change we would return NULL if e.g: pfn > highest_memmap_pfn, but now we just print the warning and call pfn_to_page() anyway. AFAIK, pfn_to_page() doesn't return NULL?

You're missing that vm_normal_page_pmd() was created as a copy from vm_normal_page() [history of the sanity check above], but as we don't have (and shouldn't have ...) print_bad_pmd(), we made the code look like this would be something that can just happen.

"
Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...
"

So we made something that should never happen a runtime sanity check without ever reporting a problem ...

Signature · 3 lines
Cheers,

David / dhildenb
#4 lore raw

Re: [PATCH RFC 01/14] mm/memory: drop highest_memmap_pfn sanity check in vm_normal_page()

Quoted 1 line · David Hildenbrand

Hi Oscar,

Hi David,

Quoted 10 lines
Quoted 5 lines

I'm confused, I'm missing something here.
Before this change we would return NULL if e.g: pfn > highest_memmap_pfn, but now we just print the warning and call pfn_to_page() anyway. AFAIK, pfn_to_page() doesn't return NULL?

You're missing that vm_normal_page_pmd() was created as a copy from vm_normal_page() [history of the sanity check above], but as we don't have (and shouldn't have ...) print_bad_pmd(), we made the code look like this would be something that can just happen.

Ok I see.

Quoted 8 lines

"
Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...
"

So we made something that should never happen a runtime sanity check without ever reporting a problem ...

all clear now, thanks :-)

Signature · 2 lines
Oscar Salvador
SUSE Labs
#5 lore raw

Re: [PATCH RFC 01/14] mm/memory: drop highest_memmap_pfn sanity check in vm_normal_page()

In reply to David Hildenbrand
Quoted 38 lines · David Hildenbrand
Quoted 14 lines · Oscar Salvador
Quoted 13 lines · David Hildenbrand

In 2009, we converted a VM_BUG_ON(!pfn_valid(pfn)) to the current highest_memmap_pfn sanity check in commit 22b31eec63e5 ("badpage:
vm_normal_page use print_bad_pte"), because highest_memmap_pfn was readily available.

Nowadays, this is the last remaining highest_memmap_pfn user, and this sanity check is not really triggering ... frequently.

Let's convert it to VM_WARN_ON_ONCE(!pfn_valid(pfn)), so we can simplify and get rid of highest_memmap_pfn. Checking for pfn_to_online_page() might be even better, but it would not handle ZONE_DEVICE properly.

Quoted 9 lines
Quoted 8 lines

Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...

What might be better in the future is having a runtime option like page-table-check to enable such checks dynamically on-demand. Something for the future.

Hi Oscar,

Quoted 4 lines

I'm confused, I'm missing something here.
Before this change we would return NULL if e.g: pfn > highest_memmap_pfn, but now we just print the warning and call pfn_to_page() anyway. AFAIK, pfn_to_page() doesn't return NULL?

You're missing that vm_normal_page_pmd() was created as a copy from vm_normal_page() [history of the sanity check above], but as we don't have (and shouldn't have ...) print_bad_pmd(), we made the code look like this would be something that can just happen.

Quoted 7 lines

"
Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...
"

So we made something that should never happen a runtime sanity check without ever reporting a problem ...

IIUC, the reasoning is that because this case should never happen, we can change the behavior from returning NULL to a "warn and continue" model?

Thanks,
Lance

#6 lore raw

Re: [PATCH RFC 01/14] mm/memory: drop highest_memmap_pfn sanity check in vm_normal_page()

Quoted 44 lines · Lance Yang
Quoted 43 lines · David Hildenbrand
Quoted 14 lines · Oscar Salvador
Quoted 13 lines · David Hildenbrand

In 2009, we converted a VM_BUG_ON(!pfn_valid(pfn)) to the current highest_memmap_pfn sanity check in commit 22b31eec63e5 ("badpage:
vm_normal_page use print_bad_pte"), because highest_memmap_pfn was readily available.

Nowadays, this is the last remaining highest_memmap_pfn user, and this sanity check is not really triggering ... frequently.

Let's convert it to VM_WARN_ON_ONCE(!pfn_valid(pfn)), so we can simplify and get rid of highest_memmap_pfn. Checking for pfn_to_online_page() might be even better, but it would not handle ZONE_DEVICE properly.

Quoted 9 lines
Quoted 8 lines

Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...

What might be better in the future is having a runtime option like page-table-check to enable such checks dynamically on-demand. Something for the future.

Hi Oscar,

Quoted 4 lines

I'm confused, I'm missing something here.
Before this change we would return NULL if e.g: pfn > highest_memmap_pfn, but now we just print the warning and call pfn_to_page() anyway. AFAIK, pfn_to_page() doesn't return NULL?

You're missing that vm_normal_page_pmd() was created as a copy from vm_normal_page() [history of the sanity check above], but as we don't have (and shouldn't have ...) print_bad_pmd(), we made the code look like this would be something that can just happen.

"
Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...
"

Quoted 5 lines
Quoted 2 lines

So we made something that should never happen a runtime sanity check without ever reporting a problem ...

IIUC, the reasoning is that because this case should never happen, we can change the behavior from returning NULL to a "warn and continue" model?

Well, yes. Point is, that check should have never been copy-pasted that way, while dropping the actual warning :)

It's a sanity check for something that should never happen, turned into something that looks like it must be handled and would be valid to encounter.

Signature · 3 lines
Cheers,

David / dhildenb
reviewer
#7 lore raw

Re: [PATCH RFC 01/14] mm/memory: drop highest_memmap_pfn sanity check in vm_normal_page()

Quoted 45 lines · David Hildenbrand
Quoted 44 lines · Lance Yang
Quoted 43 lines · David Hildenbrand
Quoted 14 lines · Oscar Salvador
Quoted 13 lines · David Hildenbrand

In 2009, we converted a VM_BUG_ON(!pfn_valid(pfn)) to the current highest_memmap_pfn sanity check in commit 22b31eec63e5 ("badpage:
vm_normal_page use print_bad_pte"), because highest_memmap_pfn was readily available.

Nowadays, this is the last remaining highest_memmap_pfn user, and this sanity check is not really triggering ... frequently.

Let's convert it to VM_WARN_ON_ONCE(!pfn_valid(pfn)), so we can simplify and get rid of highest_memmap_pfn. Checking for pfn_to_online_page() might be even better, but it would not handle ZONE_DEVICE properly.

Quoted 9 lines
Quoted 8 lines

Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...

What might be better in the future is having a runtime option like page-table-check to enable such checks dynamically on-demand. Something for the future.

Hi Oscar,

Quoted 4 lines

I'm confused, I'm missing something here.
Before this change we would return NULL if e.g: pfn > highest_memmap_pfn, but now we just print the warning and call pfn_to_page() anyway. AFAIK, pfn_to_page() doesn't return NULL?

You're missing that vm_normal_page_pmd() was created as a copy from vm_normal_page() [history of the sanity check above], but as we don't have (and shouldn't have ...) print_bad_pmd(), we made the code look like this would be something that can just happen.

"
Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...
"

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

So we made something that should never happen a runtime sanity check without ever reporting a problem ...

IIUC, the reasoning is that because this case should never happen, we can change the behavior from returning NULL to a "warn and continue" model?

Well, yes. Point is, that check should have never been copy-pasted that way, while dropping the actual warning :)

Ah, I see your point now. Thanks for clarifying!

Quoted 2 lines

It's a sanity check for something that should never happen, turned into something that looks like it must be handled and would be valid to encounter.

Yeah. Makes sense to me ;)

#8 lore raw

Re: [PATCH RFC 01/14] mm/memory: drop highest_memmap_pfn sanity check in vm_normal_page()

Quoted 54 lines · Lance Yang
Quoted 51 lines · David Hildenbrand
Quoted 50 lines · Lance Yang
Quoted 47 lines · David Hildenbrand
Quoted 17 lines · Oscar Salvador
Quoted 16 lines · David Hildenbrand

In 2009, we converted a VM_BUG_ON(!pfn_valid(pfn)) to the current highest_memmap_pfn sanity check in commit 22b31eec63e5 ("badpage:
vm_normal_page use print_bad_pte"), because highest_memmap_pfn was readily available.

Nowadays, this is the last remaining highest_memmap_pfn user, and this sanity check is not really triggering ... frequently.

Let's convert it to VM_WARN_ON_ONCE(!pfn_valid(pfn)), so we can simplify and get rid of highest_memmap_pfn. Checking for pfn_to_online_page() might be even better, but it would not handle ZONE_DEVICE properly.

Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...

Quoted 7 lines
Quoted 6 lines

What might be better in the future is having a runtime option like page-table-check to enable such checks dynamically on-demand. Something
for the future.

Hi Oscar,

Quoted 5 lines

I'm confused, I'm missing something here.
Before this change we would return NULL if e.g: pfn > highest_memmap_pfn, but
now we just print the warning and call pfn_to_page() anyway. AFAIK, pfn_to_page() doesn't return NULL?

You're missing that vm_normal_page_pmd() was created as a copy from vm_normal_page() [history of the sanity check above], but as we don't have (and shouldn't have ...) print_bad_pmd(), we made the code look like this would be something that can just happen.

"
Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...
"

So we made something that should never happen a runtime sanity check without ever reporting a problem ...

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

IIUC, the reasoning is that because this case should never happen, we can change the behavior from returning NULL to a "warn and continue" model?

Well, yes. Point is, that check should have never been copy-pasted that way, while dropping the actual warning :)

Ah, I see your point now. Thanks for clarifying!

I'll add some of that to the patch description. Thanks!

Signature · 3 lines
Cheers,

David / dhildenb
#9 lore raw

Re: [PATCH RFC 01/14] mm/memory: drop highest_memmap_pfn sanity check in vm_normal_page()

Quoted 24 lines · David Hildenbrand
Quoted 23 lines · Lance Yang
Quoted 22 lines · David Hildenbrand
Quoted 21 lines · Lance Yang
Quoted 19 lines · David Hildenbrand
Quoted 17 lines · Oscar Salvador
Quoted 16 lines · David Hildenbrand

In 2009, we converted a VM_BUG_ON(!pfn_valid(pfn)) to the current highest_memmap_pfn sanity check in commit 22b31eec63e5 ("badpage:
vm_normal_page use print_bad_pte"), because highest_memmap_pfn was readily available.

Nowadays, this is the last remaining highest_memmap_pfn user, and this sanity check is not really triggering ... frequently.

Let's convert it to VM_WARN_ON_ONCE(!pfn_valid(pfn)), so we can simplify and get rid of highest_memmap_pfn. Checking for pfn_to_online_page() might be even better, but it would not handle ZONE_DEVICE properly.

Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

What might be better in the future is having a runtime option like page-table-check to enable such checks dynamically on-demand. Something
for the future.

The author of 22b31eec63e5 thinks this is not at all an improvement. Of course the condition is not triggering frequently, of course it should not happen: but it does happen, and it still seems worthwhile to catch it in production with a "Bad page map" than to let it run on to whatever kind of crash it hits instead.

Hugh

#10 lore raw

Re: [PATCH RFC 01/14] mm/memory: drop highest_memmap_pfn sanity check in vm_normal_page()

Quoted 30 lines · Hugh Dickins
Quoted 29 lines · David Hildenbrand
Quoted 28 lines · Lance Yang
Quoted 27 lines · David Hildenbrand
Quoted 26 lines · Lance Yang
Quoted 24 lines · David Hildenbrand
Quoted 22 lines · Oscar Salvador
Quoted 21 lines · David Hildenbrand

In 2009, we converted a VM_BUG_ON(!pfn_valid(pfn)) to the current highest_memmap_pfn sanity check in commit 22b31eec63e5 ("badpage:
vm_normal_page use print_bad_pte"), because highest_memmap_pfn was readily available.

Nowadays, this is the last remaining highest_memmap_pfn user, and this sanity check is not really triggering ... frequently.

Let's convert it to VM_WARN_ON_ONCE(!pfn_valid(pfn)), so we can simplify and get rid of highest_memmap_pfn. Checking for pfn_to_online_page() might be even better, but it would not handle ZONE_DEVICE properly.

Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...

What might be better in the future is having a runtime option like page-table-check to enable such checks dynamically on-demand. Something
for the future.

Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The author of 22b31eec63e5 thinks this is not at all an improvement. Of course the condition is not triggering frequently, of course it should not happen: but it does happen, and it still seems worthwhile to catch it in production with a "Bad page map" than to let it run on to whatever kind of crash it hits instead.

Well, obviously I don't agree and was waiting for having this discussion :)

We catch corruption in a handful of PTE bits, and that's about it. You neither detect corruption of flags nor of PFN bits that result in another valid PFN.

Corruption of the "special" bit might be fun.

When I was able to trigger this during development once, the whole machine went down shortly after -- mostly because of use-after-free of something that is now a page table, which is just bad for both users of such a page!

E.g., quit that process and we will happily clear the PTE, corrupting data of the other user. Fun.

I'm sure I could find a way to unify the code while printing some comparable message, but this check as it stands is just not worth it IMHO: trying to handle something gracefully that shouldn't happen, when really we cannot handle it gracefully.

Signature · 3 lines
Cheers,

David / dhildenb
#11 lore raw

Re: [PATCH RFC 01/14] mm/memory: drop highest_memmap_pfn sanity check in vm_normal_page()

Quoted 8 lines · David Hildenbrand
Quoted 7 lines · Hugh Dickins
Quoted 6 lines · David Hildenbrand
Quoted 5 lines · Lance Yang
Quoted 4 lines · David Hildenbrand
Quoted 3 lines · Lance Yang
Quoted 1 line · David Hildenbrand
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines · Oscar Salvador
Quoted 5 lines · David Hildenbrand
Quoted 4 lines

In 2009, we converted a VM_BUG_ON(!pfn_valid(pfn)) to the current highest_memmap_pfn sanity check in commit 22b31eec63e5 ("badpage:
vm_normal_page use print_bad_pte"), because highest_memmap_pfn was readily available.

highest_memmap_pfn was introduced by that commit for this purpose.

Quoted 26 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines

Nowadays, this is the last remaining highest_memmap_pfn user, and this
sanity check is not really triggering ... frequently.

Let's convert it to VM_WARN_ON_ONCE(!pfn_valid(pfn)), so we can simplify and get rid of highest_memmap_pfn. Checking for pfn_to_online_page() might be even better, but it would not handle ZONE_DEVICE properly.

Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...

What might be better in the future is having a runtime option like page-table-check to enable such checks dynamically on-demand. Something
for the future.

Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The author of 22b31eec63e5 thinks this is not at all an improvement. Of course the condition is not triggering frequently, of course it should not happen: but it does happen, and it still seems worthwhile to catch it in production with a "Bad page map" than to let it run on to whatever kind of crash it hits instead.

Quoted 4 lines

Well, obviously I don't agree and was waiting for having this discussion :)

We catch corruption in a handful of PTE bits, and that's about it. You neither detect corruption of flags nor of PFN bits that result in another valid PFN.

Of course it's limited in what it can catch (and won't even get called if the present bit was not set - a more complete patch might unify with those various "Bad swap" messages). Of course. But it's still useful for stopping pfn_to_page() veering off the end of the memmap[] (in some configs). And it's still useful for printing out a series of "Bad page map" messages when the page table is corrupted: from which a picture can sometimes be built up (isolated instance may just be a bitflip; series of them can sometimes show e.g. ascii text, occasionally helpful for debugging).

Quoted 7 lines

Corruption of the "special" bit might be fun.

When I was able to trigger this during development once, the whole machine went down shortly after -- mostly because of use-after-free of something that is now a page table, which is just bad for both users of such a page!

Quoted 7 lines

E.g., quit that process and we will happily clear the PTE, corrupting data of the other user. Fun.

I'm sure I could find a way to unify the code while printing some comparable message, but this check as it stands is just not worth it IMHO: trying to handle something gracefully that shouldn't happen, when really we cannot handle it gracefully.

So, you have experience of a time when it didn't help you. Okay. And we have had experience of other times when it has helped, if only a little. Like with other "Bad page"s: sometimes helpful, often not; but tending to build up a big picture from repeated occurrences.

We continue to disagree. I can't argue more than append the 2.6.29 commit message, which seems to me as valid now as it was then.

From 22b31eec63e5f2e219a3ee15f456897272bc73e8 Mon Sep 17 00:00:00 2001
From: Hugh Dickins <[email protected]>
Date: Tue, 6 Jan 2009 14:40:09 -0800
Subject: [PATCH] badpage: vm_normal_page use print_bad_pte

print_bad_pte() is so far being called only when zap_pte_range() finds negative page_mapcount, or there's a fault on a pte_file where it does not belong. That's weak coverage when we suspect pagetable corruption.

Originally, it was called when vm_normal_page() found an invalid pfn: but pfn_valid is expensive on some architectures and configurations, so 2.6.24 put that under CONFIG_DEBUG_VM (which doesn't help in the field), then 2.6.26 replaced it by a VM_BUG_ON (likewise).

Reinstate the print_bad_pte() in vm_normal_page(), but use a cheaper test than pfn_valid(): memmap_init_zone() (used in bootup and hotplug) keep a __read_mostly note of the highest_memmap_pfn, vm_normal_page() then check pfn against that. We could call this pfn_plausible() or pfn_sane(), but I doubt we'll need it elsewhere: of course it's not reliable, but gives much stronger pagetable validation on many boxes.

Also use print_bad_pte() when the pte_special bit is found outside a VM_PFNMAP or VM_MIXEDMAP area, instead of VM_BUG_ON.

Cc Nick Piggin <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Rik van Riel <[email protected]>
#12 lore raw

Re: [PATCH RFC 01/14] mm/memory: drop highest_memmap_pfn sanity check in vm_normal_page()

Quoted 9 lines · Hugh Dickins
Quoted 8 lines · David Hildenbrand
Quoted 7 lines · Hugh Dickins
Quoted 6 lines · David Hildenbrand
Quoted 5 lines · Lance Yang
Quoted 4 lines · David Hildenbrand
Quoted 3 lines · Lance Yang
Quoted 1 line · David Hildenbrand
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines · Oscar Salvador
Quoted 5 lines · David Hildenbrand
Quoted 4 lines

In 2009, we converted a VM_BUG_ON(!pfn_valid(pfn)) to the current highest_memmap_pfn sanity check in commit 22b31eec63e5 ("badpage:
vm_normal_page use print_bad_pte"), because highest_memmap_pfn was readily available.

highest_memmap_pfn was introduced by that commit for this purpose.

Oh, somehow I thought it was around before that.

[...]

Quoted 7 lines
Quoted 2 lines

We catch corruption in a handful of PTE bits, and that's about it. You neither detect corruption of flags nor of PFN bits that result in another valid PFN.

Of course it's limited in what it can catch (and won't even get called if the present bit was not set - a more complete patch might unify with those various "Bad swap" messages). Of course. But it's still useful for stopping pfn_to_page() veering off the end of the memmap[] (in some configs).

Right, probably in the configs we both don't care that much about nowadays :)

Quoted 4 lines

And it's still useful for printing out a series of "Bad page map" messages when the page table is corrupted: from which a picture can sometimes be built up (isolated instance may just be a bitflip; series of them can sometimes show e.g. ascii text, occasionally helpful for debugging).

It's kind of a weird thing, because we do something very different opposed to other areas where we detect that something serious is going wrong (e.g., WARN).

But another thread just sparked whether we should WARN here, so I'll leave that discussion to the other thread.

Quoted 16 lines
Quoted 8 lines

Corruption of the "special" bit might be fun.

When I was able to trigger this during development once, the whole machine went down shortly after -- mostly because of use-after-free of something that is now a page table, which is just bad for both users of such a page!

Quoted 7 lines

E.g., quit that process and we will happily clear the PTE, corrupting data of the other user. Fun.

I'm sure I could find a way to unify the code while printing some comparable message, but this check as it stands is just not worth it IMHO: trying to handle something gracefully that shouldn't happen, when really we cannot handle it gracefully.

Quoted 4 lines

So, you have experience of a time when it didn't help you. Okay. And we have had experience of other times when it has helped, if only a little. Like with other "Bad page"s: sometimes helpful, often not; but tending to build up a big picture from repeated occurrences.

Okay. I was rather curious how often we would actually hit this one here: from my recollection, the mapcount underflows are much more frequent than the ones from vm_normal_page().

Quoted 3 lines

We continue to disagree. I can't argue more than append the 2.6.29 commit message, which seems to me as valid now as it was then.

Not that I agree that performing these sanity checks in each and every config is something reasonable, but apparently you think that they are still useful, 16 years after they were introduced.

So, let me try finding a way to unify the code while keeping that error handling for now in place.

Signature · 3 lines
Cheers,

David / dhildenb
#13 lore raw

Re: [PATCH RFC 01/14] mm/memory: drop highest_memmap_pfn sanity check in vm_normal_page()

Quoted 9 lines · David Hildenbrand
Quoted 6 lines · Hugh Dickins

Of course it's limited in what it can catch (and won't even get called if the present bit was not set - a more complete patch might unify with those various "Bad swap" messages). Of course. But it's still useful for stopping pfn_to_page() veering off the end of the memmap[] (in some configs).

Right, probably in the configs we both don't care that much about nowadays :)

I thought it was the other way round: it's useful for stopping pfn_to_page() veering off the end of the memmap[] if it's a memory model where pfn_to_page() is a simple linear conversion.

As with SPARSEMEM_VMEMMAP, which I thought was the favourite nowadays.

If you don't care about that one much (does hotplug prevent it?), then you do care about the complex pfn_to_page()s, and we should have worried more when "page++"s got unnecessarily converted to folio_page(folio, i) a year or two back (I'm thinking of in mm/rmap.c, maybe elsewhere).

Hugh

#14 lore raw

Re: [PATCH RFC 01/14] mm/memory: drop highest_memmap_pfn sanity check in vm_normal_page()

Quoted 9 lines · Hugh Dickins
Quoted 8 lines · David Hildenbrand
Quoted 6 lines · Hugh Dickins

Of course it's limited in what it can catch (and won't even get called if the present bit was not set - a more complete patch might unify with those various "Bad swap" messages). Of course. But it's still useful for stopping pfn_to_page() veering off the end of the memmap[] (in some configs).

Quoted 7 lines
Quoted 1 line

Right, probably in the configs we both don't care that much about nowadays :)

I thought it was the other way round: it's useful for stopping pfn_to_page() veering off the end of the memmap[] if it's a memory model where pfn_to_page() is a simple linear conversion.

As with SPARSEMEM_VMEMMAP, which I thought was the favourite nowadays.

Yes, you're right, I had the !SPARSEMEM model in mind, but obviously, the same applies for the SPARSEMEM_VMEMMAP case as well.

Only the SPARSEMEM && !SPARSEMEM_VMEMMAP model is weird. IIRC, it will dereference NULL when given a non-existant PFN. (__nr_to_section returning NULL and pfn_to_section_nr() not beeing happy about that)

Signature · 3 lines
Cheers,

David / dhildenb
Reviewed-by on v1 1/14
#15 lore raw

Re: [PATCH RFC 01/14] mm/memory: drop highest_memmap_pfn sanity check in vm_normal_page()

In reply to David Hildenbrand
Quoted 13 lines · David Hildenbrand

In 2009, we converted a VM_BUG_ON(!pfn_valid(pfn)) to the current highest_memmap_pfn sanity check in commit 22b31eec63e5 ("badpage:
vm_normal_page use print_bad_pte"), because highest_memmap_pfn was readily available.

Nowadays, this is the last remaining highest_memmap_pfn user, and this sanity check is not really triggering ... frequently.

Let's convert it to VM_WARN_ON_ONCE(!pfn_valid(pfn)), so we can simplify and get rid of highest_memmap_pfn. Checking for pfn_to_online_page() might be even better, but it would not handle ZONE_DEVICE properly.

Quoted 8 lines

Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...

What might be better in the future is having a runtime option like page-table-check to enable such checks dynamically on-demand. Something for the future.

Signature · 2 lines
Oscar Salvador
SUSE Labs
Reviewed-by on v1 1/14
#16 lore raw

Re: [PATCH RFC 01/14] mm/memory: drop highest_memmap_pfn sanity check in vm_normal_page()

In reply to David Hildenbrand
Quoted 14 lines · David Hildenbrand

In 2009, we converted a VM_BUG_ON(!pfn_valid(pfn)) to the current highest_memmap_pfn sanity check in commit 22b31eec63e5 ("badpage:
vm_normal_page use print_bad_pte"), because highest_memmap_pfn was readily available.

Nowadays, this is the last remaining highest_memmap_pfn user, and this sanity check is not really triggering ... frequently.

Let's convert it to VM_WARN_ON_ONCE(!pfn_valid(pfn)), so we can simplify and get rid of highest_memmap_pfn. Checking for pfn_to_online_page() might be even better, but it would not handle ZONE_DEVICE properly.

Quoted 8 lines

Do the same in vm_normal_page_pmd(), where we don't even report a problem at all ...

What might be better in the future is having a runtime option like page-table-check to enable such checks dynamically on-demand. Something for the future.

LGTM. Feel free to add:

Thanks,
Lance

#17 lore raw

RFC PATCH 2/14 mm: drop highest_memmap_pfn

In reply to David Hildenbrand

Now unused, so let's drop it.

---
 mm/internal.h | 2 --
 mm/memory.c   | 2 --
 mm/mm_init.c  | 3 ---
 mm/nommu.c    | 1 -
 4 files changed, 8 deletions(-)
mm/internal.h +0 -2
index f519eb7217c26..703871905fd6d 100644
@@ -503,8 +503,6 @@ static inline bool folio_needs_release(struct folio *folio)
 		(mapping && mapping_release_always(mapping));
 }
 
-extern unsigned long highest_memmap_pfn;
-
 /*
  * Maximum number of reclaim retries without progress before the OOM
  * killer is consider the only way forward.
mm/memory.c +0 -2
index 188b84ebf479a..a1b5575db52ac 100644
@@ -146,8 +146,6 @@ __setup("norandmaps", disable_randmaps);
 unsigned long zero_pfn __read_mostly;
 EXPORT_SYMBOL(zero_pfn);
 
-unsigned long highest_memmap_pfn __read_mostly;
-
 /*
  * CONFIG_MMU architectures set up ZERO_PAGE in their paging_init()
  */
mm/mm_init.c +0 -3
index 5c21b3af216b2..1dac66c209984 100644
@@ -881,9 +881,6 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone
 	unsigned long pfn, end_pfn = start_pfn + size;
 	struct page *page;
 
-	if (highest_memmap_pfn < end_pfn - 1)
-		highest_memmap_pfn = end_pfn - 1;
-
 #ifdef CONFIG_ZONE_DEVICE
 	/*
 	 * Honor reservation requested by the driver for this ZONE_DEVICE
mm/nommu.c +0 -1
index 38c22ea0a95c6..cd9ddbfe1af80 100644
@@ -42,7 +42,6 @@
 #include <asm/mmu_context.h>
 #include "internal.h"
 
-unsigned long highest_memmap_pfn;
 int heap_stack_gap = 0;
 
 atomic_long_t mmap_pages_allocated;
Signature · 1 line
2.49.0
#20 lore raw

RFC PATCH 3/14 mm: compare pfns only if the entry is present when inserting pfns/pages

In reply to David Hildenbrand

Doing a pte_pfn() etc. of something that is not a present page table entry is wrong. Let's check in all relevant cases where we want to upgrade write permissions when inserting pfns/pages whether the entry is actually present.

It's not expected to have caused real harm in practice, so this is more a cleanup than a fix for something that would likely trigger in some weird circumstances.

At some point, we should likely unify the two pte handling paths, similar to how we did it for pmds/puds.

---
 mm/huge_memory.c | 4 ++--
 mm/memory.c      | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)
mm/huge_memory.c +2 -2
index 8e0e3cfd9f223..e52360df87d15 100644
@@ -1392,7 +1392,7 @@ static int insert_pmd(struct vm_area_struct *vma, unsigned long addr,
 		const unsigned long pfn = fop.is_folio ? folio_pfn(fop.folio) :
 					  fop.pfn;
 
-		if (write) {
+		if (write && pmd_present(*pmd)) {
 			if (pmd_pfn(*pmd) != pfn) {
 				WARN_ON_ONCE(!is_huge_zero_pmd(*pmd));
 				return -EEXIST;
@@ -1541,7 +1541,7 @@ static void insert_pud(struct vm_area_struct *vma, unsigned long addr,
 		const unsigned long pfn = fop.is_folio ? folio_pfn(fop.folio) :
 					  fop.pfn;
 
-		if (write) {
+		if (write && pud_present(*pud)) {
 			if (WARN_ON_ONCE(pud_pfn(*pud) != pfn))
 				return;
 			entry = pud_mkyoung(*pud);
mm/memory.c +2 -2
index a1b5575db52ac..9a1acd057ce59 100644
@@ -2137,7 +2137,7 @@ static int insert_page_into_pte_locked(struct vm_area_struct *vma, pte_t *pte,
 	pte_t pteval = ptep_get(pte);
 
 	if (!pte_none(pteval)) {
-		if (!mkwrite)
+		if (!mkwrite || !pte_present(pteval))
 			return -EBUSY;
 
 		/* see insert_pfn(). */
@@ -2434,7 +2434,7 @@ static vm_fault_t insert_pfn(struct vm_area_struct *vma, unsigned long addr,
 		return VM_FAULT_OOM;
 	entry = ptep_get(pte);
 	if (!pte_none(entry)) {
-		if (mkwrite) {
+		if (mkwrite && pte_present(entry)) {
 			/*
 			 * For read faults on private mappings the PFN passed
 			 * in may not match the PFN we have mapped if the
Signature · 1 line
2.49.0
Reviewed-by on v1 3/14
#21 lore raw

Re: [PATCH RFC 03/14] mm: compare pfns only if the entry is present when inserting pfns/pages

Quoted 4 lines · David Hildenbrand

Doing a pte_pfn() etc. of something that is not a present page table entry is wrong. Let's check in all relevant cases where we want to upgrade write permissions when inserting pfns/pages whether the entry is actually present.

Maybe I would add that's because the pte can have other info like marker, swp_entry etc.

Quoted 8 lines

It's not expected to have caused real harm in practice, so this is more a cleanup than a fix for something that would likely trigger in some weird circumstances.

At some point, we should likely unify the two pte handling paths, similar to how we did it for pmds/puds.

Should we scream if someone passes us a non-present entry?

Quoted 32 lines
---
 mm/huge_memory.c | 4 ++--
 mm/memory.c      | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)
mm/huge_memory.c +2 -2
index 8e0e3cfd9f223..e52360df87d15 100644
@@ -1392,7 +1392,7 @@ static int insert_pmd(struct vm_area_struct *vma, unsigned long addr,
 		const unsigned long pfn = fop.is_folio ? folio_pfn(fop.folio) :
 					  fop.pfn;
 
-		if (write) {
+		if (write && pmd_present(*pmd)) {
 			if (pmd_pfn(*pmd) != pfn) {
 				WARN_ON_ONCE(!is_huge_zero_pmd(*pmd));
 				return -EEXIST;
@@ -1541,7 +1541,7 @@ static void insert_pud(struct vm_area_struct *vma, unsigned long addr,
 		const unsigned long pfn = fop.is_folio ? folio_pfn(fop.folio) :
 					  fop.pfn;
 
-		if (write) {
+		if (write && pud_present(*pud)) {
 			if (WARN_ON_ONCE(pud_pfn(*pud) != pfn))
 				return;
 			entry = pud_mkyoung(*pud);
mm/memory.c +0 -0
index a1b5575db52ac..9a1acd057ce59 100644

mm/memory.c static int insert_page_into_pte_locked(struct vm_area_struct *vma, pte_t *pte, · line 2141

Quoted 6 lines
mm/memory.c +1 -1
@@ -2137,7 +2137,7 @@ static int insert_page_into_pte_locked(struct vm_area_struct *vma, pte_t *pte,
 	pte_t pteval = ptep_get(pte);
 
 	if (!pte_none(pteval)) {
-		if (!mkwrite)
+		if (!mkwrite || !pte_present(pteval))
 			return -EBUSY;

Why EBUSY? because it might transitory?

Signature · 2 lines
Oscar Salvador
SUSE Labs
#22 lore raw

Re: [PATCH RFC 03/14] mm: compare pfns only if the entry is present when inserting pfns/pages

Quoted 13 lines · Oscar Salvador
Quoted 4 lines · David Hildenbrand

Doing a pte_pfn() etc. of something that is not a present page table entry is wrong. Let's check in all relevant cases where we want to upgrade write permissions when inserting pfns/pages whether the entry is actually present.

Maybe I would add that's because the pte can have other info like marker, swp_entry etc.

Quoted 4 lines

It's not expected to have caused real harm in practice, so this is more a cleanup than a fix for something that would likely trigger in some weird circumstances.

Quoted 9 lines
Quoted 4 lines

At some point, we should likely unify the two pte handling paths, similar to how we did it for pmds/puds.

Should we scream if someone passes us a non-present entry?

Probably? Good point, let me think about that.

Quoted 35 lines
Quoted 35 lines
---
  mm/huge_memory.c | 4 ++--
  mm/memory.c      | 4 ++--
  2 files changed, 4 insertions(+), 4 deletions(-)
mm/huge_memory.c +2 -2
index 8e0e3cfd9f223..e52360df87d15 100644
@@ -1392,7 +1392,7 @@ static int insert_pmd(struct vm_area_struct *vma, unsigned long addr,
  		const unsigned long pfn = fop.is_folio ? folio_pfn(fop.folio) :
  					  fop.pfn;
 
-		if (write) {
+		if (write && pmd_present(*pmd)) {
  			if (pmd_pfn(*pmd) != pfn) {
  				WARN_ON_ONCE(!is_huge_zero_pmd(*pmd));
  				return -EEXIST;
@@ -1541,7 +1541,7 @@ static void insert_pud(struct vm_area_struct *vma, unsigned long addr,
  		const unsigned long pfn = fop.is_folio ? folio_pfn(fop.folio) :
  					  fop.pfn;
 
-		if (write) {
+		if (write && pud_present(*pud)) {
  			if (WARN_ON_ONCE(pud_pfn(*pud) != pfn))
  				return;
  			entry = pud_mkyoung(*pud);
mm/memory.c +0 -0
index a1b5575db52ac..9a1acd057ce59 100644
@@ -2137,7 +2137,7 @@ static int insert_page_into_pte_locked(struct vm_area_struct *vma, pte_t *pte,
  	pte_t pteval = ptep_get(pte);
 
Quoted 6 lines

mm/memory.c static int insert_page_into_pte_locked(struct vm_area_struct *vma, pte_t *pte, · line 2141

Quoted 4 lines
mm/memory.c +1 -1
  	if (!pte_none(pteval)) {
-		if (!mkwrite)
+		if (!mkwrite || !pte_present(pteval))
  			return -EBUSY;

Why EBUSY? because it might transitory?

I was confused myself about error handling, and why it differs for all cases ... adding to me todo list to investigate that (clean it up ...) :)

Signature · 3 lines
Cheers,

David / dhildenb
reviewer Reviewed-by on v1 3/14
#23 lore raw

Re: [PATCH RFC 03/14] mm: compare pfns only if the entry is present when inserting pfns/pages

In reply to David Hildenbrand
Quoted 8 lines · David Hildenbrand

Doing a pte_pfn() etc. of something that is not a present page table entry is wrong. Let's check in all relevant cases where we want to upgrade write permissions when inserting pfns/pages whether the entry is actually present.

It's not expected to have caused real harm in practice, so this is more a cleanup than a fix for something that would likely trigger in some weird circumstances.

Couldn't we e.g have a swap entry's "pfn" accidentally match the one we're inserting? Isn't that a correctness problem?

Quoted 5 lines

At some point, we should likely unify the two pte handling paths, similar to how we did it for pmds/puds.

Nice little cleanup, thanks.

Signature · 1 line
Pedro
#24 lore raw

Re: [PATCH RFC 03/14] mm: compare pfns only if the entry is present when inserting pfns/pages

Quoted 6 lines · Pedro Falcato
Quoted 5 lines · David Hildenbrand

Doing a pte_pfn() etc. of something that is not a present page table entry is wrong. Let's check in all relevant cases where we want to upgrade write permissions when inserting pfns/pages whether the entry is actually present.

Quoted 6 lines
Quoted 3 lines

It's not expected to have caused real harm in practice, so this is more a cleanup than a fix for something that would likely trigger in some weird circumstances.

Couldn't we e.g have a swap entry's "pfn" accidentally match the one we're inserting? Isn't that a correctness problem?

In theory yes, in practice I think this will not happen.

... especially because the WARN_ON_ONCE() would already trigger in many other cases before we would find one situation where it doesn't.

That's why I decided against Fixes: and declaring this more a cleanup, because the starts really would have to align ... :)

Thanks!

Signature · 3 lines
Cheers,

David / dhildenb
#25 lore raw

RFC PATCH 4/14 mm/huge_memory: move more common code into insert_pmd()

In reply to David Hildenbrand

Let's clean it all further up.

---
 mm/huge_memory.c | 72 ++++++++++++++++--------------------------------
 1 file changed, 24 insertions(+), 48 deletions(-)
mm/huge_memory.c +24 -48
index e52360df87d15..a85e0cd455109 100644
@@ -1379,15 +1379,25 @@ struct folio_or_pfn {
 	bool is_folio;
 };
 
-static int insert_pmd(struct vm_area_struct *vma, unsigned long addr,
+static vm_fault_t insert_pmd(struct vm_area_struct *vma, unsigned long addr,
 		pmd_t *pmd, struct folio_or_pfn fop, pgprot_t prot,
-		bool write, pgtable_t pgtable)
+		bool write)
 {
 	struct mm_struct *mm = vma->vm_mm;
+	pgtable_t pgtable = NULL;
+	spinlock_t *ptl;
 	pmd_t entry;
 
-	lockdep_assert_held(pmd_lockptr(mm, pmd));
+	if (addr < vma->vm_start || addr >= vma->vm_end)
+		return VM_FAULT_SIGBUS;
 
+	if (arch_needs_pgtable_deposit()) {
+		pgtable = pte_alloc_one(vma->vm_mm);
+		if (!pgtable)
+			return VM_FAULT_OOM;
+	}
+
+	ptl = pmd_lock(mm, pmd);
 	if (!pmd_none(*pmd)) {
 		const unsigned long pfn = fop.is_folio ? folio_pfn(fop.folio) :
 					  fop.pfn;
@@ -1395,15 +1405,14 @@ static int insert_pmd(struct vm_area_struct *vma, unsigned long addr,
 		if (write && pmd_present(*pmd)) {
 			if (pmd_pfn(*pmd) != pfn) {
 				WARN_ON_ONCE(!is_huge_zero_pmd(*pmd));
-				return -EEXIST;
+				goto out_unlock;
 			}
 			entry = pmd_mkyoung(*pmd);
 			entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
 			if (pmdp_set_access_flags(vma, addr, pmd, entry, 1))
 				update_mmu_cache_pmd(vma, addr, pmd);
 		}
-
-		return -EEXIST;
+		goto out_unlock;
 	}
 
 	if (fop.is_folio) {
@@ -1424,11 +1433,17 @@ static int insert_pmd(struct vm_area_struct *vma, unsigned long addr,
 	if (pgtable) {
 		pgtable_trans_huge_deposit(mm, pmd, pgtable);
 		mm_inc_nr_ptes(mm);
+		pgtable = NULL;
 	}
 
 	set_pmd_at(mm, addr, pmd, entry);
 	update_mmu_cache_pmd(vma, addr, pmd);
-	return 0;
+
+out_unlock:
+	spin_unlock(ptl);
+	if (pgtable)
+		pte_free(mm, pgtable);
+	return VM_FAULT_NOPAGE;
 }
 
 /**
@@ -1450,9 +1465,6 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, unsigned long pfn,
 	struct folio_or_pfn fop = {
 		.pfn = pfn,
 	};
-	pgtable_t pgtable = NULL;
-	spinlock_t *ptl;
-	int error;
 
 	/*
 	 * If we had pmd_special, we could avoid all these restrictions,
@@ -1464,25 +1476,9 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, unsigned long pfn,
 						(VM_PFNMAP|VM_MIXEDMAP));
 	BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags));
 
-	if (addr < vma->vm_start || addr >= vma->vm_end)
-		return VM_FAULT_SIGBUS;
-
-	if (arch_needs_pgtable_deposit()) {
-		pgtable = pte_alloc_one(vma->vm_mm);
-		if (!pgtable)
-			return VM_FAULT_OOM;
-	}
-
 	pfnmap_setup_cachemode_pfn(pfn, &pgprot);
 
-	ptl = pmd_lock(vma->vm_mm, vmf->pmd);
-	error = insert_pmd(vma, addr, vmf->pmd, fop, pgprot, write,
-			   pgtable);
-	spin_unlock(ptl);
-	if (error && pgtable)
-		pte_free(vma->vm_mm, pgtable);
-
-	return VM_FAULT_NOPAGE;
+	return insert_pmd(vma, addr, vmf->pmd, fop, pgprot, write);
 }
 EXPORT_SYMBOL_GPL(vmf_insert_pfn_pmd);
 
@@ -1491,35 +1487,15 @@ vm_fault_t vmf_insert_folio_pmd(struct vm_fault *vmf, struct folio *folio,
 {
 	struct vm_area_struct *vma = vmf->vma;
 	unsigned long addr = vmf->address & PMD_MASK;
-	struct mm_struct *mm = vma->vm_mm;
 	struct folio_or_pfn fop = {
 		.folio = folio,
 		.is_folio = true,
 	};
-	spinlock_t *ptl;
-	pgtable_t pgtable = NULL;
-	int error;
-
-	if (addr < vma->vm_start || addr >= vma->vm_end)
-		return VM_FAULT_SIGBUS;
 
 	if (WARN_ON_ONCE(folio_order(folio) != PMD_ORDER))
 		return VM_FAULT_SIGBUS;
 
-	if (arch_needs_pgtable_deposit()) {
-		pgtable = pte_alloc_one(vma->vm_mm);
-		if (!pgtable)
-			return VM_FAULT_OOM;
-	}
-
-	ptl = pmd_lock(mm, vmf->pmd);
-	error = insert_pmd(vma, addr, vmf->pmd, fop, vma->vm_page_prot,
-			   write, pgtable);
-	spin_unlock(ptl);
-	if (error && pgtable)
-		pte_free(mm, pgtable);
-
-	return VM_FAULT_NOPAGE;
+	return insert_pmd(vma, addr, vmf->pmd, fop, vma->vm_page_prot, write);
 }
 EXPORT_SYMBOL_GPL(vmf_insert_folio_pmd);
 
Signature · 1 line
2.49.0
Reviewed-by on v1 4/14
#26 lore raw

Re: [PATCH RFC 04/14] mm/huge_memory: move more common code into insert_pmd()

Quoted 3 lines · David Hildenbrand

Let's clean it all further up.

I was thinking maybe we want to pass 'struct vm_fault' directly to insert_pmd(), and retrieve the fields in there, but since you have to retrieve some in insert_pfn_pmd().. maybe not.

Signature · 2 lines
Oscar Salvador
SUSE Labs
Reviewed-by
#27 lore raw

Re: [PATCH RFC 04/14] mm/huge_memory: move more common code into insert_pmd()

Quoted 10 lines · Oscar Salvador
Quoted 3 lines · David Hildenbrand

Let's clean it all further up.

I was thinking maybe we want to pass 'struct vm_fault' directly to insert_pmd(), and retrieve the fields in there, but since you have to retrieve some in insert_pfn_pmd().. maybe not.

Where practical I quite like having callers retrieve context struct fields rather than passing the whole struct down. It makes it very obvious what elements insert_pmd() cares about (in this case one of about fourteen fields).

Anyway looks good, thanks:

Quoted 3 lines
Signature · 2 lines
Oscar Salvador
SUSE Labs
#28 lore raw

RFC PATCH 5/14 mm/huge_memory: move more common code into insert_pud()

In reply to David Hildenbrand

Let's clean it all further up.

---
 mm/huge_memory.c | 36 +++++++++++++-----------------------
 1 file changed, 13 insertions(+), 23 deletions(-)
mm/huge_memory.c +13 -23
index a85e0cd455109..1ea23900b5adb 100644
@@ -1507,25 +1507,30 @@ static pud_t maybe_pud_mkwrite(pud_t pud, struct vm_area_struct *vma)
 	return pud;
 }
 
-static void insert_pud(struct vm_area_struct *vma, unsigned long addr,
+static vm_fault_t insert_pud(struct vm_area_struct *vma, unsigned long addr,
 		pud_t *pud, struct folio_or_pfn fop, pgprot_t prot, bool write)
 {
 	struct mm_struct *mm = vma->vm_mm;
+	spinlock_t *ptl;
 	pud_t entry;
 
+	if (addr < vma->vm_start || addr >= vma->vm_end)
+		return VM_FAULT_SIGBUS;
+
+	ptl = pud_lock(mm, pud);
 	if (!pud_none(*pud)) {
 		const unsigned long pfn = fop.is_folio ? folio_pfn(fop.folio) :
 					  fop.pfn;
 
 		if (write && pud_present(*pud)) {
 			if (WARN_ON_ONCE(pud_pfn(*pud) != pfn))
-				return;
+				goto out_unlock;
 			entry = pud_mkyoung(*pud);
 			entry = maybe_pud_mkwrite(pud_mkdirty(entry), vma);
 			if (pudp_set_access_flags(vma, addr, pud, entry, 1))
 				update_mmu_cache_pud(vma, addr, pud);
 		}
-		return;
+		goto out_unlock;
 	}
 
 	if (fop.is_folio) {
@@ -1544,6 +1549,9 @@ static void insert_pud(struct vm_area_struct *vma, unsigned long addr,
 	}
 	set_pud_at(mm, addr, pud, entry);
 	update_mmu_cache_pud(vma, addr, pud);
+out_unlock:
+	spin_unlock(ptl);
+	return VM_FAULT_NOPAGE;
 }
 
 /**
@@ -1565,7 +1573,6 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, unsigned long pfn,
 	struct folio_or_pfn fop = {
 		.pfn = pfn,
 	};
-	spinlock_t *ptl;
 
 	/*
 	 * If we had pud_special, we could avoid all these restrictions,
@@ -1577,16 +1584,9 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, unsigned long pfn,
 						(VM_PFNMAP|VM_MIXEDMAP));
 	BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags));
 
-	if (addr < vma->vm_start || addr >= vma->vm_end)
-		return VM_FAULT_SIGBUS;
-
 	pfnmap_setup_cachemode_pfn(pfn, &pgprot);
 
-	ptl = pud_lock(vma->vm_mm, vmf->pud);
-	insert_pud(vma, addr, vmf->pud, fop, pgprot, write);
-	spin_unlock(ptl);
-
-	return VM_FAULT_NOPAGE;
+	return insert_pud(vma, addr, vmf->pud, fop, pgprot, write);
 }
 EXPORT_SYMBOL_GPL(vmf_insert_pfn_pud);
 
@@ -1603,25 +1603,15 @@ vm_fault_t vmf_insert_folio_pud(struct vm_fault *vmf, struct folio *folio,
 {
 	struct vm_area_struct *vma = vmf->vma;
 	unsigned long addr = vmf->address & PUD_MASK;
-	pud_t *pud = vmf->pud;
-	struct mm_struct *mm = vma->vm_mm;
 	struct folio_or_pfn fop = {
 		.folio = folio,
 		.is_folio = true,
 	};
-	spinlock_t *ptl;
-
-	if (addr < vma->vm_start || addr >= vma->vm_end)
-		return VM_FAULT_SIGBUS;
 
 	if (WARN_ON_ONCE(folio_order(folio) != PUD_ORDER))
 		return VM_FAULT_SIGBUS;
 
-	ptl = pud_lock(mm, pud);
-	insert_pud(vma, addr, vmf->pud, fop, vma->vm_page_prot, write);
-	spin_unlock(ptl);
-
-	return VM_FAULT_NOPAGE;
+	return insert_pud(vma, addr, vmf->pud, fop, vma->vm_page_prot, write);
 }
 EXPORT_SYMBOL_GPL(vmf_insert_folio_pud);
 #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */
Signature · 1 line
2.49.0
Reviewed-by on v1 5/14
#30 lore raw

Re: [PATCH RFC 05/14] mm/huge_memory: move more common code into insert_pud()

In reply to David Hildenbrand
Quoted 1 line · David Hildenbrand

Let's clean it all further up.

Looks good:

Quoted 109 lines
---
 mm/huge_memory.c | 36 +++++++++++++-----------------------
 1 file changed, 13 insertions(+), 23 deletions(-)
mm/huge_memory.c +13 -23
index a85e0cd455109..1ea23900b5adb 100644
@@ -1507,25 +1507,30 @@ static pud_t maybe_pud_mkwrite(pud_t pud, struct vm_area_struct *vma)
 	return pud;
 }
 
-static void insert_pud(struct vm_area_struct *vma, unsigned long addr,
+static vm_fault_t insert_pud(struct vm_area_struct *vma, unsigned long addr,
 		pud_t *pud, struct folio_or_pfn fop, pgprot_t prot, bool write)
 {
 	struct mm_struct *mm = vma->vm_mm;
+	spinlock_t *ptl;
 	pud_t entry;
 
+	if (addr < vma->vm_start || addr >= vma->vm_end)
+		return VM_FAULT_SIGBUS;
+
+	ptl = pud_lock(mm, pud);
 	if (!pud_none(*pud)) {
 		const unsigned long pfn = fop.is_folio ? folio_pfn(fop.folio) :
 					  fop.pfn;
 
 		if (write && pud_present(*pud)) {
 			if (WARN_ON_ONCE(pud_pfn(*pud) != pfn))
-				return;
+				goto out_unlock;
 			entry = pud_mkyoung(*pud);
 			entry = maybe_pud_mkwrite(pud_mkdirty(entry), vma);
 			if (pudp_set_access_flags(vma, addr, pud, entry, 1))
 				update_mmu_cache_pud(vma, addr, pud);
 		}
-		return;
+		goto out_unlock;
 	}
 
 	if (fop.is_folio) {
@@ -1544,6 +1549,9 @@ static void insert_pud(struct vm_area_struct *vma, unsigned long addr,
 	}
 	set_pud_at(mm, addr, pud, entry);
 	update_mmu_cache_pud(vma, addr, pud);
+out_unlock:
+	spin_unlock(ptl);
+	return VM_FAULT_NOPAGE;
 }
 
 /**
@@ -1565,7 +1573,6 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, unsigned long pfn,
 	struct folio_or_pfn fop = {
 		.pfn = pfn,
 	};
-	spinlock_t *ptl;
 
 	/*
 	 * If we had pud_special, we could avoid all these restrictions,
@@ -1577,16 +1584,9 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, unsigned long pfn,
 						(VM_PFNMAP|VM_MIXEDMAP));
 	BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags));
 
-	if (addr < vma->vm_start || addr >= vma->vm_end)
-		return VM_FAULT_SIGBUS;
-
 	pfnmap_setup_cachemode_pfn(pfn, &pgprot);
 
-	ptl = pud_lock(vma->vm_mm, vmf->pud);
-	insert_pud(vma, addr, vmf->pud, fop, pgprot, write);
-	spin_unlock(ptl);
-
-	return VM_FAULT_NOPAGE;
+	return insert_pud(vma, addr, vmf->pud, fop, pgprot, write);
 }
 EXPORT_SYMBOL_GPL(vmf_insert_pfn_pud);
 
@@ -1603,25 +1603,15 @@ vm_fault_t vmf_insert_folio_pud(struct vm_fault *vmf, struct folio *folio,
 {
 	struct vm_area_struct *vma = vmf->vma;
 	unsigned long addr = vmf->address & PUD_MASK;
-	pud_t *pud = vmf->pud;
-	struct mm_struct *mm = vma->vm_mm;
 	struct folio_or_pfn fop = {
 		.folio = folio,
 		.is_folio = true,
 	};
-	spinlock_t *ptl;
-
-	if (addr < vma->vm_start || addr >= vma->vm_end)
-		return VM_FAULT_SIGBUS;
 
 	if (WARN_ON_ONCE(folio_order(folio) != PUD_ORDER))
 		return VM_FAULT_SIGBUS;
 
-	ptl = pud_lock(mm, pud);
-	insert_pud(vma, addr, vmf->pud, fop, vma->vm_page_prot, write);
-	spin_unlock(ptl);
-
-	return VM_FAULT_NOPAGE;
+	return insert_pud(vma, addr, vmf->pud, fop, vma->vm_page_prot, write);
 }
 EXPORT_SYMBOL_GPL(vmf_insert_folio_pud);
 #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */
Signature · 1 line
2.49.0
#31 lore raw

RFC PATCH 6/14 mm/huge_memory: support huge zero folio in vmf_insert_folio_pmd()

In reply to David Hildenbrand

Just like we do for vmf_insert_page_mkwrite() -> ... -> insert_page_into_pte_locked(), support the huge zero folio.

---
 mm/huge_memory.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
mm/huge_memory.c +5 -3
index 1ea23900b5adb..92400f3baa9ff 100644
@@ -1418,9 +1418,11 @@ static vm_fault_t insert_pmd(struct vm_area_struct *vma, unsigned long addr,
 	if (fop.is_folio) {
 		entry = folio_mk_pmd(fop.folio, vma->vm_page_prot);
 
-		folio_get(fop.folio);
-		folio_add_file_rmap_pmd(fop.folio, &fop.folio->page, vma);
-		add_mm_counter(mm, mm_counter_file(fop.folio), HPAGE_PMD_NR);
+		if (!is_huge_zero_folio(fop.folio)) {
+			folio_get(fop.folio);
+			folio_add_file_rmap_pmd(fop.folio, &fop.folio->page, vma);
+			add_mm_counter(mm, mm_counter_file(fop.folio), HPAGE_PMD_NR);
+		}
 	} else {
 		entry = pmd_mkhuge(pfn_pmd(fop.pfn, prot));
 		entry = pmd_mkspecial(entry);
Signature · 1 line
2.49.0
#32 lore raw

Re: [PATCH RFC 06/14] mm/huge_memory: support huge zero folio in vmf_insert_folio_pmd()

Quoted 4 lines · David Hildenbrand

Just like we do for vmf_insert_page_mkwrite() -> ... -> insert_page_into_pte_locked(), support the huge zero folio.

insert_page_into_pte_locked() creates a special pte in case it finds the zero folio while insert_pmd() doesn't.
I know that we didn't want to create special mappings for normal refcount folios but this seems inconsistent? I'm pretty sure there's a reason but could you elaborate on that?

Quoted 27 lines
---
 mm/huge_memory.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
mm/huge_memory.c +5 -3
index 1ea23900b5adb..92400f3baa9ff 100644
@@ -1418,9 +1418,11 @@ static vm_fault_t insert_pmd(struct vm_area_struct *vma, unsigned long addr,
 	if (fop.is_folio) {
 		entry = folio_mk_pmd(fop.folio, vma->vm_page_prot);
 
-		folio_get(fop.folio);
-		folio_add_file_rmap_pmd(fop.folio, &fop.folio->page, vma);
-		add_mm_counter(mm, mm_counter_file(fop.folio), HPAGE_PMD_NR);
+		if (!is_huge_zero_folio(fop.folio)) {
+			folio_get(fop.folio);
+			folio_add_file_rmap_pmd(fop.folio, &fop.folio->page, vma);
+			add_mm_counter(mm, mm_counter_file(fop.folio), HPAGE_PMD_NR);
+		}
 	} else {
 		entry = pmd_mkhuge(pfn_pmd(fop.pfn, prot));
 		entry = pmd_mkspecial(entry);
Signature · 1 line
2.49.0
Signature · 2 lines
Oscar Salvador
SUSE Labs
#33 lore raw

Re: [PATCH RFC 06/14] mm/huge_memory: support huge zero folio in vmf_insert_folio_pmd()

Quoted 11 lines · Oscar Salvador
Quoted 4 lines · David Hildenbrand

Just like we do for vmf_insert_page_mkwrite() -> ... -> insert_page_into_pte_locked(), support the huge zero folio.

insert_page_into_pte_locked() creates a special pte in case it finds the zero folio while insert_pmd() doesn't.
I know that we didn't want to create special mappings for normal refcount folios but this seems inconsistent? I'm pretty sure there's a reason but could you elaborate on that?

Heh, I should have checked further. I see that happening on patch#8.

Signature · 2 lines
Oscar Salvador
SUSE Labs
Reviewed-by on v1 6/14
#34 lore raw

Re: [PATCH RFC 06/14] mm/huge_memory: support huge zero folio in vmf_insert_folio_pmd()

In reply to David Hildenbrand
Quoted 2 lines · David Hildenbrand

Just like we do for vmf_insert_page_mkwrite() -> ... -> insert_page_into_pte_locked(), support the huge zero folio.

It might just be me because I don't have the full context cached, but I might have appreciated more info here :-).

Quoted 2 lines
Signature · 2 lines
Oscar Salvador
SUSE Labs
#35 lore raw

Re: [PATCH RFC 06/14] mm/huge_memory: support huge zero folio in vmf_insert_folio_pmd()

Quoted 6 lines · Oscar Salvador
Quoted 2 lines · David Hildenbrand

Just like we do for vmf_insert_page_mkwrite() -> ... -> insert_page_into_pte_locked(), support the huge zero folio.

It might just be me because I don't have the full context cached, but I might have appreciated more info here :-).

I can add something similar to what we have in patch #8, stating that we will change it to be special as well soon.

Thanks!

Signature · 3 lines
Cheers,

David / dhildenb
#36 lore raw

RFC PATCH 7/14 fs/dax: use vmf_insert_folio_pmd() to insert the huge zero folio

In reply to David Hildenbrand

Let's convert to vmf_insert_folio_pmd().

In the unlikely case there is already something mapped, we'll now still call trace_dax_pmd_load_hole() and return VM_FAULT_NOPAGE.

That should probably be fine, no need to add special cases for that.

---
 fs/dax.c | 47 ++++++++++-------------------------------------
 1 file changed, 10 insertions(+), 37 deletions(-)
fs/dax.c +10 -37
index 4229513806bea..ae90706674a3f 100644
@@ -1375,51 +1375,24 @@ static vm_fault_t dax_pmd_load_hole(struct xa_state *xas, struct vm_fault *vmf,
 		const struct iomap_iter *iter, void **entry)
 {
 	struct address_space *mapping = vmf->vma->vm_file->f_mapping;
-	unsigned long pmd_addr = vmf->address & PMD_MASK;
-	struct vm_area_struct *vma = vmf->vma;
 	struct inode *inode = mapping->host;
-	pgtable_t pgtable = NULL;
 	struct folio *zero_folio;
-	spinlock_t *ptl;
-	pmd_t pmd_entry;
-	unsigned long pfn;
+	vm_fault_t ret;
 
 	zero_folio = mm_get_huge_zero_folio(vmf->vma->vm_mm);
 
-	if (unlikely(!zero_folio))
-		goto fallback;
-
-	pfn = page_to_pfn(&zero_folio->page);
-	*entry = dax_insert_entry(xas, vmf, iter, *entry, pfn,
-				  DAX_PMD | DAX_ZERO_PAGE);
-
-	if (arch_needs_pgtable_deposit()) {
-		pgtable = pte_alloc_one(vma->vm_mm);
-		if (!pgtable)
-			return VM_FAULT_OOM;
-	}
-
-	ptl = pmd_lock(vmf->vma->vm_mm, vmf->pmd);
-	if (!pmd_none(*(vmf->pmd))) {
-		spin_unlock(ptl);
-		goto fallback;
+	if (unlikely(!zero_folio)) {
+		trace_dax_pmd_load_hole_fallback(inode, vmf, zero_folio, *entry);
+		return VM_FAULT_FALLBACK;
 	}
 
-	if (pgtable) {
-		pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, pgtable);
-		mm_inc_nr_ptes(vma->vm_mm);
-	}
-	pmd_entry = folio_mk_pmd(zero_folio, vmf->vma->vm_page_prot);
-	set_pmd_at(vmf->vma->vm_mm, pmd_addr, vmf->pmd, pmd_entry);
-	spin_unlock(ptl);
-	trace_dax_pmd_load_hole(inode, vmf, zero_folio, *entry);
-	return VM_FAULT_NOPAGE;
+	*entry = dax_insert_entry(xas, vmf, iter, *entry, folio_pfn(zero_folio),
+				  DAX_PMD | DAX_ZERO_PAGE);
 
-fallback:
-	if (pgtable)
-		pte_free(vma->vm_mm, pgtable);
-	trace_dax_pmd_load_hole_fallback(inode, vmf, zero_folio, *entry);
-	return VM_FAULT_FALLBACK;
+	ret = vmf_insert_folio_pmd(vmf, zero_folio, false);
+	if (ret == VM_FAULT_NOPAGE)
+		trace_dax_pmd_load_hole(inode, vmf, zero_folio, *entry);
+	return ret;
 }
 #else
 static vm_fault_t dax_pmd_load_hole(struct xa_state *xas, struct vm_fault *vmf,
Signature · 1 line
2.49.0
#37 lore raw

Re: [PATCH RFC 07/14] fs/dax: use vmf_insert_folio_pmd() to insert the huge zero folio

Quoted 6 lines · David Hildenbrand

Let's convert to vmf_insert_folio_pmd().

In the unlikely case there is already something mapped, we'll now still call trace_dax_pmd_load_hole() and return VM_FAULT_NOPAGE.

That should probably be fine, no need to add special cases for that.

I'm not sure about that. Consider dax_iomap_pmd_fault() -> dax_fault_iter() -> dax_pmd_load_hole(). It calls split_huge_pmd() in response to VM_FAULT_FALLBACK which will no longer happen, what makes that ok?

Quoted 74 lines
---
 fs/dax.c | 47 ++++++++++-------------------------------------
 1 file changed, 10 insertions(+), 37 deletions(-)
fs/dax.c +10 -37
index 4229513806bea..ae90706674a3f 100644
@@ -1375,51 +1375,24 @@ static vm_fault_t dax_pmd_load_hole(struct xa_state *xas, struct vm_fault *vmf,
 		const struct iomap_iter *iter, void **entry)
 {
 	struct address_space *mapping = vmf->vma->vm_file->f_mapping;
-	unsigned long pmd_addr = vmf->address & PMD_MASK;
-	struct vm_area_struct *vma = vmf->vma;
 	struct inode *inode = mapping->host;
-	pgtable_t pgtable = NULL;
 	struct folio *zero_folio;
-	spinlock_t *ptl;
-	pmd_t pmd_entry;
-	unsigned long pfn;
+	vm_fault_t ret;
 
 	zero_folio = mm_get_huge_zero_folio(vmf->vma->vm_mm);
 
-	if (unlikely(!zero_folio))
-		goto fallback;
-
-	pfn = page_to_pfn(&zero_folio->page);
-	*entry = dax_insert_entry(xas, vmf, iter, *entry, pfn,
-				  DAX_PMD | DAX_ZERO_PAGE);
-
-	if (arch_needs_pgtable_deposit()) {
-		pgtable = pte_alloc_one(vma->vm_mm);
-		if (!pgtable)
-			return VM_FAULT_OOM;
-	}
-
-	ptl = pmd_lock(vmf->vma->vm_mm, vmf->pmd);
-	if (!pmd_none(*(vmf->pmd))) {
-		spin_unlock(ptl);
-		goto fallback;
+	if (unlikely(!zero_folio)) {
+		trace_dax_pmd_load_hole_fallback(inode, vmf, zero_folio, *entry);
+		return VM_FAULT_FALLBACK;
 	}
 
-	if (pgtable) {
-		pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, pgtable);
-		mm_inc_nr_ptes(vma->vm_mm);
-	}
-	pmd_entry = folio_mk_pmd(zero_folio, vmf->vma->vm_page_prot);
-	set_pmd_at(vmf->vma->vm_mm, pmd_addr, vmf->pmd, pmd_entry);
-	spin_unlock(ptl);
-	trace_dax_pmd_load_hole(inode, vmf, zero_folio, *entry);
-	return VM_FAULT_NOPAGE;
+	*entry = dax_insert_entry(xas, vmf, iter, *entry, folio_pfn(zero_folio),
+				  DAX_PMD | DAX_ZERO_PAGE);
 
-fallback:
-	if (pgtable)
-		pte_free(vma->vm_mm, pgtable);
-	trace_dax_pmd_load_hole_fallback(inode, vmf, zero_folio, *entry);
-	return VM_FAULT_FALLBACK;
+	ret = vmf_insert_folio_pmd(vmf, zero_folio, false);
+	if (ret == VM_FAULT_NOPAGE)
+		trace_dax_pmd_load_hole(inode, vmf, zero_folio, *entry);
+	return ret;
 }
 #else
 static vm_fault_t dax_pmd_load_hole(struct xa_state *xas, struct vm_fault *vmf,
Signature · 1 line
2.49.0
#38 lore raw

Re: [PATCH RFC 07/14] fs/dax: use vmf_insert_folio_pmd() to insert the huge zero folio

Quoted 11 lines · Alistair Popple
Quoted 6 lines · David Hildenbrand

Let's convert to vmf_insert_folio_pmd().

In the unlikely case there is already something mapped, we'll now still call trace_dax_pmd_load_hole() and return VM_FAULT_NOPAGE.

That should probably be fine, no need to add special cases for that.

I'm not sure about that. Consider dax_iomap_pmd_fault() -> dax_fault_iter() -> dax_pmd_load_hole(). It calls split_huge_pmd() in response to VM_FAULT_FALLBACK which will no longer happen, what makes that ok?

My reasoning was that this is the exact same behavior other vmf_insert_folio_pmd() users here would result in.

But let me dig into the details.

Signature · 3 lines
Cheers,

David / dhildenb
#39 lore raw

Re: [PATCH RFC 07/14] fs/dax: use vmf_insert_folio_pmd() to insert the huge zero folio

Quoted 9 lines · David Hildenbrand
Quoted 8 lines · Alistair Popple
Quoted 6 lines · David Hildenbrand

Let's convert to vmf_insert_folio_pmd().

In the unlikely case there is already something mapped, we'll now still call trace_dax_pmd_load_hole() and return VM_FAULT_NOPAGE.

That should probably be fine, no need to add special cases for that.

Quoted 8 lines
Quoted 3 lines

I'm not sure about that. Consider dax_iomap_pmd_fault() -> dax_fault_iter() -> dax_pmd_load_hole(). It calls split_huge_pmd() in response to VM_FAULT_FALLBACK which will no longer happen, what makes that ok?

My reasoning was that this is the exact same behavior other vmf_insert_folio_pmd() users here would result in.

But let me dig into the details.

Okay, trying to figure out what to do here.

Assume dax_pmd_load_hole() is called and there is already something. We would have returned VM_FAULT_FALLBACK, now we would return VM_FAULT_NO_PAGE.

That obviously only happens when we have not a write fault (otherwise, the shared zeropage does not apply).

In dax_iomap_pmd_fault(), we would indeed split_huge_pmd(). In the DAX case (!anon vma), that would simply zap whatever is already mapped there.

I guess we would then return VM_FAULT_FALLBACK from huge_fault-> ... -> dax_iomap_fault() and core MM code would fallback to handle_pte_fault() etc. and ... load a single PTE mapping the shared zeropage.

BUT

why is this case handled differently than everything else?

E.g.,

(1) when we try inserting the shared zeropage through dax_load_hole()->vmf_insert_page_mkwrite() and there is already something ... we return VM_FAULT_NOPAGE.

(2) when we try inserting a PTE mapping an ordinary folio through dax_fault_iter()->vmf_insert_page_mkwrite() and there is already something ... we return VM_FAULT_NOPAGE.

(3) when we try inserting a PMD mapping an ordinary folio through dax_fault_iter()->vmf_insert_folio_pmd() and there is already something ... we return VM_FAULT_NOPAGE.

So that makes me think ... the VM_FAULT_FALLBACK right now is probably ... wrong? And probably cannot be triggered?

If there is already the huge zerofolio mapped, all good.

Anything else is really not expected I would assume?

Signature · 3 lines
Cheers,

David / dhildenb
#40 lore raw

Re: [PATCH RFC 07/14] fs/dax: use vmf_insert_folio_pmd() to insert the huge zero folio

Quoted 30 lines · David Hildenbrand
Quoted 9 lines · David Hildenbrand
Quoted 8 lines · Alistair Popple
Quoted 6 lines · David Hildenbrand

Let's convert to vmf_insert_folio_pmd().

In the unlikely case there is already something mapped, we'll now still call trace_dax_pmd_load_hole() and return VM_FAULT_NOPAGE.

That should probably be fine, no need to add special cases for that.

Quoted 8 lines
Quoted 3 lines

I'm not sure about that. Consider dax_iomap_pmd_fault() -> dax_fault_iter() -> dax_pmd_load_hole(). It calls split_huge_pmd() in response to VM_FAULT_FALLBACK which will no longer happen, what makes that ok?

My reasoning was that this is the exact same behavior other vmf_insert_folio_pmd() users here would result in.

But let me dig into the details.

Okay, trying to figure out what to do here.

Assume dax_pmd_load_hole() is called and there is already something. We would have returned VM_FAULT_FALLBACK, now we would return VM_FAULT_NO_PAGE.

That obviously only happens when we have not a write fault (otherwise, the shared zeropage does not apply).

In dax_iomap_pmd_fault(), we would indeed split_huge_pmd(). In the DAX case (!anon vma), that would simply zap whatever is already mapped there.

Quoted 7 lines

I guess we would then return VM_FAULT_FALLBACK from huge_fault-> ... -> dax_iomap_fault() and core MM code would fallback to handle_pte_fault() etc. and ... load a single PTE mapping the shared zeropage.

BUT

why is this case handled differently than everything else?

Hmm. Good question, I will have a bit more of a think about it, but your conclusion below is probably correct.

Quoted 10 lines

E.g.,

(1) when we try inserting the shared zeropage through dax_load_hole()->vmf_insert_page_mkwrite() and there is already something ... we return VM_FAULT_NOPAGE.

(2) when we try inserting a PTE mapping an ordinary folio through dax_fault_iter()->vmf_insert_page_mkwrite() and there is already something ... we return VM_FAULT_NOPAGE.

Quoted 7 lines

(3) when we try inserting a PMD mapping an ordinary folio through dax_fault_iter()->vmf_insert_folio_pmd() and there is already something ... we return VM_FAULT_NOPAGE.

So that makes me think ... the VM_FAULT_FALLBACK right now is probably ... wrong? And probably cannot be triggered?

I suspect that's true. At least I just did a full run of xfstest on a XFS DAX filesystem and was unable to trigger this path, so it's certainly not easy to trigger.

Quoted 9 lines

If there is already the huge zerofolio mapped, all good.

Anything else is really not expected I would assume?

Signature · 3 lines
Cheers,

David / dhildenb
#41 lore raw

RFC PATCH 8/14 mm/huge_memory: mark PMD mappings of the huge zero folio special

In reply to David Hildenbrand

The huge zero folio is refcounted (+mapcounted -- is that a word?) differently than "normal" folios, similarly (but different) to the ordinary shared zeropage.

For this reason, we special-case these pages in vm_normal_page*/vm_normal_folio*, and only allow selected callers to still use them (e.g., GUP can still take a reference on them).

vm_normal_page_pmd() already filters out the huge zero folio. However, so far we are not marking it as special like we do with the ordinary shared zeropage. Let's mark it as special, so we can further refactor vm_normal_page_pmd() and vm_normal_page().

While at it, update the doc regarding the shared zero folios.

---
 mm/huge_memory.c |  5 ++++-
 mm/memory.c      | 13 +++++++++----
 2 files changed, 13 insertions(+), 5 deletions(-)
mm/huge_memory.c +4 -1
index 92400f3baa9ff..8f03cd4e40397 100644
@@ -1309,6 +1309,7 @@ static void set_huge_zero_folio(pgtable_t pgtable, struct mm_struct *mm,
 {
 	pmd_t entry;
 	entry = folio_mk_pmd(zero_folio, vma->vm_page_prot);
+	entry = pmd_mkspecial(entry);
 	pgtable_trans_huge_deposit(mm, pmd, pgtable);
 	set_pmd_at(mm, haddr, pmd, entry);
 	mm_inc_nr_ptes(mm);
@@ -1418,7 +1419,9 @@ static vm_fault_t insert_pmd(struct vm_area_struct *vma, unsigned long addr,
 	if (fop.is_folio) {
 		entry = folio_mk_pmd(fop.folio, vma->vm_page_prot);
 
-		if (!is_huge_zero_folio(fop.folio)) {
+		if (is_huge_zero_folio(fop.folio)) {
+			entry = pmd_mkspecial(entry);
+		} else {
 			folio_get(fop.folio);
 			folio_add_file_rmap_pmd(fop.folio, &fop.folio->page, vma);
 			add_mm_counter(mm, mm_counter_file(fop.folio), HPAGE_PMD_NR);
mm/memory.c +9 -4
index 9a1acd057ce59..ef277dab69e33 100644
@@ -541,7 +541,13 @@ static void print_bad_pte(struct vm_area_struct *vma, unsigned long addr,
  *
  * "Special" mappings do not wish to be associated with a "struct page" (either
  * it doesn't exist, or it exists but they don't want to touch it). In this
- * case, NULL is returned here. "Normal" mappings do have a struct page.
+ * case, NULL is returned here. "Normal" mappings do have a struct page and
+ * are ordinarily refcounted.
+ *
+ * Page mappings of the shared zero folios are always considered "special", as
+ * they are not ordinarily refcounted. However, selected page table walkers
+ * (such as GUP) can still identify these mappings and work with the
+ * underlying "struct page".
  *
  * There are 2 broad cases. Firstly, an architecture may define a pte_special()
  * pte bit, in which case this function is trivial. Secondly, an architecture
@@ -571,9 +577,8 @@ static void print_bad_pte(struct vm_area_struct *vma, unsigned long addr,
  *
  * VM_MIXEDMAP mappings can likewise contain memory with or without "struct
  * page" backing, however the difference is that _all_ pages with a struct
- * page (that is, those where pfn_valid is true) are refcounted and considered
- * normal pages by the VM. The only exception are zeropages, which are
- * *never* refcounted.
+ * page (that is, those where pfn_valid is true, except the shared zero
+ * folios) are refcounted and considered normal pages by the VM.
  *
  * The disadvantage is that pages are refcounted (which can be slower and
  * simply not an option for some PFNMAP users). The advantage is that we
Signature · 1 line
2.49.0
Reviewed-by on v1 8/14
#42 lore raw

Re: [PATCH RFC 08/14] mm/huge_memory: mark PMD mappings of the huge zero folio special

Quoted 8 lines · David Hildenbrand

The huge zero folio is refcounted (+mapcounted -- is that a word?) differently than "normal" folios, similarly (but different) to the ordinary shared zeropage.

For this reason, we special-case these pages in vm_normal_page*/vm_normal_folio*, and only allow selected callers to still use them (e.g., GUP can still take a reference on them).

Quoted 8 lines

vm_normal_page_pmd() already filters out the huge zero folio. However, so far we are not marking it as special like we do with the ordinary shared zeropage. Let's mark it as special, so we can further refactor vm_normal_page_pmd() and vm_normal_page().

While at it, update the doc regarding the shared zero folios.

While doing this, would it make sense to update vm_normal_page_pmd() comments to refelect that pmd_special will also catch huge_zero_folio()? It only mentions huge pfnmaps.

It might not be worth doing since you remove that code later on, but maybe if someone stares at this commit alone..

Signature · 2 lines
Oscar Salvador
SUSE Labs
#43 lore raw

Re: [PATCH RFC 08/14] mm/huge_memory: mark PMD mappings of the huge zero folio special

Quoted 18 lines · Oscar Salvador
Quoted 8 lines · David Hildenbrand

The huge zero folio is refcounted (+mapcounted -- is that a word?) differently than "normal" folios, similarly (but different) to the ordinary shared zeropage.

For this reason, we special-case these pages in vm_normal_page*/vm_normal_folio*, and only allow selected callers to still use them (e.g., GUP can still take a reference on them).

Quoted 8 lines

vm_normal_page_pmd() already filters out the huge zero folio. However, so far we are not marking it as special like we do with the ordinary shared zeropage. Let's mark it as special, so we can further refactor vm_normal_page_pmd() and vm_normal_page().

While at it, update the doc regarding the shared zero folios.

Quoted 8 lines

While doing this, would it make sense to update vm_normal_page_pmd() comments to refelect that pmd_special will also catch huge_zero_folio()? It only mentions huge pfnmaps.

It might not be worth doing since you remove that code later on, but maybe if someone stares at this commit alone..

Yes, it should be removed in this patch, thanks!

Signature · 3 lines
Cheers,

David / dhildenb
#44 lore raw

RFC PATCH 9/14 mm/memory: introduce is_huge_zero_pfn() and use it in vm_normal_page_pmd()

In reply to David Hildenbrand

Let's avoid working with the PMD when not required. If vm_normal_page_pmd() would be called on something that is not a present pmd, it would already be a bug (pfn possibly garbage).

While at it, let's support passing in any pfn covered by the huge zero folio by masking off PFN bits -- which should be rather cheap.

---
 include/linux/huge_mm.h | 12 +++++++++++-
 mm/memory.c             |  2 +-
 2 files changed, 12 insertions(+), 2 deletions(-)
include/linux/huge_mm.h +11 -1
index 35e34e6a98a27..b260f9a1fd3f2 100644
@@ -483,9 +483,14 @@ static inline bool is_huge_zero_folio(const struct folio *folio)
 	return READ_ONCE(huge_zero_folio) == folio;
 }
 
+static inline bool is_huge_zero_pfn(unsigned long pfn)
+{
+	return READ_ONCE(huge_zero_pfn) == (pfn & ~(HPAGE_PMD_NR - 1));
+}
+
 static inline bool is_huge_zero_pmd(pmd_t pmd)
 {
-	return pmd_present(pmd) && READ_ONCE(huge_zero_pfn) == pmd_pfn(pmd);
+	return pmd_present(pmd) && is_huge_zero_pfn(pmd_pfn(pmd));
 }
 
 struct folio *mm_get_huge_zero_folio(struct mm_struct *mm);
@@ -633,6 +638,11 @@ static inline bool is_huge_zero_folio(const struct folio *folio)
 	return false;
 }
 
+static inline bool is_huge_zero_pfn(unsigned long pfn)
+{
+	return false;
+}
+
 static inline bool is_huge_zero_pmd(pmd_t pmd)
 {
 	return false;
mm/memory.c +1 -1
index ef277dab69e33..b6c069f4ad11f 100644
@@ -669,7 +669,7 @@ struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr,
 		}
 	}
 
-	if (is_huge_zero_pmd(pmd))
+	if (is_huge_zero_pfn(pfn))
 		return NULL;
 
 	/*
Signature · 1 line
2.49.0
Reviewed-by on v1 9/14
#45 lore raw

Re: [PATCH RFC 09/14] mm/memory: introduce is_huge_zero_pfn() and use it in vm_normal_page_pmd()

Quoted 8 lines · David Hildenbrand

Let's avoid working with the PMD when not required. If vm_normal_page_pmd() would be called on something that is not a present pmd, it would already be a bug (pfn possibly garbage).

While at it, let's support passing in any pfn covered by the huge zero folio by masking off PFN bits -- which should be rather cheap.

Signature · 2 lines
Oscar Salvador
SUSE Labs
#46 lore raw

RFC PATCH 10/14 mm/memory: factor out common code from vm_normal_page_*()

In reply to David Hildenbrand

Let's reduce the code duplication and factor out the non-pte/pmd related magic into vm_normal_page_pfn().

To keep it simpler, check the pfn against both zero folios. We could optimize this, but as it's only for the !CONFIG_ARCH_HAS_PTE_SPECIAL case, it's not a compelling micro-optimization.

With CONFIG_ARCH_HAS_PTE_SPECIAL we don't have to check anything else, really.

It's a good question if we can even hit the !CONFIG_ARCH_HAS_PTE_SPECIAL scenario in the PMD case in practice: but doesn't really matter, as it's now all unified in vm_normal_page_pfn().

While at it, add a check that pmd_special() is really only set where we would expect it.

No functional change intended.

---
 mm/memory.c | 104 +++++++++++++++++++++++-----------------------------
 1 file changed, 46 insertions(+), 58 deletions(-)
mm/memory.c +46 -58
index b6c069f4ad11f..3d3fa01cd217e 100644
@@ -536,6 +536,46 @@ static void print_bad_pte(struct vm_area_struct *vma, unsigned long addr,
 	add_taint(TAINT_BAD_PAGE, LOCKDEP_NOW_UNRELIABLE);
 }
 
+/* Called only if the page table entry is not marked special. */
+static inline struct page *vm_normal_page_pfn(struct vm_area_struct *vma,
+		unsigned long addr, unsigned long pfn)
+{
+	/*
+	 * With CONFIG_ARCH_HAS_PTE_SPECIAL, any special page table mappings
+	 * (incl. shared zero folios) are marked accordingly.
+	 */
+	if (IS_ENABLED(CONFIG_ARCH_HAS_PTE_SPECIAL))
+		goto normal_page;
+
+	if (unlikely(vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP))) {
+		if (vma->vm_flags & VM_MIXEDMAP) {
+			/* If it has a "struct page", it's "normal". */
+			if (!pfn_valid(pfn))
+				return NULL;
+		} else {
+			unsigned long off = (addr - vma->vm_start) >> PAGE_SHIFT;
+
+			/* Only CoW'ed anon folios are "normal". */
+			if (pfn == vma->vm_pgoff + off)
+				return NULL;
+			if (!is_cow_mapping(vma->vm_flags))
+				return NULL;
+		}
+	}
+
+	if (is_zero_pfn(pfn) || is_huge_zero_pfn(pfn))
+		return NULL;
+
+normal_page:
+	/*
+	 * NOTE! We still have PageReserved() pages in the page tables.
+	 * For example, VDSO mappings can cause them to exist.
+	 */
+	VM_WARN_ON_ONCE(!pfn_valid(pfn));
+	VM_WARN_ON_ONCE(is_zero_pfn(pfn) || is_huge_zero_pfn(pfn));
+	return pfn_to_page(pfn);
+}
+
 /*
  * vm_normal_page -- This function gets the "struct page" associated with a pte.
  *
@@ -591,9 +631,7 @@ struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr,
 {
 	unsigned long pfn = pte_pfn(pte);
 
-	if (IS_ENABLED(CONFIG_ARCH_HAS_PTE_SPECIAL)) {
-		if (likely(!pte_special(pte)))
-			goto out;
+	if (unlikely(pte_special(pte))) {
 		if (vma->vm_ops && vma->vm_ops->find_special_page)
 			return vma->vm_ops->find_special_page(vma, addr);
 		if (vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP))
@@ -604,34 +642,7 @@ struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr,
 		print_bad_pte(vma, addr, pte, NULL);
 		return NULL;
 	}
-
-	/* !CONFIG_ARCH_HAS_PTE_SPECIAL case follows: */
-
-	if (unlikely(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP))) {
-		if (vma->vm_flags & VM_MIXEDMAP) {
-			if (!pfn_valid(pfn))
-				return NULL;
-		} else {
-			unsigned long off;
-			off = (addr - vma->vm_start) >> PAGE_SHIFT;
-			if (pfn == vma->vm_pgoff + off)
-				return NULL;
-			if (!is_cow_mapping(vma->vm_flags))
-				return NULL;
-		}
-	}
-
-	if (is_zero_pfn(pfn))
-		return NULL;
-
-	/*
-	 * NOTE! We still have PageReserved() pages in the page tables.
-	 * eg. VDSO mappings can cause them to exist.
-	 */
-out:
-	VM_WARN_ON_ONCE(!pfn_valid(pfn));
-	VM_WARN_ON_ONCE(is_zero_pfn(pfn));
-	return pfn_to_page(pfn);
+	return vm_normal_page_pfn(vma, addr, pfn);
 }
 
 struct folio *vm_normal_folio(struct vm_area_struct *vma, unsigned long addr,
@@ -650,35 +661,12 @@ struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr,
 {
 	unsigned long pfn = pmd_pfn(pmd);
 
-	/* Currently it's only used for huge pfnmaps */
-	if (unlikely(pmd_special(pmd)))
+	if (unlikely(pmd_special(pmd))) {
+		VM_WARN_ON_ONCE(!(vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP)) &&
+				!is_huge_zero_pfn(pfn));
 		return NULL;
-
-	if (unlikely(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP))) {
-		if (vma->vm_flags & VM_MIXEDMAP) {
-			if (!pfn_valid(pfn))
-				return NULL;
-			goto out;
-		} else {
-			unsigned long off;
-			off = (addr - vma->vm_start) >> PAGE_SHIFT;
-			if (pfn == vma->vm_pgoff + off)
-				return NULL;
-			if (!is_cow_mapping(vma->vm_flags))
-				return NULL;
-		}
 	}
-
-	if (is_huge_zero_pfn(pfn))
-		return NULL;
-
-	/*
-	 * NOTE! We still have PageReserved() pages in the page tables.
-	 * eg. VDSO mappings can cause them to exist.
-	 */
-out:
-	VM_WARN_ON_ONCE(!pfn_valid(pfn));
-	return pfn_to_page(pfn);
+	return vm_normal_page_pfn(vma, addr, pfn);
 }
 
 struct folio *vm_normal_folio_pmd(struct vm_area_struct *vma,
Signature · 1 line
2.49.0
Reviewed-by on v1 10/14
#47 lore raw

Re: [PATCH RFC 10/14] mm/memory: factor out common code from vm_normal_page_*()

Quoted 14 lines · David Hildenbrand

Let's reduce the code duplication and factor out the non-pte/pmd related magic into vm_normal_page_pfn().

To keep it simpler, check the pfn against both zero folios. We could optimize this, but as it's only for the !CONFIG_ARCH_HAS_PTE_SPECIAL case, it's not a compelling micro-optimization.

With CONFIG_ARCH_HAS_PTE_SPECIAL we don't have to check anything else, really.

It's a good question if we can even hit the !CONFIG_ARCH_HAS_PTE_SPECIAL scenario in the PMD case in practice: but doesn't really matter, as it's now all unified in vm_normal_page_pfn().

Quoted 6 lines

While at it, add a check that pmd_special() is really only set where we would expect it.

No functional change intended.

Comment below

mm/memory.c struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr, · line 653

Quoted 6 lines

struct folio *vm_normal_folio(struct vm_area_struct *vma, unsigned long addr,

mm/memory.c +0 -1
@@ -650,35 +661,12 @@ struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr,
 {
 	unsigned long pfn = pmd_pfn(pmd);
 
-	/* Currently it's only used for huge pfnmaps */

Although the check kind of spells it out, we could leave this one and also add that huge_zero_pfn, to make it more explicit.

Signature · 2 lines
Oscar Salvador
SUSE Labs
#48 lore raw

Re: [PATCH RFC 10/14] mm/memory: factor out common code from vm_normal_page_*()

Quoted 31 lines · Oscar Salvador
Quoted 14 lines · David Hildenbrand

Let's reduce the code duplication and factor out the non-pte/pmd related magic into vm_normal_page_pfn().

To keep it simpler, check the pfn against both zero folios. We could optimize this, but as it's only for the !CONFIG_ARCH_HAS_PTE_SPECIAL case, it's not a compelling micro-optimization.

With CONFIG_ARCH_HAS_PTE_SPECIAL we don't have to check anything else, really.

It's a good question if we can even hit the !CONFIG_ARCH_HAS_PTE_SPECIAL scenario in the PMD case in practice: but doesn't really matter, as it's now all unified in vm_normal_page_pfn().

Quoted 6 lines

While at it, add a check that pmd_special() is really only set where we would expect it.

No functional change intended.

Comment below

Quoted 5 lines

struct folio *vm_normal_folio(struct vm_area_struct *vma, unsigned long addr,

mm/memory.c +0 -0
@@ -650,35 +661,12 @@ struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr,
  {
  	unsigned long pfn = pmd_pfn(pmd);
 
Quoted 4 lines

mm/memory.c struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr, · line 653

Quoted 1 line
mm/memory.c +0 -1
-	/* Currently it's only used for huge pfnmaps */

Although the check kind of spells it out, we could leave this one and also add that huge_zero_pfn, to make it more explicit.

I don't think that comment is required anymore -- we do exactly what vm_normal_page() does + documents,

What the current users are is not particularly important anymore.

Or why do you think it would still be important?

Signature · 3 lines
Cheers,

David / dhildenb
#49 lore raw

Re: [PATCH RFC 10/14] mm/memory: factor out common code from vm_normal_page_*()

Quoted 6 lines · David Hildenbrand

I don't think that comment is required anymore -- we do exactly what vm_normal_page() does + documents,

What the current users are is not particularly important anymore.

Or why do you think it would still be important?

Maybe the current users are not important, but at least a comment directing to vm_normal_page like "See comment in vm_normal_page".
Here, and in vm_normal_page_pud().

Just someone has it clear why we're only checking for X and Y when we find a pte/pmd/pud special.

But not really a strong opinion here, just I think that it might be helpful.

Signature · 2 lines
Oscar Salvador
SUSE Labs
#50 lore raw

Re: [PATCH RFC 10/14] mm/memory: factor out common code from vm_normal_page_*()

Quoted 8 lines · Oscar Salvador
Quoted 6 lines · David Hildenbrand

I don't think that comment is required anymore -- we do exactly what vm_normal_page() does + documents,

What the current users are is not particularly important anymore.

Or why do you think it would still be important?

Quoted 8 lines

Maybe the current users are not important, but at least a comment directing to vm_normal_page like "See comment in vm_normal_page".
Here, and in vm_normal_page_pud().

Just someone has it clear why we're only checking for X and Y when we find a pte/pmd/pud special.

But not really a strong opinion here, just I think that it might be helpful.

I was already debating with myself whether to add full kerneldoc for these functions ... but yeah, to me the link to "vm_normal_page()" is obvious, but we can just spell it out "see vm_normal_page()".

Signature · 3 lines
Cheers,

David / dhildenb
#51 lore raw

RFC PATCH 11/14 mm: remove "horrible special case to handle copy-on-write behaviour"

In reply to David Hildenbrand

Let's make the kernel a bit less horrible, by removing the linearity requirement in CoW PFNMAP mappings with !CONFIG_ARCH_HAS_PTE_SPECIAL. In particular, stop messing with vma->vm_pgoff in weird ways.

Simply lookup in applicable (i.e., CoW PFNMAP) mappings whether we have an anon folio.

Nobody should ever try mapping anon folios using PFNs, that just screams for other possible issues. To be sure, let's sanity-check when inserting PFNs. Are they really required? Probably not, but it's a good safety net at least for now.

The runtime overhead should be limited: there is nothing to do for !CoW mappings (common case), and archs that care about performance (i.e., GUP-fast) should be supporting CONFIG_ARCH_HAS_PTE_SPECIAL either way.

Likely the sanity checks added in mm/huge_memory.c are not required for now, because that code is probably only wired up with CONFIG_ARCH_HAS_PTE_SPECIAL, but this way is certainly cleaner and more consistent -- and doesn't really cost us anything in the cases we really care about.

---
 include/linux/mm.h |  16 ++++++
 mm/huge_memory.c   |  16 +++++-
 mm/memory.c        | 118 +++++++++++++++++++++++++--------------------
 3 files changed, 96 insertions(+), 54 deletions(-)
include/linux/mm.h +16 -0
index 98a606908307b..3f52871becd3f 100644
@@ -2339,6 +2339,22 @@ static inline bool can_do_mlock(void) { return false; }
 extern int user_shm_lock(size_t, struct ucounts *);
 extern void user_shm_unlock(size_t, struct ucounts *);
 
+#ifdef CONFIG_ARCH_HAS_PTE_SPECIAL
+static inline struct page *vm_pfnmap_normal_page_pfn(struct vm_area_struct *vma,
+		unsigned long pfn)
+{
+	/*
+	 * We don't identify normal pages using PFNs. So if we reach
+	 * this point, it's just for sanity checks that don't apply with
+	 * pte_special() etc.
+	 */
+	return NULL;
+}
+#else
+struct page *vm_pfnmap_normal_page_pfn(struct vm_area_struct *vma,
+		unsigned long pfn);
+#endif
+
 struct folio *vm_normal_folio(struct vm_area_struct *vma, unsigned long addr,
 			     pte_t pte);
 struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr,
mm/huge_memory.c +14 -2
index 8f03cd4e40397..67220c30e7818 100644
@@ -1479,7 +1479,13 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, unsigned long pfn,
 	BUG_ON(!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)));
 	BUG_ON((vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) ==
 						(VM_PFNMAP|VM_MIXEDMAP));
-	BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags));
+
+	/*
+	 * Refuse this pfn if we could mistake it as a refcounted folio
+	 * in a CoW mapping later in vm_normal_page_pmd().
+	 */
+	if ((vma->vm_flags & VM_PFNMAP) && vm_pfnmap_normal_page_pfn(vma, pfn))
+		return VM_FAULT_SIGBUS;
 
 	pfnmap_setup_cachemode_pfn(pfn, &pgprot);
 
@@ -1587,7 +1593,13 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, unsigned long pfn,
 	BUG_ON(!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)));
 	BUG_ON((vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) ==
 						(VM_PFNMAP|VM_MIXEDMAP));
-	BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags));
+
+	/*
+	 * Refuse this pfn if we could mistake it as a refcounted folio
+	 * in a CoW mapping later in vm_normal_page_pud().
+	 */
+	if ((vma->vm_flags & VM_PFNMAP) && vm_pfnmap_normal_page_pfn(vma, pfn))
+		return VM_FAULT_SIGBUS;
 
 	pfnmap_setup_cachemode_pfn(pfn, &pgprot);
 
mm/memory.c +66 -52

247 lines not on this page · raw message

Signature · 1 line
2.49.0
#52 lore raw

Re: [PATCH RFC 11/14] mm: remove "horrible special case to handle copy-on-write behaviour"

Quoted 18 lines · David Hildenbrand

Let's make the kernel a bit less horrible, by removing the linearity requirement in CoW PFNMAP mappings with !CONFIG_ARCH_HAS_PTE_SPECIAL. In particular, stop messing with vma->vm_pgoff in weird ways.

Simply lookup in applicable (i.e., CoW PFNMAP) mappings whether we have an anon folio.

Nobody should ever try mapping anon folios using PFNs, that just screams for other possible issues. To be sure, let's sanity-check when inserting PFNs. Are they really required? Probably not, but it's a good safety net at least for now.

The runtime overhead should be limited: there is nothing to do for !CoW mappings (common case), and archs that care about performance (i.e., GUP-fast) should be supporting CONFIG_ARCH_HAS_PTE_SPECIAL either way.

Quoted 8 lines

Likely the sanity checks added in mm/huge_memory.c are not required for now, because that code is probably only wired up with CONFIG_ARCH_HAS_PTE_SPECIAL, but this way is certainly cleaner and more consistent -- and doesn't really cost us anything in the cases we really care about.

---

I'm still thinking about this patch here, and will likely send out the other patches first as a v1, and come back to this one later.

Really, someone mapping random memory using /dev/mem, and then getting anonymous memory in there is the (nasty) corner case I ignored.

There are rather nasty ways of trying to detect if an anon folio really fits into a VMA, but I'd like to avoid that.

What I am thinking about right now is that we could, for these special architectures, simply disallow CoW faults on /dev/mem.

So we would still allow MAP_PRIVATE mappings (e.g., random app opening /dev/mem using MAP_PRIVATE but never actually writing to that memory), but the actual CoW faults would fail without pte_special().

Some more thinking to do ...

Signature · 3 lines
Cheers,

David / dhildenb
#53 lore raw

Re: [PATCH RFC 11/14] mm: remove "horrible special case to handle copy-on-write behaviour"

Quoted 2 lines · David Hildenbrand

I'm still thinking about this patch here, and will likely send out the other patches first as a v1, and come back to this one later.

Patch#12 depends on this one, but Patch#13 should be ok to review If I ignore the 'addr' parameter being dropped, right?

Signature · 2 lines
Oscar Salvador
SUSE Labs
#54 lore raw

Re: [PATCH RFC 11/14] mm: remove "horrible special case to handle copy-on-write behaviour"

Quoted 6 lines · Oscar Salvador
Quoted 2 lines · David Hildenbrand

I'm still thinking about this patch here, and will likely send out the other patches first as a v1, and come back to this one later.

Patch#12 depends on this one, but Patch#13 should be ok to review If I ignore the 'addr' parameter being dropped, right?

Yes, only #12 and #13 will be gone. #14 and #15 will simply have the "addr" parameter in the _pud() variant as well.

Signature · 3 lines
Cheers,

David / dhildenb
#55 lore raw

RFC PATCH 12/14 mm: drop addr parameter from vm_normal_*_pmd()

In reply to David Hildenbrand

No longer required, let's drop it.

---
 fs/proc/task_mmu.c | 6 +++---
 include/linux/mm.h | 6 ++----
 mm/huge_memory.c   | 4 ++--
 mm/memory.c        | 8 +++-----
 mm/pagewalk.c      | 2 +-
 5 files changed, 11 insertions(+), 15 deletions(-)
fs/proc/task_mmu.c +3 -3
index c4ad3083bbfa0..36ef67cdf7a3b 100644
@@ -861,7 +861,7 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr,
 	struct folio *folio;
 
 	if (pmd_present(*pmd)) {
-		page = vm_normal_page_pmd(vma, addr, *pmd);
+		page = vm_normal_page_pmd(vma, *pmd);
 		present = true;
 	} else if (unlikely(thp_migration_supported() && is_swap_pmd(*pmd))) {
 		swp_entry_t entry = pmd_to_swp_entry(*pmd);
@@ -2177,7 +2177,7 @@ static unsigned long pagemap_thp_category(struct pagemap_scan_private *p,
 			categories |= PAGE_IS_WRITTEN;
 
 		if (p->masks_of_interest & PAGE_IS_FILE) {
-			page = vm_normal_page_pmd(vma, addr, pmd);
+			page = vm_normal_page_pmd(vma, pmd);
 			if (page && !PageAnon(page))
 				categories |= PAGE_IS_FILE;
 		}
@@ -2942,7 +2942,7 @@ static struct page *can_gather_numa_stats_pmd(pmd_t pmd,
 	if (!pmd_present(pmd))
 		return NULL;
 
-	page = vm_normal_page_pmd(vma, addr, pmd);
+	page = vm_normal_page_pmd(vma, pmd);
 	if (!page)
 		return NULL;
 
include/linux/mm.h +2 -4
index 3f52871becd3f..ef709457c7076 100644
@@ -2359,10 +2359,8 @@ struct folio *vm_normal_folio(struct vm_area_struct *vma, unsigned long addr,
 			     pte_t pte);
 struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr,
 			     pte_t pte);
-struct folio *vm_normal_folio_pmd(struct vm_area_struct *vma,
-				  unsigned long addr, pmd_t pmd);
-struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr,
-				pmd_t pmd);
+struct folio *vm_normal_folio_pmd(struct vm_area_struct *vma, pmd_t pmd);
+struct page *vm_normal_page_pmd(struct vm_area_struct *vma, pmd_t pmd);
 
 void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address,
 		  unsigned long size);
mm/huge_memory.c +2 -2
index 67220c30e7818..bf2aed8d92ec2 100644
@@ -1994,7 +1994,7 @@ static inline bool can_change_pmd_writable(struct vm_area_struct *vma,
 
 	if (!(vma->vm_flags & VM_SHARED)) {
 		/* See can_change_pte_writable(). */
-		page = vm_normal_page_pmd(vma, addr, pmd);
+		page = vm_normal_page_pmd(vma, pmd);
 		return page && PageAnon(page) && PageAnonExclusive(page);
 	}
 
@@ -2033,7 +2033,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	    can_change_pmd_writable(vma, vmf->address, pmd))
 		writable = true;
 
-	folio = vm_normal_folio_pmd(vma, haddr, pmd);
+	folio = vm_normal_folio_pmd(vma, pmd);
 	if (!folio)
 		goto out_map;
 
mm/memory.c +3 -5
index ace9c59e97181..34f961024e8e6 100644
@@ -663,8 +663,7 @@ struct folio *vm_normal_folio(struct vm_area_struct *vma, unsigned long addr,
 }
 
 #ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES
-struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr,
-				pmd_t pmd)
+struct page *vm_normal_page_pmd(struct vm_area_struct *vma, pmd_t pmd)
 {
 	unsigned long pfn = pmd_pfn(pmd);
 
@@ -676,10 +675,9 @@ struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr,
 	return vm_normal_page_pfn(vma, pfn);
 }
 
-struct folio *vm_normal_folio_pmd(struct vm_area_struct *vma,
-				  unsigned long addr, pmd_t pmd)
+struct folio *vm_normal_folio_pmd(struct vm_area_struct *vma, pmd_t pmd)
 {
-	struct page *page = vm_normal_page_pmd(vma, addr, pmd);
+	struct page *page = vm_normal_page_pmd(vma, pmd);
 
 	if (page)
 		return page_folio(page);
mm/pagewalk.c +1 -1
index 648038247a8d2..0edb7240d090c 100644
@@ -944,7 +944,7 @@ struct folio *folio_walk_start(struct folio_walk *fw,
 			spin_unlock(ptl);
 			goto pte_table;
 		} else if (pmd_present(pmd)) {
-			page = vm_normal_page_pmd(vma, addr, pmd);
+			page = vm_normal_page_pmd(vma, pmd);
 			if (page) {
 				goto found;
 			} else if ((flags & FW_ZEROPAGE) &&
Signature · 1 line
2.49.0
#56 lore raw

RFC PATCH 13/14 mm: introduce and use vm_normal_page_pud()

In reply to David Hildenbrand

Let's introduce vm_normal_page_pud(), which ends up being fairly simple because of our new common helpers and there not being a PUD-sized zero folio.

Use vm_normal_page_pud() in folio_walk_start() to resolve a TODO, structuring the code like the other (pmd/pte) cases. Defer introducing vm_normal_folio_pud() until really used.

---
 include/linux/mm.h |  1 +
 mm/memory.c        | 11 +++++++++++
 mm/pagewalk.c      | 20 ++++++++++----------
 3 files changed, 22 insertions(+), 10 deletions(-)
include/linux/mm.h +1 -0
index ef709457c7076..022e8ef2c78ef 100644
@@ -2361,6 +2361,7 @@ struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr,
 			     pte_t pte);
 struct folio *vm_normal_folio_pmd(struct vm_area_struct *vma, pmd_t pmd);
 struct page *vm_normal_page_pmd(struct vm_area_struct *vma, pmd_t pmd);
+struct page *vm_normal_page_pud(struct vm_area_struct *vma, pud_t pud);
 
 void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address,
 		  unsigned long size);
mm/memory.c +11 -0
index 34f961024e8e6..6c65f51248250 100644
@@ -683,6 +683,17 @@ struct folio *vm_normal_folio_pmd(struct vm_area_struct *vma, pmd_t pmd)
 		return page_folio(page);
 	return NULL;
 }
+
+struct page *vm_normal_page_pud(struct vm_area_struct *vma, pud_t pud)
+{
+	unsigned long pfn = pud_pfn(pud);
+
+	if (unlikely(pud_special(pud))) {
+		VM_WARN_ON_ONCE(!(vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP)));
+		return NULL;
+	}
+	return vm_normal_page_pfn(vma, pfn);
+}
 #endif
 
 /**
mm/pagewalk.c +10 -10
index 0edb7240d090c..8bd95cf326872 100644
@@ -902,23 +902,23 @@ struct folio *folio_walk_start(struct folio_walk *fw,
 		fw->pudp = pudp;
 		fw->pud = pud;
 
-		/*
-		 * TODO: FW_MIGRATION support for PUD migration entries
-		 * once there are relevant users.
-		 */
-		if (!pud_present(pud) || pud_special(pud)) {
+		if (pud_none(pud)) {
 			spin_unlock(ptl);
 			goto not_found;
-		} else if (!pud_leaf(pud)) {
+		} else if (pud_present(pud) && !pud_leaf(pud)) {
 			spin_unlock(ptl);
 			goto pmd_table;
+		} else if (pud_present(pud)) {
+			page = vm_normal_page_pud(vma, pud);
+			if (page)
+				goto found;
 		}
 		/*
-		 * TODO: vm_normal_page_pud() will be handy once we want to
-		 * support PUD mappings in VM_PFNMAP|VM_MIXEDMAP VMAs.
+		 * TODO: FW_MIGRATION support for PUD migration entries
+		 * once there are relevant users.
 		 */
-		page = pud_page(pud);
-		goto found;
+		spin_unlock(ptl);
+		goto not_found;
 	}
 
 pmd_table:
Signature · 1 line
2.49.0
Reviewed-by on v1 13/14
#57 lore raw

Re: [PATCH RFC 13/14] mm: introduce and use vm_normal_page_pud()

Quoted 9 lines · David Hildenbrand

Let's introduce vm_normal_page_pud(), which ends up being fairly simple because of our new common helpers and there not being a PUD-sized zero folio.

Use vm_normal_page_pud() in folio_walk_start() to resolve a TODO, structuring the code like the other (pmd/pte) cases. Defer introducing vm_normal_folio_pud() until really used.

Ignoring the parameter stuff, I'd make the same comment I made wrt. vm_normal_page_pmd, but LGTM.

Signature · 2 lines
Oscar Salvador
SUSE Labs
#58 lore raw

RFC PATCH 14/14 mm: rename vm_ops->find_special_page() to vm_ops->find_normal_page()

In reply to David Hildenbrand

... and hide it behind a kconfig option. There is really no need for any !xen code to perform this check.

The naming is a bit off: we want to find the "normal" page when a PTE was marked "special". So it's really not "finding a special" page.

Improve the documentation, and add a comment in the code where XEN ends up performing the pte_mkspecial() through a hypercall. More details can be found in commit 923b2919e2c3 ("xen/gntdev: mark userspace PTEs as special on x86 PV guests").

---
 drivers/xen/Kconfig              |  1 +
 drivers/xen/gntdev.c             |  5 +++--
 include/linux/mm.h               | 18 +++++++++++++-----
 mm/Kconfig                       |  2 ++
 mm/memory.c                      | 10 ++++++++--
 tools/testing/vma/vma_internal.h | 18 +++++++++++++-----
 6 files changed, 40 insertions(+), 14 deletions(-)
drivers/xen/Kconfig +1 -0
index 24f485827e039..f9a35ed266ecf 100644
@@ -138,6 +138,7 @@ config XEN_GNTDEV
 	depends on XEN
 	default m
 	select MMU_NOTIFIER
+	select FIND_NORMAL_PAGE
 	help
 	  Allows userspace processes to use grants.
 
drivers/xen/gntdev.c +3 -2
index 61faea1f06630..d1bc0dae2cdf9 100644
@@ -309,6 +309,7 @@ static int find_grant_ptes(pte_t *pte, unsigned long addr, void *data)
 	BUG_ON(pgnr >= map->count);
 	pte_maddr = arbitrary_virt_to_machine(pte).maddr;
 
+	/* Note: this will perform a pte_mkspecial() through the hypercall. */
 	gnttab_set_map_op(&map->map_ops[pgnr], pte_maddr, flags,
 			  map->grants[pgnr].ref,
 			  map->grants[pgnr].domid);
@@ -516,7 +517,7 @@ static void gntdev_vma_close(struct vm_area_struct *vma)
 	gntdev_put_map(priv, map);
 }
 
-static struct page *gntdev_vma_find_special_page(struct vm_area_struct *vma,
+static struct page *gntdev_vma_find_normal_page(struct vm_area_struct *vma,
 						 unsigned long addr)
 {
 	struct gntdev_grant_map *map = vma->vm_private_data;
@@ -527,7 +528,7 @@ static struct page *gntdev_vma_find_special_page(struct vm_area_struct *vma,
 static const struct vm_operations_struct gntdev_vmops = {
 	.open = gntdev_vma_open,
 	.close = gntdev_vma_close,
-	.find_special_page = gntdev_vma_find_special_page,
+	.find_normal_page = gntdev_vma_find_normal_page,
 };
 
 /* ------------------------------------------------------------------ */
include/linux/mm.h +13 -5
index 022e8ef2c78ef..b01475f3dca99 100644
@@ -646,13 +646,21 @@ struct vm_operations_struct {
 	struct mempolicy *(*get_policy)(struct vm_area_struct *vma,
 					unsigned long addr, pgoff_t *ilx);
 #endif
+#ifdef CONFIG_FIND_NORMAL_PAGE
 	/*
-	 * Called by vm_normal_page() for special PTEs to find the
-	 * page for @addr.  This is useful if the default behavior
-	 * (using pte_page()) would not find the correct page.
+	 * Called by vm_normal_page() for special PTEs in @vma at @addr. This
+	 * allows for returning a "normal" page from vm_normal_page() even
+	 * though the PTE indicates that the "struct page" either does not exist
+	 * or should not be touched: "special".
+	 *
+	 * Do not add new users: this really only works when a "normal" page
+	 * was mapped, but then the PTE got changed to something weird (+
+	 * marked special) that would not make pte_pfn() identify the originally
+	 * inserted page.
 	 */
-	struct page *(*find_special_page)(struct vm_area_struct *vma,
-					  unsigned long addr);
+	struct page *(*find_normal_page)(struct vm_area_struct *vma,
+					 unsigned long addr);
+#endif /* CONFIG_FIND_NORMAL_PAGE */
 };
 
 #ifdef CONFIG_NUMA_BALANCING
mm/Kconfig +2 -0
index c6194d1f9d170..607a3f9672bdb 100644
@@ -1390,6 +1390,8 @@ config PT_RECLAIM
 
 	  Note: now only empty user PTE page table pages will be reclaimed.
 
+config FIND_NORMAL_PAGE
+	def_bool n
 
 source "mm/damon/Kconfig"
 
mm/memory.c +8 -2
index 6c65f51248250..1eba95fcde096 100644
@@ -619,6 +619,10 @@ static inline struct page *vm_normal_page_pfn(struct vm_area_struct *vma,
  * If an architecture does not support pte_special(), this function is less
  * trivial and more expensive in some cases.
  *
+ * With CONFIG_FIND_NORMAL_PAGE, we might have pte_special() set on PTEs that
+ * actually map "normal" pages: however, that page cannot be looked up through
+ * pte_pfn(), but instead will be looked up through vm_ops->find_normal_page().
+ *
  * A raw VM_PFNMAP mapping (ie. one that is not COWed) is always considered a
  * special mapping (even if there are underlying and valid "struct pages").
  * COWed pages of a VM_PFNMAP are always normal.
@@ -639,8 +643,10 @@ struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr,
 	unsigned long pfn = pte_pfn(pte);
 
 	if (unlikely(pte_special(pte))) {
-		if (vma->vm_ops && vma->vm_ops->find_special_page)
-			return vma->vm_ops->find_special_page(vma, addr);
+#ifdef CONFIG_FIND_NORMAL_PAGE
+		if (vma->vm_ops && vma->vm_ops->find_normal_page)
+			return vma->vm_ops->find_normal_page(vma, addr);
+#endif /* CONFIG_FIND_NORMAL_PAGE */
 		if (vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP))
 			return NULL;
 		if (is_zero_pfn(pfn))
tools/testing/vma/vma_internal.h +13 -5
index 51dd122b8d501..c5bf041036dd7 100644
@@ -470,13 +470,21 @@ struct vm_operations_struct {
 	struct mempolicy *(*get_policy)(struct vm_area_struct *vma,
 					unsigned long addr, pgoff_t *ilx);
 #endif
+#ifdef CONFIG_FIND_NORMAL_PAGE
 	/*
-	 * Called by vm_normal_page() for special PTEs to find the
-	 * page for @addr.  This is useful if the default behavior
-	 * (using pte_page()) would not find the correct page.
+	 * Called by vm_normal_page() for special PTEs in @vma at @addr. This
+	 * allows for returning a "normal" page from vm_normal_page() even
+	 * though the PTE indicates that the "struct page" either does not exist
+	 * or should not be touched: "special".
+	 *
+	 * Do not add new users: this really only works when a "normal" page
+	 * was mapped, but then the PTE got changed to something weird (+
+	 * marked special) that would not make pte_pfn() identify the originally
+	 * inserted page.
 	 */
-	struct page *(*find_special_page)(struct vm_area_struct *vma,
-					  unsigned long addr);
+	struct page *(*find_normal_page)(struct vm_area_struct *vma,
+					 unsigned long addr);
+#endif /* CONFIG_FIND_NORMAL_PAGE */
 };
 
 struct vm_unmapped_area_info {
Signature · 1 line
2.49.0
#59 lore raw

Re: [PATCH RFC 14/14] mm: rename vm_ops->find_special_page() to vm_ops->find_normal_page()

Quoted 10 lines · David Hildenbrand

... and hide it behind a kconfig option. There is really no need for any !xen code to perform this check.

The naming is a bit off: we want to find the "normal" page when a PTE was marked "special". So it's really not "finding a special" page.

Improve the documentation, and add a comment in the code where XEN ends up performing the pte_mkspecial() through a hypercall. More details can be found in commit 923b2919e2c3 ("xen/gntdev: mark userspace PTEs as special on x86 PV guests").

Looks good to me.
Since this seems a "mistake" from the past we don't want to repeat, I wonder whether we could seal FIND_NORMAL_PAGE somehow, and scream if someone tries to enable it on !XEN environments.

Signature · 2 lines
Oscar Salvador
SUSE Labs
#60 lore raw

Re: [PATCH RFC 14/14] mm: rename vm_ops->find_special_page() to vm_ops->find_normal_page()

Quoted 12 lines · Oscar Salvador
Quoted 10 lines · David Hildenbrand

... and hide it behind a kconfig option. There is really no need for any !xen code to perform this check.

The naming is a bit off: we want to find the "normal" page when a PTE was marked "special". So it's really not "finding a special" page.

Improve the documentation, and add a comment in the code where XEN ends up performing the pte_mkspecial() through a hypercall. More details can be found in commit 923b2919e2c3 ("xen/gntdev: mark userspace PTEs as special on x86 PV guests").

Quoted 4 lines

Looks good to me.
Since this seems a "mistake" from the past we don't want to repeat, I wonder whether we could seal FIND_NORMAL_PAGE somehow, and scream if someone tries to enable it on !XEN environments.

Hm, probably not that easy. I mean, as long as we cannot get rid of the XEN stuff, we don't particularly care about new users ... and it seems unlikely to get rid of that XEN stuff :(

Signature · 3 lines
Cheers,

David / dhildenb
#61 lore raw
In reply to David Hildenbrand
Quoted 16 lines · David Hildenbrand

RFC because it's based on mm-new where some things might still change around the devmap removal stuff.

While removing support for CoW PFNMAPs is a noble goal, I am not even sure if we can remove said support for e.g., /dev/mem that easily.

In the end, Cow PFNMAPs are pretty simple: everything is "special" except CoW'ed anon folios, that are "normal".

The only complication is: how to identify such pages without pte_special(). Because with pte_special(), it's easy.

Well, of course, one day all architectures might support pte_special() ... either because we added support for pte_special() or removed support for ... these architectures from Linux.

Quoted 7 lines

No need to wait for that day. Let's do some cleanups around vm_normal_page()/vm_normal_page_pmd() and handling of the huge zero folio, and remove the "horrible special case to handle copy-on-write behaviour" that does questionable things in remap_pfn_range() with a VMA, simply by

... looking for anonymous folios in CoW PFNMAPs to identify anonymous folios? I know, sounds crazy ;)

I'll mention one corner case that just occurred to me: assume someone maps arbitrary /dev/mem that is actually used by the kernel for user space, and then some of that memory gets allocated as anonymous memory, it would probably be a problem.

Hmm, I'll have to think about that, and the interaction with CONFIG_STRICT_DEVMEM.

Signature · 3 lines
Cheers,

David / dhildenb
#62 lore raw
Quoted 25 lines · David Hildenbrand
Quoted 16 lines · David Hildenbrand

RFC because it's based on mm-new where some things might still change around the devmap removal stuff.

While removing support for CoW PFNMAPs is a noble goal, I am not even sure if we can remove said support for e.g., /dev/mem that easily.

In the end, Cow PFNMAPs are pretty simple: everything is "special" except CoW'ed anon folios, that are "normal".

The only complication is: how to identify such pages without pte_special(). Because with pte_special(), it's easy.

Well, of course, one day all architectures might support pte_special() ... either because we added support for pte_special() or removed support for ... these architectures from Linux.

Quoted 7 lines

No need to wait for that day. Let's do some cleanups around vm_normal_page()/vm_normal_page_pmd() and handling of the huge zero folio, and remove the "horrible special case to handle copy-on-write behaviour" that does questionable things in remap_pfn_range() with a VMA, simply by

... looking for anonymous folios in CoW PFNMAPs to identify anonymous folios? I know, sounds crazy ;)

Quoted 7 lines

I'll mention one corner case that just occurred to me: assume someone maps arbitrary /dev/mem that is actually used by the kernel for user space, and then some of that memory gets allocated as anonymous memory, it would probably be a problem.

Hmm, I'll have to think about that, and the interaction with CONFIG_STRICT_DEVMEM.

The /dev/mem mapping of arbitrary memory is indeed the hard case. To handle all that, patch #11 is too simplistic.

I have some idea how to make it work, but have to think about a couple of corner cases.

Signature · 3 lines
Cheers,

David / dhildenb
#64 lore raw
Quoted 2 lines · Lorenzo Stoakes

David, are you planning a v2 of this soon? If so I'll hold off review until then, if not I can get stuck in when I have time?

There will probably be a v1 called "mm: vm_normal_page*()" where I drop the problematic bit, and respin the problematic bit as a new RFC once I have a better understanding.

So, no need to review right now.

Signature · 3 lines
Cheers,

David / dhildenb