Skip to contentSkip to conversation

mm: fixes for huge PFN mappings

PATCH In review Eliav Farber 6 participants 14 messages Nov 12 – Nov 13

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v2 cover mm: fixes for huge PFN mappings

Four independent fixes to the existing huge PFN mapping code. They are not dependent on each other and can be taken separately.

Patches 1-3 are three symptoms of one race. A VM_PFNMAP mapping that provides both ->fault and ->huge_fault can take faults of different orders on the same entry, because the mm's THP policy is read early in each fault and is not held across it. Flipping it while faults are in flight, for example with prctl(PR_SET_THP_DISABLE), is enough: one fault takes the PTE path while another installs a PMD leaf over the same address, or the reverse.

This is reachable today. vfio-pci has exactly that shape:
vfio_pci_mmap_ops provides .fault, which reaches insert_pfn() for order 0, and .huge_fault, which reaches insert_pmd(); mshv_vtl_main and nvgrace-gpu are built the same way.

1. walk_to_pmd() asserts that the PMD is not a leaf. With CONFIG_DEBUG_VM=y the losing fault hits that assertion instead of recovering.
2. insert_pfn() reaches the PTE through get_locked_pte(), which folds the table allocation and the mapping into one call and turns a NULL from either into VM_FAULT_OOM. The losing fault therefore reports OOM for a mapping that is healthy.
3. insert_pmd() and insert_pud() compare the PFN of a non-empty entry without checking that it is a leaf. A page table installed by a concurrent order-0 fault makes pmd_pfn()/pud_pfn() report the table's own PFN, which never matches, and the mismatch is warned about. Under panic_on_warn that is fatal.

Patch 4 is unrelated to the race. insert_pmd() and insert_pud() build a huge entry from a pgprot laid out for a PTE. On x86 the PAT bit sits at bit 7 in a PTE and at bit 12 in a large entry, and bit 7 in a large entry is _PAGE_PSE, so the attribute is silently dropped: write-through degrades to UC and write-protect degrades to write-combining, which loses the write protection. Run the pgprot through pgprot_4k_2_large(), with a generic identity fallback for architectures that encode the levels alike.

How these were observed, since it differs between them:

Patches 1-3 were reproduced on x86_64 under QEMU with four threads colliding on one address of each PMD window of a VM_PFNMAP mapping while another thread flipped PR_SET_THP_DISABLE. With patches 1 and 2 reverted and CONFIG_DEBUG_VM=y, that lands within three seconds:

kernel BUG at mm/memory.c:2392!

  RIP: 0010:walk_to_pmd+0x188/0x190
  Call Trace:
   get_locked_pte+0x1b/0x70
   insert_pfn+0x40/0x230
   vmf_insert_pfn_prot+0x94/0x100
   __do_fault+0x30/0xb0
   __handle_mm_fault+0xbb2/0x1a90
   handle_mm_fault+0x9b/0x270
   do_user_addr_fault+0x15d/0x630

and BUG() leaves mmap_lock held, so the system then wedges. With those two applied the race is survived, and patch 3's warning appears instead:

  WARNING: mm/huge_memory.c:1649 at insert_pmd.isra.0+0x1d3/0x430
  Call Trace:
   vmf_insert_pfn_pmd+0xa6/0xf0
   __handle_mm_fault+0x45b/0x1a90

With all four applied, four runs of the same test made 102656 colliding touches with no BUG, no warning, no OOM and no incorrect data. Debug counters placed at each site in a throwaway tree confirm the recovery path in patch 2 is taken rather than merely not crashing: in three of the four runs a fault reached the VM_FAULT_NOPAGE return after the page table had been allocated successfully, which is the path this version splits out, while the counter for a genuine allocation failure stayed at zero.

Patch 4 was found by inspection rather than from a failure report. The in-tree ->huge_fault providers map write-back or uncached memory, and only write-through and write-protect set _PAGE_PAT, so I have not seen the dropped attribute in practice. The entry is still built from the wrong encoding, and a caller asking for write-protect would lose it silently.

Found while adding huge_fault support for /dev/mem and UIO:
https://lore.kernel.org/all/[email protected]/
That series is not a prerequisite for any of these.

Tested on x86_64 under QEMU with CONFIG_DEBUG_VM=y. The race above needs a VM_PFNMAP mapping providing both ->fault and ->huge_fault, so the reproducer ran on these four patches applied on top of the /dev/mem and UIO series linked above, which gave it one to use; vfio-pci would serve the same purpose. A 30-check functional suite over /dev/mem and UIO mappings passes on that kernel, with the same counter totals as v1. This series on its own boots clean, and mm/ and drivers/uio/ were cross-built for riscv64 with -Werror W=1, the other architecture selecting ARCH_SUPPORTS_PUD_PFNMAP, so insert_pud() and pud_leaf() are compiled. checkpatch --strict is clean on all four.

Changes since v1:

- Patch 2: split pte_alloc_map_lock() into pte_alloc() followed by pte_offset_map_lock(), rather than calling it and then re-reading the PMD to work out which of the two failures had happened. Only a failed allocation now returns VM_FAULT_OOM, and a NULL from the map returns VM_FAULT_NOPAGE. That also covers the non-present and bad-PMD cases, which v1 still reported as OOM, and it drops the pmd_trans_huge() re-read. insert_pages() already builds the PTE the same way. Suggested by Jose A. Perez de Azpillaga.
- Patch 4: corrected the commit message. It said a write-through range ends up write-back; with the PAT slots Linux programs, write-through ends up UC and write-protect ends up write-combining, as the cover letter said.
- Patch 4: Fixes: is now 5cad465d7fa6 ("mm: add vmf_insert_pfn_pmd()"), the commit that introduced the unconverted pgprot, instead of 6857be5fecae. Pointed out by Jose A. Perez de Azpillaga.
- Patches 1 and 3: picked up Reviewed-by tags, otherwise unchanged.

v1: https://lore.kernel.org/all/[email protected]/

Based on v7.4-rc2 (067c5c24cef2).

Eliav Farber (4):
mm: drop the huge PMD assertion from walk_to_pmd()
mm: don't report OOM when a huge PMD race blocks a pfn insert mm/huge_memory: don't warn when a page table blocks a huge pfn insert mm/huge_memory: use the large-page PAT encoding for huge pfn mappings

 arch/x86/include/asm/pgtable_types.h |  2 ++
 include/linux/pgtable.h              |  9 +++++++++
 mm/huge_memory.c                     | 20 ++++++++++++++++++--
 mm/memory.c                          | 19 ++++++++++++++++---
 4 files changed, 45 insertions(+), 5 deletions(-)
Signature · 1 line
2.47.3
#1 lore raw

PATCH v2 1/4 mm: drop the huge PMD assertion from walk_to_pmd()

walk_to_pmd() asserts that the PMD it returns is not a huge entry. That held when commit c2febafc6773 ("mm: convert generic code to 5-level paging") added the assertion: nothing installed a huge PMD over the mappings its callers walk.

ARCH_SUPPORTS_HUGE_PFNMAP changed that. A VM_PFNMAP mapping whose driver provides both ->fault and ->huge_fault can have a PMD leaf installed by a concurrent huge fault over the same address, so a caller can observe one and BUG on CONFIG_DEBUG_VM=y.

The walk does not need the invariant. pmd_alloc() only allocates when the PUD is empty and otherwise returns pmd_offset(), so a leaf is returned rather than overwritten. Every caller already copes with one, because they all reach pte_offset_map_lock(), which refuses to walk a leaf and returns NULL: insert_pages() turns that into -EFAULT, get_locked_pte() passes the NULL to its callers, and insert_pfn() returns a fault code.

Fixes f9e54c3a2f5b ("vfio/pci: implement huge_fault support")
---
 mm/memory.c | 1 -
 1 file changed, 1 deletion(-)
mm/memory.c +0 -1
index ec3d9d99ccee..0f54ee0a6d88 100644
@@ -2384,7 +2384,6 @@ static pmd_t *walk_to_pmd(struct mm_struct *mm, unsigned long addr)
 	if (!pmd)
 		return NULL;
 
-	VM_BUG_ON(pmd_trans_huge(*pmd));
 	return pmd;
 }
 
Signature · 1 line
2.47.3
#2 lore raw

PATCH v2 2/4 mm: don't report OOM when a huge PMD race blocks a pfn insert

In reply to Eliav Farber

insert_pfn() maps the PTE with get_locked_pte() and treats a NULL return as an allocation failure. pte_offset_map_lock() also returns NULL when the PMD is a leaf, which a VM_PFNMAP mapping with both ->fault and ->huge_fault can hit: an order-0 fault that passed handle_pte_fault()'s pmd_none() check can reach insert_pfn() after a concurrent huge fault installed a PMD leaf over the same address. The task then takes a VM_FAULT_OOM for a mapping that is perfectly healthy. Changing an mm's THP policy while another thread faults the same PMD, for example with prctl(PR_SET_THP_DISABLE), is enough to produce the mixed orders.

get_locked_pte() cannot tell the two apart, because pte_alloc_map_lock() folds the allocation and the map into one call. Walk to the PMD, allocate, and map as three steps, so that only a failed allocation returns VM_FAULT_OOM. A NULL from the map returns VM_FAULT_NOPAGE and the access refaults, which is what a PMD leaf, a PMD that is not present and a bad PMD that __pte_offset_map() cleared all want.

Fixes f9e54c3a2f5b ("vfio/pci: implement huge_fault support")

---
v2: Split pte_alloc_map_lock() into pte_alloc() followed by
pte_offset_map_lock(), so that only a failed allocation returns VM_FAULT_OOM and a NULL from the map returns VM_FAULT_NOPAGE. v1 called pte_alloc_map_lock() and then re-read the PMD with pmd_trans_huge() to tell the two apart, which left the non-present and bad-PMD cases reported as OOM. Suggested by Jose A. Perez de Azpillaga.

 mm/memory.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)
mm/memory.c +16 -2
index 0f54ee0a6d88..05ae87518206 100644
@@ -2866,10 +2866,24 @@ static vm_fault_t insert_pfn(struct vm_area_struct *vma, unsigned long addr,
 	struct mm_struct *mm = vma->vm_mm;
 	pte_t *pte, entry;
 	spinlock_t *ptl;
+	pmd_t *pmd;
 
-	pte = get_locked_pte(mm, addr, &ptl);
-	if (!pte)
+	pmd = walk_to_pmd(mm, addr);
+	if (!pmd)
+		return VM_FAULT_OOM;
+
+	if (pte_alloc(mm, pmd))
 		return VM_FAULT_OOM;
+
+	/*
+	 * A concurrent ->huge_fault may have installed a PMD leaf over addr,
+	 * which pte_offset_map_lock() refuses to walk. The mapping is fine, so
+	 * let the access refault instead of reporting a failed allocation.
+	 */
+	pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
+	if (!pte)
+		return VM_FAULT_NOPAGE;
+
 	entry = ptep_get(pte);
 	if (!pte_none(entry)) {
 		if (mkwrite) {
Signature · 1 line
2.47.3
Reviewed-by on v2 2/4
#3 lore raw

Re: [PATCH v2 2/4] mm: don't report OOM when a huge PMD race blocks a pfn insert

Quoted 17 lines · Eliav Farber

insert_pfn() maps the PTE with get_locked_pte() and treats a NULL return as an allocation failure. pte_offset_map_lock() also returns NULL when the PMD is a leaf, which a VM_PFNMAP mapping with both ->fault and ->huge_fault can hit: an order-0 fault that passed handle_pte_fault()'s pmd_none() check can reach insert_pfn() after a concurrent huge fault installed a PMD leaf over the same address. The task then takes a VM_FAULT_OOM for a mapping that is perfectly healthy. Changing an mm's THP policy while another thread faults the same PMD, for example with prctl(PR_SET_THP_DISABLE), is enough to produce the mixed orders.

get_locked_pte() cannot tell the two apart, because pte_alloc_map_lock() folds the allocation and the map into one call. Walk to the PMD, allocate, and map as three steps, so that only a failed allocation returns VM_FAULT_OOM. A NULL from the map returns VM_FAULT_NOPAGE and the access refaults, which is what a PMD leaf, a PMD that is not present and a bad PMD that __pte_offset_map() cleared all want.

Quoted 3 lines
Fixes f9e54c3a2f5b ("vfio/pci: implement huge_fault support")

LGTM:

Signature · 2 lines
cheers,
jose a. p-a
#4 lore raw

PATCH v2 3/4 mm/huge_memory: don't warn when a page table blocks a huge pfn insert

In reply to Eliav Farber

insert_pmd() and insert_pud() treat a non-empty entry at their own level as a leaf and compare its PFN against the one being inserted. A page table installed by a concurrent order-0 fault also makes the entry non-empty, and pmd_pfn()/pud_pfn() then report the PFN of that table, which never matches. insert_pmd() warns unless the entry is the huge zero PMD, and insert_pud() warns on any mismatch.

A VM_PFNMAP mapping with both ->fault and ->huge_fault can produce the mixed orders: an order-0 fault installs a page table over an address while a concurrent huge fault on the same address reaches insert_pmd(). Changing an mm's THP policy while another thread faults the same PMD, for example with prctl(PR_SET_THP_DISABLE), is enough to do it.

Compare the PFN only for a leaf entry. A page table is not an error, so take the existing out_unlock path, which returns VM_FAULT_NOPAGE and lets the access refault onto the smaller entry.

Fixes f9e54c3a2f5b ("vfio/pci: implement huge_fault support")
---
 mm/huge_memory.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
mm/huge_memory.c +16 -0
index eb913acb03b7..052c50dadc48 100644
@@ -1643,6 +1643,14 @@ static vm_fault_t insert_pmd(struct vm_area_struct *vma, unsigned long addr,
 		const unsigned long pfn = fop.is_folio ? folio_pfn(fop.folio) :
 					  fop.pfn;
 
+		/*
+		 * A concurrent order-0 fault may have installed a PTE table
+		 * here, which pmd_pfn() would report as the PFN of the table
+		 * itself. Only a leaf entry can be compared.
+		 */
+		if (!pmd_leaf(*pmd))
+			goto out_unlock;
+
 		if (write) {
 			if (pmd_pfn(*pmd) != pfn) {
 				WARN_ON_ONCE(!is_huge_zero_pmd(*pmd));
@@ -1770,6 +1778,14 @@ static vm_fault_t insert_pud(struct vm_area_struct *vma, unsigned long addr,
 		const unsigned long pfn = fop.is_folio ? folio_pfn(fop.folio) :
 					  fop.pfn;
 
+		/*
+		 * A concurrent order-0 fault may have installed a PMD table
+		 * here, which pud_pfn() would report as the PFN of the table
+		 * itself. Only a leaf entry can be compared.
+		 */
+		if (!pud_leaf(*pud))
+			goto out_unlock;
+
 		if (write) {
 			if (WARN_ON_ONCE(pud_pfn(*pud) != pfn))
 				goto out_unlock;
Signature · 1 line
2.47.3
maintainer
#5 lore raw

Re: [PATCH v2 3/4] mm/huge_memory: don't warn when a page table blocks a huge pfn insert

Quoted 13 lines · On Thu, 12 Nov 2026 05:14:39 +0000 Eliav Farber <[email protected]> wrote:

insert_pmd() and insert_pud() treat a non-empty entry at their own level as a leaf and compare its PFN against the one being inserted. A page table installed by a concurrent order-0 fault also makes the entry non-empty, and pmd_pfn()/pud_pfn() then report the PFN of that table, which never matches. insert_pmd() warns unless the entry is the huge zero PMD, and insert_pud() warns on any mismatch.

A VM_PFNMAP mapping with both ->fault and ->huge_fault can produce the mixed orders: an order-0 fault installs a page table over an address while a concurrent huge fault on the same address reaches insert_pmd(). Changing an mm's THP policy while another thread faults the same PMD, for example with prctl(PR_SET_THP_DISABLE), is enough to do it.

Quoted 3 lines

Compare the PFN only for a leaf entry. A page table is not an error, so take the existing out_unlock path, which returns VM_FAULT_NOPAGE and lets the access refault onto the smaller entry.

Thanks. Sashiko says this fixes a cant-happen.

	https://sashiko.dev/#/patchset/[email protected]
Quoted 1 line
Fixes f9e54c3a2f5b ("vfio/pci: implement huge_fault support")

But that's because Sashiko only considered f9e54c3a2f5b.

It seems that DRM shmem and Panthor and mshv pass FAULT_FLAG_WRITE and can reach the warnings.

So... patch looks good but the Fixes: might be inaccurate.

Fixes c6f3c5ee40c1 ("mm/huge_memory.c: fix modifying of page protection by insert_pfn_pmd()")

might be better.

Also, we don't want current production kernels to do this so please consider adding an explicit cc:stable to the patch.

#6 lore raw

Re: [PATCH v2 3/4] mm/huge_memory: don't warn when a page table blocks a huge pfn insert

In reply to Eliav Farber
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] mm/huge_memory: inaccurate Fixes trailer for insert_pmd() and insert_pud() --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] mm/huge_memory: inaccurate Fixes trailer for insert_pmd() and insert_pud()
--

commit b401d2d30ea8d00390aed1659aa84d49600aa4d3
Author: Eliav Farber <[email protected]>

Quoted 15 lines

mm/huge_memory: don't warn when a page table blocks a huge pfn insert

insert_pmd() and insert_pud() treat a non-empty entry at their own level as a leaf and compare its PFN against the one being inserted. A page table installed by a concurrent order-0 fault also makes the entry non-empty, and pmd_pfn()/pud_pfn() then report the PFN of that table, which never matches. insert_pmd() warns unless the entry is the huge zero PMD, and insert_pud() warns on any mismatch.

A VM_PFNMAP mapping with both ->fault and ->huge_fault can produce the mixed orders: an order-0 fault installs a page table over an address while a concurrent huge fault on the same address reaches insert_pmd(). Changing an mm's THP policy while another thread faults the same PMD, for example with prctl(PR_SET_THP_DISABLE), is enough to do it.

Quoted 7 lines

Compare the PFN only for a leaf entry. A page table is not an error, so take the existing out_unlock path, which returns VM_FAULT_NOPAGE and lets the access refault onto the smaller entry.

Fixes f9e54c3a2f5b ("vfio/pci: implement huge_fault support")

[Severity: Low]
In insert_pmd() and insert_pud(), the pmd_pfn() and pud_pfn() comparisons and their WARN_ON_ONCE() calls only run when write is true. Commit f9e54c3a2f5b ("vfio/pci: implement huge_fault support") always passes write = false to vmf_insert_pfn_pmd() and vmf_insert_pfn_pud(), so it cannot reach this warning.

Should the Fixes: tag reference c6f3c5ee40c1 ("mm/huge_memory.c: fix modifying of page protection by insert_pfn_pmd()"), which added these checks to mm/huge_memory.c?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#7 lore raw

Re: [PATCH v2 3/4] mm/huge_memory: don't warn when a page table blocks a huge pfn insert

In reply to Eliav Farber

TL;DR is I feel like maybe I need to take this over as I'm not confident you understand this...

Quoted 1 line · Eliav Farber

insert_pmd() and insert_pud() treat a non-empty entry at their own level

Non-empty reads quite strangely. I'd say present but non-present != !pmd_none().

And the code is just... so terrible.

It seems to assume that softleaves don't exist so don't bother to care about that, for one.

I'm not sure if this fix is sufficient, actually.

Quoted 7 lines

as a leaf and compare its PFN against the one being inserted. A page table installed by a concurrent order-0 fault also makes the entry non-empty, and pmd_pfn()/pud_pfn() then report the PFN of that table, which never matches. insert_pmd() warns unless the entry is the huge zero PMD, and insert_pud() warns on any mismatch.

A VM_PFNMAP mapping with both ->fault and ->huge_fault can produce the

Please refer to non-deprecated VMA flags in new code.

Quoted 1 line

mixed orders: an order-0 fault installs a page table over an address

'the mixed orders'? What does that mean?

Quoted 3 lines

while a concurrent huge fault on the same address reaches insert_pmd(). Changing an mm's THP policy while another thread faults the same PMD, for example with prctl(PR_SET_THP_DISABLE), is enough to do it.

OK that's useful information. Terrible.

I wonder if actually there are other bugs that will surface because this takes the mmap write lock but concurrent faults are permitted.

Maybe we need to make PR_SET_THP_DISABLE take every single VMA's write lock in the mm first...

Even that is dubious as new VMAs can come into existence while you do that :))

Ah well. I guess correct thing is to handle races like this as best we can.

Quoted 4 lines

Compare the PFN only for a leaf entry. A page table is not an error, so take the existing out_unlock path, which returns VM_FAULT_NOPAGE and lets the access refault onto the smaller entry.

Sorry this paragraph is just word salad LLM-ish code-as-prose, delete it.

This commit message is really poor - it's too long, and what it goes into detail about is mostly useless.

Pare it down into a form that demonstrates understanding of the page fault code. Make the actual explainy bit as small as possible.

And you haven't answered key questions:

- What real-world workload were you doing when you encountered this?

- Or was it code inspection? Or was it an LLM? Assisted-by might be gone but you are strongly encouraged to explain this even if below the --- in the patch...

- What are the actual runtime implications for users?

- Why did you decide to not do anything about the other flaws here?

- What parts of the kernel are actually vulnerable to this? E.g.
https://elixir.bootlin.com/linux/v7.3.1/A/ident/vmf_insert_pfn_pmd

Anyway, there's lots of broken stuff here though.

insert_pmd() seems to ignore the fact that PMD softleaves exist. Maybe that's not possible in this code path... but... I don't see how it's not?

So this just seems terribly broken overall.

Quoted 2 lines
Fixes f9e54c3a2f5b ("vfio/pci: implement huge_fault support")

A commit message that goes into excruciating detail describing the code I can literally scroll down and look at, but zero comment on why this fixes target was chosen.

Presumably the first user that made this possible? I guess? Or no?

Do I as a reviewer have to go figure it out?

Please try to make our lives easier :)

Quoted 11 lines
---
 mm/huge_memory.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
mm/huge_memory.c +0 -0
index eb913acb03b7..052c50dadc48 100644

mm/huge_memory.c static vm_fault_t insert_pmd(struct vm_area_struct *vma, unsigned long addr, · line 1650

Quoted 8 lines
mm/huge_memory.c +5 -0
@@ -1643,6 +1643,14 @@ static vm_fault_t insert_pmd(struct vm_area_struct *vma, unsigned long addr,
 		const unsigned long pfn = fop.is_folio ? folio_pfn(fop.folio) :
 					  fop.pfn;
 
+		/*
+		 * A concurrent order-0 fault may have installed a PTE table
+		 * here, which pmd_pfn() would report as the PFN of the table
+		 * itself. Only a leaf entry can be compared.
+		 */

I hate this comment so much.

It's what I see so often from LLM output too... you're going on to say what _would_ have happened if you _didn't_ do what you just did on the _assumption_ that reader has zero understanding of page tables and fault handling, and that the code (of course) will _never_ change.

It's adding nothing.

'Only a leaf entry can be compared' especially... it just reads like an annotation from somebody/something that doesn't understand page table/fault handling code.

Anyway delete both please.

mm/huge_memory.c static vm_fault_t insert_pmd(struct vm_area_struct *vma, unsigned long addr, · line 1652

Quoted 2 lines
mm/huge_memory.c +2 -0
+		if (!pmd_leaf(*pmd))
+			goto out_unlock;

* @VM_FAULT_NOPAGE: ->fault installed the pte, not return page

Is this still the correct return value?

I imagine it's effectively saying 'this is not what we expected ah well carry on'.

But what if it's a softleaf... Presumably it's somehow not possible for it to be a PMD softleaf entry... but ugh.

The assumption here in the code is so bogus though. One can experience a concurrent huge fault no?

So:

			if (pmd_pfn(*pmd) != pfn) {
				WARN_ON_ONCE(!is_huge_zero_pmd(*pmd));
				goto out_unlock;
			}

Is still completely broken isn't it? You could well find a leaf, installed by a concurrent huge fault handler, and trigger a warning couldn't you?

Or is that somehow prevented?

I also don't get why an existing unexpected entry is totally fine if !write let's not check anything...

It feels like the real fix is something more substantial.

Maybe better for me to take this over...

Quoted 20 lines
mm/huge_memory.c +9 -1
+
 		if (write) {
 			if (pmd_pfn(*pmd) != pfn) {
 				WARN_ON_ONCE(!is_huge_zero_pmd(*pmd));
@@ -1770,6 +1778,14 @@ static vm_fault_t insert_pud(struct vm_area_struct *vma, unsigned long addr,
 		const unsigned long pfn = fop.is_folio ? folio_pfn(fop.folio) :
 					  fop.pfn;
 
+		/*
+		 * A concurrent order-0 fault may have installed a PMD table
+		 * here, which pud_pfn() would report as the PFN of the table
+		 * itself. Only a leaf entry can be compared.
+		 */
+		if (!pud_leaf(*pud))
+			goto out_unlock;
+
 		if (write) {
 			if (WARN_ON_ONCE(pud_pfn(*pud) != pfn))
 				goto out_unlock;
--
Quoted 2 lines

2.47.3

--
Cheers, Lorenzo

#8 lore raw

Re: [PATCH v2 3/4] mm/huge_memory: don't warn when a page table blocks a huge pfn insert

Quoted 7 lines

I also don't get why an existing unexpected entry is totally fine if !write let's not check anything...

It feels like the real fix is something more substantial.

Maybe better for me to take this over...

Please take over the entire series.

Signature · 3 lines
Cheers,

David
#9 lore raw

Re: [PATCH v2 3/4] mm/huge_memory: don't warn when a page table blocks a huge pfn insert

Quoted 9 lines · David Hildenbrand (Arm)
Quoted 7 lines

I also don't get why an existing unexpected entry is totally fine if !write let's not check anything...

It feels like the real fix is something more substantial.

Maybe better for me to take this over...

Please take over the entire series.

Ack, will do. It's clear there's no understanding of any of the code here.

--
Cheers, Lorenzo

#10 lore raw

PATCH v2 4/4 mm/huge_memory: use the large-page PAT encoding for huge pfn mappings

In reply to Eliav Farber

insert_pmd() and insert_pud() build the entry from the pgprot that vmf_insert_pfn_pmd()/_pud() took from vma->vm_page_prot and passed through pfnmap_setup_cachemode_pfn(). That pgprot carries the PTE-level cache mode encoding, but x86 keeps the high PAT bit at bit 7 for a PTE (_PAGE_BIT_PAT) and at bit 12 for a PMD or PUD (_PAGE_BIT_PAT_LARGE), where bit 7 is _PAGE_PSE.

A memtype that needs that bit, write-through or write-protect, loses it:
the cache mode index of the huge entry drops from 4-7 to 0-3. With the PAT slots Linux programs, write-through (slot 7) becomes UC (slot 3), and write-protect (slot 5) becomes write-combining (slot 1), which loses the write protection. The other x86 paths that build a large entry from a PTE pgprot convert it first, see populate_pmd() and populate_pud().

Convert in the two huge pfn paths. Add a generic pgprot_4k_2_large() that returns its argument unchanged, so that only architectures which encode the levels differently have to define it, and x86's existing definition is picked up.

The folio paths keep using vma->vm_page_prot as is. They map struct-page-backed memory, which x86 maps write-back.

Fixes 5cad465d7fa6 ("mm: add vmf_insert_pfn_pmd()")

---
v2: Corrected the second paragraph of the commit message. It said a
write-through range ends up write-back; with the PAT slots Linux programs, write-through (slot 7) ends up UC (slot 3) and write-protect (slot 5) ends up write-combining (slot 1). Changed Fixes: from 6857be5fecae to 5cad465d7fa6 ("mm: add vmf_insert_pfn_pmd()"), which is the commit that introduced the unconverted pgprot. Both pointed out by Jose A. Perez de Azpillaga. No code change.

 arch/x86/include/asm/pgtable_types.h | 2 ++
 include/linux/pgtable.h              | 9 +++++++++
 mm/huge_memory.c                     | 4 ++--
 3 files changed, 13 insertions(+), 2 deletions(-)
arch/x86/include/asm/pgtable_types.h +2 -0
index b1a669e775f4..b1ffd72e47ef 100644
@@ -497,6 +497,8 @@ static inline pgprotval_t protval_4k_2_large(pgprotval_t val)
 	return (val & ~(_PAGE_PAT | _PAGE_PAT_LARGE)) |
 		((val & _PAGE_PAT) << (_PAGE_BIT_PAT_LARGE - _PAGE_BIT_PAT));
 }
+
+#define pgprot_4k_2_large pgprot_4k_2_large
 static inline pgprot_t pgprot_4k_2_large(pgprot_t pgprot)
 {
 	return __pgprot(protval_4k_2_large(pgprot_val(pgprot)));
include/linux/pgtable.h +9 -0
index 780fe849ff8b..112814d31f0c 100644
@@ -1787,6 +1787,15 @@ void arch_sync_kernel_mappings(unsigned long start, unsigned long end);
 #define pgprot_mhp(prot)	(prot)
 #endif
 
+/*
+ * Convert a pgprot built for a PTE so that it can be used in a PMD or PUD
+ * entry. Only needed on architectures that encode attributes differently
+ * between the levels; elsewhere the pgprot is used as is.
+ */
+#ifndef pgprot_4k_2_large
+#define pgprot_4k_2_large(prot)	(prot)
+#endif
+
 #ifdef CONFIG_MMU
 #ifndef pgprot_modify
 #define pgprot_modify pgprot_modify
mm/huge_memory.c +2 -2
index 052c50dadc48..945d480c1512 100644
@@ -1675,7 +1675,7 @@ static vm_fault_t insert_pmd(struct vm_area_struct *vma, unsigned long addr,
 			add_mm_counter(mm, mm_counter_file(fop.folio), HPAGE_PMD_NR);
 		}
 	} else {
-		entry = pmd_mkhuge(pfn_pmd(fop.pfn, prot));
+		entry = pmd_mkhuge(pfn_pmd(fop.pfn, pgprot_4k_2_large(prot)));
 		entry = pmd_mkspecial(entry);
 	}
 	if (write) {
@@ -1804,7 +1804,7 @@ static vm_fault_t insert_pud(struct vm_area_struct *vma, unsigned long addr,
 		folio_add_file_rmap_pud(fop.folio, &fop.folio->page, vma);
 		add_mm_counter(mm, mm_counter_file(fop.folio), HPAGE_PUD_NR);
 	} else {
-		entry = pud_mkhuge(pfn_pud(fop.pfn, prot));
+		entry = pud_mkhuge(pfn_pud(fop.pfn, pgprot_4k_2_large(prot)));
 		entry = pud_mkspecial(entry);
 	}
 	if (write) {
Signature · 1 line
2.47.3
Reviewed-by on v2 4/4
#11 lore raw

Re: [PATCH v2 4/4] mm/huge_memory: use the large-page PAT encoding for huge pfn mappings

Quoted 19 lines · Eliav Farber

insert_pmd() and insert_pud() build the entry from the pgprot that vmf_insert_pfn_pmd()/_pud() took from vma->vm_page_prot and passed through pfnmap_setup_cachemode_pfn(). That pgprot carries the PTE-level cache mode encoding, but x86 keeps the high PAT bit at bit 7 for a PTE (_PAGE_BIT_PAT) and at bit 12 for a PMD or PUD (_PAGE_BIT_PAT_LARGE), where bit 7 is _PAGE_PSE.

A memtype that needs that bit, write-through or write-protect, loses it:
the cache mode index of the huge entry drops from 4-7 to 0-3. With the PAT slots Linux programs, write-through (slot 7) becomes UC (slot 3), and write-protect (slot 5) becomes write-combining (slot 1), which loses the write protection. The other x86 paths that build a large entry from a PTE pgprot convert it first, see populate_pmd() and populate_pud().

Convert in the two huge pfn paths. Add a generic pgprot_4k_2_large() that returns its argument unchanged, so that only architectures which encode the levels differently have to define it, and x86's existing definition is picked up.

Quoted 5 lines

The folio paths keep using vma->vm_page_prot as is. They map struct-page-backed memory, which x86 maps write-back.

Fixes 5cad465d7fa6 ("mm: add vmf_insert_pfn_pmd()")

LGTM:

Signature · 2 lines
cheers,
jose a. p-a
#12 lore raw

Re: [PATCH v2 4/4] mm/huge_memory: use the large-page PAT encoding for huge pfn mappings

In reply to Eliav Farber
Quoted 77 lines · Eliav Farber

insert_pmd() and insert_pud() build the entry from the pgprot that vmf_insert_pfn_pmd()/_pud() took from vma->vm_page_prot and passed through pfnmap_setup_cachemode_pfn(). That pgprot carries the PTE-level cache mode encoding, but x86 keeps the high PAT bit at bit 7 for a PTE (_PAGE_BIT_PAT) and at bit 12 for a PMD or PUD (_PAGE_BIT_PAT_LARGE), where bit 7 is _PAGE_PSE.

A memtype that needs that bit, write-through or write-protect, loses it:
the cache mode index of the huge entry drops from 4-7 to 0-3. With the PAT slots Linux programs, write-through (slot 7) becomes UC (slot 3), and write-protect (slot 5) becomes write-combining (slot 1), which loses the write protection. The other x86 paths that build a large entry from a PTE pgprot convert it first, see populate_pmd() and populate_pud().

Convert in the two huge pfn paths. Add a generic pgprot_4k_2_large() that returns its argument unchanged, so that only architectures which encode the levels differently have to define it, and x86's existing definition is picked up.

The folio paths keep using vma->vm_page_prot as is. They map struct-page-backed memory, which x86 maps write-back.

Fixes 5cad465d7fa6 ("mm: add vmf_insert_pfn_pmd()")

---
v2: Corrected the second paragraph of the commit message. It said a
write-through range ends up write-back; with the PAT slots Linux programs, write-through (slot 7) ends up UC (slot 3) and write-protect (slot 5) ends up write-combining (slot 1). Changed Fixes: from 6857be5fecae to 5cad465d7fa6 ("mm: add vmf_insert_pfn_pmd()"), which is the commit that introduced the unconverted pgprot. Both pointed out by Jose A. Perez de Azpillaga. No code change.

 arch/x86/include/asm/pgtable_types.h | 2 ++
 include/linux/pgtable.h              | 9 +++++++++
 mm/huge_memory.c                     | 4 ++--
 3 files changed, 13 insertions(+), 2 deletions(-)
arch/x86/include/asm/pgtable_types.h +2 -0
index b1a669e775f4..b1ffd72e47ef 100644
@@ -497,6 +497,8 @@ static inline pgprotval_t protval_4k_2_large(pgprotval_t val)
 	return (val & ~(_PAGE_PAT | _PAGE_PAT_LARGE)) |
 		((val & _PAGE_PAT) << (_PAGE_BIT_PAT_LARGE - _PAGE_BIT_PAT));
 }
+
+#define pgprot_4k_2_large pgprot_4k_2_large
 static inline pgprot_t pgprot_4k_2_large(pgprot_t pgprot)
 {
 	return __pgprot(protval_4k_2_large(pgprot_val(pgprot)));
include/linux/pgtable.h +9 -0
index 780fe849ff8b..112814d31f0c 100644
@@ -1787,6 +1787,15 @@ void arch_sync_kernel_mappings(unsigned long start, unsigned long end);
 #define pgprot_mhp(prot)	(prot)
 #endif
 
+/*
+ * Convert a pgprot built for a PTE so that it can be used in a PMD or PUD
+ * entry. Only needed on architectures that encode attributes differently
+ * between the levels; elsewhere the pgprot is used as is.
+ */
+#ifndef pgprot_4k_2_large
+#define pgprot_4k_2_large(prot)	(prot)
+#endif
+
 #ifdef CONFIG_MMU
 #ifndef pgprot_modify
 #define pgprot_modify pgprot_modify
mm/huge_memory.c +0 -0
index 052c50dadc48..945d480c1512 100644

mm/huge_memory.c static vm_fault_t insert_pmd(struct vm_area_struct *vma, unsigned long addr, · line 1678

Quoted 5 lines
mm/huge_memory.c +1 -1
@@ -1675,7 +1675,7 @@ static vm_fault_t insert_pmd(struct vm_area_struct *vma, unsigned long addr,
 			add_mm_counter(mm, mm_counter_file(fop.folio), HPAGE_PMD_NR);
 		}
 	} else {
-		entry = pmd_mkhuge(pfn_pmd(fop.pfn, prot));
+		entry = pmd_mkhuge(pfn_pmd(fop.pfn, pgprot_4k_2_large(prot)));

What on earth is 4k terminology doing in common code?! :)

Signature · 3 lines
Cheers,

David
#13 lore raw
In reply to Eliav Farber

Hi Eliav,

Thanks for the report.

As discussed inline, I'm not confident that you have a deep enough understanding of the relevant code, so I will take over the series.

--
Cheers, Lorenzo