mm: add huge pfnmap support for remap_pfn_range()
Subsystems (6)
v3 cover mm: add huge pfnmap support for remap_pfn_range()
v3:
1. Architectural Type Safety (Matthew Wilcox):
Following the insightful architectural feedback from Matthew Wilcox in v2, the approach to clearing huge page attributes has been completely redesigned. Instead of spreading the pte_clrhuge() anti-pattern to ARM64 and RISC-V, this series enforces strict type safety at the lowest level: pfn_pte() must never natively return a PTE with huge page attributes set.
To achieve this without breaking the x86 core MM, the series is structured as:
- Fix historical type-casting abuses in x86 (vmemmap, vmalloc, CPA) where pfn_pte() was wrongly used to generate huge PMDs/PUDs.
- Update pfn_pte() on x86 and ARM64 to inherently filter out huge page attributes. (RISC-V leaf PMDs and PTEs share the exact same hardware format without a specific "huge" bit, so it is naturally compliant).
- Completely eradicate pte_clrhuge() from the x86 tree and clean up
the type-casting mess in `arch/x86/mm/init_64.c`.
2. Page Table Deposit fix during clone() (syzbot):
Previously, copy_huge_pmd() was unaware of special PMDs created by pfnmap, failing to deposit a page table for the child process during clone(). This led to crashes during process teardown or PMD splitting. The logic is now updated to properly allocate and deposit pgtables for pmd_special() entries.
v2: https://lore.kernel.org/linux-mm/[email protected]/#t
- remove "nohugepfnmap" boot option and "pfnmap_max_page_shift" variable.
- zap_deposited_table for non-special pmd.
- move set_pmd_at() inside pmd_lock.
- prevent PMD mapping creation when pgtable allocation fails.
- defer the refactor of pte_clrhuge() to a separate patch series. For now, add a TODO to track this.
v1: https://lore.kernel.org/linux-mm/[email protected]/
Overview
========
This patch series adds huge page support for remap_pfn_range(), automatically creating huge mappings when prerequisites are satisfied (size, alignment, architecture support, etc.) and falling back to normal page mappings otherwise.
This work builds on Peter Xu's previous efforts on huge pfnmap support [0].
TODO
====
- Add PUD-level huge page support. Currently, only PMD-level huge pages are supported.
Tests Done
==========
- Cross-build tests.
- Core MM Regression Tests
- Booted x86 kernel with debug_pagealloc=on to heavily stress the large page splitting logic in direct mapping. No panics observed.
- Ran make -C tools/testing/selftests/vm run_tests. Both THP and Hugetlbfs tests passed successfully, proving the pfn_pte() changes do not interfere with native huge page generation.
- Functional Tests (with a custom device driver & PTDUMP):
- Verified that remap_pfn_range() successfully creates 2MB mappings by observing /sys/kernel/debug/page_tables/current_user.
- Triggered PMD splits via 4K-granular mprotect() and partial munmap(), verifying correct fallback to 512 PTEs without corrupting permissions or causing kernel crashes.
- Triggered fork()/clone() on the mapped regions, validating the syzbot fix and ensuring safe pgtable deposit/withdraw lifecycle.
- Performance tests with custom device driver implementing mmap() with remap_pfn_range():
- lat_mem_rd benchmark modified to use mmap(device_fd) instead of
malloc() shows around 40% improvement in memory access latency with
huge page support compared to normal page mappings.
numactl -C 0 lat_mem_rd -t 4096M (stride=64)
Memory Size (MB) Without Huge Mapping With Huge Mapping Improvement
---------------- ----------------- -------------- -----------
64.00 148.858 ns 100.780 ns 32.3%
128.00 164.745 ns 103.537 ns 37.2%
256.00 169.907 ns 103.179 ns 39.3%
512.00 171.285 ns 103.072 ns 39.8%
1024.00 173.054 ns 103.055 ns 40.4%
2048.00 172.820 ns 103.091 ns 40.3%
4096.00 172.877 ns 103.115 ns 40.4%
- Custom memory copy operations on mmap(device_fd) show around 18% performance
improvement with huge page support compared to normal page mappings.
numactl -C 0 memcpy_test (memory copy performance test)
Memory Size (MB) Without Huge Mapping With Huge Mapping Improvement
---------------- ----------------- -------------- -----------
1024.00 95.76 ms 77.91 ms 18.6%
2048.00 190.87 ms 155.64 ms 18.5%
4096.00 380.84 ms 311.45 ms 18.2%[0] https://lore.kernel.org/all/[email protected]/T/#u
Yin Tirui (4):
x86/mm: Use proper page table helpers for huge page generation mm/pgtable: Make pfn_pte() filter out huge page attributes x86/mm: Remove pte_clrhuge() and clean up init_64.c
mm: add PMD-level huge page support for remap_pfn_range()
arch/arm64/include/asm/pgtable.h | 4 +++- arch/x86/include/asm/pgtable.h | 9 ++++--- arch/x86/mm/init_64.c | 10 ++++---- arch/x86/mm/pat/set_memory.c | 6 ++++- arch/x86/mm/pgtable.c | 4 ++-- mm/huge_memory.c | 36 ++++++++++++++++++++++++++-- mm/memory.c | 40 ++++++++++++++++++++++++++++++++ 7 files changed, 93 insertions(+), 16 deletions(-)
Signature · 1 line
2.22.0
RFC PATCH v3 1/4 x86/mm: Use proper page table helpers for huge page generation
Historically, several core x86 mm subsystems (vmemmap, vmalloc, and CPA) have abused pfn_pte() to generate PMD and PUD entries by passing pgprot values containing the _PAGE_PSE flag, and then casting the resulting pte_t to a pmd_t or pud_t.
This violates strict type safety and prevents us from enforcing the rule that pfn_pte() should strictly generate pte without huge page attributes.
Fix these abuses by explicitly using the correct level-specific helpers (pfn_pmd() and pfn_pud()) and their corresponding setters (set_pmd(), set_pud()).
For the CPA (Change Page Attribute) code, which uses pte_t as a generic container for page table entries across all levels in __should_split_large_page(), pack the correctly generated PMD/PUD values into the pte_t container.
This cleanup prepares the ground for making pfn_pte() strictly filter out huge page attributes.
--- arch/x86/mm/init_64.c | 6 +++--- arch/x86/mm/pat/set_memory.c | 6 +++++- arch/x86/mm/pgtable.c | 4 ++-- 3 files changed, 10 insertions(+), 6 deletions(-)
arch/x86/mm/init_64.c +3 -3
@@ -1518,11 +1518,11 @@ static int __meminitdata node_start;void __meminit vmemmap_set_pmd (pmd_t *pmd,void *p,int node,unsigned long addr,unsigned long next) { -pte_t entry; +pmd_t entry; - entry =pfn_pte (__pa (p) >>PAGE_SHIFT , + entry =pfn_pmd (__pa (p) >>PAGE_SHIFT ,PAGE_KERNEL_LARGE ); -set_pmd (pmd,__pmd (pte_val (entry))); +set_pmd (pmd, entry);/* check to see if we have contiguous blocks */ if (p_end != p || node_start != node) {
arch/x86/mm/pat/set_memory.c +5 -1
@@ -1059,7 +1059,11 @@ static int __should_split_large_page(pte_t *kpte, unsigned long address,return 1 ;/* All checks passed. Update the large page mapping. */ - new_pte =pfn_pte (old_pfn, new_prot); +if (level ==PG_LEVEL_2M ) + new_pte =__pte (pmd_val (pfn_pmd (old_pfn, new_prot))); +else + new_pte =__pte (pud_val (pfn_pud (old_pfn, new_prot))); +__set_pmd_pte (kpte, address, new_pte); cpa->flags |=CPA_FLUSHTLB ;cpa_inc_lp_preserved (level);
arch/x86/mm/pgtable.c +2 -2
@@ -644,7 +644,7 @@ int pud_set_huge(pud_t *pud, phys_addr_t addr, pgprot_t prot)if (pud_present (*pud) && !pud_leaf (*pud))return 0 ; -set_pte ((pte_t *)pud,pfn_pte ( +set_pud (pud,pfn_pud ( (u64 )addr >>PAGE_SHIFT ,__pgprot (protval_4k_2_large (pgprot_val (prot)) | _PAGE_PSE)));
@@ -676,7 +676,7 @@ int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot)if (pmd_present (*pmd) && !pmd_leaf (*pmd))return 0 ; -set_pte ((pte_t *)pmd,pfn_pte ( +set_pmd (pmd,pfn_pmd ( (u64 )addr >>PAGE_SHIFT ,__pgprot (protval_4k_2_large (pgprot_val (prot)) | _PAGE_PSE)));
Signature · 1 line
2.22.0
Re: [PATCH RFC v3 1/4] x86/mm: Use proper page table helpers for huge page generation
Quoted 17 lines · On Sat, 28 Feb 2026 15:09:03 +0800 Yin Tirui <[email protected]> wrote:
Historically, several core x86 mm subsystems (vmemmap, vmalloc, and CPA) have abused
pfn_pte()to generate PMD and PUD entries by passing pgprot values containing the_PAGE_PSEflag, and then casting the resultingpte_tto apmd_torpud_t.This violates strict type safety and prevents us from enforcing the rule that
pfn_pte()should strictly generate pte without huge page attributes.Fix these abuses by explicitly using the correct level-specific helpers (
pfn_pmd()andpfn_pud()) and their corresponding setters (set_pmd(),set_pud()).For the CPA (Change Page Attribute) code, which uses
pte_tas a generic container for page table entries across all levels in__should_split_large_page(), pack the correctly generated PMD/PUD values into thepte_tcontainer.
Quoted 4 lines
This cleanup prepares the ground for making
pfn_pte()strictly filter out huge page attributes.Signed-off-by Yin Tirui <[email protected]>
Hi. A tiny drive by review comment below.
Quoted 11 lines
--- arch/x86/mm/init_64.c | 6 +++--- arch/x86/mm/pat/set_memory.c | 6 +++++- arch/x86/mm/pgtable.c | 4 ++-- 3 files changed, 10 insertions(+), 6 deletions(-)arch/x86/mm/init_64.c +0
-0
arch/x86/mm/init_64.c static int __meminitdata node_start; · line 1524
Quoted 9 lines
arch/x86/mm/init_64.c +2
-2@@ -1518,11 +1518,11 @@ static int __meminitdata node_start;void __meminit vmemmap_set_pmd (pmd_t *pmd,void *p,int node,unsigned long addr,unsigned long next) { -pte_t entry; +pmd_t entry; - entry =pfn_pte (__pa (p) >>PAGE_SHIFT , + entry =pfn_pmd (__pa (p) >>PAGE_SHIFT ,PAGE_KERNEL_LARGE );
Whilst you are here, can we make that a one liner.
entry = pfn_pmd(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL_LARGE);
Could even do
pmd_t entry = pfn_pmd(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL_LARGE);
but that's more of a question of taste.
Quoted 4 lines
arch/x86/mm/init_64.c +1
-1-set_pmd (pmd,__pmd (pte_val (entry))); +set_pmd (pmd, entry);/* check to see if we have contiguous blocks */
Re: [PATCH RFC v3 1/4] x86/mm: Use proper page table helpers for huge page generation
Quoted 7 lines · Jonathan Cameron
Whilst you are here, can we make that a one liner.
entry = pfn_pmd(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL_LARGE);Could even do
pmd_t entry = pfn_pmd(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL_LARGE);but that's more of a question of taste.
Hi Jonathan,
Thanks for the suggestion, I will fold this one-liner cleanup into the next respin.
Signature · 1 line
Yin Tirui
RFC PATCH v3 2/4 mm/pgtable: Make pfn_pte() filter out huge page attributes
In reply to Yin TiruiA fundamental principle of page table type safety is that pte_t represents the lowest level page table entry and should never carry huge page attributes.
Currently, passing a pgprot with huge page bits (e.g., extracted via pmd_pgprot()) into pfn_pte() creates a malformed PTE that retains the huge attribute, leading to the necessity of the ugly pte_clrhuge() anti-pattern.
Enforce type safety by making pfn_pte() inherently filter out huge page attributes:
- On x86: Strip the _PAGE_PSE bit.
- On ARM64: Mask out the block descriptor bits in PTE_TYPE_MASK and enforce the PTE_TYPE_PAGE format.
- On RISC-V: No changes required, as RISC-V leaf PMDs and PTEs share the exact same hardware format and do not use a distinct huge bit.
--- arch/arm64/include/asm/pgtable.h | 4 +++- arch/x86/include/asm/pgtable.h | 4 ++++ 2 files changed, 7 insertions(+), 1 deletion(-)
arch/arm64/include/asm/pgtable.h +3 -1
@@ -141,7 +141,9 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys)#define pte_pfn (pte) (__pte_to_phys(pte) >> PAGE_SHIFT)#define pfn_pte (pfn,prot) \ - __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) + __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | \ + ((pgprot_val(prot) & ~(PTE_TYPE_MASK & ~PTE_VALID)) | \ + (PTE_TYPE_PAGE & ~PTE_VALID)))#define pte_none (pte) (!pte_val(pte))#define pte_page (pte) (pfn_to_page(pte_pfn(pte)))
arch/x86/include/asm/pgtable.h +4 -0
@@ -738,6 +738,10 @@ static inline pgprotval_t check_pgprot(pgprot_t pgprot)static inline pte_t pfn_pte (unsigned long page_nr,pgprot_t pgprot) {phys_addr_t pfn = (phys_addr_t )page_nr <<PAGE_SHIFT ; + +/* Filter out _PAGE_PSE to ensure PTEs never carry the huge page bit */ + pgprot =__pgprot (pgprot_val (pgprot) & ~_PAGE_PSE); +/* This bit combination is used to mark shadow stacks */ WARN_ON_ONCE ((pgprot_val (pgprot) & (_PAGE_DIRTY | _PAGE_RW)) == _PAGE_DIRTY);
Signature · 1 line
2.22.0
Jürgen Groß
NAKRe: [PATCH RFC v3 2/4] mm/pgtable: Make pfn_pte() filter out huge page attributes
Quoted 41 lines · Yin Tirui
A fundamental principle of page table type safety is that
pte_trepresents the lowest level page table entry and should never carry huge page attributes.Currently, passing a pgprot with huge page bits (e.g., extracted via
pmd_pgprot()) intopfn_pte()creates a malformed PTE that retains the huge attribute, leading to the necessity of the uglypte_clrhuge()anti-pattern.Enforce type safety by making
pfn_pte()inherently filter out huge page attributes:
- On x86: Strip the_PAGE_PSEbit.
- On ARM64: Mask out the block descriptor bits inPTE_TYPE_MASKand enforce thePTE_TYPE_PAGEformat.
- On RISC-V: No changes required, as RISC-V leaf PMDs and PTEs share the exact same hardware format and do not use a distinct huge bit.Signed-off-by Yin Tirui <[email protected]>--- arch/arm64/include/asm/pgtable.h | 4 +++- arch/x86/include/asm/pgtable.h | 4 ++++ 2 files changed, 7 insertions(+), 1 deletion(-)arch/arm64/include/asm/pgtable.h +3
-1@@ -141,7 +141,9 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys)#define pte_pfn (pte) (__pte_to_phys(pte) >> PAGE_SHIFT)#define pfn_pte (pfn,prot) \ - __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) + __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | \ + ((pgprot_val(prot) & ~(PTE_TYPE_MASK & ~PTE_VALID)) | \ + (PTE_TYPE_PAGE & ~PTE_VALID)))#define pte_none (pte) (!pte_val(pte))#define pte_page (pte) (pfn_to_page(pte_pfn(pte)))arch/x86/include/asm/pgtable.h +0
-0
arch/x86/include/asm/pgtable.h static inline pgprotval_t check_pgprot(pgprot_t pgprot) · line 743
Quoted 6 lines
arch/x86/include/asm/pgtable.h +3
-0@@ -738,6 +738,10 @@ static inline pgprotval_t check_pgprot(pgprot_t pgprot)static inline pte_t pfn_pte (unsigned long page_nr,pgprot_t pgprot) {phys_addr_t pfn = (phys_addr_t )page_nr <<PAGE_SHIFT ; + +/* Filter out _PAGE_PSE to ensure PTEs never carry the huge page bit */ + pgprot =__pgprot (pgprot_val (pgprot) & ~_PAGE_PSE);
Is it really a good idea to silently drop the bit?
Today it can either be used for a large page (which should be a pmd, of course), or - much worse - you'd strip the _PAGE_PAT bit, which is at the same position in PTEs.
So basically you are removing the ability to use some cache modes.
NACK!
Juergen
Re: [PATCH RFC v3 2/4] mm/pgtable: Make pfn_pte() filter out huge page attributes
Quoted 57 lines · Jürgen Groß
Quoted 47 lines · Yin Tirui
A fundamental principle of page table type safety is that
pte_trepresents the lowest level page table entry and should never carry huge page attributes.Currently, passing a pgprot with huge page bits (e.g., extracted via
pmd_pgprot()) intopfn_pte()creates a malformed PTE that retains the huge attribute, leading to the necessity of the uglypte_clrhuge()anti- pattern.Enforce type safety by making
pfn_pte()inherently filter out huge page attributes:
- On x86: Strip the_PAGE_PSEbit.
- On ARM64: Mask out the block descriptor bits inPTE_TYPE_MASKand enforce thePTE_TYPE_PAGEformat.
- On RISC-V: No changes required, as RISC-V leaf PMDs and PTEs share the exact same hardware format and do not use a distinct huge bit.Signed-off-by Yin Tirui <[email protected]>---
arch/arm64/include/asm/pgtable.h| 4 +++-
arch/x86/include/asm/pgtable.h | 4 ++++2 files changed, 7 insertions(+), 1 deletion(-)arch/arm64/include/asm/pgtable.h +3
-1@@ -141,7 +141,9 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) #define pte_pfn(pte) (__pte_to_phys(pte) >> PAGE_SHIFT) #define pfn_pte(pfn,prot) \ - __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) + __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | \ + ((pgprot_val(prot) & ~(PTE_TYPE_MASK & ~PTE_VALID)) | \ + (PTE_TYPE_PAGE & ~PTE_VALID))) #define pte_none(pte) (!pte_val(pte)) #define pte_page(pte) (pfn_to_page(pte_pfn(pte)))arch/x86/include/asm/pgtable.h +2
-0@@ -738,6 +738,10 @@ static inline pgprotval_t check_pgprot(pgprot_t pgprot) static inline pte_t pfn_pte(unsigned long page_nr, pgprot_t pgprot) { phys_addr_t pfn = (phys_addr_t)page_nr << PAGE_SHIFT; + + /* Filter out _PAGE_PSE to ensure PTEs never carry the huge pageQuoted 2 lines
bit */
+ pgprot =__pgprot(pgprot_val(pgprot) & ~_PAGE_PSE);Is it really a good idea to silently drop the bit?
Today it can either be used for a large page (which should be a pmd, of course), or - much worse - you'd strip the
_PAGE_PATbit, which is at the same position in PTEs.
Quoted 6 lines
So basically you are removing the ability to use some cache modes.
NACK!
Juergen
Hi Jürgen,
You are absolutely right. I missed the fact that _PAGE_PSE aliases with _PAGE_PAT on 4K PTEs.
The intention here was to follow previous feedback to enforce type safety by filtering out huge page attributes directly inside pfn_pte(). However, doing it this way obviously breaks the cache modes on x86.
I agree with the NACK. I will drop this approach and rethink how to handle the huge-to-normal pgprot conversion safely for v4.
Signature · 2 lines
Thanks, Yin Tirui
Re: [PATCH RFC v3 2/4] mm/pgtable: Make pfn_pte() filter out huge page attributes
In reply to Jürgen GroßQuoted 57 lines · Jürgen Groß
Quoted 47 lines · Yin Tirui
A fundamental principle of page table type safety is that
pte_trepresents the lowest level page table entry and should never carry huge page attributes.Currently, passing a pgprot with huge page bits (e.g., extracted via
pmd_pgprot()) intopfn_pte()creates a malformed PTE that retains the huge attribute, leading to the necessity of the uglypte_clrhuge()anti- pattern.Enforce type safety by making
pfn_pte()inherently filter out huge page attributes:
- On x86: Strip the_PAGE_PSEbit.
- On ARM64: Mask out the block descriptor bits inPTE_TYPE_MASKand enforce thePTE_TYPE_PAGEformat.
- On RISC-V: No changes required, as RISC-V leaf PMDs and PTEs share the exact same hardware format and do not use a distinct huge bit.Signed-off-by Yin Tirui <[email protected]>---
arch/arm64/include/asm/pgtable.h| 4 +++-
arch/x86/include/asm/pgtable.h | 4 ++++2 files changed, 7 insertions(+), 1 deletion(-)arch/arm64/include/asm/pgtable.h +3
-1@@ -141,7 +141,9 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) #define pte_pfn(pte) (__pte_to_phys(pte) >> PAGE_SHIFT) #define pfn_pte(pfn,prot) \ - __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) + __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | \ + ((pgprot_val(prot) & ~(PTE_TYPE_MASK & ~PTE_VALID)) | \ + (PTE_TYPE_PAGE & ~PTE_VALID))) #define pte_none(pte) (!pte_val(pte)) #define pte_page(pte) (pfn_to_page(pte_pfn(pte)))arch/x86/include/asm/pgtable.h +2
-0@@ -738,6 +738,10 @@ static inline pgprotval_t check_pgprot(pgprot_t pgprot) static inline pte_t pfn_pte(unsigned long page_nr, pgprot_t pgprot) { phys_addr_t pfn = (phys_addr_t)page_nr << PAGE_SHIFT; + + /* Filter out _PAGE_PSE to ensure PTEs never carry the huge pageQuoted 2 lines
bit */
+ pgprot =__pgprot(pgprot_val(pgprot) & ~_PAGE_PSE);Is it really a good idea to silently drop the bit?
Today it can either be used for a large page (which should be a pmd, of course), or - much worse - you'd strip the
_PAGE_PATbit, which is at the same position in PTEs.
Quoted 6 lines
So basically you are removing the ability to use some cache modes.
NACK!
Juergen
Hi Willy and Jürgen,
Following up on the x86 _PAGE_PSE and _PAGE_PAT aliasing issue.
To achieve the goal of keeping pfn_pte() pure and completely eradicating the pte_clrhuge() anti-pattern, we need a way to ensure pfn_pte() never receives a pgprot with the huge bit set.
@Jürgen:
Just to be absolutely certain: is there any safe way to filter out the huge page attributes directly inside x86's pfn_pte() without breaking PAT? Or does the hardware bit-aliasing make this strictly impossible at the pfn_pte() level?
@Willy @Jürgen:
Assuming it is impossible to filter this safely inside pfn_pte() on x86, we must translate the pgprot before passing it down. To maintain strict type-safety and still drop pte_clrhuge(), I plan to introduce two arch-neutral wrappers:
x86:
/* Translates large prot to 4K. Shifts PAT back to bit 7, inherently clearing _PAGE_PSE */
#define pgprot_huge_to_pte(prot) pgprot_large_2_4k(prot)
/* Translates 4K prot to large. Shifts PAT to bit 12, strictly sets _PAGE_PSE */ #define pgprot_pte_to_huge(prot) __pgprot(pgprot_val(pgprot_4k_2_large(prot)) | _PAGE_PSE)
arm64:
/*
* Drops Block marker, enforces Page marker.
* Strictly preserves the PTE_VALID bit to avoid validating PROT_NONE
pages.
*/
#define pgprot_huge_to_pte(prot) \
__pgprot((pgprot_val(prot) & ~(PMD_TYPE_MASK & ~PTE_VALID)) | \
(PTE_TYPE_PAGE & ~PTE_VALID))
/*
* Drops Page marker, sets Block marker.
* Strictly preserves the PTE_VALID bit.
*/
#define pgprot_pte_to_huge(prot) \
__pgprot((pgprot_val(prot) & ~(PTE_TYPE_MASK & ~PTE_VALID)) | \
(PMD_TYPE_SECT & ~PTE_VALID))
Usage:
1. Creating a huge pfnmap (remap_try_huge_pmd) pgprot_t huge_prot = pgprot_pte_to_huge(prot);
/* No need for pmd_mkhuge() */pmd_t entry = pmd_mkspecial(pfn_pmd(pfn, huge_prot)); set_pmd_at(mm, addr, pmd, entry);
2. Splitting a huge pfnmap (__split_huge_pmd_locked) pgprot_t small_prot = pgprot_huge_to_pte(pmd_pgprot(old_pmd));
/* No need for pte_clrhuge() */pte_t entry = pfn_pte(pmd_pfn(old_pmd), small_prot); set_ptes(mm, haddr, pte, entry, HPAGE_PMD_NR);
Willy, is there a better architectural approach to handle this and satisfy the type-safety requirement given the x86 hardware constraints?
Signature · 2 lines
Thanks, Yin Tirui
Re: [PATCH RFC v3 2/4] mm/pgtable: Make pfn_pte() filter out huge page attributes
Quoted 63 lines · Yin Tirui
Quoted 54 lines · Jürgen Groß
Quoted 40 lines · Yin Tirui
A fundamental principle of page table type safety is that
pte_trepresents the lowest level page table entry and should never carry huge page attributes.Currently, passing a pgprot with huge page bits (e.g., extracted via
pmd_pgprot()) intopfn_pte()creates a malformed PTE that retains the huge attribute, leading to the necessity of the uglypte_clrhuge()anti- pattern.Enforce type safety by making
pfn_pte()inherently filter out huge page attributes:
- On x86: Strip the_PAGE_PSEbit.
- On ARM64: Mask out the block descriptor bits inPTE_TYPE_MASKand enforce thePTE_TYPE_PAGEformat.
- On RISC-V: No changes required, as RISC-V leaf PMDs and PTEs share the exact same hardware format and do not use a distinct huge bit.Signed-off-by Yin Tirui <[email protected]>---
arch/arm64/include/asm/pgtable.h| 4 +++-
arch/x86/include/asm/pgtable.h | 4 ++++2 files changed, 7 insertions(+), 1 deletion(-)arch/arm64/include/asm/pgtable.h +3
-1@@ -141,7 +141,9 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) #define pte_pfn(pte) (__pte_to_phys(pte) >> PAGE_SHIFT) #define pfn_pte(pfn,prot) \ - __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) + __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | \ + ((pgprot_val(prot) & ~(PTE_TYPE_MASK & ~PTE_VALID)) | \ + (PTE_TYPE_PAGE & ~PTE_VALID))) #define pte_none(pte) (!pte_val(pte)) #define pte_page(pte) (pfn_to_page(pte_pfn(pte)))arch/x86/include/asm/pgtable.h +0
-0arch/x86/include/asm/pgtable.h static inline pgprotval_t check_pgprot(pgprot_t pgprot) · line 743
Quoted 6 lines
arch/x86/include/asm/pgtable.h +3
-0@@ -738,6 +738,10 @@ static inline pgprotval_t check_pgprot(pgprot_t pgprot) static inline pte_t pfn_pte(unsigned long page_nr, pgprot_t pgprot) { phys_addr_t pfn = (phys_addr_t)page_nr << PAGE_SHIFT; + + /* Filter out _PAGE_PSE to ensure PTEs never carry the huge page bit */ + pgprot = __pgprot(pgprot_val(pgprot) & ~_PAGE_PSE);Is it really a good idea to silently drop the bit?
Today it can either be used for a large page (which should be a pmd, of course), or - much worse - you'd strip the
_PAGE_PATbit, which is at the same position in PTEs.Quoted 6 lines
So basically you are removing the ability to use some cache modes.
NACK!
Juergen
Quoted 8 lines
Hi Willy and Jürgen,
Following up on the x86
_PAGE_PSEand_PAGE_PATaliasing issue.To achieve the goal of keeping
pfn_pte()pure and completely eradicating thepte_clrhuge()anti-pattern, we need a way to ensurepfn_pte()never receives a pgprot with the huge bit set.@Jürgen:
Just to be absolutely certain: is there any safe way to filter out the huge page attributes directly inside x86'spfn_pte()without breaking PAT? Or does the hardware bit-aliasing make this strictly impossible at thepfn_pte()level?
There is no huge bit at the PTE level. It is existing only at the PMD and the PUD level.
So: yes, it is absolutely impossible to filter it out, as the bit has a different meaning in "real" PTEs (with "PTE" having the meaning: a translation entry in a page referenced by a PMD entry not having the PSE bit set).
Quoted 9 lines
@Willy @Jürgen:
Assuming it is impossible to filter this safely insidepfn_pte()on x86, we must translate the pgprot before passing it down. To maintain strict type-safety and still droppte_clrhuge(), I plan to introduce two arch-neutral wrappers:x86:
/* Translates large prot to 4K. Shifts PAT back to bit 7, inherently clearing_PAGE_PSE*/ #define pgprot_huge_to_pte(prot)pgprot_large_2_4k(prot)
/* Translates 4K prot to large. Shifts PAT to bit 12, strictly sets_PAGE_PSE*/ #definepgprot_pte_to_huge(prot)__pgprot(pgprot_val(pgprot_4k_2_large(prot)) | _PAGE_PSE)
Seems to be okay.
Juergen
Re: [PATCH RFC v3 2/4] mm/pgtable: Make pfn_pte() filter out huge page attributes
Quoted 9 lines · Jürgen Groß
Quoted 8 lines
Hi Willy and Jürgen,
Following up on the x86
_PAGE_PSEand_PAGE_PATaliasing issue.To achieve the goal of keeping
pfn_pte()pure and completely eradicating thepte_clrhuge()anti-pattern, we need a way to ensurepfn_pte()never receives a pgprot with the huge bit set.@Jürgen:
Just to be absolutely certain: is there any safe way to filter out the huge page attributes directly inside x86'spfn_pte()without breaking PAT? Or does the hardware bit-aliasing make this strictly impossible at thepfn_pte()level?
Quoted 7 lines
There is no huge bit at the PTE level. It is existing only at the PMD and the PUD level.
So: yes, it is absolutely impossible to filter it out, as the bit has a different meaning in "real" PTEs (with "PTE" having the meaning: a translation entry in a page referenced by a PMD entry not having the PSE bit set).
Hi Jürgen,
Thank you for your confirmation.
Quoted 11 lines
Quoted 9 lines
@Willy @Jürgen:
Assuming it is impossible to filter this safely insidepfn_pte()on x86, we must translate the pgprot before passing it down. To maintain strict type-safety and still droppte_clrhuge(), I plan to introduce two arch-neutral wrappers:x86:
/* Translates large prot to 4K. Shifts PAT back to bit 7, inherently clearing_PAGE_PSE*/ #define pgprot_huge_to_pte(prot)pgprot_large_2_4k(prot)
/* Translates 4K prot to large. Shifts PAT to bit 12, strictly sets_PAGE_PSE*/ #definepgprot_pte_to_huge(prot)__pgprot(pgprot_val(pgprot_4k_2_large(prot)) | _PAGE_PSE)Seems to be okay.
While the wrapper approach handles the aliasing, Willy recently suggested taking it a step further by embedding this translation directly into pfn_pmd() and pmd_pgprot().
I am going to explore this embedded approach for the v4 respin.
Signature · 1 line
Yin Tirui
Re: [PATCH RFC v3 2/4] mm/pgtable: Make pfn_pte() filter out huge page attributes
In reply to Yin TiruiQuoted 32 lines · Yin Tirui
Quoted 10 lines · Jürgen Groß
Today it can either be used for a large page (which should be a pmd, of course), or - much worse - you'd strip the
_PAGE_PATbit, which is at the same position in PTEs.So basically you are removing the ability to use some cache modes.
NACK!
Juergen
Hi Willy and Jürgen,
Following up on the x86
_PAGE_PSEand_PAGE_PATaliasing issue.To achieve the goal of keeping
pfn_pte()pure and completely eradicating thepte_clrhuge()anti-pattern, we need a way to ensurepfn_pte()never receives a pgprot with the huge bit set.@Jürgen:
Just to be absolutely certain: is there any safe way to filter out the huge page attributes directly inside x86'spfn_pte()without breaking PAT? Or does the hardware bit-aliasing make this strictly impossible at thepfn_pte()level?@Willy @Jürgen:
Assuming it is impossible to filter this safely insidepfn_pte()on x86, we must translate the pgprot before passing it down. To maintain strict type-safety and still droppte_clrhuge(), I plan to introduce two arch-neutral wrappers:
Quoted 8 lines
x86:
/* Translates large prot to 4K. Shifts PAT back to bit 7, inherently clearing_PAGE_PSE*/
#definepgprot_huge_to_pte(prot)pgprot_large_2_4k(prot)
/* Translates 4K prot to large. Shifts PAT to bit 12, strictly sets_PAGE_PSE*/
#definepgprot_pte_to_huge(prot)__pgprot(pgprot_val(pgprot_4k_2_large(prot)) | _PAGE_PSE)
I don't think we should have pgprot_large_2_4k(). Or rather, I think it should be embedded in pmd_pgprot() / pud_pgprot(). That is, we should have an 'ideal' pgprot which, on x86, perhaps matches that used by the 4k level. pfn_pmd() should be converting from the ideal pgprot to that actually used by PMDs (and setting _PAGE_PSE?)
Quoted 40 lines
arm64:
/*
* Drops Block marker, enforces Page marker.
* Strictly preserves thePTE_VALIDbit to avoid validatingPROT_NONEpages. */
#definepgprot_huge_to_pte(prot)\
__pgprot((pgprot_val(prot) & ~(PMD_TYPE_MASK & ~PTE_VALID)) | \
(PTE_TYPE_PAGE& ~PTE_VALID))
/*
* Drops Page marker, sets Block marker.
* Strictly preserves thePTE_VALIDbit.
*/
#definepgprot_pte_to_huge(prot)\
__pgprot((pgprot_val(prot) & ~(PTE_TYPE_MASK & ~PTE_VALID)) | \
(PMD_TYPE_SECT& ~PTE_VALID))Usage:
1. Creating a huge pfnmap (remap_try_huge_pmd)pgprot_thuge_prot=pgprot_pte_to_huge(prot);/* No need for
pmd_mkhuge()*/pmd_tentry =pmd_mkspecial(pfn_pmd(pfn, huge_prot));set_pmd_at(mm, addr, pmd, entry);2. Splitting a huge pfnmap (
__split_huge_pmd_locked)pgprot_tsmall_prot= pgprot_huge_to_pte(pmd_pgprot(old_pmd));/* No need for
pte_clrhuge()*/pte_tentry =pfn_pte(pmd_pfn(old_pmd), small_prot);set_ptes(mm, haddr, pte, entry, HPAGE_PMD_NR);Willy, is there a better architectural approach to handle this and satisfy the type-safety requirement given the x86 hardware constraints?
Signature · 2 lines
Thanks, Yin Tirui
Re: [PATCH RFC v3 2/4] mm/pgtable: Make pfn_pte() filter out huge page attributes
Quoted 7 lines · Matthew Wilcox
I don't think we should have
pgprot_large_2_4k(). Or rather, I think it should be embedded inpmd_pgprot()/pud_pgprot(). That is, we should have an 'ideal' pgprot which, on x86, perhaps matches that used by the 4k level.pfn_pmd()should be converting from the ideal pgprot to that actually used by PMDs (and setting_PAGE_PSE?)
Hi Willy,
I will take this route and implement the embedded approach for the v4 respin.
Signature · 1 line
Yin Tirui
Will Deacon
maintainerRe: [PATCH RFC v3 2/4] mm/pgtable: Make pfn_pte() filter out huge page attributes
In reply to Yin TiruiQuoted 26 lines · Yin Tirui
A fundamental principle of page table type safety is that
pte_trepresents the lowest level page table entry and should never carry huge page attributes.Currently, passing a pgprot with huge page bits (e.g., extracted via
pmd_pgprot()) intopfn_pte()creates a malformed PTE that retains the huge attribute, leading to the necessity of the uglypte_clrhuge()anti-pattern.Enforce type safety by making
pfn_pte()inherently filter out huge page attributes:
- On x86: Strip the_PAGE_PSEbit.
- On ARM64: Mask out the block descriptor bits inPTE_TYPE_MASKand enforce thePTE_TYPE_PAGEformat.
- On RISC-V: No changes required, as RISC-V leaf PMDs and PTEs share the exact same hardware format and do not use a distinct huge bit.Signed-off-by Yin Tirui <[email protected]>--- arch/arm64/include/asm/pgtable.h | 4 +++- arch/x86/include/asm/pgtable.h | 4 ++++ 2 files changed, 7 insertions(+), 1 deletion(-)arch/arm64/include/asm/pgtable.h +0
-0
arch/arm64/include/asm/pgtable.h static inline pteval_t __phys_to_pte_val(phys_addr_t phys) · line 146
Quoted 7 lines
arch/arm64/include/asm/pgtable.h +3
-1@@ -141,7 +141,9 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys)#define pte_pfn (pte) (__pte_to_phys(pte) >> PAGE_SHIFT)#define pfn_pte (pfn,prot) \ - __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) + __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | \ + ((pgprot_val(prot) & ~(PTE_TYPE_MASK & ~PTE_VALID)) | \ + (PTE_TYPE_PAGE & ~PTE_VALID)))
Why are you touching arch/arm64? We don't implement pte_clrhuge() afaict. What does this actually fix?
Will
Re: [PATCH RFC v3 2/4] mm/pgtable: Make pfn_pte() filter out huge page attributes
Hi Will,
Quoted 28 lines · Will Deacon
Quoted 27 lines · Yin Tirui
A fundamental principle of page table type safety is that
pte_trepresents the lowest level page table entry and should never carry huge page attributes.Currently, passing a pgprot with huge page bits (e.g., extracted via
pmd_pgprot()) intopfn_pte()creates a malformed PTE that retains the huge attribute, leading to the necessity of the uglypte_clrhuge()anti-pattern.Enforce type safety by making
pfn_pte()inherently filter out huge page attributes:
- On x86: Strip the_PAGE_PSEbit.
- On ARM64: Mask out the block descriptor bits inPTE_TYPE_MASKand enforce thePTE_TYPE_PAGEformat.
- On RISC-V: No changes required, as RISC-V leaf PMDs and PTEs share the exact same hardware format and do not use a distinct huge bit.Signed-off-by Yin Tirui <[email protected]>--- arch/arm64/include/asm/pgtable.h | 4 +++- arch/x86/include/asm/pgtable.h | 4 ++++ 2 files changed, 7 insertions(+), 1 deletion(-)arch/arm64/include/asm/pgtable.h +0
-0@@ -141,7 +141,9 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys)
Quoted 8 lines
arch/arm64/include/asm/pgtable.h static inline pteval_t __phys_to_pte_val(phys_addr_t phys) · line 146
Quoted 6 lines
arch/arm64/include/asm/pgtable.h +3
-1#define pte_pfn (pte) (__pte_to_phys(pte) >> PAGE_SHIFT)#define pfn_pte (pfn,prot) \ - __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) + __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | \ + ((pgprot_val(prot) & ~(PTE_TYPE_MASK & ~PTE_VALID)) | \ + (PTE_TYPE_PAGE & ~PTE_VALID)))Why are you touching
arch/arm64? We don't implementpte_clrhuge()afaict. What does this actually fix?
Originally, this patch aimed to ensure that pfn_pte() always returns a PTE without any
huge page attributes by embedding the logic of pte_clrhuge() directly into pfn_pte().
However, we found this approach doesn't work well on x86, so we've abandoned this design.
Following Matthew Wilcox's suggestion, the current approach is instead to have pmd_pgprot()
return a 4K–formatted pgprot_t (i.e., without huge page attributes), and then explicitly add the
huge page attributes when constructing a PMD via pfn_pmd().
I've already implemented this in my recent commit:
https://github.com/torvalds/linux/commit/5b8ce6d33822dd7776432e03a08fe6d2dedac079
Quoted 2 lines
Will
Signature · 1 line
Yin Tirui
RFC PATCH v3 3/4 x86/mm: Remove pte_clrhuge() and clean up init_64.c
In reply to Yin TiruiWith pfn_pte() now guaranteeing that it will natively filter out huge page attributes like _PAGE_PSE, the pte_clrhuge() helper has become obsolete.
Remove pte_clrhuge() entirely. Concurrently, clean up the ugly type-casting anti-pattern in arch/x86/mm/init_64.c where (pte_t *) was forcibly cast from pmd_t * to call pte_clrhuge(). Now, we can simply extract the pgprot directly via pmd_pgprot() and safely pass it downstream, knowing that pfn_pte() will strip the huge bit automatically.
--- arch/x86/include/asm/pgtable.h | 5 ----- arch/x86/mm/init_64.c | 4 ++-- 2 files changed, 2 insertions(+), 7 deletions(-)
arch/x86/include/asm/pgtable.h +0 -5
@@ -483,11 +483,6 @@ static inline pte_t pte_mkhuge(pte_t pte)return pte_set_flags (pte, _PAGE_PSE); } -static inline pte_t pte_clrhuge (pte_t pte) -{ -return pte_clear_flags (pte, _PAGE_PSE); -} -static inline pte_t pte_mkglobal (pte_t pte) {return pte_set_flags (pte, _PAGE_GLOBAL);
arch/x86/mm/init_64.c +2 -2
@@ -572,7 +572,7 @@ phys_pmd_init(pmd_t *pmd_page, unsigned long paddr, unsigned long paddr_end, paddr_last = paddr_next;continue ; } - new_prot =pte_pgprot (pte_clrhuge (*(pte_t *)pmd)); + new_prot =pmd_pgprot (*pmd); }if (page_size_mask & (1 <<PG_LEVEL_2M )) {
@@ -658,7 +658,7 @@ phys_pud_init(pud_t *pud_page, unsigned long paddr, unsigned long paddr_end, paddr_last = paddr_next;continue ; } - prot =pte_pgprot (pte_clrhuge (*(pte_t *)pud)); + prot =pud_pgprot (*pud); }if (page_size_mask & (1 <<PG_LEVEL_1G )) {
Signature · 1 line
2.22.0
RFC PATCH v3 4/4 mm: add PMD-level huge page support for remap_pfn_range()
In reply to Yin TiruiAdd PMD-level huge page support to remap_pfn_range(), automatically creating huge mappings when prerequisites are satisfied (size, alignment, architecture support, etc.) and falling back to normal page mappings otherwise.
Implement special huge PMD splitting by utilizing the pgtable deposit/ withdraw mechanism. When splitting is needed, the deposited pgtable is withdrawn and populated with individual PTEs created from the original huge mapping.
--- mm/huge_memory.c | 36 ++++++++++++++++++++++++++++++++++-- mm/memory.c | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+), 2 deletions(-)
mm/huge_memory.c +34 -2
@@ -1857,6 +1857,9 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmd =pmdp_get_lockless (src_pmd);if (unlikely (pmd_present (pmd) &&pmd_special (pmd) && !is_huge_zero_pmd (pmd))) { + pgtable =pte_alloc_one (dst_mm); +if (unlikely (!pgtable)) + goto out; dst_ptl =pmd_lock (dst_mm, dst_pmd); src_ptl =pmd_lockptr (src_mm, src_pmd);spin_lock_nested (src_ptl,SINGLE_DEPTH_NESTING );
@@ -1870,6 +1873,12 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,* able to wrongly write to the backend MMIO. */ VM_WARN_ON_ONCE (is_cow_mapping (src_vma->vm_flags) &&pmd_write (pmd)); + +/* dax won't reach here, it will be intercepted at vma_needs_copy() */ +VM_WARN_ON_ONCE (vma_is_dax (src_vma)); + +mm_inc_nr_ptes (dst_mm); +pgtable_trans_huge_deposit (dst_mm, dst_pmd, pgtable); goto set_pmd; }
@@ -2360,6 +2369,8 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,arch_check_zapped_pmd (vma, orig_pmd);tlb_remove_pmd_tlb_entry (tlb, pmd, addr);if (!vma_is_dax (vma) &&vma_is_special_huge (vma)) { +if (pmd_special (orig_pmd)) +zap_deposited_table (tlb->mm, pmd);if (arch_needs_pgtable_deposit ())zap_deposited_table (tlb->mm, pmd);spin_unlock (ptl);
@@ -3005,14 +3016,35 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,if (!vma_is_anonymous (vma)) { old_pmd =pmdp_huge_clear_flush (vma, haddr, pmd); + +if (!vma_is_dax (vma) &&vma_is_special_huge (vma)) { +pte_t entry; + +if (!pmd_special (old_pmd)) { +zap_deposited_table (mm, pmd); +return ; + } + pgtable =pgtable_trans_huge_withdraw (mm, pmd); +if (unlikely (!pgtable)) +return ; +pmd_populate (mm, &_pmd, pgtable); + pte =pte_offset_map (&_pmd, haddr); + entry =pfn_pte (pmd_pfn (old_pmd),pmd_pgprot (old_pmd)); +set_ptes (mm, haddr, pte, entry,HPAGE_PMD_NR ); +pte_unmap (pte); + +smp_wmb ();/* make pte visible before pmd */ +pmd_populate (mm, pmd, pgtable); +return ; + } +/* * We are going to unmap this huge page. So * just go ahead and zap it */ if (arch_needs_pgtable_deposit ())zap_deposited_table (mm, pmd); -if (!vma_is_dax (vma) &&vma_is_special_huge (vma)) -return ; +if (unlikely (pmd_is_migration_entry (old_pmd))) {const softleaf_t old_entry =softleaf_from_pmd (old_pmd);
mm/memory.c +40 -0
@@ -2890,6 +2890,40 @@ static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd,return err; } +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +static int remap_try_huge_pmd (struct mm_struct *mm,pmd_t *pmd, +unsigned long addr,unsigned long end, +unsigned long pfn,pgprot_t prot) +{ +pgtable_t pgtable; +spinlock_t *ptl; + +if ((end - addr) !=PMD_SIZE ) +return 0 ; + +if (!IS_ALIGNED (addr,PMD_SIZE )) +return 0 ; + +if (!IS_ALIGNED (pfn,HPAGE_PMD_NR )) +return 0 ; + +if (pmd_present (*pmd) && !pmd_free_pte_page (pmd, addr)) +return 0 ; + + pgtable =pte_alloc_one (mm); +if (unlikely (!pgtable)) +return 0 ; + +mm_inc_nr_ptes (mm); + ptl =pmd_lock (mm, pmd); +set_pmd_at (mm, addr, pmd,pmd_mkspecial (pmd_mkhuge (pfn_pmd (pfn, prot)))); +pgtable_trans_huge_deposit (mm, pmd, pgtable); +spin_unlock (ptl); + +return 1 ; +} +#endif +static inline int remap_pmd_range (struct mm_struct *mm,pud_t *pud,unsigned long addr,unsigned long end,unsigned long pfn,pgprot_t prot)
@@ -2905,6 +2939,12 @@ static inline int remap_pmd_range(struct mm_struct *mm, pud_t *pud,VM_BUG_ON (pmd_trans_huge (*pmd)); do { next = pmd_addr_end(addr, end); +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP + if (remap_try_huge_pmd(mm, pmd, addr, next, + pfn + (addr >> PAGE_SHIFT), prot)) { + continue; + } +#endif err = remap_pte_range(mm, pmd, addr, next, pfn + (addr >> PAGE_SHIFT), prot); if (err)
Signature · 1 line
2.22.0
David Hildenbrand (Arm)
maintainerRe: [PATCH RFC v3 4/4] mm: add PMD-level huge page support for remap_pfn_range()
Quoted 12 lines · Yin Tirui
Add PMD-level huge page support to
remap_pfn_range(), automatically creating huge mappings when prerequisites are satisfied (size, alignment, architecture support, etc.) and falling back to normal page mappings otherwise.Implement special huge PMD splitting by utilizing the pgtable deposit/ withdraw mechanism. When splitting is needed, the deposited pgtable is withdrawn and populated with individual PTEs created from the original huge mapping.
Signed-off-by Yin Tirui <[email protected]>---
[...]
mm/huge_memory.c static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, · line 3020
Quoted 5 lines
mm/huge_memory.c +2
-0if (!vma_is_anonymous (vma)) { old_pmd =pmdp_huge_clear_flush (vma, haddr, pmd); + +if (!vma_is_dax (vma) &&vma_is_special_huge (vma)) {
These magical vma checks are really bad. This all needs a cleanup (Lorenzo is doing some, hoping it will look better on top of that).
mm/huge_memory.c static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, · line 3023
Quoted 3 lines
mm/huge_memory.c +3
-0+pte_t entry; + +if (!pmd_special (old_pmd)) {
If you are using pmd_special(), you are doing something wrong.
Hint: vm_normal_page_pmd() is usually what you want.
Quoted 34 lines
mm/huge_memory.c +18
-2+zap_deposited_table (mm, pmd); +return ; + } + pgtable =pgtable_trans_huge_withdraw (mm, pmd); +if (unlikely (!pgtable)) +return ; +pmd_populate (mm, &_pmd, pgtable); + pte =pte_offset_map (&_pmd, haddr); + entry =pfn_pte (pmd_pfn (old_pmd),pmd_pgprot (old_pmd)); +set_ptes (mm, haddr, pte, entry,HPAGE_PMD_NR ); +pte_unmap (pte); + +smp_wmb ();/* make pte visible before pmd */ +pmd_populate (mm, pmd, pgtable); +return ; + } +/* * We are going to unmap this huge page. So * just go ahead and zap it */ if (arch_needs_pgtable_deposit ())zap_deposited_table (mm, pmd); -if (!vma_is_dax (vma) &&vma_is_special_huge (vma)) -return ; +if (unlikely (pmd_is_migration_entry (old_pmd))) {const softleaf_t old_entry =softleaf_from_pmd (old_pmd);mm/memory.c +0
-0
mm/memory.c static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd, · line 2893
Quoted 4 lines
mm/memory.c +1
-0@@ -2890,6 +2890,40 @@ static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd,return err; } +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP
Why exactly do we need arch support for that in form of a Kconfig.
Usually, we guard pmd support by CONFIG_TRANSPARENT_HUGEPAGE.
And then, we must check at runtime if PMD leaves are actually supported.
Luiz is working on a cleanup series:
https://lore.kernel.org/r/[email protected]
pgtable_has_pmd_leaves() is what you would want to check.
mm/memory.c static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd, · line 2896
Quoted 3 lines
mm/memory.c +3
-0+static int remap_try_huge_pmd (struct mm_struct *mm,pmd_t *pmd, +unsigned long addr,unsigned long end, +unsigned long pfn,pgprot_t prot)
Use two-tab indent. (currently 3? :) )
Also, we tend to call these things now "pmd leaves". Call it
"remap_try_pmd_leaf" or something even more expressive like
"remap_try_install_pmd_leaf()"
mm/memory.c static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd, · line 2901
Quoted 5 lines
mm/memory.c +5
-0+{ +pgtable_t pgtable; +spinlock_t *ptl; + +if ((end - addr) !=PMD_SIZE )
if (end - addr != PMD_SIZE)
Should work
mm/memory.c static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd, · line 2906
Quoted 5 lines
mm/memory.c +5
-0+return 0 ; + +if (!IS_ALIGNED (addr,PMD_SIZE )) +return 0 ; +
You could likely combine both things into a
if (!IS_ALIGNED(addr | end, PMD_SIZE))
mm/memory.c static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd, · line 2907
Quoted 1 line
mm/memory.c +1
-0+if (!IS_ALIGNED (pfn,HPAGE_PMD_NR ))
Another sign that you piggy-back on THP support ;)
mm/memory.c static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd, · line 2911
Quoted 4 lines
mm/memory.c +4
-0+return 0 ; + +if (pmd_present (*pmd) && !pmd_free_pte_page (pmd, addr)) +return 0 ;
Ripping out a page table?! That doesn't sound right :)
Why is that required? We shouldn't be doing that here. Gah.
Especially, without any pmd locks etc.
Quoted 19 lines
mm/memory.c +15
-0+ + pgtable =pte_alloc_one (mm); +if (unlikely (!pgtable)) +return 0 ; + +mm_inc_nr_ptes (mm); + ptl =pmd_lock (mm, pmd); +set_pmd_at (mm, addr, pmd,pmd_mkspecial (pmd_mkhuge (pfn_pmd (pfn, prot)))); +pgtable_trans_huge_deposit (mm, pmd, pgtable); +spin_unlock (ptl); + + return 1; +} +#endif +static inline int remap_pmd_range (struct mm_struct *mm,pud_t *pud,unsigned long addr,unsigned long end,unsigned long pfn,pgprot_t prot)
mm/memory.c static inline int remap_pmd_range(struct mm_struct *mm, pud_t *pud, · line 2944
Quoted 6 lines
mm/memory.c +3
-0@@ -2905,6 +2939,12 @@ static inline int remap_pmd_range(struct mm_struct *mm, pud_t *pud,VM_BUG_ON (pmd_trans_huge (*pmd)); do { next = pmd_addr_end(addr, end); +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP + if (remap_try_huge_pmd(mm, pmd, addr, next, + pfn + (addr >> PAGE_SHIFT), prot)) {
Please provide a stub instead so we don't end up with ifdef in this code.
Signature · 3 lines
Cheers, David
[RESEND] Re: [PATCH RFC v3 4/4] mm: add PMD-level huge page support for remap_pfn_range()
(Resending to keep the thread intact, sorry for the noise)
Hi David,
Thanks a lot for the thorough review!
Quoted 17 lines · David Hildenbrand (Arm)
Quoted 12 lines · Yin Tirui
Add PMD-level huge page support to
remap_pfn_range(), automatically creating huge mappings when prerequisites are satisfied (size, alignment, architecture support, etc.) and falling back to normal page mappings otherwise.Implement special huge PMD splitting by utilizing the pgtable deposit/ withdraw mechanism. When splitting is needed, the deposited pgtable is withdrawn and populated with individual PTEs created from the original huge mapping.
Signed-off-by Yin Tirui <[email protected]>---
[...]
Quoted 1 line
Quoted 8 lines
Review
Quoted 4 lines
(quoted diff) +2
-0if (!vma_is_anonymous(vma)) { old_pmd = pmdp_huge_clear_flush(vma, haddr, pmd); + + if (!vma_is_dax(vma) && vma_is_special_huge(vma)) {These magical vma checks are really bad. This all needs a cleanup (Lorenzo is doing some, hoping it will look better on top of that).
Agreed. I am following Lorenzo's recent cleanups closely.
Quoted 7 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ pte_t entry; + + if (!pmd_special(old_pmd)) {If you are using
pmd_special(), you are doing something wrong.Hint:
vm_normal_page_pmd()is usually what you want.
Spot on.
While looking into applying vm_normal_folio_pmd() here to avoid the magical VMA checks, I realized that both __split_huge_pmd_locked() and copy_huge_pmd() currently suffer from the same !vma_is_anonymous(vma) top-level entanglement.I think these functions could benefit from a structural refactoring similar to what Lorenzo is currently doing in zap_huge_pmd().
My idea is to flatten both functions into a pmd_present()-driven decision tree:
1. Branch strictly on pmd_present().
2. For present PMDs, use vm_normal_folio_pmd() as the single source of truth.
3. If !folio (and not a huge zero page), it cleanly identifies special mappings (like PFNMAPs) without relying on vma_is_special_huge(). We can handle the split/copy directly and return early.
4. Otherwise, proceed with the normal Anon/File THP logic, or handle non-present migration entries in the !pmd_present() branch.
I have drafted two preparation patches demonstrating this approach and appended the diffs at the end of this email. Does this direction look reasonable to you? If so, I will iron out the implementation details and include these refactoring patches in my upcoming v4 series.
Quoted 30 lines
Quoted 30 lines
(quoted diff) +18
-2+ zap_deposited_table(mm, pmd); + return; + } + pgtable = pgtable_trans_huge_withdraw(mm, pmd); + if (unlikely(!pgtable)) + return; + pmd_populate(mm, &_pmd, pgtable); + pte = pte_offset_map(&_pmd, haddr); + entry = pfn_pte(pmd_pfn(old_pmd), pmd_pgprot(old_pmd)); + set_ptes(mm, haddr, pte, entry, HPAGE_PMD_NR); + pte_unmap(pte); + + smp_wmb(); /* make pte visible before pmd */ + pmd_populate(mm, pmd, pgtable); + return; + } + /* * We are going to unmap this huge page. So * just go ahead and zap it */ if (arch_needs_pgtable_deposit()) zap_deposited_table(mm, pmd); - if (!vma_is_dax(vma) && vma_is_special_huge(vma)) - return; + if (unlikely(pmd_is_migration_entry(old_pmd))) { const softleaf_t old_entry = softleaf_from_pmd(old_pmd);
Quoted 5 lines
mm/memory.c static int remap_pte_range(struct mm_struct
Quoted 5 lines
mm/memory.c +0
-0@@ -2890,6 +2890,40 @@ static int remap_pte_range(struct mm_struct
*mm, pmd_t *pmd,
Quoted 9 lines
mm/memory.c static int remap_pte_range(struct mm_struct · line 2893
Quoted 4 lines
mm/memory.c +1
-0return err; } +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP Why exactly do we need arch support for that in form of a
Kconfig.Usually, we guard pmd support by
CONFIG_TRANSPARENT_HUGEPAGE.
Quoted 7 lines
And then, we must check at runtime if PMD leaves are actually supported.
Luiz is working on a cleanup series:
https://lore.kernel.org/r/[email protected]
pgtable_has_pmd_leaves()is what you would want to check.
Makes sense. This Kconfig was inherited from Peter Xu's earlier proposal, but depending on CONFIG_TRANSPARENT_HUGEPAGE and pgtable_has_pmd_leaves() is indeed the correct standard. I will rebase on Luiz's series.
Quoted 8 lines
mm/memory.c static int remap_pte_range(struct mm_struct · line 2896
Quoted 5 lines
mm/memory.c +3
-0+static int remap_try_huge_pmd (struct mm_struct *mm,pmd_t *pmd, +unsigned long addr,unsigned long end, +unsigned long pfn,pgprot_t prot)Use two-tab indent. (currently 3? 🙂 )
Quoted 5 lines
Also, we tend to call these things now "pmd leaves". Call it
"remap_try_pmd_leaf" or something even more expressive like"
remap_try_install_pmd_leaf()"
Noted. Will fix the indentation and rename it.
Quoted 9 lines
mm/memory.c static int remap_pte_range(struct mm_struct · line 2901
Quoted 5 lines
mm/memory.c +5
-0+{ +pgtable_t pgtable; +spinlock_t *ptl; + +if ((end - addr) !=PMD_SIZE )if (end - addr != PMD_SIZE)Should work
Noted.
Quoted 9 lines
mm/memory.c static int remap_pte_range(struct mm_struct · line 2906
Quoted 6 lines
mm/memory.c +5
-0+return 0 ; + +if (!IS_ALIGNED (addr,PMD_SIZE )) +return 0 ; +You could likely combine both things into a
Quoted 5 lines
if (!IS_ALIGNED(addr | end,
PMD_SIZE))mm/memory.c static int remap_pte_range(struct mm_struct · line 2907
Quoted 1 line
mm/memory.c +1
-0+if (!IS_ALIGNED (pfn,HPAGE_PMD_NR ))Another sign that you piggy-back on THP support 😉
Indeed! 🙂
Quoted 11 lines
mm/memory.c static int remap_pte_range(struct mm_struct · line 2911
Quoted 5 lines
mm/memory.c +4
-0+return 0 ; + +if (pmd_present (*pmd) && !pmd_free_pte_page (pmd, addr)) +return 0 ;Ripping out a page table?! That doesn't sound right 🙂
Why is that required? We shouldn't be doing that here. Gah.
Especially, without any pmd locks etc.
...oops. That is indeed a silly one. Thanks for catching it. I will fix this to:
if (!pmd_none(*pmd)) return 0;
Quoted 9 lines
mm/memory.c static int remap_pte_range(struct mm_struct · line 2919
Quoted 9 lines
mm/memory.c +8
-0+ + pgtable =pte_alloc_one (mm); +if (unlikely (!pgtable)) +return 0 ; + +mm_inc_nr_ptes (mm); + ptl =pmd_lock (mm, pmd); + set_pmd_at(mm, addr, pmd, pmd_mkspecial(pmd_mkhuge(pfn_pmd(pfn,
prot))));
Quoted 11 lines
mm/memory.c static inline int remap_pmd_range(struct
Quoted 11 lines
mm/memory.c +7
-0+pgtable_trans_huge_deposit (mm, pmd, pgtable); +spin_unlock (ptl); + + return 1; +} +#endif +static inline int remap_pmd_range (struct mm_struct *mm,pud_t *pud,unsigned long addr,unsigned long end,unsigned long pfn,pgprot_t prot)@@ -2905,6 +2939,12 @@ static inline int remap_pmd_range(struct
mm_struct *mm, pud_t *pud,
Quoted 8 lines
mm/memory.c static inline int remap_pmd_range(struct · line 2944
Quoted 6 lines
mm/memory.c +3
-0VM_BUG_ON (pmd_trans_huge (*pmd)); do { next = pmd_addr_end(addr, end); +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP + if (remap_try_huge_pmd(mm, pmd, addr, next, + pfn + (addr >> PAGE_SHIFT), prot)) {Please provide a stub instead so we don't end up with ifdef in this code.
Will do.
Quoted 1 line
Appendix:
1. copy_huge_pmd()
mm/huge_memory.c +0 -0
@@ -1912,35 +1912,11 @@ int copy_huge_pmd(struct mm_struct *dst_mm,
struct mm_struct *src_mm,
struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma)
{
spinlock_t *dst_ptl, *src_ptl;
- struct page *src_page;
struct folio *src_folio;
pmd_t pmd;
pgtable_t pgtable = NULL;
int ret = -ENOMEM;
- pmd = pmdp_get_lockless(src_pmd);
- if (unlikely(pmd_present(pmd) && pmd_special(pmd) &&
- !is_huge_zero_pmd(pmd))) {
- dst_ptl = pmd_lock(dst_mm, dst_pmd);
- src_ptl = pmd_lockptr(src_mm, src_pmd);
- spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
- /*
- * No need to recheck the pmd, it can't change with write
- * mmap lock held here.
- *
- * Meanwhile, making sure it's not a CoW VMA with writable
- * mapping, otherwise it means either the anon page wrongly
- * applied special bit, or we made the PRIVATE mapping be
- * able to wrongly write to the backend MMIO.
- */
- VM_WARN_ON_ONCE(is_cow_mapping(src_vma->vm_flags) && pmd_write(pmd));
- goto set_pmd;
- }
-
- /* Skip if can be re-fill on fault */
- if (!vma_is_anonymous(dst_vma))
- return 0;
-
pgtable = pte_alloc_one(dst_mm);
if (unlikely(!pgtable))
goto out;
@@ -1952,48 +1928,69 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
ret = -EAGAIN;
pmd = *src_pmd;
- if (unlikely(thp_migration_supported() &&
- pmd_is_valid_softleaf(pmd))) {
+ if (likely(pmd_present(pmd))) {
+ src_folio = vm_normal_folio_pmd(src_vma, addr, pmd);
+ if (unlikely(!src_folio)) {
+ /*
+ * When page table lock is held, the huge zero pmd should not be
+ * under splitting since we don't split the page itself, only pmd to
+ * a page table.
+ */
+ if (is_huge_zero_pmd(pmd)) {
+ /*
+ * mm_get_huge_zero_folio() will never allocate a new
+ * folio here, since we already have a zero page to
+ * copy. It just takes a reference.
+ */
+ mm_get_huge_zero_folio(dst_mm);
+ goto out_zero_page;
+ }
+
+ /*
+ * Making sure it's not a CoW VMA with writable
+ * mapping, otherwise it means either the anon page wrongly
+ * applied special bit, or we made the PRIVATE mapping be
+ * able to wrongly write to the backend MMIO.
+ */
+ VM_WARN_ON_ONCE(is_cow_mapping(src_vma->vm_flags) && pmd_write(pmd));
+ pte_free(dst_mm, pgtable);
+ goto set_pmd;
+ }
+
+ if (!folio_test_anon(src_folio)) {
+ pte_free(dst_mm, pgtable);
+ ret = 0;
+ goto out_unlock;
+ }
+
+ folio_get(src_folio);
+ if (unlikely(folio_try_dup_anon_rmap_pmd(src_folio, &src_folio->page,dst_vma, src_vma))) {
+ /* Page maybe pinned: split and retry the fault on PTEs. */
+ folio_put(src_folio);
+ pte_free(dst_mm, pgtable);
+ spin_unlock(src_ptl);
+ spin_unlock(dst_ptl);
+ __split_huge_pmd(src_vma, src_pmd, addr, false);
+ return -EAGAIN;
+ }
+ add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
+
+ } else if (unlikely(thp_migration_supported() &&pmd_is_valid_softleaf(pmd))) {
+ if (unlikely(!vma_is_anonymous(dst_vma))) {
+ pte_free(dst_mm, pgtable);
+ ret = 0;
+ goto out_unlock;
+ }
copy_huge_non_present_pmd(dst_mm, src_mm, dst_pmd, src_pmd, addr,
dst_vma, src_vma, pmd, pgtable);
ret = 0;
goto out_unlock;
- }
- if (unlikely(!pmd_trans_huge(pmd))) {
+ } else {
pte_free(dst_mm, pgtable);
goto out_unlock;
}
- /*
- * When page table lock is held, the huge zero pmd should not be
- * under splitting since we don't split the page itself, only pmd to
- * a page table.
- */
- if (is_huge_zero_pmd(pmd)) {
- /*
- * mm_get_huge_zero_folio() will never allocate a new
- * folio here, since we already have a zero page to
- * copy. It just takes a reference.
- */
- mm_get_huge_zero_folio(dst_mm);
- goto out_zero_page;
- }
- src_page = pmd_page(pmd);
- VM_BUG_ON_PAGE(!PageHead(src_page), src_page);
- src_folio = page_folio(src_page);
-
- folio_get(src_folio);
- if (unlikely(folio_try_dup_anon_rmap_pmd(src_folio, src_page, dst_vma,src_vma))) {
- /* Page maybe pinned: split and retry the fault on PTEs. */
- folio_put(src_folio);
- pte_free(dst_mm, pgtable);
- spin_unlock(src_ptl);
- spin_unlock(dst_ptl);
- __split_huge_pmd(src_vma, src_pmd, addr, false);
- return -EAGAIN;
- }
- add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
out_zero_page:
mm_inc_nr_ptes(dst_mm); pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
Signature · 5 lines
2.43.0 2. __split_huge_pmd_locked()
mm/huge_memory.c +0 -0
@@ -3090,98 +3090,50 @@ static void __split_huge_pmd_locked(struct
vm_area_struct *vma, pmd_t *pmd,
count_vm_event(THP_SPLIT_PMD);
- if (!vma_is_anonymous(vma)) {
- old_pmd = pmdp_huge_clear_flush(vma, haddr, pmd);
- /*
- * We are going to unmap this huge page. So
- * just go ahead and zap it
- */
- if (arch_needs_pgtable_deposit())
- zap_deposited_table(mm, pmd);
- if (vma_is_special_huge(vma))
- return;
- if (unlikely(pmd_is_migration_entry(old_pmd))) {
- const softleaf_t old_entry = softleaf_from_pmd(old_pmd);
+ if (pmd_present(*pmd)) {
+ folio = vm_normal_folio_pmd(vma, haddr, *pmd);
- folio = softleaf_to_folio(old_entry);
- } else if (is_huge_zero_pmd(old_pmd)) {
+ if (unlikely(!folio)) {
+ /* Huge Zero Page */
+ if (is_huge_zero_pmd(*pmd))
+ /*
+ * FIXME: Do we want to invalidate secondary mmu by calling
+ * mmu_notifier_arch_invalidate_secondary_tlbs() see comments below
+ * inside __split_huge_pmd() ?
+ *
+ * We are going from a zero huge page write protected to zero
+ * small page also write protected so it does not seems useful
+ * to invalidate secondary mmu at this time.
+ */
+ return __split_huge_zero_page_pmd(vma, haddr, pmd);
+
+ /* Huge PFNMAP */
+ old_pmd = pmdp_huge_clear_flush(vma, haddr, pmd);
+ if (arch_needs_pgtable_deposit())
+ zap_deposited_table(mm, pmd);
return;
- } else {
+ }
+
+ /* File/Shmem THP */
+ if (unlikely(!folio_test_anon(folio))) {
+ old_pmd = pmdp_huge_clear_flush(vma, haddr, pmd);
+ if (arch_needs_pgtable_deposit())
+ zap_deposited_table(mm, pmd);
+ if (vma_is_special_huge(vma))
+ return;
+
page = pmd_page(old_pmd);
- folio = page_folio(page);
if (!folio_test_dirty(folio) && pmd_dirty(old_pmd))
folio_mark_dirty(folio);
if (!folio_test_referenced(folio) && pmd_young(old_pmd))
folio_set_referenced(folio);
folio_remove_rmap_pmd(folio, page, vma); folio_put(folio);
+ add_mm_counter(mm, mm_counter_file(folio), -HPAGE_PMD_NR);
+ return;
}
- add_mm_counter(mm, mm_counter_file(folio), -HPAGE_PMD_NR);
- return;
- }
-
- if (is_huge_zero_pmd(*pmd)) {
- /*
- * FIXME: Do we want to invalidate secondary mmu by calling
- * mmu_notifier_arch_invalidate_secondary_tlbs() see comments below
- * inside __split_huge_pmd() ?
- *
- * We are going from a zero huge page write protected to zero
- * small page also write protected so it does not seems useful
- * to invalidate secondary mmu at this time.
- */
- return __split_huge_zero_page_pmd(vma, haddr, pmd);
- }
-
- if (pmd_is_migration_entry(*pmd)) {
- softleaf_t entry;
-
- old_pmd = *pmd;
- entry = softleaf_from_pmd(old_pmd);
- page = softleaf_to_page(entry);
- folio = page_folio(page);
-
- soft_dirty = pmd_swp_soft_dirty(old_pmd);
- uffd_wp = pmd_swp_uffd_wp(old_pmd);
-
- write = softleaf_is_migration_write(entry);
- if (PageAnon(page))
- anon_exclusive = softleaf_is_migration_read_exclusive(entry);
- young = softleaf_is_migration_young(entry);
- dirty = softleaf_is_migration_dirty(entry);
- } else if (pmd_is_device_private_entry(*pmd)) {
- softleaf_t entry;
-
- old_pmd = *pmd;
- entry = softleaf_from_pmd(old_pmd);
- page = softleaf_to_page(entry);
- folio = page_folio(page);
-
- soft_dirty = pmd_swp_soft_dirty(old_pmd);
- uffd_wp = pmd_swp_uffd_wp(old_pmd);
-
- write = softleaf_is_device_private_write(entry);
- anon_exclusive = PageAnonExclusive(page);
- /*
- * Device private THP should be treated the same as regular
- * folios w.r.t anon exclusive handling. See the comments for
- * folio handling and anon_exclusive below.
- */
- if (freeze && anon_exclusive &&
- folio_try_share_anon_rmap_pmd(folio, page))
- freeze = false;
- if (!freeze) {
- rmap_t rmap_flags = RMAP_NONE;
-
- folio_ref_add(folio, HPAGE_PMD_NR - 1);
- if (anon_exclusive)
- rmap_flags |= RMAP_EXCLUSIVE;
-
- folio_add_anon_rmap_ptes(folio, page, HPAGE_PMD_NR,
- vma, haddr, rmap_flags);
- }
- } else {
+ /* Anon THP */
/*
* Up to this point the pmd is present and huge and userland has
* the whole access to the hugepage during the split (which
@@ -3207,7 +3159,6 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
*/
old_pmd = pmdp_invalidate(vma, haddr, pmd);
page = pmd_page(old_pmd);
- folio = page_folio(page);
if (pmd_dirty(old_pmd)) {
dirty = true;
folio_set_dirty(folio);
@@ -3218,8 +3169,6 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
uffd_wp = pmd_uffd_wp(old_pmd);
VM_WARN_ON_FOLIO(!folio_ref_count(folio), folio);
- VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio);
-
/*
* Without "freeze", we'll simply split the PMD, propagating the
* PageAnonExclusive() flag for each PTE by setting it for
@@ -3236,17 +3185,82 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
* See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */
anon_exclusive = PageAnonExclusive(page);
- if (freeze && anon_exclusive &&
- folio_try_share_anon_rmap_pmd(folio, page))
+ if (freeze && anon_exclusive && folio_try_share_anon_rmap_pmd(folio,
page))
freeze = false;
if (!freeze) {
rmap_t rmap_flags = RMAP_NONE;
-
folio_ref_add(folio, HPAGE_PMD_NR - 1);
if (anon_exclusive)
rmap_flags |= RMAP_EXCLUSIVE;
- folio_add_anon_rmap_ptes(folio, page, HPAGE_PMD_NR,
- vma, haddr, rmap_flags);
+ folio_add_anon_rmap_ptes(folio, page, HPAGE_PMD_NR, vma, haddr,rmap_flags);
+ }
+ } else { /* pmd not present */
+ folio = pmd_to_softleaf_folio(*pmd);
+ if (unlikely(!folio))
+ return;
+
+ /* Migration of File/Shmem THP */
+ if (unlikely(!folio_test_anon(folio))) {
+ old_pmd = pmdp_huge_clear_flush(vma, haddr, pmd);
+ if (arch_needs_pgtable_deposit())
+ zap_deposited_table(mm, pmd);
+ if (vma_is_special_huge(vma))
+ return;
+ add_mm_counter(mm, mm_counter_file(folio), -HPAGE_PMD_NR);
+ return;
+ }
+
+ /* Migration of Anon THP or Device Private*/
+ if (pmd_is_migration_entry(*pmd)) {
+ softleaf_t entry;
+
+ old_pmd = *pmd;
+ entry = softleaf_from_pmd(old_pmd);
+ page = softleaf_to_page(entry);
+ folio = page_folio(page);
+
+ soft_dirty = pmd_swp_soft_dirty(old_pmd);
+ uffd_wp = pmd_swp_uffd_wp(old_pmd);
+
+ write = softleaf_is_migration_write(entry);
+ if (PageAnon(page))
+ anon_exclusive = softleaf_is_migration_read_exclusive(entry);
+ young = softleaf_is_migration_young(entry);
+ dirty = softleaf_is_migration_dirty(entry);
+ } else if (pmd_is_device_private_entry(*pmd)) {
+ softleaf_t entry;
+
+ old_pmd = *pmd;
+ entry = softleaf_from_pmd(old_pmd);
+ page = softleaf_to_page(entry);
+
+ soft_dirty = pmd_swp_soft_dirty(old_pmd);
+ uffd_wp = pmd_swp_uffd_wp(old_pmd);
+
+ write = softleaf_is_device_private_write(entry);
+ anon_exclusive = PageAnonExclusive(page);
+
+ /*
+ * Device private THP should be treated the same as regular
+ * folios w.r.t anon exclusive handling. See the comments for
+ * folio handling and anon_exclusive below.
+ */
+ if (freeze && anon_exclusive &&
+ folio_try_share_anon_rmap_pmd(folio, page))
+ freeze = false;
+ if (!freeze) {
+ rmap_t rmap_flags = RMAP_NONE;
+
+ folio_ref_add(folio, HPAGE_PMD_NR - 1);
+ if (anon_exclusive)
+ rmap_flags |= RMAP_EXCLUSIVE;
+
+ folio_add_anon_rmap_ptes(folio, page, HPAGE_PMD_NR,
+ vma, haddr, rmap_flags);
+ }
+ } else {
+ VM_WARN_ONCE(1, "unknown situation.");
+ return;
}
}
Signature · 5 lines
2.43.0 -- Yin Tirui
David Hildenbrand (Arm)
maintainerRe: [RESEND] Re: [PATCH RFC v3 4/4] mm: add PMD-level huge page support for remap_pfn_range()
Quoted 6 lines
Quoted 6 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ pte_t entry; + + if (!pmd_special(old_pmd)) {If you are using
pmd_special(), you are doing something wrong.
Quoted 8 lines
Quoted 1 line
Hint:
vm_normal_page_pmd()is usually what you want.Spot on.
While looking into applying
vm_normal_folio_pmd()here to avoid the magical VMA checks, I realized that both__split_huge_pmd_locked()andcopy_huge_pmd()currently suffer from the same !vma_is_anonymous(vma) top-level entanglement.
Yes, that code is an absolute mess.
Quoted 6 lines
I think these functions could benefit from a
structural refactoring similar to what Lorenzo is currently doing inzap_huge_pmd().My idea is to flatten both functions into a pmd_present()-driven decision tree:
Likely similar to how we handle PTEs.
Quoted 8 lines
1. Branch strictly on
pmd_present().
2. For present PMDs, usevm_normal_folio_pmd()as the single source of truth.
3. If !folio (and not a huge zero page), it cleanly identifies special mappings (like PFNMAPs) without relying onvma_is_special_huge(). We can handle the split/copy directly and return early.
4. Otherwise, proceed with the normal Anon/File THP logic, or handle non-present migration entries in the !pmd_present() branch.
Yes, that sounds about right.
Quoted 6 lines
I have drafted two preparation patches demonstrating this approach and appended the diffs at the end of this email. Does this direction look reasonable to you? If so, I will iron out the implementation details and include these refactoring patches in my upcoming v4 series.
Yes, sounds like nice cleanups.
Maybe the code could then benefit from moving the present vs. non-present handling into separate functions? Just a thought.
Signature · 3 lines
Cheers, David