mm/memory: optimize fork() with PTE-mapped THP
Subsystems (10)
cover mm/memory: optimize fork() with PTE-mapped THP
Now that the rmap overhaul[1] is upstream that provides a clean interface for rmap batching, let's implement PTE batching during fork when processing PTE-mapped THPs.
This series is partially based on Ryan's previous work[2] to implement cont-pte support on arm64, but its a complete rewrite based on [1] to optimize all architectures independent of any such PTE bits, and to use the new rmap batching functions that simplify the code and prepare for further rmap accounting changes.
We collect consecutive PTEs that map consecutive pages of the same large folio, making sure that the other PTE bits are compatible, and (a) adjust the refcount only once per batch, (b) call rmap handling functions only once per batch and (c) perform batch PTE setting/updates.
While this series should be beneficial for adding cont-pte support on ARM64[2], it's one of the requirements for maintaining a total mapcount[3] for large folios with minimal added overhead and further changes[4] that build up on top of the total mapcount.
Independent of all that, this series results in a speedup during fork with PTE-mapped THP, which is the default with THPs that are smaller than a PMD (for example, 16KiB to 1024KiB mTHPs for anonymous memory[5]).
On an Intel Xeon Silver 4210R CPU, fork'ing with 1GiB of PTE-mapped folios of the same size (stddev < 1%) results in the following runtimes for fork() (shorter is better):
Folio Size | v6.8-rc1 | New | Change
------------------------------------------
4KiB | 0.014328 | 0.014265 | 0%
16KiB | 0.014263 | 0.013293 | - 7%
32KiB | 0.014334 | 0.012355 | -14%
64KiB | 0.014046 | 0.011837 | -16%
128KiB | 0.014011 | 0.011536 | -18%
256KiB | 0.013993 | 0.01134 | -19%
512KiB | 0.013983 | 0.011311 | -19%
1024KiB | 0.013986 | 0.011282 | -19%
2048KiB | 0.014305 | 0.011496 | -20%Next up is PTE batching when unmapping, that I'll probably send out based on this series this/next week.
Only tested on x86-64. Compile-tested on most other architectures. Will do more testing and double-check the arch changes while this is getting some review.
[1] https://lkml.kernel.org/r/[email protected]
[2] https://lkml.kernel.org/r/[email protected]
[3] https://lkml.kernel.org/r/[email protected]
[4] https://lkml.kernel.org/r/[email protected]
[5] https://lkml.kernel.org/r/[email protected]
David Hildenbrand (11):
arm/pgtable: define PFN_PTE_SHIFT on arm and arm64 nios2/pgtable: define PFN_PTE_SHIFT
powerpc/pgtable: define PFN_PTE_SHIFT
risc: pgtable: define PFN_PTE_SHIFT
s390/pgtable: define PFN_PTE_SHIFT
sparc/pgtable: define PFN_PTE_SHIFT
mm/memory: factor out copying the actual PTE in copy_present_pte() mm/memory: pass PTE to copy_present_pte()
mm/memory: optimize fork() with PTE-mapped THP
mm/memory: ignore dirty/accessed/soft-dirty bits in folio_pte_batch() mm/memory: ignore writable bit in folio_pte_batch()
arch/arm/include/asm/pgtable.h | 2 + arch/arm64/include/asm/pgtable.h | 2 + arch/nios2/include/asm/pgtable.h | 2 + arch/powerpc/include/asm/pgtable.h | 2 + arch/riscv/include/asm/pgtable.h | 2 + arch/s390/include/asm/pgtable.h | 2 + arch/sparc/include/asm/pgtable_64.h | 2 + include/linux/pgtable.h | 17 ++- mm/memory.c | 188 +++++++++++++++++++++------- 9 files changed, 173 insertions(+), 46 deletions(-)
base-commit: 6613476e225e090cc9aad49be7fa504e290dd33d
Signature · 1 line
2.43.0
PATCH 1/11 arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
We want to make use of pte_next_pfn() outside of set_ptes(). Let's simpliy define PFN_PTE_SHIFT, required by pte_next_pfn().
--- arch/arm/include/asm/pgtable.h | 2 ++ arch/arm64/include/asm/pgtable.h | 2 ++ 2 files changed, 4 insertions(+)
arch/arm/include/asm/pgtable.h +2 -0
@@ -209,6 +209,8 @@ static inline void __sync_icache_dcache(pte_t pteval)extern void __sync_icache_dcache (pte_t pteval);#endif +#define PFN_PTE_SHIFT PAGE_SHIFT +void set_ptes (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pteval,unsigned int nr);#define set_ptes set_ptes
arch/arm64/include/asm/pgtable.h +2 -0
@@ -341,6 +341,8 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages)mte_sync_tags (pte, nr_pages); } +#define PFN_PTE_SHIFT PAGE_SHIFT +static inline void set_ptes (struct mm_struct *mm,unsigned long __always_unused addr,pte_t *ptep,pte_t pte,unsigned int nr)
Signature · 1 line
2.43.0
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
Quoted 27 lines · David Hildenbrand
We want to make use of
pte_next_pfn()outside ofset_ptes(). Let's simpliy definePFN_PTE_SHIFT, required bypte_next_pfn().Signed-off-by David Hildenbrand <[email protected]>--- arch/arm/include/asm/pgtable.h | 2 ++ arch/arm64/include/asm/pgtable.h | 2 ++ 2 files changed, 4 insertions(+)arch/arm/include/asm/pgtable.h +2
-0@@ -209,6 +209,8 @@ static inline void __sync_icache_dcache(pte_t pteval)extern void __sync_icache_dcache (pte_t pteval);#endif +#define PFN_PTE_SHIFT PAGE_SHIFT +void set_ptes (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pteval,unsigned int nr);#define set_ptes set_ptesarch/arm64/include/asm/pgtable.h +0
-0
arch/arm64/include/asm/pgtable.h static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages) · line 344
Quoted 4 lines
arch/arm64/include/asm/pgtable.h +1
-0@@ -341,6 +341,8 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages)mte_sync_tags (pte, nr_pages); } +#define PFN_PTE_SHIFT PAGE_SHIFT
I think this is buggy. And so is the arm64 implementation of set_ptes(). It works fine for 48-bit output address, but for 52-bit OAs, the high bits are not kept contigously, so if you happen to be setting a mapping for which the physical memory block straddles bit 48, this won't work.
Today, only the 64K base page config can support 52 bits, and for this, OA[51:48] are stored in PTE[15:12]. But 52 bits for 4K and 16K base pages is coming (hopefully v6.9) and in this case OA[51:50] are stored in PTE[9:8]. Fortunately we already have helpers in arm64 to abstract this.
So I think arm64 will want to define its own pte_next_pfn():
#define pte_next_pfn pte_next_pfn
static inline pte_t pte_next_pfn(pte_t pte)
{
return pfn_pte(pte_pfn(pte) + 1, pte_pgprot(pte));
}I'll do a separate patch to fix the already broken arm64 set_ptes() implementation.
I'm not sure if this type of problem might also apply to other arches?
Quoted 4 lines
arch/arm64/include/asm/pgtable.h +1
-0+static inline void set_ptes (struct mm_struct *mm,unsigned long __always_unused addr,pte_t *ptep,pte_t pte,unsigned int nr)
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
Quoted 31 lines · Ryan Roberts
Quoted 30 lines · David Hildenbrand
We want to make use of
pte_next_pfn()outside ofset_ptes(). Let's simpliy definePFN_PTE_SHIFT, required bypte_next_pfn().Signed-off-by David Hildenbrand <[email protected]>--- arch/arm/include/asm/pgtable.h | 2 ++ arch/arm64/include/asm/pgtable.h | 2 ++ 2 files changed, 4 insertions(+)arch/arm/include/asm/pgtable.h +2
-0@@ -209,6 +209,8 @@ static inline void __sync_icache_dcache(pte_t pteval)extern void __sync_icache_dcache (pte_t pteval);#endif +#define PFN_PTE_SHIFT PAGE_SHIFT +void set_ptes (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pteval,unsigned int nr);#define set_ptes set_ptesarch/arm64/include/asm/pgtable.h +0
-0@@ -341,6 +341,8 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages)mte_sync_tags (pte, nr_pages); }
Quoted 6 lines
arch/arm64/include/asm/pgtable.h static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages) · line 344
Quoted 1 line
arch/arm64/include/asm/pgtable.h +1
-0+#define PFN_PTE_SHIFT PAGE_SHIFTI think this is buggy. And so is the arm64 implementation of
set_ptes(). It works fine for 48-bit output address, but for 52-bit OAs, the high bits are not kept contigously, so if you happen to be setting a mapping for which the physical memory block straddles bit 48, this won't work.
Right, as soon as the PTE bits are not contiguous, this stops working, just like set_ptes() would, which I used as orientation.
Quoted 8 lines
Today, only the 64K base page config can support 52 bits, and for this, OA[51:48] are stored in PTE[15:12]. But 52 bits for 4K and 16K base pages is coming (hopefully v6.9) and in this case OA[51:50] are stored in PTE[9:8]. Fortunately we already have helpers in arm64 to abstract this.
So I think arm64 will want to define its own
pte_next_pfn():
Quoted 7 lines
#define
pte_next_pfnpte_next_pfn
static inlinepte_tpte_next_pfn(pte_t pte){ return pfn_pte(pte_pfn(pte) + 1, pte_pgprot(pte)); }I'll do a separate patch to fix the already broken arm64
set_ptes()implementation.
Make sense.
Quoted 2 lines
I'm not sure if this type of problem might also apply to other arches?
I saw similar handling in the PPC implementation of set_ptes, but was not able to convince me that it is actually required there.
pte_pfn on ppc does:
static inline unsigned long pte_pfn(pte_t pte)
{
return (pte_val(pte) & PTE_RPN_MASK) >> PTE_RPN_SHIFT;
}But that means that the PFNs *are* contiguous. If high bits are used for something else, then we might produce a garbage PTE on overflow, but that shouldn't really matter I concluded for folio_pte_batch() purposes, we'd not detect "belongs to this folio batch" either way.
Maybe it's likely cleaner to also have a custom pte_next_pfn() on ppc, I just hope that we don't lose any other arbitrary PTE bits by doing the pte_pgprot().
I guess pte_pfn() implementations should tell us if anything special needs to happen.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
Quoted 39 lines · David Hildenbrand
Quoted 31 lines · Ryan Roberts
Quoted 30 lines · David Hildenbrand
We want to make use of
pte_next_pfn()outside ofset_ptes(). Let's simpliy definePFN_PTE_SHIFT, required bypte_next_pfn().Signed-off-by David Hildenbrand <[email protected]>--- arch/arm/include/asm/pgtable.h | 2 ++ arch/arm64/include/asm/pgtable.h | 2 ++ 2 files changed, 4 insertions(+)arch/arm/include/asm/pgtable.h +2
-0@@ -209,6 +209,8 @@ static inline void __sync_icache_dcache(pte_t pteval)extern void __sync_icache_dcache (pte_t pteval);#endif +#define PFN_PTE_SHIFT PAGE_SHIFT +void set_ptes (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pteval,unsigned int nr);#define set_ptes set_ptesarch/arm64/include/asm/pgtable.h +0
-0@@ -341,6 +341,8 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages)mte_sync_tags (pte, nr_pages); }Quoted 6 lines
arch/arm64/include/asm/pgtable.h static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages) · line 344
Quoted 1 line
arch/arm64/include/asm/pgtable.h +1
-0+#define PFN_PTE_SHIFT PAGE_SHIFTI think this is buggy. And so is the arm64 implementation of
set_ptes(). It works fine for 48-bit output address, but for 52-bit OAs, the high bits are not kept contigously, so if you happen to be setting a mapping for which the physical memory block straddles bit 48, this won't work.
Quoted 17 lines
Right, as soon as the PTE bits are not contiguous, this stops working, just like
set_ptes()would, which I used as orientation.Quoted 14 lines
Today, only the 64K base page config can support 52 bits, and for this, OA[51:48] are stored in PTE[15:12]. But 52 bits for 4K and 16K base pages is coming (hopefully v6.9) and in this case OA[51:50] are stored in PTE[9:8]. Fortunately we already have helpers in arm64 to abstract this.
So I think arm64 will want to define its own
pte_next_pfn():#define
pte_next_pfnpte_next_pfn
static inlinepte_tpte_next_pfn(pte_t pte){ return pfn_pte(pte_pfn(pte) + 1, pte_pgprot(pte)); }
Digging into the details, on arm64 we have:
#define pte_pfn(pte) (__pte_to_phys(pte) >> PAGE_SHIFT)
and
#define __pte_to_phys(pte) (pte_val(pte) & PTE_ADDR_MASK)
But that implies, that upstream the PFN is always contiguous, no?
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
Quoted 60 lines · David Hildenbrand
Quoted 39 lines · David Hildenbrand
Quoted 29 lines · Ryan Roberts
Quoted 28 lines · David Hildenbrand
We want to make use of
pte_next_pfn()outside ofset_ptes(). Let's simpliy definePFN_PTE_SHIFT, required bypte_next_pfn().Signed-off-by David Hildenbrand <[email protected]>---
arch/arm/include/asm/pgtable.h | 2 ++
arch/arm64/include/asm/pgtable.h| 2 ++2 files changed, 4 insertions(+)arch/arm/include/asm/pgtable.h +0
-0@@ -209,6 +209,8 @@ static inline void __sync_icache_dcache(pte_t pteval) extern void __sync_icache_dcache(pte_t pteval); #endif+#define PFN_PTE_SHIFT
PAGE_SHIFT
+
voidset_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval, unsigned int nr); #defineset_ptesset_ptesarch/arm64/include/asm/pgtable.h +0
-0@@ -341,6 +341,8 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages)Quoted 8 lines
Quoted 3 lines
arch/arm64/include/asm/pgtable.h +0
-0mte_sync_tags(pte, nr_pages); }+#define PFN_PTE_SHIFT
PAGE_SHIFTI think this is buggy. And so is the arm64 implementation of
set_ptes(). It works fine for 48-bit output address, but for 52-bit OAs, the high bits are not kept contigously, so if you happen to be setting a mapping for which the physical memory block straddles bit 48, this won't work.Quoted 17 lines
Right, as soon as the PTE bits are not contiguous, this stops working, just like
set_ptes()would, which I used as orientation.Quoted 14 lines
Today, only the 64K base page config can support 52 bits, and for this, OA[51:48] are stored in PTE[15:12]. But 52 bits for 4K and 16K base pages is coming (hopefully v6.9) and in this case OA[51:50] are stored in PTE[9:8]. Fortunately we already have helpers in arm64 to abstract this.
So I think arm64 will want to define its own
pte_next_pfn():#define
pte_next_pfnpte_next_pfn
static inlinepte_tpte_next_pfn(pte_t pte)
{
returnpfn_pte(pte_pfn(pte) + 1, pte_pgprot(pte)); }Digging into the details, on arm64 we have:
Quoted 8 lines
#define pte_pfn(pte) (
__pte_to_phys(pte)>>PAGE_SHIFT)and
#define __pte_to_phys(pte) (
pte_val(pte)&PTE_ADDR_MASK)But that implies, that upstream the PFN is always contiguous, no?
But __pte_to_phys() and __phys_to_pte_val() depend on a Kconfig. If PA bits is 52, the bits are not all contiguous:
#ifdef CONFIG_ARM64_PA_BITS_52
static inline phys_addr_t __pte_to_phys(pte_t pte)
{
return (pte_val(pte) & PTE_ADDR_LOW) |
((pte_val(pte) & PTE_ADDR_HIGH) << PTE_ADDR_HIGH_SHIFT);
}
static inline pteval_t __phys_to_pte_val(phys_addr_t phys)
{
return (phys | (phys >> PTE_ADDR_HIGH_SHIFT)) & PTE_ADDR_MASK;
}#else
#define __pte_to_phys(pte) (pte_val(pte) & PTE_ADDR_MASK)
#define __phys_to_pte_val(phys) (phys)
#endif
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
Quoted 84 lines · Ryan Roberts
Quoted 60 lines · David Hildenbrand
Quoted 39 lines · David Hildenbrand
Quoted 29 lines · Ryan Roberts
Quoted 28 lines · David Hildenbrand
We want to make use of
pte_next_pfn()outside ofset_ptes(). Let's simpliy definePFN_PTE_SHIFT, required bypte_next_pfn().Signed-off-by David Hildenbrand <[email protected]>--- arch/arm/include/asm/pgtable.h | 2 ++ arch/arm64/include/asm/pgtable.h | 2 ++ 2 files changed, 4 insertions(+)arch/arm/include/asm/pgtable.h +1
-0@@ -209,6 +209,8 @@ static inline void __sync_icache_dcache(pte_t pteval) extern void __sync_icache_dcache(pte_t pteval); #endif +#define PFN_PTE_SHIFT PAGE_SHIFT + void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval, unsigned int nr); #define set_ptes set_ptesarch/arm64/include/asm/pgtable.h +0
-0@@ -341,6 +341,8 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages)Quoted 8 lines
arch/arm64/include/asm/pgtable.h static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages)
Quoted 3 lines
arch/arm64/include/asm/pgtable.h +0
-0mte_sync_tags(pte, nr_pages); } +#define PFN_PTE_SHIFT PAGE_SHIFTI think this is buggy. And so is the arm64 implementation of
set_ptes(). It works fine for 48-bit output address, but for 52-bit OAs, the high bits are not kept contigously, so if you happen to be setting a mapping for which the physical memory block straddles bit 48, this won't work.Quoted 17 lines
Right, as soon as the PTE bits are not contiguous, this stops working, just like
set_ptes()would, which I used as orientation.Quoted 14 lines
Today, only the 64K base page config can support 52 bits, and for this, OA[51:48] are stored in PTE[15:12]. But 52 bits for 4K and 16K base pages is coming (hopefully v6.9) and in this case OA[51:50] are stored in PTE[9:8]. Fortunately we already have helpers in arm64 to abstract this.
So I think arm64 will want to define its own
pte_next_pfn():#define
pte_next_pfnpte_next_pfn
static inlinepte_tpte_next_pfn(pte_t pte)
{
returnpfn_pte(pte_pfn(pte) + 1, pte_pgprot(pte));
}Digging into the details, on arm64 we have:
Quoted 8 lines
#define pte_pfn(pte) (
__pte_to_phys(pte)>>PAGE_SHIFT)and
#define __pte_to_phys(pte) (
pte_val(pte)&PTE_ADDR_MASK)But that implies, that upstream the PFN is always contiguous, no?
But
__pte_to_phys()and__phys_to_pte_val()depend on aKconfig. If PA bits is 52, the bits are not all contiguous:#ifdef
CONFIG_ARM64_PA_BITS_52
static inlinephys_addr_t__pte_to_phys(pte_t pte){ return (pte_val(pte) & PTE_ADDR_LOW) | ((pte_val(pte) & PTE_ADDR_HIGH) << PTE_ADDR_HIGH_SHIFT); } static inline pteval_t __phys_to_pte_val(phys_addr_t phys) { return (phys | (phys >> PTE_ADDR_HIGH_SHIFT)) & PTE_ADDR_MASK; }
Quoted 5 lines
#else
#define__pte_to_phys(pte)(pte_val(pte)&PTE_ADDR_MASK)
#define__phys_to_pte_val(phys)(phys)
#endif
Ah, how could I've missed that. Agreed, set_ptes() and this patch are broken.
Do you want to send a patch to implement pte_next_pfn() on arm64, and then use pte_next_pfn() in set_ptes()? Then I can drop this patch here completely from this series.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
Quoted 92 lines · David Hildenbrand
Quoted 75 lines · Ryan Roberts
Quoted 61 lines · David Hildenbrand
Quoted 40 lines · David Hildenbrand
Quoted 30 lines · Ryan Roberts
Quoted 29 lines · David Hildenbrand
We want to make use of
pte_next_pfn()outside ofset_ptes(). Let's simpliy definePFN_PTE_SHIFT, required bypte_next_pfn().Signed-off-by David Hildenbrand <[email protected]>---
arch/arm/include/asm/pgtable.h | 2 ++
arch/arm64/include/asm/pgtable.h| 2 ++2 files changed, 4 insertions(+)arch/arm/include/asm/pgtable.h +0
-0@@ -209,6 +209,8 @@ static inline void __sync_icache_dcache(pte_t pteval) extern void __sync_icache_dcache(pte_t pteval); #endif+#define PFN_PTE_SHIFT
PAGE_SHIFT
+
voidset_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval, unsigned int nr); #defineset_ptesset_ptesarch/arm64/include/asm/pgtable.h +0
-0@@ -341,6 +341,8 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages) mte_sync_tags(pte, nr_pages);Quoted 8 lines
Quoted 2 lines
arch/arm64/include/asm/pgtable.h +0
-0}+#define PFN_PTE_SHIFT
PAGE_SHIFTI think this is buggy. And so is the arm64 implementation of
set_ptes(). It works fine for 48-bit output address, but for 52-bit OAs, the high bits are not
kept contigously, so if you happen to be setting a mapping for which the physical memory block straddles bit 48, this won't work.Quoted 17 lines
Right, as soon as the PTE bits are not contiguous, this stops working, just like
set_ptes()would, which I used as orientation.Quoted 14 lines
Today, only the 64K base page config can support 52 bits, and for this, OA[51:48] are stored in PTE[15:12]. But 52 bits for 4K and 16K base pages is coming (hopefully v6.9) and in this case OA[51:50] are stored in PTE[9:8]. Fortunately we already have helpers in arm64 to abstract this.
So I think arm64 will want to define its own
pte_next_pfn():#define
pte_next_pfnpte_next_pfn
static inlinepte_tpte_next_pfn(pte_t pte)
{
returnpfn_pte(pte_pfn(pte) + 1, pte_pgprot(pte)); }Digging into the details, on arm64 we have:
Quoted 8 lines
#define pte_pfn(pte) (
__pte_to_phys(pte)>>PAGE_SHIFT)and
#define __pte_to_phys(pte) (
pte_val(pte)&PTE_ADDR_MASK)But that implies, that upstream the PFN is always contiguous, no?
But
__pte_to_phys()and__phys_to_pte_val()depend on aKconfig. If PA bits is 52, the bits are not all contiguous:Quoted 15 lines
#ifdef
CONFIG_ARM64_PA_BITS_52
static inlinephys_addr_t__pte_to_phys(pte_t pte)
{
return (pte_val(pte)&PTE_ADDR_LOW) | ((pte_val(pte)&PTE_ADDR_HIGH) <<PTE_ADDR_HIGH_SHIFT); }
static inlinepteval_t__phys_to_pte_val(phys_addr_t phys){
return (phys | (phys >>PTE_ADDR_HIGH_SHIFT)) &PTE_ADDR_MASK; }
#else
#define __pte_to_phys(pte) (pte_val(pte)&PTE_ADDR_MASK) #define __phys_to_pte_val(phys) (phys)
#endif
Quoted 5 lines
Ah, how could I've missed that. Agreed,
set_ptes()and this patch are broken.Do you want to send a patch to implement
pte_next_pfn()on arm64, and then usepte_next_pfn()inset_ptes()? Then I can drop this patch here completely from this series.
Yes good idea. I probably won't get around to it until tomorrow.
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
In reply to David HildenbrandQuoted 66 lines · David Hildenbrand
Quoted 28 lines · Ryan Roberts
Quoted 27 lines · David Hildenbrand
We want to make use of
pte_next_pfn()outside ofset_ptes(). Let's simpliy definePFN_PTE_SHIFT, required bypte_next_pfn().Signed-off-by David Hildenbrand <[email protected]>---
arch/arm/include/asm/pgtable.h | 2 ++
arch/arm64/include/asm/pgtable.h| 2 ++2 files changed, 4 insertions(+)arch/arm/include/asm/pgtable.h +0
-0@@ -209,6 +209,8 @@ static inline void __sync_icache_dcache(pte_t pteval) extern void __sync_icache_dcache(pte_t pteval); #endif+#define PFN_PTE_SHIFT
PAGE_SHIFT
+
voidset_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval, unsigned int nr); #defineset_ptesset_ptesarch/arm64/include/asm/pgtable.h +0
-0@@ -341,6 +341,8 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages)Quoted 8 lines
Quoted 3 lines
arch/arm64/include/asm/pgtable.h +0
-0mte_sync_tags(pte, nr_pages); }+#define PFN_PTE_SHIFT
PAGE_SHIFTI think this is buggy. And so is the arm64 implementation of
set_ptes(). It works fine for 48-bit output address, but for 52-bit OAs, the high bits are not kept contigously, so if you happen to be setting a mapping for which the physical memory block straddles bit 48, this won't work.Right, as soon as the PTE bits are not contiguous, this stops working, just like
set_ptes()would, which I used as orientation.Quoted 8 lines
Today, only the 64K base page config can support 52 bits, and for this, OA[51:48] are stored in PTE[15:12]. But 52 bits for 4K and 16K base pages is coming (hopefully v6.9) and in this case OA[51:50] are stored in PTE[9:8]. Fortunately we already have helpers in arm64 to abstract this.
So I think arm64 will want to define its own
pte_next_pfn():Quoted 8 lines
#define
pte_next_pfnpte_next_pfn
static inlinepte_tpte_next_pfn(pte_t pte)
{
returnpfn_pte(pte_pfn(pte) + 1, pte_pgprot(pte)); }I'll do a separate patch to fix the already broken arm64
set_ptes()implementation.Make sense.
Quoted 2 lines
I'm not sure if this type of problem might also apply to other arches?
I saw similar handling in the PPC implementation of
set_ptes, but was not able to convince me that it is actually required there.
Quoted 8 lines
pte_pfnon ppc does:static inline unsigned long
pte_pfn(pte_t pte)
{
return (pte_val(pte)&PTE_RPN_MASK) >>PTE_RPN_SHIFT; }But that means that the PFNs *are* contiguous.
all the ppc pfn_pte() implementations also only shift the pfn, so I think ppc is safe to just define PFN_PTE_SHIFT. Although 2 of the 3 implementations shift by PTE_RPN_SHIFT and the other shifts by PAGE_SIZE, so you might want to define PFN_PTE_SHIFT separately for all 3 configs?
Quoted 4 lines
If high bits are used for
something else, then we might produce a garbage PTE on overflow, but that shouldn't really matter I concluded forfolio_pte_batch()purposes, we'd not detect "belongs to this folio batch" either way.
Exactly.
Quoted 3 lines
Maybe it's likely cleaner to also have a custom
pte_next_pfn()on ppc, I just hope that we don't lose any other arbitrary PTE bits by doing thepte_pgprot().
I don't see the need for ppc to implement pte_next_pfn().
pte_pgprot() is not a "proper" arch interface (its only required by the core-mm if the arch implements a certain Kconfig IIRC). For arm64, all bits that are not pfn are pgprot, so there are no bits lost.
Quoted 5 lines
I guess
pte_pfn()implementations should tell us if anything special needs to happen.
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
Le 23/01/2024 à 12:08, Ryan Roberts a écrit :
Quoted 74 lines · David Hildenbrand
Quoted 73 lines · Ryan Roberts
Quoted 28 lines · David Hildenbrand
Quoted 27 lines
We want to make use of
pte_next_pfn()outside ofset_ptes(). Let's simpliy definePFN_PTE_SHIFT, required bypte_next_pfn().Signed-off-by David Hildenbrand <[email protected]>--- arch/arm/include/asm/pgtable.h | 2 ++ arch/arm64/include/asm/pgtable.h | 2 ++ 2 files changed, 4 insertions(+)arch/arm/include/asm/pgtable.h +1
-0@@ -209,6 +209,8 @@ static inline void __sync_icache_dcache(pte_t pteval) extern void __sync_icache_dcache(pte_t pteval); #endif +#define PFN_PTE_SHIFT PAGE_SHIFT + void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval, unsigned int nr); #define set_ptes set_ptesarch/arm64/include/asm/pgtable.h +0
-0@@ -341,6 +341,8 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages)Quoted 8 lines
arch/arm64/include/asm/pgtable.h static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages)
Quoted 3 lines
arch/arm64/include/asm/pgtable.h +0
-0mte_sync_tags(pte, nr_pages); } +#define PFN_PTE_SHIFT PAGE_SHIFTI think this is buggy. And so is the arm64 implementation of
set_ptes(). It works fine for 48-bit output address, but for 52-bit OAs, the high bits are not kept contigously, so if you happen to be setting a mapping for which the physical memory block straddles bit 48, this won't work.Right, as soon as the PTE bits are not contiguous, this stops working, just like
set_ptes()would, which I used as orientation.Quoted 8 lines
Today, only the 64K base page config can support 52 bits, and for this, OA[51:48] are stored in PTE[15:12]. But 52 bits for 4K and 16K base pages is coming (hopefully v6.9) and in this case OA[51:50] are stored in PTE[9:8]. Fortunately we already have helpers in arm64 to abstract this.
So I think arm64 will want to define its own
pte_next_pfn():Quoted 8 lines
#define
pte_next_pfnpte_next_pfn
static inlinepte_tpte_next_pfn(pte_t pte)
{
returnpfn_pte(pte_pfn(pte) + 1, pte_pgprot(pte));
}I'll do a separate patch to fix the already broken arm64
set_ptes()implementation.Make sense.
Quoted 2 lines
I'm not sure if this type of problem might also apply to other arches?
I saw similar handling in the PPC implementation of
set_ptes, but was not able to convince me that it is actually required there.
pte_pfnon ppc does:static inline unsigned long
pte_pfn(pte_t pte)
{
return (pte_val(pte)&PTE_RPN_MASK) >>PTE_RPN_SHIFT;
}
Quoted 6 lines
Quoted 1 line
But that means that the PFNs *are* contiguous.
all the ppc
pfn_pte()implementations also only shift the pfn, so I think ppc is safe to just definePFN_PTE_SHIFT. Although 2 of the 3 implementations shift byPTE_RPN_SHIFTand the other shifts byPAGE_SIZE, so you might want to definePFN_PTE_SHIFTseparately for all 3 configs?
We have PTE_RPN_SHIFT defined for all 4 implementations, for some of them you are right it is defined as PAGE_SHIFT, but I see no reason to define PFN_PTE_SHIFT separately.
Quoted 6 lines
Quoted 5 lines
If high bits are used for
something else, then we might produce a garbage PTE on overflow, but that shouldn't really matter I concluded forfolio_pte_batch()purposes, we'd not detect "belongs to this folio batch" either way.
Quoted 7 lines
Exactly.
Quoted 3 lines
Maybe it's likely cleaner to also have a custom
pte_next_pfn()on ppc, I just hope that we don't lose any other arbitrary PTE bits by doing thepte_pgprot().I don't see the need for ppc to implement
pte_next_pfn().
Agreed.
Quoted 11 lines
pte_pgprot()is not a "proper" arch interface (its only required by the core-mm if the arch implements a certainKconfigIIRC). For arm64, all bits that are not pfn are pgprot, so there are no bits lost.Quoted 5 lines
I guess
pte_pfn()implementations should tell us if anything special needs to happen.
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
If high bits are used for
something else, then we might produce a garbage PTE on overflow, but that shouldn't really matter I concluded forfolio_pte_batch()purposes, we'd not detect "belongs to this folio batch" either way.Exactly.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Maybe it's likely cleaner to also have a custom
pte_next_pfn()on ppc, I just hope that we don't lose any other arbitrary PTE bits by doing thepte_pgprot().I don't see the need for ppc to implement
pte_next_pfn().Agreed.
So likely we should then do on top for powerpc (whitespace damage):
arch/powerpc/mm/pgtable.c +1 -4
@@ -220,10 +220,7 @@ void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep,break ; ptep++; addr +=PAGE_SIZE ; -/* -* increment the pfn. -*/ - pte =pfn_pte (pte_pfn (pte) +1 ,pte_pgprot ((pte))); + pte =pte_next_pfn (pte); } }
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
Quoted 24 lines · David Hildenbrand
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
If high bits are used for
something else, then we might produce a garbage PTE on overflow, but that shouldn't really matter I concluded forfolio_pte_batch()purposes, we'd not detect "belongs to this folio batch" either way.Exactly.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Maybe it's likely cleaner to also have a custom
pte_next_pfn()on ppc, I just hope that we don't lose any other arbitrary PTE bits by doing thepte_pgprot().I don't see the need for ppc to implement
pte_next_pfn().Agreed.
So likely we should then do on top for powerpc (whitespace damage):
arch/powerpc/mm/pgtable.c +0
-0@@ -220,10 +220,7 @@ void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep,
arch/powerpc/mm/pgtable.c void set_ptes(struct mm_struct *mm, unsigned long addr, · line 226
Quoted 10 lines
arch/powerpc/mm/pgtable.c +1
-4break; ptep++; addr += PAGE_SIZE; - /* - * increment the pfn. - */ - pte = pfn_pte(pte_pfn(pte) + 1, pte_pgprot((pte))); + pte = pte_next_pfn(pte); } }
Looks like commit 47b8def9358c ("powerpc/mm: Avoid calling arch_enter/leave_lazy_mmu() in set_ptes") changed from doing the simple increment to this more complex approach, but the log doesn't say why.
Quoted 3 lines
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
Quoted 31 lines · Ryan Roberts
Quoted 30 lines · David Hildenbrand
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
If high bits are used for
something else, then we might produce a garbage PTE on overflow, but that shouldn't really matter I concluded forfolio_pte_batch()purposes, we'd not detect "belongs to this folio batch" either way.Exactly.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Maybe it's likely cleaner to also have a custom
pte_next_pfn()on ppc, I just hope that we don't lose any other arbitrary PTE bits by doing thepte_pgprot().I don't see the need for ppc to implement
pte_next_pfn().Agreed.
So likely we should then do on top for powerpc (whitespace damage):
arch/powerpc/mm/pgtable.c +0
-3@@ -220,10 +220,7 @@ void set_ptes(struct mm_struct *mm, unsigned long addr,pte_t *ptep, break; ptep++; addr += PAGE_SIZE; - /* -* increment the pfn. -*/
Quoted 8 lines
arch/powerpc/mm/pgtable.c void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, · line 225
Quoted 4 lines
arch/powerpc/mm/pgtable.c +1
-1- pte = pfn_pte(pte_pfn(pte) + 1, pte_pgprot((pte))); + pte = pte_next_pfn(pte); } }Looks like commit
47b8def9358c("powerpc/mm: Avoid calling arch_enter/leave_lazy_mmu() inset_ptes") changed from doing the simple increment to this more complex approach, but the log doesn't say why.
@Aneesh, was that change on purpose?
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
Quoted 41 lines · David Hildenbrand
Quoted 31 lines · Ryan Roberts
Quoted 30 lines · David Hildenbrand
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
If high bits are used for
something else, then we might produce a garbage PTE on overflow, but that shouldn't really matter I concluded forfolio_pte_batch()purposes, we'd not detect "belongs to this folio batch" either way.Exactly.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Maybe it's likely cleaner to also have a custom
pte_next_pfn()on ppc, I just hope that we don't lose any other arbitrary PTE bits by doing thepte_pgprot().I don't see the need for ppc to implement
pte_next_pfn().Agreed.
So likely we should then do on top for powerpc (whitespace damage):
arch/powerpc/mm/pgtable.c +0
-3@@ -220,10 +220,7 @@ void set_ptes(struct mm_struct *mm, unsigned long addr,pte_t *ptep, break; ptep++; addr += PAGE_SIZE; - /* -* increment the pfn. -*/ Quoted 8 lines
arch/powerpc/mm/pgtable.c void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, · line 225
Quoted 4 lines
arch/powerpc/mm/pgtable.c +1
-1- pte = pfn_pte(pte_pfn(pte) + 1, pte_pgprot((pte))); + pte = pte_next_pfn(pte); } }Looks like commit
47b8def9358c("powerpc/mm: Avoid calling arch_enter/leave_lazy_mmu() inset_ptes") changed from doing the simple increment to this more complex approach, but the log doesn't say why.
Quoted 2 lines
@Aneesh, was that change on purpose?
Because we had a bug with the patch that introduced the change and that line was confusing. The right thing should have been to add pte_pfn_next() to make it clear. It was confusing because not all pte format had pfn at PAGE_SHIFT offset (even though we did use the correct PTE_RPN_SHIFT in this specific case). To make it simpler I ended up switching that line to pte_pfn(pte) + 1 .
-aneesh
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
In reply to Ryan RobertsLe 23/01/2024 à 12:38, Ryan Roberts a écrit :
Quoted 31 lines · David Hildenbrand
Quoted 30 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
If high bits are used for
something else, then we might produce a garbage PTE on overflow, but that shouldn't really matter I concluded forfolio_pte_batch()purposes, we'd not detect "belongs to this folio batch" either way.Exactly.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Maybe it's likely cleaner to also have a custom
pte_next_pfn()on ppc, I just hope that we don't lose any other arbitrary PTE bits by doing thepte_pgprot().I don't see the need for ppc to implement
pte_next_pfn().Agreed.
So likely we should then do on top for powerpc (whitespace damage):
arch/powerpc/mm/pgtable.c +0
-3@@ -220,10 +220,7 @@ void set_ptes(struct mm_struct *mm, unsigned long addr,pte_t *ptep, break; ptep++; addr += PAGE_SIZE; - /* -* increment the pfn. -*/
Quoted 8 lines
arch/powerpc/mm/pgtable.c void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, · line 225
Quoted 4 lines
arch/powerpc/mm/pgtable.c +1
-1- pte = pfn_pte(pte_pfn(pte) + 1, pte_pgprot((pte))); + pte = pte_next_pfn(pte); } }Looks like commit
47b8def9358c("powerpc/mm: Avoid calling arch_enter/leave_lazy_mmu() inset_ptes") changed from doing the simple increment to this more complex approach, but the log doesn't say why.
Right. There was a discussion about it without any conclusion:
https://patchwork.ozlabs.org/project/linuxppc-dev/patch/[email protected]/
As far as understand the simple increment is better on ppc/32 but worse in ppc/64.
Christophe
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
Quoted 43 lines · Christophe Leroy
Le 23/01/2024 à 12:38, Ryan Roberts a écrit :
Quoted 31 lines · David Hildenbrand
Quoted 30 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
If high bits are used for
something else, then we might produce a garbage PTE on overflow, but that shouldn't really matter I concluded forfolio_pte_batch()purposes, we'd not detect "belongs to this folio batch" either way.Exactly.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Maybe it's likely cleaner to also have a custom
pte_next_pfn()on ppc, I just hope that we don't lose any other arbitrary PTE bits by doing thepte_pgprot().I don't see the need for ppc to implement
pte_next_pfn().Agreed.
So likely we should then do on top for powerpc (whitespace damage):
arch/powerpc/mm/pgtable.c +0
-3@@ -220,10 +220,7 @@ void set_ptes(struct mm_struct *mm, unsigned long addr,pte_t *ptep, break; ptep++; addr += PAGE_SIZE; - /* -* increment the pfn. -*/ Quoted 8 lines
arch/powerpc/mm/pgtable.c void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, · line 225
Quoted 4 lines
arch/powerpc/mm/pgtable.c +1
-1- pte = pfn_pte(pte_pfn(pte) + 1, pte_pgprot((pte))); + pte = pte_next_pfn(pte); } }Looks like commit
47b8def9358c("powerpc/mm: Avoid calling arch_enter/leave_lazy_mmu() inset_ptes") changed from doing the simple increment to this more complex approach, but the log doesn't say why.
Quoted 5 lines
Right. There was a discussion about it without any conclusion:
https://patchwork.ozlabs.org/project/linuxppc-dev/patch/[email protected]/As far as understand the simple increment is better on ppc/32 but worse in ppc/64.
Sounds like we're micro-optimizing for a specific compiler version output. Hurray.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
In reply to David HildenbrandQuoted 23 lines · David Hildenbrand
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
If high bits are used for
something else, then we might produce a garbage PTE on overflow, but that shouldn't really matter I concluded forfolio_pte_batch()purposes, we'd not detect "belongs to this folio batch" either way.Exactly.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Maybe it's likely cleaner to also have a custom
pte_next_pfn()on ppc, I just hope that we don't lose any other arbitrary PTE bits by doing thepte_pgprot().I don't see the need for ppc to implement
pte_next_pfn().Agreed.
So likely we should then do on top for powerpc (whitespace damage):
arch/powerpc/mm/pgtable.c +0
-0
arch/powerpc/mm/pgtable.c void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, · line 225
Quoted 10 lines
arch/powerpc/mm/pgtable.c +1
-4@@ -220,10 +220,7 @@ void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep,break ; ptep++; addr +=PAGE_SIZE ; -/* -* increment the pfn. -*/ - pte =pfn_pte (pte_pfn (pte) +1 ,pte_pgprot ((pte))); + pte =pte_next_pfn (pte); } }
Agreed.
-aneesh
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
In reply to David HildenbrandLe 23/01/2024 à 11:48, David Hildenbrand a écrit :
Quoted 80 lines · Ryan Roberts
Quoted 33 lines · David Hildenbrand
Quoted 32 lines
We want to make use of
pte_next_pfn()outside ofset_ptes(). Let's simpliy definePFN_PTE_SHIFT, required bypte_next_pfn().Signed-off-by David Hildenbrand <[email protected]>---
arch/arm/include/asm/pgtable.h | 2 ++
arch/arm64/include/asm/pgtable.h| 2 ++2 files changed, 4 insertions(+)arch/arm/include/asm/pgtable.h +2
-0@@ -209,6 +209,8 @@ static inline void __sync_icache_dcache(pte_t pteval) extern void __sync_icache_dcache(pte_t pteval); #endif +#define PFN_PTE_SHIFT PAGE_SHIFT + void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval, unsigned int nr); #define set_ptes set_ptesarch/arm64/include/asm/pgtable.h +0
-0@@ -341,6 +341,8 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages) mte_sync_tags(pte, nr_pages); }Quoted 8 lines
arch/arm64/include/asm/pgtable.h static inline void __sync_cache_and_tags(pte_t · line 344
Quoted 1 line
arch/arm64/include/asm/pgtable.h +1
-0+#define PFN_PTE_SHIFT PAGE_SHIFTI think this is buggy. And so is the arm64 implementation of
set_ptes(). It
works fine for 48-bit output address, but for 52-bit OAs, the high bits are not
kept contigously, so if you happen to be setting a mapping for which the physical memory block straddles bit 48, this won't work.Right, as soon as the PTE bits are not contiguous, this stops working, just like
set_ptes()would, which I used as orientation.Quoted 10 lines
Today, only the 64K base page config can support 52 bits, and for this, OA[51:48] are stored in PTE[15:12]. But 52 bits for 4K and 16K base pages is
coming (hopefully v6.9) and in this case OA[51:50] are stored in PTE[9:8].
Fortunately we already have helpers in arm64 to abstract this.So I think arm64 will want to define its own
pte_next_pfn():Quoted 8 lines
#define
pte_next_pfnpte_next_pfn
static inlinepte_tpte_next_pfn(pte_t pte)
{
returnpfn_pte(pte_pfn(pte) + 1, pte_pgprot(pte)); }I'll do a separate patch to fix the already broken arm64
set_ptes()implementation.Make sense.
Quoted 2 lines
I'm not sure if this type of problem might also apply to other arches?
I saw similar handling in the PPC implementation of
set_ptes, but was not able to convince me that it is actually required there.
pte_pfnon ppc does:static inline unsigned long
pte_pfn(pte_t pte)
{
return (pte_val(pte)&PTE_RPN_MASK) >>PTE_RPN_SHIFT;
}
Quoted 4 lines
But that means that the PFNs *are* contiguous. If high bits are used for something else, then we might produce a garbage PTE on overflow, but that shouldn't really matter I concluded for
folio_pte_batch()purposes, we'd not detect "belongs to this folio batch" either way.
Yes PFNs are contiguous. The only thing is that the PFN is not located at PAGE_SHIFT, see
https://elixir.bootlin.com/linux/v6.3-rc2/source/arch/powerpc/include/asm/nohash/pte-e500.h#L63
On powerpc e500 we have 24 PTE flags and the RPN starts above that.
The mask is then standard:
#define PTE_RPN_MASK (~((1ULL << PTE_RPN_SHIFT) - 1))
Christophe
Quoted 9 lines
Maybe it's likely cleaner to also have a custom
pte_next_pfn()on ppc, I just hope that we don't lose any other arbitrary PTE bits by doing thepte_pgprot().I guess
pte_pfn()implementations should tell us if anything special needs to happen.
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
In reply to Ryan RobertsQuoted 6 lines · Ryan Roberts
Quoted 1 line
+#define
PFN_PTE_SHIFTPAGE_SHIFTI think this is buggy. And so is the arm64 implementation of
set_ptes(). It works fine for 48-bit output address, but for 52-bit OAs, the high bits are not kept contigously, so if you happen to be setting a mapping for which the physical memory block straddles bit 48, this won't work.
I'd like to see the folio allocation that can straddle bit 48 ...
agreed, it's not workable _in general_, but specifically for a memory allocation from a power-of-two allocator, you'd have to do a 49-bit allocation (half a petabyte) to care.
Re: [PATCH v1 01/11] arm/pgtable: define PFN_PTE_SHIFT on arm and arm64
Quoted 8 lines · Matthew Wilcox
Quoted 6 lines · Ryan Roberts
Quoted 1 line
+#define
PFN_PTE_SHIFTPAGE_SHIFTI think this is buggy. And so is the arm64 implementation of
set_ptes(). It works fine for 48-bit output address, but for 52-bit OAs, the high bits are not kept contigously, so if you happen to be setting a mapping for which the physical memory block straddles bit 48, this won't work.
Quoted 5 lines
I'd like to see the folio allocation that can straddle bit 48 ...
agreed, it's not workable
_ingeneral_, but specifically for a memory allocation from a power-of-two allocator, you'd have to do a 49-bit allocation (half a petabyte) to care.
Hmm good point. So its a hypothetical bug, not an actual bug. Personally I'm still inclined to "fix" it. Although its going to cost a few more instructions. Shout if you disagree.
PATCH 2/11 nios2/pgtable: define PFN_PTE_SHIFT
In reply to David HildenbrandWe want to make use of pte_next_pfn() outside of set_ptes(). Let's simpliy define PFN_PTE_SHIFT, required by pte_next_pfn().
--- arch/nios2/include/asm/pgtable.h | 2 ++ 1 file changed, 2 insertions(+)
arch/nios2/include/asm/pgtable.h +2 -0
@@ -178,6 +178,8 @@ static inline void set_pte(pte_t *ptep, pte_t pteval) *ptep = pteval; } +#define PFN_PTE_SHIFT 0 +static inline void set_ptes (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pte,unsigned int nr) {
Signature · 1 line
2.43.0
PATCH 3/11 powerpc/pgtable: define PFN_PTE_SHIFT
In reply to David HildenbrandWe want to make use of pte_next_pfn() outside of set_ptes(). Let's simpliy define PFN_PTE_SHIFT, required by pte_next_pfn().
--- arch/powerpc/include/asm/pgtable.h | 2 ++ 1 file changed, 2 insertions(+)
arch/powerpc/include/asm/pgtable.h +2 -0
@@ -41,6 +41,8 @@ struct mm_struct;#ifndef __ASSEMBLY__ +#define PFN_PTE_SHIFT PTE_RPN_SHIFT +void set_ptes (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pte,unsigned int nr);#define set_ptes set_ptes
Signature · 1 line
2.43.0
PATCH 4/11 risc: pgtable: define PFN_PTE_SHIFT
In reply to David HildenbrandWe want to make use of pte_next_pfn() outside of set_ptes(). Let's simpliy define PFN_PTE_SHIFT, required by pte_next_pfn().
--- arch/riscv/include/asm/pgtable.h | 2 ++ 1 file changed, 2 insertions(+)
arch/riscv/include/asm/pgtable.h +2 -0
@@ -523,6 +523,8 @@ static inline void __set_pte_at(pte_t *ptep, pte_t pteval)set_pte (ptep, pteval); } +#define PFN_PTE_SHIFT _PAGE_PFN_SHIFT +static inline void set_ptes (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pteval,unsigned int nr) {
Signature · 1 line
2.43.0
Re: [PATCH v1 04/11] risc: pgtable: define PFN_PTE_SHIFT
Hi David,
Quoted 13 lines · David Hildenbrand
We want to make use of
pte_next_pfn()outside ofset_ptes(). Let's simpliy definePFN_PTE_SHIFT, required bypte_next_pfn().Signed-off-by David Hildenbrand <[email protected]>--- arch/riscv/include/asm/pgtable.h | 2 ++ 1 file changed, 2 insertions(+)arch/riscv/include/asm/pgtable.h +0
-0
arch/riscv/include/asm/pgtable.h static inline void __set_pte_at(pte_t *ptep, pte_t pteval) · line 530
Quoted 8 lines
arch/riscv/include/asm/pgtable.h +2
-0@@ -523,6 +523,8 @@ static inline void __set_pte_at(pte_t *ptep, pte_t pteval)set_pte (ptep, pteval); } +#define PFN_PTE_SHIFT _PAGE_PFN_SHIFT +static inline void set_ptes (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pteval,unsigned int nr) {
There is a typo in the commit title: risc -> riscv. Otherwise, this is right so:
Thanks,
Alex
Re: [PATCH v1 04/11] risc: pgtable: define PFN_PTE_SHIFT
Quoted 19 lines · Alexandre Ghiti
Hi David,
Quoted 16 lines · David Hildenbrand
We want to make use of
pte_next_pfn()outside ofset_ptes(). Let's simpliy definePFN_PTE_SHIFT, required bypte_next_pfn().Signed-off-by David Hildenbrand <[email protected]>--- arch/riscv/include/asm/pgtable.h | 2 ++ 1 file changed, 2 insertions(+)arch/riscv/include/asm/pgtable.h +0
-0@@ -523,6 +523,8 @@ static inline void __set_pte_at(pte_t *ptep, pte_t pteval)set_pte (ptep, pteval); }
Quoted 9 lines
arch/riscv/include/asm/pgtable.h static inline void __set_pte_at(pte_t *ptep, pte_t pteval) · line 530
Quoted 5 lines
arch/riscv/include/asm/pgtable.h +2
-0+#define PFN_PTE_SHIFT _PAGE_PFN_SHIFT +static inline void set_ptes (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pteval,unsigned int nr) {There is a typo in the commit title: risc -> riscv. Otherwise, this is right so:
Whops :)
Quoted 2 lines
Reviewed-by Alexandre Ghiti <[email protected]>
Thanks!
Signature · 3 lines
Cheers, David / dhildenb
PATCH 5/11 s390/pgtable: define PFN_PTE_SHIFT
In reply to David HildenbrandWe want to make use of pte_next_pfn() outside of set_ptes(). Let's simpliy define PFN_PTE_SHIFT, required by pte_next_pfn().
--- arch/s390/include/asm/pgtable.h | 2 ++ 1 file changed, 2 insertions(+)
arch/s390/include/asm/pgtable.h +2 -0
@@ -1316,6 +1316,8 @@ pgprot_t pgprot_writecombine(pgprot_t prot);#define pgprot_writethrough pgprot_writethroughpgprot_t pgprot_writethrough (pgprot_t prot); +#define PFN_PTE_SHIFT PAGE_SHIFT +/* * Set multiple PTEs to consecutive pages with a single call. All PTEs * are within the same folio, PMD and VMA.
Signature · 1 line
2.43.0
PATCH 6/11 sparc/pgtable: define PFN_PTE_SHIFT
In reply to David HildenbrandWe want to make use of pte_next_pfn() outside of set_ptes(). Let's simpliy define PFN_PTE_SHIFT, required by pte_next_pfn().
--- arch/sparc/include/asm/pgtable_64.h | 2 ++ 1 file changed, 2 insertions(+)
arch/sparc/include/asm/pgtable_64.h +2 -0
@@ -929,6 +929,8 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr,maybe_tlb_batch_add (mm, addr, ptep, orig, fullmm,PAGE_SHIFT ); } +#define PFN_PTE_SHIFT PAGE_SHIFT +static inline void set_ptes (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pte,unsigned int nr) {
Signature · 1 line
2.43.0
PATCH 7/11 mm/memory: factor out copying the actual PTE in copy_present_pte()
In reply to David HildenbrandLet's prepare for further changes.
--- mm/memory.c | 60 ++++++++++++++++++++++++++++------------------------- 1 file changed, 32 insertions(+), 28 deletions(-)
mm/memory.c +32 -28
@@ -930,6 +930,29 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vmareturn 0 ; } +static inline void __copy_present_pte (struct vm_area_struct *dst_vma, +struct vm_area_struct *src_vma,pte_t *dst_pte,pte_t *src_pte, +pte_t pte,unsigned long addr) +{ +struct mm_struct *src_mm = src_vma->vm_mm; + +/* If it's a COW mapping, write protect it both processes. */ +if (is_cow_mapping (src_vma->vm_flags) &&pte_write (pte)) { +ptep_set_wrprotect (src_mm, addr, src_pte); + pte =pte_wrprotect (pte); + } + +/* If it's a shared mapping, mark it clean in the child. */ +if (src_vma->vm_flags &VM_SHARED ) + pte =pte_mkclean (pte); + pte =pte_mkold (pte); + +if (!userfaultfd_wp (dst_vma)) + pte =pte_clear_uffd_wp (pte); + +set_pte_at (dst_vma->vm_mm, addr, dst_pte, pte); +} +/* * Copy one pte. Returns 0 if succeeded, or -EAGAIN if one preallocated page * is required to copy this pte.
@@ -939,16 +962,16 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, pte_t *dst_pte, pte_t *src_pte, unsigned long addr, int *rss, struct folio **prealloc) { -struct mm_struct *src_mm = src_vma->vm_mm; -unsigned long vm_flags = src_vma->vm_flags;pte_t pte =ptep_get (src_pte);struct page *page;struct folio *folio; page =vm_normal_page (src_vma, addr, pte); -if (page) - folio =page_folio (page); -if (page &&folio_test_anon (folio)) { +if (unlikely (!page)) + goto copy_pte; + + folio =page_folio (page); +if (folio_test_anon (folio)) {/* * If this page may have been pinned by the parent process, * copy the page immediately for the child so that we'll always
@@ -963,34 +986,15 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, addr, rss, prealloc, page); } rss[MM_ANONPAGES ]++; - }else if (page) { +VM_WARN_ON_FOLIO (PageAnonExclusive (page), folio); + }else {folio_get (folio);folio_dup_file_rmap_pte (folio, page); rss[mm_counter_file (page)]++; } -/* -* If it's a COW mapping, write protect it both -* in the parent and the child -*/ -if (is_cow_mapping (vm_flags) &&pte_write (pte)) { -ptep_set_wrprotect (src_mm, addr, src_pte); - pte =pte_wrprotect (pte); - } -VM_BUG_ON (page &&folio_test_anon (folio) &&PageAnonExclusive (page)); - -/* -* If it's a shared mapping, mark it clean in -* the child -*/ -if (vm_flags &VM_SHARED ) - pte =pte_mkclean (pte); - pte =pte_mkold (pte); - -if (!userfaultfd_wp (dst_vma)) - pte =pte_clear_uffd_wp (pte); - -set_pte_at (dst_vma->vm_mm, addr, dst_pte, pte); +copy_pte: +__copy_present_pte (dst_vma, src_vma, dst_pte, src_pte, pte, addr);return 0 ; }
Signature · 1 line
2.43.0
Re: [PATCH v1 07/11] mm/memory: factor out copying the actual PTE in copy_present_pte()
Quoted 3 lines · David Hildenbrand
Let's prepare for further changes.
Signed-off-by David Hildenbrand <[email protected]>
Quoted 99 lines
--- mm/memory.c | 60 ++++++++++++++++++++++++++++------------------------- 1 file changed, 32 insertions(+), 28 deletions(-)mm/memory.c +32
-28@@ -930,6 +930,29 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vmareturn 0 ; } +static inline void __copy_present_pte (struct vm_area_struct *dst_vma, +struct vm_area_struct *src_vma,pte_t *dst_pte,pte_t *src_pte, +pte_t pte,unsigned long addr) +{ +struct mm_struct *src_mm = src_vma->vm_mm; + +/* If it's a COW mapping, write protect it both processes. */ +if (is_cow_mapping (src_vma->vm_flags) &&pte_write (pte)) { +ptep_set_wrprotect (src_mm, addr, src_pte); + pte =pte_wrprotect (pte); + } + +/* If it's a shared mapping, mark it clean in the child. */ +if (src_vma->vm_flags &VM_SHARED ) + pte =pte_mkclean (pte); + pte =pte_mkold (pte); + +if (!userfaultfd_wp (dst_vma)) + pte =pte_clear_uffd_wp (pte); + +set_pte_at (dst_vma->vm_mm, addr, dst_pte, pte); +} +/* * Copy one pte. Returns 0 if succeeded, or -EAGAIN if one preallocated page * is required to copy this pte. @@ -939,16 +962,16 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, pte_t *dst_pte, pte_t *src_pte, unsigned long addr, int *rss, struct folio **prealloc) { -struct mm_struct *src_mm = src_vma->vm_mm; -unsigned long vm_flags = src_vma->vm_flags;pte_t pte =ptep_get (src_pte);struct page *page;struct folio *folio; page =vm_normal_page (src_vma, addr, pte); -if (page) - folio =page_folio (page); -if (page &&folio_test_anon (folio)) { +if (unlikely (!page)) + goto copy_pte; + + folio =page_folio (page); +if (folio_test_anon (folio)) {/* * If this page may have been pinned by the parent process, * copy the page immediately for the child so that we'll always @@ -963,34 +986,15 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, addr, rss, prealloc, page); } rss[MM_ANONPAGES ]++; - }else if (page) { +VM_WARN_ON_FOLIO (PageAnonExclusive (page), folio); + }else {folio_get (folio);folio_dup_file_rmap_pte (folio, page); rss[mm_counter_file (page)]++; } -/* -* If it's a COW mapping, write protect it both -* in the parent and the child -*/ -if (is_cow_mapping (vm_flags) &&pte_write (pte)) { -ptep_set_wrprotect (src_mm, addr, src_pte); - pte =pte_wrprotect (pte); - } -VM_BUG_ON (page &&folio_test_anon (folio) &&PageAnonExclusive (page)); - -/* -* If it's a shared mapping, mark it clean in -* the child -*/ -if (vm_flags &VM_SHARED ) - pte =pte_mkclean (pte); - pte =pte_mkold (pte); - -if (!userfaultfd_wp (dst_vma)) - pte =pte_clear_uffd_wp (pte); - -set_pte_at (dst_vma->vm_mm, addr, dst_pte, pte); +copy_pte: +__copy_present_pte (dst_vma, src_vma, dst_pte, src_pte, pte, addr);return 0 ; }
PATCH 8/11 mm/memory: pass PTE to copy_present_pte()
In reply to David HildenbrandWe already read it, let's just forward it.
This patch is based on work by Ryan Roberts.
--- mm/memory.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
mm/memory.c +3 -4
@@ -959,10 +959,9 @@ static inline void __copy_present_pte(struct vm_area_struct *dst_vma,*/ static inline int copy_present_pte (struct vm_area_struct *dst_vma,struct vm_area_struct *src_vma, -pte_t *dst_pte,pte_t *src_pte,unsigned long addr,int *rss, -struct folio **prealloc) +pte_t *dst_pte,pte_t *src_pte,pte_t pte,unsigned long addr, +int *rss,struct folio **prealloc) { -pte_t pte =ptep_get (src_pte);struct page *page;struct folio *folio;
@@ -1104,7 +1103,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, }/* copy_present_pte() will clear `*prealloc' if consumed */ ret =copy_present_pte (dst_vma, src_vma, dst_pte, src_pte, - addr, rss, &prealloc); + ptent, addr, rss, &prealloc);/* * If we need a pre-allocated page for this pte, drop the * locks, allocate, and try again.
Signature · 1 line
2.43.0
Re: [PATCH v1 08/11] mm/memory: pass PTE to copy_present_pte()
Quoted 5 lines · David Hildenbrand
We already read it, let's just forward it.
This patch is based on work by Ryan Roberts.
Signed-off-by David Hildenbrand <[email protected]>
Quoted 30 lines
--- mm/memory.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)mm/memory.c +3
-4@@ -959,10 +959,9 @@ static inline void __copy_present_pte(struct vm_area_struct *dst_vma,*/ static inline int copy_present_pte (struct vm_area_struct *dst_vma,struct vm_area_struct *src_vma, -pte_t *dst_pte,pte_t *src_pte,unsigned long addr,int *rss, -struct folio **prealloc) +pte_t *dst_pte,pte_t *src_pte,pte_t pte,unsigned long addr, +int *rss,struct folio **prealloc) { -pte_t pte =ptep_get (src_pte);struct page *page;struct folio *folio;@@ -1104,7 +1103,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, }/* copy_present_pte() will clear `*prealloc' if consumed */ ret =copy_present_pte (dst_vma, src_vma, dst_pte, src_pte, - addr, rss, &prealloc); + ptent, addr, rss, &prealloc);/* * If we need a pre-allocated page for this pte, drop the * locks, allocate, and try again.
PATCH 9/11 mm/memory: optimize fork() with PTE-mapped THP
In reply to David HildenbrandLet's implement PTE batching when consecutive (present) PTEs map consecutive pages of the same large folio, and all other PTE bits besides the PFNs are equal.
We will optimize folio_pte_batch() separately, to ignore some other PTE bits. This patch is based on work by Ryan Roberts.
Use __always_inline for __copy_present_ptes() and keep the handling for single PTEs completely separate from the multi-PTE case: we really want the compiler to optimize for the single-PTE case with small folios, to not degrade performance.
Note that PTE batching will never exceed a single page table and will always stay within VMA boundaries.
--- include/linux/pgtable.h | 17 +++++- mm/memory.c | 113 +++++++++++++++++++++++++++++++++------- 2 files changed, 109 insertions(+), 21 deletions(-)
include/linux/pgtable.h +15 -2
@@ -212,8 +212,6 @@ static inline int pmd_dirty(pmd_t pmd)#define arch_flush_lazy_mmu_mode () do {} while (0)#endif -#ifndef set_ptes -#ifndef pte_next_pfnstatic inline pte_t pte_next_pfn (pte_t pte) {
@@ -221,6 +219,7 @@ static inline pte_t pte_next_pfn(pte_t pte) }#endif +#ifndef set_ptes/** * set_ptes - Map consecutive pages to a contiguous range of addresses. * @mm: Address space to map the pages into.
@@ -650,6 +649,20 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres }#endif +#ifndef wrprotect_ptes +static inline void wrprotect_ptes (struct mm_struct *mm,unsigned long addr, +pte_t *ptep,unsigned int nr) +{ +for (;;) { +ptep_set_wrprotect (mm, addr, ptep); +if (--nr ==0 ) +break ; + ptep++; + addr +=PAGE_SIZE ; + } +} +#endif +/* * On some architectures hardware does not set page access bit when accessing * memory page, it is responsibility of software setting this bit. It brings
mm/memory.c +94 -19
191 lines not on this page · raw message
Signature · 1 line
2.43.0
Quoted 43 lines · David Hildenbrand
Let's implement PTE batching when consecutive (present) PTEs map consecutive pages of the same large folio, and all other PTE bits besides the PFNs are equal.
We will optimize
folio_pte_batch()separately, to ignore some other PTE bits. This patch is based on work by Ryan Roberts.Use
__always_inlinefor__copy_present_ptes()and keep the handling for single PTEs completely separate from the multi-PTE case: we really want the compiler to optimize for the single-PTE case with small folios, to not degrade performance.Note that PTE batching will never exceed a single page table and will always stay within VMA boundaries.
Signed-off-by David Hildenbrand <[email protected]>--- include/linux/pgtable.h | 17 +++++- mm/memory.c | 113 +++++++++++++++++++++++++++++++++------- 2 files changed, 109 insertions(+), 21 deletions(-)include/linux/pgtable.h +1
-2@@ -212,8 +212,6 @@ static inline int pmd_dirty(pmd_t pmd)#define arch_flush_lazy_mmu_mode () do {} while (0)#endif -#ifndef set_ptes -#ifndef pte_next_pfnstatic inline pte_t pte_next_pfn (pte_t pte) {@@ -221,6 +219,7 @@ static inline pte_t pte_next_pfn(pte_t pte) }#endif +#ifndef set_ptes/** * set_ptes - Map consecutive pages to a contiguous range of addresses. * @mm: Address space to map the pages into.
include/linux/pgtable.h static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres · line 652
Quoted 4 lines
include/linux/pgtable.h +1
-0@@ -650,6 +649,20 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres }#endif +#ifndef wrprotect_ptes
I wrote some documentation for this (based on Matthew's docs for set_ptes() in my version. Perhaps it makes sense to add it here, given this is overridable by the arch.
/**
* wrprotect_ptes - Write protect a consecutive set of pages.
* @mm: Address space that the pages are mapped into.
* @addr: Address of first page to write protect.
* @ptep: Page table pointer for the first entry.
* @nr: Number of pages to write protect.
*
* May be overridden by the architecture, else implemented as a loop over
* ptep_set_wrprotect().
*
* Context: The caller holds the page table lock. The PTEs are all in the same
* PMD.
*/
Quoted 21 lines
include/linux/pgtable.h +13
-0+static inline void wrprotect_ptes (struct mm_struct *mm,unsigned long addr, +pte_t *ptep,unsigned int nr) +{ +for (;;) { +ptep_set_wrprotect (mm, addr, ptep); +if (--nr ==0 ) +break ; + ptep++; + addr +=PAGE_SIZE ; + } +} +#endif +/* * On some architectures hardware does not set page access bit when accessing * memory page, it is responsibility of software setting this bit. It brings mm/memory.c +0
-0
mm/memory.c copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma · line 933
Quoted 5 lines
mm/memory.c +1
-1@@ -930,15 +930,15 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vmareturn 0 ; } -static inline void __copy_present_pte(struct vm_area_struct *dst_vma, +static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,
nit: doesn't the addition of __always_inline really belong in the patch where you factored this out? (#7)
Quoted 85 lines
mm/memory.c +50
-7struct vm_area_struct *src_vma, pte_t *dst_pte, pte_t *src_pte, - pte_t pte, unsigned long addr) + pte_t pte, unsigned long addr, int nr) { struct mm_struct *src_mm = src_vma->vm_mm;/* If it's a COW mapping, write protect it both processes. */ if (is_cow_mapping (src_vma->vm_flags) &&pte_write (pte)) { -ptep_set_wrprotect (src_mm, addr, src_pte); +wrprotect_ptes (src_mm, addr, src_pte, nr); pte =pte_wrprotect (pte); }@@ -950,26 +950,94 @@ static inline void __copy_present_pte(struct vm_area_struct *dst_vma,if (!userfaultfd_wp (dst_vma)) pte =pte_clear_uffd_wp (pte); -set_pte_at (dst_vma->vm_mm, addr, dst_pte, pte); +set_ptes (dst_vma->vm_mm, addr, dst_pte, pte, nr); +} + +/* +* Detect a PTE batch: consecutive (present) PTEs that map consecutive +* pages of the same folio. +* +* All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN. +*/ +static inline int folio_pte_batch (struct folio *folio,unsigned long addr, +pte_t *start_ptep,pte_t pte,int max_nr) +{ +unsigned long folio_end_pfn =folio_pfn (folio) +folio_nr_pages (folio); +const pte_t *end_ptep = start_ptep + max_nr; +pte_t expected_pte =pte_next_pfn (pte); +pte_t *ptep = start_ptep +1 ; + +VM_WARN_ON_FOLIO (!pte_present (pte), folio); + +while (ptep != end_ptep) { + pte =ptep_get (ptep); + +if (!pte_same (pte, expected_pte)) +break ; + +/* +* Stop immediately once we reached the end of the folio. In +* corner cases the next PFN might fall into a different +* folio. +*/ +if (pte_pfn (pte) == folio_end_pfn) +break ; + + expected_pte =pte_next_pfn (expected_pte); + ptep++; + } + +return ptep - start_ptep; }/* -* Copy one pte. Returns 0 if succeeded, or -EAGAIN if one preallocated page -* is required to copy this pte. +* Copy one present PTE, trying to batch-process subsequent PTEs that map +* consecutive pages of the same folio by copying them as well. +* +* Returns -EAGAIN if one preallocated page is required to copy the next PTE. +* Otherwise, returns the number of copied PTEs (at least 1). */ static inline int -copy_present_pte (struct vm_area_struct *dst_vma,struct vm_area_struct *src_vma, +copy_present_ptes (struct vm_area_struct *dst_vma,struct vm_area_struct *src_vma,pte_t *dst_pte,pte_t *src_pte,pte_t pte,unsigned long addr, -int *rss,struct folio **prealloc) +int max_nr,int *rss,struct folio **prealloc) {struct page *page;struct folio *folio; +int err, nr; page =vm_normal_page (src_vma, addr, pte);if (unlikely (!page)) goto copy_pte; folio =page_folio (page); + +/*
mm/memory.c static inline void __copy_present_pte(struct vm_area_struct *dst_vma, · line 1025
Quoted 10 lines
mm/memory.c +10
-0+* If we likely have to copy, just don't bother with batching. Make +* sure that the common "small folio" case stays as fast as possible +* by keeping the batching logic separate. +*/ +if (unlikely (!*prealloc &&folio_test_large (folio) && max_nr !=1 )) { + nr =folio_pte_batch (folio, addr, src_pte, pte, max_nr); +if (folio_test_anon (folio)) { +folio_ref_add (folio, nr); +if (unlikely (folio_try_dup_anon_rmap_ptes (folio, page, + nr, src_vma))) {
What happens if its not the first page of the batch that fails here? Aren't you signalling that you need a prealloc'ed page for the wrong pte? Shouldn't you still batch copy all the way up to the failing page first? Perhaps it all comes out in the wash and these events are so infrequent that we don't care about the lost batching opportunity?
mm/memory.c static inline void __copy_present_pte(struct vm_area_struct *dst_vma, · line 1032
Quoted 7 lines
mm/memory.c +7
-0+folio_ref_sub (folio, nr); +return -EAGAIN ; + } + rss[MM_ANONPAGES ] += nr; +VM_WARN_ON_FOLIO (PageAnonExclusive (page), folio); + }else { +folio_ref_add (folio, nr);
Perhaps hoist this out to immediately after folio_pte_batch() since you're calling it on both branches?
Quoted 92 lines
mm/memory.c +26
-11+folio_dup_file_rmap_ptes (folio, page, nr); + rss[mm_counter_file (page)] += nr; + } +__copy_present_ptes (dst_vma, src_vma, dst_pte, src_pte, pte, + addr, nr); +return nr; + } +if (folio_test_anon (folio)) {/* * If this page may have been pinned by the parent process, @@ -981,8 +1049,9 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,if (unlikely (folio_try_dup_anon_rmap_pte (folio, page, src_vma))) {/* Page may be pinned, we have to copy. */ folio_put (folio); -return copy_present_page (dst_vma, src_vma, dst_pte, src_pte, - addr, rss, prealloc, page); + err =copy_present_page (dst_vma, src_vma, dst_pte, src_pte, + addr, rss, prealloc, page); +return err ? err :1 ; } rss[MM_ANONPAGES ]++;VM_WARN_ON_FOLIO (PageAnonExclusive (page), folio);@@ -993,8 +1062,8 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, } copy_pte: -__copy_present_pte (dst_vma, src_vma, dst_pte, src_pte, pte, addr); -return 0 ; +__copy_present_ptes (dst_vma, src_vma, dst_pte, src_pte, pte, addr,1 ); +return 1 ; } static inline struct folio *folio_prealloc(struct mm_struct *src_mm,@@ -1031,10 +1100,11 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,pte_t *src_pte, *dst_pte;pte_t ptent;spinlock_t *src_ptl, *dst_ptl; -int progress, ret =0 ; +int progress, max_nr, ret =0 ;int rss[NR_MM_COUNTERS ];swp_entry_t entry = (swp_entry_t ){0 };struct folio *prealloc =NULL ; +int nr; again: progress =0 ;@@ -1065,6 +1135,8 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,arch_enter_lazy_mmu_mode (); do { + nr = 1; + /* * We are holding two locks at this point - either of them * could generate latencies in another task on another CPU.@@ -1101,9 +1173,10 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,*/ WARN_ON_ONCE (ret != -ENOENT ); } -/* copy_present_pte() will clear `*prealloc' if consumed */ - ret =copy_present_pte (dst_vma, src_vma, dst_pte, src_pte, - ptent, addr, rss, &prealloc); +/* copy_present_ptes() will clear `*prealloc' if consumed */ + max_nr = (end - addr) /PAGE_SIZE ; + ret =copy_present_ptes (dst_vma, src_vma, dst_pte, src_pte, + ptent, addr, max_nr, rss, &prealloc);/* * If we need a pre-allocated page for this pte, drop the * locks, allocate, and try again. @@ -1120,8 +1193,10 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,folio_put (prealloc); prealloc =NULL ; } - progress +=8 ; - }while (dst_pte++, src_pte++, addr +=PAGE_SIZE , addr != end); + nr = ret; + progress +=8 * nr; + }while (dst_pte += nr, src_pte += nr, addr +=PAGE_SIZE * nr, + addr != end);arch_leave_lazy_mmu_mode ();pte_unmap_unlock (orig_src_pte, src_ptl);@@ -1142,7 +1217,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, prealloc =folio_prealloc (src_mm, src_vma, addr, false);if (!prealloc)return -ENOMEM ; - }else if (ret) { + }else if (ret <0 ) {VM_WARN_ON_ONCE (1 ); }
[...]
Quoted 5 lines
I wrote some documentation for this (based on Matthew's docs for
set_ptes()in my version. Perhaps it makes sense to add it here, given this is overridable by the arch.
Quoted 14 lines
/**
*wrprotect_ptes- Write protect a consecutive set of pages.
* @mm: Address space that the pages are mapped into.
* @addr: Address of first page to write protect.
* @ptep: Page table pointer for the first entry.
* @nr: Number of pages to write protect.
*
* May be overridden by the architecture, else implemented as a loop over
*ptep_set_wrprotect().
*
* Context: The caller holds the page table lock. The PTEs are all in the same
* PMD.
*/
I could have sworn I had a documentation at some point. Let me add some, thanks.
[...]
Quoted 10 lines
Quoted 10 lines
mm/memory.c +10
-0+ +/* +* If we likely have to copy, just don't bother with batching. Make +* sure that the common "small folio" case stays as fast as possible +* by keeping the batching logic separate. +*/ +if (unlikely (!*prealloc &&folio_test_large (folio) && max_nr !=1 )) { + nr =folio_pte_batch (folio, addr, src_pte, pte, max_nr); +if (folio_test_anon (folio)) { +folio_ref_add (folio, nr);
Quoted 8 lines
mm/memory.c static inline void __copy_present_pte(struct vm_area_struct *dst_vma, · line 1025
Quoted 2 lines
mm/memory.c +2
-0+if (unlikely (folio_try_dup_anon_rmap_ptes (folio, page, + nr, src_vma))) {What happens if its not the first page of the batch that fails here? Aren't you signalling that you need a prealloc'ed page for the wrong pte? Shouldn't you still batch copy all the way up to the failing page first? Perhaps it all comes out in the wash and these events are so infrequent that we don't care about the lost batching opportunity?
I assume you mean the weird corner case that some folio pages in the range have PAE set, others don't -- and the folio maybe pinned.
In that case, we fallback to individual pages, and might have preallocated a page although we wouldn't have to preallocate one for processing the next page (that doesn't have PAE set).
It should all work, although not optimized to the extreme, and as it's a corner case, we don't particularly care. Hopefully, in the future we'll only have a single PAE flag per folio.
Or am I missing something?
Quoted 11 lines
mm/memory.c static inline void __copy_present_pte(struct vm_area_struct *dst_vma, · line 1032
Quoted 8 lines
mm/memory.c +7
-0+folio_ref_sub (folio, nr); +return -EAGAIN ; + } + rss[MM_ANONPAGES ] += nr; +VM_WARN_ON_FOLIO (PageAnonExclusive (page), folio); + }else { +folio_ref_add (folio, nr);Perhaps hoist this out to immediately after
folio_pte_batch()since you're calling it on both branches?
Makes sense, thanks.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 52 lines · David Hildenbrand
[...]
Quoted 5 lines
I wrote some documentation for this (based on Matthew's docs for
set_ptes()in my version. Perhaps it makes sense to add it here, given this is overridable by the arch.Quoted 14 lines
/**
*wrprotect_ptes- Write protect a consecutive set of pages.
* @mm: Address space that the pages are mapped into.
* @addr: Address of first page to write protect.
* @ptep: Page table pointer for the first entry.
* @nr: Number of pages to write protect.
*
* May be overridden by the architecture, else implemented as a loop over
*ptep_set_wrprotect().
*
* Context: The caller holds the page table lock. The PTEs are all in the same
* PMD.
*/I could have sworn I had a documentation at some point. Let me add some, thanks.
[...]
Quoted 13 lines
Quoted 4 lines
+
+ /*
+ * If we likely have to copy, just don't bother with batching. Make + * sure that the common "small folio" case stays as fast as possibleQuoted 8 lines
+ * by keeping the batching logic separate.
+ */
+ if (unlikely(!*prealloc && folio_test_large(folio) && max_nr != 1)) { + nr =folio_pte_batch(folio, addr, src_pte, pte, max_nr); + if (folio_test_anon(folio)) {
+folio_ref_add(folio, nr);
+ if (unlikely(folio_try_dup_anon_rmap_ptes(folio, page, + nr,src_vma))) {Quoted 5 lines
What happens if its not the first page of the batch that fails here? Aren't you signalling that you need a prealloc'ed page for the wrong pte? Shouldn't you still batch copy all the way up to the failing page first? Perhaps it all comes out in the wash and these events are so infrequent that we don't care about the lost batching opportunity?
I assume you mean the weird corner case that some folio pages in the range have PAE set, others don't -- and the folio maybe pinned.
In that case, we fallback to individual pages, and might have preallocated a page although we wouldn't have to preallocate one for processing the next page (that doesn't have PAE set).
Quoted 5 lines
It should all work, although not optimized to the extreme, and as it's a corner case, we don't particularly care. Hopefully, in the future we'll only have a single PAE flag per folio.
Or am I missing something?
No, your explanation makes sense. Just wanted to check this all definitely worked, because the flow is slightly different to my previous version that was doing try_dup_rmap page-by-page.
Quoted 15 lines
Quoted 10 lines
Quoted 9 lines
+
folio_ref_sub(folio, nr);
+ return-EAGAIN;
+ }
+ rss[MM_ANONPAGES] += nr;
+VM_WARN_ON_FOLIO(PageAnonExclusive(page), folio); + } else {
+folio_ref_add(folio, nr);Quoted 2 lines
Perhaps hoist this out to immediately after
folio_pte_batch()since you're calling it on both branches?Makes sense, thanks.
PATCH 10/11 mm/memory: ignore dirty/accessed/soft-dirty bits in folio_pte_batch()
In reply to David HildenbrandLet's ignore these bits: they are irrelevant for fork, and will likely be irrelevant for upcoming users such as page unmapping.
--- mm/memory.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
mm/memory.c +8 -2
@@ -953,24 +953,30 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,set_ptes (dst_vma->vm_mm, addr, dst_pte, pte, nr); } +static inline pte_t __pte_batch_clear_ignored (pte_t pte) +{ +return pte_clear_soft_dirty (pte_mkclean (pte_mkold (pte))); +} +/* * Detect a PTE batch: consecutive (present) PTEs that map consecutive * pages of the same folio. * * All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN. +* the accessed bit, dirty bit and soft-dirty bit. */ static inline int folio_pte_batch (struct folio *folio,unsigned long addr,pte_t *start_ptep,pte_t pte,int max_nr) {unsigned long folio_end_pfn =folio_pfn (folio) +folio_nr_pages (folio);const pte_t *end_ptep = start_ptep + max_nr; -pte_t expected_pte =pte_next_pfn (pte); +pte_t expected_pte =__pte_batch_clear_ignored (pte_next_pfn (pte));pte_t *ptep = start_ptep +1 ;VM_WARN_ON_FOLIO (!pte_present (pte), folio);while (ptep != end_ptep) { - pte =ptep_get (ptep); + pte =__pte_batch_clear_ignored (ptep_get (ptep));if (!pte_same (pte, expected_pte))break ;
Signature · 1 line
2.43.0
Re: [PATCH v1 10/11] mm/memory: ignore dirty/accessed/soft-dirty bits in folio_pte_batch()
Quoted 16 lines · David Hildenbrand
Let's ignore these bits: they are irrelevant for fork, and will likely be irrelevant for upcoming users such as page unmapping.
Signed-off-by David Hildenbrand <[email protected]>--- mm/memory.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)mm/memory.c +0
-0@@ -953,24 +953,30 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,set_ptes (dst_vma->vm_mm, addr, dst_pte, pte, nr); }
mm/memory.c static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma, · line 965
Quoted 10 lines
mm/memory.c +5
-0+static inline pte_t __pte_batch_clear_ignored (pte_t pte) +{ +return pte_clear_soft_dirty (pte_mkclean (pte_mkold (pte))); +} +/* * Detect a PTE batch: consecutive (present) PTEs that map consecutive * pages of the same folio. * * All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN.
nit: last char should be a comma (,) not a full stop (.)
Quoted 9 lines
mm/memory.c +2
-1+* the accessed bit, dirty bit and soft-dirty bit. */ static inline int folio_pte_batch (struct folio *folio,unsigned long addr,pte_t *start_ptep,pte_t pte,int max_nr) {unsigned long folio_end_pfn =folio_pfn (folio) +folio_nr_pages (folio);const pte_t *end_ptep = start_ptep + max_nr; -pte_t expected_pte =pte_next_pfn (pte); +pte_t expected_pte =__pte_batch_clear_ignored (pte_next_pfn (pte));
mm/memory.c static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma, · line 982
Quoted 10 lines
mm/memory.c +1
-1pte_t *ptep = start_ptep +1 ;VM_WARN_ON_FOLIO (!pte_present (pte), folio);while (ptep != end_ptep) { - pte =ptep_get (ptep); + pte =__pte_batch_clear_ignored (ptep_get (ptep));if (!pte_same (pte, expected_pte))break ;
I think you'll lose dirty information in the child for private mappings? If the first pte in a batch is clean, but a subsequent page is dirty, you will end up setting all the pages in the batch as clean in the child. Previous behavior would preserve dirty bit for private mappings.
In my version (v3) that did arbitrary batching, I had some fun and games tracking dirty, write and uffd_wp:
https://lore.kernel.org/linux-arm-kernel/[email protected]/
Also, I think you will currently either set soft dirty on all or none of the pages in the batch, depending on the value of the first. I previously convinced myself that the state was unimportant so always cleared it in the child to provide consistency.
Re: [PATCH v1 10/11] mm/memory: ignore dirty/accessed/soft-dirty bits in folio_pte_batch()
Quoted 55 lines · Ryan Roberts
Quoted 16 lines · David Hildenbrand
Let's ignore these bits: they are irrelevant for fork, and will likely be irrelevant for upcoming users such as page unmapping.
Signed-off-by David Hildenbrand <[email protected]>--- mm/memory.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)mm/memory.c +0
-0@@ -953,24 +953,30 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,set_ptes (dst_vma->vm_mm, addr, dst_pte, pte, nr); }mm/memory.c static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma, · line 965
Quoted 10 lines
mm/memory.c +5
-0+static inline pte_t __pte_batch_clear_ignored (pte_t pte) +{ +return pte_clear_soft_dirty (pte_mkclean (pte_mkold (pte))); +} +/* * Detect a PTE batch: consecutive (present) PTEs that map consecutive * pages of the same folio. * * All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN. nit: last char should be a comma (,) not a full stop (.)
Quoted 9 lines
mm/memory.c +2
-1+* the accessed bit, dirty bit and soft-dirty bit. */ static inline int folio_pte_batch (struct folio *folio,unsigned long addr,pte_t *start_ptep,pte_t pte,int max_nr) {unsigned long folio_end_pfn =folio_pfn (folio) +folio_nr_pages (folio);const pte_t *end_ptep = start_ptep + max_nr; -pte_t expected_pte =pte_next_pfn (pte); +pte_t expected_pte =__pte_batch_clear_ignored (pte_next_pfn (pte));mm/memory.c static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma, · line 982
Quoted 10 lines
mm/memory.c +1
-1pte_t *ptep = start_ptep +1 ;VM_WARN_ON_FOLIO (!pte_present (pte), folio);while (ptep != end_ptep) { - pte =ptep_get (ptep); + pte =__pte_batch_clear_ignored (ptep_get (ptep));if (!pte_same (pte, expected_pte))break ;I think you'll lose dirty information in the child for private mappings? If the first pte in a batch is clean, but a subsequent page is dirty, you will end up setting all the pages in the batch as clean in the child. Previous behavior would preserve dirty bit for private mappings.
Quoted 8 lines
In my version (v3) that did arbitrary batching, I had some fun and games tracking dirty, write and
uffd_wp:
https://lore.kernel.org/linux-arm-kernel/[email protected]/Also, I think you will currently either set soft dirty on all or none of the pages in the batch, depending on the value of the first. I previously convinced myself that the state was unimportant so always cleared it in the child to provide consistency.
Good points regarding dirty and soft-dirty. I wanted to avoid passing flags to folio_pte_batch(), but maybe that's just what we need to not change behavior.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v1 10/11] mm/memory: ignore dirty/accessed/soft-dirty bits in folio_pte_batch()
Quoted 57 lines · David Hildenbrand
Quoted 56 lines · Ryan Roberts
Quoted 16 lines · David Hildenbrand
Let's ignore these bits: they are irrelevant for fork, and will likely be irrelevant for upcoming users such as page unmapping.
Signed-off-by David Hildenbrand <[email protected]>---
mm/memory.c| 10 ++++++++--1 file changed, 8 insertions(+), 2 deletions(-)mm/memory.c +0
-0@@ -953,24 +953,30 @@ static __always_inline void __copy_present_ptes(structvm_area_struct *dst_vma, set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr); }mm/memory.c static __always_inline void __copy_present_ptes(struct · line 965
Quoted 10 lines
mm/memory.c +4
-0+static inline pte_t __pte_batch_clear_ignored(pte_t pte) +{ + return pte_clear_soft_dirty(pte_mkclean(pte_mkold(pte))); +} + /** Detect a PTE batch: consecutive (present) PTEs that map consecutive * pages of the same folio. * * All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN. nit: last char should be a comma (,) not a full stop (.)
Quoted 6 lines
mm/memory.c +1
-0+* the accessed bit, dirty bit and soft-dirty bit. */ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, pte_t *start_ptep, pte_t pte, int max_nr) { unsigned long folio_end_pfn = folio_pfn(folio) + folio_nr_pages(folio);mm/memory.c static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma, · line 982
Quoted 10 lines
mm/memory.c +2
-2const pte_t *end_ptep = start_ptep + max_nr; - pte_t expected_pte = pte_next_pfn(pte); + pte_t expected_pte = __pte_batch_clear_ignored(pte_next_pfn(pte)); pte_t *ptep = start_ptep + 1; VM_WARN_ON_FOLIO(!pte_present(pte), folio); while (ptep != end_ptep) { - pte = ptep_get(ptep); + pte = __pte_batch_clear_ignored(ptep_get(ptep)); if (!pte_same(pte, expected_pte)) break;I think you'll lose dirty information in the child for private mappings? If the first pte in a batch is clean, but a subsequent page is dirty, you will end up setting all the pages in the batch as clean in the child. Previous behavior would preserve dirty bit for private mappings.
In my version (v3) that did arbitrary batching, I had some fun and games tracking dirty, write and
uffd_wp:
https://lore.kernel.org/linux-arm-kernel/[email protected]/
Quoted 7 lines
Quoted 4 lines
Also, I think you will currently either set soft dirty on all or none of the pages in the batch, depending on the value of the first. I previously convinced myself that the state was unimportant so always cleared it in the child to provide consistency.
Good points regarding dirty and soft-dirty. I wanted to avoid passing flags to
folio_pte_batch(), but maybe that's just what we need to not change behavior.
I think you could not bother with the enforce_uffd_wp - just always enforce uffd-wp. So that's one simplification vs mine. Then you just need an any_dirty flag following the same pattern as your any_writable. Then just set dirty on the whole batch in the child if any were dirty in the parent.
Although now I'm wondering if there is a race here... What happens if a page in the parent becomes dirty after you have checked it but before you write protect it? Isn't that already a problem with the current non-batched version? Why do we even to preserve dirty in the child for private mappings?
Re: [PATCH v1 10/11] mm/memory: ignore dirty/accessed/soft-dirty bits in folio_pte_batch()
Quoted 63 lines · Ryan Roberts
Quoted 62 lines · David Hildenbrand
Quoted 60 lines · Ryan Roberts
Quoted 16 lines · David Hildenbrand
Let's ignore these bits: they are irrelevant for fork, and will likely be irrelevant for upcoming users such as page unmapping.
Signed-off-by David Hildenbrand <[email protected]>--- mm/memory.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)mm/memory.c +0
-0@@ -953,24 +953,30 @@ static __always_inline void __copy_present_ptes(structvm_area_struct *dst_vma, set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr); }mm/memory.c static __always_inline void __copy_present_ptes(struct · line 965
Quoted 10 lines
mm/memory.c +4
-0+static inline pte_t __pte_batch_clear_ignored(pte_t pte) +{ + return pte_clear_soft_dirty(pte_mkclean(pte_mkold(pte))); +} + /** Detect a PTE batch: consecutive (present) PTEs that map consecutive * pages of the same folio. * * All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN. nit: last char should be a comma (,) not a full stop (.)
Quoted 6 lines
mm/memory.c +1
-0+* the accessed bit, dirty bit and soft-dirty bit. */ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, pte_t *start_ptep, pte_t pte, int max_nr) { unsigned long folio_end_pfn = folio_pfn(folio) + folio_nr_pages(folio);mm/memory.c static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma, · line 982
Quoted 10 lines
mm/memory.c +2
-2const pte_t *end_ptep = start_ptep + max_nr; - pte_t expected_pte = pte_next_pfn(pte); + pte_t expected_pte = __pte_batch_clear_ignored(pte_next_pfn(pte)); pte_t *ptep = start_ptep + 1; VM_WARN_ON_FOLIO(!pte_present(pte), folio); while (ptep != end_ptep) { - pte = ptep_get(ptep); + pte = __pte_batch_clear_ignored(ptep_get(ptep)); if (!pte_same(pte, expected_pte)) break;I think you'll lose dirty information in the child for private mappings? If the first pte in a batch is clean, but a subsequent page is dirty, you will end up setting all the pages in the batch as clean in the child. Previous behavior would preserve dirty bit for private mappings.
In my version (v3) that did arbitrary batching, I had some fun and games tracking dirty, write and
uffd_wp:
https://lore.kernel.org/linux-arm-kernel/[email protected]/Also, I think you will currently either set soft dirty on all or none of the pages in the batch, depending on the value of the first. I previously convinced myself that the state was unimportant so always cleared it in the child to provide consistency.
Quoted 5 lines
Quoted 2 lines
Good points regarding dirty and soft-dirty. I wanted to avoid passing flags to
folio_pte_batch(), but maybe that's just what we need to not change behavior.I think you could not bother with the
enforce_uffd_wp- just always enforce uffd-wp. So that's one simplification vs mine. Then you just need anany_dirty
I think I'll just leave uffd-wp alone for now, corner case with fork/munmap that can be optimized later on top if really needed.
Regarding soft-dirty (which is set automatically much more often), I can certainly ignore the bit if !vma_soft_dirty_enabled(vma) [which is true in most of the cases]. So that's easy to handle. But likely, soft-dirty for the child is completely unexpressive and should always be cleared. Have to double check what the vmflag will be for the child process.
Quoted 2 lines
flag following the same pattern as your
any_writable. Then just set dirty on the whole batch in the child if any were dirty in the parent.
Regarding dirtying, I'm not 100% sure yet if we should just always dirty all ptes if any is dirty, or if we should preserve the state for private VMAs for now.
Quoted 5 lines
Although now I'm wondering if there is a race here... What happens if a page in the parent becomes dirty after you have checked it but before you write protect it? Isn't that already a problem with the current non-batched version? Why do we even to preserve dirty in the child for private mappings?
I suspect, because the parent could zap the anon folio. If the folio is clean, but the PTE dirty, I suspect that we could lose data of the child if we were to evict that clean folio (swapout).
So I assume we simply copy the dirty PTE bit, so the system knows that that folio is actually dirty, because one PTE is dirty.
Touching only PTEs avoids having to mess with folio flags.
But that's just pure speculation. E.g., fs/proc/task_mmu.c does some slightly different accounting if a PTE is dirty. But usually, it checks if either the PTE or the folios is dirty.
I'll have to do some more digging.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v1 10/11] mm/memory: ignore dirty/accessed/soft-dirty bits in folio_pte_batch()
Quoted 11 lines
Quoted 4 lines
Although now I'm wondering if there is a race here... What happens if a page in the parent becomes dirty after you have checked it but before you write protect it? Isn't that already a problem with the current non-batched version? Why do we even to preserve dirty in the child for private mappings?
I suspect, because the parent could zap the anon folio. If the folio is clean, but the PTE dirty, I suspect that we could lose data of the child if we were to evict that clean folio (swapout).
So I assume we simply copy the dirty PTE bit, so the system knows that that folio is actually dirty, because one PTE is dirty.
Oh, and regarding your race concern: it's undefined which page state would see if some write is racing with fork, so it also doesn't matter if we would copy the PTE dirty bit or not, if it gets set in a racy fashion.
I'll not experiment with:
From 14e83ff2a422a96ce5701f9c8454a49f9ed947e3 Mon Sep 17 00:00:00 2001
From: David Hildenbrand <[email protected]>
Date: Sat, 30 Dec 2023 12:54:35 +0100
Subject: [PATCH] mm/memory: ignore dirty/accessed/soft-dirty bits in
folio_pte_batch()
Let's always ignore the accessed/young bit: we'll always mark the PTE as old in our child process during fork, and upcoming users will similarly not care.
Ignore the dirty bit only if we don't want to duplicate the dirty bit into the child process during fork. Maybe, we could just set all PTEs in the child dirty if any PTE is dirty. For now, let's keep the behavior unchanged.
Ignore the soft-dirty bit only if the bit doesn't have any meaning in the src vma.
--- mm/memory.c | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-)
mm/memory.c +30 -4
@@ -953,24 +953,44 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,set_ptes (dst_vma->vm_mm, addr, dst_pte, pte, nr); } +/* Flags for folio_pte_batch(). */ +typedef int __bitwise fpb_t ; + +/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */ +#define FPB_IGNORE_DIRTY ((__force fpb_t)BIT(0)) + +/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */ +#define FPB_IGNORE_SOFT_DIRTY ((__force fpb_t)BIT(1)) + +static inline pte_t __pte_batch_clear_ignored (pte_t pte,fpb_t flags) +{ +if (flags &FPB_IGNORE_DIRTY ) + pte =pte_mkclean (pte); +if (likely (flags &FPB_IGNORE_SOFT_DIRTY )) + pte =pte_clear_soft_dirty (pte); +return pte_mkold (pte); +} +/* * Detect a PTE batch: consecutive (present) PTEs that map consecutive * pages of the same folio. * * All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN. +* the accessed bit, dirty bit (with FPB_IGNORE_DIRTY) and soft-dirty bit +* (with FPB_IGNORE_SOFT_DIRTY). */ static inline int folio_pte_batch (struct folio *folio,unsigned long addr, -pte_t *start_ptep,pte_t pte,int max_nr) +pte_t *start_ptep,pte_t pte,int max_nr,fpb_t flags) {unsigned long folio_end_pfn =folio_pfn (folio) +folio_nr_pages (folio);const pte_t *end_ptep = start_ptep + max_nr; -pte_t expected_pte =pte_next_pfn (pte); +pte_t expected_pte =__pte_batch_clear_ignored (pte_next_pfn (pte), flags);pte_t *ptep = start_ptep +1 ;VM_WARN_ON_FOLIO (!pte_present (pte), folio);while (ptep != end_ptep) { - pte =ptep_get (ptep); + pte =__pte_batch_clear_ignored (ptep_get (ptep), flags);if (!pte_same (pte, expected_pte))break ;
@@ -1004,6 +1024,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma {struct page *page;struct folio *folio; +fpb_t flags =0 ;int err, nr; page =vm_normal_page (src_vma, addr, pte);
@@ -1018,7 +1039,12 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma* by keeping the batching logic separate. */ if (unlikely (!*prealloc &&folio_test_large (folio) && max_nr !=1 )) { - nr =folio_pte_batch (folio, addr, src_pte, pte, max_nr); +if (src_vma->vm_flags &VM_SHARED ) + flags |=FPB_IGNORE_DIRTY ; +if (!vma_soft_dirty_enabled (src_vma)) + flags |=FPB_IGNORE_SOFT_DIRTY ; + + nr =folio_pte_batch (folio, addr, src_pte, pte, max_nr, flags);folio_ref_add (folio, nr);if (folio_test_anon (folio)) { if (unlikely(folio_try_dup_anon_rmap_ptes(folio, page,
Signature · 7 lines
2.43.0 -- Cheers, David / dhildenb
Re: [PATCH v1 10/11] mm/memory: ignore dirty/accessed/soft-dirty bits in folio_pte_batch()
Quoted 9 lines · David Hildenbrand
Quoted 9 lines
Quoted 4 lines
Although now I'm wondering if there is a race here... What happens if a page in the parent becomes dirty after you have checked it but before you write protect it? Isn't that already a problem with the current non-batched version? Why do we even to preserve dirty in the child for private mappings?
I suspect, because the parent could zap the anon folio. If the folio is clean, but the PTE dirty, I suspect that we could lose data of the child if we were to evict that clean folio (swapout).
Quoted 6 lines
Quoted 2 lines
So I assume we simply copy the dirty PTE bit, so the system knows that that folio is actually dirty, because one PTE is dirty.
Oh, and regarding your race concern: it's undefined which page state would see if some write is racing with fork, so it also doesn't matter if we would copy the PTE dirty bit or not, if it gets set in a racy fashion.
Ahh that makes sense. Thanks.
Quoted 2 lines
I'll not experiment with:
Looks good as long as its still performant.
Quoted 101 lines
From
14e83ff2a422a96ce5701f9c8454a49f9ed947e3Mon Sep 17 00:00:00 2001
From: David Hildenbrand <[email protected]>
Date: Sat, 30 Dec 2023 12:54:35 +0100
Subject: [PATCH] mm/memory: ignore dirty/accessed/soft-dirty bits infolio_pte_batch()Let's always ignore the accessed/young bit: we'll always mark the PTE as old in our child process during fork, and upcoming users will similarly not care.
Ignore the dirty bit only if we don't want to duplicate the dirty bit into the child process during fork. Maybe, we could just set all PTEs in the child dirty if any PTE is dirty. For now, let's keep the behavior unchanged.
Ignore the soft-dirty bit only if the bit doesn't have any meaning in the src vma.
Signed-off-by David Hildenbrand <[email protected]>---
mm/memory.c | 34 ++++++++++++++++++++++++++++++----1 file changed, 30 insertions(+), 4 deletions(-)mm/memory.c +0
-0@@ -953,24 +953,44 @@ static __always_inline void __copy_present_ptes(structvm_area_struct *dst_vma, set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr); }mm/memory.c +18
-0+/* Flags for folio_pte_batch(). */ +typedef int __bitwise fpb_t ; + +/* Compare PTEs after pte_mkclean(), ignoring the dirty bit. */ +#define FPB_IGNORE_DIRTY ((__force fpb_t)BIT(0)) + +/* Compare PTEs after pte_clear_soft_dirty(), ignoring the soft-dirty bit. */ +#define FPB_IGNORE_SOFT_DIRTY ((__force fpb_t)BIT(1)) + +static inline pte_t __pte_batch_clear_ignored (pte_t pte,fpb_t flags) +{ + if (flags & FPB_IGNORE_DIRTY) + pte = pte_mkclean(pte); + if (likely(flags & FPB_IGNORE_SOFT_DIRTY)) + pte = pte_clear_soft_dirty(pte); + return pte_mkold(pte); +} +/*
* Detect a PTE batch: consecutive (present) PTEs that map consecutive
* pages of the same folio.
*
* All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN.
+ * the accessed bit, dirty bit (withFPB_IGNORE_DIRTY) and soft-dirty bit
+ * (withFPB_IGNORE_SOFT_DIRTY).
*/
static inline intfolio_pte_batch(struct folio *folio, unsigned long addr, - pte_t *start_ptep, pte_t pte, int max_nr)
+pte_t*start_ptep,pte_tpte, intmax_nr,fpb_tflags)
{
unsigned longfolio_end_pfn=folio_pfn(folio)+folio_nr_pages(folio); constpte_t*end_ptep=start_ptep+max_nr;
-pte_texpected_pte=pte_next_pfn(pte);
+pte_texpected_pte=__pte_batch_clear_ignored(pte_next_pfn(pte), flags);pte_t*ptep =start_ptep+ 1;
VM_WARN_ON_FOLIO(!pte_present(pte), folio);
while (ptep !=end_ptep) {
- pte =ptep_get(ptep);
+ pte =__pte_batch_clear_ignored(ptep_get(ptep), flags);
if (!pte_same(pte,expected_pte))
break;mm/memory.c +0
-0@@ -1004,6 +1024,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct
vm_area_struct*src_vma
{
struct page *page;
struct folio *folio;
+fpb_tflags = 0;
int err, nr;
page =vm_normal_page(src_vma, addr, pte);mm/memory.c +0
-0@@ -1018,7 +1039,12 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct
vm_area_struct*src_vma
* by keeping the batching logic separate.
*/
if (unlikely(!*prealloc && folio_test_large(folio) && max_nr != 1)) { - nr =folio_pte_batch(folio, addr, src_pte, pte, max_nr); + if (src_vma->vm_flags&VM_SHARED)
+ flags |=FPB_IGNORE_DIRTY;
+ if (!vma_soft_dirty_enabled(src_vma))
+ flags |=FPB_IGNORE_SOFT_DIRTY;
+
+ nr =folio_pte_batch(folio, addr, src_pte, pte, max_nr, flags);folio_ref_add(folio, nr);
if (folio_test_anon(folio)) {
if (unlikely(folio_try_dup_anon_rmap_ptes(folio, page,
PATCH 11/11 mm/memory: ignore writable bit in folio_pte_batch()
In reply to David Hildenbrand... and conditionally return to the caller if any pte except the first one is writable. fork() has to make sure to properly write-protect in case any PTE is writable. Other users (e.g., page unmaping) won't care.
--- mm/memory.c | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-)
mm/memory.c +21 -5
@@ -955,7 +955,7 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,static inline pte_t __pte_batch_clear_ignored (pte_t pte) { -return pte_clear_soft_dirty (pte_mkclean (pte_mkold (pte))); +return pte_wrprotect (pte_clear_soft_dirty (pte_mkclean (pte_mkold (pte)))); }/*
@@ -963,20 +963,29 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte)* pages of the same folio. * * All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN. -* the accessed bit, dirty bit and soft-dirty bit. +* the accessed bit, dirty bit, soft-dirty bit and writable bit. +. If "any_writable" is set, it will indicate if any other PTE besides the +* first (given) PTE is writable. */ static inline int folio_pte_batch (struct folio *folio,unsigned long addr, -pte_t *start_ptep,pte_t pte,int max_nr) +pte_t *start_ptep,pte_t pte,int max_nr,bool *any_writable) {unsigned long folio_end_pfn =folio_pfn (folio) +folio_nr_pages (folio);const pte_t *end_ptep = start_ptep + max_nr;pte_t expected_pte =__pte_batch_clear_ignored (pte_next_pfn (pte));pte_t *ptep = start_ptep +1 ; +bool writable; + +if (any_writable) + *any_writable = false;VM_WARN_ON_FOLIO (!pte_present (pte), folio);while (ptep != end_ptep) { - pte =__pte_batch_clear_ignored (ptep_get (ptep)); + pte =ptep_get (ptep); +if (any_writable) + writable = !!pte_write (pte); + pte =__pte_batch_clear_ignored (pte);if (!pte_same (pte, expected_pte))break ;
@@ -989,6 +998,9 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr,if (pte_pfn (pte) == folio_end_pfn)break ; +if (any_writable) + *any_writable |= writable; + expected_pte =pte_next_pfn (expected_pte); ptep++; }
@@ -1010,6 +1022,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma {struct page *page;struct folio *folio; +bool any_writable;int err, nr; page =vm_normal_page (src_vma, addr, pte);
@@ -1024,7 +1037,8 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma* by keeping the batching logic separate. */ if (unlikely (!*prealloc &&folio_test_large (folio) && max_nr !=1 )) { - nr =folio_pte_batch (folio, addr, src_pte, pte, max_nr); + nr =folio_pte_batch (folio, addr, src_pte, pte, max_nr, + &any_writable);if (folio_test_anon (folio)) {folio_ref_add (folio, nr); if (unlikely(folio_try_dup_anon_rmap_ptes(folio, page,
@@ -1039,6 +1053,8 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vmafolio_dup_file_rmap_ptes (folio, page, nr); rss[mm_counter_file (page)] += nr; } +if (any_writable) + pte =pte_mkwrite (pte, src_vma);__copy_present_ptes (dst_vma, src_vma, dst_pte, src_pte, pte, addr, nr);return nr;
Signature · 1 line
2.43.0
Re: [PATCH v1 11/11] mm/memory: ignore writable bit in folio_pte_batch()
Quoted 5 lines · David Hildenbrand
... and conditionally return to the caller if any pte except the first one is writable.
fork()has to make sure to properly write-protect in case any PTE is writable. Other users (e.g., page unmaping) won't care.Signed-off-by David Hildenbrand <[email protected]>
Quoted 87 lines
--- mm/memory.c | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-)mm/memory.c +21
-5@@ -955,7 +955,7 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,static inline pte_t __pte_batch_clear_ignored (pte_t pte) { -return pte_clear_soft_dirty (pte_mkclean (pte_mkold (pte))); +return pte_wrprotect (pte_clear_soft_dirty (pte_mkclean (pte_mkold (pte)))); }/* @@ -963,20 +963,29 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte)* pages of the same folio. * * All PTEs inside a PTE batch have the same PTE bits set, excluding the PFN. -* the accessed bit, dirty bit and soft-dirty bit. +* the accessed bit, dirty bit, soft-dirty bit and writable bit. +. If "any_writable" is set, it will indicate if any other PTE besides the +* first (given) PTE is writable. */ static inline int folio_pte_batch (struct folio *folio,unsigned long addr, -pte_t *start_ptep,pte_t pte,int max_nr) +pte_t *start_ptep,pte_t pte,int max_nr,bool *any_writable) {unsigned long folio_end_pfn =folio_pfn (folio) +folio_nr_pages (folio);const pte_t *end_ptep = start_ptep + max_nr;pte_t expected_pte =__pte_batch_clear_ignored (pte_next_pfn (pte));pte_t *ptep = start_ptep +1 ; +bool writable; + +if (any_writable) + *any_writable = false;VM_WARN_ON_FOLIO (!pte_present (pte), folio);while (ptep != end_ptep) { - pte =__pte_batch_clear_ignored (ptep_get (ptep)); + pte =ptep_get (ptep); +if (any_writable) + writable = !!pte_write (pte); + pte =__pte_batch_clear_ignored (pte);if (!pte_same (pte, expected_pte))break ;@@ -989,6 +998,9 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr,if (pte_pfn (pte) == folio_end_pfn)break ; +if (any_writable) + *any_writable |= writable; + expected_pte =pte_next_pfn (expected_pte); ptep++; }@@ -1010,6 +1022,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma {struct page *page;struct folio *folio; +bool any_writable;int err, nr; page =vm_normal_page (src_vma, addr, pte);@@ -1024,7 +1037,8 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma* by keeping the batching logic separate. */ if (unlikely (!*prealloc &&folio_test_large (folio) && max_nr !=1 )) { - nr =folio_pte_batch (folio, addr, src_pte, pte, max_nr); + nr =folio_pte_batch (folio, addr, src_pte, pte, max_nr, + &any_writable);if (folio_test_anon (folio)) {folio_ref_add (folio, nr); if (unlikely(folio_try_dup_anon_rmap_ptes(folio, page,@@ -1039,6 +1053,8 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vmafolio_dup_file_rmap_ptes (folio, page, nr); rss[mm_counter_file (page)] += nr; } +if (any_writable) + pte =pte_mkwrite (pte, src_vma);__copy_present_ptes (dst_vma, src_vma, dst_pte, src_pte, pte, addr, nr);return nr;
Quoted 15 lines · David Hildenbrand
Now that the rmap overhaul[1] is upstream that provides a clean interface for rmap batching, let's implement PTE batching during fork when processing PTE-mapped THPs.
This series is partially based on Ryan's previous work[2] to implement cont-pte support on arm64, but its a complete rewrite based on [1] to optimize all architectures independent of any such PTE bits, and to use the new rmap batching functions that simplify the code and prepare for further rmap accounting changes.
We collect consecutive PTEs that map consecutive pages of the same large folio, making sure that the other PTE bits are compatible, and (a) adjust the refcount only once per batch, (b) call rmap handling functions only once per batch and (c) perform batch PTE setting/updates.
Quoted 4 lines
While this series should be beneficial for adding cont-pte support on ARM64[2], it's one of the requirements for maintaining a total mapcount[3] for large folios with minimal added overhead and further changes[4] that build up on top of the total mapcount.
I'm currently rebasing my contpte work onto this series, and have hit a problem. I need to expose the "size" of a pte (pte_size()) and skip forward to the start of the next (cont)pte every time through the folio_pte_batch() loop. But pte_next_pfn() only allows advancing by 1 pfn; I need to advance by nr pfns:
static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
pte_t *start_ptep, pte_t pte, int max_nr, bool *any_writable)
{
unsigned long folio_end_pfn = folio_pfn(folio) + folio_nr_pages(folio);
const pte_t *end_ptep = start_ptep + max_nr;
pte_t expected_pte = __pte_batch_clear_ignored(pte_next_pfn(pte));
- pte_t *ptep = start_ptep + 1;
+ pte_t *ptep = start_ptep;
+ int vfn, nr, i;
bool writable;
if (any_writable)
*any_writable = false;
VM_WARN_ON_FOLIO(!pte_present(pte), folio);+ vfn = addr >> PAGE_SIZE;
+ nr = pte_size(pte);
+ nr = ALIGN_DOWN(vfn + nr, nr) - vfn;
+ ptep += nr;
+
while (ptep != end_ptep) {
+ pte = ptep_get(ptep);
nr = pte_size(pte);
if (any_writable)
writable = !!pte_write(pte);
pte = __pte_batch_clear_ignored(pte);
if (!pte_same(pte, expected_pte))
break;
/*
* Stop immediately once we reached the end of the folio. In
* corner cases the next PFN might fall into a different
* folio.
*/
- if (pte_pfn(pte) == folio_end_pfn)
+ if (pte_pfn(pte) >= folio_end_pfn)
break;
if (any_writable)
*any_writable |= writable;- expected_pte = pte_next_pfn(expected_pte);
- ptep++;
+ for (i = 0; i < nr; i++)
+ expected_pte = pte_next_pfn(expected_pte);
+ ptep += nr; } return ptep - start_ptep; }
So I'm wondering if instead of enabling pte_next_pfn() for all the arches, perhaps its actually better to expose pte_pgprot() for all the arches. Then we can be much more flexible about generating ptes with pfn_pte(pfn, pgprot).
What do you think?
Quoted 81 lines · Ryan Roberts
Quoted 15 lines · David Hildenbrand
Now that the rmap overhaul[1] is upstream that provides a clean interface for rmap batching, let's implement PTE batching during fork when processing PTE-mapped THPs.
This series is partially based on Ryan's previous work[2] to implement cont-pte support on arm64, but its a complete rewrite based on [1] to optimize all architectures independent of any such PTE bits, and to use the new rmap batching functions that simplify the code and prepare for further rmap accounting changes.
We collect consecutive PTEs that map consecutive pages of the same large folio, making sure that the other PTE bits are compatible, and (a) adjust the refcount only once per batch, (b) call rmap handling functions only once per batch and (c) perform batch PTE setting/updates.
Quoted 4 lines
While this series should be beneficial for adding cont-pte support on ARM64[2], it's one of the requirements for maintaining a total mapcount[3] for large folios with minimal added overhead and further changes[4] that build up on top of the total mapcount.
I'm currently rebasing my contpte work onto this series, and have hit a problem. I need to expose the "size" of a pte (
pte_size()) and skip forward to the start of the next (cont)pte every time through thefolio_pte_batch()loop. Butpte_next_pfn()only allows advancing by 1 pfn; I need to advance by nr pfns:static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
pte_t*start_ptep,pte_tpte, intmax_nr, bool*any_writable){ unsigned long folio_end_pfn = folio_pfn(folio) + folio_nr_pages(folio); const pte_t *end_ptep = start_ptep + max_nr; pte_t expected_pte = __pte_batch_clear_ignored(pte_next_pfn(pte));(quoted diff) +2
-1- pte_t *ptep = start_ptep + 1; + pte_t *ptep = start_ptep; + int vfn, nr, i;bool writable; if (any_writable) *any_writable = false; VM_WARN_ON_FOLIO(!pte_present(pte), folio);(quoted diff) +5
-0+ vfn = addr >> PAGE_SIZE; + nr = pte_size(pte); + nr = ALIGN_DOWN(vfn + nr, nr) - vfn; + ptep += nr; +while (ptep != end_ptep) {(quoted diff) +1
-0+ pte = ptep_get(ptep);nr = pte_size(pte); if (any_writable) writable = !!pte_write(pte); pte = __pte_batch_clear_ignored(pte); if (!pte_same(pte, expected_pte)) break; /* * Stop immediately once we reached the end of the folio. In * corner cases the next PFN might fall into a different * folio. */(quoted diff) +1
-1- if (pte_pfn(pte) == folio_end_pfn) + if (pte_pfn(pte) >= folio_end_pfn)break; if (any_writable) *any_writable |= writable;(quoted diff) +3
-2- expected_pte = pte_next_pfn(expected_pte); - ptep++; + for (i = 0; i < nr; i++) + expected_pte = pte_next_pfn(expected_pte); + ptep += nr;} return ptep - start_ptep; }
Quoted 5 lines
So I'm wondering if instead of enabling
pte_next_pfn()for all the arches, perhaps its actually better to exposepte_pgprot()for all the arches. Then we can be much more flexible about generating ptes withpfn_pte(pfn, pgprot).What do you think?
The pte_pgprot() stuff is just nasty IMHO.
Likely it's best to simply convert pte_next_pfn() to something like pte_advance_pfns(). The we could just have
#define pte_next_pfn(pte) pte_advance_pfns(pte, 1)
That should be fairly easy to do on top (based on PFN_PTE_SHIFT). And only 3 archs (x86-64, arm64, and powerpc) need slight care to replace a hardcoded "1" by an integer we pass in.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 82 lines · David Hildenbrand
Quoted 81 lines · Ryan Roberts
Quoted 15 lines · David Hildenbrand
Now that the rmap overhaul[1] is upstream that provides a clean interface for rmap batching, let's implement PTE batching during fork when processing PTE-mapped THPs.
This series is partially based on Ryan's previous work[2] to implement cont-pte support on arm64, but its a complete rewrite based on [1] to optimize all architectures independent of any such PTE bits, and to use the new rmap batching functions that simplify the code and prepare for further rmap accounting changes.
We collect consecutive PTEs that map consecutive pages of the same large folio, making sure that the other PTE bits are compatible, and (a) adjust the refcount only once per batch, (b) call rmap handling functions only once per batch and (c) perform batch PTE setting/updates.
Quoted 4 lines
While this series should be beneficial for adding cont-pte support on ARM64[2], it's one of the requirements for maintaining a total mapcount[3] for large folios with minimal added overhead and further changes[4] that build up on top of the total mapcount.
I'm currently rebasing my contpte work onto this series, and have hit a problem. I need to expose the "size" of a pte (
pte_size()) and skip forward to the start of the next (cont)pte every time through thefolio_pte_batch()loop. Butpte_next_pfn()only allows advancing by 1 pfn; I need to advance by nr pfns:static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
pte_t*start_ptep,pte_tpte, intmax_nr, bool*any_writable) {
unsigned longfolio_end_pfn=folio_pfn(folio)+folio_nr_pages(folio); constpte_t*end_ptep=start_ptep+max_nr;
pte_texpected_pte= __pte_batch_clear_ignored(pte_next_pfn(pte)); -pte_t*ptep =start_ptep+ 1;
+pte_t*ptep =start_ptep;
+ int vfn, nr, i;
bool writable;if (
any_writable)
*any_writable= false;VM_WARN_ON_FOLIO(!pte_present(pte), folio);
+ vfn = addr >>
PAGE_SIZE;
+ nr =pte_size(pte);
+ nr =ALIGN_DOWN(vfn + nr, nr)- vfn;
+ ptep += nr;
+
while (ptep !=end_ptep) {
+ pte =ptep_get(ptep);
nr =pte_size(pte);
if (any_writable)
writable = !!pte_write(pte);
pte =__pte_batch_clear_ignored(pte);if (!pte_same(pte,
expected_pte))
break;/*
* Stop immediately once we reached the end of the folio. In
* corner cases the next PFN might fall into a different
* folio.
*/
- if (pte_pfn(pte)==folio_end_pfn)
+ if (pte_pfn(pte)>=folio_end_pfn)
break;if (
any_writable)
*any_writable|= writable;-
expected_pte=pte_next_pfn(expected_pte); - ptep++;
+ for (i = 0; i < nr; i++)
+expected_pte=pte_next_pfn(expected_pte); + ptep += nr;
}return ptep -
start_ptep;
}
Quoted 7 lines
Quoted 5 lines
So I'm wondering if instead of enabling
pte_next_pfn()for all the arches, perhaps its actually better to exposepte_pgprot()for all the arches. Then we can be much more flexible about generating ptes withpfn_pte(pfn, pgprot).What do you think?
The
pte_pgprot()stuff is just nasty IMHO.
I dunno; we have pfn_pte() which takes a pfn and a pgprot. It seems reasonable that we should be able to do the reverse.
Quoted 9 lines
Likely it's best to simply convert
pte_next_pfn()to something likepte_advance_pfns(). The we could just have#define
pte_next_pfn(pte)pte_advance_pfns(pte, 1)That should be fairly easy to do on top (based on
PFN_PTE_SHIFT). And only 3 archs (x86-64, arm64, and powerpc) need slight care to replace a hardcoded "1" by an integer we pass in.
I thought we agreed powerpc was safe to just define PFN_PTE_SHIFT? But, yeah, the principle works I guess. I guess I can do this change along with my series.
Quoted 1 line
Quoted 87 lines · Ryan Roberts
Quoted 86 lines · David Hildenbrand
Quoted 85 lines · Ryan Roberts
Quoted 15 lines · David Hildenbrand
Now that the rmap overhaul[1] is upstream that provides a clean interface for rmap batching, let's implement PTE batching during fork when processing PTE-mapped THPs.
This series is partially based on Ryan's previous work[2] to implement cont-pte support on arm64, but its a complete rewrite based on [1] to optimize all architectures independent of any such PTE bits, and to use the new rmap batching functions that simplify the code and prepare for further rmap accounting changes.
We collect consecutive PTEs that map consecutive pages of the same large folio, making sure that the other PTE bits are compatible, and (a) adjust the refcount only once per batch, (b) call rmap handling functions only once per batch and (c) perform batch PTE setting/updates.
Quoted 4 lines
While this series should be beneficial for adding cont-pte support on ARM64[2], it's one of the requirements for maintaining a total mapcount[3] for large folios with minimal added overhead and further changes[4] that build up on top of the total mapcount.
I'm currently rebasing my contpte work onto this series, and have hit a problem. I need to expose the "size" of a pte (
pte_size()) and skip forward to the start of the next (cont)pte every time through thefolio_pte_batch()loop. Butpte_next_pfn()only allows advancing by 1 pfn; I need to advance by nr pfns:static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
pte_t*start_ptep,pte_tpte, intmax_nr, bool*any_writable)
{
unsigned longfolio_end_pfn=folio_pfn(folio)+folio_nr_pages(folio); constpte_t*end_ptep=start_ptep+max_nr;
pte_texpected_pte= __pte_batch_clear_ignored(pte_next_pfn(pte));
-pte_t*ptep =start_ptep+ 1;
+pte_t*ptep =start_ptep;
+ int vfn, nr, i;
bool writable;if (
any_writable)
*any_writable= false;VM_WARN_ON_FOLIO(!pte_present(pte), folio);
+ vfn = addr >>
PAGE_SIZE;
+ nr =pte_size(pte);
+ nr =ALIGN_DOWN(vfn + nr, nr)- vfn;
+ ptep += nr;
+
while (ptep !=end_ptep) {
+ pte =ptep_get(ptep);
nr =pte_size(pte);
if (any_writable)
writable = !!pte_write(pte);
pte =__pte_batch_clear_ignored(pte);if (!pte_same(pte,
expected_pte))
break;/*
* Stop immediately once we reached the end of the folio. In
* corner cases the next PFN might fall into a different
* folio.
*/
- if (pte_pfn(pte)==folio_end_pfn)
+ if (pte_pfn(pte)>=folio_end_pfn)
break;if (
any_writable)
*any_writable|= writable;-
expected_pte=pte_next_pfn(expected_pte); - ptep++;
+ for (i = 0; i < nr; i++)
+expected_pte=pte_next_pfn(expected_pte); + ptep += nr;
}return ptep -
start_ptep;
}So I'm wondering if instead of enabling
pte_next_pfn()for all the arches, perhaps its actually better to exposepte_pgprot()for all the arches. Then we can be much more flexible about generating ptes withpfn_pte(pfn, pgprot).
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
What do you think?
The
pte_pgprot()stuff is just nasty IMHO.I dunno; we have
pfn_pte()which takes a pfn and a pgprot. It seems reasonable that we should be able to do the reverse.
But pte_pgprot() is only available on a handful of architectures, no? It would be nice to have a completely generic pte_next_pfn() / pte_advance_pfns(), though.
Anyhow, this is all "easy" to rework later. Unless I am missing something, the low hanging fruit is simply using PFN_PTE_SHIFT for now that exists on most archs already.
Quoted 5 lines
Quoted 5 lines
Likely it's best to simply convert
pte_next_pfn()to something likepte_advance_pfns(). The we could just have
Quoted 8 lines
Quoted 5 lines
#define
pte_next_pfn(pte)pte_advance_pfns(pte, 1)That should be fairly easy to do on top (based on
PFN_PTE_SHIFT). And only 3 archs (x86-64, arm64, and powerpc) need slight care to replace a hardcoded "1" by an integer we pass in.I thought we agreed powerpc was safe to just define
PFN_PTE_SHIFT? But, yeah, the principle works I guess. I guess I can do this change along with my series.
It is, if nobody insists on that micro-optimization on powerpc.
If there is good reason to invest more time and effort right now on the pte_pgprot approach, then please let me know :)
Signature · 3 lines
Cheers, David / dhildenb
Quoted 114 lines · David Hildenbrand
Quoted 88 lines · Ryan Roberts
Quoted 87 lines · David Hildenbrand
Quoted 86 lines · Ryan Roberts
Quoted 15 lines · David Hildenbrand
Now that the rmap overhaul[1] is upstream that provides a clean interface for rmap batching, let's implement PTE batching during fork when processing PTE-mapped THPs.
This series is partially based on Ryan's previous work[2] to implement cont-pte support on arm64, but its a complete rewrite based on [1] to optimize all architectures independent of any such PTE bits, and to use the new rmap batching functions that simplify the code and prepare for further rmap accounting changes.
We collect consecutive PTEs that map consecutive pages of the same large folio, making sure that the other PTE bits are compatible, and (a) adjust the refcount only once per batch, (b) call rmap handling functions only once per batch and (c) perform batch PTE setting/updates.
Quoted 4 lines
While this series should be beneficial for adding cont-pte support on ARM64[2], it's one of the requirements for maintaining a total mapcount[3] for large folios with minimal added overhead and further changes[4] that build up on top of the total mapcount.
I'm currently rebasing my contpte work onto this series, and have hit a problem.
I need to expose the "size" of a pte (pte_size()) and skip forward to the start of the next (cont)pte every time through thefolio_pte_batch()loop. Butpte_next_pfn()only allows advancing by 1 pfn; I need to advance by nr pfns:static inline int folio_pte_batch(struct folio *folio, unsigned long addr,
pte_t*start_ptep,pte_tpte, intmax_nr, bool*any_writable) {
unsigned longfolio_end_pfn=folio_pfn(folio)+folio_nr_pages(folio); constpte_t*end_ptep=start_ptep+max_nr;
pte_texpected_pte= __pte_batch_clear_ignored(pte_next_pfn(pte)); -pte_t*ptep =start_ptep+ 1;
+pte_t*ptep =start_ptep;
+ int vfn, nr, i;
bool writable;if (
any_writable)
*any_writable= false;VM_WARN_ON_FOLIO(!pte_present(pte), folio);
+ vfn = addr >>
PAGE_SIZE;
+ nr =pte_size(pte);
+ nr =ALIGN_DOWN(vfn + nr, nr)- vfn;
+ ptep += nr;
+
while (ptep !=end_ptep) {
+ pte =ptep_get(ptep);
nr =pte_size(pte);
if (any_writable)
writable = !!pte_write(pte);
pte =__pte_batch_clear_ignored(pte);if (!pte_same(pte,
expected_pte))
break;/*
* Stop immediately once we reached the end of the folio. In
* corner cases the next PFN might fall into a different
* folio.
*/
- if (pte_pfn(pte)==folio_end_pfn)
+ if (pte_pfn(pte)>=folio_end_pfn)
break;if (
any_writable)
*any_writable|= writable;-
expected_pte=pte_next_pfn(expected_pte); - ptep++;
+ for (i = 0; i < nr; i++)
+expected_pte=pte_next_pfn(expected_pte); + ptep += nr;
}return ptep -
start_ptep;
}So I'm wondering if instead of enabling
pte_next_pfn()for all the arches, perhaps its actually better to exposepte_pgprot()for all the arches. Then we can be much more flexible about generating ptes withpfn_pte(pfn, pgprot).Quoted 6 lines
Quoted 3 lines
Quoted 1 line
What do you think?
The
pte_pgprot()stuff is just nasty IMHO.I dunno; we have
pfn_pte()which takes a pfn and a pgprot. It seems reasonable that we should be able to do the reverse.But
pte_pgprot()is only available on a handful of architectures, no? It would be nice to have a completely genericpte_next_pfn()/pte_advance_pfns(), though.Anyhow, this is all "easy" to rework later. Unless I am missing something, the low hanging fruit is simply using
PFN_PTE_SHIFTfor now that exists on most archs already.Quoted 11 lines
Quoted 10 lines
Likely it's best to simply convert
pte_next_pfn()to something likepte_advance_pfns(). The we could just have#define
pte_next_pfn(pte)pte_advance_pfns(pte, 1)That should be fairly easy to do on top (based on
PFN_PTE_SHIFT). And only 3 archs (x86-64, arm64, and powerpc) need slight care to replace a hardcoded "1" by an integer we pass in.
Quoted 8 lines
Quoted 2 lines
I thought we agreed powerpc was safe to just define
PFN_PTE_SHIFT? But, yeah, the principle works I guess. I guess I can do this change along with my series.It is, if nobody insists on that micro-optimization on powerpc.
If there is good reason to invest more time and effort right now on the
pte_pgprotapproach, then please let me know :)
No I think you're right. I thought pte_pgprot() was implemented by more arches, but there are 13 without it, so clearly a lot of effort to plug that gap. I'll take the approach you suggest with pte_advance_pfns(). It'll just require mods to x86 and arm64, +/- ppc.