Skip to contentSkip to conversation

mm/memory: reuse the whole exclusive large folio on a write fault

PATCH Superseded Yuan-Hao Hsu 4 participants 18 messages Sep 18 – Sep 29

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm/memory: reuse the whole exclusive large folio on a write fault

fork() maps the anonymous pages of the parent read-only and clears PageAnonExclusive on them. Once the child has exec'ed or exited, a write fault of the parent ends up in the reuse path of do_wp_page():
wp_can_reuse_anon_folio() finds that all references to the folio come from mappings in this MM, the page is marked exclusive again and its PTE is made writable.

For a large folio that check is about the folio, and what it finds holds for every page of it. Still only the page that faulted is marked exclusive and only its PTE becomes writable, so each of the other pages takes a write fault of its own, and each of those takes the large mapcount lock to find out the same thing again: 16 faults for a 64K folio, 512 for a 2M THP that is mapped by PTEs. The same THP mapped by a PMD is reused by one fault in do_huge_pmd_wp_page(), do_swap_page() maps all PTEs of an exclusive large folio writable at once, and numa_rebuild_large_mapping() upgrades all PTEs of the folio from one hinting fault.

Commit 1da190f4d0a6 ("mm: Copy-on-Write (COW) reuse support for PTE-mapped THP") left this for later because faulting around might increase the COW latency. Numbers for that are below.

When a large folio has been found exclusive, walk the part of it that this page table maps inside the VMA. PTEs that still map it read-only are batched with folio_pte_batch_flags(), their pages are marked exclusive, and where can_change_pte_writable() agrees the batch is made writable with modify_prot_start_ptes()/modify_prot_commit_ptes(), the way mprotect() does it. That leaves NUMA hinting and uffd-wp PTEs alone, keeps soft-dirty tracking exact, and on arm64 writes a contpte block back as a whole where ptep_set_access_flags() on a single PTE has to unfold it. Pages that cannot be made writable are marked exclusive all the same, so their own fault skips the folio check. The PTE that faulted is in one of the batches and is then completed by wp_page_reuse() as before. Small folios, PMD-mapped THPs, unsharing faults and the copy path are not changed.

x86-64, i7-12700KF, 256 MiB of anonymous memory, fork(), the child exits, then the parent stores to the memory. Medians of 15 runs, two boots of each kernel, taken alternately:

v7.3-rc3+ patched
one byte per page, a clock_gettime() between the stores

    write faults
      4K pages            65,601              65,601
      64K mTHP            65,601               4,161
      1M mTHP             65,601                 321
      2M THP, PTE-mapped  65,601                 193
      2M THP, PMD-mapped     193                 193
    time (ms)
      4K pages            29.7 / 30.1         30.0 / 31.1
      64K mTHP            29.4 / 29.7          5.5 /  5.6
      1M mTHP             29.3 / 29.8          3.9 /  3.9
      2M THP, PTE-mapped  29.9 / 29.2          3.8 /  3.9
      2M THP, PMD-mapped   2.5 /  2.5          2.5 /  2.6
  memset() of all of it (ms)
      4K pages            57.2 / 56.5         56.9 / 57.1
      64K mTHP            56.5 / 56.3         39.0 / 38.7
      2M THP, PTE-mapped  58.6 / 56.5         36.8 / 37.7
      2M THP, PMD-mapped  37.5 / 35.9         35.9 / 36.5
  8 threads, one byte per page, random order (ms)
      64K mTHP             6.1 /  6.2          0.9 /  1.0
      2M THP, PTE-mapped   6.0 /  6.7          0.7 /  0.7

The latency of the one fault that now does the work for the folio, measured as the time of the store that takes it, against 420 ns for a reuse fault today (medians, ns):

                          reuse fault   fault that     COW fault that
                          (patched)     allocated it   copies 4K
      64K mTHP                   730          4,600          1,500
      1M mTHP                  5,500         63,000          1,500
      2M THP, PTE-mapped      10,100        126,000          1,500

That is 14 to 20 ns per PTE. Builds that differ only by NOPs in front of the new function take either 10,100 or 7,500 ns for the 2M folio, with a period of 32 bytes: it is the loop of modify_prot_commit_ptes() that changes speed with its address.

Capping the walk to the 16 PTEs around the fault instead was measured as
well: it takes 5.4 ms where the above takes 3.9 ms on 1M and 2M folios, it is slower than today when only one page per 64K is written (3.1 ms against 1.9 ms, the whole folio takes 1.5 ms), and it is only ahead when no more than one page per 2M is ever written (0.1 ms against 1.3 ms for the 256 MiB).

Redis 7.0.15 with 1.3 M keys of 512 bytes on 64K mTHP, BGSAVE and then 1,000,000 SETs: the faults of redis-server during the SETs go from 262,100 to 18,500, its CPU time from 1.48-1.53 s to 1.25-1.33 s, and redis-benchmark reports 742,000 to 794,000 requests per second instead of 652,000 to 658,000. With THP off all three stay where they were.

arm64 was only run under QEMU, for the counters and with DEBUG_VM and
PAGE_TABLE_CHECK: the faults are the same as above, and with 64K folios the first pass after fork() unfolds every contpte block today (512 contpte_convert() calls for 512 blocks, and nothing folds them again) while none is unfolded with this patch.

What does not get faster on x86 are stores to pages that this CPU still has a read-only TLB entry for. The fault makes the PTEs writable but, like mprotect(), does not flush, so such a page still takes a fault, a spurious one that costs about the same as the reuse fault it replaces. That happens to pages that were read since fork(): reading the 16 pages of every 64K folio before storing to them takes 65,500 faults and 28 ms before and after. And it happens in a loop that does nothing but store one byte to every page in ascending order, which is what the reuse-byte mode of David's pte-mapped-folio-benchmarks does (120 ms before and after for 1 GiB of 64K folios; 2M folios: 119 ms to 18 ms; the reuse mode, a memset(), goes from 232 ms to 157 ms with 64K folios): while the first store of a folio is faulting, the CPU has already run the next stores speculatively and has filled the TLB with the read-only translations of their pages. Counting with kprobes, such a run enters handle_mm_fault() 135,687 times and do_wp_page() 8,457 times; with an LFENCE after every store the faults are 4,100 instead of 65,400 and the loop takes 4 ms instead of 29 ms. The PMD-mapped case, which this patch does not touch, shows the same: 8,800 faults for 128 THPs, 133 with the LFENCE.

With a flush_tlb_local() in the new function, as an experiment, the ascending loop takes 4.1 ms instead of 28 ms on 64K folios, the read-then-store loop 5.3 ms instead of 28 ms and the memset() 21 ms instead of 38 ms, for a fault of 870 instead of 730 ns and 11% more time for the pass with the clock_gettime(). Generic code has no way to ask x86 for a flush that stays on this CPU, so that is left for later.

Assisted-by LLM sparse
---
 mm/memory.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 66 insertions(+), 2 deletions(-)
mm/memory.c +66 -2
index 8b0c2c735d3d..85c883d1e558 100644
@@ -4355,6 +4355,64 @@ static bool wp_can_reuse_anon_folio(struct folio *folio,
 	return true;
 }
 
+/*
+ * wp_can_reuse_anon_folio() found a large folio to be exclusive to this MM.
+ * That holds for all of its pages and not only for the one that faulted: mark
+ * the ones that this page table maps exclusive as well and map them writable,
+ * like mprotect() would. Each of them would otherwise take a write fault of
+ * its own that repeats the check on the very same folio.
+ *
+ * The PTE that faulted is among them; wp_page_reuse() completes it.
+ */
+static void wp_reuse_large_anon_folio(struct vm_fault *vmf,
+		struct folio *folio)
+{
+	const fpb_t flags = FPB_RESPECT_WRITE | FPB_RESPECT_SOFT_DIRTY;
+	const unsigned long idx = folio_page_idx(folio, vmf->page);
+	struct vm_area_struct *vma = vmf->vma;
+	unsigned long addr = vmf->address;
+	unsigned long pt_start = ALIGN_DOWN(addr, PMD_SIZE);
+	unsigned long nr_before, nr_after, end;
+	struct page *page;
+	unsigned int nr, i;
+	pte_t *ptep, pte;
+
+	/* Stay within the folio, the VMA and the page table. */
+	nr_before = min3(idx, (addr - pt_start) >> PAGE_SHIFT,
+			 (addr - vma->vm_start) >> PAGE_SHIFT);
+	nr_after = min3(folio_nr_pages(folio) - idx,
+			(pt_start + PMD_SIZE - addr) >> PAGE_SHIFT,
+			(vma->vm_end - addr) >> PAGE_SHIFT);
+	end = addr + (nr_after << PAGE_SHIFT);
+	addr -= nr_before << PAGE_SHIFT;
+	ptep = vmf->pte - nr_before;
+	page = vmf->page - nr_before;
+
+	for (; addr != end; addr += nr * PAGE_SIZE, ptep += nr, page += nr) {
+		pte = ptep_get(ptep);
+		nr = 1;
+
+		/* Unmapped or replaced since, or writable already. */
+		if (!pte_present(pte) || pte_pfn(pte) != page_to_pfn(page) ||
+		    pte_write(pte))
+			continue;
+
+		nr = folio_pte_batch_flags(folio, NULL, ptep, &pte,
+					   (end - addr) >> PAGE_SHIFT, flags);
+		for (i = 0; i < nr; i++)
+			if (!PageAnonExclusive(page + i))
+				SetPageAnonExclusive(page + i);
+
+		/* The PTEs of a batch agree on everything this looks at. */
+		if (!can_change_pte_writable(vma, addr, pte))
+			continue;
+
+		pte = modify_prot_start_ptes(vma, addr, ptep, nr);
+		modify_prot_commit_ptes(vma, addr, ptep, pte,
+					pte_mkwrite(pte, vma), nr);
+	}
+}
+
 /*
  * This routine handles present pages, when
  * * users try to write to a shared page (FAULT_FLAG_WRITE)
@@ -4449,8 +4507,14 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
 	 */
 	if (folio && folio_test_anon(folio) &&
 	    (PageAnonExclusive(vmf->page) || wp_can_reuse_anon_folio(folio, vma))) {
-		if (!PageAnonExclusive(vmf->page))
-			SetPageAnonExclusive(vmf->page);
+		if (!PageAnonExclusive(vmf->page)) {
+			if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
+			    folio_test_large(folio) && likely(!unshare) &&
+			    likely(vma->vm_flags & VM_WRITE))
+				wp_reuse_large_anon_folio(vmf, folio);
+			else
+				SetPageAnonExclusive(vmf->page);
+		}
 		if (unlikely(unshare)) {
 			pte_unmap_unlock(vmf->pte, vmf->ptl);
 			return 0;

base-commit: 238650ef6c7c7cca08e032527329424c9fbd70e5

Signature · 1 line
2.43.0
#1 lore raw
Quoted 117 lines · Yuan-Hao Hsu

fork() maps the anonymous pages of the parent read-only and clears PageAnonExclusive on them. Once the child has exec'ed or exited, a write fault of the parent ends up in the reuse path of do_wp_page():
wp_can_reuse_anon_folio() finds that all references to the folio come from mappings in this MM, the page is marked exclusive again and its PTE is made writable.

For a large folio that check is about the folio, and what it finds holds for every page of it. Still only the page that faulted is marked exclusive and only its PTE becomes writable, so each of the other pages takes a write fault of its own, and each of those takes the large mapcount lock to find out the same thing again: 16 faults for a 64K folio, 512 for a 2M THP that is mapped by PTEs. The same THP mapped by a PMD is reused by one fault in do_huge_pmd_wp_page(), do_swap_page() maps all PTEs of an exclusive large folio writable at once, and numa_rebuild_large_mapping() upgrades all PTEs of the folio from one hinting fault.

Commit 1da190f4d0a6 ("mm: Copy-on-Write (COW) reuse support for PTE-mapped THP") left this for later because faulting around might increase the COW latency. Numbers for that are below.

When a large folio has been found exclusive, walk the part of it that this page table maps inside the VMA. PTEs that still map it read-only are batched with folio_pte_batch_flags(), their pages are marked exclusive, and where can_change_pte_writable() agrees the batch is made writable with modify_prot_start_ptes()/modify_prot_commit_ptes(), the way mprotect() does it. That leaves NUMA hinting and uffd-wp PTEs alone, keeps soft-dirty tracking exact, and on arm64 writes a contpte block back as a whole where ptep_set_access_flags() on a single PTE has to unfold it. Pages that cannot be made writable are marked exclusive all the same, so their own fault skips the folio check. The PTE that faulted is in one of the batches and is then completed by wp_page_reuse() as before. Small folios, PMD-mapped THPs, unsharing faults and the copy path are not changed.

x86-64, i7-12700KF, 256 MiB of anonymous memory, fork(), the child exits, then the parent stores to the memory. Medians of 15 runs, two boots of each kernel, taken alternately:

v7.3-rc3+ patched
one byte per page, a clock_gettime() between the stores

    write faults
      4K pages            65,601              65,601
      64K mTHP            65,601               4,161
      1M mTHP             65,601                 321
      2M THP, PTE-mapped  65,601                 193
      2M THP, PMD-mapped     193                 193
    time (ms)
      4K pages            29.7 / 30.1         30.0 / 31.1
      64K mTHP            29.4 / 29.7          5.5 /  5.6
      1M mTHP             29.3 / 29.8          3.9 /  3.9
      2M THP, PTE-mapped  29.9 / 29.2          3.8 /  3.9
      2M THP, PMD-mapped   2.5 /  2.5          2.5 /  2.6
  memset() of all of it (ms)
      4K pages            57.2 / 56.5         56.9 / 57.1
      64K mTHP            56.5 / 56.3         39.0 / 38.7
      2M THP, PTE-mapped  58.6 / 56.5         36.8 / 37.7
      2M THP, PMD-mapped  37.5 / 35.9         35.9 / 36.5
  8 threads, one byte per page, random order (ms)
      64K mTHP             6.1 /  6.2          0.9 /  1.0
      2M THP, PTE-mapped   6.0 /  6.7          0.7 /  0.7

The latency of the one fault that now does the work for the folio, measured as the time of the store that takes it, against 420 ns for a reuse fault today (medians, ns):

                          reuse fault   fault that     COW fault that
                          (patched)     allocated it   copies 4K
      64K mTHP                   730          4,600          1,500
      1M mTHP                  5,500         63,000          1,500
      2M THP, PTE-mapped      10,100        126,000          1,500

That is 14 to 20 ns per PTE. Builds that differ only by NOPs in front of the new function take either 10,100 or 7,500 ns for the 2M folio, with a period of 32 bytes: it is the loop of modify_prot_commit_ptes() that changes speed with its address.

Capping the walk to the 16 PTEs around the fault instead was measured as
well: it takes 5.4 ms where the above takes 3.9 ms on 1M and 2M folios, it is slower than today when only one page per 64K is written (3.1 ms against 1.9 ms, the whole folio takes 1.5 ms), and it is only ahead when no more than one page per 2M is ever written (0.1 ms against 1.3 ms for the 256 MiB).

Redis 7.0.15 with 1.3 M keys of 512 bytes on 64K mTHP, BGSAVE and then 1,000,000 SETs: the faults of redis-server during the SETs go from 262,100 to 18,500, its CPU time from 1.48-1.53 s to 1.25-1.33 s, and redis-benchmark reports 742,000 to 794,000 requests per second instead of 652,000 to 658,000. With THP off all three stay where they were.

arm64 was only run under QEMU, for the counters and with DEBUG_VM and
PAGE_TABLE_CHECK: the faults are the same as above, and with 64K folios the first pass after fork() unfolds every contpte block today (512 contpte_convert() calls for 512 blocks, and nothing folds them again) while none is unfolded with this patch.

What does not get faster on x86 are stores to pages that this CPU still has a read-only TLB entry for. The fault makes the PTEs writable but, like mprotect(), does not flush, so such a page still takes a fault, a spurious one that costs about the same as the reuse fault it replaces. That happens to pages that were read since fork(): reading the 16 pages of every 64K folio before storing to them takes 65,500 faults and 28 ms before and after. And it happens in a loop that does nothing but store one byte to every page in ascending order, which is what the reuse-byte mode of David's pte-mapped-folio-benchmarks does (120 ms before and after for 1 GiB of 64K folios; 2M folios: 119 ms to 18 ms; the reuse mode, a memset(), goes from 232 ms to 157 ms with 64K folios): while the first store of a folio is faulting, the CPU has already run the next stores speculatively and has filled the TLB with the read-only translations of their pages. Counting with kprobes, such a run enters handle_mm_fault() 135,687 times and do_wp_page() 8,457 times; with an LFENCE after every store the faults are 4,100 instead of 65,400 and the loop takes 4 ms instead of 29 ms. The PMD-mapped case, which this patch does not touch, shows the same: 8,800 faults for 128 THPs, 133 with the LFENCE.

Quoted 8 lines

With a flush_tlb_local() in the new function, as an experiment, the ascending loop takes 4.1 ms instead of 28 ms on 64K folios, the read-then-store loop 5.3 ms instead of 28 ms and the memset() 21 ms instead of 38 ms, for a fault of 870 instead of 730 ns and 11% more time for the pass with the clock_gettime(). Generic code has no way to ask x86 for a flush that stays on this CPU, so that is left for later.

Assisted-by LLM sparse

My review backlog is large enough for me to just go through this wall of text.

There were previous discussions on this, in particular around how much we should actually try operating around the target PTE.

How did you use the LLM for coming up with this patch + description?

Signature · 3 lines
Cheers,

David
#2 lore raw
Quoted 2 lines · David Hildenbrand (Arm)

There were previous discussions on this, in particular around how much we should actually try operating around the target PTE.

Yes, Barry's RFC from 2024 and your replies to it, where you were fine with contpte granularity and not convinced about anything bigger. That is why the description has the numbers for both: capped to 16 PTEs, the 1M and 2M cases take 5.4 ms instead of 3.9 ms, and the only pattern where the cap wins is one store per 2M. If you would rather start with the contpte-sized version, that is a two-line change and I can send that version instead.

Quoted 1 line

How did you use the LLM for coming up with this patch + description?

I did the initial investigation and made the design decisions myself. Once I had a clear idea of the approach and the overall structure, I used an LLM to help with parts of the implementation and the test programs. I also used it to help with drafting the commit message.

Before sending the patch, I went through it line by line and checked the test results and measurements to make sure they matched what the code was actually doing and that there were no obvious issues.

Lorenzo, for your bot's question: the reuse decision in wp_can_reuse_anon_folio() is per folio. The patch applies it to the other PTEs of the folio within the same VMA and page table using the mprotect helpers (can_change_pte_writable() and modify_prot_*_ptes()), so soft-dirty, uffd-wp and NUMA hinting PTEs are handled the way mprotect() handles them, and the PTE that faulted is finished by wp_page_reuse() as before.

The part I spent most of the time on was the measurements, including the cases where it does not help.

Yuan-Hao Hsu

#3 lore raw
Quoted 10 lines · Yuan-Hao Hsu
Quoted 2 lines · David Hildenbrand (Arm)

There were previous discussions on this, in particular around how much we should actually try operating around the target PTE.

Yes, Barry's RFC from 2024 and your replies to it, where you were fine with contpte granularity and not convinced about anything bigger. That is why the description has the numbers for both: capped to 16 PTEs, the 1M and 2M cases take 5.4 ms instead of 3.9 ms, and the only pattern where

Hi Yuan-Hao,

I'm fine with your follow-up work, but I'd appreciate it if you could mention the previous work and include the link in your changelog, along with an explanation of how you address David's concerns.

Quoted 3 lines

the cap wins is one store per 2M. If you would rather start with the contpte-sized version, that is a two-line change and I can send that version instead.

I think a major concern is that we may spend too much time scanning PTEs to determine whether we can batch them. If we don't support CONT-PTE, we may simply scan fewer PTEs; if we do support CONT-PTE, we may end up scanning more PTEs. So we need to show that the scanning cost is controlled and explain how it is bounded.

Quoted 11 lines
Quoted 2 lines

How did you use the LLM for coming up with this patch + description?

I did the initial investigation and made the design decisions myself. Once I had a clear idea of the approach and the overall structure, I used an LLM to help with parts of the implementation and the test programs. I also used it to help with drafting the commit message.

Before sending the patch, I went through it line by line and checked the test results and measurements to make sure they matched what the code was actually doing and that there were no obvious issues.

I also personally feel that both the changelog and the code are too long to read. Could we somehow split the changes into smaller pieces and describe how each of those concerns is addressed at a finer granularity?

Quoted 9 lines

Lorenzo, for your bot's question: the reuse decision in wp_can_reuse_anon_folio() is per folio. The patch applies it to the other PTEs of the folio within the same VMA and page table using the mprotect helpers (can_change_pte_writable() and modify_prot_*_ptes()), so soft-dirty, uffd-wp and NUMA hinting PTEs are handled the way mprotect() handles them, and the PTE that faulted is finished by wp_page_reuse() as before.

Quoted 3 lines

The part I spent most of the time on was the measurements, including the cases where it does not help.

Best Regards
Barry

#4 lore raw
Quoted 3 lines · Barry Song

I'm fine with your follow-up work, but I'd appreciate it if you could mention the previous work and include the link in your changelog, along with an explanation of how you address David's concerns.

Will do in v2.

Quoted 5 lines

I think a major concern is that we may spend too much time scanning PTEs to determine whether we can batch them. If we don't support CONT-PTE, we may simply scan fewer PTEs; if we do support CONT-PTE, we may end up scanning more PTEs. So we need to show that the scanning cost is controlled and explain how it is bounded.

The walk stays within the folio, the VMA and the page table, so it touches at most the PTEs that map this folio in this page table: 16 for a 64K folio, PTRS_PER_PTE (512 with 4K pages) at the most. Each PTE is read once by folio_pte_batch_flags(), the same scan fork() and mprotect() already run over these PTEs.

Measured, that is 14-20 ns per PTE. For 512 PTEs the fault takes 10 us, 1 us of it the scan, the rest marking the pages exclusive and writing the PTEs. Those are PTEs that would otherwise each take a 420 ns fault. The pattern where the work is wasted, one store per folio, is the one the description gives the numbers for.

Quoted 4 lines

I also personally feel that both the changelog and the code are too long to read. Could we somehow split the changes into smaller pieces and describe how each of those concerns is addressed at a finer granularity?

Yes, and sorry for the wall of text. v2 is two patches: 1/2 handles an aligned block of 16 PTEs, the contpte-sized version David was fine with in 2024; 2/2 lifts that to the folio. Each has its own numbers, the earlier discussion is linked, and the changelogs are cut to what is needed to judge the change.

#5 lore raw
In reply to Yuan-Hao Hsu

This is the friendly patch-bot of Lorenzo Stoakes.

You have sent him a patch/series that has triggered this response.

He used to manually respond to these common problems, but in order to save his sanity (he kept writing the same thing over and over, yet to different people), I was created.

Hopefully you will not take offence and will fix the problem in your patch and resubmit it so that it can be accepted into the Linux kernel tree.

When sending emails to mm:

For one of several possible reasons this mail has triggered an AI detector script.

Thank you for using the Assisted-by tag to acknowledge this.

However, please do note the policy at
https://docs.kernel.org/process/generated-content.html most notably:

If tools permit you to generate a contribution automatically, expect additional scrutiny in proportion to how much of it was generated.

As with the output of any tooling, the result may be incorrect or inappropriate. You are expected to understand and to be able to defend everything you submit. If you are unable to do so, then do not submit the resulting changes.

If you do so anyway, maintainers are entitled to reject your series without detailed review.

This change is significant, and you are a relative newcomer to mm. As a result, I must ask that you provide some details as to how you solved this problem and your overall understanding of these changes and mm as a whole.

In general, if you are a newcomer to mm, we expect you to do smaller work before moving on to larger changes, so you build understanding of both the technical aspects of mm and how we do things.

If you wish to discuss this problem further, or you have questions about how to resolve this issue, please feel free to respond to this email and Lorenzo will reply once he has dug out from the pending patches received from other developers.

thanks,

Lorenzo's patch email bot

[ Idea shamelessly stolen from greg-kh ]

--
Cheers, Lorenzo

#6 lore raw

v2 cover mm/memory: reuse the whole exclusive large folio on a write fault

In reply to Yuan-Hao Hsu

After fork() and the child's exit, the parent's write faults reuse a large anonymous folio one PTE at a time, although wp_can_reuse_anon_folio() has already found the whole folio exclusive.

v2, after the comments on v1 [1]:
- Split in two. 1/2 handles an aligned block of 16 PTEs around the fault, the contpte-sized version David was fine with in the 2024 discussion of Barry's RFC [2]. 2/2 lifts that to the folio.
- The earlier discussion is linked; the changelogs say how its two reservations, the latency of the individual fault and how far to go around it, are answered.
- The bound of the walk is spelled out (Barry): folio, VMA and block or page table, each PTE read once by folio_pte_batch_flags().
- Changelogs cut to what is needed to judge the change.
- Same base as v1; 1/2 + 2/2 is the code of v1.

Two points from the AI review of v1, both done as mprotect() does them:
change_pte_range() does not flush_cache_range() before making PTEs writable, and it makes clean exclusive anonymous PTEs writable without pte_mkdirty() (the dirty rule is for shared file mappings, see can_change_shared_pte_writable()).

Controls, unchanged: order-0 pages, PMD-mapped THPs, the COW copy path and the order-0 and cow/fork/write-fault modes of David's pte-mapped-folio-benchmarks.

What does not get faster on x86: stores to pages this CPU still holds a read-only TLB entry for. The fault makes the PTEs writable but, like mprotect(), does not flush, so such a page takes one spurious fault, about the cost of the reuse fault it replaces. That is the case for pages read since fork(), and for a loop that only stores one byte per page in ascending order (David's reuse-byte mode): the CPU runs the next stores speculatively while the first one faults and caches their read-only translations. Shown with kprobes (135,687 handle_mm_fault() for 8,457 do_wp_page()) and an LFENCE after every store (4,100 faults instead of 65,400); the untouched PMD-mapped case behaves the same. A flush_tlb_local() in the helper would fix it (that loop 28 -> 4 ms, memset() 38 -> 21 ms, +140 ns per fault), but generic code has no way to ask x86 for a flush that stays on this CPU, so that is for later.

Tested with DEBUG_VM, DEBUG_VM_PGTABLE, PROVE_LOCKING and
PAGE_TABLE_CHECK_ENFORCED: the mm selftests, a 12-scenario COW test (child alive, vmsplice, PROT_READ VMA inside the folio, soft-dirty and uffd-wp counts, mremap, holes, pageout, FOLL_FORCE), a fork/pageout/mprotect/vmsplice stress, NUMA balancing on numa=fake=2 (protnone PTEs left alone), and arm64 under QEMU for the counters. Cross-built for arm64 4K/16K/64K, i386 with and without PAE, x86 without THP, arm, arm nommu, riscv64, powerpc64le and s390x.

[1] https://lore.kernel.org/r/[email protected]
[2] https://lore.kernel.org/r/[email protected]

Yuan-Hao Hsu (2):
mm/memory: reuse 16 PTEs of an exclusive large folio on a write fault mm/memory: reuse the whole exclusive large folio on a write fault

 mm/memory.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 66 insertions(+), 2 deletions(-)

base-commit: 238650ef6c7c7cca08e032527329424c9fbd70e5

Signature · 1 line
2.43.0
#7 lore raw

PATCH v2 1/2 mm/memory: reuse 16 PTEs of an exclusive large folio on a write fault

fork() maps the anonymous pages of the parent read-only and clears PageAnonExclusive on them. Once the child has exec'ed or exited, the parent's write fault takes the reuse path of do_wp_page():
wp_can_reuse_anon_folio() finds that all references to the folio come from this MM, the page is marked exclusive again and its PTE is made writable.

For a large folio that check is about the folio and holds for every page of it, but only the page that faulted is marked exclusive and made writable. Each other page takes a write fault of its own and takes the large mapcount lock to find out the same thing again: 16 faults for a 64K folio. The same THP mapped by a PMD is reused by one fault in do_huge_pmd_wp_page(), and do_swap_page() maps all PTEs of an exclusive large folio writable at once.

Barry proposed reusing the whole mTHP from one fault in 2024 [1]. The reservations then were the latency of the individual write fault and how far to go around the faulting PTE: a contpte-sized block was fine, anything bigger not yet convincing (David's replies, linked below). Commit 1da190f4d0a6 ("mm: Copy-on-Write (COW) reuse support for PTE-mapped THP") then added the per-folio check and left faulting around for later.

This is the contpte-sized part. Once the folio is known to be exclusive, walk the aligned block of 16 PTEs around the fault, within the folio and the VMA. PTEs that still map the folio read-only are batched with folio_pte_batch_flags(), their pages are marked exclusive, and where can_change_pte_writable() agrees the batch is made writable with modify_prot_start_ptes()/modify_prot_commit_ptes(), as mprotect() does it. That leaves NUMA hinting and uffd-wp PTEs alone, keeps soft-dirty tracking exact, and on arm64 writes a contpte block back as a whole where ptep_set_access_flags() on one PTE has to unfold it. Pages that cannot be made writable are still marked exclusive, so their own fault skips the folio check. The PTE that faulted is in one of the batches and is completed by wp_page_reuse() as before. Small folios, PMD-mapped THPs, unsharing faults and the copy path are unchanged.

The cost is bounded by the block: at most 16 PTEs read once by folio_pte_batch_flags(), the scan fork() and mprotect() already run over them, 16 pages marked exclusive and 16 PTEs written. On x86-64 (i7-12700KF; medians of 15 runs, two boots of each kernel taken alternately) the fault that does that for a 64K folio takes 750 ns, against 420-440 ns for a reuse fault today and 4,600-5,000 ns for the fault that allocated the folio.

256 MiB of 64K folios after fork() and the child's exit, faults and time of the pass:

v7.3-rc3+ patched

  one byte per page, seq    65,601  30.8/30.5 ms   4,164   5.7/ 5.7 ms
  one byte per page, random 65,601  39.5/41.6 ms   4,161   7.4/ 7.6 ms
  memset()                  65,541  57.4/59.5 ms   4,102  39.5/39.8 ms
  8 threads, random order   65,541   6.1/ 6.2 ms   4,155   0.9/ 0.9 ms
  one store per 64K folio    4,101   1.9/ 2.0 ms   4,101   3.5/ 3.3 ms

Redis 7.0.15, 1.3 M keys of 512 bytes on 64K mTHP, BGSAVE and then 1,000,000 SETs: the faults of redis-server during the SETs go from 262,100 to 18,550 and its CPU time from 1.48-1.53 s to 1.31-1.41 s; the requests per second stay within the boot-to-boot spread. With THP off nothing changes.

arm64, under QEMU for the counters: the first pass after fork() unfolds every contpte block of the 64K folios today (512 contpte_convert() calls for 512 blocks, and nothing folds them again); with this patch none is unfolded.

[1] https://lore.kernel.org/r/[email protected]

---
 mm/memory.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 73 insertions(+), 2 deletions(-)
mm/memory.c +73 -2
index 8b0c2c735d3d..73e5691b3ed8 100644
@@ -4355,6 +4355,71 @@ static bool wp_can_reuse_anon_folio(struct folio *folio,
 	return true;
 }
 
+/*
+ * The pages of the folio around the one that faulted are handled in aligned
+ * blocks of this many PTEs: a 64K folio with 4K pages, and the size of a
+ * contpte block on arm64.
+ */
+#define WP_REUSE_NR_PTES	16
+
+/*
+ * wp_can_reuse_anon_folio() found a large folio to be exclusive to this MM.
+ * That holds for all of its pages and not only for the one that faulted: mark
+ * the ones in the same block exclusive as well and map them writable, like
+ * mprotect() would. Each of them would otherwise take a write fault of its own
+ * that repeats the check on the very same folio.
+ *
+ * The PTE that faulted is among them; wp_page_reuse() completes it.
+ */
+static void wp_reuse_large_anon_folio(struct vm_fault *vmf,
+		struct folio *folio)
+{
+	const fpb_t flags = FPB_RESPECT_WRITE | FPB_RESPECT_SOFT_DIRTY;
+	const unsigned long idx = folio_page_idx(folio, vmf->page);
+	struct vm_area_struct *vma = vmf->vma;
+	unsigned long addr = vmf->address;
+	unsigned long block = ALIGN_DOWN(addr, WP_REUSE_NR_PTES * PAGE_SIZE);
+	unsigned long nr_before, nr_after, end;
+	struct page *page;
+	unsigned int nr, i;
+	pte_t *ptep, pte;
+
+	/* Stay within the folio, the VMA and the block. */
+	nr_before = min3(idx, (addr - block) >> PAGE_SHIFT,
+			 (addr - vma->vm_start) >> PAGE_SHIFT);
+	nr_after = min3(folio_nr_pages(folio) - idx,
+			(block + WP_REUSE_NR_PTES * PAGE_SIZE - addr) >> PAGE_SHIFT,
+			(vma->vm_end - addr) >> PAGE_SHIFT);
+	end = addr + (nr_after << PAGE_SHIFT);
+	addr -= nr_before << PAGE_SHIFT;
+	ptep = vmf->pte - nr_before;
+	page = vmf->page - nr_before;
+
+	for (; addr != end; addr += nr * PAGE_SIZE, ptep += nr, page += nr) {
+		pte = ptep_get(ptep);
+		nr = 1;
+
+		/* Unmapped or replaced since, or writable already. */
+		if (!pte_present(pte) || pte_pfn(pte) != page_to_pfn(page) ||
+		    pte_write(pte))
+			continue;
+
+		nr = folio_pte_batch_flags(folio, NULL, ptep, &pte,
+					   (end - addr) >> PAGE_SHIFT, flags);
+		for (i = 0; i < nr; i++)
+			if (!PageAnonExclusive(page + i))
+				SetPageAnonExclusive(page + i);
+
+		/* The PTEs of a batch agree on everything this looks at. */
+		if (!can_change_pte_writable(vma, addr, pte))
+			continue;
+
+		pte = modify_prot_start_ptes(vma, addr, ptep, nr);
+		modify_prot_commit_ptes(vma, addr, ptep, pte,
+					pte_mkwrite(pte, vma), nr);
+	}
+}
+
 /*
  * This routine handles present pages, when
  * * users try to write to a shared page (FAULT_FLAG_WRITE)
@@ -4449,8 +4514,14 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
 	 */
 	if (folio && folio_test_anon(folio) &&
 	    (PageAnonExclusive(vmf->page) || wp_can_reuse_anon_folio(folio, vma))) {
-		if (!PageAnonExclusive(vmf->page))
-			SetPageAnonExclusive(vmf->page);
+		if (!PageAnonExclusive(vmf->page)) {
+			if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
+			    folio_test_large(folio) && likely(!unshare) &&
+			    likely(vma->vm_flags & VM_WRITE))
+				wp_reuse_large_anon_folio(vmf, folio);
+			else
+				SetPageAnonExclusive(vmf->page);
+		}
 		if (unlikely(unshare)) {
 			pte_unmap_unlock(vmf->pte, vmf->ptl);
 			return 0;
Signature · 1 line
2.43.0
#8 lore raw

Re: [PATCH v2 1/2] mm/memory: reuse 16 PTEs of an exclusive large folio on a write fault

Quoted 15 lines · Yuan-Hao Hsu

fork() maps the anonymous pages of the parent read-only and clears PageAnonExclusive on them. Once the child has exec'ed or exited, the parent's write fault takes the reuse path of do_wp_page():
wp_can_reuse_anon_folio() finds that all references to the folio come from this MM, the page is marked exclusive again and its PTE is made writable.

For a large folio that check is about the folio and holds for every page of it, but only the page that faulted is marked exclusive and made writable. Each other page takes a write fault of its own and takes the large mapcount lock to find out the same thing again: 16 faults for a 64K folio. The same THP mapped by a PMD is reused by one fault in do_huge_pmd_wp_page(), and do_swap_page() maps all PTEs of an exclusive large folio writable at once.

Quoted 7 lines

Barry proposed reusing the whole mTHP from one fault in 2024 [1]. The reservations then were the latency of the individual write fault and how far to go around the faulting PTE: a contpte-sized block was fine, anything bigger not yet convincing (David's replies, linked below). Commit 1da190f4d0a6 ("mm: Copy-on-Write (COW) reuse support for PTE-mapped THP") then added the per-folio check and left faulting around for later.

I'm fine with the original idea of limiting this to reasonable chunk sizes, reducing the work we do in a single page fault.

The patch needs work. I disagree with various decisions either you or the LLM came up with like

* Uglifying do_wp_page
* Not handling unshare
* Calling wp_reuse_large_anon_folio() to do some batching to then call wp_reuse_page in same page fault
* Batching multiple pieces in a loop
* Using can_change_pte_writable()

I tried to see how to implement it cleaner. I think we should definitely start with:

From 3d16faebcbbd1b970edb51ef2f710ca51e9709af Mon Sep 17 00:00:00 2001
From: "David Hildenbrand (Arm)" <[email protected]>
Date: Thu, 24 Sep 2026 19:43:15 +0200
Subject: [PATCH 1/2] mm/memory: factor out anon reuse logic into
wp_try_reuse_anon_page()

Let's move the core logic from do_wp_page() into wp_try_reuse_anon_page() to prepare for further changes.

No functional change intended.

---
 mm/memory.c | 45 +++++++++++++++++++++++++++++----------------
 1 file changed, 29 insertions(+), 16 deletions(-)
mm/memory.c +29 -16
index 338fce99e7119..67fcf67bc64fd 100644
@@ -4414,6 +4414,34 @@ static bool wp_can_reuse_anon_folio(struct folio *folio,
 	return true;
 }
 
+static bool wp_try_reuse_anon_page(struct vm_fault *vmf, struct folio *folio)
+	__cond_releases(true, vmf->ptl)
+{
+	const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE;
+
+	VM_WARN_ON_ONCE(!folio_test_anon(folio));
+
+	/*
+	 * Private mapping: create an exclusive anonymous page copy if reuse
+	 * is impossible. We might miss VM_WRITE for FOLL_FORCE handling.
+	 *
+	 * If we encounter a page that is marked exclusive, we must reuse
+	 * the page without further checks.
+	 */
+	if (!PageAnonExclusive(vmf->page)) {
+		if (!wp_can_reuse_anon_folio(folio, vmf->vma))
+			return false;
+		SetPageAnonExclusive(vmf->page);
+	}
+
+	if (unlikely(unshare)) {
+		pte_unmap_unlock(vmf->pte, vmf->ptl);
+		return true;
+	}
+	wp_page_reuse(vmf, folio);
+	return true;
+}
+
 /*
  * This routine handles present pages, when
  * * users try to write to a shared page (FAULT_FLAG_WRITE)
@@ -4499,24 +4527,9 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
 		return wp_page_shared(vmf, folio);
 	}
 
-	/*
-	 * Private mapping: create an exclusive anonymous page copy if reuse
-	 * is impossible. We might miss VM_WRITE for FOLL_FORCE handling.
-	 *
-	 * If we encounter a page that is marked exclusive, we must reuse
-	 * the page without further checks.
-	 */
 	if (folio && folio_test_anon(folio) &&
-	    (PageAnonExclusive(vmf->page) || wp_can_reuse_anon_folio(folio, vma))) {
-		if (!PageAnonExclusive(vmf->page))
-			SetPageAnonExclusive(vmf->page);
-		if (unlikely(unshare)) {
-			pte_unmap_unlock(vmf->pte, vmf->ptl);
-			return 0;
-		}
-		wp_page_reuse(vmf, folio);
+	    wp_try_reuse_anon_page(vmf, folio))
 		return 0;
-	}
 	/*
 	 * Ok, we need to copy. Oh, well..
 	 */
Signature · 32 lines
2.43.0


And the maybe go into this direction, where we really only try to
batch exactly once, and include in that patch out PTE of interest. IOW, optimize
for the common case and also take care of unsharing.


Some things I am not sure about
* Hardcoding WP_REUSE_MAX_NR_PTES, likely should be determine differently.
* Marking all 16 PTEs young+dirty. It's somewhat the same thing as we do in
  map_anon_folio_pte_pf(). On arm64 it's already fuzzy with cont-pte. With
  transparent coalescing we'd actually allow it directly. So it does feel like the right thing.

I also wonder whether some part of the function could be factored out as helpers for
other code to use in the future. I also suspect that there are more cleanups to be had.

Long story short, needs more work, but I am out of time.

Entirely untested:


From 1db2c61661cc854eabb8619a2717d1ddf43a95f5 Mon Sep 17 00:00:00 2001
From: "David Hildenbrand (Arm)" <[email protected]>
Date: Thu, 24 Sep 2026 20:03:01 +0200
Subject: [PATCH 2/2] mm/memory: reuse 16 PTEs of an exclusive large folio on a
 write fault

Signed-off-by: David Hildenbrand (Arm) <[email protected]>
---
 mm/memory.c | 120 +++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 115 insertions(+), 5 deletions(-)
mm/memory.c +115 -5

139 lines not on this page · raw message

Signature · 8 lines
2.43.0



-- 
Cheers,

David
#9 lore raw

Re: [PATCH v2 1/2] mm/memory: reuse 16 PTEs of an exclusive large folio on a write fault

Quoted 2 lines · David Hildenbrand (Arm)

The patch needs work. I disagree with various decisions either you or the LLM came up with like

[...]

Quoted 2 lines

I tried to see how to implement it cleaner. I think we should definitely start with:

[...]

Quoted 1 line

Entirely untested:

Thanks for writing it out. I took your two patches as they are, on 238650ef6c7c, and ran them through what I had for v1/v2. Nothing broke:

- builds on x86-64, arm64 4K/16K/64K, i386 (+PAE), x86 without THP, arm, arm nommu, riscv64, powerpc64le, s390x; sparse and W=1 clean
- DEBUG_VM + DEBUG_VM_PGTABLE + PROVE_LOCKING + PAGE_TABLE_CHECK_ENFORCED, with the mm selftests, my 12 COW scenarios (vmsplice, PROT_READ VMA inside the folio, soft-dirty/uffd-wp counts, mremap, holes, pageout, FOLL_FORCE) and a fork/pageout/mprotect/vmsplice stress: no warnings
- arm64 under QEMU: 64K folios go from 8,200 to 520 faults per pass, contpte_convert() from 512 to 0, and contpte_ptep_set_access_flags() from 512 to 0
- x86-64 (i7-12700KF): same fault counts and pass times as my 16-PTE version; the fault that handles the block takes 720-730 ns instead of 740-750
- Redis on 64K mTHP (BGSAVE, then 1M SETs): faults 262,100 -> 18,900, CPU 1.53 -> 1.38 s, 640k -> 715k req/s; THP-off control unchanged

Quoted 1 line

* Hardcoding WP_REUSE_MAX_NR_PTES, likely should be determine differently.

Same patch, only the constant changed; 256 MiB of PTE-mapped 2M folios after fork() + child exit:

WP_REUSE_MAX_NR_PTES 16 32 64 128 512
one byte per page (seq)

    faults                   4,161  2,113  1,089    577    193
    ms                         5.6    4.7    4.2    4.1    3.9
  one byte per page, random    7.0    6.0    5.4    5.2    5.0
  one store per 64K            3.2    2.3    1.9    1.7    1.6
  one store per 2M: the fault
    that walks the block (us)  0.84   1.13   1.71   3.01  10.32

So roughly 0.35 us per block plus 20 ns per PTE, and the gain is flat from 64 on. For where the number could come from: arm64's CONT_PTES is 16/128/32 for 4K/16K/64K pages, and fault_around_pages defaults to 64K worth, which is 16 on 4K pages but 4 and 1 on the larger ones. An arch-overridable default of 16, with arm64 using CONT_PTES, would fit the numbers. Your call.

Quoted 3 lines

* Marking all 16 PTEs young+dirty. It's somewhat the same thing as we do in map_anon_folio_pte_pf(). On arm64 it's already fuzzy with cont-pte. With transparent coalescing we'd actually allow it directly. So it does feel like the right thing.

I think it's right, and for a simpler reason: nothing looks at young or dirty per PTE within a folio. try_to_unmap_one() marks the folio dirty from the merged pteval of the batch, ttu_anon_lazyfree_folio() checks folio_test_dirty(), folio_referenced_one() adds the young bits up. The faulting PTE is young+dirty anyway, so the other 15 can't change any decision. I tried the most sensitive case I could think of, a MADV_FREE'd folio with one store per folio after fork(): two MADV_PAGEOUT passes keep and swap the whole 64K/2M folio on today's kernel exactly as with your patch.

Quoted 2 lines

I also wonder whether some part of the function could be factored out as helpers for other code to use in the future. I also suspect that there are more cleanups to be had.

One candidate: numa_rebuild_large_mapping() does the same folio/VMA/page-table bounded walk by hand, so a "batch around this PTE within the folio" helper could serve both.

Thanks,
Yuan-Hao

#10 lore raw

PATCH v2 2/2 mm/memory: reuse the whole exclusive large folio on a write fault

In reply to Yuan-Hao Hsu

With blocks of 16 PTEs a PTE-mapped 2M THP still takes 32 reuse faults and a 1M folio 16. Lift the bound to the page table: the walk then covers the PTEs that map the folio in this page table, at most PTRS_PER_PTE of them, and one fault does the work for the folio.

Its cost, measured as the time of the store that takes it, against 420 ns for a reuse fault today (x86-64, i7-12700KF, medians, ns):

                          reuse fault   fault that     COW fault that
                          (patched)     allocated it   copies 4K
      1M mTHP            5,400- 5,700   63,000-68,000          1,500
      2M THP, PTE-mapped  10,000-10,500         126,000          1,500

That is 14-20 ns per PTE, about 2 ns of it the scan. Builds that differ only by NOPs in front of the function take either 10,100 or 7,500 ns for the 2M folio, with a period of 32 bytes: the loop of modify_prot_commit_ptes() changes speed with its address.

What it buys, 256 MiB after fork() and the child's exit, medians of 15 runs, two boots of each kernel:

16 PTEs whole folio
one byte per page, seq

      1M mTHP                     5.7 /  5.8 ms    4.1 /  4.0 ms
      2M THP, PTE-mapped          5.7 /  5.8 ms    3.9 /  3.9 ms
  one byte per page, random order
      2M THP, PTE-mapped          7.4 /  7.5 ms    4.9 /  5.1 ms
  memset()
      2M THP, PTE-mapped         40.2 / 39.4 ms   38.1 / 37.2 ms
  8 threads, random order
      2M THP, PTE-mapped          0.9 /  1.0 ms    0.7 /  0.7 ms
  one store per 64K, 2M folios    3.3 /  3.3 ms    1.5 /  1.6 ms
  one store per 2M, 2M folios     0.1 /  0.1 ms    1.4 /  1.4 ms

The last row is the pattern where the whole-folio fault is pure cost:
511 pages made writable that nobody writes, 1.3 ms more per 256 MiB, paid once per fork(). Anything that stores to more than one page per 64K comes out ahead.

---
 mm/memory.c | 21 +++++++--------------
 1 file changed, 7 insertions(+), 14 deletions(-)
mm/memory.c +7 -14
index 73e5691b3ed8..85c883d1e558 100644
@@ -4355,19 +4355,12 @@ static bool wp_can_reuse_anon_folio(struct folio *folio,
 	return true;
 }
 
-/*
- * The pages of the folio around the one that faulted are handled in aligned
- * blocks of this many PTEs: a 64K folio with 4K pages, and the size of a
- * contpte block on arm64.
- */
-#define WP_REUSE_NR_PTES	16
-
 /*
  * wp_can_reuse_anon_folio() found a large folio to be exclusive to this MM.
  * That holds for all of its pages and not only for the one that faulted: mark
- * the ones in the same block exclusive as well and map them writable, like
- * mprotect() would. Each of them would otherwise take a write fault of its own
- * that repeats the check on the very same folio.
+ * the ones that this page table maps exclusive as well and map them writable,
+ * like mprotect() would. Each of them would otherwise take a write fault of
+ * its own that repeats the check on the very same folio.
  *
  * The PTE that faulted is among them; wp_page_reuse() completes it.
  */
@@ -4378,17 +4371,17 @@ static void wp_reuse_large_anon_folio(struct vm_fault *vmf,
 	const unsigned long idx = folio_page_idx(folio, vmf->page);
 	struct vm_area_struct *vma = vmf->vma;
 	unsigned long addr = vmf->address;
-	unsigned long block = ALIGN_DOWN(addr, WP_REUSE_NR_PTES * PAGE_SIZE);
+	unsigned long pt_start = ALIGN_DOWN(addr, PMD_SIZE);
 	unsigned long nr_before, nr_after, end;
 	struct page *page;
 	unsigned int nr, i;
 	pte_t *ptep, pte;
 
-	/* Stay within the folio, the VMA and the block. */
-	nr_before = min3(idx, (addr - block) >> PAGE_SHIFT,
+	/* Stay within the folio, the VMA and the page table. */
+	nr_before = min3(idx, (addr - pt_start) >> PAGE_SHIFT,
 			 (addr - vma->vm_start) >> PAGE_SHIFT);
 	nr_after = min3(folio_nr_pages(folio) - idx,
-			(block + WP_REUSE_NR_PTES * PAGE_SIZE - addr) >> PAGE_SHIFT,
+			(pt_start + PMD_SIZE - addr) >> PAGE_SHIFT,
 			(vma->vm_end - addr) >> PAGE_SHIFT);
 	end = addr + (nr_after << PAGE_SHIFT);
 	addr -= nr_before << PAGE_SHIFT;
Signature · 1 line
2.43.0
#11 lore raw
Quoted 23 lines · Yuan-Hao Hsu

With blocks of 16 PTEs a PTE-mapped 2M THP still takes 32 reuse faults and a 1M folio 16. Lift the bound to the page table: the walk then covers the PTEs that map the folio in this page table, at most PTRS_PER_PTE of them, and one fault does the work for the folio.

Its cost, measured as the time of the store that takes it, against 420 ns for a reuse fault today (x86-64, i7-12700KF, medians, ns):

                          reuse fault   fault that     COW fault that
                          (patched)     allocated it   copies 4K
      1M mTHP            5,400- 5,700   63,000-68,000          1,500
      2M THP, PTE-mapped  10,000-10,500         126,000          1,500

That is 14-20 ns per PTE, about 2 ns of it the scan. Builds that differ only by NOPs in front of the function take either 10,100 or 7,500 ns for the 2M folio, with a period of 32 bytes: the loop of modify_prot_commit_ptes() changes speed with its address.

What it buys, 256 MiB after fork() and the child's exit, medians of 15 runs, two boots of each kernel:

16 PTEs whole folio
one byte per page, seq

Quoted 11 lines
      1M mTHP                     5.7 /  5.8 ms    4.1 /  4.0 ms
      2M THP, PTE-mapped          5.7 /  5.8 ms    3.9 /  3.9 ms
  one byte per page, random order
      2M THP, PTE-mapped          7.4 /  7.5 ms    4.9 /  5.1 ms
  memset()
      2M THP, PTE-mapped         40.2 / 39.4 ms   38.1 / 37.2 ms
  8 threads, random order
      2M THP, PTE-mapped          0.9 /  1.0 ms    0.7 /  0.7 ms
  one store per 64K, 2M folios    3.3 /  3.3 ms    1.5 /  1.6 ms
  one store per 2M, 2M folios     0.1 /  0.1 ms    1.4 /  1.4 ms

Are you measuring the worst-case latency of a single write fault or only the average over all writes?

Signature · 3 lines
Cheers,

David
#12 lore raw
Quoted 2 lines · David Hildenbrand (Arm)

Are you measuring the worst-case latency of a single write fault or only the average over all writes?

The single fault. Each store is timed by itself (clock_gettime() before and after), and the latency numbers come from a pattern where every store takes one of these faults (one store per folio), so there is nothing else in the average. The tables show the median of those faults; I also have p99 and max (i7-12700KF, in a VM, us):

v7.3-rc3+ patched
p50 p99 max p50 p99 max

  64K, 1 fault      0.42  0.57  25-26      0.71-0.74  0.96-1.10  28-60
  2M, 1 fault       0.50  0.67  0.7-1.2    10.0-10.2  12.2-15.8  13-22

max is the slowest store of a run, median over 15 runs. The 25 us on the base kernel are interrupts; the VM has those on both sides. The 2M row is the worst case by construction, every one of those faults walks all 512 PTEs.

#13 lore raw
Quoted 12 lines · Yuan-Hao Hsu
Quoted 2 lines · David Hildenbrand (Arm)

Are you measuring the worst-case latency of a single write fault or only the average over all writes?

The single fault. Each store is timed by itself (clock_gettime() before and after), and the latency numbers come from a pattern where every store takes one of these faults (one store per folio), so there is nothing else in the average. The tables show the median of those faults; I also have p99 and max (i7-12700KF, in a VM, us):

v7.3-rc3+ patched
p50 p99 max p50 p99 max

Quoted 7 lines
  64K, 1 fault      0.42  0.57  25-26      0.71-0.74  0.96-1.10  28-60
  2M, 1 fault       0.50  0.67  0.7-1.2    10.0-10.2  12.2-15.8  13-22

max is the slowest store of a run, median over 15 runs. The 25 us on the base kernel are interrupts; the VM has those on both sides. The 2M row is the worst case by construction, every one of those faults walks all 512 PTEs.

Thanks. See my other mail, we should focus on keeping the fault path as fast as possible and minimizing the work in there.

Signature · 3 lines
Cheers,

David
#14 lore raw
In reply to Yuan-Hao Hsu
Quoted 5 lines · Yuan-Hao Hsu

After fork() and the child's exit, the parent's write faults reuse a large anonymous folio one PTE at a time, although wp_can_reuse_anon_folio() has already found the whole folio exclusive.

v2, after the comments on v1 [1]:

Please don't send out a new version

* Unless a maintainer asked you to
* At least one week passed

Signature · 3 lines
Cheers,

David
#15 lore raw
Quoted 4 lines · David Hildenbrand (Arm)

Please don't send out a new version

* Unless a maintainer asked you to
* At least one week passed

Understood, sorry about that. I'll sit tight and wait for the review of this version.

#16 lore raw
Quoted 8 lines · Yuan-Hao Hsu
Quoted 4 lines · David Hildenbrand (Arm)

Please don't send out a new version

* Unless a maintainer asked you to
* At least one week passed

Understood, sorry about that. I'll sit tight and wait for the review of this version.

Also, and I am struggling to believe I'm having to say this, don't completely ignore feedback you've already received which told you about multiple process fails?

https://lore.kernel.org/all/aq1CjdaPUg_Fos0t@gremlin/

--
Cheers, Lorenzo

#17 lore raw
Quoted 5 lines · Lorenzo Stoakes (ARM)

Also, and I am struggling to believe I'm having to say this, don't completely ignore feedback you've already received which told you about multiple process fails?

https://lore.kernel.org/all/aq1CjdaPUg_Fos0t@gremlin/

I'm sorry. I'll go back and read the process documentation properly from the start before I send anything else. Really sorry for the trouble.