Create 'old' ptes for faultaround mappings on arm64 with hardware access flag

Hi all,

This is version two of the series I originally posted here:

https://lore.kernel.org/r/[email protected]

The patches allow architectures to opt-in at runtime for faultaround mappings to be created as 'old' instead of 'young'. Although there have been previous attempts at this, they failed either because the decision was deferred to userspace [1] or because it was done unconditionally and shown to regress benchmarks for particular architectures [2].

The big difference in this version is that I have reworked it based on Kirill's patch which he posted as a follow-up to the original. However, I can't tell where we've landed on that -- Linus seemed to like it, but Hugh was less enthusiastic. I think that my subsequent patches are an awful lot cleaner after the rework, but I don't think I necessarily depend on everything in there so if that patch is likely to be a sticking point then I can try to extract the parts I need.

Feedback welcome.

Cheers,

Will

[1] https://www.spinics.net/lists/linux-mm/msg143831.html
[2] 315d09bf30c2 ("Revert "mm: make faultaround produce old ptes"")

--->8

Kirill A. Shutemov (1):
mm: Cleanup faultaround and finish_fault() codepaths

Will Deacon (2):
mm: Allow architectures to request 'old' entries when prefaulting
arm64: mm: Implement arch_wants_old_prefaulted_pte()

 arch/arm64/include/asm/pgtable.h |  12 +-
 fs/xfs/xfs_file.c                |   6 +-
 include/linux/mm.h               |  17 ++-
 include/linux/pgtable.h          |  11 ++
 mm/filemap.c                     | 181 +++++++++++++++++++------
 mm/memory.c                      | 219 +++++++++++--------------------
 6 files changed, 251 insertions(+), 195 deletions(-)

2.29.2.729.g45daf8777d-goog

From: "Kirill A. Shutemov" <[email protected]>

alloc_set_pte() has two users with different requirements: in the faultaround code, it called from an atomic context and PTE page table has to be preallocated. finish_fault() can sleep and allocate page table as needed.

PTL locking rules are also strange, hard to follow and overkill for finish_fault().

Let's untangle the mess. alloc_set_pte() has gone now. All locking is explicit.

The price is some code duplication to handle huge pages in faultaround path, but it should be fine, having overall improvement in readability.

---
 fs/xfs/xfs_file.c       |   6 +-
 include/linux/mm.h      |  12 ++-
 include/linux/pgtable.h |  11 +++
 mm/filemap.c            | 177 ++++++++++++++++++++++++++---------
 mm/memory.c             | 199 ++++++++++++----------------------------
 5 files changed, 213 insertions(+), 192 deletions(-)

index 5b0f93f73837..111fe73bb8a7 100644
@@ -1319,17 +1319,19 @@ xfs_filemap_pfn_mkwrite(
 	return __xfs_filemap_fault(vmf, PE_SIZE_PTE, true);
 }
 
-static void
+static vm_fault_t
 xfs_filemap_map_pages(
 	struct vm_fault		*vmf,
 	pgoff_t			start_pgoff,
 	pgoff_t			end_pgoff)
 {
 	struct inode		*inode = file_inode(vmf->vma->vm_file);
+	vm_fault_t ret;
 
 	xfs_ilock(XFS_I(inode), XFS_MMAPLOCK_SHARED);
-	filemap_map_pages(vmf, start_pgoff, end_pgoff);
+	ret = filemap_map_pages(vmf, start_pgoff, end_pgoff);
 	xfs_iunlock(XFS_I(inode), XFS_MMAPLOCK_SHARED);
+	return ret;
 }
 
 static const struct vm_operations_struct xfs_file_vm_ops = {

index ecdf8a8cd6ae..801dd99f733c 100644
@@ -542,8 +542,8 @@ struct vm_fault {
 					 * is not NULL, otherwise pmd.
 					 */
 	pgtable_t prealloc_pte;		/* Pre-allocated pte page table.
-					 * vm_ops->map_pages() calls
-					 * alloc_set_pte() from atomic context.
+					 * vm_ops->map_pages() sets up a page
+					 * table from from atomic context.
 					 * do_fault_around() pre-allocates
 					 * page table to avoid allocation from
 					 * atomic context.

@@ -578,7 +578,7 @@ struct vm_operations_struct {
 	vm_fault_t (*fault)(struct vm_fault *vmf);
 	vm_fault_t (*huge_fault)(struct vm_fault *vmf,
 			enum page_entry_size pe_size);
-	void (*map_pages)(struct vm_fault *vmf,
+	vm_fault_t (*map_pages)(struct vm_fault *vmf,
 			pgoff_t start_pgoff, pgoff_t end_pgoff);
 	unsigned long (*pagesize)(struct vm_area_struct * area);
 

@@ -988,7 +988,9 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
 	return pte;
 }
 
-vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct page *page);
+vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page);
+void do_set_pte(struct vm_fault *vmf, struct page *page);
+
 vm_fault_t finish_fault(struct vm_fault *vmf);
 vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf);
 #endif

@@ -2622,7 +2624,7 @@ extern void truncate_inode_pages_final(struct address_space *);
 
 /* generic vm_area_ops exported for stackable file systems */
 extern vm_fault_t filemap_fault(struct vm_fault *vmf);
-extern void filemap_map_pages(struct vm_fault *vmf,
+extern vm_fault_t filemap_map_pages(struct vm_fault *vmf,
 		pgoff_t start_pgoff, pgoff_t end_pgoff);
 extern vm_fault_t filemap_page_mkwrite(struct vm_fault *vmf);
 

index 8fcdfa52eb4b..36eb748f3c97 100644
@@ -1314,6 +1314,17 @@ static inline int pmd_trans_unstable(pmd_t *pmd)
 #endif
 }
 
+/*
+ * the ordering of these checks is important for pmds with _page_devmap set.
+ * if we check pmd_trans_unstable() first we will trip the bad_pmd() check
+ * inside of pmd_none_or_trans_huge_or_clear_bad(). this will end up correctly
+ * returning 1 but not before it spams dmesg with the pmd_clear_bad() output.
+ */
+static inline int pmd_devmap_trans_unstable(pmd_t *pmd)
+{
+	return pmd_devmap(*pmd) || pmd_trans_unstable(pmd);
+}
+
 #ifndef CONFIG_NUMA_BALANCING
 /*
  * Technically a PTE can be PROTNONE even when not doing NUMA balancing but

2.29.2.729.g45daf8777d-goog

Commit 5c0a85fad949 ("mm: make faultaround produce old ptes") changed the "faultaround" behaviour to initialise prefaulted PTEs as 'old', since this avoids vmscan wrongly assuming that they are hot, despite having never been explicitly accessed by userspace. The change has been shown to benefit numerous arm64 micro-architectures (with hardware access flag) running Android, where both application launch latency and direct reclaim time are significantly reduced.

Unfortunately, commit 315d09bf30c2 ("Revert "mm: make faultaround produce old ptes"") reverted the change to it being identified as the cause of a ~6% regression in unixbench on x86. Experiments on a variety of recent arm64 micro-architectures indicate that unixbench is not affected by the original commit, yielding a 0-1% performance improvement.

Since one size does not fit all for the initial state of prefaulted PTEs, introduce arch_wants_old_prefaulted_pte(), which allows an architecture to opt-in to 'old' prefaulted PTEs at runtime based on whatever criteria it may have.

---
 include/linux/mm.h |  5 ++++-
 mm/filemap.c       | 12 ++++++++----
 mm/memory.c        | 20 +++++++++++++++++++-
 3 files changed, 31 insertions(+), 6 deletions(-)

index 801dd99f733c..873e410d6238 100644
@@ -434,6 +434,7 @@ extern pgprot_t protection_map[16];
  * @FAULT_FLAG_REMOTE: The fault is not for current task/mm.
  * @FAULT_FLAG_INSTRUCTION: The fault was during an instruction fetch.
  * @FAULT_FLAG_INTERRUPTIBLE: The fault can be interrupted by non-fatal signals.
+ * @FAULT_FLAG_PREFAULT: Fault was a prefault.
  *
  * About @FAULT_FLAG_ALLOW_RETRY and @FAULT_FLAG_TRIED: we can specify
  * whether we would allow page faults to retry by specifying these two

@@ -464,6 +465,7 @@ extern pgprot_t protection_map[16];
 #define FAULT_FLAG_REMOTE			0x80
 #define FAULT_FLAG_INSTRUCTION  		0x100
 #define FAULT_FLAG_INTERRUPTIBLE		0x200
+#define FAULT_FLAG_PREFAULT			0x400
 
 /*
  * The default fault flags that should be used by most of the

@@ -501,7 +503,8 @@ static inline bool fault_flag_allow_retry_first(unsigned int flags)
 	{ FAULT_FLAG_USER,		"USER" }, \
 	{ FAULT_FLAG_REMOTE,		"REMOTE" }, \
 	{ FAULT_FLAG_INSTRUCTION,	"INSTRUCTION" }, \
-	{ FAULT_FLAG_INTERRUPTIBLE,	"INTERRUPTIBLE" }
+	{ FAULT_FLAG_INTERRUPTIBLE,	"INTERRUPTIBLE" }, \
+	{ FAULT_FLAG_PREFAULT,		"PREFAULT" }
 
 /*
  * vm_fault is filled by the pagefault handler and passed to the vma's

index c1f2dc89b8a7..0fb9d1714797 100644
@@ -3051,14 +3051,18 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,
 		if (!pte_none(*vmf->pte))
 			goto unlock;
 
+		/* We're about to handle the fault */
+		if (vmf->address == address) {
+			vmf->flags &= ~FAULT_FLAG_PREFAULT;
+			ret = VM_FAULT_NOPAGE;
+		} else {
+			vmf->flags |= FAULT_FLAG_PREFAULT;
+		}
+
 		do_set_pte(vmf, page);
 		/* no need to invalidate: a not-present page won't be cached */
 		update_mmu_cache(vma, vmf->address, vmf->pte);
 		unlock_page(head);
-
-		/* The fault is handled */
-		if (vmf->address == address)
-			ret = VM_FAULT_NOPAGE;
 		continue;
 unlock:
 		unlock_page(head);

index 3e2fc2950ad7..f0e7c589ca9d 100644
@@ -134,6 +134,18 @@ static inline bool arch_faults_on_old_pte(void)
 }
 #endif
 
+#ifndef arch_wants_old_prefaulted_pte
+static inline bool arch_wants_old_prefaulted_pte(void)
+{
+	/*
+	 * Transitioning a PTE from 'old' to 'young' can be expensive on
+	 * some architectures, even if it's performed in hardware. By
+	 * default, "false" means prefaulted entries will be 'young'.
+	 */
+	return false;
+}
+#endif
+
 static int __init disable_randmaps(char *s)
 {
 	randomize_va_space = 0;

@@ -3725,11 +3737,17 @@ void do_set_pte(struct vm_fault *vmf, struct page *page)
 {
 	struct vm_area_struct *vma = vmf->vma;
 	bool write = vmf->flags & FAULT_FLAG_WRITE;
+	bool prefault = vmf->flags & FAULT_FLAG_PREFAULT;
 	pte_t entry;
 
 	flush_icache_page(vma, page);
 	entry = mk_pte(page, vma->vm_page_prot);
-	entry = pte_sw_mkyoung(entry);
+
+	if (prefault && arch_wants_old_prefaulted_pte())
+		entry = pte_mkold(entry);
+	else
+		entry = pte_sw_mkyoung(entry);
+
 	if (write)
 		entry = maybe_mkwrite(pte_mkdirty(entry), vma);
 	/* copy-on-write page */

2.29.2.729.g45daf8777d-goog

I can certainly add that, although it doesn't look like we do that for vmf->pte, so it's hard to tell what the rules are here. It certainly feels odd to restore some fields but not others, as it looks like vmf->address will be out-of-whack with vmf->pte when filemap_map_pages() returns. Am I missing something?

Will

Unlike vmf->flags or vmf->address, vmf->pte is not going to be reused. finish_fault() will overwrite it.

Yeah, it's messy.

 Kirill A. Shutemov

On CPUs with hardware AF/DBM, initialising prefaulted PTEs as 'old' improves vmscan behaviour and does not appear to introduce any overhead.

Implement arch_wants_old_prefaulted_pte() to return 'true' if we detect hardware access flag support at runtime. This can be extended in future based on MIDR matching if necessary.

---
 arch/arm64/include/asm/pgtable.h | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

index 501562793ce2..e17b96d0e4b5 100644
@@ -980,7 +980,17 @@ static inline bool arch_faults_on_old_pte(void)
 
 	return !cpu_has_hw_af();
 }
-#define arch_faults_on_old_pte arch_faults_on_old_pte
+#define arch_faults_on_old_pte		arch_faults_on_old_pte
+
+/*
+ * Experimentally, it's cheap to set the access flag in hardware and we
+ * benefit from prefaulting mappings as 'old' to start with.
+ */
+static inline bool arch_wants_old_prefaulted_pte(void)
+{
+	return !arch_faults_on_old_pte();
+}
+#define arch_wants_old_prefaulted_pte	arch_wants_old_prefaulted_pte
 
 #endif /* !__ASSEMBLY__ */
 

2.29.2.729.g45daf8777d-goog

Yeah, I like it, but I have to admit that it had a disturbingly high number of small details wrong for several versions. I hope you picked up the final version of the code.

At the same time, I do think that the "disturbingly high number of issues" was primarily exactly _because_ the old code was so incomprehensible, and I think the end result is much cleaner, so I still like it.

Yeah, I think that's a side effect of "now the code really makes a lot more sense". Your subsequent patches 2-3 certainly are much simpler now, although I'd be inclined to add an argument to "do_set_pte()" that has the "write" and "pretault" bits in it, instead of having to modify the 'vmf' structure.

I still dislike how we basically randomly modify the information in that 'vmf' thing.

That said, now it's just a small detail - not really objectionable, just a "this could be cleaner, I think".

I think it was Kirill who pointed out that we sadly cannot make 'vmf' read-only anyway, because it does also contain those pre-allocation details etc (vmf->pte etc) that are very much about what the current "state" of the fault is. So while I would hope it could be more read-only than it is, my wish that it could _actually_ be 'const' is clearly just an irrelevant dream.

Linus

On that note - they could be simpler still if this was just done entirely unconditionally..

I'm taking your word for "it makes sense", but when you say

On CPUs with hardware AF/DBM, initialising prefaulted PTEs as 'old' improves vmscan behaviour and does not appear to introduce any overhead.

in the description for patch 3, it makes me wonder how noticeable the overhead is on the hardware that _does_ take a fault on old pte's..

IOW, it would be lovely to see numbers if you have any like that..

Both ways, actually. Because I also wonder how noticeable the vmscan improvement is. You say there's no measurable overhead for platforms with hardware dirty/accessed bits, but maybe there's not a lot of measurable improvements from a more exact accessed bit either?

Linus

[Vinayak -- please chime in if I miss anything here, as you've posted these
numbers before]

The initial posting in 2016 had some numbers based on a 3.18 kernel, which didn't have support for hardware AF/DBM:

https://lore.kernel.org/lkml/[email protected]

(note that "Kirill's-fix" in the last column was a quick hack and didn't
make the faulting pte young)

So yes, for the cases we care about in Android (where the vmscan behaviour seems to be the important thing), then this patch makes sense for non-hardware AF/DBM CPUs too. In either case, we see ~80% reduction in direct reclaim time according to mmtests [1] and double-digit percentage reductions in app launch latency (some of this is mentioned in the link above). The actual fault cost isn't especially relevant.

*However...*

For machines with lots of memory, the increased fault cost when hardware AF/DBM is not available may well be measurable, and I suspect it would hurt unixbench (which was the reason behind reverting this on x86 [2], although I must admit that the diagnosis wasn't particularly satisfactory [3]). We could run those numbers on arm64 but, due to the wide diversity of micro-architectures we have to deal with, I would like to keep our options open to detecting this dynamically anyway, just in case somebody builds a CPU which struggles in this area.

Cheers,

Will

[1] https://github.com/gormanm/mmtests
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/

I picked the version from here:

https://lore.kernel.org/r/20201229132819.najtavneutnf7ajp@box

and actually, I just noticed that willy spotted a typo in a comment, so I'll fix that locally as well as adding the above to a 'Link:' tag for reference.

I played with a few different ways of doing this, but I can't say I prefer them over what I ended up posting. Having a bunch of 'bool' arguments makes the callers hard to read and brings into question what exactly vmf->flags is for. I also tried adding a separate 'address' parameter so that vmf->address is always the real faulting address, but 'address' is the thing to use for the pte (i.e. prefault is when 'address != vmf->address'). That wasn't too bad, but it made the normal finish_fault() case look weird.

So I think I'll leave it as-is and see if anybody wants to change it later on.

Will

Just to report that I gave this v2 set a spin on a few (x86_64 and i386) machines, and found nothing objectionable this time around.

And the things that I'm unenthusiastic about are exactly those details that you and Kirill and Linus find unsatisfactory, but awkward to
eliminate: expect no new insights from me!

Hugh

Thanks, Hugh.

Well, I'll keep you on CC for v3 -- just in case!

Will

I wounder if it would be acceptable to pass down to faultaround a copy of vmf, so it mess with it without risking to corrupt the original one?

We would need to transfer back prealloc_pte, but the rest can be safely discarded, I believe.

But it's somewhat wasteful of stack. sizeof(vmf) == 96 on x86-64.

 Kirill A. Shutemov

I'd almost prefer to split vmf into two parts: the 'this is the fault info' part and the 'this is the fault handling state' part.

So the first one would be filled in by the actual page faulter (or GUP) - and then be 'const' during the lookup, while the second one would be set up by handle_mm_fault() and would contain that "this is the current state of my fault state machine" and contain things like that ->pte thing.

And then if somebody actually needs to pass in "modified fault state" (ie that whole "I'm doing fault-around, so I'll use multiple addresses") they'd never modify the address in the fault info, they'd just pass the address as an explicit argument (like most cases already do - the "change addr or flags in vmf" is actually already _fairly_ rare).

Linus

Alright then, I'll take another crack at this for v3 and see how far I get.

Will

But please do keep it as a separate patch, and if it turns out to be ugly for whatever reason, treat it as just a "maybe that would help" suggestion rather than anything stronger..

Linus