Skip to contentSkip to conversation

KVM: x86/mmu: don't zap secondary MMU on permission upgrade

RFC PATCH Superseded Carlos López 5 participants 19 messages Oct 29 – Nov 4

Subsystems (8)
Memory management x86 KVM MM core +4 more subsystems

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover KVM: x86/mmu: don't zap secondary MMU on permission upgrade

Prevent x86 KVM from over-zapping its secondary MMU on PTE permission upgrades, which leave the PFNs unchanged and do not revoke access. Permissions for SPTEs that KVM holds are updated on the next fault, so dropping them early gains nothing.

Do this by adding a new MMU notifier flag to signal that a MMU_NOTIFY_PROTECTION_VMA strictly upgrades (loosens) permissions on a particular range.

See below for benchmark results.

Background
==========

change_protection() pairs every protection change with an MMU_NOTIFY_PROTECTION_VMA invalidation, and KVM zaps its SPTEs in response, as it cannot tell whether permissions widened or narrowed. This is costly e.g. during a QEMU background snapshot, which relies on uffd-wp to write-protect guest RAM and then restore permissions as each page is saved. Without this series, the permission restore causes the guest to re-fault read-only memory needlessly.

Note, the reverse situation is already handled for secondary MMUs that reuse the CPU page tables (e.g. ARM SMMU, IOMMU/ATS). They cannot refault, so they are notified of a permission upgrade on the CPU TLB-flush path [1].

KVM, however, owns separate page tables and faults permissions on demand, so it needs no such notification (as Sean noted when that signalling was first proposed on the fault path [2]). This series applies the same principle to the change_protection() path, signalling the permission upgrade so KVM can avoid the unnecessary zap.

RFC Considerations
==================

1. Having the zap skip in KVM's arch-independent code does not seem obviously correct, as s390 caches additional translation state that may not be rebuilt by refaults. Maybe an expert can correct me. The skip could be extended to arm64 and RISC-V, but I have not benchmarked the benefits and I'd like to get feedback first. Hence, x86-only for now.

2. change_protection() could simply not emit the invalidation at all for a pure permission upgrade (shared-page-table MMUs invalidate on the CPU TLB-flush path, as noted above, so they do not need it). But again, the s390 case could be broken, and even then, that would seem too much intrusion for KVM's benefit.

3. The flag reaches KVM as a bool on struct kvm_gfn_range. Alternatively, KVM could forward the notifier event, as Claudio Imbrenda suggested at LPC [3] and others have proposed in the past for different purposes [4] [5]. That, along with the event flags, could serve the same purpose. I'd be happy to take that approach if there is agreement that other consumers would want it.

(The LPC talk recording has some audio issues so I am not too sure on what the final agreement was)

Benchmarks
==========

AMD EPYC 9654 96-Core Processor host running a recent Linux kernel (6b4b543fa187), with a 64GB, 64-vCPU guest running stress-ng --vm while the host takes a QEMU background snapshot. All results are aggregated over 5 runs, and compared between the stock kernel and one with this patch series. Guest is rebooted across each individual run, and NUMA balancing is disabled.

pf/s and tlb/s (per-second deltas of KVM's pf_taken and remote_tlb_flush debugfs stats) clearly improve:

guest pattern baseline patched
-----------------------------------------------------------------------

  read64   (read)        pf/s=950,015   tlb/s=1,440  pf/s=2,503 tlb/s=0.1
  rand-set (read+write)  pf/s=1,004,353 tlb/s=862    pf/s=4,073 tlb/s=0.3
  write64  (write)       pf/s=968,018   tlb/s=94     pf/s=2,292 tlb/s=0.3

Guest read latency was sampled with a paced probe thread in the guest, reading one byte per page with an idle gap between reads, while stress-ng ran with read64. Each figure is the median across 5 runs, with the run-to-run min and max, in ns:

metric baseline patched
------------------------------------------------------
p50 (min max) 340 (270 380) 260 (250 260)

  p99 (min max)    19789 (16885 20200)  5207 (4076 5638)
  p99.9 (min max)  23445 (22845 24036)  6850 (6259 7110)

The median improves modestly and the refault cost shows in the tail, where p99 drops ~3.8x.

A separate stress-ng run with write64 and --verify showed no data corruption and no livelocks.

References
==========

[1] https://lore.kernel.org/linux-mm/cover.1eca029b8603ef4eebe5b41eae51facfc5920c41.1690292440.git-series.apopple@nvidia.com/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lpc.events/event/20/contributions/2552/
[4] https://lore.kernel.org/all/[email protected]
[5] https://lore.kernel.org/all/[email protected]

Carlos López (4):
mm/mprotect: add MM_CP_PERM_UPGRADE
mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE
KVM: plumb MMU_NOTIFIER_RANGE_PERM_UPGRADE into kvm_gfn_range
KVM: x86/mmu: don't zap the secondary MMU on permission upgrade

 arch/x86/kvm/mmu/mmu.c       |  7 +++++++
 include/linux/kvm_host.h     |  1 +
 include/linux/mm.h           |  8 ++++++++
 include/linux/mmu_notifier.h |  5 +++++
 mm/hugetlb.c                 |  6 +++++-
 mm/mprotect.c                | 20 +++++++++++++++++++-
 virt/kvm/kvm_main.c          |  4 ++++
 7 files changed, 49 insertions(+), 2 deletions(-)
Signature · 1 line
2.51.0
#1 lore raw

RFC PATCH 1/4 mm/mprotect: add MM_CP_PERM_UPGRADE

Add a change_protection() flag for changes that only add permissions:
the PFN stays the same and no access is revoked. mprotect() sets it when the new VMA permissions are a superset of the old, and the protection key is unchanged.

This will allow MMU notifier users to perform optimizations, as they can avoid invalidating PTEs whose permissions can be faulted in.

---
 include/linux/mm.h |  2 ++
 mm/mprotect.c      | 13 +++++++++++++
 2 files changed, 15 insertions(+)
include/linux/mm.h +2 -0
index 691cef038d1e..f406ce9d1032 100644
@@ -3415,6 +3415,8 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen);
 #define  MM_CP_UFFD_RWP_RESOLVE            (1UL << 5) /* resolve rwp */
 #define  MM_CP_UFFD_RWP_ALL                (MM_CP_UFFD_RWP | \
 					    MM_CP_UFFD_RWP_RESOLVE)
+/* Whether this change only upgrades/adds permissions */
+#define  MM_CP_PERM_UPGRADE                (1UL << 6)
 
 bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,
 			     pte_t pte);
mm/mprotect.c +13 -0
index 2888ee638d87..e9b636809c42 100644
@@ -756,6 +756,17 @@ static const struct mm_walk_ops prot_none_walk_ops = {
 	.walk_lock		= PGWALK_WRLOCK,
 };
 
+static bool vma_flags_is_perm_upgrade(vma_flags_t old_vma_flags,
+				       vma_flags_t new_vma_flags)
+{
+	vm_flags_t old = vma_flags_to_legacy(old_vma_flags);
+	vm_flags_t new = vma_flags_to_legacy(new_vma_flags);
+	vm_flags_t rwx = VM_READ | VM_WRITE | VM_EXEC;
+
+	return !(old & rwx & ~new) &&
+	       (old & ARCH_VM_PKEY_FLAGS) == (new & ARCH_VM_PKEY_FLAGS);
+}
+
 int
 mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,
 	       struct vm_area_struct *vma, struct vm_area_struct **pprev,
@@ -837,6 +848,8 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,
 	vma_flags_reset_once(vma, &new_vma_flags);
 	if (vma_wants_manual_pte_write_upgrade(vma))
 		mm_cp_flags |= MM_CP_TRY_CHANGE_WRITABLE;
+	if (vma_flags_is_perm_upgrade(old_vma_flags, new_vma_flags))
+		mm_cp_flags |= MM_CP_PERM_UPGRADE;
 	vma_set_page_prot(vma);
 
 	change_protection(tlb, vma, start, end, mm_cp_flags);
Signature · 1 line
2.51.0
#2 lore raw

Re: [RFC PATCH 1/4] mm/mprotect: add MM_CP_PERM_UPGRADE

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] mm/mprotect: MM_CP_PERM_UPGRADE set when change_protection() revokes write - [Medium] mm/mprotect…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] mm/mprotect: MM_CP_PERM_UPGRADE set when change_protection() revokes write
- [Medium] mm/mprotect: vma_flags_is_perm_upgrade() ignores arch-specific prot flags --

commit f3d5a380cac712469a433d985427420721eea496
Author: Carlos López <[email protected]>

mm/mprotect: add MM_CP_PERM_UPGRADE

This commit introduces the MM_CP_PERM_UPGRADE flag for change_protection() when mprotect() updates VMA permissions to a superset of the previous permissions without changing the protection key, allowing MMU notifier consumers to avoid invalidating PTEs.

Quoted 8 lines
mm/mprotect.c +0 -0
index 754333698c1ea..96e32e0159166 100644
@@ -756,6 +756,17 @@ static const struct mm_walk_ops prot_none_walk_ops = {
 	.walk_lock		= PGWALK_WRLOCK,
 };
 

mm/mprotect.c static const struct mm_walk_ops prot_none_walk_ops = { · line 768

Quoted 10 lines
mm/mprotect.c +10 -0
+static bool vma_flags_is_perm_upgrade(vma_flags_t old_vma_flags,
+				       vma_flags_t new_vma_flags)
+{
+	vm_flags_t old = vma_flags_to_legacy(old_vma_flags);
+	vm_flags_t new = vma_flags_to_legacy(new_vma_flags);
+	vm_flags_t rwx = VM_READ | VM_WRITE | VM_EXEC;
+
+	return !(old & rwx & ~new) &&
+	       (old & ARCH_VM_PKEY_FLAGS) == (new & ARCH_VM_PKEY_FLAGS);
+}

[Severity: Medium]
Does vma_flags_is_perm_upgrade() also need to check architecture-specific protection flags that mprotect() can modify?

In do_mprotect_pkey(), newflags is computed via calc_vm_prot_bits() (which calls arch_calc_vm_prot_bits()) and masks off VM_FLAGS_CLEAR (which includes VM_ARCH_CLEAR):

mm/mprotect.c:do_mprotect_pkey() {
    ...
    mask_off_old_flags = VM_ACCESS_FLAGS | VM_FLAGS_CLEAR;

    new_vma_pkey = arch_override_mprotect_pkey(vma, prot, pkey);
    newflags = calc_vm_prot_bits(prot, new_vma_pkey);
    newflags |= (vma->vm_flags & ~mask_off_old_flags);
    ...
}

Several architectures expose PROT_* flags in mprotect() that set or clear VMA flags affecting hardware PTE protection or attribute bits in vm_get_page_prot():

- On arm64, arch_calc_vm_prot_bits() maps PROT_BTI to VM_ARM64_BTI (cleared via VM_ARCH_CLEAR when omitted) and PROT_MTE to VM_MTE. In arch/arm64/mm/mmap.c:vm_get_page_prot(), VM_ARM64_BTI sets PTE_GP (which restricts execution by faulting on indirect branches to non-BTI landing pads) and VM_MTE sets PTE_ATTRINDX(MT_NORMAL_TAGGED) (which enables MTE tag check faults).
- On sparc64, PROT_ADI sets VM_SPARC_ADI (cleared via VM_ARCH_CLEAR when omitted), which sets _PAGE_MCD_4V in arch/sparc/mm/init_64.c:vm_get_page_prot() to enable hardware ADI tag checking traps.
- On powerpc64, PROT_SAO sets VM_SAO, which sets _PAGE_SAO in arch/powerpc/mm/book3s64/pgtable.c:vm_get_page_prot() to alter hardware memory ordering attributes.

Because vma_flags_is_perm_upgrade() only inspects VM_READ | VM_WRITE | VM_EXEC and ARCH_VM_PKEY_FLAGS, if userspace calls mprotect() to enable PROT_BTI, PROT_MTE, PROT_ADI, or PROT_SAO while keeping the read/write/exec bits identical (or a superset) and the pkey unchanged, won't this set MM_CP_PERM_UPGRADE and allow MMU notifier consumers to skip secondary MMU invalidation when access restrictions were tightened or PTE attributes changed?

mm/mprotect.c mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, · line 854

Quoted 13 lines
mm/mprotect.c +3 -0
+
 int
 mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,
 	       struct vm_area_struct *vma, struct vm_area_struct **pprev,
@@ -836,6 +847,8 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb,
 	vma_flags_reset_once(vma, &new_vma_flags);
 	if (vma_wants_manual_pte_write_upgrade(vma))
 		mm_cp_flags |= MM_CP_TRY_CHANGE_WRITABLE;
+	if (vma_flags_is_perm_upgrade(old_vma_flags, new_vma_flags))
+		mm_cp_flags |= MM_CP_PERM_UPGRADE;
 	vma_set_page_prot(vma);
 
 	change_protection(tlb, vma, start, end, mm_cp_flags);

[Severity: High]
Can setting MM_CP_PERM_UPGRADE when the VMA already had VM_WRITE (or on a hugetlb VMA) violate the invariant that no access is revoked and the PFN stays the same?

Follow-up commits ("mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE", "KVM: plumb MMU_NOTIFIER_RANGE_PERM_UPGRADE into kvm_gfn_range", and "KVM: x86/mmu: don't zap the secondary MMU on permission upgrade") wire MM_CP_PERM_UPGRADE to MMU_NOTIFIER_RANGE_PERM_UPGRADE so that kvm_unmap_gfn_range() skips zapping secondary MMU SPTEs.

However, when a VMA already has VM_WRITE and mprotect() adds permissions (for example, upgrading PROT_READ | PROT_WRITE to
PROT_READ | PROT_WRITE | PROT_EXEC, or PROT_WRITE to PROT_READ | PROT_WRITE), vma_set_page_prot() computes vma->vm_page_prot (newprot) without write permission for all MAP_PRIVATE mappings (COW) and for MAP_SHARED mappings where vma_wants_writenotify() is true.

When change_protection() applies this read-only newprot across the VMA, there are several paths where write permission is stripped from existing entries or entries are unmapped in the primary MMU while MM_CP_PERM_UPGRADE is set:

1. In mprotect_fixup(), vma_wants_manual_pte_write_upgrade(vma) is called after vma_flags_reset_once() but before vma_set_page_prot(vma). For MAP_SHARED VMAs without vm_ops->page_mkwrite that use writenotify for VM_UFFD_WP or soft-dirty tracking (such as shmem), vma_wants_writenotify(vma, vma->vm_page_prot) compares the old read-only vma->vm_page_prot against vma_pgprot_modify(vm_page_prot, vma->flags) (which still has VMA_SHARED_BIT | VMA_WRITE_BIT) and returns false. As a result, MM_CP_TRY_CHANGE_WRITABLE is not set, vma_set_page_prot() computes a read-only vma->vm_page_prot, and change_present_ptes() / change_huge_pmd() strip write permission from all writable PTEs and PMDs in the VMA.

2. In mm/hugetlb.c:hugetlb_change_protection(), MM_CP_TRY_CHANGE_WRITABLE is not handled, so huge_pte_modify(old_pte, newprot) unconditionally strips write permission from already-writable COW huge PTEs in MAP_PRIVATE hugetlb VMAs; in addition, huge_pmd_unshare() unmaps shared PMD tables via pud_clear().

3. In mm/huge_memory.c:change_huge_pud(), MM_CP_TRY_CHANGE_WRITABLE is not handled, so pud_modify(oldpud, newprot) strips write permission from writable huge PUDs in writenotify DAX mappings.

4. In mm/mprotect.c:change_softleaf_pte(), mm/huge_memory.c:change_non_present_huge_pmd(), and mm/hugetlb.c:hugetlb_change_protection(), writable migration entries (softleaf_is_migration_write()) and writable device-private entries (softleaf_is_device_private_write()) are unconditionally downgraded to read-only entries via make_readable_exclusive_migration_entry(), make_readable_migration_entry(), or make_readable_device_private_entry().

5. In change_present_ptes() and change_huge_pmd(), pte_modify() / pmd_modify() strip write permission and do not restore it under MM_CP_TRY_CHANGE_WRITABLE when !folio in a private PFN/mixed mapping or when a shared writenotify PTE was cleaned (!pte_dirty(ptent)).

Wouldn't skipping secondary MMU invalidation in these cases leave secondary MMUs (such as KVM SPTEs) with stale writable mappings to pages that change_protection() has write-protected or unmapped in the primary MMU?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#3 lore raw

RFC PATCH 2/4 mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE

In reply to Carlos López

Add a range flag marking an invalidation that only upgrades permissions (that is, no PFN change and no revoked access). change_protection() sets it for MM_CP_PERM_UPGRADE and for the userfaultfd permission resolves, which unprotect a page and thus only restore access.

A secondary MMU can use the flag to keep its mappings rather than zap them. Consumers that ignore it are unaffected.

---
 include/linux/mm.h           | 6 ++++++
 include/linux/mmu_notifier.h | 5 +++++
 mm/hugetlb.c                 | 6 +++++-
 mm/mprotect.c                | 7 ++++++-
 4 files changed, 22 insertions(+), 2 deletions(-)
include/linux/mm.h +6 -0
index f406ce9d1032..c4fc3876a4b7 100644
@@ -3418,6 +3418,12 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen);
 /* Whether this change only upgrades/adds permissions */
 #define  MM_CP_PERM_UPGRADE                (1UL << 6)
 
+static inline bool mm_cp_is_perm_upgrade(unsigned long cp_flags)
+{
+	return cp_flags & (MM_CP_PERM_UPGRADE | MM_CP_UFFD_WP_RESOLVE |
+			   MM_CP_UFFD_RWP_RESOLVE);
+}
+
 bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,
 			     pte_t pte);
 extern long change_protection(struct mmu_gather *tlb,
include/linux/mmu_notifier.h +5 -0
index a11a44eef521..112be28cdf6b 100644
@@ -60,6 +60,11 @@ enum mmu_notifier_event {
 };
 
 #define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0)
+/*
+ * Set on an invalidation that only expands permissions without
+ * revoking any existing access, nor freeing/moving the underlying page.
+ */
+#define MMU_NOTIFIER_RANGE_PERM_UPGRADE (1 << 1)
 
 struct mmu_notifier_ops {
 	/*
mm/hugetlb.c +5 -1
index b017ae1dfd6f..68d44044fdd2 100644
@@ -6548,19 +6548,23 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
 	long pages = 0, psize = huge_page_size(h);
 	struct mmu_notifier_range range;
 	unsigned long last_addr_mask;
+	unsigned int mn_flags = 0;
 	bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
 	bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
 	bool uffd_rwp = cp_flags & MM_CP_UFFD_RWP;
 	bool uffd_rwp_resolve = cp_flags & MM_CP_UFFD_RWP_RESOLVE;
 	struct mmu_gather tlb;
 
+	if (mm_cp_is_perm_upgrade(cp_flags))
+		mn_flags |= MMU_NOTIFIER_RANGE_PERM_UPGRADE;
+
 	/*
 	 * In the case of shared PMDs, the area to flush could be beyond
 	 * start/end.  Set range.start/range.end to cover the maximum possible
 	 * range if PMD sharing is possible.
 	 */
 	mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA,
-				0, mm, start, end);
+				mn_flags, mm, start, end);
 	adjust_range_if_pmd_sharing_possible(vma, &range.start, &range.end);
 
 	BUG_ON(address >= end);
mm/mprotect.c +6 -1
index e9b636809c42..b48d2fd3b7b5 100644
@@ -597,8 +597,13 @@ static inline long change_pud_range(struct mmu_gather *tlb,
 			continue;
 
 		if (!range.start) {
+			unsigned int mn_flags = 0;
+
+			if (mm_cp_is_perm_upgrade(cp_flags))
+				mn_flags |= MMU_NOTIFIER_RANGE_PERM_UPGRADE;
+
 			mmu_notifier_range_init(&range,
-						MMU_NOTIFY_PROTECTION_VMA, 0,
+						MMU_NOTIFY_PROTECTION_VMA, mn_flags,
 						vma->vm_mm, addr, end);
 			mmu_notifier_invalidate_range_start(&range);
 		}
Signature · 1 line
2.51.0
#4 lore raw

Re: [RFC PATCH 2/4] mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider: - [Critical] mm/hugetlb: hugetlb_change_protection() sets PERM_UPGRADE when revoking access - [High]…

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Critical] mm/hugetlb: hugetlb_change_protection() sets PERM_UPGRADE when revoking access
- [High] mm/mmu_notifier: change_protection() sets PERM_UPGRADE when revoking write
- [High] mm: change_protection() revokes write access under MMU_NOTIFIER_RANGE_PERM_UPGRADE
- [Low] mm/mmu_notifier: unused-but-set mn_flags when CONFIG_MMU_NOTIFIER=n --

commit 22cbc27b0ff3d2d6c7ee718ba2fc52e905cded0f
Author: Carlos López <[email protected]>

mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE

This commit adds the MMU_NOTIFIER_RANGE_PERM_UPGRADE range flag to mark MMU notifier invalidations that only upgrade permissions without revoking existing access or changing PFNs. It sets this flag in hugetlb_change_protection() and change_pud_range() when mm_cp_is_perm_upgrade(cp_flags) is true.

Quoted 16 lines
mm/hugetlb.c +2 -0
index da5d04edf7c10..3a988ca364e6c 100644
@@ -6596,19 +6596,23 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
 	long pages = 0, psize = huge_page_size(h);
 	struct mmu_notifier_range range;
 	unsigned long last_addr_mask;
+	unsigned int mn_flags = 0;
 	bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
 	bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
 	bool uffd_rwp = cp_flags & MM_CP_UFFD_RWP;
 	bool uffd_rwp_resolve = cp_flags & MM_CP_UFFD_RWP_RESOLVE;
 	struct mmu_gather tlb;
 
+	if (mm_cp_is_perm_upgrade(cp_flags))

mm/hugetlb.c long hugetlb_change_protection(struct vm_area_struct *vma, · line 6615

Quoted 10 lines
mm/hugetlb.c +3 -1
+		mn_flags |= MMU_NOTIFIER_RANGE_PERM_UPGRADE;
+
 	/*
 	 * In the case of shared PMDs, the area to flush could be beyond
 	 * start/end.  Set range.start/range.end to cover the maximum possible
 	 * range if PMD sharing is possible.
 	 */
 	mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA,
-				0, mm, start, end);
+				mn_flags, mm, start, end);

[Severity: Critical]
Can setting MMU_NOTIFIER_RANGE_PERM_UPGRADE in hugetlb_change_protection() leave dangling mappings in a secondary MMU or leave writable secondary MMU mappings to pages that the host unmapped or downgraded to read-only?

First, when PMD sharing is active on a MAP_SHARED hugetlbfs mapping (for example, two processes sharing a PUD_SIZE-aligned PROT_READ mapping of a 2MB hugetlbfs folio where only one process holds the folio rmap count on the shared PMD table) and the process with a secondary MMU calls mprotect() to upgrade to PROT_READ | PROT_WRITE, mprotect_fixup() sets MM_CP_PERM_UPGRADE. hugetlb_change_protection() then advertises MMU_NOTIFIER_RANGE_PERM_UPGRADE, telling secondary MMUs such as KVM to keep their existing SPTEs, and proceeds to unshare and unmap the PUD entry:

mm/hugetlb.c:hugetlb_change_protection()
-> huge_pmd_unshare()

    -> __huge_pmd_unshare()
         pud_clear(pud);

Because the PUD in this mm is now cleared and detached from the shared PMD page table without zapping the secondary MMU, when the other process later unmaps the file, unmap_hugepage_range() only notifies that other mm and drops the shared folio's mapcount to zero. If the folio is subsequently migrated or freed in migrate_hugetlb_folio(), folio_mapped(src) is false and try_to_migrate() is skipped, freeing or reusing the physical page while the first process's secondary MMU still retains a live mapping to it.

Second, unlike change_present_ptes() and change_huge_pmd(), hugetlb_change_protection() ignores MM_CP_TRY_CHANGE_WRITABLE and unconditionally overwrites the huge PTE protection with newprot (vma->vm_page_prot):

mm/hugetlb.c:hugetlb_change_protection() {
    ...
    old_pte = huge_ptep_modify_prot_start(vma, address, ptep);
    pte = huge_pte_modify(old_pte, newprot);
    pte = arch_make_huge_pte(pte, shift, vma->vm_flags);
    ...
}

For any MAP_PRIVATE hugetlb VMA, and for any MAP_SHARED hugetlb VMA with VM_UFFD_WP enabled (where vma_wants_writenotify() is true and vma_set_page_prot() clears VMA_SHARED_BIT from vma->vm_page_prot), newprot is read-only. When mprotect() upgrades permissions (for example adding PROT_EXEC with MM_CP_PERM_UPGRADE) or userfaultfd resolves write protection (MM_CP_UFFD_WP_RESOLVE via uffd_wp_range() or userfaultfd_clear_vma()), any already-writable huge PTE in the range has its write bit stripped to read-only while MMU_NOTIFIER_RANGE_PERM_UPGRADE tells secondary MMUs not to zap their writable mappings.

Similarly, hugetlb_change_protection() unconditionally downgrades writable migration entries to readable migration entries even when mm_cp_is_perm_upgrade(cp_flags) is true:

mm/hugetlb.c:hugetlb_change_protection() {
    ...
    if (softleaf_is_migration_write(entry)) {
        if (folio_test_anon(folio))
            entry = make_readable_exclusive_migration_entry(
                        swp_offset(entry));
        else
            entry = make_readable_migration_entry(
                        swp_offset(entry));
        newpte = swp_entry_to_pte(entry);
        pages++;
    }
    ...
}
Quoted 9 lines
mm/mprotect.c +1 -0
index 96e32e0159166..8c17875cdcfa6 100644
@@ -597,8 +597,13 @@ static inline long change_pud_range(struct mmu_gather *tlb,
 			continue;
 
 		if (!range.start) {
+			unsigned int mn_flags = 0;

mm/mprotect.c static inline long change_pud_range(struct mmu_gather *tlb, · line 609

Quoted 10 lines
mm/mprotect.c +5 -1
+
+			if (mm_cp_is_perm_upgrade(cp_flags))
+				mn_flags |= MMU_NOTIFIER_RANGE_PERM_UPGRADE;
+
 			mmu_notifier_range_init(&range,
-						MMU_NOTIFY_PROTECTION_VMA, 0,
+						MMU_NOTIFY_PROTECTION_VMA, mn_flags,
 						vma->vm_mm, addr, end);
 			mmu_notifier_invalidate_range_start(&range);
 		}

[Severity: High]
Does change_pud_range() also set MMU_NOTIFIER_RANGE_PERM_UPGRADE in cases where the underlying page table walk actually revokes write access?

1. In change_softleaf_pte() and change_non_present_huge_pmd() (as well as hugetlb_change_protection()), cp_flags is not checked when handling writable non-present entries. Both SOFTLEAF_MIGRATION_WRITE and SOFTLEAF_DEVICE_PRIVATE_WRITE entries are unconditionally downgraded to readable entries during permission upgrades:

mm/mprotect.c:change_softleaf_pte() {
    ...
    if (softleaf_is_migration_write(entry)) {
        ...
        if (folio_test_anon(folio))
            entry = make_readable_exclusive_migration_entry(swp_offset(entry));
        else
            entry = make_readable_migration_entry(swp_offset(entry));
        ...
    } else if (softleaf_is_device_private_write(entry)) {
        entry = make_readable_device_private_entry(swp_offset(entry));
        ...
    }
    ...
}

2. In change_huge_pud() (as in hugetlb_change_protection()), MM_CP_TRY_CHANGE_WRITABLE is not handled:

mm/huge_memory.c:change_huge_pud() {
    ...
    oldpud = pudp_invalidate(vma, addr, pudp);
    entry = pud_modify(oldpud, newprot);
    set_pud_at(mm, addr, pudp, entry);
    ...
}

Whenever vma->vm_page_prot is read-only (such as a MAP_SHARED DAX PUD VMA where vma_wants_writenotify() is true), pud_modify(oldpud, newprot) strips write permission from already-writable huge PUDs while MMU_NOTIFIER_RANGE_PERM_UPGRADE is advertised.

3. In change_present_ptes() and change_huge_pmd(), pte_modify() and pmd_modify() first apply newprot (which is read-only when !VM_SHARED or vma_wants_writenotify()), and only restore write permission if MM_CP_TRY_CHANGE_WRITABLE is set and can_change_pte_writable() or can_change_pmd_writable() returns true. For already-writable PTEs or PMDs where can_change_shared_pte_writable() returns false (for example, a writable shared PTE cleaned by clean_record_pte(), or a clean writable VM_PFNMAP/VM_MIXEDMAP entry) or where vm_normal_page() is NULL in a !VM_SHARED VMA, write permission is stripped while secondary MMUs are told no access was revoked.

[Severity: High]
What happens when change_protection() runs with MM_CP_PERM_UPGRADE, MM_CP_UFFD_WP_RESOLVE, or MM_CP_UFFD_RWP_RESOLVE on a writable MAP_SHARED VMA without vm_ops->page_mkwrite/pfn_mkwrite (such as shmem/tmpfs) after vma->vm_page_prot has been downgraded to PAGE_COPY for userfaultfd-wp or soft-dirty tracking?

When userfaultfd-wp is registered or soft-dirty is cleared on a writable MAP_SHARED shmem VMA, vma_set_page_prot() sees vma_wants_writenotify() return true, clears VMA_SHARED_BIT on its local flags copy, and stores read-only PAGE_COPY in vma->vm_page_prot while vma->flags retains VMA_SHARED_BIT | VMA_WRITE_BIT.

Later, when uffd_wp_range(), userfaultfd_clear_vma(), or mprotect_fixup() checks whether MM_CP_TRY_CHANGE_WRITABLE should be set, it calls vma_wants_manual_pte_write_upgrade(vma):

mm/vma.h:vma_wants_manual_pte_write_upgrade()
-> vma_wants_writenotify(vma, vma->vm_page_prot)

In vma_wants_writenotify(), because shmem has no page_mkwrite or pfn_mkwrite callback, execution reaches:

mm/vma.c:vma_wants_writenotify() {
    ...
    /* The open routine did something to the protections that pgprot_modify
     * won't preserve? */
    if (pgprot_val(vm_page_prot) !=
        pgprot_val(vma_pgprot_modify(vm_page_prot, vma->flags)))
        return false;
    ...
}

Because vm_page_prot is already PAGE_COPY whereas vma_pgprot_modify(vm_page_prot, vma->flags) computes PAGE_SHARED (and in mprotect_fixup(), vma_wants_manual_pte_write_upgrade(vma) is also called after vma_flags_reset_once() updates vma->flags but before vma_set_page_prot(vma) updates vma->vm_page_prot), vma_wants_writenotify() returns false. As a result, MM_CP_TRY_CHANGE_WRITABLE is not set, and change_present_ptes() and change_huge_pmd() apply read-only newprot (PAGE_COPY), stripping _PAGE_RW from all present writable PTEs and PMDs in the range while change_pud_range() sets MMU_NOTIFIER_RANGE_PERM_UPGRADE.

[Severity: Low]
When CONFIG_MMU_NOTIFIER=n, does mn_flags in change_pud_range() and hugetlb_change_protection() trigger a -Wunused-but-set-variable warning under W=1?

In include/linux/mmu_notifier.h, when CONFIG_MMU_NOTIFIER is disabled, mmu_notifier_range_init() is a macro that drops the flags argument:

#define mmu_notifier_range_init(range,event,flags,mm,start,end) \
_mmu_notifier_range_init(range, start, end)

So mn_flags is initialized and modified with |=, but never read after preprocessing when CONFIG_MMU_NOTIFIER=n.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#5 lore raw

Re: [RFC PATCH 2/4] mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE

In reply to Carlos López
kernel test robot noticed the following build warnings:

Hi Carlos,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:

[auto build test WARNING on kvm/queue]
[also build test WARNING on kvm/next mst-vhost/linux-next linus/master v7.3 next-20261030] [cannot apply to akpm-mm/mm-everything kvm/linux-next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Carlos-L-pez/mm-mprotect-add-MM_CP_PERM_UPGRADE/20261029-153334
base:   https://git.kernel.org/pub/scm/virt/kvm/kvm.git queue
patch link:    https://lore.kernel.org/r/20261029143336.390231-5-clopez%40suse.de
patch subject: [RFC PATCH 2/4] mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE
config: openrisc-allnoconfig (https://download.01.org/0day-ci/archive/20261105/[email protected]/config)
compiler: or1k-linux-gcc (GCC) 16.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261105/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

mm/mprotect.c: In function 'change_pud_range':

>> mm/mprotect.c:600:38: warning: variable 'mn_flags' set but not used [-Wunused-but-set-variable=]
     600 |                         unsigned int mn_flags = 0;
         |                                      ^~~~~~~~

vim +/mn_flags +600 mm/mprotect.c

573
574 static inline long change_pud_range(struct mmu_gather *tlb,
575 struct vm_area_struct *vma, p4d_t *p4d, unsigned long addr,
576 unsigned long end, pgprot_t newprot, unsigned long cp_flags)
577 {
578 struct mmu_notifier_range range;
579 pud_t *pudp, pud;
580 unsigned long next;
581 long pages = 0, ret;
582
583 range.start = 0;
584
585 pudp = pud_offset(p4d, addr);
586 do {
587 again:
588 next = pud_addr_end(addr, end);
589 ret = change_prepare(vma, pudp, pmd, addr, cp_flags);
590 if (ret) {
591 pages = ret;
592 break;
593 }
594
595 pud = pudp_get(pudp);
596 if (pud_none(pud))
597 continue;
598
599 if (!range.start) {

 > 600				unsigned int mn_flags = 0;

601
602 if (mm_cp_is_perm_upgrade(cp_flags))
603 mn_flags |= MMU_NOTIFIER_RANGE_PERM_UPGRADE;
604
605 mmu_notifier_range_init(&range,
606 MMU_NOTIFY_PROTECTION_VMA, mn_flags,
607 vma->vm_mm, addr, end);
608 mmu_notifier_invalidate_range_start(&range);
609 }
610
611 if (pud_leaf(pud)) {
612 if ((next - addr != PUD_SIZE) ||
613 pgtable_split_needed(vma, cp_flags)) {
614 __split_huge_pud(vma, pudp, addr);
615 goto again;
616 } else {
617 ret = change_huge_pud(tlb, vma, pudp,
618 addr, newprot, cp_flags);
619 if (ret == 0)
620 goto again;
621 /* huge pud was handled */
622 if (ret == HPAGE_PUD_NR)
623 pages += HPAGE_PUD_NR;
624 continue;
625 }
626 }
627
628 pages += change_pmd_range(tlb, vma, pudp, addr, next, newprot,
629 cp_flags);
630 } while (pudp++, addr = next, addr != end);
631
632 if (range.start)
633 mmu_notifier_invalidate_range_end(&range);
634
635 return pages;
636 }
637

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

#6 lore raw

Re: [RFC PATCH 2/4] mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE

In reply to Carlos López
kernel test robot noticed the following build warnings:

Hi Carlos,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:

[auto build test WARNING on kvm/queue]
[also build test WARNING on kvm/next mst-vhost/linux-next linus/master v7.3 next-20261030] [cannot apply to akpm-mm/mm-everything kvm/linux-next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Carlos-L-pez/mm-mprotect-add-MM_CP_PERM_UPGRADE/20261029-153334
base:   https://git.kernel.org/pub/scm/virt/kvm/kvm.git queue
patch link:    https://lore.kernel.org/r/20261029143336.390231-5-clopez%40suse.de
patch subject: [RFC PATCH 2/4] mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE
config: sh-defconfig (https://download.01.org/0day-ci/archive/20261105/[email protected]/config)
compiler: sh4-linux-gcc (GCC) 16.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261105/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

mm/hugetlb.c: In function 'hugetlb_change_protection':

>> mm/hugetlb.c:6509:22: warning: variable 'mn_flags' set but not used [-Wunused-but-set-variable=]
    6509 |         unsigned int mn_flags = 0;
         |                      ^~~~~~~~

vim +/mn_flags +6509 mm/hugetlb.c

6496
6497 long hugetlb_change_protection(struct vm_area_struct *vma,
6498 unsigned long address, unsigned long end,
6499 pgprot_t newprot, unsigned long cp_flags)
6500 {
6501 struct mm_struct *mm = vma->vm_mm;
6502 unsigned long start = address;
6503 pte_t *ptep;
6504 pte_t pte;
6505 struct hstate *h = hstate_vma(vma);
6506 long pages = 0, psize = huge_page_size(h);
6507 struct mmu_notifier_range range;
6508 unsigned long last_addr_mask;

> 6509		unsigned int mn_flags = 0;

6510 bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
6511 bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
6512 bool uffd_rwp = cp_flags & MM_CP_UFFD_RWP;
6513 bool uffd_rwp_resolve = cp_flags & MM_CP_UFFD_RWP_RESOLVE;
6514 struct mmu_gather tlb;
6515
6516 if (mm_cp_is_perm_upgrade(cp_flags))
6517 mn_flags |= MMU_NOTIFIER_RANGE_PERM_UPGRADE;
6518
6519 /*
6520 * In the case of shared PMDs, the area to flush could be beyond
6521 * start/end. Set range.start/range.end to cover the maximum possible
6522 * range if PMD sharing is possible.
6523 */
6524 mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA,
6525 mn_flags, mm, start, end);
6526 adjust_range_if_pmd_sharing_possible(vma, &range.start, &range.end);
6527
6528 BUG_ON(address >= end);
6529 flush_cache_range(vma, range.start, range.end);
6530 tlb_gather_mmu_vma(&tlb, vma);
6531
6532 mmu_notifier_invalidate_range_start(&range);
6533 hugetlb_vma_lock_write(vma);
6534 i_mmap_lock_write(vma->vm_file->f_mapping);
6535 last_addr_mask = hugetlb_mask_last_page(h);
6536 for (; address < end; address += psize) {
6537 softleaf_t entry;
6538 spinlock_t *ptl;
6539
6540 ptep = hugetlb_walk(vma, address, psize);
6541 if (!ptep) {
6542 /*
6543 * uffd_wp installs a pte marker on the unpopulated
6544 * entry; uffd_rwp does not install markers so the
6545 * allocation is unnecessary for it.
6546 */
6547 if (!uffd_wp) {
6548 address |= last_addr_mask;
6549 continue;
6550 }
6551 /*
6552 * Userfaultfd wr-protect requires pgtable
6553 * pre-allocations to install pte markers.
6554 */
6555 ptep = huge_pte_alloc(mm, vma, address, psize);
6556 if (!ptep) {
6557 pages = -ENOMEM;
6558 break;
6559 }
6560 }
6561 ptl = huge_pte_lock(h, mm, ptep);
6562 if (huge_pmd_unshare(&tlb, vma, address, ptep)) {
6563 /*
6564 * When uffd-wp is enabled on the vma, unshare
6565 * shouldn't happen at all. Warn about it if it
6566 * happened due to some reason.
6567 */
6568 WARN_ON_ONCE(uffd_wp || uffd_wp_resolve ||
6569 uffd_rwp || uffd_rwp_resolve);
6570 pages++;
6571 spin_unlock(ptl);
6572 address |= last_addr_mask;
6573 continue;
6574 }
6575 pte = huge_ptep_get(mm, address, ptep);
6576 if (huge_pte_none(pte)) {
6577 if (unlikely(uffd_wp))
6578 /* Safe to modify directly (none->non-present). */
6579 set_huge_pte_at(mm, address, ptep,
6580 make_pte_marker(PTE_MARKER_UFFD_WP),
6581 psize);
6582 goto next;
6583 }
6584
6585 entry = softleaf_from_pte(pte);
6586 if (unlikely(softleaf_is_hwpoison(entry))) {
6587 /* Nothing to do. */
6588 } else if (unlikely(softleaf_is_migration(entry))) {
6589 struct folio *folio = softleaf_to_folio(entry);
6590 pte_t newpte = pte;
6591
6592 if (softleaf_is_migration_write(entry)) {
6593 if (folio_test_anon(folio))
6594 entry = make_readable_exclusive_migration_entry(
6595 swp_offset(entry));
6596 else
6597 entry = make_readable_migration_entry(
6598 swp_offset(entry));
6599 newpte = swp_entry_to_pte(entry);
6600 pages++;
6601 }
6602
6603 if (uffd_wp || uffd_rwp)
6604 newpte = pte_swp_mkuffd(newpte);
6605 else if (uffd_wp_resolve || uffd_rwp_resolve)
6606 newpte = pte_swp_clear_uffd(newpte);
6607 if (!pte_same(pte, newpte))
6608 set_huge_pte_at(mm, address, ptep, newpte, psize);
6609 } else if (unlikely(pte_is_marker(pte))) {
6610 /*
6611 * Do nothing on a poison marker; page is
6612 * corrupted, permissions do not apply. Here
6613 * pte_marker_uffd_wp()==true implies !poison
6614 * because they're mutual exclusive.
6615 */
6616 if (pte_is_uffd_wp_marker(pte) &&
6617 (uffd_wp_resolve || uffd_rwp_resolve))
6618 /* Safe to modify directly (non-present->none). */
6619 huge_pte_clear(mm, address, ptep, psize);
6620 } else {
6621 pte_t old_pte;
6622 unsigned int shift = huge_page_shift(hstate_vma(vma));
6623
6624 /* Already protnone with uffd bit set? Nothing to do. */
6625 if (uffd_rwp && pte_protnone(pte) && huge_pte_uffd(pte))
6626 goto next;
6627
6628 old_pte = huge_ptep_modify_prot_start(vma, address, ptep);
6629 pte = huge_pte_modify(old_pte, newprot);
6630 pte = arch_make_huge_pte(pte, shift, vma->vm_flags);
6631 if (uffd_wp || uffd_rwp)
6632 pte = huge_pte_mkuffd(pte);
6633 else if (uffd_wp_resolve || uffd_rwp_resolve)
6634 pte = huge_pte_clear_uffd(pte);
6635
6636 /* Preserve RWP protection across mprotect() */
6637 if (userfaultfd_rwp(vma) && huge_pte_uffd(pte)) {
6638 pte = huge_pte_modify(pte, PAGE_NONE);
6639 pte = arch_make_huge_pte(pte, shift, vma->vm_flags);
6640 }
6641
6642 huge_ptep_modify_prot_commit(vma, address, ptep, old_pte, pte);
6643 pages++;
6644 tlb_remove_huge_tlb_entry(h, &tlb, ptep, address);
6645 }
6646
6647 next:
6648 spin_unlock(ptl);
6649 cond_resched();
6650 }
6651
6652 tlb_flush_mmu_tlbonly(&tlb);
6653 huge_pmd_unshare_flush(&tlb, vma);
6654 /*
6655 * No need to call mmu_notifier_arch_invalidate_secondary_tlbs() we are
6656 * downgrading page table protection not changing it to point to a new
6657 * page.
6658 *
6659 * See Documentation/mm/mmu_notifier.rst
6660 */
6661 i_mmap_unlock_write(vma->vm_file->f_mapping);
6662 hugetlb_vma_unlock_write(vma);
6663 mmu_notifier_invalidate_range_end(&range);
6664 tlb_finish_mmu(&tlb);
6665
6666 return pages > 0 ? (pages << h->order) : pages;
6667 }
6668

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

#7 lore raw

RFC PATCH 3/4 KVM: plumb MMU_NOTIFIER_RANGE_PERM_UPGRADE into kvm_gfn_range

In reply to Carlos López

Propagate the permission upgrade flag from the mmu_notifier range into struct kvm_gfn_range, so the unmap handler can react accordingly. The value is not consumed yet, so no functional change.

---
 include/linux/kvm_host.h | 1 +
 virt/kvm/kvm_main.c      | 4 ++++
 2 files changed, 5 insertions(+)
include/linux/kvm_host.h +1 -0
index e7fc94e2910a..aa109818cede 100644
@@ -274,6 +274,7 @@ struct kvm_gfn_range {
 	enum kvm_gfn_range_filter attr_filter;
 	bool may_block;
 	bool lockless;
+	bool perm_upgrade;
 };
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
virt/kvm/kvm_main.c +4 -0
index cf4cb586f140..0d0073615fa8 100644
@@ -532,6 +532,7 @@ struct kvm_mmu_notifier_range {
 	bool flush_on_ret;
 	bool may_block;
 	bool lockless;
+	bool perm_upgrade;
 };
 
 /*
@@ -613,6 +614,7 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
 			 */
 			gfn_range.arg = range->arg;
 			gfn_range.may_block = range->may_block;
+			gfn_range.perm_upgrade = range->perm_upgrade;
 			/*
 			 * HVA-based notifications aren't relevant to private
 			 * mappings as they don't have a userspace mapping.
@@ -738,6 +740,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
 		.on_lock	= kvm_mmu_invalidate_start,
 		.flush_on_ret	= true,
 		.may_block	= mmu_notifier_range_blockable(range),
+		.perm_upgrade = !!(range->flags & MMU_NOTIFIER_RANGE_PERM_UPGRADE),
 	};
 
 	trace_kvm_unmap_hva_range(range->start, range->end);
@@ -2552,6 +2555,7 @@ static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
 
 	gfn_range.arg = range->arg;
 	gfn_range.may_block = range->may_block;
+	gfn_range.perm_upgrade = range->perm_upgrade;
 
 	/*
 	 * If/when KVM supports more attributes beyond private .vs shared, this
Signature · 1 line
2.51.0
#8 lore raw

Re: [RFC PATCH 3/4] KVM: plumb MMU_NOTIFIER_RANGE_PERM_UPGRADE into kvm_gfn_range

Sashiko has reviewed this patch and found no issues.
Quoted 5 lines

Propagate the permission upgrade flag from the mmu_notifier range into struct kvm_gfn_range, so the unmap handler can react accordingly. The value is not consumed yet, so no functional change.

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#9 lore raw

RFC PATCH 4/4 KVM: x86/mmu: don't zap the secondary MMU on permission upgrade

In reply to Carlos López

When an mmu_notifier invalidation only upgrades permissions (MMU_NOTIFIER_RANGE_PERM_UPGRADE), keep the existing SPTEs instead of zapping them. The PFN is unchanged and no access is revoked, so a held SPTE is at most as permissive as before. Permissions for the SPTE are updated on the next fault, so dropping them early gains nothing.

The motivating case is a userfaultfd-wp dirty tracker such as QEMU's background snapshot, which unprotects every guest page as it saves it. Zapping on each unprotect makes the guest refault its whole working set. On a 64GB guest under a background snapshot, a read-mostly workload drops secondary-MMU refaults from ~1M/s to ~2.5K/s and cross-vCPU TLB shootdowns from ~1400/s to near zero. Uses of mprotect() that only add access benefit likewise.

---
 arch/x86/kvm/mmu/mmu.c | 7 +++++++
 1 file changed, 7 insertions(+)
arch/x86/kvm/mmu/mmu.c +7 -0
index dc2aef72ffdd..a4258a8c0c89 100644
@@ -1719,6 +1719,13 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 	lockdep_assert_once(kvm->mmu_invalidate_in_progress ||
 			    lockdep_is_held(&kvm->slots_lock));
 
+	/*
+	 * Do not bother zapping SPTEs if this is only a protection upgrade.
+	 * Permissions will be faulted-in on demand.
+	 */
+	if (range->perm_upgrade)
+		return false;
+
 	if (kvm_memslots_have_rmaps(kvm))
 		flush = __kvm_rmap_zap_gfn_range(kvm, range->slot,
 						 range->start, range->end,
Signature · 1 line
2.51.0
#10 lore raw

Re: [RFC PATCH 4/4] KVM: x86/mmu: don't zap the secondary MMU on permission upgrade

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Critical] KVM: x86/mmu: stale SPTEs in kvm_unmap_gfn_range() on hugetlb PMD unshare - [High] mm…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] KVM: x86/mmu: stale SPTEs in kvm_unmap_gfn_range() on hugetlb PMD unshare
- [High] mm: change_protection() revokes _PAGE_RW under MMU_NOTIFIER_RANGE_PERM_UPGRADE
- [Medium] KVM: x86/mmu: kvm_unmap_gfn_range() skips APIC page reload on perm_upgrade --

commit eb63ff65a947838d12f405a5148038708c2ed0d7
Author: Carlos López <[email protected]>

KVM: x86/mmu: don't zap the secondary MMU on permission upgrade

This commit skips zapping secondary MMU SPTEs in kvm_unmap_gfn_range() when an mmu_notifier invalidation has range->perm_upgrade set, leaving permission upgrades to be faulted in on demand.

Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +0 -0
index dc2aef72ffddc..a4258a8c0c899 100644

arch/x86/kvm/mmu/mmu.c bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) · line 1727

Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +6 -0
@@ -1719,6 +1719,13 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 	lockdep_assert_once(kvm->mmu_invalidate_in_progress ||
 			    lockdep_is_held(&kvm->slots_lock));
 
+	/*
+	 * Do not bother zapping SPTEs if this is only a protection upgrade.
+	 * Permissions will be faulted-in on demand.
+	 */
+	if (range->perm_upgrade)
+		return false;

[Severity: Critical]
Can skipping the zap in arch/x86/kvm/mmu/mmu.c:kvm_unmap_gfn_range() leave stale SPTEs pointing to freed 2MB hugetlbfs folios when hugetlb PMD page table sharing is active (CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING)?

When a guest backed by a MAP_SHARED hugetlbfs VMA faults in a 2MB page, huge_pmd_share() can map a shared PMD table into kvm->mm's PUD while KVM installs a present 2MB leaf SPTE and drops the GUP reference in kvm_mmu_finish_page_fault().

If host userspace later upgrades permissions via mprotect() over the VMA or a PUD-aligned range (where hugetlb_split() does not unshare beforehand), mprotect_fixup() sets MM_CP_PERM_UPGRADE and mm/hugetlb.c:hugetlb_change_protection() marks the mmu_notifier range with MMU_NOTIFIER_RANGE_PERM_UPGRADE:

mm/hugetlb.c:hugetlb_change_protection() {
	if (mm_cp_is_perm_upgrade(cp_flags))
		mn_flags |= MMU_NOTIFIER_RANGE_PERM_UPGRADE;
	...
	mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA,
				mn_flags, mm, start, end);
	adjust_range_if_pmd_sharing_possible(vma, &range.start, &range.end);
	...
		ptl = huge_pte_lock(h, mm, ptep);
		if (huge_pmd_unshare(&tlb, vma, address, ptep)) {
			...
			pages++;
			spin_unlock(ptl);
			address |= last_addr_mask;
			continue;
		}
}

With range->perm_upgrade set, kvm_unmap_gfn_range() returns false without zapping its SPTEs. However, inside the loop, hugetlb_change_protection() -> huge_pmd_unshare() -> __huge_pmd_unshare() clears the PUD entry (pud_clear(pud)) in kvm->mm, decrements the shared PMD count, and skips the rest of the 1GB PUD range without re-populating PTEs or issuing a non-upgrade mmu_notifier invalidation.

Because the shared PMD table holds only a single rmap count across all sharing VMAs, once the other sharing VMA unmaps the range via __unmap_hugepage_range() -> hugetlb_remove_rmap(folio), folio_mapped(folio) becomes false while kvm->mm is already pud_none() and KVM still holds live SPTEs.

When that folio is subsequently migrated or truncated, mm/migrate.c:unmap_and_move_huge_page() (or try_to_migrate() -> mm/rmap.c:__rmap_walk_file() stopping once rwc->done / folio_not_mapped() is true, or fs/hugetlbfs/inode.c:remove_inode_single_folio() -> hugetlb_unmap_file_folio()) sees !folio_mapped(src) and skips mmu_notifier invalidation on kvm->mm:

mm/migrate.c:unmap_and_move_huge_page() {
	if (folio_mapped(src)) {
		...
		try_to_migrate(src, ttu);
		was_mapped = 1;
	}

	if (!folio_mapped(src))
		rc = move_to_new_folio(dst, src, mode);
}

This leaves the guest with direct read/write access to the migrated or freed folio, and if the guest later faults on that GFN after a new folio is mapped, KVM attempts to replace the still-present leaf SPTE with a different PFN and hits BUG() in arch/x86/kvm/mmu/tdp_mmu.c:__handle_changed_spte():

	if (was_leaf && is_leaf && pfn_changed) {
		...
		BUG();
	}

[Severity: High]
Is it always guaranteed that primary MMU entries retain _PAGE_RW when range->perm_upgrade is set?

In mm/mprotect.c:change_protection(), newprot (vma->vm_page_prot) is passed to pte_modify(), pmd_modify(), pud_modify(), and huge_pte_modify(). Because _PAGE_RW is not in _COMMON_PAGE_CHG_MASK, those helpers overwrite _PAGE_RW from newprot. Whenever vma->vm_page_prot has _PAGE_RW == 0 (such as MAP_PRIVATE VMAs or MAP_SHARED VMAs where vma_wants_writenotify() was true due to VM_UFFD_WP, soft-dirty tracking, vm_ops_needs_writenotify, or vma_fs_can_writeback), _PAGE_RW is stripped from any already-writable entry in the range unless explicitly restored:

1. In mm/hugetlb.c:hugetlb_change_protection() and mm/huge_memory.c:change_huge_pud(), MM_CP_TRY_CHANGE_WRITABLE is not implemented:

mm/hugetlb.c:hugetlb_change_protection() {
	...
	old_pte = huge_ptep_modify_prot_start(vma, address, ptep);
	pte = huge_pte_modify(old_pte, newprot);
	pte = arch_make_huge_pte(pte, shift, vma->vm_flags);
}

Any already-writable hugetlb PTE or huge PUD has _PAGE_RW cleared during mprotect() permission upgrades (e.g. PROT_READ | PROT_WRITE to PROT_READ | PROT_WRITE | PROT_EXEC), UFFDIO_WRITEPROTECT(enable_wp=false) across a range containing already-writable hugetlb pages, or mm/userfaultfd.c:userfaultfd_clear_vma() on UFFDIO_UNREGISTER / close(uffd) (which calls change_protection() with MM_CP_UFFD_WP_RESOLVE while VM_UFFD_WP is still set before userfaultfd_reset_ctx() updates vma->vm_page_prot).

2. For 4KB PTEs in mm/mprotect.c:change_present_ptes() and 2MB THP PMDs in mm/huge_memory.c:change_huge_pmd(), restoring _PAGE_RW depends on MM_CP_TRY_CHANGE_WRITABLE from vma_wants_manual_pte_write_upgrade(vma), which calls mm/vma.c:vma_wants_writenotify(vma, vma->vm_page_prot):

mm/vma.c:vma_wants_writenotify() {
	if (pgprot_val(vm_page_prot) !=
	    pgprot_val(vma_pgprot_modify(vm_page_prot, vma->flags)))
		return false;
}

On a MAP_SHARED VMA where vm_ops_needs_writenotify(vma->vm_ops) is false (such as shmem/memfd) and vma->vm_page_prot was set to read-only due to VM_UFFD_WP or soft-dirty tracking, or in mm/mprotect.c:mprotect_fixup() where vma_flags_reset_once() updates vma->flags before calling vma_wants_manual_pte_write_upgrade(vma) and vma_set_page_prot(vma):

mm/mprotect.c:mprotect_fixup() {
	vma_flags_reset_once(vma, &new_vma_flags);
	if (vma_wants_manual_pte_write_upgrade(vma))
		mm_cp_flags |= MM_CP_TRY_CHANGE_WRITABLE;
	if (vma_flags_is_perm_upgrade(old_vma_flags, new_vma_flags))
		mm_cp_flags |= MM_CP_PERM_UPGRADE;
	vma_set_page_prot(vma);
}

vma_wants_writenotify() returns false and MM_CP_TRY_CHANGE_WRITABLE is not set, so change_present_ptes() and change_huge_pmd() strip _PAGE_RW from already-writable PTEs and PMDs. Even when MM_CP_TRY_CHANGE_WRITABLE is set, can_change_shared_pte_writable() and can_change_pmd_writable() only restore _PAGE_RW if the entry is dirty, downgrading writable-but-clean shared PTEs/PMDs to read-only.

Because kvm_unmap_gfn_range() skips zapping on range->perm_upgrade, KVM keeps writable leaf SPTEs (PT_WRITABLE_MASK, shadow_host_writable_mask, and shadow_mmu_writable_mask) while the host PTE is now read-only, bypassing host write-fault tracking. If a subsequent non-write KVM fault (such as an access-tracking fault or shadow MMU exec fault) resolves the read-only host PTE with fault->map_writable = false, make_spte() clears shadow_host_writable_mask and shadow_mmu_writable_mask on the present leaf SPTE and triggers the WARN_ON_ONCE in arch/x86/kvm/mmu/tdp_mmu.c:tdp_mmu_map_handle_target_level():

	else if (is_shadow_present_pte(iter->old_spte) &&
		 (!is_last_spte(iter->old_spte, iter->level) ||
		  WARN_ON_ONCE(leaf_spte_change_needs_tlb_flush(iter->old_spte, new_spte))))
		kvm_flush_remote_tlbs_gfn(vcpu->kvm, iter->gfn, iter->level);

[Severity: Medium]
What happens to the APIC access page reload in arch/x86/kvm/mmu/mmu.c:kvm_unmap_gfn_range() when range->perm_upgrade is true?

Further down in kvm_unmap_gfn_range(), invalidations on APIC_ACCESS_PAGE_PRIVATE_MEMSLOT issue KVM_REQ_APIC_PAGE_RELOAD:

	if (kvm_x86_ops.set_apic_access_page_addr &&
	    range->slot->id == APIC_ACCESS_PAGE_PRIVATE_MEMSLOT)
		kvm_make_all_cpus_request(kvm, KVM_REQ_APIC_PAGE_RELOAD);

When arch/x86/kvm/vmx/vmx.c:vmx_set_apic_access_page_addr() handles KVM_REQ_APIC_PAGE_RELOAD, it resolves the PFN with FOLL_WRITE:

	pfn = __kvm_faultin_pfn(slot, gfn, FOLL_WRITE, &writable, &refcounted_page);
	if (is_error_noslot_pfn(pfn))
		return;

	read_lock(&vcpu->kvm->mmu_lock);
	if (mmu_invalidate_retry_gfn(kvm, mmu_seq, gfn))
		kvm_make_request(KVM_REQ_APIC_PAGE_RELOAD, vcpu);
	else
		vmcs_write64(APIC_ACCESS_ADDR, pfn_to_hpa(pfn));

If the VMA backing APIC_ACCESS_PAGE_PRIVATE_MEMSLOT is made read-only (e.g. via mprotect(..., PROT_READ)) before the page is faulted in or while the backing page is migrated, swapped out, or unmapped, __kvm_faultin_pfn() fails after kvm_check_request() has already cleared KVM_REQ_APIC_PAGE_RELOAD, leaving APIC_ACCESS_ADDR uninitialized or holding a stale HPA.

When write permission on that VMA is later restored via mprotect(..., PROT_READ | PROT_WRITE), range->perm_upgrade is true and the early return in kvm_unmap_gfn_range() skips requesting KVM_REQ_APIC_PAGE_RELOAD. Since APIC_ACCESS_ADDR in the VMCS is not updated on demand during kvm_mmu_do_page_fault(), wouldn't this leave APIC_ACCESS_ADDR stale?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
#11 lore raw
In reply to Carlos López
Quoted 56 lines · On Thu, 29 Oct 2026 15:33:33 +0100 Carlos López <[email protected]> wrote:

Prevent x86 KVM from over-zapping its secondary MMU on PTE permission upgrades, which leave the PFNs unchanged and do not revoke access. Permissions for SPTEs that KVM holds are updated on the next fault, so dropping them early gains nothing.

Do this by adding a new MMU notifier flag to signal that a MMU_NOTIFY_PROTECTION_VMA strictly upgrades (loosens) permissions on a particular range.

See below for benchmark results.

Background
==========

change_protection() pairs every protection change with an MMU_NOTIFY_PROTECTION_VMA invalidation, and KVM zaps its SPTEs in response, as it cannot tell whether permissions widened or narrowed. This is costly e.g. during a QEMU background snapshot, which relies on uffd-wp to write-protect guest RAM and then restore permissions as each page is saved. Without this series, the permission restore causes the guest to re-fault read-only memory needlessly.

Note, the reverse situation is already handled for secondary MMUs that reuse the CPU page tables (e.g. ARM SMMU, IOMMU/ATS). They cannot refault, so they are notified of a permission upgrade on the CPU TLB-flush path [1].

KVM, however, owns separate page tables and faults permissions on demand, so it needs no such notification (as Sean noted when that signalling was first proposed on the fault path [2]). This series applies the same principle to the change_protection() path, signalling the permission upgrade so KVM can avoid the unnecessary zap.

RFC Considerations
==================

1. Having the zap skip in KVM's arch-independent code does not seem obviously correct, as s390 caches additional translation state that may not be rebuilt by refaults. Maybe an expert can correct me. The skip could be extended to arm64 and RISC-V, but I have not benchmarked the benefits and I'd like to get feedback first. Hence, x86-only for now.

2. change_protection() could simply not emit the invalidation at all for a pure permission upgrade (shared-page-table MMUs invalidate on the CPU TLB-flush path, as noted above, so they do not need it). But again, the s390 case could be broken, and even then, that would seem too much intrusion for KVM's benefit.

3. The flag reaches KVM as a bool on struct kvm_gfn_range. Alternatively, KVM could forward the notifier event, as Claudio Imbrenda suggested at LPC [3] and others have proposed in the past for different purposes [4] [5]. That, along with the event flags, could serve the same purpose. I'd be happy to take that approach if there is agreement that other consumers would want it.

Quoted 2 lines

(The LPC talk recording has some audio issues so I am not too sure on what the final agreement was)

AFAIR the consensus was:
- adding the reason to the kvm kvm_gfn_range is fine.
- having a return value from the notifier to pass information back is _not_ fine; I should add a new callback instead.

regarding this series:

on x86 you don't care what the new permissions are, because you're updating when re-faulting (if I understood correctly).

for s390 it might be beneficial to know what the new permissions are; of course we could look up the vma and find out, but since you are already checking the permissions, maybe you can find a way to also convey the new permission bits?

that would make our life on s390 easier :)

Quoted 61 lines

Benchmarks
==========

AMD EPYC 9654 96-Core Processor host running a recent Linux kernel (6b4b543fa187), with a 64GB, 64-vCPU guest running stress-ng --vm while the host takes a QEMU background snapshot. All results are aggregated over 5 runs, and compared between the stock kernel and one with this patch series. Guest is rebooted across each individual run, and NUMA balancing is disabled.

pf/s and tlb/s (per-second deltas of KVM's pf_taken and remote_tlb_flush debugfs stats) clearly improve:

guest pattern baseline patched
-----------------------------------------------------------------------

  read64   (read)        pf/s=950,015   tlb/s=1,440  pf/s=2,503 tlb/s=0.1
  rand-set (read+write)  pf/s=1,004,353 tlb/s=862    pf/s=4,073 tlb/s=0.3
  write64  (write)       pf/s=968,018   tlb/s=94     pf/s=2,292 tlb/s=0.3

Guest read latency was sampled with a paced probe thread in the guest, reading one byte per page with an idle gap between reads, while stress-ng ran with read64. Each figure is the median across 5 runs, with the run-to-run min and max, in ns:

metric baseline patched
------------------------------------------------------
p50 (min max) 340 (270 380) 260 (250 260)

  p99 (min max)    19789 (16885 20200)  5207 (4076 5638)
  p99.9 (min max)  23445 (22845 24036)  6850 (6259 7110)

The median improves modestly and the refault cost shows in the tail, where p99 drops ~3.8x.

A separate stress-ng run with write64 and --verify showed no data corruption and no livelocks.

References
==========

[1] https://lore.kernel.org/linux-mm/cover.1eca029b8603ef4eebe5b41eae51facfc5920c41.1690292440.git-series.apopple@nvidia.com/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lpc.events/event/20/contributions/2552/
[4] https://lore.kernel.org/all/[email protected]
[5] https://lore.kernel.org/all/[email protected]

Carlos López (4):
mm/mprotect: add MM_CP_PERM_UPGRADE
mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE
KVM: plumb MMU_NOTIFIER_RANGE_PERM_UPGRADE into kvm_gfn_range
KVM: x86/mmu: don't zap the secondary MMU on permission upgrade

 arch/x86/kvm/mmu/mmu.c       |  7 +++++++
 include/linux/kvm_host.h     |  1 +
 include/linux/mm.h           |  8 ++++++++
 include/linux/mmu_notifier.h |  5 +++++
 mm/hugetlb.c                 |  6 +++++-
 mm/mprotect.c                | 20 +++++++++++++++++++-
 virt/kvm/kvm_main.c          |  4 ++++
 7 files changed, 49 insertions(+), 2 deletions(-)
#12 lore raw
Quoted 67 lines · Claudio Imbrenda
Quoted 56 lines · On Thu, 29 Oct 2026 15:33:33 +0100 Carlos López <[email protected]> wrote:

Prevent x86 KVM from over-zapping its secondary MMU on PTE permission upgrades, which leave the PFNs unchanged and do not revoke access. Permissions for SPTEs that KVM holds are updated on the next fault, so dropping them early gains nothing.

Do this by adding a new MMU notifier flag to signal that a MMU_NOTIFY_PROTECTION_VMA strictly upgrades (loosens) permissions on a particular range.

See below for benchmark results.

Background
==========

change_protection() pairs every protection change with an MMU_NOTIFY_PROTECTION_VMA invalidation, and KVM zaps its SPTEs in response, as it cannot tell whether permissions widened or narrowed. This is costly e.g. during a QEMU background snapshot, which relies on uffd-wp to write-protect guest RAM and then restore permissions as each page is saved. Without this series, the permission restore causes the guest to re-fault read-only memory needlessly.

Note, the reverse situation is already handled for secondary MMUs that reuse the CPU page tables (e.g. ARM SMMU, IOMMU/ATS). They cannot refault, so they are notified of a permission upgrade on the CPU TLB-flush path [1].

KVM, however, owns separate page tables and faults permissions on demand, so it needs no such notification (as Sean noted when that signalling was first proposed on the fault path [2]). This series applies the same principle to the change_protection() path, signalling the permission upgrade so KVM can avoid the unnecessary zap.

RFC Considerations
==================

1. Having the zap skip in KVM's arch-independent code does not seem obviously correct, as s390 caches additional translation state that may not be rebuilt by refaults. Maybe an expert can correct me. The skip could be extended to arm64 and RISC-V, but I have not benchmarked the benefits and I'd like to get feedback first. Hence, x86-only for now.

2. change_protection() could simply not emit the invalidation at all for a pure permission upgrade (shared-page-table MMUs invalidate on the CPU TLB-flush path, as noted above, so they do not need it). But again, the s390 case could be broken, and even then, that would seem too much intrusion for KVM's benefit.

3. The flag reaches KVM as a bool on struct kvm_gfn_range. Alternatively, KVM could forward the notifier event, as Claudio Imbrenda suggested at LPC [3] and others have proposed in the past for different purposes [4] [5]. That, along with the event flags, could serve the same purpose. I'd be happy to take that approach if there is agreement that other consumers would want it.

Quoted 2 lines

(The LPC talk recording has some audio issues so I am not too sure on what the final agreement was)

AFAIR the consensus was:
- adding the reason to the kvm kvm_gfn_range is fine.
- having a return value from the notifier to pass information back is _not_ fine; I should add a new callback instead.

Quoted 4 lines

regarding this series:

on x86 you don't care what the new permissions are, because you're updating when re-faulting (if I understood correctly).

Correct.

Quoted 6 lines

for s390 it might be beneficial to know what the new permissions are; of course we could look up the vma and find out, but since you are already checking the permissions, maybe you can find a way to also convey the new permission bits?

that would make our life on s390 easier :)

Right, getting the new bits from vma->flags should be trivially done from the MMU_NOTIFY_PROTECTION_VMA callsites. There'd be no users in this series though. The rest of architectures still need the notifier to tell them if the change is an upgrade, since they don't have an easy way to grab the old RWX bits. Passing the old RWX bits from the MMU notifier does not seem trivial to me either.

So what I can do is to plumb down the event and flags into KVM, and leave it open for you to come back and forward the permission bits you need via the range flags, if the MM maintainers are okay with it.

However, I tried to understand the s390 code better with AI, and it seems to think that skipping the zap is also safe there (and an improvement). It could be further optimized by passing down the new W bit to eagerly fix the gmap when restoring write permissions.

Quoted 62 lines
Quoted 61 lines

Benchmarks
==========

AMD EPYC 9654 96-Core Processor host running a recent Linux kernel (6b4b543fa187), with a 64GB, 64-vCPU guest running stress-ng --vm while the host takes a QEMU background snapshot. All results are aggregated over 5 runs, and compared between the stock kernel and one with this patch series. Guest is rebooted across each individual run, and NUMA balancing is disabled.

pf/s and tlb/s (per-second deltas of KVM's pf_taken and remote_tlb_flush debugfs stats) clearly improve:

guest pattern baseline patched
-----------------------------------------------------------------------

  read64   (read)        pf/s=950,015   tlb/s=1,440  pf/s=2,503 tlb/s=0.1
  rand-set (read+write)  pf/s=1,004,353 tlb/s=862    pf/s=4,073 tlb/s=0.3
  write64  (write)       pf/s=968,018   tlb/s=94     pf/s=2,292 tlb/s=0.3

Guest read latency was sampled with a paced probe thread in the guest, reading one byte per page with an idle gap between reads, while stress-ng ran with read64. Each figure is the median across 5 runs, with the run-to-run min and max, in ns:

metric baseline patched
------------------------------------------------------
p50 (min max) 340 (270 380) 260 (250 260)

  p99 (min max)    19789 (16885 20200)  5207 (4076 5638)
  p99.9 (min max)  23445 (22845 24036)  6850 (6259 7110)

The median improves modestly and the refault cost shows in the tail, where p99 drops ~3.8x.

A separate stress-ng run with write64 and --verify showed no data corruption and no livelocks.

References
==========

[1] https://lore.kernel.org/linux-mm/cover.1eca029b8603ef4eebe5b41eae51facfc5920c41.1690292440.git-series.apopple@nvidia.com/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lpc.events/event/20/contributions/2552/
[4] https://lore.kernel.org/all/[email protected]
[5] https://lore.kernel.org/all/[email protected]

Carlos López (4):
mm/mprotect: add MM_CP_PERM_UPGRADE
mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE
KVM: plumb MMU_NOTIFIER_RANGE_PERM_UPGRADE into kvm_gfn_range
KVM: x86/mmu: don't zap the secondary MMU on permission upgrade

 arch/x86/kvm/mmu/mmu.c       |  7 +++++++
 include/linux/kvm_host.h     |  1 +
 include/linux/mm.h           |  8 ++++++++
 include/linux/mmu_notifier.h |  5 +++++
 mm/hugetlb.c                 |  6 +++++-
 mm/mprotect.c                | 20 +++++++++++++++++++-
 virt/kvm/kvm_main.c          |  4 ++++
 7 files changed, 49 insertions(+), 2 deletions(-)
#13 lore raw

On Fri, 30 Oct 2026 12:11:07 +0100
Carlos López <[email protected]> wrote:

[...]

Quoted 21 lines
Quoted 4 lines

regarding this series:

on x86 you don't care what the new permissions are, because you're updating when re-faulting (if I understood correctly).

Correct.

Quoted 6 lines

for s390 it might be beneficial to know what the new permissions are; of course we could look up the vma and find out, but since you are already checking the permissions, maybe you can find a way to also convey the new permission bits?

that would make our life on s390 easier :)

Right, getting the new bits from vma->flags should be trivially done from the MMU_NOTIFY_PROTECTION_VMA callsites. There'd be no users in this series though. The rest of architectures still need the notifier to tell them if the change is an upgrade, since they don't have an easy way to grab the old RWX bits. Passing the old RWX bits from the MMU notifier does not seem trivial to me either.

Quoted 8 lines

So what I can do is to plumb down the event and flags into KVM, and leave it open for you to come back and forward the permission bits you need via the range flags, if the MM maintainers are okay with it.

However, I tried to understand the s390 code better with AI, and it seems to think that skipping the zap is also safe there (and an improvement). It could be further optimized by passing down the new W bit to eagerly fix the gmap when restoring write permissions.

right, the fast path would fail (gmap_try_fixup_minor()), and the slow path will call __kvm_faultin_pfn(), which will update the flags as needed.

knowing whether the new mapping is writable would allow us to set the w softbit (which is a fast operation), thus allowing the fast path to handle the fault afterwards, instead of the slow path.

but even just avoiding the zap would surely be beneficial already

I can try to test this next week and see if anything explodes

Quoted 64 lines
Quoted 63 lines
Quoted 62 lines

Benchmarks
==========

AMD EPYC 9654 96-Core Processor host running a recent Linux kernel (6b4b543fa187), with a 64GB, 64-vCPU guest running stress-ng --vm while the host takes a QEMU background snapshot. All results are aggregated over 5 runs, and compared between the stock kernel and one with this patch series. Guest is rebooted across each individual run, and NUMA balancing is disabled.

pf/s and tlb/s (per-second deltas of KVM's pf_taken and remote_tlb_flush debugfs stats) clearly improve:

guest pattern baseline patched
-----------------------------------------------------------------------

  read64   (read)        pf/s=950,015   tlb/s=1,440  pf/s=2,503 tlb/s=0.1
  rand-set (read+write)  pf/s=1,004,353 tlb/s=862    pf/s=4,073 tlb/s=0.3
  write64  (write)       pf/s=968,018   tlb/s=94     pf/s=2,292 tlb/s=0.3

Guest read latency was sampled with a paced probe thread in the guest, reading one byte per page with an idle gap between reads, while stress-ng ran with read64. Each figure is the median across 5 runs, with the run-to-run min and max, in ns:

metric baseline patched
------------------------------------------------------
p50 (min max) 340 (270 380) 260 (250 260)

  p99 (min max)    19789 (16885 20200)  5207 (4076 5638)
  p99.9 (min max)  23445 (22845 24036)  6850 (6259 7110)

The median improves modestly and the refault cost shows in the tail, where p99 drops ~3.8x.

A separate stress-ng run with write64 and --verify showed no data corruption and no livelocks.

References
==========

[1] https://lore.kernel.org/linux-mm/cover.1eca029b8603ef4eebe5b41eae51facfc5920c41.1690292440.git-series.apopple@nvidia.com/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lpc.events/event/20/contributions/2552/
[4] https://lore.kernel.org/all/[email protected]
[5] https://lore.kernel.org/all/[email protected]

Carlos López (4):
mm/mprotect: add MM_CP_PERM_UPGRADE
mm/mmu_notifier: add MMU_NOTIFIER_RANGE_PERM_UPGRADE
KVM: plumb MMU_NOTIFIER_RANGE_PERM_UPGRADE into kvm_gfn_range
KVM: x86/mmu: don't zap the secondary MMU on permission upgrade

 arch/x86/kvm/mmu/mmu.c       |  7 +++++++
 include/linux/kvm_host.h     |  1 +
 include/linux/mm.h           |  8 ++++++++
 include/linux/mmu_notifier.h |  5 +++++
 mm/hugetlb.c                 |  6 +++++-
 mm/mprotect.c                | 20 +++++++++++++++++++-
 virt/kvm/kvm_main.c          |  4 ++++
 7 files changed, 49 insertions(+), 2 deletions(-)
#14 lore raw
Quoted 25 lines · Claudio Imbrenda

On Fri, 30 Oct 2026 12:11:07 +0100
Carlos López <[email protected]> wrote:

[...]

Quoted 20 lines
Quoted 4 lines

regarding this series:

on x86 you don't care what the new permissions are, because you're updating when re-faulting (if I understood correctly).

Correct.

Quoted 6 lines

for s390 it might be beneficial to know what the new permissions are; of course we could look up the vma and find out, but since you are already checking the permissions, maybe you can find a way to also convey the new permission bits?

that would make our life on s390 easier :)

Right, getting the new bits from vma->flags should be trivially done from the MMU_NOTIFY_PROTECTION_VMA callsites. There'd be no users in this series though. The rest of architectures still need the notifier to tell them if the change is an upgrade, since they don't have an easy way to grab the old RWX bits. Passing the old RWX bits from the MMU notifier does not seem trivial to me either.

But why is core MM even doing mmu_notifier_invalidate_range_{start,end}() in the first place? I.e. why add MMU_NOTIFIER_RANGE_PERM_UPGRADE? Just skip the "invalidation".

#15 lore raw
Quoted 27 lines · Sean Christopherson
Quoted 25 lines · Claudio Imbrenda

On Fri, 30 Oct 2026 12:11:07 +0100
Carlos López <[email protected]> wrote:

[...]

Quoted 20 lines
Quoted 4 lines

regarding this series:

on x86 you don't care what the new permissions are, because you're updating when re-faulting (if I understood correctly).

Correct.

Quoted 6 lines

for s390 it might be beneficial to know what the new permissions are; of course we could look up the vma and find out, but since you are already checking the permissions, maybe you can find a way to also convey the new permission bits?

that would make our life on s390 easier :)

Right, getting the new bits from vma->flags should be trivially done from the MMU_NOTIFY_PROTECTION_VMA callsites. There'd be no users in this series though. The rest of architectures still need the notifier to tell them if the change is an upgrade, since they don't have an easy way to grab the old RWX bits. Passing the old RWX bits from the MMU notifier does not seem trivial to me either.

Quoted 3 lines

But why is core MM even doing mmu_notifier_invalidate_range_{start,end}() in the first place? I.e. why add MMU_NOTIFIER_RANGE_PERM_UPGRADE? Just skip the "invalidation".

I briefly mention this in the cover letter, but in short, denying other users the notification seems like an overly intrusive change for KVM's benefit. A flag allows users to opt in instead.

It is also not clear to me that secondary MMUs are not relying on the notification in some subtle way. In fact, Sashiko points out in reply to patch 4 that the notification lets KVM retry the APIC access page reload in case userspace shoots itself in the foot by calling mprotect(PROT_READ) on it. This would require a really weird VMM, as it would need to go looking for the page's address, but breaking this would be a "regression" nonetheless. I will address this in v2.

#16 lore raw
Quoted 28 lines · Carlos López
Quoted 27 lines · Sean Christopherson
Quoted 25 lines · Claudio Imbrenda

On Fri, 30 Oct 2026 12:11:07 +0100
Carlos López <[email protected]> wrote:

[...]

Quoted 20 lines
Quoted 4 lines

regarding this series:

on x86 you don't care what the new permissions are, because you're updating when re-faulting (if I understood correctly).

Correct.

Quoted 6 lines

for s390 it might be beneficial to know what the new permissions are; of course we could look up the vma and find out, but since you are already checking the permissions, maybe you can find a way to also convey the new permission bits?

that would make our life on s390 easier :)

Right, getting the new bits from vma->flags should be trivially done from the MMU_NOTIFY_PROTECTION_VMA callsites. There'd be no users in this series though. The rest of architectures still need the notifier to tell them if the change is an upgrade, since they don't have an easy way to grab the old RWX bits. Passing the old RWX bits from the MMU notifier does not seem trivial to me either.

Quoted 7 lines
Quoted 3 lines

But why is core MM even doing mmu_notifier_invalidate_range_{start,end}() in the first place? I.e. why add MMU_NOTIFIER_RANGE_PERM_UPGRADE? Just skip the "invalidation".

I briefly mention this in the cover letter, but in short, denying other users the notification seems like an overly intrusive change for KVM's benefit.

There are all of two other users:

  $ git grep -w "\.invalidate_range_start"
  drivers/gpu/drm/nouveau/nouveau_svm.c:  .invalidate_range_start = nouveau_svmm_invalidate_range_start,
  drivers/infiniband/hw/hfi1/mmu_rb.c:    .invalidate_range_start = mmu_notifier_range_start,
  virt/kvm/kvm_main.c:    .invalidate_range_start = kvm_mmu_notifier_invalidate_range_start,

and AFAICT neither of them wants an invalidation even for protection upgrades either.

Quoted 7 lines

A flag allows users to opt in instead.

It is also not clear to me that secondary MMUs are not relying on the notification in some subtle way. In fact, Sashiko points out in reply to patch 4 that the notification lets KVM retry the APIC access page reload in case userspace shoots itself in the foot by calling mprotect(PROT_READ) on it. This would require a really weird VMM,

s/weird/misbehaving

Quoted 2 lines

as it would need to go looking for the page's address, but breaking this would be a "regression" nonetheless. I will address this in v2.

Nah, there's zero chance any real VMM is doing mprotect() on the APIC access page.

More importantly, doing nothing if updating the mapping fails and APICv is enabled is wrong. KVM needs to ensure vmcs.APIC_ACCESS_ADDR is fresh when running a vCPU with APICv enabled, even if that means putting the vCPU into an infinite loop. VMX doesn't support "inhibiting" the APIC access page, i.e. the memslot can't disappear while the VM is alive, and if userspace mucks with the VMA, then KVM needs to prevent running the vCPU until the mapping is restored, irrespective of mmu_notifiers.

arch/x86/kvm/vmx/vmx.c +6 -2
index 464d6ed862d5..11b24a42e482 100644
@@ -6928,8 +6928,10 @@ void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu)
 	 * APICv is enabled, but paranoia won't hurt in this case.
 	 */
 	slot = id_to_memslot(slots, APIC_ACCESS_PAGE_PRIVATE_MEMSLOT);
-	if (!slot || slot->flags & KVM_MEMSLOT_INVALID)
+	if (WARN_ON_ONCE(!slot || slot->flags & KVM_MEMSLOT_INVALID)) {
+		kvm_make_request(KVM_REQ_APIC_PAGE_RELOAD, vcpu);
 		return;
+	}
 
 	/*
 	 * Ensure that the mmu_notifier sequence count is read before KVM
@@ -6946,8 +6948,10 @@ void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu)
 	 * if APICv is permanently inhibited, i.e. the memslot won't reappear.
 	 */
 	pfn = __kvm_faultin_pfn(slot, gfn, FOLL_WRITE, &writable, &refcounted_page);
-	if (is_error_noslot_pfn(pfn))
+	if (is_error_noslot_pfn(pfn)) {
+		kvm_make_request(KVM_REQ_APIC_PAGE_RELOAD, vcpu);
 		return;
+	}
 
 	read_lock(&vcpu->kvm->mmu_lock);
 	if (mmu_invalidate_retry_gfn(kvm, mmu_seq, gfn))
#17 lore raw
Quoted 39 lines · Sean Christopherson
Quoted 28 lines · Carlos López
Quoted 27 lines · Sean Christopherson
Quoted 25 lines · Claudio Imbrenda

On Fri, 30 Oct 2026 12:11:07 +0100
Carlos López <[email protected]> wrote:

[...]

Quoted 20 lines
Quoted 4 lines

regarding this series:

on x86 you don't care what the new permissions are, because you're updating when re-faulting (if I understood correctly).

Correct.

Quoted 6 lines

for s390 it might be beneficial to know what the new permissions are; of course we could look up the vma and find out, but since you are already checking the permissions, maybe you can find a way to also convey the new permission bits?

that would make our life on s390 easier :)

Right, getting the new bits from vma->flags should be trivially done from the MMU_NOTIFY_PROTECTION_VMA callsites. There'd be no users in this series though. The rest of architectures still need the notifier to tell them if the change is an upgrade, since they don't have an easy way to grab the old RWX bits. Passing the old RWX bits from the MMU notifier does not seem trivial to me either.

Quoted 7 lines
Quoted 3 lines

But why is core MM even doing mmu_notifier_invalidate_range_{start,end}() in the first place? I.e. why add MMU_NOTIFIER_RANGE_PERM_UPGRADE? Just skip the "invalidation".

I briefly mention this in the cover letter, but in short, denying other users the notification seems like an overly intrusive change for KVM's benefit.

There are all of two other users:

Quoted 7 lines
  $ git grep -w "\.invalidate_range_start"
  drivers/gpu/drm/nouveau/nouveau_svm.c:  .invalidate_range_start = nouveau_svmm_invalidate_range_start,
  drivers/infiniband/hw/hfi1/mmu_rb.c:    .invalidate_range_start = mmu_notifier_range_start,
  virt/kvm/kvm_main.c:    .invalidate_range_start = kvm_mmu_notifier_invalidate_range_start,

and AFAICT neither of them wants an invalidation even for protection upgrades either.

There are more users that register for notifications via mmu_interval_notifier_insert():

  $ git grep -l -e "mmu_interval_notifier_insert(" \
          -e "mmu_interval_notifier_insert_locked(" -- drivers/ | wc -l
  12

I haven't audited all of them, but I'd say that at least amdkfd relies on the invalidation for upgrades. Without XNACK, GPU page faults are not retried, so the driver maps ranges up front with the VMA permissions (svm_range_validate_and_map()) and relies on an invalidation to remap them when those change. So mprotect(PROT_READ) followed by mprotect(PROT_READ|PROT_WRITE) would cause the GPU mapping to stay read-only and the next GPU write to fault. Unlike the APIC access page, this is the application's own memory, mprotect() should be perfectly valid.

This is the kind of subtle reliance that I mentioned before, and why I'd rather not open that can of worms.

Quoted 15 lines
Quoted 7 lines

A flag allows users to opt in instead.

It is also not clear to me that secondary MMUs are not relying on the notification in some subtle way. In fact, Sashiko points out in reply to patch 4 that the notification lets KVM retry the APIC access page reload in case userspace shoots itself in the foot by calling mprotect(PROT_READ) on it. This would require a really weird VMM,

s/weird/misbehaving

Quoted 2 lines

as it would need to go looking for the page's address, but breaking this would be a "regression" nonetheless. I will address this in v2.

Nah, there's zero chance any real VMM is doing mprotect() on the APIC access page.

Quoted 7 lines

More importantly, doing nothing if updating the mapping fails and APICv is enabled is wrong. KVM needs to ensure vmcs.APIC_ACCESS_ADDR is fresh when running a vCPU with APICv enabled, even if that means putting the vCPU into an infinite loop. VMX doesn't support "inhibiting" the APIC access page, i.e. the memslot can't disappear while the VM is alive, and if userspace mucks with the VMA, then KVM needs to prevent running the vCPU until the mapping is restored, irrespective of mmu_notifiers.

Fair point. Should I just carry the diff below in v2 with your Signed-off-by?

Quoted 28 lines
arch/x86/kvm/vmx/vmx.c +6 -2
index 464d6ed862d5..11b24a42e482 100644
@@ -6928,8 +6928,10 @@ void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu)
 	 * APICv is enabled, but paranoia won't hurt in this case.
 	 */
 	slot = id_to_memslot(slots, APIC_ACCESS_PAGE_PRIVATE_MEMSLOT);
-	if (!slot || slot->flags & KVM_MEMSLOT_INVALID)
+	if (WARN_ON_ONCE(!slot || slot->flags & KVM_MEMSLOT_INVALID)) {
+		kvm_make_request(KVM_REQ_APIC_PAGE_RELOAD, vcpu);
 		return;
+	}
 
 	/*
 	 * Ensure that the mmu_notifier sequence count is read before KVM
@@ -6946,8 +6948,10 @@ void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu)
 	 * if APICv is permanently inhibited, i.e. the memslot won't reappear.
 	 */
 	pfn = __kvm_faultin_pfn(slot, gfn, FOLL_WRITE, &writable, &refcounted_page);
-	if (is_error_noslot_pfn(pfn))
+	if (is_error_noslot_pfn(pfn)) {
+		kvm_make_request(KVM_REQ_APIC_PAGE_RELOAD, vcpu);
 		return;
+	}
 
 	read_lock(&vcpu->kvm->mmu_lock);
 	if (mmu_invalidate_retry_gfn(kvm, mmu_seq, gfn))
#18 lore raw
Quoted 45 lines · Carlos López
Quoted 44 lines · Sean Christopherson
Quoted 28 lines · Carlos López
Quoted 27 lines · Sean Christopherson
Quoted 25 lines · Claudio Imbrenda

On Fri, 30 Oct 2026 12:11:07 +0100
Carlos López <[email protected]> wrote:

[...]

Quoted 20 lines
Quoted 4 lines

regarding this series:

on x86 you don't care what the new permissions are, because you're updating when re-faulting (if I understood correctly).

Correct.

Quoted 6 lines

for s390 it might be beneficial to know what the new permissions are; of course we could look up the vma and find out, but since you are already checking the permissions, maybe you can find a way to also convey the new permission bits?

that would make our life on s390 easier :)

Right, getting the new bits from vma->flags should be trivially done from the MMU_NOTIFY_PROTECTION_VMA callsites. There'd be no users in this series though. The rest of architectures still need the notifier to tell them if the change is an upgrade, since they don't have an easy way to grab the old RWX bits. Passing the old RWX bits from the MMU notifier does not seem trivial to me either.

Quoted 7 lines
Quoted 3 lines

But why is core MM even doing mmu_notifier_invalidate_range_{start,end}() in the first place? I.e. why add MMU_NOTIFIER_RANGE_PERM_UPGRADE? Just skip the "invalidation".

I briefly mention this in the cover letter, but in short, denying other users the notification seems like an overly intrusive change for KVM's benefit.

There are all of two other users:

  $ git grep -w "\.invalidate_range_start"
  drivers/gpu/drm/nouveau/nouveau_svm.c:  .invalidate_range_start = nouveau_svmm_invalidate_range_start,
  drivers/infiniband/hw/hfi1/mmu_rb.c:    .invalidate_range_start = mmu_notifier_range_start,
  virt/kvm/kvm_main.c:    .invalidate_range_start = kvm_mmu_notifier_invalidate_range_start,
Quoted 4 lines
Quoted 2 lines

and AFAICT neither of them wants an invalidation even for protection upgrades either.

There are more users that register for notifications via

Ugh, I didn't realize we gained a variant of .invalidate_range_{start,end}(), I thought those were for what is now arch_invalidate_secondary_tlbs().

Quoted 15 lines

mmu_interval_notifier_insert():

  $ git grep -l -e "mmu_interval_notifier_insert(" \
          -e "mmu_interval_notifier_insert_locked(" -- drivers/ | wc -l
  12

I haven't audited all of them, but I'd say that at least amdkfd relies on the invalidation for upgrades. Without XNACK, GPU page faults are not retried, so the driver maps ranges up front with the VMA permissions (svm_range_validate_and_map()) and relies on an invalidation to remap them when those change. So mprotect(PROT_READ) followed by mprotect(PROT_READ|PROT_WRITE) would cause the GPU mapping to stay read-only and the next GPU write to fault. Unlike the APIC access page, this is the application's own memory, mprotect() should be perfectly valid.

Quoted 2 lines

This is the kind of subtle reliance that I mentioned before, and why I'd rather not open that can of worms.

Yeah, agreed.

Quoted 23 lines
Quoted 15 lines
Quoted 7 lines

A flag allows users to opt in instead.

It is also not clear to me that secondary MMUs are not relying on the notification in some subtle way. In fact, Sashiko points out in reply to patch 4 that the notification lets KVM retry the APIC access page reload in case userspace shoots itself in the foot by calling mprotect(PROT_READ) on it. This would require a really weird VMM,

s/weird/misbehaving

Quoted 2 lines

as it would need to go looking for the page's address, but breaking this would be a "regression" nonetheless. I will address this in v2.

Nah, there's zero chance any real VMM is doing mprotect() on the APIC access page.

Quoted 7 lines

More importantly, doing nothing if updating the mapping fails and APICv is enabled is wrong. KVM needs to ensure vmcs.APIC_ACCESS_ADDR is fresh when running a vCPU with APICv enabled, even if that means putting the vCPU into an infinite loop. VMX doesn't support "inhibiting" the APIC access page, i.e. the memslot can't disappear while the VM is alive, and if userspace mucks with the VMA, then KVM needs to prevent running the vCPU until the mapping is restored, irrespective of mmu_notifiers.

Quoted 2 lines

Fair point. Should I just carry the diff below in v2 with your Signed-off-by?

Nah, I'll send a standalone patch. It doesn't have a direct relationship to your series, and the bug is pre-existing despite what Sashiko thinks.

Thanks!