KVM: x86/mmu: Fix pre-fault and map private loops
Sean Christopherson
maintainercover KVM: x86/mmu: Fix pre-fault and map private loops
Fix a bug in the pre-fault path where KVM fails to reload an invalidated MMU root, which puts the KVM_PRE_FAULT_MEMORY task into an infinite loop (although it's breakable, so not fatal to the host). My best guess is that the test started failing once PREEMPT_LAZY was enabled by default.
Then harden the similar "map private PFN" to also guard against unexpected root invalidations, because Sashiko keeps pointing out that it's theoretically possible for that code to end up in the same type of infinite loop.
Sean Christopherson (4):
KVM: x86/mmu: Reload MMU on *every* page pre-fault attempt/iteration
KVM: x86/mmu: Harden "map private PFN" against unexpected root
invalidation
KVM: x86/mmu: Top-up memory caches when retrying "map private PFN"
KVM: x86/mmu: Add sanity check to detect stale page faults in "map
private PFN"
arch/x86/kvm/mmu/mmu.c | 45 +++++++++++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 16 deletions(-)
base-commit: a806d364ef288a6443a1337820ea8410a7ccc6b3
Signature · 1 line
2.55.0.679.g6767b8d81c-goog
Sean Christopherson
maintainerPATCH 1/4 KVM: x86/mmu: Reload MMU on *every* page pre-fault attempt/iteration
Reload the MMU (which is a nop if the MMU doesn't need to be reloaded) on every attempt to pre-fault a guest page, i.e. when the page fault path signals that the caller should retry. If the synchronize_srcu_expedited() in kvm_invalidate_memslot() completes before kvm_vcpu_pre_fault_memory() grabs SRCU, but kvm_mmu_reload() in the pre-fault path completes before kvm_invalidate_memslot() triggers x86's "fast zap all", then the pre-fault task will reach kvm_tdp_page_prefault() with an invalid root.
Attempting to fault-in memory with an invalid root ultimately puts kvm_tdp_page_prefault() into an infinite (breakable) retry loop, which manifests most obviously as a hang in the pre_fault_memory_test selftest, but also eventually causes RCU (SRCU?) to complain.
INFO: rcu_tasks detected stalls on tasks: 000000000cda47bd: .. nvcsw: 6/6 holdout: 1 idle_cpu: -1/25 task:pre_fault_memor state:R running task stack:12696 pid:95588 tgid:95588 ppid:95584 task_flags:0x400000 flags:0x00080801 Call Trace: <TASK> lock_release+0x4e/0x320 __get_user_pages+0x546/0xcd0 up_read+0x1b/0x30 get_user_pages_unlocked+0xee/0x350 hva_to_pfn+0xd3/0x3d0 [kvm] lock_release+0x4e/0x320 xa_load+0x5c/0x170 xa_load+0x14c/0x170 __kvm_faultin_pfn+0xd9/0x130 [kvm] lock_acquire+0x65/0x2b0 lock_release+0x4e/0x320 kvm_mmu_faultin_pfn+0x1e1/0x690 [kvm] gup_fast_fallback+0x63e/0xdf0 kvm_tdp_page_fault+0xeb/0x140 [kvm] kvm_mmu_do_page_fault+0x12e/0x200 [kvm] kvm_arch_vcpu_pre_fault_memory+0x16e/0x200 [kvm] kvm_vcpu_pre_fault_memory+0xc1/0x1f0 [kvm] kvm_vcpu_pre_fault_memory+0x116/0x1f0 [kvm] kvm_vcpu_ioctl+0x3a4/0x6b0 [kvm] clockevents_program_event+0x5d/0x170 __se_sys_ioctl+0x6d/0xb0 entry_SYSCALL_64_after_hwframe+0x4b/0x53 do_syscall_64+0x10a/0x480 __irq_exit_rcu+0x8e/0x140 entry_SYSCALL_64_after_hwframe+0x4b/0x53 </TASK>
--- arch/x86/kvm/mmu/mmu.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)
arch/x86/kvm/mmu/mmu.c +4 -8
@@ -5036,6 +5036,10 @@ static int kvm_tdp_page_prefault(struct kvm_vcpu *vcpu, gpa_t gpa,if (kvm_check_request (KVM_REQ_VM_DEAD , vcpu))return -EIO ; + r =kvm_mmu_reload (vcpu); +if (r) +return r; +cond_resched (); r =kvm_mmu_do_page_fault (vcpu, gpa, error_code, true,NULL , level); }while (r ==RET_PF_RETRY );
@@ -5076,14 +5080,6 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,if (kvm_is_gfn_alias (vcpu->kvm,gpa_to_gfn (range->gpa)))return -EINVAL ; -/* -* reload is efficient when called repeatedly, so we can do it on -* every iteration. -*/ - r =kvm_mmu_reload (vcpu); -if (r) -return r; - direct_bits =0 ;if (kvm_arch_has_private_mem (vcpu->kvm) &&kvm_mem_is_private (vcpu->kvm,gpa_to_gfn (range->gpa)))
Signature · 1 line
2.55.0.679.g6767b8d81c-goog
Re: [PATCH 1/4] KVM: x86/mmu: Reload MMU on *every* page pre-fault attempt/iteration
Quoted 46 lines · Sean Christopherson
Reload the MMU (which is a nop if the MMU doesn't need to be reloaded) on every attempt to pre-fault a guest page, i.e. when the page fault path signals that the caller should retry. If the
synchronize_srcu_expedited()inkvm_invalidate_memslot()completes beforekvm_vcpu_pre_fault_memory()grabs SRCU, butkvm_mmu_reload()in the pre-fault path completes beforekvm_invalidate_memslot()triggers x86's "fast zap all", then the pre-fault task will reachkvm_tdp_page_prefault()with an invalid root.Attempting to fault-in memory with an invalid root ultimately puts
kvm_tdp_page_prefault()into an infinite (breakable) retry loop, which manifests most obviously as a hang in thepre_fault_memory_testselftest, but also eventually causes RCU (SRCU?) to complain.INFO: rcu_tasks detected stalls on tasks: 000000000cda47bd: .. nvcsw: 6/6 holdout: 1 idle_cpu: -1/25 task:pre_fault_memor state:R running task stack:12696 pid:95588 tgid:95588 ppid:95584 task_flags:0x400000 flags:0x00080801 Call Trace: <TASK> lock_release+0x4e/0x320 __get_user_pages+0x546/0xcd0 up_read+0x1b/0x30 get_user_pages_unlocked+0xee/0x350 hva_to_pfn+0xd3/0x3d0 [kvm] lock_release+0x4e/0x320 xa_load+0x5c/0x170 xa_load+0x14c/0x170 __kvm_faultin_pfn+0xd9/0x130 [kvm] lock_acquire+0x65/0x2b0 lock_release+0x4e/0x320 kvm_mmu_faultin_pfn+0x1e1/0x690 [kvm] gup_fast_fallback+0x63e/0xdf0 kvm_tdp_page_fault+0xeb/0x140 [kvm] kvm_mmu_do_page_fault+0x12e/0x200 [kvm] kvm_arch_vcpu_pre_fault_memory+0x16e/0x200 [kvm] kvm_vcpu_pre_fault_memory+0xc1/0x1f0 [kvm] kvm_vcpu_pre_fault_memory+0x116/0x1f0 [kvm] kvm_vcpu_ioctl+0x3a4/0x6b0 [kvm] clockevents_program_event+0x5d/0x170 __se_sys_ioctl+0x6d/0xb0 entry_SYSCALL_64_after_hwframe+0x4b/0x53 do_syscall_64+0x10a/0x480 __irq_exit_rcu+0x8e/0x140 entry_SYSCALL_64_after_hwframe+0x4b/0x53 </TASK>
Quoted 3 lines
Fixes 6e01b7601dfe ("KVM: x86: Implement kvm_arch_vcpu_pre_fault_memory()")Signed-off-by Sean Christopherson <[email protected]>
Re: [PATCH 1/4] KVM: x86/mmu: Reload MMU on *every* page pre-fault attempt/iteration
In reply to Sean ChristophersonQuoted 46 lines · Sean Christopherson
Reload the MMU (which is a nop if the MMU doesn't need to be reloaded) on every attempt to pre-fault a guest page, i.e. when the page fault path signals that the caller should retry. If the
synchronize_srcu_expedited()inkvm_invalidate_memslot()completes beforekvm_vcpu_pre_fault_memory()grabs SRCU, butkvm_mmu_reload()in the pre-fault path completes beforekvm_invalidate_memslot()triggers x86's "fast zap all", then the pre-fault task will reachkvm_tdp_page_prefault()with an invalid root.Attempting to fault-in memory with an invalid root ultimately puts
kvm_tdp_page_prefault()into an infinite (breakable) retry loop, which manifests most obviously as a hang in thepre_fault_memory_testselftest, but also eventually causes RCU (SRCU?) to complain.INFO: rcu_tasks detected stalls on tasks: 000000000cda47bd: .. nvcsw: 6/6 holdout: 1 idle_cpu: -1/25 task:pre_fault_memor state:R running task stack:12696 pid:95588 tgid:95588 ppid:95584 task_flags:0x400000 flags:0x00080801 Call Trace: <TASK> lock_release+0x4e/0x320 __get_user_pages+0x546/0xcd0 up_read+0x1b/0x30 get_user_pages_unlocked+0xee/0x350 hva_to_pfn+0xd3/0x3d0 [kvm] lock_release+0x4e/0x320 xa_load+0x5c/0x170 xa_load+0x14c/0x170 __kvm_faultin_pfn+0xd9/0x130 [kvm] lock_acquire+0x65/0x2b0 lock_release+0x4e/0x320 kvm_mmu_faultin_pfn+0x1e1/0x690 [kvm] gup_fast_fallback+0x63e/0xdf0 kvm_tdp_page_fault+0xeb/0x140 [kvm] kvm_mmu_do_page_fault+0x12e/0x200 [kvm] kvm_arch_vcpu_pre_fault_memory+0x16e/0x200 [kvm] kvm_vcpu_pre_fault_memory+0xc1/0x1f0 [kvm] kvm_vcpu_pre_fault_memory+0x116/0x1f0 [kvm] kvm_vcpu_ioctl+0x3a4/0x6b0 [kvm] clockevents_program_event+0x5d/0x170 __se_sys_ioctl+0x6d/0xb0 entry_SYSCALL_64_after_hwframe+0x4b/0x53 do_syscall_64+0x10a/0x480 __irq_exit_rcu+0x8e/0x140 entry_SYSCALL_64_after_hwframe+0x4b/0x53 </TASK>
Quoted 2 lines
Fixes 6e01b7601dfe ("KVM: x86: Implement kvm_arch_vcpu_pre_fault_memory()")Signed-off-by Sean Christopherson <[email protected]>
But, did you hit this somehow with an unmodified test and kernel? The log makes me think it was hit in a normal test run. (which I failed to reproduce)
Sean Christopherson
maintainerRe: [PATCH 1/4] KVM: x86/mmu: Reload MMU on *every* page pre-fault attempt/iteration
Quoted 47 lines · Rick P Edgecombe
Quoted 46 lines · Sean Christopherson
Reload the MMU (which is a nop if the MMU doesn't need to be reloaded) on every attempt to pre-fault a guest page, i.e. when the page fault path signals that the caller should retry. If the
synchronize_srcu_expedited()inkvm_invalidate_memslot()completes beforekvm_vcpu_pre_fault_memory()grabs SRCU, butkvm_mmu_reload()in the pre-fault path completes beforekvm_invalidate_memslot()triggers x86's "fast zap all", then the pre-fault task will reachkvm_tdp_page_prefault()with an invalid root.Attempting to fault-in memory with an invalid root ultimately puts
kvm_tdp_page_prefault()into an infinite (breakable) retry loop, which manifests most obviously as a hang in thepre_fault_memory_testselftest, but also eventually causes RCU (SRCU?) to complain.INFO: rcu_tasks detected stalls on tasks: 000000000cda47bd: .. nvcsw: 6/6 holdout: 1 idle_cpu: -1/25 task:pre_fault_memor state:R running task stack:12696 pid:95588 tgid:95588 ppid:95584 task_flags:0x400000 flags:0x00080801 Call Trace: <TASK> lock_release+0x4e/0x320 __get_user_pages+0x546/0xcd0 up_read+0x1b/0x30 get_user_pages_unlocked+0xee/0x350 hva_to_pfn+0xd3/0x3d0 [kvm] lock_release+0x4e/0x320 xa_load+0x5c/0x170 xa_load+0x14c/0x170 __kvm_faultin_pfn+0xd9/0x130 [kvm] lock_acquire+0x65/0x2b0 lock_release+0x4e/0x320 kvm_mmu_faultin_pfn+0x1e1/0x690 [kvm] gup_fast_fallback+0x63e/0xdf0 kvm_tdp_page_fault+0xeb/0x140 [kvm] kvm_mmu_do_page_fault+0x12e/0x200 [kvm] kvm_arch_vcpu_pre_fault_memory+0x16e/0x200 [kvm] kvm_vcpu_pre_fault_memory+0xc1/0x1f0 [kvm] kvm_vcpu_pre_fault_memory+0x116/0x1f0 [kvm] kvm_vcpu_ioctl+0x3a4/0x6b0 [kvm] clockevents_program_event+0x5d/0x170 __se_sys_ioctl+0x6d/0xb0 entry_SYSCALL_64_after_hwframe+0x4b/0x53 do_syscall_64+0x10a/0x480 __irq_exit_rcu+0x8e/0x140 entry_SYSCALL_64_after_hwframe+0x4b/0x53 </TASK>
Quoted 8 lines
Quoted 2 lines
Fixes 6e01b7601dfe ("KVM: x86: Implement kvm_arch_vcpu_pre_fault_memory()")Signed-off-by Sean Christopherson <[email protected]>Reviewed-by Rick Edgecombe <[email protected]>But, did you hit this somehow with an unmodified test and kernel? The log makes me think it was hit in a normal test run. (which I failed to reproduce)
Yes and yes. I often test by running all selftests in parallel. My guess is that PREEMPT_LAZY plus oversubscribed CPUs allowed the timing condition to be hit. It wasn't anywhere near 100% reproducible on my end either, maybe 1 out of every 10 runs? 20 runs? (of the entire suite of selftests).
P.S. I was very proud of myself: because the failure was so flaky, I used perf and bpftrace to debug the issue instead of my usual hack-the-kernel approach. :-)
Sean Christopherson
maintainerPATCH 2/4 KVM: x86/mmu: Harden "map private PFN" against unexpected root invalidation
In reply to Sean ChristophersonMove kvm_tdp_mmu_map_private_pfn()'s reload of the MMU into its tight loop so that an unexpected root invalidation has a better chance of being handled gracefully, even though it should be impossible for the vCPU's root to be invalidated after the initial reload. As is, encountering an invalid root is *guaranteed* to put the task into an infinite loop (albeit a breakable loop that honors NEED_RESCHED).
Add a WARN to try and detect bugs that break KVM's expectations, along with a comment to explain why it should be impossible for the root to be invalidated.
--- arch/x86/kvm/mmu/mmu.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
arch/x86/kvm/mmu/mmu.c +11 -4
@@ -5184,10 +5184,6 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)if (kvm_gfn_is_write_tracked (kvm, fault.slot, fault.gfn))return -EPERM ; - r =kvm_mmu_reload (vcpu); -if (r) -return r; - r =mmu_topup_memory_caches (vcpu, false);if (r)return r;
@@ -5199,10 +5195,21 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)if (kvm_test_request (KVM_REQ_VM_DEAD , vcpu))return -EIO ; + r =kvm_mmu_reload (vcpu); +if (r) +return r; +cond_resched ();guard (read_lock)(&kvm->mmu_lock); +/* +* Because slots_lock is held, it should be impossible for roots +* to be invalidated after the initial MMU reload. WARN, but +* continue and re-reload the MMU to try and keep the VM alive. +*/ +WARN_ON_ONCE (kvm_test_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu)); + r =kvm_tdp_mmu_map (vcpu, &fault); }while (r ==RET_PF_RETRY );
Signature · 1 line
2.55.0.679.g6767b8d81c-goog
[email protected]
AI reviewRe: [PATCH 2/4] KVM: x86/mmu: Harden "map private PFN" against unexpected root invalidation
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
New issues:
- [High] Moving kvm_mmu_reload() inside the retry loop after mmu_topup_memory_caches() causes deterministic memory cache underflows on the first page fault if roots are reloaded.
- [Medium] Failure to re-reload the MMU upon detecting an obsolete root.
Pre-existing issues:
- [High] Memory cache depletion during the RET_PF_RETRY loop due to missing mmu_topup_memory_caches() inside the loop. --
--- Patch [2]: [PATCH 2/4] KVM: x86/mmu: Harden "map private PFN" against unexpected root invalidation --- commit 22d402057a7ea26b753baa57f8845602fe5a1b81
Author: Sean Christopherson <[email protected]>
KVM: x86/mmu: Harden "map private PFN" against unexpected root invalidation
This commit moves the MMU reload inside the tight loop in kvm_tdp_mmu_map_private_pfn() to better handle unexpected root invalidations. It also adds a warning if the root is invalidated after the initial reload.
Quoted 4 lines
arch/x86/kvm/mmu/mmu.c +0
-0
[ ... ]
arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5189
Quoted 11 lines
arch/x86/kvm/mmu/mmu.c +0
-4@@ -5184,10 +5184,6 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)if (kvm_gfn_is_write_tracked (kvm, fault.slot, fault.gfn))return -EPERM ; - r =kvm_mmu_reload (vcpu); -if (r) -return r; - r =mmu_topup_memory_caches (vcpu, false);if (r)return r;
[Severity: High]
Since mmu_topup_memory_caches() is outside the loop, but kvm_mmu_reload() was moved inside, is it possible for the reload to consume cache items when allocating new roots?
If the root is invalid and kvm_mmu_load() allocates new roots, this consumes items from mmu_page_header_cache and mmu_shadow_page_cache. When execution proceeds to kvm_tdp_mmu_map() expecting fully topped up caches, could this lead to a cache underflow and a host warning or panic?
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +4
-0@@ -5199,10 +5195,21 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)if (kvm_test_request (KVM_REQ_VM_DEAD , vcpu))return -EIO ; + r =kvm_mmu_reload (vcpu); +if (r) +return r; +cond_resched ();
arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5212
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +7
-0guard (read_lock)(&kvm->mmu_lock); +/* +* Because slots_lock is held, it should be impossible for roots +* to be invalidated after the initial MMU reload. WARN, but +* continue and re-reload the MMU to try and keep the VM alive. +*/ +WARN_ON_ONCE (kvm_test_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu)); +
[Severity: Medium]
The comment mentions "continue and re-reload the MMU", but does this actually fall through directly to kvm_tdp_mmu_map() with an obsolete root?
If the mapping succeeds, it might return RET_PF_FIXED. This would exit the loop instead of forcing a retry. Should there be a continue statement after the WARN_ON_ONCE to enforce the retry behavior described in the comment?
arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5215
Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +0
-0r =kvm_tdp_mmu_map (vcpu, &fault); }while (r ==RET_PF_RETRY );
[Severity: High]
This isn't a bug introduced by this patch, but retrying on RET_PF_RETRY in this loop seems vulnerable to memory cache depletion without the cache topup.
If kvm_tdp_mmu_map() encounters contention, frees its locally allocated shadow pages, and returns RET_PF_RETRY, the loop repeats without calling mmu_topup_memory_caches() again. Could this underflow the cache during repeated retries under high contention?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Sean Christopherson
maintainerRe: [PATCH 2/4] KVM: x86/mmu: Harden "map private PFN" against unexpected root invalidation
Quoted 17 lines · On Thu, Aug 06, 2026, [email protected] wrote:
(quoted diff) +0
-0@@ -5184,10 +5184,6 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) > if (kvm_gfn_is_write_tracked(kvm, fault.slot, fault.gfn)) > return -EPERM;arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5189
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c +0
-4- r =kvm_mmu_reload (vcpu); -if (r) -return r; - r =mmu_topup_memory_caches (vcpu, false);if (r)return r;[Severity: High]
Sincemmu_topup_memory_caches()is outside the loop, butkvm_mmu_reload()was moved inside, is it possible for the reload to consume cache items when allocating new roots?
Quoted 4 lines
If the root is invalid and
kvm_mmu_load()allocates new roots, this consumes items frommmu_page_header_cacheandmmu_shadow_page_cache. When execution proceeds tokvm_tdp_mmu_map()expecting fully topped up caches, could this lead to a cache underflow and a host warning or panic?
Literally the next patch...
Quoted 12 lines
Quoted 12 lines
arch/x86/kvm/mmu/mmu.c +4
-0@@ -5199,10 +5195,21 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)if (kvm_test_request (KVM_REQ_VM_DEAD , vcpu))return -EIO ; + r =kvm_mmu_reload (vcpu); +if (r) +return r; +cond_resched ();guard (read_lock)(&kvm->mmu_lock);
Quoted 11 lines
arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5212
Quoted 7 lines
arch/x86/kvm/mmu/mmu.c +7
-0+/* +* Because slots_lock is held, it should be impossible for roots +* to be invalidated after the initial MMU reload. WARN, but +* continue and re-reload the MMU to try and keep the VM alive. +*/ +WARN_ON_ONCE (kvm_test_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu)); +[Severity: Medium]
The comment mentions "continue and re-reload the MMU", but does this actually fall through directly tokvm_tdp_mmu_map()with an obsolete root?
Yes, addressed two patches from now.
Quoted 3 lines
If the mapping succeeds, it might return
RET_PF_FIXED. This would exit the loop instead of forcing a retry. Should there be a continue statement after theWARN_ON_ONCEto enforce the retry behavior described in the comment?
Eh, I'd rather hedge in the changelog.
Re: [PATCH 2/4] KVM: x86/mmu: Harden "map private PFN" against unexpected root invalidation
In reply to Sean ChristophersonQuoted 12 lines · Sean Christopherson
arch/x86/kvm/mmu/mmu.c +0
-0@@ -5199,10 +5195,21 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu)) return -EIO;arch/x86/kvm/mmu/mmu.c +4
-0+ r =kvm_mmu_reload (vcpu); +if (r) +return r; +
cond_resched();
guard(read_lock)(&kvm->mmu_lock);
arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5210
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +5
-0+/* +* Because slots_lock is held, it should be impossible for roots +* to be invalidated after the initial MMU reload. WARN, but +* continue and re-reload the MMU to try and keep the VM alive. +*/
It might be a little confusing when no MMU reload follows the comment. Maybe mention that the MMU reload is above? Either way ok to me.
Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +2
-0+WARN_ON_ONCE (kvm_test_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu)); +r =
kvm_tdp_mmu_map(vcpu, &fault);
Sean Christopherson
maintainerRe: [PATCH 2/4] KVM: x86/mmu: Harden "map private PFN" against unexpected root invalidation
Quoted 13 lines · Rick P Edgecombe
Quoted 12 lines · Sean Christopherson
arch/x86/kvm/mmu/mmu.c +0
-0@@ -5199,10 +5195,21 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu)) return -EIO;arch/x86/kvm/mmu/mmu.c +4
-0+ r =kvm_mmu_reload (vcpu); +if (r) +return r; +
cond_resched();
guard(read_lock)(&kvm->mmu_lock);
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5210
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +5
-0+/* +* Because slots_lock is held, it should be impossible for roots +* to be invalidated after the initial MMU reload. WARN, but +* continue and re-reload the MMU to try and keep the VM alive. +*/ It might be a little confusing when no MMU reload follows the comment. Maybe mention that the MMU reload is above? Either way ok to me.
Yeah, I don't love the comment either. The subtlety that it doesn't capture is that KVM_REQ_MMU_FREE_OBSOLETE_ROOTS doesn't actually mean the current root is invalid, just that it might be invalid and the vCPU needs to check.
How about this?
/*
* Because slots_lock is held, it should be impossible for *any*
* roots to be invalidated after the initial MMU reload. WARN,
* but continue on; the above MMU reload will do the right thing
* if the current root is actually invalid.
*/
Quoted 4 lines
Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +2
-0+WARN_ON_ONCE (kvm_test_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu)); +r =
kvm_tdp_mmu_map(vcpu, &fault);
Re: [PATCH 2/4] KVM: x86/mmu: Harden "map private PFN" against unexpected root invalidation
Quoted 4 lines · Sean Christopherson
Yeah, I don't love the comment either. The subtlety that it doesn't capture is that
KVM_REQ_MMU_FREE_OBSOLETE_ROOTSdoesn't actually mean the current root is invalid, just that it might be invalid and the vCPU needs to check.
Quoted 8 lines
How about this?
/*
* Becauseslots_lockis held, it should be impossible for *any*
* roots to be invalidated after the initial MMU reload. WARN,
* but continue on; the above MMU reload will do the right thing
* if the current root is actually invalid.
*/
Looks good! thanks
Re: [PATCH 2/4] KVM: x86/mmu: Harden "map private PFN" against unexpected root invalidation
In reply to Sean ChristophersonQuoted 6 lines · Sean Christopherson
Move kvm_tdp_mmu_map_private_pfn()'s reload of the MMU into its tight loop so that an unexpected root invalidation has a better chance of being handled gracefully, even though it should be impossible for the vCPU's root to be invalidated after the initial reload. As is, encountering an invalid root is *guaranteed* to put the task into an infinite loop (albeit a breakable loop that honors
NEED_RESCHED).
Note: without the newly added is_page_fault_stale() check in patch 4, an invalid root would not put the task into an infinite loop :)
BTW: As noted in [1], is_page_fault_stale() only checks !mirror roots, and kvm_mmu_reload() reloads mirror roots only when !mirror roots are also invalid, since an invalid mirror root was considered impossible. (up to now, no?)
[1] https://lore.kernel.org/all/[email protected]
Quoted 3 lines
Add a WARN to try and detect bugs that break KVM's expectations, along with a comment to explain why it should be impossible for the root to be invalidated.
And there's already a warning in kvm_tdp_mmu_map():
"KVM_MMU_WARN_ON(!root || root->role.invalid);". So the warning also seems redundant.
Quoted 24 lines
Cc Kai Huang <[email protected]>Cc Yan Zhao <[email protected]>Cc Rick Edgecombe <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)arch/x86/kvm/mmu/mmu.c +0
-4@@ -5184,10 +5184,6 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)if (kvm_gfn_is_write_tracked (kvm, fault.slot, fault.gfn))return -EPERM ; - r =kvm_mmu_reload (vcpu); -if (r) -return r; - r =mmu_topup_memory_caches (vcpu, false);if (r)return r;
arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5201
Quoted 7 lines
arch/x86/kvm/mmu/mmu.c +4
-0@@ -5199,10 +5195,21 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)if (kvm_test_request (KVM_REQ_VM_DEAD , vcpu))return -EIO ; + r =kvm_mmu_reload (vcpu); +if (r) +return r; +
Since kvm_mmu_reload() is moved inside the loop, should we also move kvm_gfn_is_write_tracked() inside the loop to prevent unexpected bugs?
Quoted 17 lines
arch/x86/kvm/mmu/mmu.c +6
-0cond_resched ();guard (read_lock)(&kvm->mmu_lock); +/* +* Because slots_lock is held, it should be impossible for roots +* to be invalidated after the initial MMU reload. WARN, but +* continue and re-reload the MMU to try and keep the VM alive. +*/ +WARN_ON_ONCE (kvm_test_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu)); r =kvm_tdp_mmu_map (vcpu, &fault); }while (r ==RET_PF_RETRY );Signature · 2 lines
-- 2.55.0.679.g6767b8d81c-goog
Sean Christopherson
maintainerRe: [PATCH 2/4] KVM: x86/mmu: Harden "map private PFN" against unexpected root invalidation
Quoted 14 lines · Yan Zhao
Quoted 6 lines · Sean Christopherson
Move kvm_tdp_mmu_map_private_pfn()'s reload of the MMU into its tight loop so that an unexpected root invalidation has a better chance of being handled gracefully, even though it should be impossible for the vCPU's root to be invalidated after the initial reload. As is, encountering an invalid root is *guaranteed* to put the task into an infinite loop (albeit a breakable loop that honors
NEED_RESCHED).Note: without the newly added
is_page_fault_stale()check in patch 4, an invalid root would not put the task into an infinite loop :)BTW: As noted in [1],
is_page_fault_stale()only checks !mirror roots, andkvm_mmu_reload()reloads mirror roots only when !mirror roots are also invalid, since an invalid mirror root was considered impossible. (up to now, no?)
Quoted 8 lines
[1] https://lore.kernel.org/all/[email protected]
Quoted 3 lines
Add a WARN to try and detect bugs that break KVM's expectations, along with a comment to explain why it should be impossible for the root to be invalidated.
And there's already a warning in
kvm_tdp_mmu_map():
"KVM_MMU_WARN_ON(!root || root->role.invalid);". So the warning also seems redundant.
No, KVM_REQ_MMU_FREE_OBSOLETE_ROOTS can be pending even if the current root is valid. And once the is_page_fault_stale() check comes along, the WARN in kvm_tdp_mmu_map() is effectively unreachable. The patch ordering is weird, but there wasn't a great solution because adding is_page_fault_stale() first would create an obvious infinite loop.
Quoted 27 lines
Quoted 27 lines
Cc Kai Huang <[email protected]>Cc Yan Zhao <[email protected]>Cc Rick Edgecombe <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)arch/x86/kvm/mmu/mmu.c +0
-4@@ -5184,10 +5184,6 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)if (kvm_gfn_is_write_tracked (kvm, fault.slot, fault.gfn))return -EPERM ; - r =kvm_mmu_reload (vcpu); -if (r) -return r; - r =mmu_topup_memory_caches (vcpu, false);if (r)return r;@@ -5199,10 +5195,21 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)if (kvm_test_request (KVM_REQ_VM_DEAD , vcpu))return -EIO ;
Quoted 6 lines
arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5201
Quoted 4 lines
arch/x86/kvm/mmu/mmu.c +4
-0+ r =kvm_mmu_reload (vcpu); +if (r) +return r; +Since
kvm_mmu_reload()is moved inside the loop, should we also movekvm_gfn_is_write_tracked()inside the loop to prevent unexpected bugs?
I'm leaning no? That check was extreme paranoia in the first place.
Re: [PATCH 2/4] KVM: x86/mmu: Harden "map private PFN" against unexpected root invalidation
Quoted 24 lines · Sean Christopherson
Quoted 14 lines · Yan Zhao
Quoted 6 lines · Sean Christopherson
Move kvm_tdp_mmu_map_private_pfn()'s reload of the MMU into its tight loop so that an unexpected root invalidation has a better chance of being handled gracefully, even though it should be impossible for the vCPU's root to be invalidated after the initial reload. As is, encountering an invalid root is *guaranteed* to put the task into an infinite loop (albeit a breakable loop that honors
NEED_RESCHED).Note: without the newly added
is_page_fault_stale()check in patch 4, an invalid root would not put the task into an infinite loop :)BTW: As noted in [1],
is_page_fault_stale()only checks !mirror roots, andkvm_mmu_reload()reloads mirror roots only when !mirror roots are also invalid, since an invalid mirror root was considered impossible. (up to now, no?)Quoted 8 lines
[1] https://lore.kernel.org/all/[email protected]
Quoted 3 lines
Add a WARN to try and detect bugs that break KVM's expectations, along with a comment to explain why it should be impossible for the root to be invalidated.
And there's already a warning in
kvm_tdp_mmu_map():
"KVM_MMU_WARN_ON(!root || root->role.invalid);". So the warning also seems redundant.
Quoted 5 lines
No,
KVM_REQ_MMU_FREE_OBSOLETE_ROOTScan be pending even if the current root is valid. And once theis_page_fault_stale()check comes along, the WARN inkvm_tdp_mmu_map()is effectively unreachable. The patch ordering is weird, but there wasn't a great solution because addingis_page_fault_stale()first would create an obvious infinite loop.
Ok. When KVM_REQ_MMU_FREE_OBSOLETE_ROOTS is pending, it is only for direct roots. This patch still holds that assumption true, right?
If so, since mirror roots cannot be invalid after the first reload, any subsequent kvm_mmu_reload() calls triggered later in the loop would only reload direct roots.
Is this necessary for kvm_tdp_mmu_map_private_pfn(), which only maps private pfn? Or is the purpose of this patch simply to avoid the potential infinite loop after patch 4?
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Cc Kai Huang <[email protected]>Cc Yan Zhao <[email protected]>Cc Rick Edgecombe <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)arch/x86/kvm/mmu/mmu.c +0
-4@@ -5184,10 +5184,6 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)if (kvm_gfn_is_write_tracked (kvm, fault.slot, fault.gfn))return -EPERM ; - r =kvm_mmu_reload (vcpu); -if (r) -return r; - r =mmu_topup_memory_caches (vcpu, false);if (r)return r;@@ -5199,10 +5195,21 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)if (kvm_test_request (KVM_REQ_VM_DEAD , vcpu))return -EIO ;
Quoted 8 lines
Quoted 6 lines
arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5201
Quoted 4 lines
arch/x86/kvm/mmu/mmu.c +4
-0+ r =kvm_mmu_reload (vcpu); +if (r) +return r; +Since
kvm_mmu_reload()is moved inside the loop, should we also movekvm_gfn_is_write_tracked()inside the loop to prevent unexpected bugs?I'm leaning no? That check was extreme paranoia in the first place.
Hmm, ok.
Sean Christopherson
maintainerRe: [PATCH 2/4] KVM: x86/mmu: Harden "map private PFN" against unexpected root invalidation
Quoted 25 lines · Yan Zhao
Quoted 24 lines · Sean Christopherson
Quoted 22 lines · Yan Zhao
Quoted 6 lines · Sean Christopherson
Move kvm_tdp_mmu_map_private_pfn()'s reload of the MMU into its tight loop so that an unexpected root invalidation has a better chance of being handled gracefully, even though it should be impossible for the vCPU's root to be invalidated after the initial reload. As is, encountering an invalid root is *guaranteed* to put the task into an infinite loop (albeit a breakable loop that honors
NEED_RESCHED).Note: without the newly added
is_page_fault_stale()check in patch 4, an invalid root would not put the task into an infinite loop :)BTW: As noted in [1],
is_page_fault_stale()only checks !mirror roots, andkvm_mmu_reload()reloads mirror roots only when !mirror roots are also invalid, since an invalid mirror root was considered impossible. (up to now, no?)[1] https://lore.kernel.org/all/[email protected]
Quoted 3 lines
Add a WARN to try and detect bugs that break KVM's expectations, along with a comment to explain why it should be impossible for the root to be invalidated.
And there's already a warning in
kvm_tdp_mmu_map():
"KVM_MMU_WARN_ON(!root || root->role.invalid);". So the warning also seems redundant.
Quoted 6 lines
Quoted 5 lines
No,
KVM_REQ_MMU_FREE_OBSOLETE_ROOTScan be pending even if the current root is valid. And once theis_page_fault_stale()check comes along, the WARN inkvm_tdp_mmu_map()is effectively unreachable. The patch ordering is weird, but there wasn't a great solution because addingis_page_fault_stale()first would create an obvious infinite loop.Ok. When
KVM_REQ_MMU_FREE_OBSOLETE_ROOTSis pending, it is only for direct roots.
No? KVM_REQ_MMU_FREE_OBSOLETE_ROOTS is also used by the shadow MMU, in __kvm_mmu_prepare_zap_page() and in FNAME(fetch). Or did I misunderstand the question?
Quoted 9 lines
This patch still holds that assumption true, right?
If so, since mirror roots cannot be invalid after the first reload, any subsequent
kvm_mmu_reload()calls triggered later in the loop would only reload direct roots.Is this necessary for
kvm_tdp_mmu_map_private_pfn(), which only maps private pfn? Or is the purpose of this patch simply to avoid the potential infinite loop after patch 4?
Yes, though I would still want this change even without patch 4, as there are no guarantees that KVM won't gain an equivalent check in kvm_tdp_mmu_map() in the future, e.g. as hardening.
Re: [PATCH 2/4] KVM: x86/mmu: Harden "map private PFN" against unexpected root invalidation
Quoted 31 lines · Sean Christopherson
Quoted 30 lines · Yan Zhao
Quoted 29 lines · Sean Christopherson
Quoted 14 lines · Yan Zhao
Quoted 6 lines · Sean Christopherson
Move kvm_tdp_mmu_map_private_pfn()'s reload of the MMU into its tight loop so that an unexpected root invalidation has a better chance of being handled gracefully, even though it should be impossible for the vCPU's root to be invalidated after the initial reload. As is, encountering an invalid root is *guaranteed* to put the task into an infinite loop (albeit a breakable loop that honors
NEED_RESCHED).Note: without the newly added
is_page_fault_stale()check in patch 4, an invalid root would not put the task into an infinite loop :)BTW: As noted in [1],
is_page_fault_stale()only checks !mirror roots, andkvm_mmu_reload()reloads mirror roots only when !mirror roots are also invalid, since an invalid mirror root was considered impossible. (up to now, no?)Quoted 8 lines
[1] https://lore.kernel.org/all/[email protected]
Quoted 3 lines
Add a WARN to try and detect bugs that break KVM's expectations, along with a comment to explain why it should be impossible for the root to be invalidated.
And there's already a warning in
kvm_tdp_mmu_map():
"KVM_MMU_WARN_ON(!root || root->role.invalid);". So the warning also seems redundant.No,
KVM_REQ_MMU_FREE_OBSOLETE_ROOTScan be pending even if the current root is valid. And once theis_page_fault_stale()check comes along, the WARN inkvm_tdp_mmu_map()is effectively unreachable. The patch ordering is weird, but there wasn't a great solution because addingis_page_fault_stale()first would create an obvious infinite loop.
Quoted 5 lines
Quoted 1 line
Ok. When
KVM_REQ_MMU_FREE_OBSOLETE_ROOTSis pending, it is only for direct roots.No?
KVM_REQ_MMU_FREE_OBSOLETE_ROOTSis also used by the shadow MMU, in__kvm_mmu_prepare_zap_page()and in FNAME(fetch). Or did I misunderstand the question?
Oh, I forgot about the shadow MMU. By "direct roots", I was referring to "!mirror roots".kvm_tdp_mmu_map_private_pfn() is currently solely called by TDX. So, even when a vCPU has KVM_REQ_MMU_FREE_OBSOLETE_ROOTS pending, it should only be caused by invalidating !mirror roots.
Therefore, I didn't understand why kvm_tdp_mmu_map_private_pfn() cares about invalidating a !mirror root, which cannot be the same root to be mapped in kvm_tdp_mmu_map().
Quoted 6 lines
Quoted 6 lines
This patch still holds that assumption true, right?
If so, since mirror roots cannot be invalid after the first reload, any subsequent
kvm_mmu_reload()calls triggered later in the loop would only reload direct roots.
Quoted 7 lines
Quoted 3 lines
Is this necessary for
kvm_tdp_mmu_map_private_pfn(), which only maps private pfn? Or is the purpose of this patch simply to avoid the potential infinite loop after patch 4?Yes, though I would still want this change even without patch 4, as there are no guarantees that KVM won't gain an equivalent check in
kvm_tdp_mmu_map()in the future, e.g. as hardening.
Ok.
Sean Christopherson
maintainerPATCH 3/4 KVM: x86/mmu: Top-up memory caches when retrying "map private PFN"
In reply to Sean ChristophersonWhen mapping a private PFN in TDX's post-populate callback, top-up the memory caches on every attempt to map the PFN to harden against bugs in the map flow that could consume cache entries even if mapping ultimately fails. E.g. as pointed out by Sashiko, the in-progress Dynamic PAMT support could consume PAMT cache entries on TDX-Module lock contention.
Harden KVM even though consuming an entry on failure is considered a KVM bug, as retry is uncommon, top-up is "free" if there's no work to be done, and populating a TDX guest's memory is a slow path, i.e. there's no meaningful downside to the hardening.
--- arch/x86/kvm/mmu/mmu.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/kvm/mmu/mmu.c +4 -4
@@ -5184,10 +5184,6 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)if (kvm_gfn_is_write_tracked (kvm, fault.slot, fault.gfn))return -EPERM ; - r =mmu_topup_memory_caches (vcpu, false); -if (r) -return r; - do { if (signal_pending(current)) return -EINTR;
@@ -5199,6 +5195,10 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)if (r)return r; + r =mmu_topup_memory_caches (vcpu, false); +if (r) +return r; +cond_resched ();guard (read_lock)(&kvm->mmu_lock);
Signature · 1 line
2.55.0.679.g6767b8d81c-goog
[email protected]
AI reviewRe: [PATCH 3/4] KVM: x86/mmu: Top-up memory caches when retrying "map private PFN"
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The WARN_ON_ONCE check for obsolete roots in kvm_tdp_mmu_map_private_pfn is largely ineffective…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The WARN_ON_ONCE check for obsolete roots in kvm_tdp_mmu_map_private_pfn is largely ineffective due to being placed after kvm_mmu_reload, which clears the request bit. --
--- Patch [3]: [PATCH 3/4] KVM: x86/mmu: Top-up memory caches when retrying "map private PFN" --- commit 7a1e8c6872a95084f82e79ad3438f87300a97a91
Author: Sean Christopherson <[email protected]>
KVM: x86/mmu: Top-up memory caches when retrying "map private PFN"
This commit moves the top-up of memory caches into the retry loop in kvm_tdp_mmu_map_private_pfn(). This ensures that the caches are replenished on every attempt to map the PFN, hardening KVM against potential cache consumption issues during TDX post-populate callbacks.
Quoted 15 lines
arch/x86/kvm/mmu/mmu.c +0
-4@@ -5184,10 +5184,6 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)if (kvm_gfn_is_write_tracked (kvm, fault.slot, fault.gfn))return -EPERM ; - r =mmu_topup_memory_caches (vcpu, false); -if (r) -return r; - do { if (signal_pending(current)) return -EINTR;
[ ... ]
arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5204
Quoted 11 lines
arch/x86/kvm/mmu/mmu.c +4
-0@@ -5199,6 +5195,10 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)if (r)return r; + r =mmu_topup_memory_caches (vcpu, false); +if (r) +return r; +cond_resched ();guard (read_lock)(&kvm->mmu_lock);
[Severity: Medium]
In kvm_tdp_mmu_map_private_pfn(), is the WARN_ON_ONCE() check for obsolete roots still effective here?
Just a few lines down from this chunk:
WARN_ON_ONCE(kvm_test_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu));
Because kvm_mmu_reload() is called higher up in the loop, which internally invokes kvm_check_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu), it will clear the request bit.
If an invalidation occurs during the kvm_tdp_mmu_map() execution, the loop correctly retries, but the next iteration's kvm_mmu_reload() will silently clear the request bit before execution reaches the WARN_ON_ONCE() statement.
This seems to leave the assertion only covering the small window between the reload and the WARN itself, masking potential regressions where roots are unexpectedly invalidated during the map operation.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Sean Christopherson
maintainerRe: [PATCH 3/4] KVM: x86/mmu: Top-up memory caches when retrying "map private PFN"
Quoted 43 lines · On Thu, Aug 06, 2026, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Quoted 15 lines
arch/x86/kvm/mmu/mmu.c +0
-4@@ -5184,10 +5184,6 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)if (kvm_gfn_is_write_tracked (kvm, fault.slot, fault.gfn))return -EPERM ; - r =mmu_topup_memory_caches (vcpu, false); -if (r) -return r; - do { if (signal_pending(current)) return -EINTR;[ ... ]
arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5204
Quoted 11 lines
arch/x86/kvm/mmu/mmu.c +4
-0@@ -5199,6 +5195,10 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)if (r)return r; + r =mmu_topup_memory_caches (vcpu, false); +if (r) +return r; +cond_resched ();guard (read_lock)(&kvm->mmu_lock);[Severity: Medium]
Inkvm_tdp_mmu_map_private_pfn(), is theWARN_ON_ONCE()check for obsolete roots still effective here?Just a few lines down from this chunk:
WARN_ON_ONCE(kvm_test_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu));Because
kvm_mmu_reload()is called higher up in the loop, which internally invokeskvm_check_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu), it will clear the request bit.
Quoted 7 lines
If an invalidation occurs during the
kvm_tdp_mmu_map()execution, the loop correctly retries, but the next iteration'skvm_mmu_reload()will silently clear the request bit before execution reaches theWARN_ON_ONCE()statement.This seems to leave the assertion only covering the small window between the reload and the WARN itself, masking potential regressions where roots are unexpectedly invalidated during the map operation.
Yep. It's honestly more for documentation than it is for actually catching bugs. Though I do think that if a bug were introduced, there would actually be a decent chance of catching it because KVM_REQ_MMU_FREE_OBSOLETE_ROOTS is set while holding mmu_lock for write.
Moving the WARN up isn't an option, because it could get a false positive on the first iteration. And the odds of this firing are stupidly low, i.e. it's not worth reworking the code to skip the WARN on the first iteration.
Quoted 3 lines
Signature · 2 lines
-- Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Re: [PATCH 3/4] KVM: x86/mmu: Top-up memory caches when retrying "map private PFN"
In reply to Sean ChristophersonQuoted 5 lines · Sean Christopherson
When mapping a private PFN in TDX's post-populate callback, top-up the memory caches on every attempt to map the PFN to harden against bugs in the map flow that could consume cache entries even if mapping ultimately fails. E.g. as pointed out by Sashiko, the in-progress Dynamic PAMT support could consume PAMT cache entries on TDX-Module lock contention.
I think it is the same for the other caches consumed by the fault. I guess "e.g." covers it. But it's not new after DPAMT.
Quoted 4 lines
Harden KVM even though consuming an entry on failure is considered a KVM bug, as retry is uncommon
The locks held by the sole call path will prevent retries from being needed due to TDX-specific details. So in the place where this code lives, it is a bug. But can't really be hit. To me "retry is uncommon" sounds like it's a rare case that is hittable. I guess you mean only in the uncommon case of bugs.
Quoted 8 lines
, top-up is "free" if there's no work to be done,
and populating a TDX guest's memory is a slow path, i.e. there's no meaningful downside to the hardening.Reported-by Sashiko Bot <[email protected]>Signed-off-by Sean Christopherson <[email protected]>---
Thanks! And...
Sean Christopherson
maintainerRe: [PATCH 3/4] KVM: x86/mmu: Top-up memory caches when retrying "map private PFN"
Quoted 9 lines · Rick P Edgecombe
Quoted 5 lines · Sean Christopherson
When mapping a private PFN in TDX's post-populate callback, top-up the memory caches on every attempt to map the PFN to harden against bugs in the map flow that could consume cache entries even if mapping ultimately fails. E.g. as pointed out by Sashiko, the in-progress Dynamic PAMT support could consume PAMT cache entries on TDX-Module lock contention.
I think it is the same for the other caches consumed by the fault. I guess "e.g." covers it. But it's not new after DPAMT.
I don't think so? Especially since as you point out below, nothing else can muck with the SPTEs. The TDP MMU only consumes an cache entry if it successfully creates a SPTE, and since nothing can muck with SPTEs, anything created on the first attempt will still be there on subsequent attempts. I.e. the TDP MMU might create SPTEs that are ultimately unused, but I don't think it can exhaust a cache.
Quoted 8 lines
Quoted 3 lines
Harden KVM even though consuming an entry on failure is considered a KVM bug, as retry is uncommon
The locks held by the sole call path will prevent retries from being needed due to TDX-specific details. So in the place where this code lives, it is a bug. But can't really be hit. To me "retry is uncommon" sounds like it's a rare case that is hittable. I guess you mean only in the uncommon case of bugs.
Ah, I was thinking a different task could pre-fault memory, but pre-fault isn't allowed until the VM is TD_STATE_RUNNABLE, and KVM_TDX_INIT_MEM_REGION is only usable if the VM is *not* TD_STATE_RUNNABLE.
Re: [PATCH 3/4] KVM: x86/mmu: Top-up memory caches when retrying "map private PFN"
Quoted 9 lines · Sean Christopherson
Quoted 2 lines
I think it is the same for the other caches consumed by the fault. I guess "e.g." covers it. But it's not new after DPAMT.
I don't think so? Especially since as you point out below, nothing else can muck with the SPTEs. The TDP MMU only consumes an cache entry if it successfully creates a SPTE, and since nothing can muck with SPTEs, anything created on the first attempt will still be there on subsequent attempts. I.e. the TDP MMU might create SPTEs that are ultimately unused, but I don't think it can exhaust a cache.
Functionally we won't see multiple kvm_tdp_mmu_map() calls during kvm_tdp_mmu_map_private_pfn() because of the locks as we discussed. But... if we did, then I think we would have the same pattern of freeing but not topping up before retrying:
kvm_tdp_mmu_map():
...
/*
* The SPTE is either non-present or points to a huge page that
* needs to be split.
*/
sp = tdp_mmu_alloc_sp(vcpu);
tdp_mmu_init_child_sp(sp, &iter); <- shrink mirror
if (is_mirror_sp(sp))
kvm_mmu_alloc_external_spt(vcpu, sp); <- shrink external
sp->nx_huge_page_disallowed = fault->huge_page_disallowed;
if (is_shadow_present_pte(iter.old_spte)) {
/* Don't support large page for mirrored roots (TDX) */
KVM_BUG_ON(is_mirror_sptep(iter.sptep), vcpu->kvm);
r = tdp_mmu_split_huge_page(kvm, &iter, sp, true);
} else {
r = tdp_mmu_link_sp(kvm, &iter, sp, true);<- TDX
error(BUSY,etc)
}
/*
* Force the guest to retry if installing an upper level SPTE
* failed, e.g. because a different task modified the SPTE.
*/
if (r) {
tdp_mmu_free_unused_sp(sp); <- free_page() external and mirror
goto retry; <- return to kvm_tdp_mmu_map_private_pfn()
}...
Without the additional top up in kvm_tdp_mmu_map_private_pfn(), the retry wouldn't have enough, right? The 'sp' goes back to the cache, but the external_pt doesn't. Actually, hmm...
But that is why I thought: No functional issue with or without DPAMT. And the brittleness is existing.
Sean Christopherson
maintainerRe: [PATCH 3/4] KVM: x86/mmu: Top-up memory caches when retrying "map private PFN"
Quoted 46 lines · Rick P Edgecombe
Quoted 9 lines · Sean Christopherson
Quoted 2 lines
I think it is the same for the other caches consumed by the fault. I guess "e.g." covers it. But it's not new after DPAMT.
I don't think so? Especially since as you point out below, nothing else can muck with the SPTEs. The TDP MMU only consumes an cache entry if it successfully creates a SPTE, and since nothing can muck with SPTEs, anything created on the first attempt will still be there on subsequent attempts. I.e. the TDP MMU might create SPTEs that are ultimately unused, but I don't think it can exhaust a cache.
Functionally we won't see multiple
kvm_tdp_mmu_map()calls duringkvm_tdp_mmu_map_private_pfn()because of the locks as we discussed. But... if we did, then I think we would have the same pattern of freeing but not topping up before retrying:
kvm_tdp_mmu_map():
.../* * The SPTE is either non-present or points to a huge page that * needs to be split. */ sp = tdp_mmu_alloc_sp(vcpu); tdp_mmu_init_child_sp(sp, &iter); <- shrink mirror if (is_mirror_sp(sp)) kvm_mmu_alloc_external_spt(vcpu, sp); <- shrink external sp->nx_huge_page_disallowed = fault->huge_page_disallowed; if (is_shadow_present_pte(iter.old_spte)) { /* Don't support large page for mirrored roots (TDX) */ KVM_BUG_ON(is_mirror_sptep(iter.sptep), vcpu->kvm); r = tdp_mmu_split_huge_page(kvm, &iter, sp, true); } else { r = tdp_mmu_link_sp(kvm, &iter, sp, true);<- TDX error(BUSY,etc) } /* * Force the guest to retry if installing an upper level SPTE * failed, e.g. because a different task modified the SPTE. */ if (r) { tdp_mmu_free_unused_sp(sp); <- free_page() external and mirror goto retry; <- return to kvm_tdp_mmu_map_private_pfn() }
Quoted 5 lines
...
Without the additional top up in
kvm_tdp_mmu_map_private_pfn(), the retry wouldn't have enough, right? The 'sp' goes back to the cache, but theexternal_ptdoesn't. Actually, hmm...
Huh. Right you are. I completely forgot that flow existed. Seems stupidly obvious in hindsight that something like that would have to exist.
Quoted 2 lines
But that is why I thought: No functional issue with or without DPAMT. And the brittleness is existing.
Ya, agreed.
Sean Christopherson
maintainerPATCH 4/4 KVM: x86/mmu: Add sanity check to detect stale page faults in "map private PFN"
In reply to Sean ChristophersonHarden the "map private PFN" flow against potentially-fatal bugs or future KVM changes by checking for a stale "fault" prior to actually mapping the PFN into the guest. While it should be impossible for the "page fault" to become stale, the sanity check is cheap, whereas a broken assumption would have a high probability of leading to a guest-expoitable use-after-free.
Snapshot the invalidation sequence after acquiring mmu_lock to avoid false positives, even though doing so completely voids anys and all protection against unexpected invalidations. Pretty much the entire point of kvm_tdp_mmu_map_private_pfn() is that it allows mapping a PFN that was gifted by the caller, i.e. the caller would have to mess up its one and only responsibility.
--- arch/x86/kvm/mmu/mmu.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/x86/kvm/mmu/mmu.c +10 -0
@@ -5210,6 +5210,16 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)*/ WARN_ON_ONCE (kvm_test_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu)); +/* +* Snapshot the invalidation sequence counter after acquiring +* mmu_lock, as guest_memfd guarantees the validity of the pfn, +* i.e. any concurrent invalidations are guaranteed to be +* irrelevant. +*/ + fault.mmu_seq = vcpu->kvm->mmu_invalidate_seq; +if (is_page_fault_stale (vcpu, &fault)) +continue ; + r =kvm_tdp_mmu_map (vcpu, &fault); }while (r ==RET_PF_RETRY );
Signature · 1 line
2.55.0.679.g6767b8d81c-goog
[email protected]
AI reviewRe: [PATCH 4/4] KVM: x86/mmu: Add sanity check to detect stale page faults in "map private PFN"
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] The `continue` statement executed on a stale page fault inadvertently terminates the retry loop…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] The continue statement executed on a stale page fault inadvertently terminates the retry loop, causing a fatal -EIO failure instead of retrying.
- [High] The KVM MMU invalidation retry protocol order is violated by capturing the sequence after acquiring mmu_lock and after the PFN is resolved. --
--- Patch [4]: [PATCH 4/4] KVM: x86/mmu: Add sanity check to detect stale page faults in "map private PFN" --- commit 8ecbe739b111e2504c32794278902d5137202a16
Author: Sean Christopherson <[email protected]>
KVM: x86/mmu: Add sanity check to detect stale page faults in "map private PFN"
This commit hardens the "map private PFN" flow against stale page faults prior to mapping the PFN into the guest. It adds a sanity check using a sequence counter snapshot taken under the mmu_lock.
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +0
-0
arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5219
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +7
-0@@ -5210,6 +5210,16 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)*/ WARN_ON_ONCE (kvm_test_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu)); +/* +* Snapshot the invalidation sequence counter after acquiring +* mmu_lock, as guest_memfd guarantees the validity of the pfn, +* i.e. any concurrent invalidations are guaranteed to be +* irrelevant. +*/ + fault.mmu_seq = vcpu->kvm->mmu_invalidate_seq;
[Severity: High]
Does capturing the invalidation sequence after acquiring mmu_lock violate the KVM MMU invalidation retry protocol order?
The sequence is explicitly captured inside the mmu_lock critical section after the PFN has already been resolved by the caller. This bypasses the mandatory generation safety sequence (capture sequence, resolve PFN, acquire lock).
Additionally, could this still cause false positives since mmu_invalidate_retry_gfn() checks mmu_invalidate_in_progress?
arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5221
Quoted 2 lines
arch/x86/kvm/mmu/mmu.c +2
-0+if (is_page_fault_stale (vcpu, &fault)) +continue ;
[Severity: High]
Does this continue statement inadvertently exit the loop without retrying?
In C, a continue statement within a do-while loop jumps to the while condition evaluation, not the beginning of the loop block. If r is not RET_PF_RETRY when this continue is hit, the condition r == RET_PF_RETRY will evaluate to false.
Will this cause the function to exit the loop prematurely and return -EIO?
Quoted 2 lines
arch/x86/kvm/mmu/mmu.c +0
-0r =kvm_tdp_mmu_map (vcpu, &fault); }while (r ==RET_PF_RETRY );
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Sean Christopherson
maintainerRe: [PATCH 4/4] KVM: x86/mmu: Add sanity check to detect stale page faults in "map private PFN"
Quoted 8 lines · On Thu, Aug 06, 2026, [email protected] wrote:
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c +0
-0@@ -5210,6 +5210,16 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)*/ WARN_ON_ONCE (kvm_test_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu));
Quoted 11 lines
arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5219
Quoted 7 lines
arch/x86/kvm/mmu/mmu.c +7
-0+/* +* Snapshot the invalidation sequence counter after acquiring +* mmu_lock, as guest_memfd guarantees the validity of the pfn, +* i.e. any concurrent invalidations are guaranteed to be +* irrelevant. +*/ + fault.mmu_seq = vcpu->kvm->mmu_invalidate_seq;[Severity: High]
Does capturing the invalidation sequence after acquiringmmu_lockviolate the KVM MMU invalidation retry protocol order?
Yes, see the comment and changelog.
Quoted 7 lines
The sequence is explicitly captured inside the
mmu_lockcritical section after the PFN has already been resolved by the caller. This bypasses the mandatory generation safety sequence (capture sequence, resolve PFN, acquire lock).Additionally, could this still cause false positives since
mmu_invalidate_retry_gfn()checksmmu_invalidate_in_progress?
Yes, it's possible, but it would require zappings GFNs on either side of the GFN being mapped, such that the target GFN is contained by mmu_invalidate_range_{start,end}. The odds of that happening in practice are low enough that I'm comfortable eating false positives even though strictly speaking they _should_ be irrelevant.
Quoted 6 lines
arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5221
Quoted 2 lines
arch/x86/kvm/mmu/mmu.c +2
-0+if (is_page_fault_stale (vcpu, &fault)) +continue ;[Severity: High]
Does this continue statement inadvertently exit the loop without retrying?
Quoted 6 lines
In C, a continue statement within a do-while loop jumps to the while condition evaluation, not the beginning of the loop block. If r is not
RET_PF_RETRYwhen this continue is hit, the condition r ==RET_PF_RETRYwill evaluate to false.Will this cause the function to exit the loop prematurely and return
-EIO?
Doh. And this is why I'm more than happy to deal with Sashiko's paranoia :-)
Re: [PATCH 4/4] KVM: x86/mmu: Add sanity check to detect stale page faults in "map private PFN"
In reply to Sean ChristophersonQuoted 23 lines · Sean Christopherson
Harden the "map private PFN" flow against potentially-fatal bugs or future KVM changes by checking for a stale "fault" prior to actually mapping the PFN into the guest. While it should be impossible for the "page fault" to become stale, the sanity check is cheap, whereas a broken assumption would have a high probability of leading to a guest-expoitable use-after-free.
Snapshot the invalidation sequence after acquiring
mmu_lockto avoid false positives, even though doing so completely voids anys and all protection against unexpected invalidations. Pretty much the entire point ofkvm_tdp_mmu_map_private_pfn()is that it allows mapping a PFN that was gifted by the caller, i.e. the caller would have to mess up its one and only responsibility.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 10 ++++++++++ 1 file changed, 10 insertions(+)arch/x86/kvm/mmu/mmu.c +0
-0
arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5219
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +7
-0@@ -5210,6 +5210,16 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)*/ WARN_ON_ONCE (kvm_test_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu)); +/* +* Snapshot the invalidation sequence counter after acquiring +* mmu_lock, as guest_memfd guarantees the validity of the pfn, +* i.e. any concurrent invalidations are guaranteed to be +* irrelevant. +*/ + fault.mmu_seq = vcpu->kvm->mmu_invalidate_seq;
Could you explain more about the conditions under which a fault is stale while guest_memfd guarantees the validity of the pfn?
Given that kvm_tdp_mmu_map_private_pfn() already asserts holding slots_lock and invalidate_lock, I can't think of one. If we add the sanity check because it's cheap, why don't we save fault.mmu_seq before getting the pfn to guard against stale pfn as well?
Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +3
-0+if (is_page_fault_stale (vcpu, &fault)) +continue ; + r =kvm_tdp_mmu_map (vcpu, &fault); }while (r ==RET_PF_RETRY );
Re: [PATCH 4/4] KVM: x86/mmu: Add sanity check to detect stale page faults in "map private PFN"
Quoted 37 lines · Yan Zhao
Quoted 23 lines · Sean Christopherson
Harden the "map private PFN" flow against potentially-fatal bugs or future KVM changes by checking for a stale "fault" prior to actually mapping the PFN into the guest. While it should be impossible for the "page fault" to become stale, the sanity check is cheap, whereas a broken assumption would have a high probability of leading to a guest-expoitable use-after-free.
Snapshot the invalidation sequence after acquiring
mmu_lockto avoid false positives, even though doing so completely voids anys and all protection against unexpected invalidations. Pretty much the entire point ofkvm_tdp_mmu_map_private_pfn()is that it allows mapping a PFN that was gifted by the caller, i.e. the caller would have to mess up its one and only responsibility.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 10 ++++++++++ 1 file changed, 10 insertions(+)arch/x86/kvm/mmu/mmu.c +0
-0arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5219
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +7
-0@@ -5210,6 +5210,16 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)*/ WARN_ON_ONCE (kvm_test_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu)); +/* +* Snapshot the invalidation sequence counter after acquiring +* mmu_lock, as guest_memfd guarantees the validity of the pfn, +* i.e. any concurrent invalidations are guaranteed to be +* irrelevant. +*/ + fault.mmu_seq = vcpu->kvm->mmu_invalidate_seq;Could you explain more about the conditions under which a fault is stale while
guest_memfdguarantees the validity of the pfn?
Quoted 6 lines
Given that
kvm_tdp_mmu_map_private_pfn()already asserts holdingslots_lockandinvalidate_lock, I can't think of one. If we add the sanity check because it's cheap, why don't we save fault.mmu_seq before getting the pfn to guard against stale pfn as well?arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5220
Quoted 1 line
arch/x86/kvm/mmu/mmu.c +1
-0+if (is_page_fault_stale (vcpu, &fault))
is_page_fault_stale() gets sp from vcpu->arch.mmu->root.hpa:
"struct kvm_mmu_page *sp = root_to_sp(vcpu->arch.mmu->root.hpa);".
For TDX, sp should be from vcpu->arch.mmu->mirror_root_hpa?
Quoted 5 lines
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +2
-0+continue ; + r =kvm_tdp_mmu_map (vcpu, &fault); }while (r ==RET_PF_RETRY );
Sean Christopherson
maintainerRe: [PATCH 4/4] KVM: x86/mmu: Add sanity check to detect stale page faults in "map private PFN"
In reply to Yan ZhaoQuoted 37 lines · Yan Zhao
Quoted 23 lines · Sean Christopherson
Harden the "map private PFN" flow against potentially-fatal bugs or future KVM changes by checking for a stale "fault" prior to actually mapping the PFN into the guest. While it should be impossible for the "page fault" to become stale, the sanity check is cheap, whereas a broken assumption would have a high probability of leading to a guest-expoitable use-after-free.
Snapshot the invalidation sequence after acquiring
mmu_lockto avoid false positives, even though doing so completely voids anys and all protection against unexpected invalidations. Pretty much the entire point ofkvm_tdp_mmu_map_private_pfn()is that it allows mapping a PFN that was gifted by the caller, i.e. the caller would have to mess up its one and only responsibility.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 10 ++++++++++ 1 file changed, 10 insertions(+)arch/x86/kvm/mmu/mmu.c +0
-0arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5219
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +7
-0@@ -5210,6 +5210,16 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)*/ WARN_ON_ONCE (kvm_test_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu)); +/* +* Snapshot the invalidation sequence counter after acquiring +* mmu_lock, as guest_memfd guarantees the validity of the pfn, +* i.e. any concurrent invalidations are guaranteed to be +* irrelevant. +*/ + fault.mmu_seq = vcpu->kvm->mmu_invalidate_seq;Could you explain more about the conditions under which a fault is stale while
guest_memfdguarantees the validity of the pfn?
Quoted 2 lines
Given that
kvm_tdp_mmu_map_private_pfn()already asserts holdingslots_lockandinvalidate_lock, I can't think of one.
A non-guest_memfd mmu_notifier invalidation bumps mmu_invalidate_seq. And because the range-based invalidation checks are deliberately coarse, in-flight invalidations could also trigger a false positive if GFNs N and N+2 are being invalidate, while kvm_tdp_mmu_map_private_pfn() is trying to map N+1.
Quoted 2 lines
If we add the sanity check because it's cheap, why don't we save fault.mmu_seq before getting the pfn to guard against stale pfn as well?
Because as the changelog says, the whole point of this API is to provide a stable PFN. I want to add an is_page_fault_stale() check as defense-in-depth against bugs, and against future changes, e.g. if we extend is_page_fault_stale() to cover more reason why a page fault can become stale. The downside is that is_page_fault_stale() is susceptible to false positives; the funky code here is to minimize the chances of a false positives, while maintaining a reasonable level of safety.
Re: [PATCH 4/4] KVM: x86/mmu: Add sanity check to detect stale page faults in "map private PFN"
Sorry for the late reply!
Quoted 38 lines · Sean Christopherson
Quoted 37 lines · Yan Zhao
Quoted 23 lines · Sean Christopherson
Harden the "map private PFN" flow against potentially-fatal bugs or future KVM changes by checking for a stale "fault" prior to actually mapping the PFN into the guest. While it should be impossible for the "page fault" to become stale, the sanity check is cheap, whereas a broken assumption would have a high probability of leading to a guest-expoitable use-after-free.
Snapshot the invalidation sequence after acquiring
mmu_lockto avoid false positives, even though doing so completely voids anys and all protection against unexpected invalidations. Pretty much the entire point ofkvm_tdp_mmu_map_private_pfn()is that it allows mapping a PFN that was gifted by the caller, i.e. the caller would have to mess up its one and only responsibility.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 10 ++++++++++ 1 file changed, 10 insertions(+)arch/x86/kvm/mmu/mmu.c +0
-0arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5219
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +7
-0@@ -5210,6 +5210,16 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)*/ WARN_ON_ONCE (kvm_test_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu)); +/* +* Snapshot the invalidation sequence counter after acquiring +* mmu_lock, as guest_memfd guarantees the validity of the pfn, +* i.e. any concurrent invalidations are guaranteed to be +* irrelevant. +*/ + fault.mmu_seq = vcpu->kvm->mmu_invalidate_seq;Could you explain more about the conditions under which a fault is stale while
guest_memfdguarantees the validity of the pfn?
Quoted 7 lines
Quoted 2 lines
Given that
kvm_tdp_mmu_map_private_pfn()already asserts holdingslots_lockandinvalidate_lock, I can't think of one.A non-guest_memfd
mmu_notifierinvalidation bumpsmmu_invalidate_seq. And because the range-based invalidation checks are deliberately coarse, in-flight invalidations could also trigger a false positive if GFNs N and N+2 are being invalidate, whilekvm_tdp_mmu_map_private_pfn()is trying to map N+1.
Could updating fault.mmu_seq in each iteration cause a false negative? I may be wrong, as I am unaware of the specific bugs or future changes you mentioned below in 'defense-in-depth against bugs and against future changes'. Could you elaborate a little more on those bugs or future changes?
Quoted 5 lines
Quoted 2 lines
If we add the sanity check because it's cheap, why don't we save fault.mmu_seq before getting the pfn to guard against stale pfn as well?
Because as the changelog says, the whole point of this API is to provide a stable PFN. I want to add an
is_page_fault_stale()check as defense-in-depth against bugs,
Ok. So adding is_page_fault_stale() is not intended to ensure that a private PFN is stable (i.e., not stale).
Any background on why we cannot update the code when the bug/future changes actually arise? :)
Quoted 4 lines
and against future changes, e.g. if we extend
is_page_fault_stale()to cover more reason why a page fault can become stale. The downside is thatis_page_fault_stale()is susceptible to false positives; the funky code here is to minimize the chances of a false positives, while maintaining a reasonable level of safety.
Sean Christopherson
maintainerRe: [PATCH 4/4] KVM: x86/mmu: Add sanity check to detect stale page faults in "map private PFN"
Quoted 41 lines · Yan Zhao
Sorry for the late reply!
Quoted 38 lines · Sean Christopherson
Quoted 37 lines · Yan Zhao
Quoted 23 lines · Sean Christopherson
Harden the "map private PFN" flow against potentially-fatal bugs or future KVM changes by checking for a stale "fault" prior to actually mapping the PFN into the guest. While it should be impossible for the "page fault" to become stale, the sanity check is cheap, whereas a broken assumption would have a high probability of leading to a guest-expoitable use-after-free.
Snapshot the invalidation sequence after acquiring
mmu_lockto avoid false positives, even though doing so completely voids anys and all protection against unexpected invalidations. Pretty much the entire point ofkvm_tdp_mmu_map_private_pfn()is that it allows mapping a PFN that was gifted by the caller, i.e. the caller would have to mess up its one and only responsibility.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 10 ++++++++++ 1 file changed, 10 insertions(+)arch/x86/kvm/mmu/mmu.c +0
-0arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5219
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +7
-0@@ -5210,6 +5210,16 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)*/ WARN_ON_ONCE (kvm_test_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu)); +/* +* Snapshot the invalidation sequence counter after acquiring +* mmu_lock, as guest_memfd guarantees the validity of the pfn, +* i.e. any concurrent invalidations are guaranteed to be +* irrelevant. +*/ + fault.mmu_seq = vcpu->kvm->mmu_invalidate_seq;Could you explain more about the conditions under which a fault is stale while
guest_memfdguarantees the validity of the pfn?
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Given that
kvm_tdp_mmu_map_private_pfn()already asserts holdingslots_lockandinvalidate_lock, I can't think of one.A non-guest_memfd
mmu_notifierinvalidation bumpsmmu_invalidate_seq. And because the range-based invalidation checks are deliberately coarse, in-flight invalidations could also trigger a false positive if GFNs N and N+2 are being invalidate, whilekvm_tdp_mmu_map_private_pfn()is trying to map N+1.Could updating fault.mmu_seq in each iteration cause a false negative?
Not really. I mean, yes, technically it could, but I addressed this in changelog.
: Snapshot the invalidation sequence after acquiring mmu_lock to avoid false : positives, even though doing so completely voids anys and all protection : against unexpected invalidations. Pretty much the entire point of : kvm_tdp_mmu_map_private_pfn() is that it allows mapping a PFN that was : gifted by the caller, i.e. the caller would have to mess up its one and : only responsibility.
Quoted 3 lines
I may be wrong, as I am unaware of the specific bugs or future changes you mentioned below in 'defense-in-depth against bugs and against future changes'. Could you elaborate a little more on those bugs or future changes?
There are non, AFAIK. It's a statement saying "things may change at some point in the future".
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
If we add the sanity check because it's cheap, why don't we save fault.mmu_seq before getting the pfn to guard against stale pfn as well?
Because as the changelog says, the whole point of this API is to provide a stable PFN. I want to add an
is_page_fault_stale()check as defense-in-depth against bugs,Ok. So adding
is_page_fault_stale()is not intended to ensure that a private PFN is stable (i.e., not stale).
Any background on why we cannot update the code when the bug/future changes actually arise? :)
Because finding bugs of this nature after they've been shipped to production isn't very fun.
Re: [PATCH 4/4] KVM: x86/mmu: Add sanity check to detect stale page faults in "map private PFN"
Quoted 51 lines · Sean Christopherson
Quoted 41 lines · Yan Zhao
Sorry for the late reply!
Quoted 38 lines · Sean Christopherson
Quoted 37 lines · Yan Zhao
Quoted 23 lines · Sean Christopherson
Harden the "map private PFN" flow against potentially-fatal bugs or future KVM changes by checking for a stale "fault" prior to actually mapping the PFN into the guest. While it should be impossible for the "page fault" to become stale, the sanity check is cheap, whereas a broken assumption would have a high probability of leading to a guest-expoitable use-after-free.
Snapshot the invalidation sequence after acquiring
mmu_lockto avoid false positives, even though doing so completely voids anys and all protection against unexpected invalidations. Pretty much the entire point ofkvm_tdp_mmu_map_private_pfn()is that it allows mapping a PFN that was gifted by the caller, i.e. the caller would have to mess up its one and only responsibility.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 10 ++++++++++ 1 file changed, 10 insertions(+)arch/x86/kvm/mmu/mmu.c +0
-0arch/x86/kvm/mmu/mmu.c int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) · line 5219
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +7
-0@@ -5210,6 +5210,16 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)*/ WARN_ON_ONCE (kvm_test_request (KVM_REQ_MMU_FREE_OBSOLETE_ROOTS , vcpu)); +/* +* Snapshot the invalidation sequence counter after acquiring +* mmu_lock, as guest_memfd guarantees the validity of the pfn, +* i.e. any concurrent invalidations are guaranteed to be +* irrelevant. +*/ + fault.mmu_seq = vcpu->kvm->mmu_invalidate_seq;Could you explain more about the conditions under which a fault is stale while
guest_memfdguarantees the validity of the pfn?Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Given that
kvm_tdp_mmu_map_private_pfn()already asserts holdingslots_lockandinvalidate_lock, I can't think of one.A non-guest_memfd
mmu_notifierinvalidation bumpsmmu_invalidate_seq. And because the range-based invalidation checks are deliberately coarse, in-flight invalidations could also trigger a false positive if GFNs N and N+2 are being invalidate, whilekvm_tdp_mmu_map_private_pfn()is trying to map N+1.Could updating fault.mmu_seq in each iteration cause a false negative?
Quoted 8 lines
Not really. I mean, yes, technically it could, but I addressed this in changelog.
: Snapshot the invalidation sequence after acquiring
mmu_lockto avoid false : positives, even though doing so completely voids anys and all protection : against unexpected invalidations. Pretty much the entire point of :kvm_tdp_mmu_map_private_pfn()is that it allows mapping a PFN that was : gifted by the caller, i.e. the caller would have to mess up its one and : only responsibility.
Thanks for the explanation. I wasn't able to gather that meaning from this paragraph before :)
Quoted 6 lines
Quoted 3 lines
I may be wrong, as I am unaware of the specific bugs or future changes you mentioned below in 'defense-in-depth against bugs and against future changes'. Could you elaborate a little more on those bugs or future changes?
There are non, AFAIK. It's a statement saying "things may change at some point in the future".
I see.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
If we add the sanity check because it's cheap, why don't we save fault.mmu_seq before getting the pfn to guard against stale pfn as well?
Because as the changelog says, the whole point of this API is to provide a stable PFN. I want to add an
is_page_fault_stale()check as defense-in-depth against bugs,
Quoted 7 lines
Quoted 4 lines
Ok. So adding
is_page_fault_stale()is not intended to ensure that a private PFN is stable (i.e., not stale).
Any background on why we cannot update the code when the bug/future changes actually arise? :)Because finding bugs of this nature after they've been shipped to production isn't very fun.
Ok.