KVM: pfncache: track HVA invalidations for HVA-based caches
PATCH 1/1 KVM: pfncache: track HVA invalidations for HVA-based caches
HVA-based gfn_to_pfn caches are not necessarily backed by a KVM memslot. When an MMU notifier invalidation targets such an HVA, KVM's global mmu_invalidate_seq is not guaranteed to change because that sequence is advanced through the memslot-based invalidation path.
This matters during hva_to_pfn_retry(). The refresh path temporarily marks the cache invalid and drops gpc->lock while resolving the HVA and creating a kernel mapping. If an overlapping HVA invalidation completes in that window, the notifier may observe gpc->valid == false and therefore leave no state behind for the in-progress refresh. For an HVA outside all memslots, the refresh cannot rely on mmu_invalidate_seq to detect the event either.
To prevent this, we must add a per-cache HVA invalidation sequence. Bump the sequence whenever the cached HVA overlaps an MMU notifier range, regardless of the current valid state. Snapshot the sequence before dropping gpc->lock in hva_to_pfn_retry(), and retry the refresh if it changes before the new mapping is published.
--- include/linux/kvm_types.h | 1 + virt/kvm/pfncache.c | 42 ++++++++++++++++++++++++++++++++------- 2 files changed, 36 insertions(+), 7 deletions(-)
include/linux/kvm_types.h +1 -0
@@ -85,6 +85,7 @@ struct gfn_to_pfn_cache {u64 generation;gpa_t gpa;unsigned long uhva; +unsigned long hva_invalidate_seq;struct kvm_memory_slot *memslot;struct kvm *kvm;struct list_head list;
virt/kvm/pfncache.c +35 -7
98 lines not on this page · raw message
--
Sean Christopherson
reviewerQuoted 13 lines · Jeongjun Park
HVA-based
gfn_to_pfncaches are not necessarily backed by a KVM memslot. When an MMU notifier invalidation targets such an HVA, KVM's globalmmu_invalidate_seqis not guaranteed to change because that sequence is advanced through the memslot-based invalidation path.This matters during
hva_to_pfn_retry(). The refresh path temporarily marks the cache invalid and dropsgpc->lockwhile resolving the HVA and creating a kernel mapping. If an overlapping HVA invalidation completes in that window, the notifier may observegpc->valid== false and therefore leave no state behind for the in-progress refresh. For an HVA outside all memslots, the refresh cannot rely onmmu_invalidate_seqto detect the event either.
Quoted 1 line
To prevent this, we must add a per-cache HVA invalidation sequence.
No, there are multiple ways this could be handled. Instead of saying "we must", state what change is being (as you did below), and then briefly explain the alternatives and _why_ the chosen approach was deemed to be the best approach.
Quoted 7 lines
Bump the sequence whenever the cached HVA overlaps an MMU notifier range, regardless of the current valid state. Snapshot the sequence before dropping
gpc->lockinhva_to_pfn_retry(), and retry the refresh if it changes before the new mapping is published.Reported-by [email protected]
Given that there's no reproducer, how did you test this?
Quoted 25 lines
Fixes b9220d32799a ("KVM: x86/xen: allow shared_info to be mapped by fixed HVA")Signed-off-by Jeongjun Park <[email protected]>--- include/linux/kvm_types.h | 1 + virt/kvm/pfncache.c | 42 ++++++++++++++++++++++++++++++++------- 2 files changed, 36 insertions(+), 7 deletions(-)include/linux/kvm_types.h +1
-0@@ -85,6 +85,7 @@ struct gfn_to_pfn_cache {u64 generation;gpa_t gpa;unsigned long uhva; +unsigned long hva_invalidate_seq;struct kvm_memory_slot *memslot;struct kvm *kvm;struct list_head list;virt/kvm/pfncache.c +0
-0@@ -19,6 +19,24 @@
virt/kvm/pfncache.c line 29
Quoted 10 lines
virt/kvm/pfncache.c +8
-0#include "kvm_mm.h" +static inline bool gpc_uhva_in_range (struct gfn_to_pfn_cache *gpc, +unsigned long start,unsigned long end) +{ +return gpc->uhva >= start && gpc->uhva < end; +} + +static inline bool gpc_should_invalidate (struct gfn_to_pfn_cache *gpc, +unsigned long start,unsigned long end)
Rather than providing a "should invalidate" helper, which is makes it difficult to understand the conditions for invalidation, i.e. requires jumping around the code base, I think we should provide:
static bool gpc_hva_is_valid(struct gfn_to_pfn_cache *gpc)
{
return kvm_gpc_is_hva_active(gpc) || gpc->valid;}
and open code the start/end checks. We could keep gpc_uhva_in_range() (but drop the 'u'), but IMO it's not a net positive. E.g. yield:
/* Only a single page so no need to care about length */
if (gpc_hva_is_valid(gpc) &&
gpc->uhva >= start && gpc->uhva < end) {
read_unlock_irq(&gpc->lock);Which, for me, makes it easier to understand _this_ flow. I might have questions about what constitutes a "valid hva", but I don't need to learn those details to understand the invalidation flow.
Quoted 24 lines
virt/kvm/pfncache.c +10
-2+{ +if (!gpc_uhva_in_range (gpc, start, end)) +return false; + +if (kvm_gpc_is_hva_active (gpc)) +return true; + +return gpc->valid && !is_error_noslot_pfn (gpc->pfn); +}/* * MMU notifier 'invalidate_range_start' hook. */ @@ -32,8 +50,7 @@ void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start,read_lock_irq (&gpc->lock);/* Only a single page so no need to care about length */ -if (gpc->valid && !is_error_noslot_pfn (gpc->pfn) && - gpc->uhva >= start && gpc->uhva < end) { +if (gpc_should_invalidate (gpc, start, end)) {read_unlock_irq (&gpc->lock);/*
virt/kvm/pfncache.c void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start, · line 48
Quoted 4 lines
virt/kvm/pfncache.c +0
-1@@ -45,9 +62,11 @@ void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start,*/ write_lock_irq (&gpc->lock); - if (gpc->valid && !is_error_noslot_pfn(gpc->pfn) &&
Unless I'm forgetting/missing something, it should be impossible for gpc->pfn to be garbage if gpc->valid is true. As a follow-up patch, turn that into a WARN_ON_ONCE()? Assuming I'm not missing something...
virt/kvm/pfncache.c void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start, · line 67
Quoted 4 lines
virt/kvm/pfncache.c +3
-1- gpc->uhva >= start && gpc->uhva < end) +if (gpc_should_invalidate (gpc, start, end)) { +if (kvm_gpc_is_hva_active (gpc)) + gpc->hva_invalidate_seq++;
Hmm, I think I would rather just add a per-gpc mmu_invalidate_seq and drop usage of kvm->mmu_invalidate_seq entirely in the gpc code. Well, if we can get this working, which I'm not sure we can without a pile of surgery I don't really want to do.
CPU0 (hva_to_pfn_retry()) CPU1 (invalidate)
------------------------- -------------------------
invalidate_range_start() mn_active_invalidate_count = 1 hva_invalidate_seq = X write_unlock_irq()
write_lock_irq()
gpc->active = true
gpc->uhva = A
hva_seq = X
write_unlock_irq()
pfn = Z
invalidate_range_end() mn_active_invalidate_count = 0
mn_active_invalidate_count == 0
hva_seq == hva_invalidate_seq
use-after-free
In other words, the mmu_invalidate_seq scheme requires that the sequence counter be incremented in the same critical section as mmu_invalidate_in_progress.
Note, the current use of mn_active_invalidate_count is "fine" because it outlives mmu_invalidate_in_progress, i.e. mn_active_invalidate_count achieves the "same critical section" rule by proxy.
We could add gfn_to_pfn_cache_invalidate_end(), but I really, really don't want to add more serialization to the mmu_notifier flow when it's all but guaranteed to be unnecessary. And we can't optimize it to skip bumping sequence counters if gfn_to_pfn_cache_invalidate_start() doesn't detect relevant GPS, unless we block GPC refresh while mn_active_invalidate_count is elevated, as is done for memslots, otherwise KVM would incorrectly skip bumping counters if a GPC becomes relevant after the fact. And blocking memslot updates is a bit of a disaster because it's inherently unfair, and that's for a path that's already slow.
Oh! But we can simply piggyback mn_invalidate_lock and use a per-VM GPC sequence counter, then the end() path doesn't need to re-walk GPCs. That will regress GPCs in the presense of unrelated mmu_notifier events, but we already know that's a lurking problem, and we know how to fix it[*] (I think).
There's a "sleeping while atomic" problem that will occur on RT if/when GPCs take mn_invalidate_lock when refreshing the PFN, thanks to IRQs being disabled, but that's not a problem until we implement the range-based optimization, because until then, the reader side can be lockless.
[*] https://lore.kernel.org/all/[email protected]
Not yet tested, and I think I'd want to land this at the same time as the range-based optimzation, to avoid regressing "normal" GPC usage (only AWS uses the HVA-based Xen API, so getting a fix to LTS kernels isn't a priority, AFAIK), but this?
include/linux/kvm_host.h +2 -0
@@ -854,6 +854,8 @@ struct kvm {gfn_t mmu_invalidate_range_start;gfn_t mmu_invalidate_range_end; +unsigned long gpc_invalidate_seq; +struct list_head devices;u64 manual_dirty_log_protect;struct dentry *debugfs_dentry;
virt/kvm/kvm_main.c +9 -0
@@ -812,6 +812,15 @@ static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn,/* Pairs with the increment in range_start(). */ spin_lock (&kvm->mn_invalidate_lock); + kvm->gpc_invalidate_seq++; + +/* +* As with the MMU sequence counter and mmu_invalidate_in_progress, the +* GPC sequence increase must be visible before the invalidate count +* goes to zero. Pairs with the smp_rmb() in gpc_invalidate_retry(). +*/ +smp_wmb (); +if (!WARN_ON_ONCE (!kvm->mn_active_invalidate_count)) --kvm->mn_active_invalidate_count; wake = !kvm->mn_active_invalidate_count;
virt/kvm/pfncache.c +8 -8
@@ -124,7 +124,7 @@ static void gpc_unmap(kvm_pfn_t pfn, void *khva)#endif } -static inline bool mmu_notifier_retry_cache (struct kvm *kvm,unsigned long mmu_seq) +static inline bool gpc_invalidate_retry (struct kvm *kvm,unsigned long mmu_seq) {/* * mn_active_invalidate_count acts for all intents and purposes
@@ -142,14 +142,14 @@ static inline bool mmu_notifier_retry_cache(struct kvm *kvm, unsigned long mmu_sreturn true;/* -* Ensure mn_active_invalidate_count is read before -* mmu_invalidate_seq. This pairs with the smp_wmb() in -* mmu_notifier_invalidate_range_end() to guarantee either the -* old (non-zero) value of mn_active_invalidate_count or the -* new (incremented) value of mmu_invalidate_seq is observed. +* Ensure mn_active_invalidate_count is read before gpc_invalidate_seq. +* This pairs with the smp_wmb() in +* kvm_mmu_notifier_invalidate_range_end() to guarantee either the +* old (non-zero) value of mn_active_invalidate_count or the new +* (incremented) value of gpc_invalidate_seq is observed. */ smp_rmb (); -return kvm->mmu_invalidate_seq != mmu_seq; +return kvm->gpc_invalidate_seq != mmu_seq; }static kvm_pfn_t hva_to_pfn_retry (struct gfn_to_pfn_cache *gpc)
@@ -232,7 +232,7 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)* attempting to refresh. */ WARN_ON_ONCE (gpc->valid); - }while (mmu_notifier_retry_cache (gpc->kvm, mmu_seq)); + }while (gpc_invalidate_retry (gpc->kvm, mmu_seq)); gpc->valid = true; gpc->pfn = new_pfn;
Hi,
Quoted 4 lines · Sean Christopherson
Quoted 2 lines · Jeongjun Park
HVA-based
gfn_to_pfncaches are not necessarily backed by a KVM memslot. When an MMU notifier invalidation targets such an HVA, KVM's global
[....]
Quoted 6 lines
Quoted 3 lines
Reported-by [email protected]Given that there's no reproducer, how did you test this?
For some unknown reason, repro was not generated in syzbot, so I analyzed the KASAN logs separately to predict a sequence similar to the race sequence you suggested, and then wrote the repro code myself to test it.
Quoted 9 lines
Quoted 9 lines
Fixes b9220d32799a ("KVM: x86/xen: allow shared_info to be mapped by fixed HVA")Signed-off-by Jeongjun Park <[email protected]>--- include/linux/kvm_types.h | 1 + virt/kvm/pfncache.c | 42 ++++++++++++++++++++++++++++++++------- 2 files changed, 36 insertions(+), 7 deletions(-)include/linux/kvm_types.h +0
-0
[....]
Quoted 11 lines
include/linux/kvm_types.h void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start, · line 48
Quoted 6 lines
/*include/linux/kvm_types.h +0
-1@@ -45,9 +62,11 @@ void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start,*/ write_lock_irq (&gpc->lock); - if (gpc->valid && !is_error_noslot_pfn(gpc->pfn) &&Unless I'm forgetting/missing something, it should be impossible for
gpc->pfnto be garbage ifgpc->validis true. As a follow-up patch, turn that into aWARN_ON_ONCE()? Assuming I'm not missing something...
I agree. As far as I know, if gpc->valid is true, !is_error_noslot_pfn() cannot be false. Since this check exists in the existing code, I considered keeping it in gpc_hva_is_valid(), but the method you suggested seems more accurate, so I think it would be better to just remove !is_error_noslot_pfn().
Quoted 3 lines
Quoted 3 lines
include/linux/kvm_types.h +2
-1- gpc->uhva >= start && gpc->uhva < end) +if (gpc_should_invalidate (gpc, start, end)) { +if (kvm_gpc_is_hva_active (gpc))
[....]
Quoted 11 lines
Oh! But we can simply piggyback
mn_invalidate_lockand use a per-VM GPC sequence counter, then theend()path doesn't need to re-walk GPCs. That will regress GPCs in the presense of unrelatedmmu_notifierevents, but we already know that's a lurking problem, and we know how to fix it[*] (I think).There's a "sleeping while atomic" problem that will occur on RT if/when GPCs take
mn_invalidate_lockwhen refreshing the PFN, thanks to IRQs being disabled, but that's not a problem until we implement the range-based optimization, because until then, the reader side can be lockless.
Quoted 7 lines
[*] https://lore.kernel.org/all/[email protected]
Not yet tested, and I think I'd want to land this at the same time as the range-based optimzation, to avoid regressing "normal" GPC usage (only AWS uses the HVA-based Xen API, so getting a fix to LTS kernels isn't a priority, AFAIK), but this?
The approach you suggested seems most appropriate for resolving this bug. Of course, as you mentioned, modifying it this way would cause a regress when unrelated mmu_notifier events occur; however, since resolving this issue would likely require a significant number of patches, this method appears more suitable for the time being.
I'll send a v2 patch reflecting these changes soon.
include/linux/kvm_host.h struct kvm { · line 857
Quoted 9 lines
include/linux/kvm_host.h +1
-0@@ -854,6 +854,8 @@ struct kvm {gfn_t mmu_invalidate_range_start;gfn_t mmu_invalidate_range_end; +unsigned long gpc_invalidate_seq;
[....]
Regards,
Jeongjun Park
Sean Christopherson
reviewerQuoted 12 lines · Jeongjun Park
Hi,
Quoted 4 lines · Sean Christopherson
Quoted 2 lines · Jeongjun Park
HVA-based
gfn_to_pfncaches are not necessarily backed by a KVM memslot. When an MMU notifier invalidation targets such an HVA, KVM's global[....]
Quoted 4 lines
Quoted 3 lines
Reported-by [email protected]
Quoted 6 lines
Quoted 2 lines
Given that there's no reproducer, how did you test this?
For some unknown reason, repro was not generated in syzbot, so I analyzed the KASAN logs separately to predict a sequence similar to the race sequence you suggested, and then wrote the repro code myself to test it.
Can you provide the reproducer? That would be super helpful for reviewing and iterating on this fix, and probably as a regression test for future changes as well.
Thanks!
(Spent the day prodding at AI on this while I'm in meetings; I've let it draft the below for me which isn't entirely my voice but I'm mostly OK with it...)
Quoted 3 lines · Sean Christopherson
Can you provide the reproducer? That would be super helpful for reviewing and iterating on this fix, and probably as a regression test for future changes as well.
There's one at https://david.woodhou.se/xen_shinfo_race.c now — no memslots, just five processes racing KVM_XEN_HVM_SET_ATTR(SHARED_INFO_HVA) against MAP_FIXED shmem remaps over the cached HVA. It fires the syzbot splat within five seconds in a QEMU guest running an unfixed kernel. Tracing confirms the
diagnosis: complete invalidations (start *and* end) land entirely within the window where hva_to_pfn_retry() has dropped gpc->lock for the GUP, so by the time it re-checks, mn_active_invalidate_count is back to zero and mmu_invalidate_seq never moved because nothing overlapped a memslot. (Note syzbot also found its own reproducer last week¹.)
The same bug is also reachable through the vcpu_info cache, where the stale mapping is then dereferenced by kvm_setup_guest_pvclock() on the next KVM_RUN — which is syzbot+fb7c2dd166d3ea63df2a, the one Heitor posted a sibling-GPC workaround for². Reproducer for that variant:
https://david.woodhou.se/vcpu_info_race.c
Since you said you'd want to land the sequence fix together with the range-based optimization: I've taken a shot at that. The branch at
https://git.infradead.org/?p=users/dwmw2/linux.git;a=shortlog;h=refs/heads/pfncache-fixes
has your gpc_invalidate_seq patch (with the sampling side in hva_to_pfn_retry() also switched over, and both syzbot tags; Cc:
stable and deliberately minimal for backporting) followed by your Oct 2024 patches³ rebased on top, with the range check's second arm now checking gpc_invalidate_seq so it catches invalidations which complete within the refresh window without touching a memslot. I've dropped the sleep-based wait for now.
David Woodhouse (2):
KVM: pfncache: use a dedicated invalidation sequence for cache refresh
KVM: pfncache: Implement range-based invalidation check for hva_to_pfn_retry()
Sean Christopherson (3):
KVM: pfncache: Snapshot invalidation sequence immediately before hva_to_pfn()
KVM: pfncache: Wait for in-progress invalidations to complete during refresh
KVM: pfncache: Add lockless checking of cache invalidation events
include/linux/kvm_host.h | 4 ++ virt/kvm/kvm_main.c | 17 +++++- virt/kvm/pfncache.c | 144 +++++++++++++++++++++++++++++++---------------- 3 files changed, 114 insertions(+), 51 deletions(-)
Both reproducers fire in seconds on the unfixed base and survive ten-minute soaks under KASAN+lockdep with the first patch alone and with the full series.
On the RT concern with taking mn_invalidate_lock across the GPC walk:
I don't think the "sleeping while atomic" problem exists once the GPC locks are no longer irqsave, which v2 of the rwlock_t series⁴ already does.
¹ https://lore.kernel.org/all/[email protected]/
² https://lore.kernel.org/all/[email protected]/
³ https://lore.kernel.org/all/[email protected]/
⁴ https://lore.kernel.org/all/[email protected]/