KVM: Don't let dirty ring resets pass dirty_index
cover KVM: Don't let dirty ring resets pass dirty_index
Fix a userspace-triggerable WARN in kvm_dirty_ring_push() reported by syzbot [1], and tidy up the memory ordering and annotations around the dirty ring indices while in the area.
Patch 1 is the fix. The dirty ring is mapped writable into userspace, so userspace can set KVM_DIRTY_GFN_F_RESET on entries that KVM never pushed. KVM_RESET_DIRTY_RINGS then advances reset_index past dirty_index, the u32 math in kvm_dirty_ring_used() underflows, and the next push hits WARN_ON_ONCE(kvm_dirty_ring_full(ring)). Stop resetting entries once reset_index catches up to dirty_index, and bump dirty_index before publishing the DIRTY flag, so that a VMM that reaps the ring while the vCPU is running never has a legitimately harvested entry skipped.
Patch 2 converts the DIRTY side of the flags handshake with userspace to smp_store_release(), to match the acquire/release semantics already used for the RESET side. No functional change intended.
Patch 3 is optional. It marks the remaining concurrent accesses to the ring indices: WRITE_ONCE() for reset_index, which the vCPU reads with READ_ONCE(), and READ_ONCE() for the reads of the other task's index in the kvm_dirty_ring_push and kvm_dirty_ring_reset tracepoints. I'm not aware of any KCSAN reports or miscompiles, so this mostly documents the concurrency and ensures the compiler can't tear the accesses. Feel free to drop it if that is appropriate.
Patch 4 adds a regression test for patch 1 to dirty_log_test. It fails without patch 1 (KVM resets two entries it never pushed), and passes with the series applied, including with a minimally sized ring so that the vCPU repeatedly exits with KVM_EXIT_DIRTY_RING_FULL while the ring is being reaped.
[1] https://lore.kernel.org/all/[email protected]
James Houghton (4):
KVM: Don't reset dirty ring entries that KVM hasn't pushed
KVM: Use smp_store_release() to mark dirty ring entries as dirtied
KVM: Annotate concurrent accesses to dirty ring indices
KVM: selftests: Verify KVM doesn't reset unused dirty ring entries
include/trace/events/kvm.h | 4 ++-- tools/testing/selftests/kvm/dirty_log_test.c | 18 +++++++++++++++ virt/kvm/dirty_ring.c | 23 +++++++++++++++----- 3 files changed, 37 insertions(+), 8 deletions(-)
base-commit: 666539eecdf4db623f5e3c3e55cf9f2dfad24aa4
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
PATCH 1/4 KVM: Don't reset dirty ring entries that KVM hasn't pushed
When resetting a vCPU's dirty ring, stop if reset_index has caught up to dirty_index instead of relying solely on the RESET flag of the next entry. The ring is mapped writable into userspace, so userspace can set KVM_DIRTY_GFN_F_RESET on entries that KVM never pushed, in which case KVM advances reset_index past dirty_index. The u32 math in kvm_dirty_ring_used() then underflows, and the next push hits the WARN_ON_ONCE() in kvm_dirty_ring_push():
WARNING: virt/kvm/dirty_ring.c:225 at kvm_dirty_ring_push+0x2b2/0x5e0 Call Trace: make_spte+0x71a/0xc80 kvm_tdp_mmu_map+0x140e/0x1f40 kvm_tdp_page_fault+0x2f7/0x3a0 kvm_mmu_page_fault+0x1f8/0xc00 vcpu_run+0x4da1/0x7890 kvm_arch_vcpu_ioctl_run+0xe83/0x18b0 kvm_vcpu_ioctl+0xa61/0xfd0
Userspace that scribbles on its own dirty ring gets to keep the pieces, but KVM shouldn't WARN, and keeping reset_index from passing dirty_index is trivial.
To avoid skipping entries that were legitimately harvested while the vCPU is running, bump dirty_index before publishing the DIRTY flag in kvm_dirty_ring_push(), and check dirty_index after the load-acquire of the RESET flag in kvm_dirty_ring_reset(). If KVM observes the RESET flag for an entry, i.e. userspace observed the DIRTY flag, then KVM is guaranteed to also observe the updated dirty_index.
--- virt/kvm/dirty_ring.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)
virt/kvm/dirty_ring.c +15 -3
@@ -140,6 +140,16 @@ int kvm_dirty_ring_reset(struct kvm *kvm, struct kvm_dirty_ring *ring,if (!kvm_dirty_gfn_harvested (entry))break ; +/* +* Don't reset entries that KVM hasn't pushed, e.g. if userspace +* set the RESET flag on an unused entry. Check the number of +* used entries after checking the flag, so that dirty_index is +* up-to-date for entries that were harvested (pairs with the +* ordering in kvm_dirty_ring_push()). +*/ +if (!kvm_dirty_ring_used (ring)) +break ; + next_slot =READ_ONCE (entry->slot); next_offset =READ_ONCE (entry->offset);
@@ -229,12 +239,14 @@ void kvm_dirty_ring_push(struct kvm_vcpu *vcpu, u32 slot, u64 offset) entry->slot = slot; entry->offset = offset;/* -* Make sure the data is filled in before we publish this to -* the userspace program. There's no paired kernel-side reader. +* Bump dirty_index before publishing the entry, so that +* kvm_dirty_ring_reset() sees the new dirty_index if it sees the entry +* as harvested, and make sure the data is filled in before publishing +* the entry to userspace. */ +WRITE_ONCE (ring->dirty_index, ring->dirty_index +1 );smp_wmb ();kvm_dirty_gfn_set_dirtied (entry); - ring->dirty_index++;trace_kvm_dirty_ring_push (ring, slot, offset);if (kvm_dirty_ring_soft_full (ring))
base-commit: 666539eecdf4db623f5e3c3e55cf9f2dfad24aa4
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
PATCH 2/4 KVM: Use smp_store_release() to mark dirty ring entries as dirtied
In reply to James HoughtonPublish dirty ring entries to userspace with smp_store_release() in kvm_dirty_gfn_set_dirtied(), instead of an smp_wmb() in kvm_dirty_ring_push() followed by a plain store in the helper.
Setting the DIRTY flag is the first half of a handshake with userspace:
userspace load-acquires the DIRTY flag, harvests the entry, and store-releases the RESET flag, which KVM then load-acquires in kvm_dirty_gfn_harvested(). Commit 8929bc965964 ("KVM: Use acquire/release semantics when accessing dirty ring GFN state") converted KVM's side of the RESET flag to acquire/release, but left the DIRTY side split between a barrier in the caller and a plain store in the helper. As a result, the helper doesn't convey that its store must be ordered, and the store isn't marked even though userspace can access the flags concurrently.
Now that kvm_dirty_ring_reset() relies on this handshake to observe an up-to-date dirty_index, make the ordering explicit and self-contained in the helper, so that all three flag accessors use matching acquire/release semantics.
No functional change intended.
--- virt/kvm/dirty_ring.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
virt/kvm/dirty_ring.c +1 -2
@@ -95,7 +95,7 @@ static inline void kvm_dirty_gfn_set_invalid(struct kvm_dirty_gfn *gfn)static inline void kvm_dirty_gfn_set_dirtied (struct kvm_dirty_gfn *gfn) { - gfn->flags =KVM_DIRTY_GFN_F_DIRTY ; +smp_store_release (&gfn->flags,KVM_DIRTY_GFN_F_DIRTY ); }static inline bool kvm_dirty_gfn_harvested (struct kvm_dirty_gfn *gfn)
@@ -245,7 +245,6 @@ void kvm_dirty_ring_push(struct kvm_vcpu *vcpu, u32 slot, u64 offset)* the entry to userspace. */ WRITE_ONCE (ring->dirty_index, ring->dirty_index +1 ); -smp_wmb ();kvm_dirty_gfn_set_dirtied (entry);trace_kvm_dirty_ring_push (ring, slot, offset);
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
PATCH 3/4 KVM: Annotate concurrent accesses to dirty ring indices
In reply to James HoughtonA vCPU's dirty ring indices are accessed concurrently by two tasks: the vCPU pushes entries and updates dirty_index, while KVM_RESET_DIRTY_RINGS updates reset_index, and can run at any time (VMMs commonly reap the ring from a separate thread while the vCPU is running).
Use WRITE_ONCE() to update reset_index, which the vCPU reads with READ_ONCE() in kvm_dirty_ring_used(). KCSAN doesn't assume that a plain read-modify-write is atomic, so it can report the increment as a data race.
Use READ_ONCE() to read the other task's index in the kvm_dirty_ring_push and kvm_dirty_ring_reset tracepoints, so that the compiler can't tear or refetch the loads. The values are only used for tracing, but the indices are part of the ring's concurrency design, so mark the reads the same way as kvm_dirty_ring_used().
--- include/trace/events/kvm.h | 4 ++-- virt/kvm/dirty_ring.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-)
include/trace/events/kvm.h +2 -2
@@ -316,7 +316,7 @@ TRACE_EVENT(kvm_dirty_ring_push,TP_fast_assign ( __entry->index = ring->index; __entry->dirty_index = ring->dirty_index; - __entry->reset_index = ring->reset_index; + __entry->reset_index =READ_ONCE (ring->reset_index); __entry->slot = slot; __entry->offset = offset; ),
@@ -340,7 +340,7 @@ TRACE_EVENT(kvm_dirty_ring_reset,TP_fast_assign ( __entry->index = ring->index; - __entry->dirty_index = ring->dirty_index; + __entry->dirty_index =READ_ONCE (ring->dirty_index); __entry->reset_index = ring->reset_index; ),
virt/kvm/dirty_ring.c +1 -1
@@ -156,7 +156,7 @@ int kvm_dirty_ring_reset(struct kvm *kvm, struct kvm_dirty_ring *ring,/* Update the flags to reflect that this GFN is reset */ kvm_dirty_gfn_set_invalid (entry); - ring->reset_index++; +WRITE_ONCE (ring->reset_index, ring->reset_index +1 ); (*nr_entries_reset)++;if (mask) {
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
PATCH 4/4 KVM: selftests: Verify KVM doesn't reset unused dirty ring entries
In reply to James HoughtonBefore running the vCPU in the dirty ring test, mark the first two (unused) entries of the ring as harvested and verify that KVM_RESET_DIRTY_RINGS doesn't reset any entries, i.e. that KVM doesn't advance reset_index beyond dirty_index.
--- tools/testing/selftests/kvm/dirty_log_test.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
tools/testing/selftests/kvm/dirty_log_test.c +18 -0
@@ -375,6 +375,21 @@ static void dirty_ring_collect_dirty_pages(struct kvm_vcpu *vcpu, int slot, "with collected (%u)", cleared, count); } +static void dirty_ring_reset_unused_entries (struct kvm_vcpu *vcpu) +{ +struct kvm_dirty_gfn *dirty_gfns =vcpu_map_dirty_ring (vcpu); + +/* +* Mark unused entries as harvested, and verify KVM doesn't reset any +* entries, as KVM hasn't pushed anything to the ring. KVM will +* overwrite the flags when it pushes to the entries. +*/ +dirty_gfn_set_collected (&dirty_gfns[0 ]); +dirty_gfn_set_collected (&dirty_gfns[1 ]); + +TEST_ASSERT_EQ (kvm_vm_reset_dirty_ring (vcpu->vm),0 ); +} +static void dirty_ring_after_vcpu_run (struct kvm_vcpu *vcpu) {struct kvm_run *run = vcpu->run;
@@ -687,6 +702,9 @@ static void run_test(enum vm_guest_mode mode, void *arg)TEST_ASSERT_EQ (vcpu_stop, false); +if (host_log_mode ==LOG_MODE_DIRTY_RING ) +dirty_ring_reset_unused_entries (vcpu); +kvm_pthread_create (&vcpu_thread,NULL , vcpu_worker, vcpu);for (iteration =1 ; iteration <= p->iterations; iteration++) {
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
[email protected]
AI reviewRe: [PATCH 4/4] KVM: selftests: Verify KVM doesn't reset unused dirty ring entries
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] KVM: selftests: clear RESET flags in dirty_ring_reset_unused_entries() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] KVM: selftests: clear RESET flags in dirty_ring_reset_unused_entries()
--
commit 494a32b29217257f8a046ec9364c749fef56211c
Author: James Houghton <[email protected]>
KVM: selftests: Verify KVM doesn't reset unused dirty ring entries
This commit adds a helper to dirty_log_test that marks the first two unused dirty ring entries as harvested before running the vCPU and asserts that KVM_RESET_DIRTY_RINGS resets zero entries.
Quoted 12 lines
tools/testing/selftests/kvm/dirty_log_test.c +4
-0@@ -375,6 +375,21 @@ static void dirty_ring_collect_dirty_pages(struct kvm_vcpu *vcpu, int slot, "with collected (%u)", cleared, count); } +static void dirty_ring_reset_unused_entries (struct kvm_vcpu *vcpu) +{ +struct kvm_dirty_gfn *dirty_gfns =vcpu_map_dirty_ring (vcpu); +
tools/testing/selftests/kvm/dirty_log_test.c static void dirty_ring_collect_dirty_pages(struct kvm_vcpu *vcpu, int slot, · line 391
Quoted 10 lines
tools/testing/selftests/kvm/dirty_log_test.c +10
-0+/* +* Mark unused entries as harvested, and verify KVM doesn't reset any +* entries, as KVM hasn't pushed anything to the ring. KVM will +* overwrite the flags when it pushes to the entries. +*/ +dirty_gfn_set_collected (&dirty_gfns[0 ]); +dirty_gfn_set_collected (&dirty_gfns[1 ]); + +TEST_ASSERT_EQ (kvm_vm_reset_dirty_ring (vcpu->vm),0 ); +}
[Severity: Low]
Should dirty_ring_reset_unused_entries() clear flags back to 0 on dirty_gfns[0] and dirty_gfns[1] after kvm_vm_reset_dirty_ring() returns?
When kvm_dirty_ring_used(ring) is 0, kvm_dirty_ring_reset() breaks out of its loop before calling kvm_dirty_gfn_set_invalid(entry), leaving both entries with KVM_DIRTY_GFN_F_RESET set when vcpu_worker starts. In kvm_dirty_ring_push(), KVM increments ring->dirty_index before overwriting entry->flags with KVM_DIRTY_GFN_F_DIRTY:
virt/kvm/dirty_ring.c:kvm_dirty_ring_push() {
...
WRITE_ONCE(ring->dirty_index, ring->dirty_index + 1);
kvm_dirty_gfn_set_dirtied(entry);
...
}If the host reaping loop runs while the vCPU is pushing its first two dirty ring entries (for example when reap_i is 0), kvm_dirty_ring_reset() can observe the stale KVM_DIRTY_GFN_F_RESET flag together with the incremented ring->dirty_index and reset an entry that userspace never collected:
host vcpu_worker
---- -----------dirty_ring_collect_dirty_pages()
dirty_ring_collect_one()
/* flags == KVM_DIRTY_GFN_F_RESET,
stops with count == 0 */
kvm_dirty_ring_push()
ring->dirty_index++ kvm_vm_reset_dirty_ring()
kvm_dirty_ring_reset()
kvm_dirty_gfn_harvested(entry) kvm_dirty_ring_used(ring) > 0
ring->reset_index++
cleared++
kvm_dirty_gfn_set_dirtied(entry)
TEST_ASSERT(cleared == count) <-- fails (1 != 0)
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Re: [PATCH 4/4] KVM: selftests: Verify KVM doesn't reset unused dirty ring entries
Quoted 9 lines · On 17 November 2026, [email protected] wrote:
[Severity: Low]
Shoulddirty_ring_reset_unused_entries()clear flags back to 0 on dirty_gfns[0] and dirty_gfns[1] afterkvm_vm_reset_dirty_ring()returns?When
kvm_dirty_ring_used(ring)is 0,kvm_dirty_ring_reset()breaks out of its loop before callingkvm_dirty_gfn_set_invalid(entry), leaving both entries withKVM_DIRTY_GFN_F_RESETset whenvcpu_workerstarts. Inkvm_dirty_ring_push(), KVM incrementsring->dirty_indexbefore overwritingentry->flagswithKVM_DIRTY_GFN_F_DIRTY:
[...]
Quoted 4 lines
If the host reaping loop runs while the vCPU is pushing its first two dirty ring entries (for example when
reap_iis 0),kvm_dirty_ring_reset()can observe the staleKVM_DIRTY_GFN_F_RESETflag together with the incrementedring->dirty_indexand reset an entry that userspace never collected:
Yeah, this is possible. Sashiko's suggestion is the right one, I think.
Sean/Paolo, if nothing else in the series needs a v2, would you mind squashing in the below when applying? Otherwise I'll fold it into v2.
Thanks!
---
tools/testing/selftests/kvm/dirty_log_test.c +9 -2
@@ -381,13 +381,20 @@ static void dirty_ring_reset_unused_entries(struct kvm_vcpu *vcpu)/* * Mark unused entries as harvested, and verify KVM doesn't reset any -* entries, as KVM hasn't pushed anything to the ring. KVM will -* overwrite the flags when it pushes to the entries. +* entries, as KVM hasn't pushed anything to the ring. */ dirty_gfn_set_collected (&dirty_gfns[0 ]);dirty_gfn_set_collected (&dirty_gfns[1 ]);TEST_ASSERT_EQ (kvm_vm_reset_dirty_ring (vcpu->vm),0 ); + +/* +* Clear the bogus RESET flags, otherwise a reset that runs while KVM +* is pushing to the entries could reset them before userspace has +* collected them. +*/ + dirty_gfns[0 ].flags =0 ; + dirty_gfns[1 ].flags =0 ; }static void dirty_ring_after_vcpu_run (struct kvm_vcpu *vcpu)