KVM: guest_memfd: Avoid deadlock in error_remove_folio()
Subsystems (2)
PATCH v3 1/1 KVM: guest_memfd: Avoid deadlock in error_remove_folio()
From: Hao Zhang <[email protected]>
memory_failure() invokes ->error_remove_folio() while holding the poisoned folio lock. KVM's guest_memfd callback then takes mapping->invalidate_lock for read before invalidating guest mappings.
This can deadlock with paths that hold mapping->invalidate_lock for write and then try to lock the same folio, e.g. truncation from MADV_REMOVE or FALLOC_FL_PUNCH_HOLE. Because rwsems are fair, the problem is not limited to guest_memfd punch-hole: a pending writer can also block future readers, so ->error_remove_folio() must not block on the invalidate lock while holding the poisoned folio lock.
Use filemap_invalidate_trylock_shared() in kvm_gmem_error_folio(). If the invalidate lock is contended, fail the recovery instead of potentially deadlocking the memory-failure path.
---
Changes in v3:
- Drop the guest_memfd-private bindings_lock approach from v2 and go back to the trylock stopgap from v1, as suggested by Sean.
- Rewrite the changelog and code comment to describe the problematic lock dependency as the poisoned folio lock vs. mapping->invalidate_lock, and avoid implying that mf_mutex is part of the actual deadlock.
- Account for rwsem fairness: a pending writer can block future readers, so the issue is not limited to guest_memfd punch-hole.
Link to v2: https://lore.kernel.org/all/[email protected]/
virt/kvm/guest_memfd.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
virt/kvm/guest_memfd.c +11 -1
@@ -500,7 +500,17 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol {pgoff_t start, end; -filemap_invalidate_lock_shared (mapping); +/* +* memory_failure() invokes ->error_remove_folio() while holding the +* poisoned folio lock. Do not block on mapping->invalidate_lock, as +* a writer that already holds the invalidate lock can be waiting for +* the same folio, e.g. truncation from MADV_REMOVE/FALLOC_FL_PUNCH_HOLE. +* +* If the invalidate lock is contended, fail the recovery instead of +* risking a deadlock in the memory-failure path. +*/ +if (!filemap_invalidate_trylock_shared (mapping)) +return -EBUSY ; start = folio->index; end = start +folio_nr_pages (folio);
base-commit: 58717b2a1365d06c8c64b72aa948541b53fe31eb
Signature · 1 line
2.39.2
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Asynchronous memslot unbinding can leak kvm->mmu_invalidate_in_progress when racing with kvm_gmem_error_folio(), causing a permanent KVM page fault hang. --
--- Patch [1]: [PATCH v3] KVM: guest_memfd: Avoid deadlock in error_remove_folio() --- commit db14f829c7993aef911f52935c9113e37c1a49ea
Author: Hao Zhang <[email protected]>
KVM: guest_memfd: Avoid deadlock in error_remove_folio()
memory_failure() invokes ->error_remove_folio() while holding the poisoned folio lock. KVM's guest_memfd callback then takes mapping->invalidate_lock for read before invalidating guest mappings. This can deadlock with paths that hold mapping->invalidate_lock for write.
Quoted 13 lines
virt/kvm/guest_memfd.c +4
-1@@ -500,7 +500,17 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol {pgoff_t start, end; -filemap_invalidate_lock_shared (mapping); +/* +* memory_failure() invokes ->error_remove_folio() while holding the +* poisoned folio lock. Do not block on mapping->invalidate_lock, as +* a writer that already holds the invalidate lock can be waiting for
virt/kvm/guest_memfd.c static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol · line 516
Quoted 10 lines
virt/kvm/guest_memfd.c +7
-0+* the same folio, e.g. truncation from MADV_REMOVE/FALLOC_FL_PUNCH_HOLE. +* +* If the invalidate lock is contended, fail the recovery instead of +* risking a deadlock in the memory-failure path. +*/ +if (!filemap_invalidate_trylock_shared (mapping)) +return -EBUSY ; start = folio->index; end = start +folio_nr_pages (folio);
[Severity: High]
This is a pre-existing issue, but can this function race with memslot unbinding and permanently leak kvm->mmu_invalidate_in_progress, causing a KVM page fault hang?
If userspace closes the guest_memfd file descriptor, dropping its refcount to 0, and concurrently deletes the memslot, kvm_gmem_unbind() gets called. Because the file refcount is 0, it takes the lockless path and modifies f->bindings without holding mapping->invalidate_lock:
virt/kvm/guest_memfd.c:kvm_gmem_unbind() {
...
if (!file) {
__kvm_gmem_unbind(slot, slot->gmem.file->private_data);
return;
}
...
}Concurrently, if a hardware memory failure triggers kvm_gmem_error_folio(), it acquires the invalidate_lock via filemap_invalidate_trylock_shared(). It then calls kvm_gmem_invalidate_start(), which finds the memslot and increments the invalidation counter:
virt/kvm/guest_memfd.c:__kvm_gmem_invalidate_start() {
...
if (!found_memslot) {
found_memslot = true;
KVM_MMU_LOCK(kvm);
kvm_mmu_invalidate_start(kvm);
}
...
}But because __kvm_gmem_unbind() concurrently removes the binding from f->bindings without synchronization against invalidate_lock readers, __kvm_gmem_invalidate_end() then fails to find the binding and skips the decrement:
virt/kvm/guest_memfd.c:__kvm_gmem_invalidate_end() {
...
if (xa_find(&f->bindings, &start, end - 1, XA_PRESENT)) {
KVM_MMU_LOCK(kvm);
kvm_mmu_invalidate_end(kvm);
KVM_MMU_UNLOCK(kvm);
}
...
}Could this permanent leak cause the KVM page fault retry protocol to unconditionally return true, throwing the vCPU into an infinite page fault loop?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Sean Christopherson
maintainerQuoted 4 lines · Hao Zhang
memory_failure()invokes ->error_remove_folio() while holding the poisoned folio lock. KVM'sguest_memfdcallback then takesmapping->invalidate_lockfor read before invalidating guest mappings.
Quoted 8 lines
This can deadlock with paths that hold
mapping->invalidate_lockfor write and then try to lock the same folio, e.g. truncation fromMADV_REMOVEorFALLOC_FL_PUNCH_HOLE. Because rwsems are fair, the problem is not limited toguest_memfdpunch-hole: a pending writer can also block future readers, so ->error_remove_folio() must not block on the invalidate lock while holding the poisoned folio lock.[...]
(finally) Applied to kvm-x86 coco, thanks!
[1/1] KVM: guest_memfd: Avoid deadlock in error_remove_folio()
https://github.com/kvm-x86/linux/commit/ad06cf5338bc