KVM: arm64: vgic-its: Serialize translation cache invalidation under its_lock
Subsystems (3)
PATCH v2 1/1 KVM: arm64: vgic-its: Serialize translation cache invalidation under its_lock
vgic_its_invalidate_cache() walks the per-ITS translation cache with xa_for_each() and drops the cache's reference on each entry with vgic_put_irq().
It must be called with its_lock held. The ITS command handlers and the ITS teardown path hold it, but two paths that also invalidate the cache do
not: the GITS_CTLR write path holds only cmd_lock, and the path that clears EnableLPIs in a redistributor's GICR_CTLR holds neither lock. Two contexts without a common lock, such as two vCPUs clearing EnableLPIs or an EnableLPIs clear racing an ITS command, can drain the same cache at once. If both observe an entry, erase it and then put it, the single reference the cache holds on that entry is dropped more than once, and the entry can be freed while an ITE still maps it.
Take its_lock in the two paths that lacked it: the GITS_CTLR write path and vgic_its_invalidate_all_caches(), which clears EnableLPIs. Since vgic_its_invalidate_all_caches() now takes a mutex, it can no longer walk kvm->devices under rcu_read_lock(), so walk it under kvm->lock instead. With its_lock held across every invalidation, each entry is erased and put by a single context, so the cache reference is dropped exactly once.
---
Changes in v2:
- Serialize the invalidation under its_lock as suggested by Oliver, instead of v1's gating of the put on the xa_erase() return value.
- v1: https://lore.kernel.org/all/ah2c5lu4JbUg7dj-@v4bel/
--- arch/arm64/kvm/vgic/vgic-its.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +8 -5
@@ -596,6 +596,8 @@ static void vgic_its_invalidate_cache(struct vgic_its *its)struct vgic_irq *irq;unsigned long idx; +lockdep_assert_held (&its->its_lock); +xa_for_each (&its->translation_cache, idx, irq) {xa_erase (&its->translation_cache, idx);vgic_put_irq (kvm, irq);
@@ -607,17 +609,16 @@ void vgic_its_invalidate_all_caches(struct kvm *kvm)struct kvm_device *dev;struct vgic_its *its; -rcu_read_lock (); +guard (mutex)(&kvm->lock); -list_for_each_entry_rcu (dev, &kvm->devices, vm_node) { +list_for_each_entry (dev, &kvm->devices, vm_node) {if (dev->ops != &kvm_arm_vgic_its_ops)continue ; its = dev->private; +guard (mutex)(&its->its_lock);vgic_its_invalidate_cache (its); } - -rcu_read_unlock (); } int vgic_its_resolve_lpi(struct kvm *kvm, struct vgic_its *its,
@@ -1725,8 +1726,10 @@ static void vgic_mmio_write_its_ctlr(struct kvm *kvm, struct vgic_its *its, goto out; its->enabled = !!(val &GITS_CTLR_ENABLE ); -if (!its->enabled) +if (!its->enabled) { +guard (mutex)(&its->its_lock);vgic_its_invalidate_cache (its); + }/* * Try to process any pending commands. This function bails out early
Signature · 1 line
2.43.0
Oliver Upton
maintainerHi Hyunwoo,
Thanks for respinning.
arch/arm64/kvm/vgic/vgic-its.c void vgic_its_invalidate_all_caches(struct kvm *kvm) · line 612
Quoted 6 lines · Hyunwoo Kim
arch/arm64/kvm/vgic/vgic-its.c +1
-1@@ -607,17 +609,16 @@ void vgic_its_invalidate_all_caches(struct kvm *kvm)struct kvm_device *dev;struct vgic_its *its; -rcu_read_lock (); +guard (mutex)(&kvm->lock);
Urgh, entirely my mistake but we can get here holding the vcpu->mutex which is an ordering bug. We can defer the locking cleanup as a long term fix (which I still want) and take what you had before in v1.
Sorry for the noise.
Thanks,
Oliver
Quoted 29 lines
arch/arm64/kvm/vgic/vgic-its.c +5
-4-list_for_each_entry_rcu (dev, &kvm->devices, vm_node) { +list_for_each_entry (dev, &kvm->devices, vm_node) {if (dev->ops != &kvm_arm_vgic_its_ops)continue ; its = dev->private; +guard (mutex)(&its->its_lock);vgic_its_invalidate_cache (its); } - -rcu_read_unlock (); } int vgic_its_resolve_lpi(struct kvm *kvm, struct vgic_its *its,@@ -1725,8 +1726,10 @@ static void vgic_mmio_write_its_ctlr(struct kvm *kvm, struct vgic_its *its, goto out; its->enabled = !!(val &GITS_CTLR_ENABLE ); -if (!its->enabled) +if (!its->enabled) { +guard (mutex)(&its->its_lock);vgic_its_invalidate_cache (its); + }/* * Try to process any pending commands. This function bails out early Signature · 1 line
2.43.0