KVM: arm64: vgic-its: Drop the translation cache reference only for the erased entry
PATCH 1/1 KVM: arm64: vgic-its: Drop the translation cache reference only for the erased entry
vgic_its_invalidate_cache() walks the per-ITS translation cache with xa_for_each() and drops the cache's reference on each entry with vgic_put_irq(). It puts the iterated pointer, though, rather than the value returned by xa_erase().
The function is called from contexts that do not exclude one another: the ITS command handlers hold its_lock, the GITS_CTLR write path holds cmd_lock, and the path that clears EnableLPIs in a redistributor's GICR_CTLR holds neither. Two or more of them can drain the same cache concurrently, and if each one observes the same entry, erases it and then puts it, the single reference the cache holds on that entry is dropped more than once. The entry can then be freed while an ITE still maps it.
xa_erase() is atomic and returns the previous entry, so put only the entry that this context actually removed. The cache reference is then dropped exactly once per entry even when the invalidations run concurrently, and the behavior is unchanged when only one context runs.
--- arch/arm64/kvm/vgic/vgic-its.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +4 -2
@@ -597,8 +597,10 @@ static void vgic_its_invalidate_cache(struct vgic_its *its)unsigned long idx;xa_for_each (&its->translation_cache, idx, irq) { -xa_erase (&its->translation_cache, idx); -vgic_put_irq (kvm, irq); +/* Only the context that erases the entry drops its cache ref. */ + irq =xa_erase (&its->translation_cache, idx); +if (irq) +vgic_put_irq (kvm, irq); } }
Signature · 1 line
2.43.0
Oliver Upton
maintainerHi Hyunwoo,
Nice find.
Quoted 13 lines · Hyunwoo Kim
vgic_its_invalidate_cache()walks the per-ITS translation cache withxa_for_each()and drops the cache's reference on each entry withvgic_put_irq(). It puts the iterated pointer, though, rather than the value returned byxa_erase().The function is called from contexts that do not exclude one another: the ITS command handlers hold
its_lock, theGITS_CTLRwrite path holdscmd_lock, and the path that clears EnableLPIs in a redistributor'sGICR_CTLRholds neither. Two or more of them can drain the same cache concurrently, and if each one observes the same entry, erases it and then puts it, the single reference the cache holds on that entry is dropped more than once. The entry can then be freed while an ITE still maps it.
Quoted 4 lines
xa_erase()is atomic and returns the previous entry, so put only the entry that this context actually removed. The cache reference is then dropped exactly once per entry even when the invalidations run concurrently, and the behavior is unchanged when only one context runs.
Next time:
Quoted 12 lines
Fixes 8201d1028caa ("KVM: arm64: vgic-its: Maintain a translation cache per ITS")Signed-off-by Hyunwoo Kim <[email protected]>--- arch/arm64/kvm/vgic/vgic-its.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)arch/arm64/kvm/vgic/vgic-its.c +0
-0@@ -597,8 +597,10 @@ static void vgic_its_invalidate_cache(struct vgic_its *its)unsigned long idx;
arch/arm64/kvm/vgic/vgic-its.c static void vgic_its_invalidate_cache(struct vgic_its *its) · line 605
Quoted 10 lines
arch/arm64/kvm/vgic/vgic-its.c +4
-2xa_for_each (&its->translation_cache, idx, irq) { -xa_erase (&its->translation_cache, idx); -vgic_put_irq (kvm, irq); +/* Only the context that erases the entry drops its cache ref. */ + irq =xa_erase (&its->translation_cache, idx); +if (irq) +vgic_put_irq (kvm, irq); } }
This definitely works but TBH I'd rather just plug the subtle race and do invalidations behind the its_lock since it already nests with the cmd_lock.
Could you give this a spin?
arch/arm64/kvm/vgic/vgic-its.c +8 -5
@@ -596,6 +596,8 @@ static void vgic_its_invalidate_cache(struct vgic_its *its)struct vgic_irq *irq;unsigned long idx; +lockdep_assert_held (&its->its_lock); +xa_for_each (&its->translation_cache, idx, irq) {xa_erase (&its->translation_cache, idx);vgic_put_irq (kvm, irq);
@@ -607,17 +609,16 @@ void vgic_its_invalidate_all_caches(struct kvm *kvm)struct kvm_device *dev;struct vgic_its *its; -rcu_read_lock (); +guard (mutex)(&kvm->lock); -list_for_each_entry_rcu (dev, &kvm->devices, vm_node) { +list_for_each_entry (dev, &kvm->devices, vm_node) {if (dev->ops != &kvm_arm_vgic_its_ops)continue ; its = dev->private; +guard (mutex)(&its->its_lock);vgic_its_invalidate_cache (its); } - -rcu_read_unlock (); } int vgic_its_resolve_lpi(struct kvm *kvm, struct vgic_its *its,
@@ -1725,8 +1726,10 @@ static void vgic_mmio_write_its_ctlr(struct kvm *kvm, struct vgic_its *its, goto out; its->enabled = !!(val &GITS_CTLR_ENABLE ); -if (!its->enabled) +if (!its->enabled) { +guard (mutex)(&its->its_lock);vgic_its_invalidate_cache (its); + }/* * Try to process any pending commands. This function bails out early
--
Thanks,
Oliver
Quoted 40 lines · Oliver Upton
Hi Hyunwoo,
Nice find.
Quoted 13 lines · Hyunwoo Kim
vgic_its_invalidate_cache()walks the per-ITS translation cache withxa_for_each()and drops the cache's reference on each entry withvgic_put_irq(). It puts the iterated pointer, though, rather than the value returned byxa_erase().The function is called from contexts that do not exclude one another: the ITS command handlers hold
its_lock, theGITS_CTLRwrite path holdscmd_lock, and the path that clears EnableLPIs in a redistributor'sGICR_CTLRholds neither. Two or more of them can drain the same cache concurrently, and if each one observes the same entry, erases it and then puts it, the single reference the cache holds on that entry is dropped more than once. The entry can then be freed while an ITE still maps it.Quoted 4 lines
xa_erase()is atomic and returns the previous entry, so put only the entry that this context actually removed. The cache reference is then dropped exactly once per entry even when the invalidations run concurrently, and the behavior is unchanged when only one context runs.Next time:
Quoted 13 lines
Fixes 8201d1028caa ("KVM: arm64: vgic-its: Maintain a translation cache per ITS")Signed-off-by Hyunwoo Kim <[email protected]>--- arch/arm64/kvm/vgic/vgic-its.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)arch/arm64/kvm/vgic/vgic-its.c +0
-0@@ -597,8 +597,10 @@ static void vgic_its_invalidate_cache(struct vgic_its *its)unsigned long idx;
Quoted 13 lines
arch/arm64/kvm/vgic/vgic-its.c static void vgic_its_invalidate_cache(struct vgic_its *its) · line 605
Quoted 9 lines
arch/arm64/kvm/vgic/vgic-its.c +4
-2xa_for_each (&its->translation_cache, idx, irq) { -xa_erase (&its->translation_cache, idx); -vgic_put_irq (kvm, irq); +/* Only the context that erases the entry drops its cache ref. */ + irq =xa_erase (&its->translation_cache, idx); +if (irq) +vgic_put_irq (kvm, irq); } }This definitely works but TBH I'd rather just plug the subtle race and do invalidations behind the
its_locksince it already nests with thecmd_lock.
Thank you for the review.
Quoted 2 lines
Could you give this a spin?
After testing, I've confirmed that this patch approach works well too.
Shall I submit v2 based on this fix?
Quoted 49 lines
arch/arm64/kvm/vgic/vgic-its.c +8
-6@@ -596,6 +596,8 @@ static void vgic_its_invalidate_cache(struct vgic_its *its)struct vgic_irq *irq;unsigned long idx; +lockdep_assert_held (&its->its_lock); +xa_for_each (&its->translation_cache, idx, irq) {xa_erase (&its->translation_cache, idx);vgic_put_irq (kvm, irq);@@ -607,17 +609,16 @@ void vgic_its_invalidate_all_caches(struct kvm *kvm)struct kvm_device *dev;struct vgic_its *its; -rcu_read_lock (); +guard (mutex)(&kvm->lock); -list_for_each_entry_rcu (dev, &kvm->devices, vm_node) { +list_for_each_entry (dev, &kvm->devices, vm_node) {if (dev->ops != &kvm_arm_vgic_its_ops)continue ; its = dev->private; +guard (mutex)(&its->its_lock);vgic_its_invalidate_cache (its); } - -rcu_read_unlock (); } int vgic_its_resolve_lpi(struct kvm *kvm, struct vgic_its *its,@@ -1725,8 +1726,10 @@ static void vgic_mmio_write_its_ctlr(struct kvm *kvm, struct vgic_its *its, goto out; its->enabled = !!(val &GITS_CTLR_ENABLE ); -if (!its->enabled) +if (!its->enabled) { +guard (mutex)(&its->its_lock);vgic_its_invalidate_cache (its); + }/* * Try to process any pending commands. This function bails out early --
Quoted 2 lines
Thanks,
Oliver
Best regards,
Hyunwoo Kim
Oliver Upton
maintainerQuoted 55 lines · Hyunwoo Kim
Quoted 40 lines · Oliver Upton
Hi Hyunwoo,
Nice find.
Quoted 13 lines · Hyunwoo Kim
vgic_its_invalidate_cache()walks the per-ITS translation cache withxa_for_each()and drops the cache's reference on each entry withvgic_put_irq(). It puts the iterated pointer, though, rather than the value returned byxa_erase().The function is called from contexts that do not exclude one another: the ITS command handlers hold
its_lock, theGITS_CTLRwrite path holdscmd_lock, and the path that clears EnableLPIs in a redistributor'sGICR_CTLRholds neither. Two or more of them can drain the same cache concurrently, and if each one observes the same entry, erases it and then puts it, the single reference the cache holds on that entry is dropped more than once. The entry can then be freed while an ITE still maps it.Quoted 4 lines
xa_erase()is atomic and returns the previous entry, so put only the entry that this context actually removed. The cache reference is then dropped exactly once per entry even when the invalidations run concurrently, and the behavior is unchanged when only one context runs.Next time:
Quoted 13 lines
Fixes 8201d1028caa ("KVM: arm64: vgic-its: Maintain a translation cache per ITS")Signed-off-by Hyunwoo Kim <[email protected]>--- arch/arm64/kvm/vgic/vgic-its.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)arch/arm64/kvm/vgic/vgic-its.c +0
-0@@ -597,8 +597,10 @@ static void vgic_its_invalidate_cache(struct vgic_its *its)unsigned long idx;Quoted 13 lines
arch/arm64/kvm/vgic/vgic-its.c static void vgic_its_invalidate_cache(struct vgic_its *its) · line 605
Quoted 9 lines
arch/arm64/kvm/vgic/vgic-its.c +4
-2xa_for_each (&its->translation_cache, idx, irq) { -xa_erase (&its->translation_cache, idx); -vgic_put_irq (kvm, irq); +/* Only the context that erases the entry drops its cache ref. */ + irq =xa_erase (&its->translation_cache, idx); +if (irq) +vgic_put_irq (kvm, irq); } }This definitely works but TBH I'd rather just plug the subtle race and do invalidations behind the
its_locksince it already nests with thecmd_lock.
Quoted 8 lines
Thank you for the review.
Quoted 2 lines
Could you give this a spin?
After testing, I've confirmed that this patch approach works well too.
Shall I submit v2 based on this fix?
Yes, please do.
Thanks,
Oliver
Quoted 13 lines · Hyunwoo Kim
vgic_its_invalidate_cache()walks the per-ITS translation cache withxa_for_each()and drops the cache's reference on each entry withvgic_put_irq(). It puts the iterated pointer, though, rather than the value returned byxa_erase().The function is called from contexts that do not exclude one another: the ITS command handlers hold
its_lock, theGITS_CTLRwrite path holdscmd_lock, and the path that clears EnableLPIs in a redistributor'sGICR_CTLRholds neither. Two or more of them can drain the same cache concurrently, and if each one observes the same entry, erases it and then puts it, the single reference the cache holds on that entry is dropped more than once. The entry can then be freed while an ITE still maps it.
Quoted 7 lines
xa_erase()is atomic and returns the previous entry, so put only the entry that this context actually removed. The cache reference is then dropped exactly once per entry even when the invalidations run concurrently, and the behavior is unchanged when only one context runs.Fixes 8201d1028caa ("KVM: arm64: vgic-its: Maintain a translation cache per ITS")Signed-off-by Hyunwoo Kim <[email protected]>
Thanks,
Oliver
Quoted 24 lines
--- arch/arm64/kvm/vgic/vgic-its.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)arch/arm64/kvm/vgic/vgic-its.c +4
-2@@ -597,8 +597,10 @@ static void vgic_its_invalidate_cache(struct vgic_its *its)unsigned long idx;xa_for_each (&its->translation_cache, idx, irq) { -xa_erase (&its->translation_cache, idx); -vgic_put_irq (kvm, irq); +/* Only the context that erases the entry drops its cache ref. */ + irq =xa_erase (&its->translation_cache, idx); +if (irq) +vgic_put_irq (kvm, irq); } }Signature · 1 line
2.43.0