Skip to contentSkip to conversation

KVM: arm/arm64: vgic: ITS translation cache

PATCH In review Marc Zyngier 4 participants 17 messages Jul 25, 2019 – Aug 18, 2019

Subsystems (2)
lore
Share & tools
lore

Marc Zyngier

author maintainer
#0 lore raw

v3 cover KVM: arm/arm64: vgic: ITS translation cache

From: Marc Zyngier <[email protected]>

It recently became apparent[1] that our LPI injection path is not as efficient as it could be when injecting interrupts coming from a VFIO assigned device.

Although the proposed patch wasn't 100% correct, it outlined at least two issues:

(1) Injecting an LPI from VFIO always results in a context switch to a worker thread: no good

(2) We have no way of amortising the cost of translating a DID+EID pair to an LPI number

The reason for (1) is that we may sleep when translating an LPI, so we do need a context process. A way to fix that is to implement a small LPI translation cache that could be looked up from an atomic context. It would also solve (2).

This is what this small series proposes. It implements a very basic LRU cache of pre-translated LPIs, which gets used to implement kvm_arch_set_irq_inatomic. The size of the cache is currently hard-coded at 16 times the number of vcpus, a number I have picked under the influence of Ali Saidi. If that's not enough for you, blame me, though.

Does it work? well, it doesn't crash, and is thus perfect. More seriously, I don't really have a way to benchmark it directly, so my observations are only indirect:

On a TX2 system, I run a 4 vcpu VM with an Ethernet interface passed to it directly. From the host, I inject interrupts using debugfs. In parallel, I look at the number of context switch, and the number of interrupts on the host. Without this series, I get the same number for both IRQ and CS (about half a million of each per second is pretty easy to reach). With this series, the number of context switches drops to something pretty small (in the low 2k), while the number of interrupts stays the same.

Yes, this is a pretty rubbish benchmark, what did you expect? ;-)

Andre did run some benchmarks of his own, with some rather positive results[2]. So I'm putting this out for people with real workloads to try out and report what they see.

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://www.spinics.net/lists/arm-kernel/msg742655.html

* From v2:

- Added invalidation on turning the ITS off
- Added invalidation on MAPC with V=0
- Added Rb's from Eric

* From v1:

- Fixed race on allocation, where the same LPI could be cached multiple times
- Now invalidate the cache on vgic teardown, avoiding memory leaks
- Change patch split slightly, general reshuffling
- Small cleanups here and there
- Rebased on 5.2-rc4

Marc Zyngier (10):
KVM: arm/arm64: vgic: Add LPI translation cache definition
KVM: arm/arm64: vgic: Add __vgic_put_lpi_locked primitive
KVM: arm/arm64: vgic-its: Add MSI-LPI translation cache invalidation
KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on
specific commands
KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on
disabling LPIs
KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on ITS
disable
KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on vgic
teardown
KVM: arm/arm64: vgic-its: Cache successful MSI->LPI translation
KVM: arm/arm64: vgic-its: Check the LPI translation cache on MSI
injection
KVM: arm/arm64: vgic-irqfd: Implement kvm_arch_set_irq_inatomic

 include/kvm/arm_vgic.h           |   3 +
 virt/kvm/arm/vgic/vgic-init.c    |   5 +
 virt/kvm/arm/vgic/vgic-irqfd.c   |  36 +++++-
 virt/kvm/arm/vgic/vgic-its.c     | 207 +++++++++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic-mmio-v3.c |   4 +-
 virt/kvm/arm/vgic/vgic.c         |  26 ++--
 virt/kvm/arm/vgic/vgic.h         |   5 +
 7 files changed, 270 insertions(+), 16 deletions(-)
Signature · 1 line
2.20.1
author maintainer
#1 lore raw

PATCH v3 1/10 KVM: arm/arm64: vgic: Add LPI translation cache definition

From: Marc Zyngier <[email protected]>

Add the basic data structure that expresses an MSI to LPI translation as well as the allocation/release hooks.

The size of the cache is arbitrarily defined as 16*nr_vcpus.

---
 include/kvm/arm_vgic.h        |  3 +++
 virt/kvm/arm/vgic/vgic-init.c |  5 ++++
 virt/kvm/arm/vgic/vgic-its.c  | 49 +++++++++++++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic.h      |  2 ++
 4 files changed, 59 insertions(+)
include/kvm/arm_vgic.h +3 -0
index 46bbc949c20a..298cfc38e4a1 100644
@@ -249,6 +249,9 @@ struct vgic_dist {
 	struct list_head	lpi_list_head;
 	int			lpi_list_count;
 
+	/* LPI translation cache */
+	struct list_head	lpi_translation_cache;
+
 	/* used by vgic-debug */
 	struct vgic_state_iter *iter;
 
virt/kvm/arm/vgic/vgic-init.c +5 -0
index bdbc297d06fb..80127ca9269f 100644
@@ -53,6 +53,7 @@ void kvm_vgic_early_init(struct kvm *kvm)
 	struct vgic_dist *dist = &kvm->arch.vgic;
 
 	INIT_LIST_HEAD(&dist->lpi_list_head);
+	INIT_LIST_HEAD(&dist->lpi_translation_cache);
 	raw_spin_lock_init(&dist->lpi_list_lock);
 }
 
@@ -294,6 +295,7 @@ int vgic_init(struct kvm *kvm)
 	}
 
 	if (vgic_has_its(kvm)) {
+		vgic_lpi_translation_cache_init(kvm);
 		ret = vgic_v4_init(kvm);
 		if (ret)
 			goto out;
@@ -335,6 +337,9 @@ static void kvm_vgic_dist_destroy(struct kvm *kvm)
 		INIT_LIST_HEAD(&dist->rd_regions);
 	}
 
+	if (vgic_has_its(kvm))
+		vgic_lpi_translation_cache_destroy(kvm);
+
 	if (vgic_supports_direct_msis(kvm))
 		vgic_v4_teardown(kvm);
 }
virt/kvm/arm/vgic/vgic-its.c +49 -0

67 lines not on this page · raw message

virt/kvm/arm/vgic/vgic.h +2 -0
index 57205beaa981..11291a9c8c1c 100644
@@ -305,6 +305,8 @@ int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr);
 int vgic_its_resolve_lpi(struct kvm *kvm, struct vgic_its *its,
 			 u32 devid, u32 eventid, struct vgic_irq **irq);
 struct vgic_its *vgic_msi_to_its(struct kvm *kvm, struct kvm_msi *msi);
+void vgic_lpi_translation_cache_init(struct kvm *kvm);
+void vgic_lpi_translation_cache_destroy(struct kvm *kvm);
 
 bool vgic_supports_direct_msis(struct kvm *kvm);
 int vgic_v4_init(struct kvm *kvm);
Signature · 1 line
2.20.1
author maintainer
#2 lore raw

PATCH v3 2/10 KVM: arm/arm64: vgic: Add __vgic_put_lpi_locked primitive

In reply to Marc Zyngier

From: Marc Zyngier <[email protected]>

Our LPI translation cache needs to be able to drop the refcount on an LPI whilst already holding the lpi_list_lock.

Let's add a new primitive for this.

---
 virt/kvm/arm/vgic/vgic.c | 26 +++++++++++++++++---------
 virt/kvm/arm/vgic/vgic.h |  1 +
 2 files changed, 18 insertions(+), 9 deletions(-)
virt/kvm/arm/vgic/vgic.c +17 -9
index 04786c8ec77e..deb84712f550 100644
@@ -119,6 +119,22 @@ static void vgic_irq_release(struct kref *ref)
 {
 }
 
+/*
+ * Drop the refcount on the LPI. Must be called with lpi_list_lock held.
+ */
+void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+
+	if (!kref_put(&irq->refcount, vgic_irq_release))
+		return;
+
+	list_del(&irq->lpi_list);
+	dist->lpi_list_count--;
+
+	kfree(irq);
+}
+
 void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq)
 {
 	struct vgic_dist *dist = &kvm->arch.vgic;
@@ -128,16 +144,8 @@ void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq)
 		return;
 
 	raw_spin_lock_irqsave(&dist->lpi_list_lock, flags);
-	if (!kref_put(&irq->refcount, vgic_irq_release)) {
-		raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
-		return;
-	};
-
-	list_del(&irq->lpi_list);
-	dist->lpi_list_count--;
+	__vgic_put_lpi_locked(kvm, irq);
 	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
-
-	kfree(irq);
 }
 
 void vgic_flush_pending_lpis(struct kvm_vcpu *vcpu)
virt/kvm/arm/vgic/vgic.h +1 -0
index 11291a9c8c1c..e523b3a54590 100644
@@ -161,6 +161,7 @@ vgic_get_mmio_region(struct kvm_vcpu *vcpu, struct vgic_io_device *iodev,
 		     gpa_t addr, int len);
 struct vgic_irq *vgic_get_irq(struct kvm *kvm, struct kvm_vcpu *vcpu,
 			      u32 intid);
+void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq);
 void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq);
 bool vgic_get_phys_line_level(struct vgic_irq *irq);
 void vgic_irq_set_phys_pending(struct vgic_irq *irq, bool pending);
Signature · 1 line
2.20.1
author maintainer
#3 lore raw

PATCH v3 3/10 KVM: arm/arm64: vgic-its: Add MSI-LPI translation cache invalidation

In reply to Marc Zyngier

From: Marc Zyngier <[email protected]>

There's a number of cases where we need to invalidate the caching of translations, so let's add basic support for that.

---
 virt/kvm/arm/vgic/vgic-its.c | 23 +++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic.h     |  1 +
 2 files changed, 24 insertions(+)
virt/kvm/arm/vgic/vgic-its.c +23 -0
index 0e5c1519bbe2..cc6b5e49a312 100644
@@ -535,6 +535,29 @@ static unsigned long vgic_mmio_read_its_idregs(struct kvm *kvm,
 	return 0;
 }
 
+void vgic_its_invalidate_cache(struct kvm *kvm)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	struct vgic_translation_cache_entry *cte;
+	unsigned long flags;
+
+	raw_spin_lock_irqsave(&dist->lpi_list_lock, flags);
+
+	list_for_each_entry(cte, &dist->lpi_translation_cache, entry) {
+		/*
+		 * If we hit a NULL entry, there is nothing after this
+		 * point.
+		 */
+		if (!cte->irq)
+			break;
+
+		__vgic_put_lpi_locked(kvm, cte->irq);
+		cte->irq = NULL;
+	}
+
+	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
+}
+
 int vgic_its_resolve_lpi(struct kvm *kvm, struct vgic_its *its,
 			 u32 devid, u32 eventid, struct vgic_irq **irq)
 {
virt/kvm/arm/vgic/vgic.h +1 -0
index e523b3a54590..09908b80fb1e 100644
@@ -308,6 +308,7 @@ int vgic_its_resolve_lpi(struct kvm *kvm, struct vgic_its *its,
 struct vgic_its *vgic_msi_to_its(struct kvm *kvm, struct kvm_msi *msi);
 void vgic_lpi_translation_cache_init(struct kvm *kvm);
 void vgic_lpi_translation_cache_destroy(struct kvm *kvm);
+void vgic_its_invalidate_cache(struct kvm *kvm);
 
 bool vgic_supports_direct_msis(struct kvm *kvm);
 int vgic_v4_init(struct kvm *kvm);
Signature · 1 line
2.20.1
author maintainer
#4 lore raw

PATCH v3 4/10 KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on specific commands

In reply to Marc Zyngier

From: Marc Zyngier <[email protected]>

The LPI translation cache needs to be discarded when an ITS command may affect the translation of an LPI (DISCARD, MAPC and MAPD with V=0) or the routing of an LPI to a redistributor with disabled LPIs (MOVI, MOVALL).

We decide to perform a full invalidation of the cache, irrespective of the LPI that is affected. Commands are supposed to be rare enough that it doesn't matter.

---
 virt/kvm/arm/vgic/vgic-its.c | 9 +++++++++
 1 file changed, 9 insertions(+)
virt/kvm/arm/vgic/vgic-its.c +9 -0
index cc6b5e49a312..09a179820816 100644
@@ -722,6 +722,8 @@ static int vgic_its_cmd_handle_discard(struct kvm *kvm, struct vgic_its *its,
 		 * don't bother here since we clear the ITTE anyway and the
 		 * pending state is a property of the ITTE struct.
 		 */
+		vgic_its_invalidate_cache(kvm);
+
 		its_free_ite(kvm, ite);
 		return 0;
 	}
@@ -757,6 +759,8 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
 	ite->collection = collection;
 	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
 
+	vgic_its_invalidate_cache(kvm);
+
 	return update_affinity(ite->irq, vcpu);
 }
 
@@ -985,6 +989,8 @@ static void vgic_its_free_device(struct kvm *kvm, struct its_device *device)
 	list_for_each_entry_safe(ite, temp, &device->itt_head, ite_list)
 		its_free_ite(kvm, ite);
 
+	vgic_its_invalidate_cache(kvm);
+
 	list_del(&device->dev_list);
 	kfree(device);
 }
@@ -1090,6 +1096,7 @@ static int vgic_its_cmd_handle_mapc(struct kvm *kvm, struct vgic_its *its,
 
 	if (!valid) {
 		vgic_its_free_collection(its, coll_id);
+		vgic_its_invalidate_cache(kvm);
 	} else {
 		collection = find_collection(its, coll_id);
 
@@ -1238,6 +1245,8 @@ static int vgic_its_cmd_handle_movall(struct kvm *kvm, struct vgic_its *its,
 		vgic_put_irq(kvm, irq);
 	}
 
+	vgic_its_invalidate_cache(kvm);
+
 	kfree(intids);
 	return 0;
 }
Signature · 1 line
2.20.1
author maintainer
#5 lore raw

PATCH v3 5/10 KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on disabling LPIs

In reply to Marc Zyngier

From: Marc Zyngier <[email protected]>

If a vcpu disables LPIs at its redistributor level, we need to make sure we won't pend more interrupts. For this, we need to invalidate the LPI translation cache.

---
 virt/kvm/arm/vgic/vgic-mmio-v3.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
virt/kvm/arm/vgic/vgic-mmio-v3.c +3 -1
index 936962abc38d..cb60da48810d 100644
@@ -192,8 +192,10 @@ static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu,
 
 	vgic_cpu->lpis_enabled = val & GICR_CTLR_ENABLE_LPIS;
 
-	if (was_enabled && !vgic_cpu->lpis_enabled)
+	if (was_enabled && !vgic_cpu->lpis_enabled) {
 		vgic_flush_pending_lpis(vcpu);
+		vgic_its_invalidate_cache(vcpu->kvm);
+	}
 
 	if (!was_enabled && vgic_cpu->lpis_enabled)
 		vgic_enable_lpis(vcpu);
Signature · 1 line
2.20.1
Reviewed-by on v3 5/10
#6 lore raw

Re: [PATCH v3 05/10] KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on disabling LPIs

Hi Marc,

Quoted 7 lines · Marc Zyngier

From: Marc Zyngier <[email protected]>

If a vcpu disables LPIs at its redistributor level, we need to make sure we won't pend more interrupts. For this, we need to invalidate the LPI translation cache.

Eric

Quoted 21 lines
---
 virt/kvm/arm/vgic/vgic-mmio-v3.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
virt/kvm/arm/vgic/vgic-mmio-v3.c +3 -1
index 936962abc38d..cb60da48810d 100644
@@ -192,8 +192,10 @@ static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu,
 
 	vgic_cpu->lpis_enabled = val & GICR_CTLR_ENABLE_LPIS;
 
-	if (was_enabled && !vgic_cpu->lpis_enabled)
+	if (was_enabled && !vgic_cpu->lpis_enabled) {
 		vgic_flush_pending_lpis(vcpu);
+		vgic_its_invalidate_cache(vcpu->kvm);
+	}
 
 	if (!was_enabled && vgic_cpu->lpis_enabled)
 		vgic_enable_lpis(vcpu);
author maintainer
#7 lore raw

PATCH v3 6/10 KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on ITS disable

In reply to Marc Zyngier

From: Marc Zyngier <[email protected]>

If an ITS gets disabled, we need to make sure that further interrupts won't hit in the cache. For that, we invalidate the translation cache when the ITS is disabled.

---
 virt/kvm/arm/vgic/vgic-its.c | 2 ++
 1 file changed, 2 insertions(+)
virt/kvm/arm/vgic/vgic-its.c +2 -0
index 09a179820816..05406bd92ce9 100644
@@ -1597,6 +1597,8 @@ static void vgic_mmio_write_its_ctlr(struct kvm *kvm, struct vgic_its *its,
 		goto out;
 
 	its->enabled = !!(val & GITS_CTLR_ENABLE);
+	if (!its->enabled)
+		vgic_its_invalidate_cache(kvm);
 
 	/*
 	 * Try to process any pending commands. This function bails out early
Signature · 1 line
2.20.1
Reviewed-by on v3 6/10
#8 lore raw

Re: [PATCH v3 06/10] KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on ITS disable

Hi Marc,

Quoted 7 lines · Marc Zyngier

From: Marc Zyngier <[email protected]>

If an ITS gets disabled, we need to make sure that further interrupts won't hit in the cache. For that, we invalidate the translation cache when the ITS is disabled.

Eric

Quoted 18 lines
---
 virt/kvm/arm/vgic/vgic-its.c | 2 ++
 1 file changed, 2 insertions(+)
virt/kvm/arm/vgic/vgic-its.c +2 -0
index 09a179820816..05406bd92ce9 100644
@@ -1597,6 +1597,8 @@ static void vgic_mmio_write_its_ctlr(struct kvm *kvm, struct vgic_its *its,
 		goto out;
 
 	its->enabled = !!(val & GITS_CTLR_ENABLE);
+	if (!its->enabled)
+		vgic_its_invalidate_cache(kvm);
 
 	/*
 	 * Try to process any pending commands. This function bails out early
author maintainer
#9 lore raw

PATCH v3 7/10 KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on vgic teardown

In reply to Marc Zyngier

From: Marc Zyngier <[email protected]>

In order to avoid leaking vgic_irq structures on teardown, we need to drop all references to LPIs before deallocating the cache itself.

This is done by invalidating the cache on vgic teardown.

---
 virt/kvm/arm/vgic/vgic-its.c | 2 ++
 1 file changed, 2 insertions(+)
virt/kvm/arm/vgic/vgic-its.c +2 -0
index 05406bd92ce9..d3e90a9d0a7a 100644
@@ -1731,6 +1731,8 @@ void vgic_lpi_translation_cache_destroy(struct kvm *kvm)
 	struct vgic_dist *dist = &kvm->arch.vgic;
 	struct vgic_translation_cache_entry *cte, *tmp;
 
+	vgic_its_invalidate_cache(kvm);
+
 	list_for_each_entry_safe(cte, tmp,
 				 &dist->lpi_translation_cache, entry) {
 		list_del(&cte->entry);
Signature · 1 line
2.20.1
Reviewed-by on v3 7/10
#10 lore raw

Re: [PATCH v3 07/10] KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on vgic teardown

Hi Marc,

Quoted 8 lines · Marc Zyngier

From: Marc Zyngier <[email protected]>

In order to avoid leaking vgic_irq structures on teardown, we need to drop all references to LPIs before deallocating the cache itself.

This is done by invalidating the cache on vgic teardown.

Eric

Quoted 18 lines
---
 virt/kvm/arm/vgic/vgic-its.c | 2 ++
 1 file changed, 2 insertions(+)
virt/kvm/arm/vgic/vgic-its.c +2 -0
index 05406bd92ce9..d3e90a9d0a7a 100644
@@ -1731,6 +1731,8 @@ void vgic_lpi_translation_cache_destroy(struct kvm *kvm)
 	struct vgic_dist *dist = &kvm->arch.vgic;
 	struct vgic_translation_cache_entry *cte, *tmp;
 
+	vgic_its_invalidate_cache(kvm);
+
 	list_for_each_entry_safe(cte, tmp,
 				 &dist->lpi_translation_cache, entry) {
 		list_del(&cte->entry);
author maintainer
#11 lore raw

PATCH v3 8/10 KVM: arm/arm64: vgic-its: Cache successful MSI->LPI translation

In reply to Marc Zyngier

From: Marc Zyngier <[email protected]>

On a successful translation, preserve the parameters in the LPI translation cache. Each translation is reusing the last slot in the list, naturally evicting the least recently used entry.

---
 virt/kvm/arm/vgic/vgic-its.c | 86 ++++++++++++++++++++++++++++++++++++
 1 file changed, 86 insertions(+)
virt/kvm/arm/vgic/vgic-its.c +86 -0
index d3e90a9d0a7a..e61d3ea0ab40 100644
@@ -535,6 +535,90 @@ static unsigned long vgic_mmio_read_its_idregs(struct kvm *kvm,
 	return 0;
 }
 
+static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist,
+					       phys_addr_t db,
+					       u32 devid, u32 eventid)
+{
+	struct vgic_translation_cache_entry *cte;
+
+	list_for_each_entry(cte, &dist->lpi_translation_cache, entry) {
+		/*
+		 * If we hit a NULL entry, there is nothing after this
+		 * point.
+		 */
+		if (!cte->irq)
+			break;
+
+		if (cte->db != db || cte->devid != devid ||
+		    cte->eventid != eventid)
+			continue;
+
+		/*
+		 * Move this entry to the head, as it is the most
+		 * recently used.
+		 */
+		if (!list_is_first(&cte->entry, &dist->lpi_translation_cache))
+			list_move(&cte->entry, &dist->lpi_translation_cache);
+
+		return cte->irq;
+	}
+
+	return NULL;
+}
+
+static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,
+				       u32 devid, u32 eventid,
+				       struct vgic_irq *irq)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	struct vgic_translation_cache_entry *cte;
+	unsigned long flags;
+	phys_addr_t db;
+
+	/* Do not cache a directly injected interrupt */
+	if (irq->hw)
+		return;
+
+	raw_spin_lock_irqsave(&dist->lpi_list_lock, flags);
+
+	if (unlikely(list_empty(&dist->lpi_translation_cache)))
+		goto out;
+
+	/*
+	 * We could have raced with another CPU caching the same
+	 * translation behind our back, so let's check it is not in
+	 * already
+	 */
+	db = its->vgic_its_base + GITS_TRANSLATER;
+	if (__vgic_its_check_cache(dist, db, devid, eventid))
+		goto out;
+
+	/* Always reuse the last entry (LRU policy) */
+	cte = list_last_entry(&dist->lpi_translation_cache,
+			      typeof(*cte), entry);
+
+	/*
+	 * Caching the translation implies having an extra reference
+	 * to the interrupt, so drop the potential reference on what
+	 * was in the cache, and increment it on the new interrupt.
+	 */
+	if (cte->irq)
+		__vgic_put_lpi_locked(kvm, cte->irq);
+
+	vgic_get_irq_kref(irq);
+
+	cte->db		= db;
+	cte->devid	= devid;
+	cte->eventid	= eventid;
+	cte->irq	= irq;
+
+	/* Move the new translation to the head of the list */
+	list_move(&cte->entry, &dist->lpi_translation_cache);
+
+out:
+	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
+}
+
 void vgic_its_invalidate_cache(struct kvm *kvm)
 {
 	struct vgic_dist *dist = &kvm->arch.vgic;
@@ -578,6 +662,8 @@ int vgic_its_resolve_lpi(struct kvm *kvm, struct vgic_its *its,
 	if (!vcpu->arch.vgic_cpu.lpis_enabled)
 		return -EBUSY;
 
+	vgic_its_cache_translation(kvm, its, devid, eventid, ite->irq);
+
 	*irq = ite->irq;
 	return 0;
 }
Signature · 1 line
2.20.1
author maintainer
#12 lore raw

PATCH v3 9/10 KVM: arm/arm64: vgic-its: Check the LPI translation cache on MSI injection

In reply to Marc Zyngier

From: Marc Zyngier <[email protected]>

When performing an MSI injection, let's first check if the translation is already in the cache. If so, let's inject it quickly without going through the whole translation process.

---
 virt/kvm/arm/vgic/vgic-its.c | 36 ++++++++++++++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic.h     |  1 +
 2 files changed, 37 insertions(+)
virt/kvm/arm/vgic/vgic-its.c +36 -0
index e61d3ea0ab40..2be6b66b3856 100644
@@ -566,6 +566,20 @@ static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist,
 	return NULL;
 }
 
+static struct vgic_irq *vgic_its_check_cache(struct kvm *kvm, phys_addr_t db,
+					     u32 devid, u32 eventid)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	struct vgic_irq *irq;
+	unsigned long flags;
+
+	raw_spin_lock_irqsave(&dist->lpi_list_lock, flags);
+	irq = __vgic_its_check_cache(dist, db, devid, eventid);
+	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
+
+	return irq;
+}
+
 static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,
 				       u32 devid, u32 eventid,
 				       struct vgic_irq *irq)
@@ -725,6 +739,25 @@ static int vgic_its_trigger_msi(struct kvm *kvm, struct vgic_its *its,
 	return 0;
 }
 
+int vgic_its_inject_cached_translation(struct kvm *kvm, struct kvm_msi *msi)
+{
+	struct vgic_irq *irq;
+	unsigned long flags;
+	phys_addr_t db;
+
+	db = (u64)msi->address_hi << 32 | msi->address_lo;
+	irq = vgic_its_check_cache(kvm, db, msi->devid, msi->data);
+
+	if (!irq)
+		return -1;
+
+	raw_spin_lock_irqsave(&irq->irq_lock, flags);
+	irq->pending_latch = true;
+	vgic_queue_irq_unlock(kvm, irq, flags);
+
+	return 0;
+}
+
 /*
  * Queries the KVM IO bus framework to get the ITS pointer from the given
  * doorbell address.
@@ -736,6 +769,9 @@ int vgic_its_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
 	struct vgic_its *its;
 	int ret;
 
+	if (!vgic_its_inject_cached_translation(kvm, msi))
+		return 1;
+
 	its = vgic_msi_to_its(kvm, msi);
 	if (IS_ERR(its))
 		return PTR_ERR(its);
virt/kvm/arm/vgic/vgic.h +1 -0
index 09908b80fb1e..db308a62b34d 100644
@@ -306,6 +306,7 @@ int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr);
 int vgic_its_resolve_lpi(struct kvm *kvm, struct vgic_its *its,
 			 u32 devid, u32 eventid, struct vgic_irq **irq);
 struct vgic_its *vgic_msi_to_its(struct kvm *kvm, struct kvm_msi *msi);
+int vgic_its_inject_cached_translation(struct kvm *kvm, struct kvm_msi *msi);
 void vgic_lpi_translation_cache_init(struct kvm *kvm);
 void vgic_lpi_translation_cache_destroy(struct kvm *kvm);
 void vgic_its_invalidate_cache(struct kvm *kvm);
Signature · 1 line
2.20.1
Reviewed-by on v3 9/10
#13 lore raw

Re: [PATCH v3 09/10] KVM: arm/arm64: vgic-its: Check the LPI translation cache on MSI injection

Hi Marc,

Quoted 7 lines · Marc Zyngier

From: Marc Zyngier <[email protected]>

When performing an MSI injection, let's first check if the translation is already in the cache. If so, let's inject it quickly without going through the whole translation process.

Thanks

Eric

Quoted 79 lines
---
 virt/kvm/arm/vgic/vgic-its.c | 36 ++++++++++++++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic.h     |  1 +
 2 files changed, 37 insertions(+)
virt/kvm/arm/vgic/vgic-its.c +36 -0
index e61d3ea0ab40..2be6b66b3856 100644
@@ -566,6 +566,20 @@ static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist,
 	return NULL;
 }
 
+static struct vgic_irq *vgic_its_check_cache(struct kvm *kvm, phys_addr_t db,
+					     u32 devid, u32 eventid)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	struct vgic_irq *irq;
+	unsigned long flags;
+
+	raw_spin_lock_irqsave(&dist->lpi_list_lock, flags);
+	irq = __vgic_its_check_cache(dist, db, devid, eventid);
+	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
+
+	return irq;
+}
+
 static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,
 				       u32 devid, u32 eventid,
 				       struct vgic_irq *irq)
@@ -725,6 +739,25 @@ static int vgic_its_trigger_msi(struct kvm *kvm, struct vgic_its *its,
 	return 0;
 }
 
+int vgic_its_inject_cached_translation(struct kvm *kvm, struct kvm_msi *msi)
+{
+	struct vgic_irq *irq;
+	unsigned long flags;
+	phys_addr_t db;
+
+	db = (u64)msi->address_hi << 32 | msi->address_lo;
+	irq = vgic_its_check_cache(kvm, db, msi->devid, msi->data);
+
+	if (!irq)
+		return -1;
+
+	raw_spin_lock_irqsave(&irq->irq_lock, flags);
+	irq->pending_latch = true;
+	vgic_queue_irq_unlock(kvm, irq, flags);
+
+	return 0;
+}
+
 /*
  * Queries the KVM IO bus framework to get the ITS pointer from the given
  * doorbell address.
@@ -736,6 +769,9 @@ int vgic_its_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
 	struct vgic_its *its;
 	int ret;
 
+	if (!vgic_its_inject_cached_translation(kvm, msi))
+		return 1;
+
 	its = vgic_msi_to_its(kvm, msi);
 	if (IS_ERR(its))
 		return PTR_ERR(its);
virt/kvm/arm/vgic/vgic.h +1 -0
index 09908b80fb1e..db308a62b34d 100644
@@ -306,6 +306,7 @@ int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr);
 int vgic_its_resolve_lpi(struct kvm *kvm, struct vgic_its *its,
 			 u32 devid, u32 eventid, struct vgic_irq **irq);
 struct vgic_its *vgic_msi_to_its(struct kvm *kvm, struct kvm_msi *msi);
+int vgic_its_inject_cached_translation(struct kvm *kvm, struct kvm_msi *msi);
 void vgic_lpi_translation_cache_init(struct kvm *kvm);
 void vgic_lpi_translation_cache_destroy(struct kvm *kvm);
 void vgic_its_invalidate_cache(struct kvm *kvm);
author maintainer
#14 lore raw

PATCH v3 10/10 KVM: arm/arm64: vgic-irqfd: Implement kvm_arch_set_irq_inatomic

In reply to Marc Zyngier

From: Marc Zyngier <[email protected]>

Now that we have a cache of MSI->LPI translations, it is pretty easy to implement kvm_arch_set_irq_inatomic (this cache can be parsed without sleeping).

Hopefully, this will improve some LPI-heavy workloads.

---
 virt/kvm/arm/vgic/vgic-irqfd.c | 36 ++++++++++++++++++++++++++++------
 1 file changed, 30 insertions(+), 6 deletions(-)
virt/kvm/arm/vgic/vgic-irqfd.c +30 -6
index c9304b88e720..d8cdfea5cc96 100644
@@ -66,6 +66,15 @@ int kvm_set_routing_entry(struct kvm *kvm,
 	return r;
 }
 
+static void kvm_populate_msi(struct kvm_kernel_irq_routing_entry *e,
+			     struct kvm_msi *msi)
+{
+	msi->address_lo = e->msi.address_lo;
+	msi->address_hi = e->msi.address_hi;
+	msi->data = e->msi.data;
+	msi->flags = e->msi.flags;
+	msi->devid = e->msi.devid;
+}
 /**
  * kvm_set_msi: inject the MSI corresponding to the
  * MSI routing entry
@@ -79,21 +88,36 @@ int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
 {
 	struct kvm_msi msi;
 
-	msi.address_lo = e->msi.address_lo;
-	msi.address_hi = e->msi.address_hi;
-	msi.data = e->msi.data;
-	msi.flags = e->msi.flags;
-	msi.devid = e->msi.devid;
-
 	if (!vgic_has_its(kvm))
 		return -ENODEV;
 
 	if (!level)
 		return -1;
 
+	kvm_populate_msi(e, &msi);
 	return vgic_its_inject_msi(kvm, &msi);
 }
 
+/**
+ * kvm_arch_set_irq_inatomic: fast-path for irqfd injection
+ *
+ * Currently only direct MSI injection is supported.
+ */
+int kvm_arch_set_irq_inatomic(struct kvm_kernel_irq_routing_entry *e,
+			      struct kvm *kvm, int irq_source_id, int level,
+			      bool line_status)
+{
+	if (e->type == KVM_IRQ_ROUTING_MSI && vgic_has_its(kvm) && level) {
+		struct kvm_msi msi;
+
+		kvm_populate_msi(e, &msi);
+		if (!vgic_its_inject_cached_translation(kvm, &msi))
+			return 0;
+	}
+
+	return -EWOULDBLOCK;
+}
+
 int kvm_vgic_setup_default_irq_routing(struct kvm *kvm)
 {
 	struct kvm_irq_routing_entry *entries;
Signature · 1 line
2.20.1
Tested-by on the v3 series
#15 lore raw
In reply to Marc Zyngier

On Thu, 25 Jul 2019 16:35:33 +0100
Marc Zyngier <[email protected]> wrote:

Hi,

Quoted 40 lines

From: Marc Zyngier <[email protected]>

It recently became apparent[1] that our LPI injection path is not as efficient as it could be when injecting interrupts coming from a VFIO assigned device.

Although the proposed patch wasn't 100% correct, it outlined at least two issues:

(1) Injecting an LPI from VFIO always results in a context switch to a worker thread: no good

(2) We have no way of amortising the cost of translating a DID+EID pair to an LPI number

The reason for (1) is that we may sleep when translating an LPI, so we do need a context process. A way to fix that is to implement a small LPI translation cache that could be looked up from an atomic context. It would also solve (2).

This is what this small series proposes. It implements a very basic LRU cache of pre-translated LPIs, which gets used to implement kvm_arch_set_irq_inatomic. The size of the cache is currently hard-coded at 16 times the number of vcpus, a number I have picked under the influence of Ali Saidi. If that's not enough for you, blame me, though.

Does it work? well, it doesn't crash, and is thus perfect. More seriously, I don't really have a way to benchmark it directly, so my observations are only indirect:

On a TX2 system, I run a 4 vcpu VM with an Ethernet interface passed to it directly. From the host, I inject interrupts using debugfs. In parallel, I look at the number of context switch, and the number of interrupts on the host. Without this series, I get the same number for both IRQ and CS (about half a million of each per second is pretty easy to reach). With this series, the number of context switches drops to something pretty small (in the low 2k), while the number of interrupts stays the same.

Quoted 5 lines

Yes, this is a pretty rubbish benchmark, what did you expect? ;-)

Andre did run some benchmarks of his own, with some rather positive results[2]. So I'm putting this out for people with real workloads to try out and report what they see.

And I gave this series a try, on top of 5.3-rc2, with Robin's patch [3] to fix a VFIO IRQ breakage.
The results were very similar, though at least one performance number was slightly worse than compared to this series on top of v5.2. But nevertheless there is still the big improvement compared to the baseline without this series, so:

(for the whole series):

Cheers,
Andre.

[3] http://lists.infradead.org/pipermail/linux-arm-kernel/2019-July/669468.html

Quoted 43 lines

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://www.spinics.net/lists/arm-kernel/msg742655.html

* From v2:

- Added invalidation on turning the ITS off
- Added invalidation on MAPC with V=0
- Added Rb's from Eric

* From v1:

- Fixed race on allocation, where the same LPI could be cached multiple times
- Now invalidate the cache on vgic teardown, avoiding memory leaks
- Change patch split slightly, general reshuffling
- Small cleanups here and there
- Rebased on 5.2-rc4

Marc Zyngier (10):
KVM: arm/arm64: vgic: Add LPI translation cache definition
KVM: arm/arm64: vgic: Add __vgic_put_lpi_locked primitive
KVM: arm/arm64: vgic-its: Add MSI-LPI translation cache invalidation
KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on
specific commands
KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on
disabling LPIs
KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on ITS
disable
KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on vgic
teardown
KVM: arm/arm64: vgic-its: Cache successful MSI->LPI translation
KVM: arm/arm64: vgic-its: Check the LPI translation cache on MSI
injection
KVM: arm/arm64: vgic-irqfd: Implement kvm_arch_set_irq_inatomic

 include/kvm/arm_vgic.h           |   3 +
 virt/kvm/arm/vgic/vgic-init.c    |   5 +
 virt/kvm/arm/vgic/vgic-irqfd.c   |  36 +++++-
 virt/kvm/arm/vgic/vgic-its.c     | 207 +++++++++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic-mmio-v3.c |   4 +-
 virt/kvm/arm/vgic/vgic.c         |  26 ++--
 virt/kvm/arm/vgic/vgic.h         |   5 +
 7 files changed, 270 insertions(+), 16 deletions(-)
#16 lore raw
In reply to Marc Zyngier
Quoted 80 lines · On Thu, 25 Jul 2019 16:35:33 +0100 Marc Zyngier <[email protected]> wrote:

From: Marc Zyngier <[email protected]>

It recently became apparent[1] that our LPI injection path is not as efficient as it could be when injecting interrupts coming from a VFIO assigned device.

Although the proposed patch wasn't 100% correct, it outlined at least two issues:

(1) Injecting an LPI from VFIO always results in a context switch to a worker thread: no good

(2) We have no way of amortising the cost of translating a DID+EID pair to an LPI number

The reason for (1) is that we may sleep when translating an LPI, so we do need a context process. A way to fix that is to implement a small LPI translation cache that could be looked up from an atomic context. It would also solve (2).

This is what this small series proposes. It implements a very basic LRU cache of pre-translated LPIs, which gets used to implement kvm_arch_set_irq_inatomic. The size of the cache is currently hard-coded at 16 times the number of vcpus, a number I have picked under the influence of Ali Saidi. If that's not enough for you, blame me, though.

Does it work? well, it doesn't crash, and is thus perfect. More seriously, I don't really have a way to benchmark it directly, so my observations are only indirect:

On a TX2 system, I run a 4 vcpu VM with an Ethernet interface passed to it directly. From the host, I inject interrupts using debugfs. In parallel, I look at the number of context switch, and the number of interrupts on the host. Without this series, I get the same number for both IRQ and CS (about half a million of each per second is pretty easy to reach). With this series, the number of context switches drops to something pretty small (in the low 2k), while the number of interrupts stays the same.

Yes, this is a pretty rubbish benchmark, what did you expect? ;-)

Andre did run some benchmarks of his own, with some rather positive results[2]. So I'm putting this out for people with real workloads to try out and report what they see.

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://www.spinics.net/lists/arm-kernel/msg742655.html

* From v2:

- Added invalidation on turning the ITS off
- Added invalidation on MAPC with V=0
- Added Rb's from Eric

* From v1:

- Fixed race on allocation, where the same LPI could be cached multiple times
- Now invalidate the cache on vgic teardown, avoiding memory leaks
- Change patch split slightly, general reshuffling
- Small cleanups here and there
- Rebased on 5.2-rc4

Marc Zyngier (10):
KVM: arm/arm64: vgic: Add LPI translation cache definition
KVM: arm/arm64: vgic: Add __vgic_put_lpi_locked primitive
KVM: arm/arm64: vgic-its: Add MSI-LPI translation cache invalidation
KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on
specific commands
KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on
disabling LPIs
KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on ITS
disable
KVM: arm/arm64: vgic-its: Invalidate MSI-LPI translation cache on vgic
teardown
KVM: arm/arm64: vgic-its: Cache successful MSI->LPI translation
KVM: arm/arm64: vgic-its: Check the LPI translation cache on MSI
injection
KVM: arm/arm64: vgic-irqfd: Implement kvm_arch_set_irq_inatomic

Quoted 9 lines
 include/kvm/arm_vgic.h           |   3 +
 virt/kvm/arm/vgic/vgic-init.c    |   5 +
 virt/kvm/arm/vgic/vgic-irqfd.c   |  36 +++++-
 virt/kvm/arm/vgic/vgic-its.c     | 207 +++++++++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic-mmio-v3.c |   4 +-
 virt/kvm/arm/vgic/vgic.c         |  26 ++--
 virt/kvm/arm/vgic/vgic.h         |   5 +
 7 files changed, 270 insertions(+), 16 deletions(-)

FWIW, I've now queued this for 5.4, with Eric's RBs and Andre's TBs.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.