Skip to contentSkip to conversation

KVM: arm64: Improvements to LPI injection

PATCH Superseded Oliver Upton 2 participants 41 messages Feb 13, 2024 – Feb 15, 2024

Subsystems (4)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover KVM: arm64: Improvements to LPI injection

For full details on the what/why, please see the cover letter in v1.

Apologies for the delay on v2, I wanted to spend some time to get a microbenchmark in place to slam the ITS code pretty hard, and based on the results I'm glad I did.

The test is built around having vCPU threads and device threads, where each device can signal a particular number of event IDs (alien, right?)

Anyway, here are the results of that test in some carefully-selected examples:

+----------------------------+---------------------+------------------------------+
|           Config           | v6.8-rc1 (LPIs/sec) | v6.8-rc1 + series (LPIs/sec) |
+----------------------------+---------------------+------------------------------+
| -v 1 -d 1 -e 1 -i 1000000  |           780151.37 |                   1255291.00 |
| -v 16 -d 16 -e 16 -i 10000 |           437081.55 |                   5078225.70 |
| -v 16 -d 16 -e 17 -i 10000 |           506446.50 |                   1126118.10 |
| -v 64 -d 64 -e 1 -i 100000 |           295097.03 |                   5565964.87 |
| -v 1 -d 1 -e 17 -i 1000    |           732934.43 |                       149.24 |
+----------------------------+---------------------+------------------------------+

While there is an 18x improvement in the scaled-out config (64 vCPUs, 64 devices, 1 event per device), there is an extremely disappointing 4911x regression in the example that effectively forces a cache eviction for every lookup.

Clearly the RCU synchronization is a bounding issue in this case. I think other scenarios where the cache is overcommitted (16 vCPUs, 16 devices, 17 events / device) are able to hide effects somewhat, as other threads can make forward progress while others are stuck waiting on RCU.

A few ideas on next steps:

1) Rework the lpi_list_lock as an rwlock. This would obviate the need for RCU protection in the LPI cache as well as memory allocations on the injection path. This is actually what I had in the internal version of the series, although it was very incomplete.

I'd expect this to nullify the improvement on the slightly-overcommitted case and 'fix' the pathological case.

2) call_rcu() and move on. This feels somewhat abusive of the API, as the guest can flood the host with RCU callbacks, but I wasn't able to make my machine fall over in any mean configuration of the test.

I haven't studied the degree to which such a malicious VM could adversely affect neighboring workloads.

3) Redo the whole ITS representation with xarrays and allow RCU readers outside of the ITS lock. I haven't fully thought this out, and if we pursue this option then we will need a secondary data structure to track where ITSes have been placed in guest memory to avoid taking the SRCU lock. We can then stick RCU synchronization in ITS command processing, which feels right to me, and dump the translation cache altogether.

I'd expect slightly worse average case performance in favor of more consistent performance.

Even though it is more work, I'm slightly in favor of (3) as it is a net reduction in overall complexity of the ITS implementation. But, I wanted to send out what I had to guage opinions on these options, and get feedback on the first 10 patches which are an overall win.

v1: https://lore.kernel.org/kvmarm/[email protected]/

v1 -> v2:
- Add the microbenchmark
- Add tracepoints / VM stats for the important bits of LPI injection. This was extremely useful for making sense of test results.
- Fix a silly lock imbalance on error path in vgic_add_lpi() (Dan)
- Constrain xas_for_each() based on the properties of the INTID space (Marc)
- Remove some missed vestiges of the LPI linked-list (Marc)
- Explicitly free unused cache entry on failed insertion race (Marc)
- Don't explode people's machines with a boatload of xchg() (I said it was WIP!) (Marc)

Oliver Upton (23):
KVM: arm64: Add tracepoints + stats for LPI cache effectiveness
KVM: arm64: vgic: Store LPIs in an xarray
KVM: arm64: vgic: Use xarray to find LPI in vgic_get_lpi()
KVM: arm64: vgic-v3: Iterate the xarray to find pending LPIs
KVM: arm64: vgic-its: Walk the LPI xarray in vgic_copy_lpi_list()
KVM: arm64: vgic: Get rid of the LPI linked-list
KVM: arm64: vgic: Use atomics to count LPIs
KVM: arm64: vgic: Free LPI vgic_irq structs in an RCU-safe manner
KVM: arm64: vgic: Rely on RCU protection in vgic_get_lpi()
KVM: arm64: vgic: Ensure the irq refcount is nonzero when taking a ref
KVM: arm64: vgic: Don't acquire the lpi_list_lock in vgic_put_irq()
KVM: arm64: vgic-its: Lazily allocate LPI translation cache
KVM: arm64: vgic-its: Pick cache victim based on usage count
KVM: arm64: vgic-its: Protect cached vgic_irq pointers with RCU
KVM: arm64: vgic-its: Treat the LPI translation cache as an rculist
KVM: arm64: vgic-its: Rely on RCU to protect translation cache reads
KVM: selftests: Align with kernel's GIC definitions
KVM: selftests: Standardise layout of GIC frames
KVM: selftests: Add a minimal library for interacting with an ITS
KVM: selftests: Add helper for enabling LPIs on a redistributor
KVM: selftests: Use MPIDR_HWID_BITMASK from cputype.h
KVM: selftests: Hack in support for aligned page allocations
KVM: selftests: Add stress test for LPI injection

 arch/arm64/include/asm/kvm_host.h             |   3 +
 arch/arm64/kvm/guest.c                        |   5 +-
 arch/arm64/kvm/vgic/trace.h                   |  66 ++
 arch/arm64/kvm/vgic/vgic-debug.c              |   2 +-
 arch/arm64/kvm/vgic/vgic-init.c               |   7 +-
 arch/arm64/kvm/vgic/vgic-its.c                | 220 ++++---
 arch/arm64/kvm/vgic/vgic-v3.c                 |   3 +-
 arch/arm64/kvm/vgic/vgic.c                    |  56 +-
 arch/arm64/kvm/vgic/vgic.h                    |  15 +-
 include/kvm/arm_vgic.h                        |  10 +-
 include/linux/kvm_host.h                      |   4 +
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../selftests/kvm/aarch64/arch_timer.c        |   8 +-
 .../testing/selftests/kvm/aarch64/psci_test.c |   2 +
 .../testing/selftests/kvm/aarch64/vgic_irq.c  |  15 +-
 .../selftests/kvm/aarch64/vgic_lpi_stress.c   | 388 ++++++++++++
 .../kvm/aarch64/vpmu_counter_access.c         |   6 +-
 .../selftests/kvm/dirty_log_perf_test.c       |   5 +-
 .../selftests/kvm/include/aarch64/gic.h       |  15 +-
 .../selftests/kvm/include/aarch64/gic_v3.h    | 586 +++++++++++++++++-
 .../selftests/kvm/include/aarch64/processor.h |   2 -
 .../selftests/kvm/include/aarch64/vgic.h      |  27 +-
 .../selftests/kvm/include/kvm_util_base.h     |   2 +
 tools/testing/selftests/kvm/lib/aarch64/gic.c |  18 +-
 .../selftests/kvm/lib/aarch64/gic_private.h   |   4 +-
 .../selftests/kvm/lib/aarch64/gic_v3.c        |  69 ++-
 .../testing/selftests/kvm/lib/aarch64/vgic.c  | 337 +++++++++-
 tools/testing/selftests/kvm/lib/kvm_util.c    |  27 +-
 28 files changed, 1641 insertions(+), 262 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/aarch64/vgic_lpi_stress.c

base-commit: 6613476e225e090cc9aad49be7fa504e290dd33d

Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#1 lore raw

PATCH v2 1/23 KVM: arm64: Add tracepoints + stats for LPI cache effectiveness

LPI translation and injection has been shown to have a significant impact on the performance of VM workloads, so it probably makes sense to add some signals in this area.

Introduce the concept of a KVM tracepoint that associates with a VM stat and use it for the LPI translation cache tracepoints. It isn't too uncommon for a kernel hacker to attach to tracepoints, while at the same time userspace may open a 'binary stats' FD to peek at the corresponding VM stats.

---
 arch/arm64/include/asm/kvm_host.h |  3 ++
 arch/arm64/kvm/guest.c            |  5 ++-
 arch/arm64/kvm/vgic/trace.h       | 66 +++++++++++++++++++++++++++++++
 arch/arm64/kvm/vgic/vgic-its.c    | 14 ++++++-
 include/linux/kvm_host.h          |  4 ++
 5 files changed, 89 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_host.h +3 -0
index 21c57b812569..6f88b76373a5 100644
@@ -966,6 +966,9 @@ static inline bool __vcpu_write_sys_reg_to_cpu(u64 val, int reg)
 
 struct kvm_vm_stat {
 	struct kvm_vm_stat_generic generic;
+	u64 vgic_its_trans_cache_hit;
+	u64 vgic_its_trans_cache_miss;
+	u64 vgic_its_trans_cache_victim;
 };
 
 struct kvm_vcpu_stat {
arch/arm64/kvm/guest.c +4 -1
index aaf1d4939739..354d67251fc2 100644
@@ -30,7 +30,10 @@
 #include "trace.h"
 
 const struct _kvm_stats_desc kvm_vm_stats_desc[] = {
-	KVM_GENERIC_VM_STATS()
+	KVM_GENERIC_VM_STATS(),
+	STATS_DESC_COUNTER(VM, vgic_its_trans_cache_hit),
+	STATS_DESC_COUNTER(VM, vgic_its_trans_cache_miss),
+	STATS_DESC_COUNTER(VM, vgic_its_trans_cache_victim)
 };
 
 const struct kvm_stats_header kvm_vm_stats_header = {
arch/arm64/kvm/vgic/trace.h +66 -0

72 lines not on this page · raw message

arch/arm64/kvm/vgic/vgic-its.c +12 -2
index e2764d0ffa9f..59179268ac2d 100644
@@ -20,6 +20,7 @@
 #include <asm/kvm_arm.h>
 #include <asm/kvm_mmu.h>
 
+#include "trace.h"
 #include "vgic.h"
 #include "vgic-mmio.h"
 
@@ -636,8 +637,11 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,
 	 * to the interrupt, so drop the potential reference on what
 	 * was in the cache, and increment it on the new interrupt.
 	 */
-	if (cte->irq)
+	if (cte->irq) {
+		KVM_VM_TRACE_EVENT(kvm, vgic_its_trans_cache_victim, cte->db,
+				   cte->devid, cte->eventid, cte->irq->intid);
 		__vgic_put_lpi_locked(kvm, cte->irq);
+	}
 
 	vgic_get_irq_kref(irq);
 
@@ -767,8 +771,14 @@ int vgic_its_inject_cached_translation(struct kvm *kvm, struct kvm_msi *msi)
 
 	db = (u64)msi->address_hi << 32 | msi->address_lo;
 	irq = vgic_its_check_cache(kvm, db, msi->devid, msi->data);
-	if (!irq)
+	if (!irq) {
+		KVM_VM_TRACE_EVENT(kvm, vgic_its_trans_cache_miss, db, msi->devid,
+				   msi->data);
 		return -EWOULDBLOCK;
+	}
+
+	KVM_VM_TRACE_EVENT(kvm, vgic_its_trans_cache_hit, db, msi->devid,
+			   msi->data, irq->intid);
 
 	raw_spin_lock_irqsave(&irq->irq_lock, flags);
 	irq->pending_latch = true;
include/linux/kvm_host.h +4 -0
index 7e7fd25b09b3..846b447b6798 100644
@@ -1927,6 +1927,10 @@ struct _kvm_stats_desc {
 			HALT_POLL_HIST_COUNT),				       \
 	STATS_DESC_IBOOLEAN(VCPU_GENERIC, blocking)
 
+#define KVM_VM_TRACE_EVENT(vm, event, ...)					\
+	((vm)->stat.event)++; trace_## event(__VA_ARGS__)
+
+
 extern struct dentry *kvm_debugfs_dir;
 
 ssize_t kvm_stats_read(char *id, const struct kvm_stats_header *header,
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
maintainer
#2 lore raw

Re: [PATCH v2 01/23] KVM: arm64: Add tracepoints + stats for LPI cache effectiveness

Quoted 39 lines · Oliver Upton

LPI translation and injection has been shown to have a significant impact on the performance of VM workloads, so it probably makes sense to add some signals in this area.

Introduce the concept of a KVM tracepoint that associates with a VM stat and use it for the LPI translation cache tracepoints. It isn't too uncommon for a kernel hacker to attach to tracepoints, while at the same time userspace may open a 'binary stats' FD to peek at the corresponding VM stats.

---
 arch/arm64/include/asm/kvm_host.h |  3 ++
 arch/arm64/kvm/guest.c            |  5 ++-
 arch/arm64/kvm/vgic/trace.h       | 66 +++++++++++++++++++++++++++++++
 arch/arm64/kvm/vgic/vgic-its.c    | 14 ++++++-
 include/linux/kvm_host.h          |  4 ++
 5 files changed, 89 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_host.h +3 -0
index 21c57b812569..6f88b76373a5 100644
@@ -966,6 +966,9 @@ static inline bool __vcpu_write_sys_reg_to_cpu(u64 val, int reg)
 
 struct kvm_vm_stat {
 	struct kvm_vm_stat_generic generic;
+	u64 vgic_its_trans_cache_hit;
+	u64 vgic_its_trans_cache_miss;
+	u64 vgic_its_trans_cache_victim;
 };
 
 struct kvm_vcpu_stat {
arch/arm64/kvm/guest.c +0 -0
index aaf1d4939739..354d67251fc2 100644

arch/arm64/kvm/guest.c line 37

Quoted 9 lines
arch/arm64/kvm/guest.c +4 -1
@@ -30,7 +30,10 @@
 #include "trace.h"
 
 const struct _kvm_stats_desc kvm_vm_stats_desc[] = {
-	KVM_GENERIC_VM_STATS()
+	KVM_GENERIC_VM_STATS(),
+	STATS_DESC_COUNTER(VM, vgic_its_trans_cache_hit),
+	STATS_DESC_COUNTER(VM, vgic_its_trans_cache_miss),
+	STATS_DESC_COUNTER(VM, vgic_its_trans_cache_victim)
 };

We've talked about this offline, but I thought I'd make my position public on these.

I've very concerned that exposing these statistic, however useful they may be at a given point in time, will eventually become all wrong and/or misleading.

Case in point, our discussion about this very series, where we landed on a potential reimplementation of the translation cache as a per-ITS xarray. If this comes to fruition, these stats will probably be totally useless (hit monotonically increasing at the rate of 1 per interrupt, miss being 0 or 1, victim being stuck to 0).

I'm *not* saying that such stats are totally useless. Just that they are, by definition, tied to a kernel-side implementation choice, which will evolve. The trouble is that they create an ABI that we need to support forever.

I wish we could work on something that would allow such statistics to be *extracted* (as opposed to published). Something like this as a discussion subject for a future KVM Forum or LPC uConf would get my full backing (and I'm pretty sure the networking folks have solved that problem a long time ago).

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#3 lore raw

PATCH v2 2/23 KVM: arm64: vgic: Store LPIs in an xarray

In reply to Oliver Upton

Using a linked-list for LPIs is less than ideal as it of course requires iterative searches to find a particular entry. An xarray is a better data structure for this use case, as it provides faster searches and can still handle a potentially sparse range of INTID allocations.

Start by storing LPIs in an xarray, punting usage of the xarray to a subsequent change.

---
 arch/arm64/kvm/vgic/vgic-init.c |  3 +++
 arch/arm64/kvm/vgic/vgic-its.c  | 16 ++++++++++++++++
 arch/arm64/kvm/vgic/vgic.c      |  1 +
 include/kvm/arm_vgic.h          |  2 ++
 4 files changed, 22 insertions(+)
arch/arm64/kvm/vgic/vgic-init.c +3 -0
index e949e1d0fd9f..411719053107 100644
@@ -56,6 +56,7 @@ void kvm_vgic_early_init(struct kvm *kvm)
 	INIT_LIST_HEAD(&dist->lpi_list_head);
 	INIT_LIST_HEAD(&dist->lpi_translation_cache);
 	raw_spin_lock_init(&dist->lpi_list_lock);
+	xa_init_flags(&dist->lpi_xa, XA_FLAGS_LOCK_IRQ);
 }
 
 /* CREATION */
@@ -366,6 +367,8 @@ static void kvm_vgic_dist_destroy(struct kvm *kvm)
 
 	if (vgic_supports_direct_msis(kvm))
 		vgic_v4_teardown(kvm);
+
+	xa_destroy(&dist->lpi_xa);
 }
 
 static void __kvm_vgic_vcpu_destroy(struct kvm_vcpu *vcpu)
arch/arm64/kvm/vgic/vgic-its.c +16 -0
index 59179268ac2d..0265cd1f2d6e 100644
@@ -53,6 +53,12 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,
 	if (!irq)
 		return ERR_PTR(-ENOMEM);
 
+	ret = xa_reserve_irq(&dist->lpi_xa, intid, GFP_KERNEL_ACCOUNT);
+	if (ret) {
+		kfree(irq);
+		return ERR_PTR(ret);
+	}
+
 	INIT_LIST_HEAD(&irq->lpi_list);
 	INIT_LIST_HEAD(&irq->ap_list);
 	raw_spin_lock_init(&irq->irq_lock);
@@ -87,10 +93,20 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,
 		goto out_unlock;
 	}
 
+	ret = xa_err(xa_store(&dist->lpi_xa, intid, irq, 0));
+	if (ret) {
+		xa_release(&dist->lpi_xa, intid);
+		kfree(irq);
+		goto out_unlock;
+	}
+
 	list_add_tail(&irq->lpi_list, &dist->lpi_list_head);
 	dist->lpi_list_count++;
 
 out_unlock:
+	if (ret)
+		return ERR_PTR(ret);
+
 	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
 
 	/*
arch/arm64/kvm/vgic/vgic.c +1 -0
index db2a95762b1b..c126014f8395 100644
@@ -131,6 +131,7 @@ void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq)
 		return;
 
 	list_del(&irq->lpi_list);
+	xa_erase(&dist->lpi_xa, irq->intid);
 	dist->lpi_list_count--;
 
 	kfree(irq);
include/kvm/arm_vgic.h +2 -0
index 8cc38e836f54..795b35656b54 100644
@@ -13,6 +13,7 @@
 #include <linux/spinlock.h>
 #include <linux/static_key.h>
 #include <linux/types.h>
+#include <linux/xarray.h>
 #include <kvm/iodev.h>
 #include <linux/list.h>
 #include <linux/jump_label.h>
@@ -275,6 +276,7 @@ struct vgic_dist {
 
 	/* Protects the lpi_list and the count value below. */
 	raw_spinlock_t		lpi_list_lock;
+	struct xarray		lpi_xa;
 	struct list_head	lpi_list_head;
 	int			lpi_list_count;
 
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#4 lore raw

Re: [PATCH v2 02/23] KVM: arm64: vgic: Store LPIs in an xarray

On Tue, Feb 13, 2024 at 09:32:39AM +0000, Oliver Upton wrote:

[...]

Quoted 9 lines
(quoted diff) +5 -0
@@ -87,10 +93,20 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,
 		goto out_unlock;
 	}
 
+	ret = xa_err(xa_store(&dist->lpi_xa, intid, irq, 0));
+	if (ret) {
+		xa_release(&dist->lpi_xa, intid);
+		kfree(irq);
+		goto out_unlock;

Review static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, · line 110

Quoted 10 lines
(quoted diff) +5 -0
+	}
+
 	list_add_tail(&irq->lpi_list, &dist->lpi_list_head);
 	dist->lpi_list_count++;
 
 out_unlock:
+	if (ret)
+		return ERR_PTR(ret);
+
 	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);

Hah, I need to get my head checked. This is *still* wrong!

---

arch/arm64/kvm/vgic/vgic-its.c +2 -2
index 0265cd1f2d6e..5579660abf7a 100644
@@ -104,11 +104,11 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,
 	dist->lpi_list_count++;
 
 out_unlock:
+	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
+
 	if (ret)
 		return ERR_PTR(ret);
 
-	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
-
 	/*
 	 * We "cache" the configuration table entries in our struct vgic_irq's.
 	 * However we only have those structs for mapped IRQs, so we read in
Signature · 2 lines
Thanks,
Oliver
#5 lore raw

PATCH v2 3/23 KVM: arm64: vgic: Use xarray to find LPI in vgic_get_lpi()

In reply to Oliver Upton

Iterating over the LPI linked-list is less than ideal when the desired index is already known. Use the INTID to index the LPI xarray instead.

---
 arch/arm64/kvm/vgic/vgic.c | 19 +++++--------------
 1 file changed, 5 insertions(+), 14 deletions(-)
arch/arm64/kvm/vgic/vgic.c +5 -14
index c126014f8395..d90c42ff051d 100644
@@ -54,8 +54,9 @@ struct vgic_global kvm_vgic_global_state __ro_after_init = {
  */
 
 /*
- * Iterate over the VM's list of mapped LPIs to find the one with a
- * matching interrupt ID and return a reference to the IRQ structure.
+ * Index the VM's xarray of mapped LPIs and return a reference to the IRQ
+ * structure. The caller is expected to call vgic_put_irq() later once it's
+ * finished with the IRQ.
  */
 static struct vgic_irq *vgic_get_lpi(struct kvm *kvm, u32 intid)
 {
@@ -65,20 +66,10 @@ static struct vgic_irq *vgic_get_lpi(struct kvm *kvm, u32 intid)
 
 	raw_spin_lock_irqsave(&dist->lpi_list_lock, flags);
 
-	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
-		if (irq->intid != intid)
-			continue;
-
-		/*
-		 * This increases the refcount, the caller is expected to
-		 * call vgic_put_irq() later once it's finished with the IRQ.
-		 */
+	irq = xa_load(&dist->lpi_xa, intid);
+	if (irq)
 		vgic_get_irq_kref(irq);
-		goto out_unlock;
-	}
-	irq = NULL;
 
-out_unlock:
 	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
 
 	return irq;
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#6 lore raw

PATCH v2 4/23 KVM: arm64: vgic-v3: Iterate the xarray to find pending LPIs

In reply to Oliver Upton

Start walking the LPI xarray to find pending LPIs in preparation for the removal of the LPI linked-list. Note that the 'basic' iterator is chosen here as each iteration needs to drop the xarray read lock (RCU) as reads/writes to guest memory can potentially block.

---
 arch/arm64/kvm/vgic/vgic-v3.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/arm64/kvm/vgic/vgic-v3.c +2 -1
index 9465d3706ab9..4ea3340786b9 100644
@@ -380,6 +380,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
 	struct vgic_irq *irq;
 	gpa_t last_ptr = ~(gpa_t)0;
 	bool vlpi_avail = false;
+	unsigned long index;
 	int ret = 0;
 	u8 val;
 
@@ -396,7 +397,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
 		vlpi_avail = true;
 	}
 
-	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
+	xa_for_each(&dist->lpi_xa, index, irq) {
 		int byte_offset, bit_nr;
 		struct kvm_vcpu *vcpu;
 		gpa_t pendbase, ptr;
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#7 lore raw

PATCH v2 5/23 KVM: arm64: vgic-its: Walk the LPI xarray in vgic_copy_lpi_list()

In reply to Oliver Upton

Start iterating the LPI xarray in anticipation of removing the LPI linked-list.

---
 arch/arm64/kvm/vgic/vgic-its.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
arch/arm64/kvm/vgic/vgic-its.c +6 -1
index 0265cd1f2d6e..aefe0794b71c 100644
@@ -336,6 +336,7 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
 int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr)
 {
 	struct vgic_dist *dist = &kvm->arch.vgic;
+	XA_STATE(xas, &dist->lpi_xa, GIC_LPI_OFFSET);
 	struct vgic_irq *irq;
 	unsigned long flags;
 	u32 *intids;
@@ -354,7 +355,9 @@ int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr)
 		return -ENOMEM;
 
 	raw_spin_lock_irqsave(&dist->lpi_list_lock, flags);
-	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
+	rcu_read_lock();
+
+	xas_for_each(&xas, irq, INTERRUPT_ID_BITS_ITS) {
 		if (i == irq_count)
 			break;
 		/* We don't need to "get" the IRQ, as we hold the list lock. */
@@ -362,6 +365,8 @@ int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr)
 			continue;
 		intids[i++] = irq->intid;
 	}
+
+	rcu_read_unlock();
 	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
 
 	*intid_ptr = intids;
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#8 lore raw

PATCH v2 6/23 KVM: arm64: vgic: Get rid of the LPI linked-list

In reply to Oliver Upton

All readers of LPI configuration have been transitioned to use the LPI xarray. Get rid of the linked-list altogether.

---
 arch/arm64/kvm/vgic/vgic-init.c | 1 -
 arch/arm64/kvm/vgic/vgic-its.c  | 8 ++------
 arch/arm64/kvm/vgic/vgic.c      | 1 -
 include/kvm/arm_vgic.h          | 2 --
 4 files changed, 2 insertions(+), 10 deletions(-)
arch/arm64/kvm/vgic/vgic-init.c +0 -1
index 411719053107..e25672d6e846 100644
@@ -53,7 +53,6 @@ void kvm_vgic_early_init(struct kvm *kvm)
 {
 	struct vgic_dist *dist = &kvm->arch.vgic;
 
-	INIT_LIST_HEAD(&dist->lpi_list_head);
 	INIT_LIST_HEAD(&dist->lpi_translation_cache);
 	raw_spin_lock_init(&dist->lpi_list_lock);
 	xa_init_flags(&dist->lpi_xa, XA_FLAGS_LOCK_IRQ);
arch/arm64/kvm/vgic/vgic-its.c +2 -6
index aefe0794b71c..c68164d6cba0 100644
@@ -59,7 +59,6 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,
 		return ERR_PTR(ret);
 	}
 
-	INIT_LIST_HEAD(&irq->lpi_list);
 	INIT_LIST_HEAD(&irq->ap_list);
 	raw_spin_lock_init(&irq->irq_lock);
 
@@ -75,10 +74,8 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,
 	 * There could be a race with another vgic_add_lpi(), so we need to
 	 * check that we don't add a second list entry with the same LPI.
 	 */
-	list_for_each_entry(oldirq, &dist->lpi_list_head, lpi_list) {
-		if (oldirq->intid != intid)
-			continue;
-
+	oldirq = xa_load(&dist->lpi_xa, intid);
+	if (oldirq) {
 		/* Someone was faster with adding this LPI, lets use that. */
 		kfree(irq);
 		irq = oldirq;
@@ -100,7 +97,6 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,
 		goto out_unlock;
 	}
 
-	list_add_tail(&irq->lpi_list, &dist->lpi_list_head);
 	dist->lpi_list_count++;
 
 out_unlock:
arch/arm64/kvm/vgic/vgic.c +0 -1
index d90c42ff051d..e58ce68e325c 100644
@@ -121,7 +121,6 @@ void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq)
 	if (!kref_put(&irq->refcount, vgic_irq_release))
 		return;
 
-	list_del(&irq->lpi_list);
 	xa_erase(&dist->lpi_xa, irq->intid);
 	dist->lpi_list_count--;
 
include/kvm/arm_vgic.h +0 -2
index 795b35656b54..aeff363e3ba6 100644
@@ -117,7 +117,6 @@ struct irq_ops {
 
 struct vgic_irq {
 	raw_spinlock_t irq_lock;	/* Protects the content of the struct */
-	struct list_head lpi_list;	/* Used to link all LPIs together */
 	struct list_head ap_list;
 
 	struct kvm_vcpu *vcpu;		/* SGIs and PPIs: The VCPU
@@ -277,7 +276,6 @@ struct vgic_dist {
 	/* Protects the lpi_list and the count value below. */
 	raw_spinlock_t		lpi_list_lock;
 	struct xarray		lpi_xa;
-	struct list_head	lpi_list_head;
 	int			lpi_list_count;
 
 	/* LPI translation cache */
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#9 lore raw

PATCH v2 7/23 KVM: arm64: vgic: Use atomics to count LPIs

In reply to Oliver Upton

Switch to using atomics for LPI accounting, allowing vgic_irq references to be dropped in parallel.

---
 arch/arm64/kvm/vgic/vgic-debug.c | 2 +-
 arch/arm64/kvm/vgic/vgic-its.c   | 4 ++--
 arch/arm64/kvm/vgic/vgic.c       | 2 +-
 include/kvm/arm_vgic.h           | 4 ++--
 4 files changed, 6 insertions(+), 6 deletions(-)
arch/arm64/kvm/vgic/vgic-debug.c +1 -1
index 85606a531dc3..389025ce7749 100644
@@ -149,7 +149,7 @@ static void print_dist_state(struct seq_file *s, struct vgic_dist *dist)
 	seq_printf(s, "vgic_model:\t%s\n", v3 ? "GICv3" : "GICv2");
 	seq_printf(s, "nr_spis:\t%d\n", dist->nr_spis);
 	if (v3)
-		seq_printf(s, "nr_lpis:\t%d\n", dist->lpi_list_count);
+		seq_printf(s, "nr_lpis:\t%d\n", atomic_read(&dist->lpi_count));
 	seq_printf(s, "enabled:\t%d\n", dist->enabled);
 	seq_printf(s, "\n");
 
arch/arm64/kvm/vgic/vgic-its.c +2 -2
index c68164d6cba0..048226812974 100644
@@ -97,7 +97,7 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,
 		goto out_unlock;
 	}
 
-	dist->lpi_list_count++;
+	atomic_inc(&dist->lpi_count);
 
 out_unlock:
 	if (ret)
@@ -345,7 +345,7 @@ int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr)
 	 * command). If coming from another path (such as enabling LPIs),
 	 * we must be careful not to overrun the array.
 	 */
-	irq_count = READ_ONCE(dist->lpi_list_count);
+	irq_count = atomic_read(&dist->lpi_count);
 	intids = kmalloc_array(irq_count, sizeof(intids[0]), GFP_KERNEL_ACCOUNT);
 	if (!intids)
 		return -ENOMEM;
arch/arm64/kvm/vgic/vgic.c +1 -1
index e58ce68e325c..5988d162b765 100644
@@ -122,7 +122,7 @@ void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq)
 		return;
 
 	xa_erase(&dist->lpi_xa, irq->intid);
-	dist->lpi_list_count--;
+	atomic_dec(&dist->lpi_count);
 
 	kfree(irq);
 }
include/kvm/arm_vgic.h +2 -2
index aeff363e3ba6..71e9d719533b 100644
@@ -273,10 +273,10 @@ struct vgic_dist {
 	 */
 	u64			propbaser;
 
-	/* Protects the lpi_list and the count value below. */
+	/* Protects the lpi_list. */
 	raw_spinlock_t		lpi_list_lock;
 	struct xarray		lpi_xa;
-	int			lpi_list_count;
+	atomic_t		lpi_count;
 
 	/* LPI translation cache */
 	struct list_head	lpi_translation_cache;
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
maintainer
#10 lore raw

Re: [PATCH v2 07/23] KVM: arm64: vgic: Use atomics to count LPIs

Quoted 39 lines · Oliver Upton

Switch to using atomics for LPI accounting, allowing vgic_irq references to be dropped in parallel.

---
 arch/arm64/kvm/vgic/vgic-debug.c | 2 +-
 arch/arm64/kvm/vgic/vgic-its.c   | 4 ++--
 arch/arm64/kvm/vgic/vgic.c       | 2 +-
 include/kvm/arm_vgic.h           | 4 ++--
 4 files changed, 6 insertions(+), 6 deletions(-)
arch/arm64/kvm/vgic/vgic-debug.c +1 -1
index 85606a531dc3..389025ce7749 100644
@@ -149,7 +149,7 @@ static void print_dist_state(struct seq_file *s, struct vgic_dist *dist)
 	seq_printf(s, "vgic_model:\t%s\n", v3 ? "GICv3" : "GICv2");
 	seq_printf(s, "nr_spis:\t%d\n", dist->nr_spis);
 	if (v3)
-		seq_printf(s, "nr_lpis:\t%d\n", dist->lpi_list_count);
+		seq_printf(s, "nr_lpis:\t%d\n", atomic_read(&dist->lpi_count));
 	seq_printf(s, "enabled:\t%d\n", dist->enabled);
 	seq_printf(s, "\n");
 
arch/arm64/kvm/vgic/vgic-its.c +1 -1
index c68164d6cba0..048226812974 100644
@@ -97,7 +97,7 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,
 		goto out_unlock;
 	}
 
-	dist->lpi_list_count++;
+	atomic_inc(&dist->lpi_count);
 
 out_unlock:
 	if (ret)

arch/arm64/kvm/vgic/vgic-its.c int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr) · line 348

Quoted 5 lines
arch/arm64/kvm/vgic/vgic-its.c +1 -1
@@ -345,7 +345,7 @@ int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr)
 	 * command). If coming from another path (such as enabling LPIs),
 	 * we must be careful not to overrun the array.
 	 */
-	irq_count = READ_ONCE(dist->lpi_list_count);
+	irq_count = atomic_read(&dist->lpi_count);

I'd like to propose an alternative approach here. I've always hated this "copy a bunch of INTIDs" thing, and the only purpose of this silly counter is to dimension the resulting array.

Could we instead rely on an xarray marking a bunch of entries (the ones we want to 'copy'), and get the reader to clear these marks once done?

Of course, we only have 3 marks, so that's a bit restrictive from a concurrency perspective, but since most callers hold a lock, it should be OK.

What do you think?

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#11 lore raw

Re: [PATCH v2 07/23] KVM: arm64: vgic: Use atomics to count LPIs

Hey,

Quoted 2 lines · Marc Zyngier

I'd like to propose an alternative approach here. I've always hated this "copy a bunch of INTIDs" thing,

Agree.

Quoted 2 lines

and the only purpose of this
silly counter is to dimension the resulting array.

Well, we also use it to trivially print the number of LPIs for a particular vgic in the debug interface.

Quoted 3 lines

Could we instead rely on an xarray marking a bunch of entries (the ones we want to 'copy'), and get the reader to clear these marks once done?

I think that'd work. I'm trying to convince myself we don't have bugs lurking in some of the existing usage of vgic_copy_lpi_list()...

Quoted 3 lines

Of course, we only have 3 marks, so that's a bit restrictive from a concurrency perspective, but since most callers hold a lock, it should be OK.

They all hold *a* lock, but maybe not the same one! :)

Maybe we should serialize the use of markers on the LPI list on the config_lock. A slight misuse, but we need a mutex since we're poking at guest memory. Then we can go through the whole N-dimensional locking puzzle and convince ourselves it is still correct.

Signature · 2 lines
Thanks,
Oliver
maintainer
#12 lore raw

Re: [PATCH v2 07/23] KVM: arm64: vgic: Use atomics to count LPIs

Quoted 7 lines · Oliver Upton

Hey,

Quoted 2 lines · Marc Zyngier

I'd like to propose an alternative approach here. I've always hated this "copy a bunch of INTIDs" thing,

Quoted 7 lines

Agree.

Quoted 2 lines

and the only purpose of this
silly counter is to dimension the resulting array.

Well, we also use it to trivially print the number of LPIs for a particular vgic in the debug interface.

I think we can get survive this... ;-)

Quoted 5 lines
Quoted 4 lines

Could we instead rely on an xarray marking a bunch of entries (the ones we want to 'copy'), and get the reader to clear these marks once done?

Quoted 8 lines

I think that'd work. I'm trying to convince myself we don't have bugs lurking in some of the existing usage of vgic_copy_lpi_list()...

Quoted 3 lines

Of course, we only have 3 marks, so that's a bit restrictive from a concurrency perspective, but since most callers hold a lock, it should be OK.

They all hold *a* lock, but maybe not the same one! :)

Indeed. But as long as there isn't more than 3 locks (and that the xarray is OK being concurrently updated with marks), we're good!

Quoted 4 lines

Maybe we should serialize the use of markers on the LPI list on the config_lock. A slight misuse, but we need a mutex since we're poking at guest memory. Then we can go through the whole N-dimensional locking puzzle and convince ourselves it is still correct.

Maybe. This thing is already seeing so many abuses that one more may not matter much. Need to see how it fits in the whole hierarchy of GIC-related locks...

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#13 lore raw

Re: [PATCH v2 07/23] KVM: arm64: vgic: Use atomics to count LPIs

Quoted 8 lines · Marc Zyngier
Quoted 5 lines
Quoted 3 lines

Of course, we only have 3 marks, so that's a bit restrictive from a concurrency perspective, but since most callers hold a lock, it should be OK.

They all hold *a* lock, but maybe not the same one! :)

Indeed. But as long as there isn't more than 3 locks (and that the xarray is OK being concurrently updated with marks), we're good!

Oh, you mean to give each existing caller their own mark?

Quoted 8 lines
Quoted 4 lines

Maybe we should serialize the use of markers on the LPI list on the config_lock. A slight misuse, but we need a mutex since we're poking at guest memory. Then we can go through the whole N-dimensional locking puzzle and convince ourselves it is still correct.

Maybe. This thing is already seeing so many abuses that one more may not matter much. Need to see how it fits in the whole hierarchy of GIC-related locks...

It doesn't work. We have it that the config_lock needs to be taken outside the its_lock.

Too many damn locks!

Signature · 2 lines
Thanks,
Oliver
maintainer
#14 lore raw

Re: [PATCH v2 07/23] KVM: arm64: vgic: Use atomics to count LPIs

Quoted 6 lines · Oliver Upton
Quoted 4 lines · Marc Zyngier
Quoted 4 lines
Quoted 3 lines

Of course, we only have 3 marks, so that's a bit restrictive from a concurrency perspective, but since most callers hold a lock, it should be OK.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

They all hold *a* lock, but maybe not the same one! :)

Indeed. But as long as there isn't more than 3 locks (and that the xarray is OK being concurrently updated with marks), we're good!

Oh, you mean to give each existing caller their own mark?

Well, each caller "class". Where "class" means "holding look 'foo'". Same lock, same mark. With a maximum of 3 (and I think we can get away with 2).

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Maybe we should serialize the use of markers on the LPI list on the config_lock. A slight misuse, but we need a mutex since we're poking at guest memory. Then we can go through the whole N-dimensional locking puzzle and convince ourselves it is still correct.

Quoted 8 lines
Quoted 3 lines

Maybe. This thing is already seeing so many abuses that one more may not matter much. Need to see how it fits in the whole hierarchy of GIC-related locks...

It doesn't work. We have it that the config_lock needs to be taken outside the its_lock.

Too many damn locks!

Well, the joys of emulating highly complex HW with a braindead programming interface. I'd explore the above suggestion to avoid introducing a new lock, if at all possible.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#15 lore raw

PATCH v2 8/23 KVM: arm64: vgic: Free LPI vgic_irq structs in an RCU-safe manner

In reply to Oliver Upton

Free the vgic_irq structs in an RCU-safe manner to allow reads of the LPI configuration data to happen in parallel with the release of LPIs.

---
 arch/arm64/kvm/vgic/vgic.c | 2 +-
 include/kvm/arm_vgic.h     | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
arch/arm64/kvm/vgic/vgic.c +1 -1
index 5988d162b765..be3ed4c5e1fa 100644
@@ -124,7 +124,7 @@ void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq)
 	xa_erase(&dist->lpi_xa, irq->intid);
 	atomic_dec(&dist->lpi_count);
 
-	kfree(irq);
+	kfree_rcu(irq, rcu);
 }
 
 void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq)
include/kvm/arm_vgic.h +1 -0
index 71e9d719533b..47035946648e 100644
@@ -117,6 +117,7 @@ struct irq_ops {
 
 struct vgic_irq {
 	raw_spinlock_t irq_lock;	/* Protects the content of the struct */
+	struct rcu_head rcu;
 	struct list_head ap_list;
 
 	struct kvm_vcpu *vcpu;		/* SGIs and PPIs: The VCPU
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#16 lore raw

PATCH v2 9/23 KVM: arm64: vgic: Rely on RCU protection in vgic_get_lpi()

In reply to Oliver Upton

Stop acquiring the lpi_list_lock in favor of RCU for protecting the read-side critical section in vgic_get_lpi(). In order for this to be safe, we also need to be careful not to take a reference on an irq with a refcount of 0, as it is about to be freed.

---
 arch/arm64/kvm/vgic/vgic.c |  9 ++++-----
 arch/arm64/kvm/vgic/vgic.h | 14 +++++++++++---
 2 files changed, 15 insertions(+), 8 deletions(-)
arch/arm64/kvm/vgic/vgic.c +4 -5
index be3ed4c5e1fa..128ae53a0a55 100644
@@ -62,15 +62,14 @@ static struct vgic_irq *vgic_get_lpi(struct kvm *kvm, u32 intid)
 {
 	struct vgic_dist *dist = &kvm->arch.vgic;
 	struct vgic_irq *irq = NULL;
-	unsigned long flags;
 
-	raw_spin_lock_irqsave(&dist->lpi_list_lock, flags);
+	rcu_read_lock();
 
 	irq = xa_load(&dist->lpi_xa, intid);
-	if (irq)
-		vgic_get_irq_kref(irq);
+	if (!vgic_try_get_irq_kref(irq))
+		irq = NULL;
 
-	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
+	rcu_read_unlock();
 
 	return irq;
 }
arch/arm64/kvm/vgic/vgic.h +11 -3
index 8d134569d0a1..f874b9932c5a 100644
@@ -220,12 +220,20 @@ void vgic_v2_vmcr_sync(struct kvm_vcpu *vcpu);
 void vgic_v2_save_state(struct kvm_vcpu *vcpu);
 void vgic_v2_restore_state(struct kvm_vcpu *vcpu);
 
-static inline void vgic_get_irq_kref(struct vgic_irq *irq)
+static inline bool vgic_try_get_irq_kref(struct vgic_irq *irq)
 {
+	if (!irq)
+		return false;
+
 	if (irq->intid < VGIC_MIN_LPI)
-		return;
+		return true;
 
-	kref_get(&irq->refcount);
+	return kref_get_unless_zero(&irq->refcount);
+}
+
+static inline void vgic_get_irq_kref(struct vgic_irq *irq)
+{
+	WARN_ON_ONCE(!vgic_try_get_irq_kref(irq));
 }
 
 void vgic_v3_fold_lr_state(struct kvm_vcpu *vcpu);
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#17 lore raw

PATCH v2 10/23 KVM: arm64: vgic: Ensure the irq refcount is nonzero when taking a ref

In reply to Oliver Upton

It will soon be possible for get() and put() calls to happen in parallel, which means in most cases we must ensure the refcount is nonzero when taking a new reference. Switch to using vgic_try_get_irq_kref() where necessary, and document the few conditions where an IRQ's refcount is guaranteed to be nonzero.

---
 arch/arm64/kvm/vgic/vgic-its.c | 18 ++++++++----------
 arch/arm64/kvm/vgic/vgic.c     |  3 ++-
 2 files changed, 10 insertions(+), 11 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +8 -10
index 048226812974..6b9634cec77f 100644
@@ -75,18 +75,11 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,
 	 * check that we don't add a second list entry with the same LPI.
 	 */
 	oldirq = xa_load(&dist->lpi_xa, intid);
-	if (oldirq) {
+	if (vgic_try_get_irq_kref(oldirq)) {
 		/* Someone was faster with adding this LPI, lets use that. */
 		kfree(irq);
 		irq = oldirq;
 
-		/*
-		 * This increases the refcount, the caller is expected to
-		 * call vgic_put_irq() on the returned pointer once it's
-		 * finished with the IRQ.
-		 */
-		vgic_get_irq_kref(irq);
-
 		goto out_unlock;
 	}
 
@@ -610,8 +603,8 @@ static struct vgic_irq *vgic_its_check_cache(struct kvm *kvm, phys_addr_t db,
 	raw_spin_lock_irqsave(&dist->lpi_list_lock, flags);
 
 	irq = __vgic_its_check_cache(dist, db, devid, eventid);
-	if (irq)
-		vgic_get_irq_kref(irq);
+	if (!vgic_try_get_irq_kref(irq))
+		irq = NULL;
 
 	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
 
@@ -660,6 +653,11 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,
 		__vgic_put_lpi_locked(kvm, cte->irq);
 	}
 
+	/*
+	 * The irq refcount is guaranteed to be nonzero while holding the
+	 * its_lock, as the ITE (and the reference it holds) cannot be freed.
+	 */
+	lockdep_assert_held(&its->its_lock);
 	vgic_get_irq_kref(irq);
 
 	cte->db		= db;
arch/arm64/kvm/vgic/vgic.c +2 -1
index 128ae53a0a55..2a288d6c0be7 100644
@@ -394,7 +394,8 @@ bool vgic_queue_irq_unlock(struct kvm *kvm, struct vgic_irq *irq,
 
 	/*
 	 * Grab a reference to the irq to reflect the fact that it is
-	 * now in the ap_list.
+	 * now in the ap_list. This is safe as the caller must already hold a
+	 * reference on the irq.
 	 */
 	vgic_get_irq_kref(irq);
 	list_add_tail(&irq->ap_list, &vcpu->arch.vgic_cpu.ap_list_head);
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#18 lore raw

PATCH v2 11/23 KVM: arm64: vgic: Don't acquire the lpi_list_lock in vgic_put_irq()

In reply to Oliver Upton

The LPI xarray's xa_lock is sufficient for synchronizing writers when freeing a given LPI. Furthermore, readers can only take a new reference on an IRQ if it was already nonzero.

Stop taking the lpi_list_lock unnecessarily and get rid of __vgic_put_lpi_locked().

---
 arch/arm64/kvm/vgic/vgic-its.c |  4 ++--
 arch/arm64/kvm/vgic/vgic.c     | 21 ++++-----------------
 arch/arm64/kvm/vgic/vgic.h     |  1 -
 3 files changed, 6 insertions(+), 20 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +2 -2
index 6b9634cec77f..50a9addebeed 100644
@@ -650,7 +650,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,
 	if (cte->irq) {
 		KVM_VM_TRACE_EVENT(kvm, vgic_its_trans_cache_victim, cte->db,
 				   cte->devid, cte->eventid, cte->irq->intid);
-		__vgic_put_lpi_locked(kvm, cte->irq);
+		vgic_put_irq(kvm, cte->irq);
 	}
 
 	/*
@@ -688,7 +688,7 @@ void vgic_its_invalidate_cache(struct kvm *kvm)
 		if (!cte->irq)
 			break;
 
-		__vgic_put_lpi_locked(kvm, cte->irq);
+		vgic_put_irq(kvm, cte->irq);
 		cte->irq = NULL;
 	}
 
arch/arm64/kvm/vgic/vgic.c +4 -17
index 2a288d6c0be7..c8208f81fdc8 100644
@@ -110,13 +110,13 @@ static void vgic_irq_release(struct kref *ref)
 {
 }
 
-/*
- * Drop the refcount on the LPI. Must be called with lpi_list_lock held.
- */
-void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq)
+void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq)
 {
 	struct vgic_dist *dist = &kvm->arch.vgic;
 
+	if (irq->intid < VGIC_MIN_LPI)
+		return;
+
 	if (!kref_put(&irq->refcount, vgic_irq_release))
 		return;
 
@@ -126,19 +126,6 @@ void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq)
 	kfree_rcu(irq, rcu);
 }
 
-void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq)
-{
-	struct vgic_dist *dist = &kvm->arch.vgic;
-	unsigned long flags;
-
-	if (irq->intid < VGIC_MIN_LPI)
-		return;
-
-	raw_spin_lock_irqsave(&dist->lpi_list_lock, flags);
-	__vgic_put_lpi_locked(kvm, irq);
-	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
-}
-
 void vgic_flush_pending_lpis(struct kvm_vcpu *vcpu)
 {
 	struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
arch/arm64/kvm/vgic/vgic.h +0 -1
index f874b9932c5a..0c2b82de8fa3 100644
@@ -180,7 +180,6 @@ vgic_get_mmio_region(struct kvm_vcpu *vcpu, struct vgic_io_device *iodev,
 		     gpa_t addr, int len);
 struct vgic_irq *vgic_get_irq(struct kvm *kvm, struct kvm_vcpu *vcpu,
 			      u32 intid);
-void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq);
 void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq);
 bool vgic_get_phys_line_level(struct vgic_irq *irq);
 void vgic_irq_set_phys_pending(struct vgic_irq *irq, bool pending);
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#19 lore raw

PATCH v2 12/23 KVM: arm64: vgic-its: Lazily allocate LPI translation cache

In reply to Oliver Upton

Reusing translation cache entries within a read-side critical section is fundamentally incompatible with an rculist. As such, we need to allocate a new entry to replace an eviction and free the removed entry afterwards.

Take this as an opportunity to remove the eager allocation of translation cache entries altogether in favor of a lazy allocation model on cache miss.

---
 arch/arm64/kvm/vgic/vgic-init.c |  3 --
 arch/arm64/kvm/vgic/vgic-its.c  | 96 +++++++++++++++------------------
 include/kvm/arm_vgic.h          |  1 +
 3 files changed, 45 insertions(+), 55 deletions(-)
arch/arm64/kvm/vgic/vgic-init.c +0 -3
index e25672d6e846..660d5ce3b610 100644
@@ -305,9 +305,6 @@ int vgic_init(struct kvm *kvm)
 		}
 	}
 
-	if (vgic_has_its(kvm))
-		vgic_lpi_translation_cache_init(kvm);
-
 	/*
 	 * If we have GICv4.1 enabled, unconditionnaly request enable the
 	 * v4 support so that we get HW-accelerated vSGIs. Otherwise, only
arch/arm64/kvm/vgic/vgic-its.c +44 -52

145 lines not on this page · raw message

include/kvm/arm_vgic.h +1 -0
index 47035946648e..431d05c01a53 100644
@@ -281,6 +281,7 @@ struct vgic_dist {
 
 	/* LPI translation cache */
 	struct list_head	lpi_translation_cache;
+	unsigned int		lpi_cache_count;
 
 	/* used by vgic-debug */
 	struct vgic_state_iter *iter;
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#20 lore raw

PATCH v2 13/23 KVM: arm64: vgic-its: Pick cache victim based on usage count

In reply to Oliver Upton

To date the translation cache LRU policy relies on the ordering of the linked-list to pick the victim, as entries are moved to the head of the list on every cache hit. These sort of transformations are incompatible with an rculist, necessitating a different strategy for recording usage in-place.

Count the number of cache hits since the last translation cache miss for every entry. The preferences for selecting a victim are as follows:

- Invalid entries over valid entries

- Valid entry with the lowest usage count

- In the case of a tie, pick the entry closest to the tail (oldest)

---
 arch/arm64/kvm/vgic/vgic-its.c | 46 ++++++++++++++++++++++++++--------
 1 file changed, 36 insertions(+), 10 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +36 -10
index a7ba20b57264..35926d5ae561 100644
@@ -157,6 +157,7 @@ struct vgic_translation_cache_entry {
 	u32			devid;
 	u32			eventid;
 	struct vgic_irq		*irq;
+	atomic64_t		usage_count;
 };
 
 /**
@@ -580,13 +581,7 @@ static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist,
 		    cte->eventid != eventid)
 			continue;
 
-		/*
-		 * Move this entry to the head, as it is the most
-		 * recently used.
-		 */
-		if (!list_is_first(&cte->entry, &dist->lpi_translation_cache))
-			list_move(&cte->entry, &dist->lpi_translation_cache);
-
+		atomic64_inc(&cte->usage_count);
 		return cte->irq;
 	}
 
@@ -619,6 +614,36 @@ static unsigned int vgic_its_max_cache_size(struct kvm *kvm)
 	return atomic_read(&kvm->online_vcpus) * LPI_DEFAULT_PCPU_CACHE_SIZE;
 }
 
+static struct vgic_translation_cache_entry *vgic_its_cache_victim(struct vgic_dist *dist,
+								  s64 *max_usage)
+{
+	struct vgic_translation_cache_entry *cte, *victim = NULL;
+	s64 min, tmp, max = S64_MIN;
+
+	/*
+	 * Find the least used cache entry since the last cache miss, preferring
+	 * older entries in the case of a tie. Return the max usage count seen
+	 * during the scan to initialize the new cache entry.
+	 */
+	list_for_each_entry(cte, &dist->lpi_translation_cache, entry) {
+		tmp = atomic64_read(&cte->usage_count);
+		max = max(max, tmp);
+
+		if (!cte->irq) {
+			victim = cte;
+			break;
+		}
+
+		if (!victim || tmp <= min) {
+			victim = cte;
+			min = tmp;
+		}
+	}
+
+	*max_usage = max;
+	return victim;
+}
+
 static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,
 				       u32 devid, u32 eventid,
 				       struct vgic_irq *irq)
@@ -627,6 +652,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,
 	struct vgic_dist *dist = &kvm->arch.vgic;
 	unsigned long flags;
 	phys_addr_t db;
+	s64 usage = 0;
 
 	/* Do not cache a directly injected interrupt */
 	if (irq->hw)
@@ -650,9 +676,8 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,
 	}
 
 	if (dist->lpi_cache_count >= vgic_its_max_cache_size(kvm)) {
-		/* Always reuse the last entry (LRU policy) */
-		victim = list_last_entry(&dist->lpi_translation_cache,
-				      typeof(*cte), entry);
+		victim = vgic_its_cache_victim(dist, &usage);
+
 		list_del(&victim->entry);
 		dist->lpi_cache_count--;
 	}
@@ -664,6 +689,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,
 	lockdep_assert_held(&its->its_lock);
 	vgic_get_irq_kref(irq);
 
+	atomic64_set(&new->usage_count, usage);
 	new->db		= db;
 	new->devid	= devid;
 	new->eventid	= eventid;
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#21 lore raw

PATCH v2 14/23 KVM: arm64: vgic-its: Protect cached vgic_irq pointers with RCU

In reply to Oliver Upton

RCU readers of the LPI translation cache will be able to run in parallel with a cache invalidation, which clears the RCU pointer. Start using RCU protection on the cached irq pointer in light of this.

---
 arch/arm64/kvm/vgic/vgic-its.c | 28 ++++++++++++++++++----------
 1 file changed, 18 insertions(+), 10 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +18 -10
index 35926d5ae561..99042ecc9c85 100644
@@ -156,7 +156,7 @@ struct vgic_translation_cache_entry {
 	phys_addr_t		db;
 	u32			devid;
 	u32			eventid;
-	struct vgic_irq		*irq;
+	struct vgic_irq __rcu	*irq;
 	atomic64_t		usage_count;
 };
 
@@ -574,7 +574,7 @@ static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist,
 		 * If we hit a NULL entry, there is nothing after this
 		 * point.
 		 */
-		if (!cte->irq)
+		if (!rcu_access_pointer(cte->irq))
 			break;
 
 		if (cte->db != db || cte->devid != devid ||
@@ -582,7 +582,7 @@ static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist,
 			continue;
 
 		atomic64_inc(&cte->usage_count);
-		return cte->irq;
+		return rcu_dereference(cte->irq);
 	}
 
 	return NULL;
@@ -629,7 +629,7 @@ static struct vgic_translation_cache_entry *vgic_its_cache_victim(struct vgic_di
 		tmp = atomic64_read(&cte->usage_count);
 		max = max(max, tmp);
 
-		if (!cte->irq) {
+		if (!rcu_access_pointer(cte->irq)) {
 			victim = cte;
 			break;
 		}
@@ -663,6 +663,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,
 		return;
 
 	raw_spin_lock_irqsave(&dist->lpi_list_lock, flags);
+	rcu_read_lock();
 
 	/*
 	 * We could have raced with another CPU caching the same
@@ -693,27 +694,32 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,
 	new->db		= db;
 	new->devid	= devid;
 	new->eventid	= eventid;
-	new->irq	= irq;
+	rcu_assign_pointer(new->irq, irq);
 
 	/* Move the new translation to the head of the list */
 	list_add(&new->entry, &dist->lpi_translation_cache);
 	dist->lpi_cache_count++;
 
 out:
+	rcu_read_unlock();
 	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
 
 	if (!victim)
 		return;
 
+	synchronize_rcu();
+
 	/*
 	 * Caching the translation implies having an extra reference
 	 * to the interrupt, so drop the potential reference on what
-	 * was in the cache, and increment it on the new interrupt.
+	 * was in the cache.
 	 */
 	if (victim->irq) {
+		struct vgic_irq *irq = rcu_dereference_raw(victim->irq);
+
 		KVM_VM_TRACE_EVENT(kvm, vgic_its_trans_cache_victim, victim->db,
-				   victim->devid, victim->eventid, victim->irq->intid);
-		vgic_put_irq(kvm, victim->irq);
+				   victim->devid, victim->eventid, irq->intid);
+		vgic_put_irq(kvm, irq);
 	}
 
 	kfree(victim);
@@ -726,19 +732,21 @@ void vgic_its_invalidate_cache(struct kvm *kvm)
 	unsigned long flags;
 
 	raw_spin_lock_irqsave(&dist->lpi_list_lock, flags);
+	rcu_read_lock();
 
 	list_for_each_entry(cte, &dist->lpi_translation_cache, entry) {
 		/*
 		 * If we hit a NULL entry, there is nothing after this
 		 * point.
 		 */
-		if (!cte->irq)
+		if (!rcu_access_pointer(cte->irq))
 			break;
 
 		vgic_put_irq(kvm, cte->irq);
-		cte->irq = NULL;
+		rcu_assign_pointer(cte->irq, NULL);
 	}
 
+	rcu_read_unlock();
 	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
 }
 
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#22 lore raw

PATCH v2 15/23 KVM: arm64: vgic-its: Treat the LPI translation cache as an rculist

In reply to Oliver Upton

Convert the LPI translation cache to an rculist such that readers can walk it while only holding the RCU read lock.

---
 arch/arm64/kvm/vgic/vgic-its.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +6 -6
index 99042ecc9c85..55463eb84763 100644
@@ -569,7 +569,7 @@ static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist,
 {
 	struct vgic_translation_cache_entry *cte;
 
-	list_for_each_entry(cte, &dist->lpi_translation_cache, entry) {
+	list_for_each_entry_rcu(cte, &dist->lpi_translation_cache, entry) {
 		/*
 		 * If we hit a NULL entry, there is nothing after this
 		 * point.
@@ -625,7 +625,7 @@ static struct vgic_translation_cache_entry *vgic_its_cache_victim(struct vgic_di
 	 * older entries in the case of a tie. Return the max usage count seen
 	 * during the scan to initialize the new cache entry.
 	 */
-	list_for_each_entry(cte, &dist->lpi_translation_cache, entry) {
+	list_for_each_entry_rcu(cte, &dist->lpi_translation_cache, entry) {
 		tmp = atomic64_read(&cte->usage_count);
 		max = max(max, tmp);
 
@@ -679,7 +679,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,
 	if (dist->lpi_cache_count >= vgic_its_max_cache_size(kvm)) {
 		victim = vgic_its_cache_victim(dist, &usage);
 
-		list_del(&victim->entry);
+		list_del_rcu(&victim->entry);
 		dist->lpi_cache_count--;
 	}
 
@@ -697,7 +697,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,
 	rcu_assign_pointer(new->irq, irq);
 
 	/* Move the new translation to the head of the list */
-	list_add(&new->entry, &dist->lpi_translation_cache);
+	list_add_rcu(&new->entry, &dist->lpi_translation_cache);
 	dist->lpi_cache_count++;
 
 out:
@@ -734,7 +734,7 @@ void vgic_its_invalidate_cache(struct kvm *kvm)
 	raw_spin_lock_irqsave(&dist->lpi_list_lock, flags);
 	rcu_read_lock();
 
-	list_for_each_entry(cte, &dist->lpi_translation_cache, entry) {
+	list_for_each_entry_rcu(cte, &dist->lpi_translation_cache, entry) {
 		/*
 		 * If we hit a NULL entry, there is nothing after this
 		 * point.
@@ -1981,7 +1981,7 @@ void vgic_lpi_translation_cache_destroy(struct kvm *kvm)
 
 	list_for_each_entry_safe(cte, tmp,
 				 &dist->lpi_translation_cache, entry) {
-		list_del(&cte->entry);
+		list_del_rcu(&cte->entry);
 		kfree(cte);
 	}
 }
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#23 lore raw

PATCH v2 16/23 KVM: arm64: vgic-its: Rely on RCU to protect translation cache reads

In reply to Oliver Upton

Stop taking the lpi_list_lock when reading the LPI translation cache.

---
 arch/arm64/kvm/vgic/vgic-its.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +2 -3
index 55463eb84763..0e449f8c1f01 100644
@@ -593,15 +593,14 @@ static struct vgic_irq *vgic_its_check_cache(struct kvm *kvm, phys_addr_t db,
 {
 	struct vgic_dist *dist = &kvm->arch.vgic;
 	struct vgic_irq *irq;
-	unsigned long flags;
 
-	raw_spin_lock_irqsave(&dist->lpi_list_lock, flags);
+	rcu_read_lock();
 
 	irq = __vgic_its_check_cache(dist, db, devid, eventid);
 	if (!vgic_try_get_irq_kref(irq))
 		irq = NULL;
 
-	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
+	rcu_read_unlock();
 
 	return irq;
 }
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#24 lore raw

PATCH v2 17/23 KVM: selftests: Align with kernel's GIC definitions

In reply to Oliver Upton

There are a few subtle incongruencies between the GIC definitions used by the kernel and selftests. Furthermore, the selftests header blends implementation detail (e.g. default priority) with the architectural definitions.

This is all rather annoying, since bulk imports of the kernel header is not possible. Move selftests-specific definitions out of the offending header and realign tests on the canonical definitions for things like sysregs. Finally, haul in a fresh copy of the gicv3 header to enable a forthcoming ITS selftest.

---
 .../testing/selftests/kvm/aarch64/vgic_irq.c  |   4 +-
 .../selftests/kvm/include/aarch64/gic_v3.h    | 586 +++++++++++++++++-
 .../selftests/kvm/lib/aarch64/gic_v3.c        |  13 +-
 3 files changed, 568 insertions(+), 35 deletions(-)
tools/testing/selftests/kvm/aarch64/vgic_irq.c +2 -2
index 2e64b4856e38..d61a6302f467 100644
@@ -152,7 +152,7 @@ static void reset_stats(void)
 
 static uint64_t gic_read_ap1r0(void)
 {
-	uint64_t reg = read_sysreg_s(SYS_ICV_AP1R0_EL1);
+	uint64_t reg = read_sysreg_s(SYS_ICC_AP1R0_EL1);
 
 	dsb(sy);
 	return reg;
@@ -160,7 +160,7 @@ static uint64_t gic_read_ap1r0(void)
 
 static void gic_write_ap1r0(uint64_t val)
 {
-	write_sysreg_s(val, SYS_ICV_AP1R0_EL1);
+	write_sysreg_s(val, SYS_ICC_AP1R0_EL1);
 	isb();
 }
 
tools/testing/selftests/kvm/include/aarch64/gic_v3.h +554 -32

636 lines not on this page · raw message

tools/testing/selftests/kvm/lib/aarch64/gic_v3.c +12 -1
index 263bf3ed8fd5..cd8f0e209599 100644
@@ -9,9 +9,20 @@
 #include "processor.h"
 #include "delay.h"
 
+#include "gic.h"
 #include "gic_v3.h"
 #include "gic_private.h"
 
+#define GICV3_MAX_CPUS			512
+
+#define GICD_INT_DEF_PRI		0xa0
+#define GICD_INT_DEF_PRI_X4		((GICD_INT_DEF_PRI << 24) |\
+					(GICD_INT_DEF_PRI << 16) |\
+					(GICD_INT_DEF_PRI << 8) |\
+					GICD_INT_DEF_PRI)
+
+#define ICC_PMR_DEF_PRIO		0xf0
+
 struct gicv3_data {
 	void *dist_base;
 	void *redist_base[GICV3_MAX_CPUS];
@@ -320,7 +331,7 @@ static void gicv3_cpu_init(unsigned int cpu, void *redist_base)
 	write_sysreg_s(ICC_PMR_DEF_PRIO, SYS_ICC_PMR_EL1);
 
 	/* Enable non-secure Group-1 interrupts */
-	write_sysreg_s(ICC_IGRPEN1_EL1_ENABLE, SYS_ICC_GRPEN1_EL1);
+	write_sysreg_s(ICC_IGRPEN1_EL1_MASK, SYS_ICC_IGRPEN1_EL1);
 
 	gicv3_data.redist_base[cpu] = redist_base_cpu;
 }
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#25 lore raw

PATCH v2 18/23 KVM: selftests: Standardise layout of GIC frames

In reply to Oliver Upton

It would appear that all of the selftests are using the same exact layout for the GIC frames. Fold this back into the library implementation to avoid defining magic values all over the selftests.

This is an extension of Colton's change, ripping out parameterization of from the library internals in addition to the public interfaces.

---
 .../selftests/kvm/aarch64/arch_timer.c        |  8 +--
 .../testing/selftests/kvm/aarch64/vgic_irq.c  | 11 +---
 .../kvm/aarch64/vpmu_counter_access.c         |  6 +-
 .../selftests/kvm/dirty_log_perf_test.c       |  5 +-
 .../selftests/kvm/include/aarch64/gic.h       | 10 +++-
 .../selftests/kvm/include/aarch64/vgic.h      |  3 +-
 tools/testing/selftests/kvm/lib/aarch64/gic.c | 18 +++---
 .../selftests/kvm/lib/aarch64/gic_private.h   |  4 +-
 .../selftests/kvm/lib/aarch64/gic_v3.c        | 56 +++++++++----------
 .../testing/selftests/kvm/lib/aarch64/vgic.c  | 18 +++---
 10 files changed, 57 insertions(+), 82 deletions(-)
tools/testing/selftests/kvm/aarch64/arch_timer.c +2 -6
index 274b8465b42a..f5101898c46a 100644
@@ -59,9 +59,6 @@ static struct test_args test_args = {
 
 #define msecs_to_usecs(msec)		((msec) * 1000LL)
 
-#define GICD_BASE_GPA			0x8000000ULL
-#define GICR_BASE_GPA			0x80A0000ULL
-
 enum guest_stage {
 	GUEST_STAGE_VTIMER_CVAL = 1,
 	GUEST_STAGE_VTIMER_TVAL,
@@ -204,8 +201,7 @@ static void guest_code(void)
 
 	local_irq_disable();
 
-	gic_init(GIC_V3, test_args.nr_vcpus,
-		(void *)GICD_BASE_GPA, (void *)GICR_BASE_GPA);
+	gic_init(GIC_V3, test_args.nr_vcpus);
 
 	timer_set_ctl(VIRTUAL, CTL_IMASK);
 	timer_set_ctl(PHYSICAL, CTL_IMASK);
@@ -391,7 +387,7 @@ static struct kvm_vm *test_vm_create(void)
 		vcpu_init_descriptor_tables(vcpus[i]);
 
 	test_init_timer_irq(vm);
-	gic_fd = vgic_v3_setup(vm, nr_vcpus, 64, GICD_BASE_GPA, GICR_BASE_GPA);
+	gic_fd = vgic_v3_setup(vm, nr_vcpus, 64);
 	__TEST_REQUIRE(gic_fd >= 0, "Failed to create vgic-v3");
 
 	/* Make all the test's cmdline args visible to the guest */
tools/testing/selftests/kvm/aarch64/vgic_irq.c +2 -9
index d61a6302f467..a51dbd2a5f84 100644
@@ -19,9 +19,6 @@
 #include "gic_v3.h"
 #include "vgic.h"
 
-#define GICD_BASE_GPA		0x08000000ULL
-#define GICR_BASE_GPA		0x080A0000ULL
-
 /*
  * Stores the user specified args; it's passed to the guest and to every test
  * function.
@@ -49,9 +46,6 @@ struct test_args {
 #define IRQ_DEFAULT_PRIO	(LOWEST_PRIO - 1)
 #define IRQ_DEFAULT_PRIO_REG	(IRQ_DEFAULT_PRIO << KVM_PRIO_SHIFT) /* 0xf0 */
 
-static void *dist = (void *)GICD_BASE_GPA;
-static void *redist = (void *)GICR_BASE_GPA;
-
 /*
  * The kvm_inject_* utilities are used by the guest to ask the host to inject
  * interrupts (e.g., using the KVM_IRQ_LINE ioctl).
@@ -478,7 +472,7 @@ static void guest_code(struct test_args *args)
 	bool level_sensitive = args->level_sensitive;
 	struct kvm_inject_desc *f, *inject_fns;
 
-	gic_init(GIC_V3, 1, dist, redist);
+	gic_init(GIC_V3, 1);
 
 	for (i = 0; i < nr_irqs; i++)
 		gic_irq_enable(i);
@@ -764,8 +758,7 @@ static void test_vgic(uint32_t nr_irqs, bool level_sensitive, bool eoi_split)
 	memcpy(addr_gva2hva(vm, args_gva), &args, sizeof(args));
 	vcpu_args_set(vcpu, 1, args_gva);
 
-	gic_fd = vgic_v3_setup(vm, 1, nr_irqs,
-			GICD_BASE_GPA, GICR_BASE_GPA);
+	gic_fd = vgic_v3_setup(vm, 1, nr_irqs);
 	__TEST_REQUIRE(gic_fd >= 0, "Failed to create vgic-v3, skipping");
 
 	vm_install_exception_handler(vm, VECTOR_IRQ_CURRENT,
tools/testing/selftests/kvm/aarch64/vpmu_counter_access.c +1 -5
index 9d51b5691349..496a7a3c615b 100644
@@ -420,9 +420,6 @@ static void guest_code(uint64_t expected_pmcr_n)
 	GUEST_DONE();
 }
 
-#define GICD_BASE_GPA	0x8000000ULL
-#define GICR_BASE_GPA	0x80A0000ULL
-
 /* Create a VM that has one vCPU with PMUv3 configured. */
 static void create_vpmu_vm(void *guest_code)
 {
@@ -454,8 +451,7 @@ static void create_vpmu_vm(void *guest_code)
 	init.features[0] |= (1 << KVM_ARM_VCPU_PMU_V3);
 	vpmu_vm.vcpu = aarch64_vcpu_add(vpmu_vm.vm, 0, &init, guest_code);
 	vcpu_init_descriptor_tables(vpmu_vm.vcpu);
-	vpmu_vm.gic_fd = vgic_v3_setup(vpmu_vm.vm, 1, 64,
-					GICD_BASE_GPA, GICR_BASE_GPA);
+	vpmu_vm.gic_fd = vgic_v3_setup(vpmu_vm.vm, 1, 64);
 	__TEST_REQUIRE(vpmu_vm.gic_fd >= 0,
 		       "Failed to create vgic-v3, skipping");
 
tools/testing/selftests/kvm/dirty_log_perf_test.c +1 -4
index d374dbcf9a53..53c9d7a2611d 100644
@@ -22,9 +22,6 @@
 #ifdef __aarch64__
 #include "aarch64/vgic.h"
 
-#define GICD_BASE_GPA			0x8000000ULL
-#define GICR_BASE_GPA			0x80A0000ULL
-
 static int gic_fd;
 
 static void arch_setup_vm(struct kvm_vm *vm, unsigned int nr_vcpus)
@@ -33,7 +30,7 @@ static void arch_setup_vm(struct kvm_vm *vm, unsigned int nr_vcpus)
 	 * The test can still run even if hardware does not support GICv3, as it
 	 * is only an optimization to reduce guest exits.
 	 */
-	gic_fd = vgic_v3_setup(vm, nr_vcpus, 64, GICD_BASE_GPA, GICR_BASE_GPA);
+	gic_fd = vgic_v3_setup(vm, nr_vcpus, 64);
 }
 
 static void arch_cleanup_vm(struct kvm_vm *vm)
tools/testing/selftests/kvm/include/aarch64/gic.h +8 -2
index b217ea17cac5..16d944486e9c 100644
@@ -11,6 +11,13 @@ enum gic_type {
 	GIC_TYPE_MAX,
 };
 
+#define GICD_BASE_GPA		0x8000000ULL
+#define GICR_BASE_GPA		(GICD_BASE_GPA + SZ_64K)
+
+/* The GIC is identity-mapped into the guest at the time of setup. */
+#define GICD_BASE_GVA		((void *)GICD_BASE_GPA)
+#define GICR_BASE_GVA		((void *)GICR_BASE_GPA)
+
 #define MIN_SGI			0
 #define MIN_PPI			16
 #define MIN_SPI			32
@@ -21,8 +28,7 @@ enum gic_type {
 #define INTID_IS_PPI(intid)	(MIN_PPI <= (intid) && (intid) < MIN_SPI)
 #define INTID_IS_SPI(intid)	(MIN_SPI <= (intid) && (intid) <= MAX_SPI)
 
-void gic_init(enum gic_type type, unsigned int nr_cpus,
-		void *dist_base, void *redist_base);
+void gic_init(enum gic_type type, unsigned int nr_cpus);
 void gic_irq_enable(unsigned int intid);
 void gic_irq_disable(unsigned int intid);
 unsigned int gic_get_and_ack_irq(void);
tools/testing/selftests/kvm/include/aarch64/vgic.h +1 -2
index 0ac6f05c63f9..ce19aa0a8360 100644
@@ -16,8 +16,7 @@
 	((uint64_t)(flags) << 12) | \
 	index)
 
-int vgic_v3_setup(struct kvm_vm *vm, unsigned int nr_vcpus, uint32_t nr_irqs,
-		uint64_t gicd_base_gpa, uint64_t gicr_base_gpa);
+int vgic_v3_setup(struct kvm_vm *vm, unsigned int nr_vcpus, uint32_t nr_irqs);
 
 #define VGIC_MAX_RESERVED	1023
 
tools/testing/selftests/kvm/lib/aarch64/gic.c +7 -11
index 55668631d546..7abbf8866512 100644
@@ -17,13 +17,12 @@
 static const struct gic_common_ops *gic_common_ops;
 static struct spinlock gic_lock;
 
-static void gic_cpu_init(unsigned int cpu, void *redist_base)
+static void gic_cpu_init(unsigned int cpu)
 {
-	gic_common_ops->gic_cpu_init(cpu, redist_base);
+	gic_common_ops->gic_cpu_init(cpu);
 }
 
-static void
-gic_dist_init(enum gic_type type, unsigned int nr_cpus, void *dist_base)
+static void gic_dist_init(enum gic_type type, unsigned int nr_cpus)
 {
 	const struct gic_common_ops *gic_ops = NULL;
 
@@ -40,7 +39,7 @@ gic_dist_init(enum gic_type type, unsigned int nr_cpus, void *dist_base)
 
 	GUEST_ASSERT(gic_ops);
 
-	gic_ops->gic_init(nr_cpus, dist_base);
+	gic_ops->gic_init(nr_cpus);
 	gic_common_ops = gic_ops;
 
 	/* Make sure that the initialized data is visible to all the vCPUs */
@@ -49,18 +48,15 @@ gic_dist_init(enum gic_type type, unsigned int nr_cpus, void *dist_base)
 	spin_unlock(&gic_lock);
 }
 
-void gic_init(enum gic_type type, unsigned int nr_cpus,
-		void *dist_base, void *redist_base)
+void gic_init(enum gic_type type, unsigned int nr_cpus)
 {
 	uint32_t cpu = guest_get_vcpuid();
 
 	GUEST_ASSERT(type < GIC_TYPE_MAX);
-	GUEST_ASSERT(dist_base);
-	GUEST_ASSERT(redist_base);
 	GUEST_ASSERT(nr_cpus);
 
-	gic_dist_init(type, nr_cpus, dist_base);
-	gic_cpu_init(cpu, redist_base);
+	gic_dist_init(type, nr_cpus);
+	gic_cpu_init(cpu);
 }
 
 void gic_irq_enable(unsigned int intid)
tools/testing/selftests/kvm/lib/aarch64/gic_private.h +2 -2
index 75d07313c893..d24e9ecc96c6 100644
@@ -8,8 +8,8 @@
 #define SELFTEST_KVM_GIC_PRIVATE_H
 
 struct gic_common_ops {
-	void (*gic_init)(unsigned int nr_cpus, void *dist_base);
-	void (*gic_cpu_init)(unsigned int cpu, void *redist_base);
+	void (*gic_init)(unsigned int nr_cpus);
+	void (*gic_cpu_init)(unsigned int cpu);
 	void (*gic_irq_enable)(unsigned int intid);
 	void (*gic_irq_disable)(unsigned int intid);
 	uint64_t (*gic_read_iar)(void);
tools/testing/selftests/kvm/lib/aarch64/gic_v3.c +25 -31

149 lines not on this page · raw message

tools/testing/selftests/kvm/lib/aarch64/vgic.c +8 -10
index b5f28d21a947..ac55b6c2e915 100644
@@ -19,8 +19,6 @@
  * Input args:
  *	vm - KVM VM
  *	nr_vcpus - Number of vCPUs supported by this VM
- *	gicd_base_gpa - Guest Physical Address of the Distributor region
- *	gicr_base_gpa - Guest Physical Address of the Redistributor region
  *
  * Output args: None
  *
@@ -30,11 +28,10 @@
  * redistributor regions of the guest. Since it depends on the number of
  * vCPUs for the VM, it must be called after all the vCPUs have been created.
  */
-int vgic_v3_setup(struct kvm_vm *vm, unsigned int nr_vcpus, uint32_t nr_irqs,
-		uint64_t gicd_base_gpa, uint64_t gicr_base_gpa)
+int vgic_v3_setup(struct kvm_vm *vm, unsigned int nr_vcpus, uint32_t nr_irqs)
 {
 	int gic_fd;
-	uint64_t redist_attr;
+	uint64_t attr;
 	struct list_head *iter;
 	unsigned int nr_gic_pages, nr_vcpus_created = 0;
 
@@ -60,18 +57,19 @@ int vgic_v3_setup(struct kvm_vm *vm, unsigned int nr_vcpus, uint32_t nr_irqs,
 	kvm_device_attr_set(gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL,
 			    KVM_DEV_ARM_VGIC_CTRL_INIT, NULL);
 
+	attr = GICD_BASE_GPA;
 	kvm_device_attr_set(gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR,
-			    KVM_VGIC_V3_ADDR_TYPE_DIST, &gicd_base_gpa);
+			    KVM_VGIC_V3_ADDR_TYPE_DIST, &attr);
 	nr_gic_pages = vm_calc_num_guest_pages(vm->mode, KVM_VGIC_V3_DIST_SIZE);
-	virt_map(vm, gicd_base_gpa, gicd_base_gpa,  nr_gic_pages);
+	virt_map(vm, GICD_BASE_GPA, GICD_BASE_GPA, nr_gic_pages);
 
 	/* Redistributor setup */
-	redist_attr = REDIST_REGION_ATTR_ADDR(nr_vcpus, gicr_base_gpa, 0, 0);
+	attr = REDIST_REGION_ATTR_ADDR(nr_vcpus, GICR_BASE_GPA, 0, 0);
 	kvm_device_attr_set(gic_fd, KVM_DEV_ARM_VGIC_GRP_ADDR,
-			    KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION, &redist_attr);
+			    KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION, &attr);
 	nr_gic_pages = vm_calc_num_guest_pages(vm->mode,
 						KVM_VGIC_V3_REDIST_SIZE * nr_vcpus);
-	virt_map(vm, gicr_base_gpa, gicr_base_gpa,  nr_gic_pages);
+	virt_map(vm, GICR_BASE_GPA, GICR_BASE_GPA, nr_gic_pages);
 
 	kvm_device_attr_set(gic_fd, KVM_DEV_ARM_VGIC_GRP_CTRL,
 			    KVM_DEV_ARM_VGIC_CTRL_INIT, NULL);
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#26 lore raw

PATCH v2 19/23 KVM: selftests: Add a minimal library for interacting with an ITS

In reply to Oliver Upton

A prerequisite of testing LPI injection performance is of course instantiating an ITS for the guest. Add a small library for creating an ITS and interacting with it *from userspace*.

Yep, you read that right. KVM unintentionally allows userspace to send commands to the virtual ITS via the command queue. Besides adding test coverage for an elusive UAPI, interacting with the ITS in userspace simplifies the handling of commands that need to allocate memory, like a MAPD command with an ITT.

---
 .../selftests/kvm/include/aarch64/gic.h       |   7 +-
 .../selftests/kvm/include/aarch64/vgic.h      |  20 ++
 .../testing/selftests/kvm/lib/aarch64/vgic.c  | 241 ++++++++++++++++++
 3 files changed, 267 insertions(+), 1 deletion(-)
tools/testing/selftests/kvm/include/aarch64/gic.h +6 -1
index 16d944486e9c..abb41d67880c 100644
@@ -11,7 +11,12 @@ enum gic_type {
 	GIC_TYPE_MAX,
 };
 
-#define GICD_BASE_GPA		0x8000000ULL
+/*
+ * Note that the redistributor frames are at the end, as the range scales
+ * with the number of vCPUs in the VM.
+ */
+#define GITS_BASE_GPA		0x8000000ULL
+#define GICD_BASE_GPA		(GITS_BASE_GPA + SZ_128K)
 #define GICR_BASE_GPA		(GICD_BASE_GPA + SZ_64K)
 
 /* The GIC is identity-mapped into the guest at the time of setup. */
tools/testing/selftests/kvm/include/aarch64/vgic.h +20 -0
index ce19aa0a8360..d45b2902439d 100644
@@ -32,4 +32,24 @@ void kvm_irq_write_isactiver(int gic_fd, uint32_t intid, struct kvm_vcpu *vcpu);
 
 #define KVM_IRQCHIP_NUM_PINS	(1020 - 32)
 
+struct vgic_its {
+	int	its_fd;
+	void 	*cmdq_hva;
+	size_t	cmdq_size;
+};
+
+struct vgic_its *vgic_its_setup(struct kvm_vm *vm,
+				vm_paddr_t coll_tbl, size_t coll_tbl_sz,
+				vm_paddr_t device_tbl, size_t device_tbl_sz,
+				vm_paddr_t cmdq, size_t cmdq_size);
+void vgic_its_destroy(struct vgic_its *its);
+
+void vgic_its_send_mapd_cmd(struct vgic_its *its, u32 device_id,
+		            vm_paddr_t itt_base, size_t itt_size, bool valid);
+void vgic_its_send_mapc_cmd(struct vgic_its *its, struct kvm_vcpu *vcpu,
+			    u32 collection_id, bool valid);
+void vgic_its_send_mapti_cmd(struct vgic_its *its, u32 device_id,
+			     u32 event_id, u32 collection_id, u32 intid);
+void vgic_its_send_invall_cmd(struct vgic_its *its, u32 collection_id);
+
 #endif // SELFTEST_KVM_VGIC_H
tools/testing/selftests/kvm/lib/aarch64/vgic.c +241 -0

250 lines not on this page · raw message

Signature · 1 line
2.43.0.687.g38aa6559b0-goog
maintainer
#27 lore raw

Re: [PATCH v2 19/23] KVM: selftests: Add a minimal library for interacting with an ITS

Quoted 10 lines · Oliver Upton

A prerequisite of testing LPI injection performance is of course instantiating an ITS for the guest. Add a small library for creating an ITS and interacting with it *from userspace*.

Yep, you read that right. KVM unintentionally allows userspace to send commands to the virtual ITS via the command queue. Besides adding test coverage for an elusive UAPI, interacting with the ITS in userspace simplifies the handling of commands that need to allocate memory, like a MAPD command with an ITT.

I don't mean to derail the party, but I really think we should plug this hole. Either that, or we make it an official interface for state restore. And don't we all love to have multiple interfaces to do the same thing?

Quoted 301 lines
---
 .../selftests/kvm/include/aarch64/gic.h       |   7 +-
 .../selftests/kvm/include/aarch64/vgic.h      |  20 ++
 .../testing/selftests/kvm/lib/aarch64/vgic.c  | 241 ++++++++++++++++++
 3 files changed, 267 insertions(+), 1 deletion(-)
tools/testing/selftests/kvm/include/aarch64/gic.h +6 -1
index 16d944486e9c..abb41d67880c 100644
@@ -11,7 +11,12 @@ enum gic_type {
 	GIC_TYPE_MAX,
 };
 
-#define GICD_BASE_GPA		0x8000000ULL
+/*
+ * Note that the redistributor frames are at the end, as the range scales
+ * with the number of vCPUs in the VM.
+ */
+#define GITS_BASE_GPA		0x8000000ULL
+#define GICD_BASE_GPA		(GITS_BASE_GPA + SZ_128K)
 #define GICR_BASE_GPA		(GICD_BASE_GPA + SZ_64K)
 
 /* The GIC is identity-mapped into the guest at the time of setup. */
tools/testing/selftests/kvm/include/aarch64/vgic.h +20 -0
index ce19aa0a8360..d45b2902439d 100644
@@ -32,4 +32,24 @@ void kvm_irq_write_isactiver(int gic_fd, uint32_t intid, struct kvm_vcpu *vcpu);
 
 #define KVM_IRQCHIP_NUM_PINS	(1020 - 32)
 
+struct vgic_its {
+	int	its_fd;
+	void 	*cmdq_hva;
+	size_t	cmdq_size;
+};
+
+struct vgic_its *vgic_its_setup(struct kvm_vm *vm,
+				vm_paddr_t coll_tbl, size_t coll_tbl_sz,
+				vm_paddr_t device_tbl, size_t device_tbl_sz,
+				vm_paddr_t cmdq, size_t cmdq_size);
+void vgic_its_destroy(struct vgic_its *its);
+
+void vgic_its_send_mapd_cmd(struct vgic_its *its, u32 device_id,
+		            vm_paddr_t itt_base, size_t itt_size, bool valid);
+void vgic_its_send_mapc_cmd(struct vgic_its *its, struct kvm_vcpu *vcpu,
+			    u32 collection_id, bool valid);
+void vgic_its_send_mapti_cmd(struct vgic_its *its, u32 device_id,
+			     u32 event_id, u32 collection_id, u32 intid);
+void vgic_its_send_invall_cmd(struct vgic_its *its, u32 collection_id);
+
 #endif // SELFTEST_KVM_VGIC_H
tools/testing/selftests/kvm/lib/aarch64/vgic.c +231 -0
index ac55b6c2e915..fc7b4fbe6453 100644
@@ -12,6 +12,7 @@
 #include "vgic.h"
 #include "gic.h"
 #include "gic_v3.h"
+#include "processor.h"
 
 /*
  * vGIC-v3 default host setup
@@ -166,3 +167,243 @@ void kvm_irq_write_isactiver(int gic_fd, uint32_t intid, struct kvm_vcpu *vcpu)
 {
 	vgic_poke_irq(gic_fd, intid, vcpu, GICD_ISACTIVER);
 }
+
+static u64 vgic_its_read_reg(int its_fd, unsigned long offset)
+{
+	u64 attr;
+
+	kvm_device_attr_get(its_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS,
+			    offset, &attr);
+	return attr;
+}
+
+static void vgic_its_write_reg(int its_fd, unsigned long offset, u64 val)
+{
+	kvm_device_attr_set(its_fd, KVM_DEV_ARM_VGIC_GRP_ITS_REGS,
+			    offset, &val);
+}
+
+static unsigned long vgic_its_find_baser(int its_fd, unsigned int type)
+{
+	int i;
+
+	for (i = 0; i < GITS_BASER_NR_REGS; i++) {
+		u64 baser;
+		unsigned long offset = GITS_BASER + (i * sizeof(baser));
+
+		baser = vgic_its_read_reg(its_fd, offset);
+		if (GITS_BASER_TYPE(baser) == type)
+			return offset;
+	}
+
+	TEST_FAIL("Couldn't find an ITS BASER of type %u", type);
+	return -1;
+}
+
+static void vgic_its_install_table(int its_fd, unsigned int type, vm_paddr_t base,
+				   size_t size)
+{
+	unsigned long offset = vgic_its_find_baser(its_fd, type);
+	u64 baser;
+
+	baser = ((size / SZ_64K) - 1) |
+		GITS_BASER_PAGE_SIZE_64K |
+		GITS_BASER_InnerShareable |
+		base |
+		GITS_BASER_RaWaWb |
+		GITS_BASER_VALID;
+
+	vgic_its_write_reg(its_fd, offset, baser);
+}
+
+static void vgic_its_install_cmdq(int its_fd, vm_paddr_t base, size_t size)
+{
+	u64 cbaser;
+
+	cbaser = ((size / SZ_4K) - 1) |
+		 GITS_CBASER_InnerShareable |
+		 base |
+		 GITS_CBASER_RaWaWb |
+		 GITS_CBASER_VALID;
+
+	vgic_its_write_reg(its_fd, GITS_CBASER, cbaser);
+}
+
+struct vgic_its *vgic_its_setup(struct kvm_vm *vm,
+				vm_paddr_t coll_tbl, size_t coll_tbl_sz,
+				vm_paddr_t device_tbl, size_t device_tbl_sz,
+				vm_paddr_t cmdq, size_t cmdq_size)
+{
+	int its_fd = kvm_create_device(vm, KVM_DEV_TYPE_ARM_VGIC_ITS);
+	struct vgic_its *its = malloc(sizeof(struct vgic_its));
+	u64 attr, ctlr;
+
+	attr = GITS_BASE_GPA;
+	kvm_device_attr_set(its_fd, KVM_DEV_ARM_VGIC_GRP_ADDR,
+			    KVM_VGIC_ITS_ADDR_TYPE, &attr);
+
+	kvm_device_attr_set(its_fd, KVM_DEV_ARM_VGIC_GRP_CTRL,
+			    KVM_DEV_ARM_VGIC_CTRL_INIT, NULL);
+
+	vgic_its_install_table(its_fd, GITS_BASER_TYPE_COLLECTION, coll_tbl,
+			       coll_tbl_sz);
+	vgic_its_install_table(its_fd, GITS_BASER_TYPE_DEVICE, device_tbl,
+			       device_tbl_sz);
+
+	vgic_its_install_cmdq(its_fd, cmdq, cmdq_size);
+
+	ctlr = vgic_its_read_reg(its_fd, GITS_CTLR);
+	ctlr |= GITS_CTLR_ENABLE;
+	vgic_its_write_reg(its_fd, GITS_CTLR, ctlr);
+
+	*its = (struct vgic_its) {
+		.its_fd		= its_fd,
+		.cmdq_hva	= addr_gpa2hva(vm, cmdq),
+		.cmdq_size	= cmdq_size,
+	};
+
+	return its;
+}
+
+void vgic_its_destroy(struct vgic_its *its)
+{
+	close(its->its_fd);
+	free(its);
+}
+
+struct its_cmd_block {
+	union {
+		u64	raw_cmd[4];
+		__le64	raw_cmd_le[4];
+	};
+};
+
+static inline void its_fixup_cmd(struct its_cmd_block *cmd)
+{
+	/* Let's fixup BE commands */
+	cmd->raw_cmd_le[0] = cpu_to_le64(cmd->raw_cmd[0]);
+	cmd->raw_cmd_le[1] = cpu_to_le64(cmd->raw_cmd[1]);
+	cmd->raw_cmd_le[2] = cpu_to_le64(cmd->raw_cmd[2]);
+	cmd->raw_cmd_le[3] = cpu_to_le64(cmd->raw_cmd[3]);
+}
+
+static void its_mask_encode(u64 *raw_cmd, u64 val, int h, int l)
+{
+	u64 mask = GENMASK_ULL(h, l);
+	*raw_cmd &= ~mask;
+	*raw_cmd |= (val << l) & mask;
+}
+
+static void its_encode_cmd(struct its_cmd_block *cmd, u8 cmd_nr)
+{
+	its_mask_encode(&cmd->raw_cmd[0], cmd_nr, 7, 0);
+}
+
+static void its_encode_devid(struct its_cmd_block *cmd, u32 devid)
+{
+	its_mask_encode(&cmd->raw_cmd[0], devid, 63, 32);
+}
+
+static void its_encode_event_id(struct its_cmd_block *cmd, u32 id)
+{
+	its_mask_encode(&cmd->raw_cmd[1], id, 31, 0);
+}
+
+static void its_encode_phys_id(struct its_cmd_block *cmd, u32 phys_id)
+{
+	its_mask_encode(&cmd->raw_cmd[1], phys_id, 63, 32);
+}
+
+static void its_encode_size(struct its_cmd_block *cmd, u8 size)
+{
+	its_mask_encode(&cmd->raw_cmd[1], size, 4, 0);
+}
+
+static void its_encode_itt(struct its_cmd_block *cmd, u64 itt_addr)
+{
+	its_mask_encode(&cmd->raw_cmd[2], itt_addr >> 8, 51, 8);
+}
+
+static void its_encode_valid(struct its_cmd_block *cmd, int valid)
+{
+	its_mask_encode(&cmd->raw_cmd[2], !!valid, 63, 63);
+}
+
+static void its_encode_target(struct its_cmd_block *cmd, u64 target_addr)
+{
+	its_mask_encode(&cmd->raw_cmd[2], target_addr >> 16, 51, 16);
+}
+
+static void its_encode_collection(struct its_cmd_block *cmd, u16 col)
+{
+	its_mask_encode(&cmd->raw_cmd[2], col, 15, 0);
+}
+
+static void vgic_its_send_cmd(struct vgic_its *its, struct its_cmd_block *cmd)
+{
+	u64 cwriter = vgic_its_read_reg(its->its_fd, GITS_CWRITER);
+	struct its_cmd_block *dst = its->cmdq_hva + cwriter;
+	u64 next;
+
+	its_fixup_cmd(cmd);
+
+	WRITE_ONCE(*dst, *cmd);
+	dsb(ishst);
+
+	next = (cwriter + sizeof(*cmd)) % its->cmdq_size;
+	vgic_its_write_reg(its->its_fd, GITS_CWRITER, next);
+
+	TEST_ASSERT(vgic_its_read_reg(its->its_fd, GITS_CREADR) == next,
+		    "ITS didn't process command at offset: %lu\n", cwriter);
+}
+
+void vgic_its_send_mapd_cmd(struct vgic_its *its, u32 device_id,
+		            vm_paddr_t itt_base, size_t itt_size, bool valid)
+{
+	struct its_cmd_block cmd = {};
+
+	its_encode_cmd(&cmd, GITS_CMD_MAPD);
+	its_encode_devid(&cmd, device_id);
+	its_encode_size(&cmd, ilog2(itt_size) - 1);
+	its_encode_itt(&cmd, itt_base);
+	its_encode_valid(&cmd, valid);
+
+	vgic_its_send_cmd(its, &cmd);
+}
+
+void vgic_its_send_mapc_cmd(struct vgic_its *its, struct kvm_vcpu *vcpu,
+			    u32 collection_id, bool valid)
+{
+	struct its_cmd_block cmd = {};
+
+	its_encode_cmd(&cmd, GITS_CMD_MAPC);
+	its_encode_collection(&cmd, collection_id);
+	its_encode_target(&cmd, vcpu->id);
+	its_encode_valid(&cmd, valid);
+
+	vgic_its_send_cmd(its, &cmd);
+}
+
+void vgic_its_send_mapti_cmd(struct vgic_its *its, u32 device_id,
+			     u32 event_id, u32 collection_id, u32 intid)
+{
+	struct its_cmd_block cmd = {};
+
+	its_encode_cmd(&cmd, GITS_CMD_MAPTI);
+	its_encode_devid(&cmd, device_id);
+	its_encode_event_id(&cmd, event_id);
+	its_encode_phys_id(&cmd, intid);
+	its_encode_collection(&cmd, collection_id);
+
+	vgic_its_send_cmd(its, &cmd);
+}

tools/testing/selftests/kvm/lib/aarch64/vgic.c void kvm_irq_write_isactiver(int gic_fd, uint32_t intid, struct kvm_vcpu *vcpu) · line 409

Quoted 10 lines
tools/testing/selftests/kvm/lib/aarch64/vgic.c +10 -0
+
+void vgic_its_send_invall_cmd(struct vgic_its *its, u32 collection_id)
+{
+	struct its_cmd_block cmd = {};
+
+	its_encode_cmd(&cmd, GITS_CMD_INVALL);
+	its_encode_collection(&cmd, collection_id);
+
+	vgic_its_send_cmd(its, &cmd);
+}

Holy crap, that's a whole ITS driver in loserspace. *mindblown*.

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#28 lore raw

Re: [PATCH v2 19/23] KVM: selftests: Add a minimal library for interacting with an ITS

Quoted 13 lines · Marc Zyngier
Quoted 10 lines · Oliver Upton

A prerequisite of testing LPI injection performance is of course instantiating an ITS for the guest. Add a small library for creating an ITS and interacting with it *from userspace*.

Yep, you read that right. KVM unintentionally allows userspace to send commands to the virtual ITS via the command queue. Besides adding test coverage for an elusive UAPI, interacting with the ITS in userspace simplifies the handling of commands that need to allocate memory, like a MAPD command with an ITT.

Quoted 4 lines

I don't mean to derail the party, but I really think we should plug this hole. Either that, or we make it an official interface for state restore. And don't we all love to have multiple interfaces to do the same thing?

Ok, I've thought about it a bit more and I'm fully convinced we need to shut the door on this stupidity.

We expect CREADR == CWRITER at the time userspace saves the ITS registers, but we have a *hideous* ordering issue on the restore path.

If the order of restore from userspace is CBASER, CWRITER, CREADR then we **wind up replaying the entire command queue**. While insane, I'm pretty sure it is legal for the guest to write garbage after the read pointer has moved past a particular command index.

Fsck!!!

So, how about we do this:

- Provide a uaccess hook for CWRITER that changes the write-pointer without processing any commands

- Assert an invariant that at any time CWRITER or CREADR are read from userspace that CREADR == CWRITER. Fail the ioctl and scream if that isn't the case, so that way we never need to worry about processing 'in-flight' commands at the destination.

Signature · 2 lines
Thanks,
Oliver
maintainer
#29 lore raw

Re: [PATCH v2 19/23] KVM: selftests: Add a minimal library for interacting with an ITS

Quoted 26 lines · Oliver Upton
Quoted 13 lines · Marc Zyngier
Quoted 10 lines · Oliver Upton

A prerequisite of testing LPI injection performance is of course instantiating an ITS for the guest. Add a small library for creating an ITS and interacting with it *from userspace*.

Yep, you read that right. KVM unintentionally allows userspace to send commands to the virtual ITS via the command queue. Besides adding test coverage for an elusive UAPI, interacting with the ITS in userspace simplifies the handling of commands that need to allocate memory, like a MAPD command with an ITT.

Quoted 4 lines

I don't mean to derail the party, but I really think we should plug this hole. Either that, or we make it an official interface for state restore. And don't we all love to have multiple interfaces to do the same thing?

Ok, I've thought about it a bit more and I'm fully convinced we need to shut the door on this stupidity.

We expect CREADR == CWRITER at the time userspace saves the ITS registers, but we have a *hideous* ordering issue on the restore path.

Quoted 6 lines

If the order of restore from userspace is CBASER, CWRITER, CREADR then we **wind up replaying the entire command queue**. While insane, I'm pretty sure it is legal for the guest to write garbage after the read pointer has moved past a particular command index.

Fsck!!!

This is documented Documentation/virt/kvm/devices/arm-vgic-its.rst to some extent, and it is allowed for the guest to crap itself on behalf of userspace if the ordering isn't respected.

Quoted 9 lines

So, how about we do this:

- Provide a uaccess hook for CWRITER that changes the write-pointer without processing any commands

- Assert an invariant that at any time CWRITER or CREADR are read from userspace that CREADR == CWRITER. Fail the ioctl and scream if that isn't the case, so that way we never need to worry about processing 'in-flight' commands at the destination.

Are we guaranteed that we cannot ever see CWRITER != CREADR at VM dumping time? I'm not convinced that we cannot preempt the vcpu thread at the right spot, specially given that you can have an arbitrary large batch of commands to execute.

Just add a page-fault to the mix, and a signal pending. Pronto, you see a guest exit and you should be able to start dumping things without the ITS having processed much. I haven't tried, but that doesn't seem totally unlikely.

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#30 lore raw

Re: [PATCH v2 19/23] KVM: selftests: Add a minimal library for interacting with an ITS

Quoted 10 lines · Marc Zyngier
Quoted 6 lines

If the order of restore from userspace is CBASER, CWRITER, CREADR then we **wind up replaying the entire command queue**. While insane, I'm pretty sure it is legal for the guest to write garbage after the read pointer has moved past a particular command index.

Fsck!!!

This is documented Documentation/virt/kvm/devices/arm-vgic-its.rst to some extent, and it is allowed for the guest to crap itself on behalf of userspace if the ordering isn't respected.

Ah, fair, I missed the documentation here. If we require userspace to write CTLR last then we _should_ be fine, but damn is this a tricky set of expectations.

Quoted 15 lines
Quoted 9 lines

So, how about we do this:

- Provide a uaccess hook for CWRITER that changes the write-pointer without processing any commands

- Assert an invariant that at any time CWRITER or CREADR are read from userspace that CREADR == CWRITER. Fail the ioctl and scream if that isn't the case, so that way we never need to worry about processing 'in-flight' commands at the destination.

Are we guaranteed that we cannot ever see CWRITER != CREADR at VM dumping time? I'm not convinced that we cannot preempt the vcpu thread at the right spot, specially given that you can have an arbitrary large batch of commands to execute.

Quoted 4 lines

Just add a page-fault to the mix, and a signal pending. Pronto, you see a guest exit and you should be able to start dumping things without the ITS having processed much. I haven't tried, but that doesn't seem totally unlikely.

Well, we would need to run all userspace reads and writes through the cmd_lock in this case, which is what we already do for the CREADR uaccess hook. To me the 'racy' queue accessors only make sense for guest accesses, since the driver is expecting to poll for completion in that case.

Otherwise we decide the existing rules for restoring the ITS are fine and I get to keep my funky driver :)

Signature · 2 lines
Thanks,
Oliver
#31 lore raw

Re: [PATCH v2 19/23] KVM: selftests: Add a minimal library for interacting with an ITS

Quoted 36 lines · Oliver Upton
Quoted 10 lines · Marc Zyngier
Quoted 6 lines

If the order of restore from userspace is CBASER, CWRITER, CREADR then we **wind up replaying the entire command queue**. While insane, I'm pretty sure it is legal for the guest to write garbage after the read pointer has moved past a particular command index.

Fsck!!!

This is documented Documentation/virt/kvm/devices/arm-vgic-its.rst to some extent, and it is allowed for the guest to crap itself on behalf of userspace if the ordering isn't respected.

Ah, fair, I missed the documentation here. If we require userspace to write CTLR last then we _should_ be fine, but damn is this a tricky set of expectations.

Quoted 15 lines
Quoted 9 lines

So, how about we do this:

- Provide a uaccess hook for CWRITER that changes the write-pointer without processing any commands

- Assert an invariant that at any time CWRITER or CREADR are read from userspace that CREADR == CWRITER. Fail the ioctl and scream if that isn't the case, so that way we never need to worry about processing 'in-flight' commands at the destination.

Are we guaranteed that we cannot ever see CWRITER != CREADR at VM dumping time? I'm not convinced that we cannot preempt the vcpu thread at the right spot, specially given that you can have an arbitrary large batch of commands to execute.

Quoted 4 lines

Just add a page-fault to the mix, and a signal pending. Pronto, you see a guest exit and you should be able to start dumping things without the ITS having processed much. I haven't tried, but that doesn't seem totally unlikely.

Quoted 5 lines

Well, we would need to run all userspace reads and writes through the cmd_lock in this case, which is what we already do for the CREADR uaccess hook. To me the 'racy' queue accessors only make sense for guest accesses, since the driver is expecting to poll for completion in that case.

My proposed invariant cannot be maintained, of course, since userspace can do whatever it pleases on the cmdq pointers.

Quoted 7 lines

Otherwise we decide the existing rules for restoring the ITS are fine and I get to keep my funky driver :)

Signature · 2 lines
Thanks,
Oliver
#32 lore raw

PATCH v2 20/23 KVM: selftests: Add helper for enabling LPIs on a redistributor

In reply to Oliver Upton

The selftests GIC library presently does not support LPIs. Add a userspace helper for configuring a redistributor for LPIs, installing an LPI configuration table and LPI pending table.

---
 .../selftests/kvm/include/aarch64/vgic.h      |  4 +
 .../testing/selftests/kvm/lib/aarch64/vgic.c  | 78 +++++++++++++++++++
 2 files changed, 82 insertions(+)
tools/testing/selftests/kvm/include/aarch64/vgic.h +4 -0
index d45b2902439d..ae6a54453c62 100644
@@ -32,6 +32,10 @@ void kvm_irq_write_isactiver(int gic_fd, uint32_t intid, struct kvm_vcpu *vcpu);
 
 #define KVM_IRQCHIP_NUM_PINS	(1020 - 32)
 
+void vgic_rdist_enable_lpis(int gic_fd, struct kvm_vcpu *vcpu,
+			    vm_paddr_t cfg_table, size_t cfg_table_size,
+			    vm_paddr_t pend_table);
+
 struct vgic_its {
 	int	its_fd;
 	void 	*cmdq_hva;
tools/testing/selftests/kvm/lib/aarch64/vgic.c +78 -0

92 lines not on this page · raw message

Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#33 lore raw

PATCH v2 21/23 KVM: selftests: Use MPIDR_HWID_BITMASK from cputype.h

In reply to Oliver Upton

No need for a home-rolled definition, just rely on the common header.

---
 tools/testing/selftests/kvm/aarch64/psci_test.c         | 2 ++
 tools/testing/selftests/kvm/include/aarch64/processor.h | 2 --
 2 files changed, 2 insertions(+), 2 deletions(-)
tools/testing/selftests/kvm/aarch64/psci_test.c +2 -0
index 9b004905d1d3..9fa3578d47d5 100644
@@ -13,7 +13,9 @@
 
 #define _GNU_SOURCE
 
+#include <linux/kernel.h>
 #include <linux/psci.h>
+#include <asm/cputype.h>
 
 #include "kvm_util.h"
 #include "processor.h"
tools/testing/selftests/kvm/include/aarch64/processor.h +0 -2
index cf20e44e86f2..94e8f1892754 100644
@@ -58,8 +58,6 @@
 	 MAIR_ATTRIDX(MAIR_ATTR_NORMAL, MT_NORMAL) |				\
 	 MAIR_ATTRIDX(MAIR_ATTR_NORMAL_WT, MT_NORMAL_WT))
 
-#define MPIDR_HWID_BITMASK (0xff00fffffful)
-
 void aarch64_vcpu_setup(struct kvm_vcpu *vcpu, struct kvm_vcpu_init *init);
 struct kvm_vcpu *aarch64_vcpu_add(struct kvm_vm *vm, uint32_t vcpu_id,
 				  struct kvm_vcpu_init *init, void *guest_code);
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#34 lore raw

PATCH v2 22/23 KVM: selftests: Hack in support for aligned page allocations

In reply to Oliver Upton

Many of the memory allocations handed off to a GIC ITS require 64K alignment. One would think that requesting 64K of pages would follow natural alignment in selftests (much like alloc_pages() in the kernel), however that is unfortunately not the case.

Add a new helper to allow a caller to decide if they want naturally-aligned page allocations. Deliberately avoid making this the default in case this subtly breaks assumptions or memory overheads in other selftests.

---
 .../selftests/kvm/include/kvm_util_base.h     |  2 ++
 tools/testing/selftests/kvm/lib/kvm_util.c    | 27 ++++++++++++++++---
 2 files changed, 25 insertions(+), 4 deletions(-)
tools/testing/selftests/kvm/include/kvm_util_base.h +2 -0
index 9e5afc472c14..750638cfa849 100644
@@ -830,6 +830,8 @@ vm_paddr_t vm_phy_page_alloc(struct kvm_vm *vm, vm_paddr_t paddr_min,
 vm_paddr_t vm_phy_pages_alloc(struct kvm_vm *vm, size_t num,
 			      vm_paddr_t paddr_min, uint32_t memslot);
 vm_paddr_t vm_alloc_page_table(struct kvm_vm *vm);
+vm_paddr_t vm_phy_pages_alloc_aligned(struct kvm_vm *vm, size_t num,
+				      vm_paddr_t paddr_min, uint32_t memslot);
 
 /*
  * ____vm_create() does KVM_CREATE_VM and little else.  __vm_create() also
tools/testing/selftests/kvm/lib/kvm_util.c +23 -4
index e066d584c656..60948a004012 100644
@@ -1984,8 +1984,9 @@ const char *exit_reason_str(unsigned int exit_reason)
  * and their base address is returned. A TEST_ASSERT failure occurs if
  * not enough pages are available at or above paddr_min.
  */
-vm_paddr_t vm_phy_pages_alloc(struct kvm_vm *vm, size_t num,
-			      vm_paddr_t paddr_min, uint32_t memslot)
+static vm_paddr_t __vm_phy_pages_alloc(struct kvm_vm *vm, size_t num,
+				       vm_paddr_t paddr_min, uint32_t memslot,
+				       bool aligned)
 {
 	struct userspace_mem_region *region;
 	sparsebit_idx_t pg, base;
@@ -1997,14 +1998,20 @@ vm_paddr_t vm_phy_pages_alloc(struct kvm_vm *vm, size_t num,
 		"  paddr_min: 0x%lx page_size: 0x%x",
 		paddr_min, vm->page_size);
 
+	TEST_ASSERT(!aligned || (paddr_min % vm->page_size * num) == 0,
+		    "Min physical address isn't naturally aligned.\n"
+		    "  paddr_min: 0x%lx page_size: 0x%x num: %lu",
+		    paddr_min, vm->page_size, num);
+
 	region = memslot2region(vm, memslot);
 	base = pg = paddr_min >> vm->page_shift;
 
 	do {
 		for (; pg < base + num; ++pg) {
 			if (!sparsebit_is_set(region->unused_phy_pages, pg)) {
-				base = pg = sparsebit_next_set(region->unused_phy_pages, pg);
-				break;
+				do {
+					base = pg = sparsebit_next_set(region->unused_phy_pages, pg);
+				} while (aligned && ((pg % num) != 0));
 			}
 		}
 	} while (pg && pg != base + num);
@@ -2024,6 +2031,12 @@ vm_paddr_t vm_phy_pages_alloc(struct kvm_vm *vm, size_t num,
 	return base * vm->page_size;
 }
 
+vm_paddr_t vm_phy_pages_alloc(struct kvm_vm *vm, size_t num,
+			      vm_paddr_t paddr_min, uint32_t memslot)
+{
+	return __vm_phy_pages_alloc(vm, num, paddr_min, memslot, false);
+}
+
 vm_paddr_t vm_phy_page_alloc(struct kvm_vm *vm, vm_paddr_t paddr_min,
 			     uint32_t memslot)
 {
@@ -2036,6 +2049,12 @@ vm_paddr_t vm_alloc_page_table(struct kvm_vm *vm)
 				 vm->memslots[MEM_REGION_PT]);
 }
 
+vm_paddr_t vm_phy_pages_alloc_aligned(struct kvm_vm *vm, size_t num,
+				      vm_paddr_t paddr_min, uint32_t memslot)
+{
+	return __vm_phy_pages_alloc(vm, num, paddr_min, memslot, true);
+}
+
 /*
  * Address Guest Virtual to Host Virtual
  *
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#35 lore raw

PATCH v2 23/23 KVM: selftests: Add stress test for LPI injection

In reply to Oliver Upton

Now that all the infrastructure is in place, add a test to stress KVM's LPI injection. Keep a 1:1 mapping of device IDs to signalling threads, allowing the user to scale up/down the sender side of an LPI. Make use of the new VM stats for the translation cache to estimate the translation hit rate.

Since the primary focus of the test is on performance, you'll notice that the guest code is not pedantic about the LPIs it receives. Counting the number of LPIs would require synchronization between the device and vCPU threads to avoid coalescing and would get in the way of performance numbers.

---
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../selftests/kvm/aarch64/vgic_lpi_stress.c   | 388 ++++++++++++++++++
 2 files changed, 389 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/aarch64/vgic_lpi_stress.c
tools/testing/selftests/kvm/Makefile +1 -0
index 492e937fab00..8a240d20ec5e 100644
@@ -153,6 +153,7 @@ TEST_GEN_PROGS_aarch64 += aarch64/smccc_filter
 TEST_GEN_PROGS_aarch64 += aarch64/vcpu_width_config
 TEST_GEN_PROGS_aarch64 += aarch64/vgic_init
 TEST_GEN_PROGS_aarch64 += aarch64/vgic_irq
+TEST_GEN_PROGS_aarch64 += aarch64/vgic_lpi_stress
 TEST_GEN_PROGS_aarch64 += aarch64/vpmu_counter_access
 TEST_GEN_PROGS_aarch64 += access_tracking_perf_test
 TEST_GEN_PROGS_aarch64 += demand_paging_test
tools/testing/selftests/kvm/aarch64/vgic_lpi_stress.c new file +388 -0

388 lines not on this page · raw message

Signature · 1 line
2.43.0.687.g38aa6559b0-goog
#36 lore raw
In reply to Oliver Upton

On Tue, Feb 13, 2024 at 09:32:37AM +0000, Oliver Upton wrote:

[...]

Quoted 30 lines

Clearly the RCU synchronization is a bounding issue in this case. I think other scenarios where the cache is overcommitted (16 vCPUs, 16 devices, 17 events / device) are able to hide effects somewhat, as other threads can make forward progress while others are stuck waiting on RCU.

A few ideas on next steps:

1) Rework the lpi_list_lock as an rwlock. This would obviate the need for RCU protection in the LPI cache as well as memory allocations on the injection path. This is actually what I had in the internal version of the series, although it was very incomplete.

I'd expect this to nullify the improvement on the slightly-overcommitted case and 'fix' the pathological case.

2) call_rcu() and move on. This feels somewhat abusive of the API, as the guest can flood the host with RCU callbacks, but I wasn't able to make my machine fall over in any mean configuration of the test.

I haven't studied the degree to which such a malicious VM could adversely affect neighboring workloads.

3) Redo the whole ITS representation with xarrays and allow RCU readers outside of the ITS lock. I haven't fully thought this out, and if we pursue this option then we will need a secondary data structure to track where ITSes have been placed in guest memory to avoid taking the SRCU lock. We can then stick RCU synchronization in ITS command processing, which feels right to me, and dump the translation cache altogether.

Quoted 2 lines

I'd expect slightly worse average case performance in favor of more consistent performance.

Marc and I had an off-list conversation about this and agreed on option 4!

It is somewhat similar in spirit to (3), in that KVM will maintain an xarray translation cache per ITS, indexed by (device_id, event_id). This will be a perfect cache that can fit the entire range addressed by the ITS. The memory overheads of the xarray are not anticipated to be consequential, as the ITS memory footprint already scales linearly with the number of entries in the ITS.

Separately the DB -> ITS translation will be resolved by walking the ITSes present in the VM.

The existing invalidation calls will be scoped to an ITS besides the case where the guest disables LPIs on a redistributor.

Signature · 2 lines
Thanks,
Oliver
maintainer
#37 lore raw
In reply to Oliver Upton
Quoted 6 lines · Oliver Upton

For full details on the what/why, please see the cover letter in v1.

Apologies for the delay on v2, I wanted to spend some time to get a microbenchmark in place to slam the ITS code pretty hard, and based on the results I'm glad I did.

[...]

Buglets and potential improvements aside, I like the smell of this. At least the first handful of patches could easily be taken as a separate improvement series.

Let me know how you'd like to play this.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#38 lore raw
Quoted 9 lines · Marc Zyngier
Quoted 6 lines · Oliver Upton

For full details on the what/why, please see the cover letter in v1.

Apologies for the delay on v2, I wanted to spend some time to get a microbenchmark in place to slam the ITS code pretty hard, and based on the results I'm glad I did.

Quoted 7 lines

[...]

Buglets and potential improvements aside, I like the smell of this. At least the first handful of patches could easily be taken as a separate improvement series.

Let me know how you'd like to play this.

Yeah, I think there's 3 independent series here if we want to take the initial improvements:

- Address contention around vgic_get_irq() / vgic_put_irq() with the first 10 patches. Appears there is violent agreement these are good to go.

- Changing out the translation cache into a per-ITS xarray

- A final series cleaning up a lot of the warts we have in LPI management, like vgic_copy_lpi_list(). I believe we can get rid of the lpi_list_lock as well, but this needs to be ordered after the first 2.

I'd really like to de-risk the performance changes from the cleanups, as I'm convinced they're going to have their own respective piles of bugs.

How does that sound?

Signature · 2 lines
Thanks,
Oliver
maintainer
#39 lore raw
Quoted 33 lines · Oliver Upton
Quoted 9 lines · Marc Zyngier
Quoted 6 lines · Oliver Upton

For full details on the what/why, please see the cover letter in v1.

Apologies for the delay on v2, I wanted to spend some time to get a microbenchmark in place to slam the ITS code pretty hard, and based on the results I'm glad I did.

Quoted 7 lines

[...]

Buglets and potential improvements aside, I like the smell of this. At least the first handful of patches could easily be taken as a separate improvement series.

Let me know how you'd like to play this.

Yeah, I think there's 3 independent series here if we want to take the initial improvements:

- Address contention around vgic_get_irq() / vgic_put_irq() with the first 10 patches. Appears there is violent agreement these are good to go.

- Changing out the translation cache into a per-ITS xarray

- A final series cleaning up a lot of the warts we have in LPI management, like vgic_copy_lpi_list(). I believe we can get rid of the lpi_list_lock as well, but this needs to be ordered after the first 2.

Quoted 4 lines

I'd really like to de-risk the performance changes from the cleanups, as I'm convinced they're going to have their own respective piles of bugs.

How does that sound?

Yup, I'd be on board with that. If you can respin the first part with bugs fixed and without the stats, that'd be great. We can further bikeshed on the rest in the 6.10 time frame.

Also please Cc: Eric Auger, as he dealt with a lot of the ITS save/restore stuff.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#40 lore raw
Quoted 8 lines · Marc Zyngier
Quoted 4 lines

I'd really like to de-risk the performance changes from the cleanups, as I'm convinced they're going to have their own respective piles of bugs.

How does that sound?

Yup, I'd be on board with that. If you can respin the first part with bugs fixed and without the stats, that'd be great. We can further bikeshed on the rest in the 6.10 time frame.

Cool. That makes things much easier to manage. I'll respin the get / put improvements shortly, and hopefully this time I actually fix the stupid lock imbalance :)

Quoted 2 lines

Also please Cc: Eric Auger, as he dealt with a lot of the ITS save/restore stuff.

Always happy to designate another victim to review my crap.

Signature · 2 lines
Thanks,
Oliver