KVM: arm64: Improvements to LPI injection
v2 cover KVM: arm64: Improvements to LPI injection
For full details on the what/why, please see the cover letter in v1.
Apologies for the delay on v2, I wanted to spend some time to get a microbenchmark in place to slam the ITS code pretty hard, and based on the results I'm glad I did.
The test is built around having vCPU threads and device threads, where each device can signal a particular number of event IDs (alien, right?)
Anyway, here are the results of that test in some carefully-selected examples:
+----------------------------+---------------------+------------------------------+ | Config | v6.8-rc1 (LPIs/sec) | v6.8-rc1 + series (LPIs/sec) | +----------------------------+---------------------+------------------------------+ | -v 1 -d 1 -e 1 -i 1000000 | 780151.37 | 1255291.00 | | -v 16 -d 16 -e 16 -i 10000 | 437081.55 | 5078225.70 | | -v 16 -d 16 -e 17 -i 10000 | 506446.50 | 1126118.10 | | -v 64 -d 64 -e 1 -i 100000 | 295097.03 | 5565964.87 | | -v 1 -d 1 -e 17 -i 1000 | 732934.43 | 149.24 | +----------------------------+---------------------+------------------------------+
While there is an 18x improvement in the scaled-out config (64 vCPUs, 64 devices, 1 event per device), there is an extremely disappointing 4911x regression in the example that effectively forces a cache eviction for every lookup.
Clearly the RCU synchronization is a bounding issue in this case. I think other scenarios where the cache is overcommitted (16 vCPUs, 16 devices, 17 events / device) are able to hide effects somewhat, as other threads can make forward progress while others are stuck waiting on RCU.
A few ideas on next steps:
1) Rework the lpi_list_lock as an rwlock. This would obviate the need for RCU protection in the LPI cache as well as memory allocations on the injection path. This is actually what I had in the internal version of the series, although it was very incomplete.
I'd expect this to nullify the improvement on the slightly-overcommitted case and 'fix' the pathological case.
2) call_rcu() and move on. This feels somewhat abusive of the API, as the guest can flood the host with RCU callbacks, but I wasn't able to make my machine fall over in any mean configuration of the test.
I haven't studied the degree to which such a malicious VM could adversely affect neighboring workloads.
3) Redo the whole ITS representation with xarrays and allow RCU readers outside of the ITS lock. I haven't fully thought this out, and if we pursue this option then we will need a secondary data structure to track where ITSes have been placed in guest memory to avoid taking the SRCU lock. We can then stick RCU synchronization in ITS command processing, which feels right to me, and dump the translation cache altogether.
I'd expect slightly worse average case performance in favor of more consistent performance.
Even though it is more work, I'm slightly in favor of (3) as it is a net reduction in overall complexity of the ITS implementation. But, I wanted to send out what I had to guage opinions on these options, and get feedback on the first 10 patches which are an overall win.
v1: https://lore.kernel.org/kvmarm/[email protected]/
v1 -> v2:
- Add the microbenchmark
- Add tracepoints / VM stats for the important bits of LPI injection. This was extremely useful for making sense of test results.
- Fix a silly lock imbalance on error path in vgic_add_lpi() (Dan)
- Constrain xas_for_each() based on the properties of the INTID space (Marc)
- Remove some missed vestiges of the LPI linked-list (Marc)
- Explicitly free unused cache entry on failed insertion race (Marc)
- Don't explode people's machines with a boatload of xchg() (I said it was WIP!) (Marc)
Oliver Upton (23):
KVM: arm64: Add tracepoints + stats for LPI cache effectiveness
KVM: arm64: vgic: Store LPIs in an xarray
KVM: arm64: vgic: Use xarray to find LPI in vgic_get_lpi()
KVM: arm64: vgic-v3: Iterate the xarray to find pending LPIs
KVM: arm64: vgic-its: Walk the LPI xarray in vgic_copy_lpi_list()
KVM: arm64: vgic: Get rid of the LPI linked-list
KVM: arm64: vgic: Use atomics to count LPIs
KVM: arm64: vgic: Free LPI vgic_irq structs in an RCU-safe manner
KVM: arm64: vgic: Rely on RCU protection in vgic_get_lpi()
KVM: arm64: vgic: Ensure the irq refcount is nonzero when taking a ref
KVM: arm64: vgic: Don't acquire the lpi_list_lock in vgic_put_irq()
KVM: arm64: vgic-its: Lazily allocate LPI translation cache
KVM: arm64: vgic-its: Pick cache victim based on usage count
KVM: arm64: vgic-its: Protect cached vgic_irq pointers with RCU
KVM: arm64: vgic-its: Treat the LPI translation cache as an rculist
KVM: arm64: vgic-its: Rely on RCU to protect translation cache reads
KVM: selftests: Align with kernel's GIC definitions
KVM: selftests: Standardise layout of GIC frames
KVM: selftests: Add a minimal library for interacting with an ITS
KVM: selftests: Add helper for enabling LPIs on a redistributor
KVM: selftests: Use MPIDR_HWID_BITMASK from cputype.h
KVM: selftests: Hack in support for aligned page allocations
KVM: selftests: Add stress test for LPI injection
arch/arm64/include/asm/kvm_host.h | 3 + arch/arm64/kvm/guest.c | 5 +- arch/arm64/kvm/vgic/trace.h | 66 ++ arch/arm64/kvm/vgic/vgic-debug.c | 2 +- arch/arm64/kvm/vgic/vgic-init.c | 7 +- arch/arm64/kvm/vgic/vgic-its.c | 220 ++++--- arch/arm64/kvm/vgic/vgic-v3.c | 3 +- arch/arm64/kvm/vgic/vgic.c | 56 +- arch/arm64/kvm/vgic/vgic.h | 15 +- include/kvm/arm_vgic.h | 10 +- include/linux/kvm_host.h | 4 + tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/aarch64/arch_timer.c | 8 +- .../testing/selftests/kvm/aarch64/psci_test.c | 2 + .../testing/selftests/kvm/aarch64/vgic_irq.c | 15 +- .../selftests/kvm/aarch64/vgic_lpi_stress.c | 388 ++++++++++++ .../kvm/aarch64/vpmu_counter_access.c | 6 +- .../selftests/kvm/dirty_log_perf_test.c | 5 +- .../selftests/kvm/include/aarch64/gic.h | 15 +- .../selftests/kvm/include/aarch64/gic_v3.h | 586 +++++++++++++++++- .../selftests/kvm/include/aarch64/processor.h | 2 - .../selftests/kvm/include/aarch64/vgic.h | 27 +- .../selftests/kvm/include/kvm_util_base.h | 2 + tools/testing/selftests/kvm/lib/aarch64/gic.c | 18 +- .../selftests/kvm/lib/aarch64/gic_private.h | 4 +- .../selftests/kvm/lib/aarch64/gic_v3.c | 69 ++- .../testing/selftests/kvm/lib/aarch64/vgic.c | 337 +++++++++- tools/testing/selftests/kvm/lib/kvm_util.c | 27 +- 28 files changed, 1641 insertions(+), 262 deletions(-) create mode 100644 tools/testing/selftests/kvm/aarch64/vgic_lpi_stress.c
base-commit: 6613476e225e090cc9aad49be7fa504e290dd33d
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 1/23 KVM: arm64: Add tracepoints + stats for LPI cache effectiveness
LPI translation and injection has been shown to have a significant impact on the performance of VM workloads, so it probably makes sense to add some signals in this area.
Introduce the concept of a KVM tracepoint that associates with a VM stat and use it for the LPI translation cache tracepoints. It isn't too uncommon for a kernel hacker to attach to tracepoints, while at the same time userspace may open a 'binary stats' FD to peek at the corresponding VM stats.
--- arch/arm64/include/asm/kvm_host.h | 3 ++ arch/arm64/kvm/guest.c | 5 ++- arch/arm64/kvm/vgic/trace.h | 66 +++++++++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic-its.c | 14 ++++++- include/linux/kvm_host.h | 4 ++ 5 files changed, 89 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_host.h +3 -0
@@ -966,6 +966,9 @@ static inline bool __vcpu_write_sys_reg_to_cpu(u64 val, int reg)struct kvm_vm_stat {struct kvm_vm_stat_generic generic; +u64 vgic_its_trans_cache_hit; +u64 vgic_its_trans_cache_miss; +u64 vgic_its_trans_cache_victim; };struct kvm_vcpu_stat {
arch/arm64/kvm/guest.c +4 -1
@@ -30,7 +30,10 @@#include "trace.h" const struct _kvm_stats_desc kvm_vm_stats_desc[] = { -KVM_GENERIC_VM_STATS () +KVM_GENERIC_VM_STATS (), +STATS_DESC_COUNTER (VM , vgic_its_trans_cache_hit), +STATS_DESC_COUNTER (VM , vgic_its_trans_cache_miss), +STATS_DESC_COUNTER (VM , vgic_its_trans_cache_victim) };const struct kvm_stats_header kvm_vm_stats_header = {
arch/arm64/kvm/vgic/trace.h +66 -0
72 lines not on this page · raw message
arch/arm64/kvm/vgic/vgic-its.c +12 -2
@@ -20,6 +20,7 @@#include <asm/kvm_arm.h> #include <asm/kvm_mmu.h> +#include "trace.h" #include "vgic.h" #include "vgic-mmio.h"
@@ -636,8 +637,11 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,* to the interrupt, so drop the potential reference on what * was in the cache, and increment it on the new interrupt. */ -if (cte->irq) +if (cte->irq) { +KVM_VM_TRACE_EVENT (kvm, vgic_its_trans_cache_victim, cte->db, + cte->devid, cte->eventid, cte->irq->intid);__vgic_put_lpi_locked (kvm, cte->irq); + }vgic_get_irq_kref (irq);
@@ -767,8 +771,14 @@ int vgic_its_inject_cached_translation(struct kvm *kvm, struct kvm_msi *msi) db = (u64 )msi->address_hi <<32 | msi->address_lo; irq =vgic_its_check_cache (kvm, db, msi->devid, msi->data); -if (!irq) +if (!irq) { +KVM_VM_TRACE_EVENT (kvm, vgic_its_trans_cache_miss, db, msi->devid, + msi->data);return -EWOULDBLOCK ; + } + +KVM_VM_TRACE_EVENT (kvm, vgic_its_trans_cache_hit, db, msi->devid, + msi->data, irq->intid);raw_spin_lock_irqsave (&irq->irq_lock, flags); irq->pending_latch = true;
include/linux/kvm_host.h +4 -0
@@ -1927,6 +1927,10 @@ struct _kvm_stats_desc { HALT_POLL_HIST_COUNT), \ STATS_DESC_IBOOLEAN(VCPU_GENERIC, blocking) +#define KVM_VM_TRACE_EVENT (vm, event, ...) \ + ((vm)->stat.event)++; trace_## event(__VA_ARGS__) + +extern struct dentry *kvm_debugfs_dir; ssize_t kvm_stats_read(char *id, const struct kvm_stats_header *header,
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
Marc Zyngier
maintainerRe: [PATCH v2 01/23] KVM: arm64: Add tracepoints + stats for LPI cache effectiveness
Quoted 39 lines · Oliver Upton
LPI translation and injection has been shown to have a significant impact on the performance of VM workloads, so it probably makes sense to add some signals in this area.
Introduce the concept of a KVM tracepoint that associates with a VM stat and use it for the LPI translation cache tracepoints. It isn't too uncommon for a kernel hacker to attach to tracepoints, while at the same time userspace may open a 'binary stats' FD to peek at the corresponding VM stats.
Signed-off-by Oliver Upton <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 3 ++ arch/arm64/kvm/guest.c | 5 ++- arch/arm64/kvm/vgic/trace.h | 66 +++++++++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic-its.c | 14 ++++++- include/linux/kvm_host.h | 4 ++ 5 files changed, 89 insertions(+), 3 deletions(-)arch/arm64/include/asm/kvm_host.h +3
-0@@ -966,6 +966,9 @@ static inline bool __vcpu_write_sys_reg_to_cpu(u64 val, int reg)struct kvm_vm_stat {struct kvm_vm_stat_generic generic; +u64 vgic_its_trans_cache_hit; +u64 vgic_its_trans_cache_miss; +u64 vgic_its_trans_cache_victim; };struct kvm_vcpu_stat {arch/arm64/kvm/guest.c +0
-0
arch/arm64/kvm/guest.c line 37
Quoted 9 lines
arch/arm64/kvm/guest.c +4
-1@@ -30,7 +30,10 @@#include "trace.h" const struct _kvm_stats_desc kvm_vm_stats_desc[] = { -KVM_GENERIC_VM_STATS () +KVM_GENERIC_VM_STATS (), +STATS_DESC_COUNTER (VM , vgic_its_trans_cache_hit), +STATS_DESC_COUNTER (VM , vgic_its_trans_cache_miss), +STATS_DESC_COUNTER (VM , vgic_its_trans_cache_victim) };
We've talked about this offline, but I thought I'd make my position public on these.
I've very concerned that exposing these statistic, however useful they may be at a given point in time, will eventually become all wrong and/or misleading.
Case in point, our discussion about this very series, where we landed on a potential reimplementation of the translation cache as a per-ITS xarray. If this comes to fruition, these stats will probably be totally useless (hit monotonically increasing at the rate of 1 per interrupt, miss being 0 or 1, victim being stuck to 0).
I'm *not* saying that such stats are totally useless. Just that they are, by definition, tied to a kernel-side implementation choice, which will evolve. The trouble is that they create an ABI that we need to support forever.
I wish we could work on something that would allow such statistics to be *extracted* (as opposed to published). Something like this as a discussion subject for a future KVM Forum or LPC uConf would get my full backing (and I'm pretty sure the networking folks have solved that problem a long time ago).
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
PATCH v2 2/23 KVM: arm64: vgic: Store LPIs in an xarray
In reply to Oliver UptonUsing a linked-list for LPIs is less than ideal as it of course requires iterative searches to find a particular entry. An xarray is a better data structure for this use case, as it provides faster searches and can still handle a potentially sparse range of INTID allocations.
Start by storing LPIs in an xarray, punting usage of the xarray to a subsequent change.
--- arch/arm64/kvm/vgic/vgic-init.c | 3 +++ arch/arm64/kvm/vgic/vgic-its.c | 16 ++++++++++++++++ arch/arm64/kvm/vgic/vgic.c | 1 + include/kvm/arm_vgic.h | 2 ++ 4 files changed, 22 insertions(+)
arch/arm64/kvm/vgic/vgic-init.c +3 -0
@@ -56,6 +56,7 @@ void kvm_vgic_early_init(struct kvm *kvm)INIT_LIST_HEAD (&dist->lpi_list_head);INIT_LIST_HEAD (&dist->lpi_translation_cache);raw_spin_lock_init (&dist->lpi_list_lock); +xa_init_flags (&dist->lpi_xa,XA_FLAGS_LOCK_IRQ ); }/* CREATION */
@@ -366,6 +367,8 @@ static void kvm_vgic_dist_destroy(struct kvm *kvm)if (vgic_supports_direct_msis (kvm))vgic_v4_teardown (kvm); + +xa_destroy (&dist->lpi_xa); }static void __kvm_vgic_vcpu_destroy (struct kvm_vcpu *vcpu)
arch/arm64/kvm/vgic/vgic-its.c +16 -0
@@ -53,6 +53,12 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,if (!irq)return ERR_PTR (-ENOMEM ); + ret =xa_reserve_irq (&dist->lpi_xa, intid,GFP_KERNEL_ACCOUNT ); +if (ret) { +kfree (irq); +return ERR_PTR (ret); + } +INIT_LIST_HEAD (&irq->lpi_list);INIT_LIST_HEAD (&irq->ap_list);raw_spin_lock_init (&irq->irq_lock);
@@ -87,10 +93,20 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, goto out_unlock; } + ret =xa_err (xa_store (&dist->lpi_xa, intid, irq,0 )); +if (ret) { +xa_release (&dist->lpi_xa, intid); +kfree (irq); + goto out_unlock; + } +list_add_tail (&irq->lpi_list, &dist->lpi_list_head); dist->lpi_list_count++; out_unlock: +if (ret) +return ERR_PTR (ret); +raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags);/*
arch/arm64/kvm/vgic/vgic.c +1 -0
@@ -131,6 +131,7 @@ void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq)return ;list_del (&irq->lpi_list); +xa_erase (&dist->lpi_xa, irq->intid); dist->lpi_list_count--;kfree (irq);
include/kvm/arm_vgic.h +2 -0
@@ -13,6 +13,7 @@#include <linux/spinlock.h> #include <linux/static_key.h> #include <linux/types.h> +#include <linux/xarray.h> #include <kvm/iodev.h> #include <linux/list.h> #include <linux/jump_label.h>
@@ -275,6 +276,7 @@ struct vgic_dist {/* Protects the lpi_list and the count value below. */ raw_spinlock_t lpi_list_lock; +struct xarray lpi_xa;struct list_head lpi_list_head;int lpi_list_count;
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
Re: [PATCH v2 02/23] KVM: arm64: vgic: Store LPIs in an xarray
On Tue, Feb 13, 2024 at 09:32:39AM +0000, Oliver Upton wrote:
[...]
Quoted 9 lines
(quoted diff) +5
-0@@ -87,10 +93,20 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, goto out_unlock; } + ret = xa_err(xa_store(&dist->lpi_xa, intid, irq, 0)); + if (ret) { + xa_release(&dist->lpi_xa, intid); + kfree(irq); + goto out_unlock;
Review static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, · line 110
Quoted 10 lines
(quoted diff) +5
-0+ } + list_add_tail(&irq->lpi_list, &dist->lpi_list_head); dist->lpi_list_count++; out_unlock: + if (ret) + return ERR_PTR(ret); + raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
Hah, I need to get my head checked. This is *still* wrong!
---
arch/arm64/kvm/vgic/vgic-its.c +2 -2
@@ -104,11 +104,11 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, dist->lpi_list_count++; out_unlock: +raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); +if (ret)return ERR_PTR (ret); -raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); -/* * We "cache" the configuration table entries in our struct vgic_irq's. * However we only have those structs for mapped IRQs, so we read in
Signature · 2 lines
Thanks, Oliver
PATCH v2 3/23 KVM: arm64: vgic: Use xarray to find LPI in vgic_get_lpi()
In reply to Oliver UptonIterating over the LPI linked-list is less than ideal when the desired index is already known. Use the INTID to index the LPI xarray instead.
--- arch/arm64/kvm/vgic/vgic.c | 19 +++++-------------- 1 file changed, 5 insertions(+), 14 deletions(-)
arch/arm64/kvm/vgic/vgic.c +5 -14
@@ -54,8 +54,9 @@ struct vgic_global kvm_vgic_global_state __ro_after_init = {*/ /* -* Iterate over the VM's list of mapped LPIs to find the one with a -* matching interrupt ID and return a reference to the IRQ structure. +* Index the VM's xarray of mapped LPIs and return a reference to the IRQ +* structure. The caller is expected to call vgic_put_irq() later once it's +* finished with the IRQ. */ static struct vgic_irq *vgic_get_lpi (struct kvm *kvm,u32 intid) {
@@ -65,20 +66,10 @@ static struct vgic_irq *vgic_get_lpi(struct kvm *kvm, u32 intid)raw_spin_lock_irqsave (&dist->lpi_list_lock, flags); -list_for_each_entry (irq, &dist->lpi_list_head, lpi_list) { -if (irq->intid != intid) -continue ; - -/* -* This increases the refcount, the caller is expected to -* call vgic_put_irq() later once it's finished with the IRQ. -*/ + irq =xa_load (&dist->lpi_xa, intid); +if (irq)vgic_get_irq_kref (irq); - goto out_unlock; - } - irq =NULL ; -out_unlock:raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags);return irq;
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 4/23 KVM: arm64: vgic-v3: Iterate the xarray to find pending LPIs
In reply to Oliver UptonStart walking the LPI xarray to find pending LPIs in preparation for the removal of the LPI linked-list. Note that the 'basic' iterator is chosen here as each iteration needs to drop the xarray read lock (RCU) as reads/writes to guest memory can potentially block.
--- arch/arm64/kvm/vgic/vgic-v3.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/arm64/kvm/vgic/vgic-v3.c +2 -1
@@ -380,6 +380,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)struct vgic_irq *irq;gpa_t last_ptr = ~(gpa_t )0 ;bool vlpi_avail = false; +unsigned long index;int ret =0 ;u8 val;
@@ -396,7 +397,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm) vlpi_avail = true; } -list_for_each_entry (irq, &dist->lpi_list_head, lpi_list) { +xa_for_each (&dist->lpi_xa, index, irq) {int byte_offset, bit_nr;struct kvm_vcpu *vcpu;gpa_t pendbase, ptr;
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 5/23 KVM: arm64: vgic-its: Walk the LPI xarray in vgic_copy_lpi_list()
In reply to Oliver UptonStart iterating the LPI xarray in anticipation of removing the LPI linked-list.
--- arch/arm64/kvm/vgic/vgic-its.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
arch/arm64/kvm/vgic/vgic-its.c +6 -1
@@ -336,6 +336,7 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,int vgic_copy_lpi_list (struct kvm *kvm,struct kvm_vcpu *vcpu,u32 **intid_ptr) {struct vgic_dist *dist = &kvm->arch.vgic; +XA_STATE (xas, &dist->lpi_xa,GIC_LPI_OFFSET );struct vgic_irq *irq;unsigned long flags;u32 *intids;
@@ -354,7 +355,9 @@ int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr)return -ENOMEM ;raw_spin_lock_irqsave (&dist->lpi_list_lock, flags); -list_for_each_entry (irq, &dist->lpi_list_head, lpi_list) { +rcu_read_lock (); + +xas_for_each (&xas, irq,INTERRUPT_ID_BITS_ITS ) {if (i == irq_count)break ;/* We don't need to "get" the IRQ, as we hold the list lock. */
@@ -362,6 +365,8 @@ int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr)continue ; intids[i++] = irq->intid; } + +rcu_read_unlock ();raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); *intid_ptr = intids;
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 6/23 KVM: arm64: vgic: Get rid of the LPI linked-list
In reply to Oliver UptonAll readers of LPI configuration have been transitioned to use the LPI xarray. Get rid of the linked-list altogether.
--- arch/arm64/kvm/vgic/vgic-init.c | 1 - arch/arm64/kvm/vgic/vgic-its.c | 8 ++------ arch/arm64/kvm/vgic/vgic.c | 1 - include/kvm/arm_vgic.h | 2 -- 4 files changed, 2 insertions(+), 10 deletions(-)
arch/arm64/kvm/vgic/vgic-init.c +0 -1
@@ -53,7 +53,6 @@ void kvm_vgic_early_init(struct kvm *kvm) {struct vgic_dist *dist = &kvm->arch.vgic; -INIT_LIST_HEAD (&dist->lpi_list_head);INIT_LIST_HEAD (&dist->lpi_translation_cache);raw_spin_lock_init (&dist->lpi_list_lock);xa_init_flags (&dist->lpi_xa,XA_FLAGS_LOCK_IRQ );
arch/arm64/kvm/vgic/vgic-its.c +2 -6
@@ -59,7 +59,6 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,return ERR_PTR (ret); } -INIT_LIST_HEAD (&irq->lpi_list);INIT_LIST_HEAD (&irq->ap_list);raw_spin_lock_init (&irq->irq_lock);
@@ -75,10 +74,8 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,* There could be a race with another vgic_add_lpi(), so we need to * check that we don't add a second list entry with the same LPI. */ -list_for_each_entry (oldirq, &dist->lpi_list_head, lpi_list) { -if (oldirq->intid != intid) -continue ; - + oldirq =xa_load (&dist->lpi_xa, intid); +if (oldirq) {/* Someone was faster with adding this LPI, lets use that. */ kfree (irq); irq = oldirq;
@@ -100,7 +97,6 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, goto out_unlock; } -list_add_tail (&irq->lpi_list, &dist->lpi_list_head); dist->lpi_list_count++; out_unlock:
arch/arm64/kvm/vgic/vgic.c +0 -1
@@ -121,7 +121,6 @@ void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq)if (!kref_put (&irq->refcount, vgic_irq_release))return ; -list_del (&irq->lpi_list);xa_erase (&dist->lpi_xa, irq->intid); dist->lpi_list_count--;
include/kvm/arm_vgic.h +0 -2
@@ -117,7 +117,6 @@ struct irq_ops {struct vgic_irq {raw_spinlock_t irq_lock;/* Protects the content of the struct */ -struct list_head lpi_list;/* Used to link all LPIs together */ struct list_head ap_list;struct kvm_vcpu *vcpu;/* SGIs and PPIs: The VCPU
@@ -277,7 +276,6 @@ struct vgic_dist {/* Protects the lpi_list and the count value below. */ raw_spinlock_t lpi_list_lock;struct xarray lpi_xa; -struct list_head lpi_list_head;int lpi_list_count;/* LPI translation cache */
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 7/23 KVM: arm64: vgic: Use atomics to count LPIs
In reply to Oliver UptonSwitch to using atomics for LPI accounting, allowing vgic_irq references to be dropped in parallel.
--- arch/arm64/kvm/vgic/vgic-debug.c | 2 +- arch/arm64/kvm/vgic/vgic-its.c | 4 ++-- arch/arm64/kvm/vgic/vgic.c | 2 +- include/kvm/arm_vgic.h | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-)
arch/arm64/kvm/vgic/vgic-debug.c +1 -1
@@ -149,7 +149,7 @@ static void print_dist_state(struct seq_file *s, struct vgic_dist *dist)seq_printf (s,"vgic_model:\t%s\n" , v3 ?"GICv3" :"GICv2" );seq_printf (s,"nr_spis:\t%d\n" , dist->nr_spis);if (v3) -seq_printf (s,"nr_lpis:\t%d\n" , dist->lpi_list_count); +seq_printf (s,"nr_lpis:\t%d\n" ,atomic_read (&dist->lpi_count));seq_printf (s,"enabled:\t%d\n" , dist->enabled);seq_printf (s,"\n" );
arch/arm64/kvm/vgic/vgic-its.c +2 -2
@@ -97,7 +97,7 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, goto out_unlock; } - dist->lpi_list_count++; +atomic_inc (&dist->lpi_count); out_unlock:if (ret)
@@ -345,7 +345,7 @@ int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr)* command). If coming from another path (such as enabling LPIs), * we must be careful not to overrun the array. */ - irq_count =READ_ONCE (dist->lpi_list_count); + irq_count =atomic_read (&dist->lpi_count); intids =kmalloc_array (irq_count,sizeof (intids[0 ]),GFP_KERNEL_ACCOUNT );if (!intids)return -ENOMEM ;
arch/arm64/kvm/vgic/vgic.c +1 -1
@@ -122,7 +122,7 @@ void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq)return ;xa_erase (&dist->lpi_xa, irq->intid); - dist->lpi_list_count--; +atomic_dec (&dist->lpi_count);kfree (irq); }
include/kvm/arm_vgic.h +2 -2
@@ -273,10 +273,10 @@ struct vgic_dist {*/ u64 propbaser; -/* Protects the lpi_list and the count value below. */ +/* Protects the lpi_list. */ raw_spinlock_t lpi_list_lock;struct xarray lpi_xa; -int lpi_list_count; +atomic_t lpi_count;/* LPI translation cache */ struct list_head lpi_translation_cache;
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
Marc Zyngier
maintainerRe: [PATCH v2 07/23] KVM: arm64: vgic: Use atomics to count LPIs
Quoted 39 lines · Oliver Upton
Switch to using atomics for LPI accounting, allowing
vgic_irqreferences to be dropped in parallel.Signed-off-by Oliver Upton <[email protected]>--- arch/arm64/kvm/vgic/vgic-debug.c | 2 +- arch/arm64/kvm/vgic/vgic-its.c | 4 ++-- arch/arm64/kvm/vgic/vgic.c | 2 +- include/kvm/arm_vgic.h | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-)arch/arm64/kvm/vgic/vgic-debug.c +1
-1@@ -149,7 +149,7 @@ static void print_dist_state(struct seq_file *s, struct vgic_dist *dist)seq_printf (s,"vgic_model:\t%s\n" , v3 ?"GICv3" :"GICv2" );seq_printf (s,"nr_spis:\t%d\n" , dist->nr_spis);if (v3) -seq_printf (s,"nr_lpis:\t%d\n" , dist->lpi_list_count); +seq_printf (s,"nr_lpis:\t%d\n" ,atomic_read (&dist->lpi_count));seq_printf (s,"enabled:\t%d\n" , dist->enabled);seq_printf (s,"\n" );arch/arm64/kvm/vgic/vgic-its.c +1
-1@@ -97,7 +97,7 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, goto out_unlock; } - dist->lpi_list_count++; +atomic_inc (&dist->lpi_count); out_unlock:if (ret)
arch/arm64/kvm/vgic/vgic-its.c int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr) · line 348
Quoted 5 lines
arch/arm64/kvm/vgic/vgic-its.c +1
-1@@ -345,7 +345,7 @@ int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr)* command). If coming from another path (such as enabling LPIs), * we must be careful not to overrun the array. */ - irq_count =READ_ONCE (dist->lpi_list_count); + irq_count =atomic_read (&dist->lpi_count);
I'd like to propose an alternative approach here. I've always hated this "copy a bunch of INTIDs" thing, and the only purpose of this silly counter is to dimension the resulting array.
Could we instead rely on an xarray marking a bunch of entries (the ones we want to 'copy'), and get the reader to clear these marks once done?
Of course, we only have 3 marks, so that's a bit restrictive from a concurrency perspective, but since most callers hold a lock, it should be OK.
What do you think?
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Re: [PATCH v2 07/23] KVM: arm64: vgic: Use atomics to count LPIs
Hey,
Quoted 2 lines · Marc Zyngier
I'd like to propose an alternative approach here. I've always hated this "copy a bunch of INTIDs" thing,
Agree.
Quoted 2 lines
and the only purpose of this
silly counter is to dimension the resulting array.
Well, we also use it to trivially print the number of LPIs for a particular vgic in the debug interface.
Quoted 3 lines
Could we instead rely on an xarray marking a bunch of entries (the ones we want to 'copy'), and get the reader to clear these marks once done?
I think that'd work. I'm trying to convince myself we don't have bugs lurking in some of the existing usage of vgic_copy_lpi_list()...
Quoted 3 lines
Of course, we only have 3 marks, so that's a bit restrictive from a concurrency perspective, but since most callers hold a lock, it should be OK.
They all hold *a* lock, but maybe not the same one! :)
Maybe we should serialize the use of markers on the LPI list on the config_lock. A slight misuse, but we need a mutex since we're poking at guest memory. Then we can go through the whole N-dimensional locking puzzle and convince ourselves it is still correct.
Signature · 2 lines
Thanks, Oliver
Marc Zyngier
maintainerRe: [PATCH v2 07/23] KVM: arm64: vgic: Use atomics to count LPIs
Quoted 7 lines · Oliver Upton
Hey,
Quoted 2 lines · Marc Zyngier
I'd like to propose an alternative approach here. I've always hated this "copy a bunch of INTIDs" thing,
Quoted 7 lines
Agree.
Quoted 2 lines
and the only purpose of this
silly counter is to dimension the resulting array.Well, we also use it to trivially print the number of LPIs for a particular vgic in the debug interface.
I think we can get survive this... ;-)
Quoted 5 lines
Quoted 4 lines
Could we instead rely on an xarray marking a bunch of entries (the ones we want to 'copy'), and get the reader to clear these marks once done?
Quoted 8 lines
I think that'd work. I'm trying to convince myself we don't have bugs lurking in some of the existing usage of
vgic_copy_lpi_list()...Quoted 3 lines
Of course, we only have 3 marks, so that's a bit restrictive from a concurrency perspective, but since most callers hold a lock, it should be OK.
They all hold *a* lock, but maybe not the same one! :)
Indeed. But as long as there isn't more than 3 locks (and that the xarray is OK being concurrently updated with marks), we're good!
Quoted 4 lines
Maybe we should serialize the use of markers on the LPI list on the
config_lock. A slight misuse, but we need a mutex since we're poking at guest memory. Then we can go through the whole N-dimensional locking puzzle and convince ourselves it is still correct.
Maybe. This thing is already seeing so many abuses that one more may not matter much. Need to see how it fits in the whole hierarchy of GIC-related locks...
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Re: [PATCH v2 07/23] KVM: arm64: vgic: Use atomics to count LPIs
Quoted 8 lines · Marc Zyngier
Quoted 5 lines
Quoted 3 lines
Of course, we only have 3 marks, so that's a bit restrictive from a concurrency perspective, but since most callers hold a lock, it should be OK.
They all hold *a* lock, but maybe not the same one! :)
Indeed. But as long as there isn't more than 3 locks (and that the xarray is OK being concurrently updated with marks), we're good!
Oh, you mean to give each existing caller their own mark?
Quoted 8 lines
Quoted 4 lines
Maybe we should serialize the use of markers on the LPI list on the
config_lock. A slight misuse, but we need a mutex since we're poking at guest memory. Then we can go through the whole N-dimensional locking puzzle and convince ourselves it is still correct.Maybe. This thing is already seeing so many abuses that one more may not matter much. Need to see how it fits in the whole hierarchy of GIC-related locks...
It doesn't work. We have it that the config_lock needs to be taken outside the its_lock.
Too many damn locks!
Signature · 2 lines
Thanks, Oliver
Marc Zyngier
maintainerRe: [PATCH v2 07/23] KVM: arm64: vgic: Use atomics to count LPIs
Quoted 6 lines · Oliver Upton
Quoted 4 lines · Marc Zyngier
Quoted 4 lines
Quoted 3 lines
Of course, we only have 3 marks, so that's a bit restrictive from a concurrency perspective, but since most callers hold a lock, it should be OK.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
They all hold *a* lock, but maybe not the same one! :)
Indeed. But as long as there isn't more than 3 locks (and that the xarray is OK being concurrently updated with marks), we're good!
Oh, you mean to give each existing caller their own mark?
Well, each caller "class". Where "class" means "holding look 'foo'". Same lock, same mark. With a maximum of 3 (and I think we can get away with 2).
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Maybe we should serialize the use of markers on the LPI list on the
config_lock. A slight misuse, but we need a mutex since we're poking at guest memory. Then we can go through the whole N-dimensional locking puzzle and convince ourselves it is still correct.
Quoted 8 lines
Quoted 3 lines
Maybe. This thing is already seeing so many abuses that one more may not matter much. Need to see how it fits in the whole hierarchy of GIC-related locks...
It doesn't work. We have it that the
config_lockneeds to be taken outside theits_lock.Too many damn locks!
Well, the joys of emulating highly complex HW with a braindead programming interface. I'd explore the above suggestion to avoid introducing a new lock, if at all possible.
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
PATCH v2 8/23 KVM: arm64: vgic: Free LPI vgic_irq structs in an RCU-safe manner
In reply to Oliver UptonFree the vgic_irq structs in an RCU-safe manner to allow reads of the LPI configuration data to happen in parallel with the release of LPIs.
--- arch/arm64/kvm/vgic/vgic.c | 2 +- include/kvm/arm_vgic.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
arch/arm64/kvm/vgic/vgic.c +1 -1
@@ -124,7 +124,7 @@ void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq)xa_erase (&dist->lpi_xa, irq->intid);atomic_dec (&dist->lpi_count); -kfree (irq); +kfree_rcu (irq, rcu); }void vgic_put_irq (struct kvm *kvm,struct vgic_irq *irq)
include/kvm/arm_vgic.h +1 -0
@@ -117,6 +117,7 @@ struct irq_ops {struct vgic_irq {raw_spinlock_t irq_lock;/* Protects the content of the struct */ +struct rcu_head rcu;struct list_head ap_list;struct kvm_vcpu *vcpu;/* SGIs and PPIs: The VCPU
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 9/23 KVM: arm64: vgic: Rely on RCU protection in vgic_get_lpi()
In reply to Oliver UptonStop acquiring the lpi_list_lock in favor of RCU for protecting the read-side critical section in vgic_get_lpi(). In order for this to be safe, we also need to be careful not to take a reference on an irq with a refcount of 0, as it is about to be freed.
--- arch/arm64/kvm/vgic/vgic.c | 9 ++++----- arch/arm64/kvm/vgic/vgic.h | 14 +++++++++++--- 2 files changed, 15 insertions(+), 8 deletions(-)
arch/arm64/kvm/vgic/vgic.c +4 -5
@@ -62,15 +62,14 @@ static struct vgic_irq *vgic_get_lpi(struct kvm *kvm, u32 intid) {struct vgic_dist *dist = &kvm->arch.vgic;struct vgic_irq *irq =NULL ; -unsigned long flags; -raw_spin_lock_irqsave (&dist->lpi_list_lock, flags); +rcu_read_lock (); irq =xa_load (&dist->lpi_xa, intid); -if (irq) -vgic_get_irq_kref (irq); +if (!vgic_try_get_irq_kref (irq)) + irq =NULL ; -raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); +rcu_read_unlock ();return irq; }
arch/arm64/kvm/vgic/vgic.h +11 -3
@@ -220,12 +220,20 @@ void vgic_v2_vmcr_sync(struct kvm_vcpu *vcpu);void vgic_v2_save_state (struct kvm_vcpu *vcpu);void vgic_v2_restore_state (struct kvm_vcpu *vcpu); -static inline void vgic_get_irq_kref (struct vgic_irq *irq) +static inline bool vgic_try_get_irq_kref (struct vgic_irq *irq) { +if (!irq) +return false; +if (irq->intid <VGIC_MIN_LPI ) -return ; +return true; -kref_get (&irq->refcount); +return kref_get_unless_zero (&irq->refcount); +} + +static inline void vgic_get_irq_kref (struct vgic_irq *irq) +{ +WARN_ON_ONCE (!vgic_try_get_irq_kref (irq)); }void vgic_v3_fold_lr_state (struct kvm_vcpu *vcpu);
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 10/23 KVM: arm64: vgic: Ensure the irq refcount is nonzero when taking a ref
In reply to Oliver UptonIt will soon be possible for get() and put() calls to happen in parallel, which means in most cases we must ensure the refcount is nonzero when taking a new reference. Switch to using vgic_try_get_irq_kref() where necessary, and document the few conditions where an IRQ's refcount is guaranteed to be nonzero.
--- arch/arm64/kvm/vgic/vgic-its.c | 18 ++++++++---------- arch/arm64/kvm/vgic/vgic.c | 3 ++- 2 files changed, 10 insertions(+), 11 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +8 -10
@@ -75,18 +75,11 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,* check that we don't add a second list entry with the same LPI. */ oldirq =xa_load (&dist->lpi_xa, intid); -if (oldirq) { +if (vgic_try_get_irq_kref (oldirq)) {/* Someone was faster with adding this LPI, lets use that. */ kfree (irq); irq = oldirq; -/* -* This increases the refcount, the caller is expected to -* call vgic_put_irq() on the returned pointer once it's -* finished with the IRQ. -*/ -vgic_get_irq_kref (irq); - goto out_unlock; }
@@ -610,8 +603,8 @@ static struct vgic_irq *vgic_its_check_cache(struct kvm *kvm, phys_addr_t db,raw_spin_lock_irqsave (&dist->lpi_list_lock, flags); irq =__vgic_its_check_cache (dist, db, devid, eventid); -if (irq) -vgic_get_irq_kref (irq); +if (!vgic_try_get_irq_kref (irq)) + irq =NULL ;raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags);
@@ -660,6 +653,11 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,__vgic_put_lpi_locked (kvm, cte->irq); } +/* +* The irq refcount is guaranteed to be nonzero while holding the +* its_lock, as the ITE (and the reference it holds) cannot be freed. +*/ +lockdep_assert_held (&its->its_lock);vgic_get_irq_kref (irq); cte->db = db;
arch/arm64/kvm/vgic/vgic.c +2 -1
@@ -394,7 +394,8 @@ bool vgic_queue_irq_unlock(struct kvm *kvm, struct vgic_irq *irq,/* * Grab a reference to the irq to reflect the fact that it is -* now in the ap_list. +* now in the ap_list. This is safe as the caller must already hold a +* reference on the irq. */ vgic_get_irq_kref (irq);list_add_tail (&irq->ap_list, &vcpu->arch.vgic_cpu.ap_list_head);
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 11/23 KVM: arm64: vgic: Don't acquire the lpi_list_lock in vgic_put_irq()
In reply to Oliver UptonThe LPI xarray's xa_lock is sufficient for synchronizing writers when freeing a given LPI. Furthermore, readers can only take a new reference on an IRQ if it was already nonzero.
Stop taking the lpi_list_lock unnecessarily and get rid of __vgic_put_lpi_locked().
--- arch/arm64/kvm/vgic/vgic-its.c | 4 ++-- arch/arm64/kvm/vgic/vgic.c | 21 ++++----------------- arch/arm64/kvm/vgic/vgic.h | 1 - 3 files changed, 6 insertions(+), 20 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +2 -2
@@ -650,7 +650,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,if (cte->irq) {KVM_VM_TRACE_EVENT (kvm, vgic_its_trans_cache_victim, cte->db, cte->devid, cte->eventid, cte->irq->intid); -__vgic_put_lpi_locked (kvm, cte->irq); +vgic_put_irq (kvm, cte->irq); }/*
@@ -688,7 +688,7 @@ void vgic_its_invalidate_cache(struct kvm *kvm)if (!cte->irq)break ; -__vgic_put_lpi_locked (kvm, cte->irq); +vgic_put_irq (kvm, cte->irq); cte->irq =NULL ; }
arch/arm64/kvm/vgic/vgic.c +4 -17
@@ -110,13 +110,13 @@ static void vgic_irq_release(struct kref *ref) { } -/* -* Drop the refcount on the LPI. Must be called with lpi_list_lock held. -*/ -void __vgic_put_lpi_locked (struct kvm *kvm,struct vgic_irq *irq) +void vgic_put_irq (struct kvm *kvm,struct vgic_irq *irq) {struct vgic_dist *dist = &kvm->arch.vgic; +if (irq->intid <VGIC_MIN_LPI ) +return ; +if (!kref_put (&irq->refcount, vgic_irq_release))return ;
@@ -126,19 +126,6 @@ void __vgic_put_lpi_locked(struct kvm *kvm, struct vgic_irq *irq)kfree_rcu (irq, rcu); } -void vgic_put_irq (struct kvm *kvm,struct vgic_irq *irq) -{ -struct vgic_dist *dist = &kvm->arch.vgic; -unsigned long flags; - -if (irq->intid <VGIC_MIN_LPI ) -return ; - -raw_spin_lock_irqsave (&dist->lpi_list_lock, flags); -__vgic_put_lpi_locked (kvm, irq); -raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); -} -void vgic_flush_pending_lpis (struct kvm_vcpu *vcpu) {struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
arch/arm64/kvm/vgic/vgic.h +0 -1
@@ -180,7 +180,6 @@ vgic_get_mmio_region(struct kvm_vcpu *vcpu, struct vgic_io_device *iodev, gpa_t addr, int len);struct vgic_irq *vgic_get_irq (struct kvm *kvm,struct kvm_vcpu *vcpu,u32 intid); -void __vgic_put_lpi_locked (struct kvm *kvm,struct vgic_irq *irq);void vgic_put_irq (struct kvm *kvm,struct vgic_irq *irq);bool vgic_get_phys_line_level (struct vgic_irq *irq);void vgic_irq_set_phys_pending (struct vgic_irq *irq,bool pending);
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 12/23 KVM: arm64: vgic-its: Lazily allocate LPI translation cache
In reply to Oliver UptonReusing translation cache entries within a read-side critical section is fundamentally incompatible with an rculist. As such, we need to allocate a new entry to replace an eviction and free the removed entry afterwards.
Take this as an opportunity to remove the eager allocation of translation cache entries altogether in favor of a lazy allocation model on cache miss.
--- arch/arm64/kvm/vgic/vgic-init.c | 3 -- arch/arm64/kvm/vgic/vgic-its.c | 96 +++++++++++++++------------------ include/kvm/arm_vgic.h | 1 + 3 files changed, 45 insertions(+), 55 deletions(-)
arch/arm64/kvm/vgic/vgic-init.c +0 -3
@@ -305,9 +305,6 @@ int vgic_init(struct kvm *kvm) } } -if (vgic_has_its (kvm)) -vgic_lpi_translation_cache_init (kvm); -/* * If we have GICv4.1 enabled, unconditionnaly request enable the * v4 support so that we get HW-accelerated vSGIs. Otherwise, only
arch/arm64/kvm/vgic/vgic-its.c +44 -52
145 lines not on this page · raw message
include/kvm/arm_vgic.h +1 -0
@@ -281,6 +281,7 @@ struct vgic_dist {/* LPI translation cache */ struct list_head lpi_translation_cache; +unsigned int lpi_cache_count;/* used by vgic-debug */ struct vgic_state_iter *iter;
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 13/23 KVM: arm64: vgic-its: Pick cache victim based on usage count
In reply to Oliver UptonTo date the translation cache LRU policy relies on the ordering of the linked-list to pick the victim, as entries are moved to the head of the list on every cache hit. These sort of transformations are incompatible with an rculist, necessitating a different strategy for recording usage in-place.
Count the number of cache hits since the last translation cache miss for every entry. The preferences for selecting a victim are as follows:
- Invalid entries over valid entries
- Valid entry with the lowest usage count
- In the case of a tie, pick the entry closest to the tail (oldest)
--- arch/arm64/kvm/vgic/vgic-its.c | 46 ++++++++++++++++++++++++++-------- 1 file changed, 36 insertions(+), 10 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +36 -10
@@ -157,6 +157,7 @@ struct vgic_translation_cache_entry {u32 devid;u32 eventid;struct vgic_irq *irq; +atomic64_t usage_count; };/**
@@ -580,13 +581,7 @@ static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist, cte->eventid != eventid)continue ; -/* -* Move this entry to the head, as it is the most -* recently used. -*/ -if (!list_is_first (&cte->entry, &dist->lpi_translation_cache)) -list_move (&cte->entry, &dist->lpi_translation_cache); - +atomic64_inc (&cte->usage_count);return cte->irq; }
@@ -619,6 +614,36 @@ static unsigned int vgic_its_max_cache_size(struct kvm *kvm)return atomic_read (&kvm->online_vcpus) *LPI_DEFAULT_PCPU_CACHE_SIZE ; } +static struct vgic_translation_cache_entry *vgic_its_cache_victim (struct vgic_dist *dist, +s64 *max_usage) +{ +struct vgic_translation_cache_entry *cte, *victim =NULL ; +s64 min, tmp, max =S64_MIN ; + +/* +* Find the least used cache entry since the last cache miss, preferring +* older entries in the case of a tie. Return the max usage count seen +* during the scan to initialize the new cache entry. +*/ +list_for_each_entry (cte, &dist->lpi_translation_cache, entry) { + tmp =atomic64_read (&cte->usage_count); + max =max (max, tmp); + +if (!cte->irq) { + victim = cte; +break ; + } + +if (!victim || tmp <= min) { + victim = cte; + min = tmp; + } + } + + *max_usage = max; +return victim; +} +static void vgic_its_cache_translation (struct kvm *kvm,struct vgic_its *its,u32 devid,u32 eventid,struct vgic_irq *irq)
@@ -627,6 +652,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,struct vgic_dist *dist = &kvm->arch.vgic;unsigned long flags;phys_addr_t db; +s64 usage =0 ;/* Do not cache a directly injected interrupt */ if (irq->hw)
@@ -650,9 +676,8 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, }if (dist->lpi_cache_count >=vgic_its_max_cache_size (kvm)) { -/* Always reuse the last entry (LRU policy) */ - victim =list_last_entry (&dist->lpi_translation_cache, -typeof (*cte), entry); + victim =vgic_its_cache_victim (dist, &usage); +list_del (&victim->entry); dist->lpi_cache_count--; }
@@ -664,6 +689,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,lockdep_assert_held (&its->its_lock);vgic_get_irq_kref (irq); +atomic64_set (&new->usage_count, usage); new->db = db; new->devid = devid; new->eventid = eventid;
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 14/23 KVM: arm64: vgic-its: Protect cached vgic_irq pointers with RCU
In reply to Oliver UptonRCU readers of the LPI translation cache will be able to run in parallel with a cache invalidation, which clears the RCU pointer. Start using RCU protection on the cached irq pointer in light of this.
--- arch/arm64/kvm/vgic/vgic-its.c | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +18 -10
@@ -156,7 +156,7 @@ struct vgic_translation_cache_entry {phys_addr_t db;u32 devid;u32 eventid; -struct vgic_irq *irq; +struct vgic_irq __rcu *irq;atomic64_t usage_count; };
@@ -574,7 +574,7 @@ static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist,* If we hit a NULL entry, there is nothing after this * point. */ -if (!cte->irq) +if (!rcu_access_pointer (cte->irq))break ; if (cte->db != db || cte->devid != devid ||
@@ -582,7 +582,7 @@ static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist,continue ;atomic64_inc (&cte->usage_count); -return cte->irq; +return rcu_dereference (cte->irq); }return NULL ;
@@ -629,7 +629,7 @@ static struct vgic_translation_cache_entry *vgic_its_cache_victim(struct vgic_di tmp =atomic64_read (&cte->usage_count); max =max (max, tmp); -if (!cte->irq) { +if (!rcu_access_pointer (cte->irq)) { victim = cte;break ; }
@@ -663,6 +663,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,return ;raw_spin_lock_irqsave (&dist->lpi_list_lock, flags); +rcu_read_lock ();/* * We could have raced with another CPU caching the same
@@ -693,27 +694,32 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, new->db = db; new->devid = devid; new->eventid = eventid; - new->irq = irq; +rcu_assign_pointer (new->irq, irq);/* Move the new translation to the head of the list */ list_add (&new->entry, &dist->lpi_translation_cache); dist->lpi_cache_count++; out: +rcu_read_unlock ();raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags);if (!victim)return ; +synchronize_rcu (); +/* * Caching the translation implies having an extra reference * to the interrupt, so drop the potential reference on what -* was in the cache, and increment it on the new interrupt. +* was in the cache. */ if (victim->irq) { +struct vgic_irq *irq =rcu_dereference_raw (victim->irq); +KVM_VM_TRACE_EVENT (kvm, vgic_its_trans_cache_victim, victim->db, - victim->devid, victim->eventid, victim->irq->intid); -vgic_put_irq (kvm, victim->irq); + victim->devid, victim->eventid, irq->intid); +vgic_put_irq (kvm, irq); }kfree (victim);
@@ -726,19 +732,21 @@ void vgic_its_invalidate_cache(struct kvm *kvm)unsigned long flags;raw_spin_lock_irqsave (&dist->lpi_list_lock, flags); +rcu_read_lock ();list_for_each_entry (cte, &dist->lpi_translation_cache, entry) {/* * If we hit a NULL entry, there is nothing after this * point. */ -if (!cte->irq) +if (!rcu_access_pointer (cte->irq))break ;vgic_put_irq (kvm, cte->irq); - cte->irq =NULL ; +rcu_assign_pointer (cte->irq,NULL ); } +rcu_read_unlock ();raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); }
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 15/23 KVM: arm64: vgic-its: Treat the LPI translation cache as an rculist
In reply to Oliver UptonConvert the LPI translation cache to an rculist such that readers can walk it while only holding the RCU read lock.
--- arch/arm64/kvm/vgic/vgic-its.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +6 -6
@@ -569,7 +569,7 @@ static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist, {struct vgic_translation_cache_entry *cte; -list_for_each_entry (cte, &dist->lpi_translation_cache, entry) { +list_for_each_entry_rcu (cte, &dist->lpi_translation_cache, entry) {/* * If we hit a NULL entry, there is nothing after this * point.
@@ -625,7 +625,7 @@ static struct vgic_translation_cache_entry *vgic_its_cache_victim(struct vgic_di* older entries in the case of a tie. Return the max usage count seen * during the scan to initialize the new cache entry. */ -list_for_each_entry (cte, &dist->lpi_translation_cache, entry) { +list_for_each_entry_rcu (cte, &dist->lpi_translation_cache, entry) { tmp =atomic64_read (&cte->usage_count); max =max (max, tmp);
@@ -679,7 +679,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,if (dist->lpi_cache_count >=vgic_its_max_cache_size (kvm)) { victim =vgic_its_cache_victim (dist, &usage); -list_del (&victim->entry); +list_del_rcu (&victim->entry); dist->lpi_cache_count--; }
@@ -697,7 +697,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,rcu_assign_pointer (new->irq, irq);/* Move the new translation to the head of the list */ -list_add (&new->entry, &dist->lpi_translation_cache); +list_add_rcu (&new->entry, &dist->lpi_translation_cache); dist->lpi_cache_count++; out:
@@ -734,7 +734,7 @@ void vgic_its_invalidate_cache(struct kvm *kvm)raw_spin_lock_irqsave (&dist->lpi_list_lock, flags);rcu_read_lock (); -list_for_each_entry (cte, &dist->lpi_translation_cache, entry) { +list_for_each_entry_rcu (cte, &dist->lpi_translation_cache, entry) {/* * If we hit a NULL entry, there is nothing after this * point.
@@ -1981,7 +1981,7 @@ void vgic_lpi_translation_cache_destroy(struct kvm *kvm)list_for_each_entry_safe (cte, tmp, &dist->lpi_translation_cache, entry) { -list_del (&cte->entry); +list_del_rcu (&cte->entry);kfree (cte); } }
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 16/23 KVM: arm64: vgic-its: Rely on RCU to protect translation cache reads
In reply to Oliver UptonStop taking the lpi_list_lock when reading the LPI translation cache.
--- arch/arm64/kvm/vgic/vgic-its.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +2 -3
@@ -593,15 +593,14 @@ static struct vgic_irq *vgic_its_check_cache(struct kvm *kvm, phys_addr_t db, {struct vgic_dist *dist = &kvm->arch.vgic;struct vgic_irq *irq; -unsigned long flags; -raw_spin_lock_irqsave (&dist->lpi_list_lock, flags); +rcu_read_lock (); irq =__vgic_its_check_cache (dist, db, devid, eventid);if (!vgic_try_get_irq_kref (irq)) irq =NULL ; -raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); +rcu_read_unlock ();return irq; }
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 17/23 KVM: selftests: Align with kernel's GIC definitions
In reply to Oliver UptonThere are a few subtle incongruencies between the GIC definitions used by the kernel and selftests. Furthermore, the selftests header blends implementation detail (e.g. default priority) with the architectural definitions.
This is all rather annoying, since bulk imports of the kernel header is not possible. Move selftests-specific definitions out of the offending header and realign tests on the canonical definitions for things like sysregs. Finally, haul in a fresh copy of the gicv3 header to enable a forthcoming ITS selftest.
--- .../testing/selftests/kvm/aarch64/vgic_irq.c | 4 +- .../selftests/kvm/include/aarch64/gic_v3.h | 586 +++++++++++++++++- .../selftests/kvm/lib/aarch64/gic_v3.c | 13 +- 3 files changed, 568 insertions(+), 35 deletions(-)
tools/testing/selftests/kvm/aarch64/vgic_irq.c +2 -2
@@ -152,7 +152,7 @@ static void reset_stats(void)static uint64_t gic_read_ap1r0 (void ) { -uint64_t reg =read_sysreg_s (SYS_ICV_AP1R0_EL1 ); +uint64_t reg =read_sysreg_s (SYS_ICC_AP1R0_EL1 );dsb (sy);return reg;
@@ -160,7 +160,7 @@ static uint64_t gic_read_ap1r0(void)static void gic_write_ap1r0 (uint64_t val) { -write_sysreg_s (val,SYS_ICV_AP1R0_EL1 ); +write_sysreg_s (val,SYS_ICC_AP1R0_EL1 );isb (); }
tools/testing/selftests/kvm/include/aarch64/gic_v3.h +554 -32
636 lines not on this page · raw message
tools/testing/selftests/kvm/lib/aarch64/gic_v3.c +12 -1
@@ -9,9 +9,20 @@#include "processor.h" #include "delay.h" +#include "gic.h" #include "gic_v3.h" #include "gic_private.h" +#define GICV3_MAX_CPUS 512 + +#define GICD_INT_DEF_PRI 0xa0 +#define GICD_INT_DEF_PRI_X4 ((GICD_INT_DEF_PRI << 24) |\ + (GICD_INT_DEF_PRI << 16) |\ + (GICD_INT_DEF_PRI << 8) |\ + GICD_INT_DEF_PRI) + +#define ICC_PMR_DEF_PRIO 0xf0 +struct gicv3_data {void *dist_base;void *redist_base[GICV3_MAX_CPUS ];
@@ -320,7 +331,7 @@ static void gicv3_cpu_init(unsigned int cpu, void *redist_base)write_sysreg_s (ICC_PMR_DEF_PRIO ,SYS_ICC_PMR_EL1 );/* Enable non-secure Group-1 interrupts */ -write_sysreg_s (ICC_IGRPEN1_EL1_ENABLE ,SYS_ICC_GRPEN1_EL1 ); +write_sysreg_s (ICC_IGRPEN1_EL1_MASK ,SYS_ICC_IGRPEN1_EL1 ); gicv3_data.redist_base[cpu] = redist_base_cpu; }
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 18/23 KVM: selftests: Standardise layout of GIC frames
In reply to Oliver UptonIt would appear that all of the selftests are using the same exact layout for the GIC frames. Fold this back into the library implementation to avoid defining magic values all over the selftests.
This is an extension of Colton's change, ripping out parameterization of from the library internals in addition to the public interfaces.
--- .../selftests/kvm/aarch64/arch_timer.c | 8 +-- .../testing/selftests/kvm/aarch64/vgic_irq.c | 11 +--- .../kvm/aarch64/vpmu_counter_access.c | 6 +- .../selftests/kvm/dirty_log_perf_test.c | 5 +- .../selftests/kvm/include/aarch64/gic.h | 10 +++- .../selftests/kvm/include/aarch64/vgic.h | 3 +- tools/testing/selftests/kvm/lib/aarch64/gic.c | 18 +++--- .../selftests/kvm/lib/aarch64/gic_private.h | 4 +- .../selftests/kvm/lib/aarch64/gic_v3.c | 56 +++++++++---------- .../testing/selftests/kvm/lib/aarch64/vgic.c | 18 +++--- 10 files changed, 57 insertions(+), 82 deletions(-)
tools/testing/selftests/kvm/aarch64/arch_timer.c +2 -6
@@ -59,9 +59,6 @@ static struct test_args test_args = {#define msecs_to_usecs (msec) ((msec) * 1000LL) -#define GICD_BASE_GPA 0x8000000ULL -#define GICR_BASE_GPA 0x80A0000ULL -enum guest_stage {GUEST_STAGE_VTIMER_CVAL =1 ,GUEST_STAGE_VTIMER_TVAL ,
@@ -204,8 +201,7 @@ static void guest_code(void)local_irq_disable (); -gic_init (GIC_V3 , test_args.nr_vcpus, - (void *)GICD_BASE_GPA , (void *)GICR_BASE_GPA ); +gic_init (GIC_V3 , test_args.nr_vcpus);timer_set_ctl (VIRTUAL ,CTL_IMASK );timer_set_ctl (PHYSICAL ,CTL_IMASK );
@@ -391,7 +387,7 @@ static struct kvm_vm *test_vm_create(void)vcpu_init_descriptor_tables (vcpus[i]);test_init_timer_irq (vm); - gic_fd =vgic_v3_setup (vm, nr_vcpus,64 ,GICD_BASE_GPA ,GICR_BASE_GPA ); + gic_fd =vgic_v3_setup (vm, nr_vcpus,64 );__TEST_REQUIRE (gic_fd >=0 ,"Failed to create vgic-v3" );/* Make all the test's cmdline args visible to the guest */
tools/testing/selftests/kvm/aarch64/vgic_irq.c +2 -9
@@ -19,9 +19,6 @@#include "gic_v3.h" #include "vgic.h" -#define GICD_BASE_GPA 0x08000000ULL -#define GICR_BASE_GPA 0x080A0000ULL -/* * Stores the user specified args; it's passed to the guest and to every test * function.
@@ -49,9 +46,6 @@ struct test_args {#define IRQ_DEFAULT_PRIO (LOWEST_PRIO - 1)#define IRQ_DEFAULT_PRIO_REG (IRQ_DEFAULT_PRIO << KVM_PRIO_SHIFT)/* 0xf0 */ -static void *dist = (void *)GICD_BASE_GPA ; -static void *redist = (void *)GICR_BASE_GPA ; -/* * The kvm_inject_* utilities are used by the guest to ask the host to inject * interrupts (e.g., using the KVM_IRQ_LINE ioctl).
@@ -478,7 +472,7 @@ static void guest_code(struct test_args *args)bool level_sensitive = args->level_sensitive;struct kvm_inject_desc *f, *inject_fns; -gic_init (GIC_V3 ,1 , dist, redist); +gic_init (GIC_V3 ,1 );for (i =0 ; i < nr_irqs; i++)gic_irq_enable (i);
@@ -764,8 +758,7 @@ static void test_vgic(uint32_t nr_irqs, bool level_sensitive, bool eoi_split)memcpy (addr_gva2hva (vm, args_gva), &args,sizeof (args));vcpu_args_set (vcpu,1 , args_gva); - gic_fd =vgic_v3_setup (vm,1 , nr_irqs, -GICD_BASE_GPA ,GICR_BASE_GPA ); + gic_fd =vgic_v3_setup (vm,1 , nr_irqs);__TEST_REQUIRE (gic_fd >=0 ,"Failed to create vgic-v3, skipping" ); vm_install_exception_handler(vm, VECTOR_IRQ_CURRENT,
tools/testing/selftests/kvm/aarch64/vpmu_counter_access.c +1 -5
@@ -420,9 +420,6 @@ static void guest_code(uint64_t expected_pmcr_n)GUEST_DONE (); } -#define GICD_BASE_GPA 0x8000000ULL -#define GICR_BASE_GPA 0x80A0000ULL -/* Create a VM that has one vCPU with PMUv3 configured. */ static void create_vpmu_vm (void *guest_code) {
@@ -454,8 +451,7 @@ static void create_vpmu_vm(void *guest_code) init.features[0 ] |= (1 <<KVM_ARM_VCPU_PMU_V3 ); vpmu_vm.vcpu =aarch64_vcpu_add (vpmu_vm.vm,0 , &init, guest_code);vcpu_init_descriptor_tables (vpmu_vm.vcpu); - vpmu_vm.gic_fd =vgic_v3_setup (vpmu_vm.vm,1 ,64 , -GICD_BASE_GPA ,GICR_BASE_GPA ); + vpmu_vm.gic_fd =vgic_v3_setup (vpmu_vm.vm,1 ,64 );__TEST_REQUIRE (vpmu_vm.gic_fd >=0 ,"Failed to create vgic-v3, skipping" );
tools/testing/selftests/kvm/dirty_log_perf_test.c +1 -4
@@ -22,9 +22,6 @@#ifdef __aarch64__#include "aarch64/vgic.h" -#define GICD_BASE_GPA 0x8000000ULL -#define GICR_BASE_GPA 0x80A0000ULL -static int gic_fd;static void arch_setup_vm (struct kvm_vm *vm,unsigned int nr_vcpus)
@@ -33,7 +30,7 @@ static void arch_setup_vm(struct kvm_vm *vm, unsigned int nr_vcpus)* The test can still run even if hardware does not support GICv3, as it * is only an optimization to reduce guest exits. */ - gic_fd =vgic_v3_setup (vm, nr_vcpus,64 ,GICD_BASE_GPA ,GICR_BASE_GPA ); + gic_fd =vgic_v3_setup (vm, nr_vcpus,64 ); }static void arch_cleanup_vm (struct kvm_vm *vm)
tools/testing/selftests/kvm/include/aarch64/gic.h +8 -2
@@ -11,6 +11,13 @@ enum gic_type { GIC_TYPE_MAX, }; +#define GICD_BASE_GPA 0x8000000ULL +#define GICR_BASE_GPA (GICD_BASE_GPA + SZ_64K) + +/* The GIC is identity-mapped into the guest at the time of setup. */ +#define GICD_BASE_GVA ((void *)GICD_BASE_GPA) +#define GICR_BASE_GVA ((void *)GICR_BASE_GPA) +#define MIN_SGI 0#define MIN_PPI 16#define MIN_SPI 32
@@ -21,8 +28,7 @@ enum gic_type {#define INTID_IS_PPI (intid) (MIN_PPI <= (intid) && (intid) < MIN_SPI)#define INTID_IS_SPI (intid) (MIN_SPI <= (intid) && (intid) <= MAX_SPI) -void gic_init (enum gic_type type,unsigned int nr_cpus, -void *dist_base,void *redist_base); +void gic_init (enum gic_type type,unsigned int nr_cpus);void gic_irq_enable (unsigned int intid);void gic_irq_disable (unsigned int intid);unsigned int gic_get_and_ack_irq (void );
tools/testing/selftests/kvm/include/aarch64/vgic.h +1 -2
@@ -16,8 +16,7 @@ ((uint64_t)(flags) << 12) | \ index) -int vgic_v3_setup (struct kvm_vm *vm,unsigned int nr_vcpus,uint32_t nr_irqs, -uint64_t gicd_base_gpa,uint64_t gicr_base_gpa); +int vgic_v3_setup (struct kvm_vm *vm,unsigned int nr_vcpus,uint32_t nr_irqs);#define VGIC_MAX_RESERVED 1023
tools/testing/selftests/kvm/lib/aarch64/gic.c +7 -11
@@ -17,13 +17,12 @@static const struct gic_common_ops *gic_common_ops;static struct spinlock gic_lock; -static void gic_cpu_init (unsigned int cpu,void *redist_base) +static void gic_cpu_init (unsigned int cpu) { - gic_common_ops->gic_cpu_init (cpu, redist_base); + gic_common_ops->gic_cpu_init (cpu); } -static void -gic_dist_init (enum gic_type type,unsigned int nr_cpus,void *dist_base) +static void gic_dist_init (enum gic_type type,unsigned int nr_cpus) {const struct gic_common_ops *gic_ops =NULL ;
@@ -40,7 +39,7 @@ gic_dist_init(enum gic_type type, unsigned int nr_cpus, void *dist_base)GUEST_ASSERT (gic_ops); - gic_ops->gic_init (nr_cpus, dist_base); + gic_ops->gic_init (nr_cpus); gic_common_ops = gic_ops;/* Make sure that the initialized data is visible to all the vCPUs */
@@ -49,18 +48,15 @@ gic_dist_init(enum gic_type type, unsigned int nr_cpus, void *dist_base)spin_unlock (&gic_lock); } -void gic_init (enum gic_type type,unsigned int nr_cpus, -void *dist_base,void *redist_base) +void gic_init (enum gic_type type,unsigned int nr_cpus) {uint32_t cpu =guest_get_vcpuid ();GUEST_ASSERT (type <GIC_TYPE_MAX ); -GUEST_ASSERT (dist_base); -GUEST_ASSERT (redist_base);GUEST_ASSERT (nr_cpus); -gic_dist_init (type, nr_cpus, dist_base); -gic_cpu_init (cpu, redist_base); +gic_dist_init (type, nr_cpus); +gic_cpu_init (cpu); }void gic_irq_enable (unsigned int intid)
tools/testing/selftests/kvm/lib/aarch64/gic_private.h +2 -2
@@ -8,8 +8,8 @@#define SELFTEST_KVM_GIC_PRIVATE_H struct gic_common_ops { -void (*gic_init)(unsigned int nr_cpus,void *dist_base); -void (*gic_cpu_init)(unsigned int cpu,void *redist_base); +void (*gic_init)(unsigned int nr_cpus); +void (*gic_cpu_init)(unsigned int cpu);void (*gic_irq_enable)(unsigned int intid);void (*gic_irq_disable)(unsigned int intid);uint64_t (*gic_read_iar)(void );
tools/testing/selftests/kvm/lib/aarch64/gic_v3.c +25 -31
149 lines not on this page · raw message
tools/testing/selftests/kvm/lib/aarch64/vgic.c +8 -10
@@ -19,8 +19,6 @@* Input args: * vm - KVM VM * nr_vcpus - Number of vCPUs supported by this VM -* gicd_base_gpa - Guest Physical Address of the Distributor region -* gicr_base_gpa - Guest Physical Address of the Redistributor region * * Output args: None *
@@ -30,11 +28,10 @@* redistributor regions of the guest. Since it depends on the number of * vCPUs for the VM, it must be called after all the vCPUs have been created. */ -int vgic_v3_setup (struct kvm_vm *vm,unsigned int nr_vcpus,uint32_t nr_irqs, -uint64_t gicd_base_gpa,uint64_t gicr_base_gpa) +int vgic_v3_setup (struct kvm_vm *vm,unsigned int nr_vcpus,uint32_t nr_irqs) {int gic_fd; -uint64_t redist_attr; +uint64_t attr;struct list_head *iter;unsigned int nr_gic_pages, nr_vcpus_created =0 ;
@@ -60,18 +57,19 @@ int vgic_v3_setup(struct kvm_vm *vm, unsigned int nr_vcpus, uint32_t nr_irqs,kvm_device_attr_set (gic_fd,KVM_DEV_ARM_VGIC_GRP_CTRL ,KVM_DEV_ARM_VGIC_CTRL_INIT ,NULL ); + attr =GICD_BASE_GPA ;kvm_device_attr_set (gic_fd,KVM_DEV_ARM_VGIC_GRP_ADDR , -KVM_VGIC_V3_ADDR_TYPE_DIST , &gicd_base_gpa); +KVM_VGIC_V3_ADDR_TYPE_DIST , &attr); nr_gic_pages =vm_calc_num_guest_pages (vm->mode,KVM_VGIC_V3_DIST_SIZE ); -virt_map (vm, gicd_base_gpa, gicd_base_gpa, nr_gic_pages); +virt_map (vm,GICD_BASE_GPA ,GICD_BASE_GPA , nr_gic_pages);/* Redistributor setup */ - redist_attr =REDIST_REGION_ATTR_ADDR (nr_vcpus, gicr_base_gpa,0 ,0 ); + attr =REDIST_REGION_ATTR_ADDR (nr_vcpus,GICR_BASE_GPA ,0 ,0 );kvm_device_attr_set (gic_fd,KVM_DEV_ARM_VGIC_GRP_ADDR , -KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION , &redist_attr); +KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION , &attr); nr_gic_pages =vm_calc_num_guest_pages (vm->mode,KVM_VGIC_V3_REDIST_SIZE * nr_vcpus); -virt_map (vm, gicr_base_gpa, gicr_base_gpa, nr_gic_pages); +virt_map (vm,GICR_BASE_GPA ,GICR_BASE_GPA , nr_gic_pages);kvm_device_attr_set (gic_fd,KVM_DEV_ARM_VGIC_GRP_CTRL ,KVM_DEV_ARM_VGIC_CTRL_INIT ,NULL );
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 19/23 KVM: selftests: Add a minimal library for interacting with an ITS
In reply to Oliver UptonA prerequisite of testing LPI injection performance is of course instantiating an ITS for the guest. Add a small library for creating an ITS and interacting with it *from userspace*.
Yep, you read that right. KVM unintentionally allows userspace to send commands to the virtual ITS via the command queue. Besides adding test coverage for an elusive UAPI, interacting with the ITS in userspace simplifies the handling of commands that need to allocate memory, like a MAPD command with an ITT.
--- .../selftests/kvm/include/aarch64/gic.h | 7 +- .../selftests/kvm/include/aarch64/vgic.h | 20 ++ .../testing/selftests/kvm/lib/aarch64/vgic.c | 241 ++++++++++++++++++ 3 files changed, 267 insertions(+), 1 deletion(-)
tools/testing/selftests/kvm/include/aarch64/gic.h +6 -1
@@ -11,7 +11,12 @@ enum gic_type { GIC_TYPE_MAX, }; -#define GICD_BASE_GPA 0x8000000ULL +/* +* Note that the redistributor frames are at the end, as the range scales +* with the number of vCPUs in the VM. +*/ +#define GITS_BASE_GPA 0x8000000ULL +#define GICD_BASE_GPA (GITS_BASE_GPA + SZ_128K)#define GICR_BASE_GPA (GICD_BASE_GPA + SZ_64K)/* The GIC is identity-mapped into the guest at the time of setup. */
tools/testing/selftests/kvm/include/aarch64/vgic.h +20 -0
@@ -32,4 +32,24 @@ void kvm_irq_write_isactiver(int gic_fd, uint32_t intid, struct kvm_vcpu *vcpu);#define KVM_IRQCHIP_NUM_PINS (1020 - 32) +struct vgic_its { +int its_fd; +void *cmdq_hva; +size_t cmdq_size; +}; + +struct vgic_its *vgic_its_setup (struct kvm_vm *vm, +vm_paddr_t coll_tbl,size_t coll_tbl_sz, +vm_paddr_t device_tbl,size_t device_tbl_sz, +vm_paddr_t cmdq,size_t cmdq_size); +void vgic_its_destroy (struct vgic_its *its); + +void vgic_its_send_mapd_cmd (struct vgic_its *its,u32 device_id, +vm_paddr_t itt_base,size_t itt_size,bool valid); +void vgic_its_send_mapc_cmd (struct vgic_its *its,struct kvm_vcpu *vcpu, +u32 collection_id,bool valid); +void vgic_its_send_mapti_cmd (struct vgic_its *its,u32 device_id, +u32 event_id,u32 collection_id,u32 intid); +void vgic_its_send_invall_cmd (struct vgic_its *its,u32 collection_id); +#endif // SELFTEST_KVM_VGIC_H
tools/testing/selftests/kvm/lib/aarch64/vgic.c +241 -0
250 lines not on this page · raw message
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
Marc Zyngier
maintainerRe: [PATCH v2 19/23] KVM: selftests: Add a minimal library for interacting with an ITS
Quoted 10 lines · Oliver Upton
A prerequisite of testing LPI injection performance is of course instantiating an ITS for the guest. Add a small library for creating an ITS and interacting with it *from userspace*.
Yep, you read that right. KVM unintentionally allows userspace to send commands to the virtual ITS via the command queue. Besides adding test coverage for an elusive UAPI, interacting with the ITS in userspace simplifies the handling of commands that need to allocate memory, like a MAPD command with an ITT.
I don't mean to derail the party, but I really think we should plug this hole. Either that, or we make it an official interface for state restore. And don't we all love to have multiple interfaces to do the same thing?
Quoted 301 lines
Signed-off-by Oliver Upton <[email protected]>--- .../selftests/kvm/include/aarch64/gic.h | 7 +- .../selftests/kvm/include/aarch64/vgic.h | 20 ++ .../testing/selftests/kvm/lib/aarch64/vgic.c | 241 ++++++++++++++++++ 3 files changed, 267 insertions(+), 1 deletion(-)tools/testing/selftests/kvm/include/aarch64/gic.h +6
-1@@ -11,7 +11,12 @@ enum gic_type { GIC_TYPE_MAX, }; -#define GICD_BASE_GPA 0x8000000ULL +/* +* Note that the redistributor frames are at the end, as the range scales +* with the number of vCPUs in the VM. +*/ +#define GITS_BASE_GPA 0x8000000ULL +#define GICD_BASE_GPA (GITS_BASE_GPA + SZ_128K)#define GICR_BASE_GPA (GICD_BASE_GPA + SZ_64K)/* The GIC is identity-mapped into the guest at the time of setup. */ tools/testing/selftests/kvm/include/aarch64/vgic.h +20
-0@@ -32,4 +32,24 @@ void kvm_irq_write_isactiver(int gic_fd, uint32_t intid, struct kvm_vcpu *vcpu);#define KVM_IRQCHIP_NUM_PINS (1020 - 32) +struct vgic_its { +int its_fd; +void *cmdq_hva; +size_t cmdq_size; +}; + +struct vgic_its *vgic_its_setup (struct kvm_vm *vm, +vm_paddr_t coll_tbl,size_t coll_tbl_sz, +vm_paddr_t device_tbl,size_t device_tbl_sz, +vm_paddr_t cmdq,size_t cmdq_size); +void vgic_its_destroy (struct vgic_its *its); + +void vgic_its_send_mapd_cmd (struct vgic_its *its,u32 device_id, +vm_paddr_t itt_base,size_t itt_size,bool valid); +void vgic_its_send_mapc_cmd (struct vgic_its *its,struct kvm_vcpu *vcpu, +u32 collection_id,bool valid); +void vgic_its_send_mapti_cmd (struct vgic_its *its,u32 device_id, +u32 event_id,u32 collection_id,u32 intid); +void vgic_its_send_invall_cmd (struct vgic_its *its,u32 collection_id); +#endif // SELFTEST_KVM_VGIC_H tools/testing/selftests/kvm/lib/aarch64/vgic.c +231
-0@@ -12,6 +12,7 @@#include "vgic.h" #include "gic.h" #include "gic_v3.h" +#include "processor.h" /* * vGIC-v3 default host setup @@ -166,3 +167,243 @@ void kvm_irq_write_isactiver(int gic_fd, uint32_t intid, struct kvm_vcpu *vcpu) {vgic_poke_irq (gic_fd, intid, vcpu,GICD_ISACTIVER ); } + +static u64 vgic_its_read_reg (int its_fd,unsigned long offset) +{ +u64 attr; + +kvm_device_attr_get (its_fd,KVM_DEV_ARM_VGIC_GRP_ITS_REGS , + offset, &attr); +return attr; +} + +static void vgic_its_write_reg (int its_fd,unsigned long offset,u64 val) +{ +kvm_device_attr_set (its_fd,KVM_DEV_ARM_VGIC_GRP_ITS_REGS , + offset, &val); +} + +static unsigned long vgic_its_find_baser (int its_fd,unsigned int type) +{ +int i; + +for (i =0 ; i <GITS_BASER_NR_REGS ; i++) { +u64 baser; +unsigned long offset =GITS_BASER + (i *sizeof (baser)); + + baser =vgic_its_read_reg (its_fd, offset); +if (GITS_BASER_TYPE (baser) == type) +return offset; + } + +TEST_FAIL ("Couldn't find an ITS BASER of type %u" , type); +return -1 ; +} + +static void vgic_its_install_table (int its_fd,unsigned int type,vm_paddr_t base, +size_t size) +{ +unsigned long offset =vgic_its_find_baser (its_fd, type); +u64 baser; + + baser = ((size /SZ_64K ) -1 ) | +GITS_BASER_PAGE_SIZE_64K | + GITS_BASER_InnerShareable | + base | + GITS_BASER_RaWaWb | +GITS_BASER_VALID ; + +vgic_its_write_reg (its_fd, offset, baser); +} + +static void vgic_its_install_cmdq (int its_fd,vm_paddr_t base,size_t size) +{ +u64 cbaser; + + cbaser = ((size /SZ_4K ) -1 ) | + GITS_CBASER_InnerShareable | + base | + GITS_CBASER_RaWaWb | +GITS_CBASER_VALID ; + +vgic_its_write_reg (its_fd,GITS_CBASER , cbaser); +} + +struct vgic_its *vgic_its_setup (struct kvm_vm *vm, +vm_paddr_t coll_tbl,size_t coll_tbl_sz, +vm_paddr_t device_tbl,size_t device_tbl_sz, +vm_paddr_t cmdq,size_t cmdq_size) +{ +int its_fd =kvm_create_device (vm,KVM_DEV_TYPE_ARM_VGIC_ITS ); +struct vgic_its *its =malloc (sizeof (struct vgic_its )); +u64 attr, ctlr; + + attr =GITS_BASE_GPA ; +kvm_device_attr_set (its_fd,KVM_DEV_ARM_VGIC_GRP_ADDR , +KVM_VGIC_ITS_ADDR_TYPE , &attr); + +kvm_device_attr_set (its_fd,KVM_DEV_ARM_VGIC_GRP_CTRL , +KVM_DEV_ARM_VGIC_CTRL_INIT ,NULL ); + +vgic_its_install_table (its_fd,GITS_BASER_TYPE_COLLECTION , coll_tbl, + coll_tbl_sz); +vgic_its_install_table (its_fd,GITS_BASER_TYPE_DEVICE , device_tbl, + device_tbl_sz); + +vgic_its_install_cmdq (its_fd, cmdq, cmdq_size); + + ctlr =vgic_its_read_reg (its_fd,GITS_CTLR ); + ctlr |=GITS_CTLR_ENABLE ; +vgic_its_write_reg (its_fd,GITS_CTLR , ctlr); + + *its = (struct vgic_its ) { + .its_fd = its_fd, + .cmdq_hva =addr_gpa2hva (vm, cmdq), + .cmdq_size = cmdq_size, + }; + +return its; +} + +void vgic_its_destroy (struct vgic_its *its) +{ +close (its->its_fd); +free (its); +} + +struct its_cmd_block { +union { +u64 raw_cmd[4 ]; +__le64 raw_cmd_le[4 ]; + }; +}; + +static inline void its_fixup_cmd (struct its_cmd_block *cmd) +{ +/* Let's fixup BE commands */ + cmd->raw_cmd_le[0 ] =cpu_to_le64 (cmd->raw_cmd[0 ]); + cmd->raw_cmd_le[1 ] =cpu_to_le64 (cmd->raw_cmd[1 ]); + cmd->raw_cmd_le[2 ] =cpu_to_le64 (cmd->raw_cmd[2 ]); + cmd->raw_cmd_le[3 ] =cpu_to_le64 (cmd->raw_cmd[3 ]); +} + +static void its_mask_encode (u64 *raw_cmd,u64 val,int h,int l) +{ +u64 mask =GENMASK_ULL (h, l); + *raw_cmd &= ~mask; + *raw_cmd |= (val << l) & mask; +} + +static void its_encode_cmd (struct its_cmd_block *cmd,u8 cmd_nr) +{ +its_mask_encode (&cmd->raw_cmd[0 ], cmd_nr,7 ,0 ); +} + +static void its_encode_devid (struct its_cmd_block *cmd,u32 devid) +{ +its_mask_encode (&cmd->raw_cmd[0 ], devid,63 ,32 ); +} + +static void its_encode_event_id (struct its_cmd_block *cmd,u32 id) +{ +its_mask_encode (&cmd->raw_cmd[1 ], id,31 ,0 ); +} + +static void its_encode_phys_id (struct its_cmd_block *cmd,u32 phys_id) +{ +its_mask_encode (&cmd->raw_cmd[1 ], phys_id,63 ,32 ); +} + +static void its_encode_size (struct its_cmd_block *cmd,u8 size) +{ +its_mask_encode (&cmd->raw_cmd[1 ], size,4 ,0 ); +} + +static void its_encode_itt (struct its_cmd_block *cmd,u64 itt_addr) +{ +its_mask_encode (&cmd->raw_cmd[2 ], itt_addr >>8 ,51 ,8 ); +} + +static void its_encode_valid (struct its_cmd_block *cmd,int valid) +{ +its_mask_encode (&cmd->raw_cmd[2 ], !!valid,63 ,63 ); +} + +static void its_encode_target (struct its_cmd_block *cmd,u64 target_addr) +{ +its_mask_encode (&cmd->raw_cmd[2 ], target_addr >>16 ,51 ,16 ); +} + +static void its_encode_collection (struct its_cmd_block *cmd,u16 col) +{ +its_mask_encode (&cmd->raw_cmd[2 ], col,15 ,0 ); +} + +static void vgic_its_send_cmd (struct vgic_its *its,struct its_cmd_block *cmd) +{ +u64 cwriter =vgic_its_read_reg (its->its_fd,GITS_CWRITER ); +struct its_cmd_block *dst = its->cmdq_hva + cwriter; +u64 next; + +its_fixup_cmd (cmd); + +WRITE_ONCE (*dst, *cmd); +dsb (ishst); + + next = (cwriter +sizeof (*cmd)) % its->cmdq_size; +vgic_its_write_reg (its->its_fd,GITS_CWRITER , next); + +TEST_ASSERT (vgic_its_read_reg (its->its_fd,GITS_CREADR ) == next, +"ITS didn't process command at offset: %lu\n" , cwriter); +} + +void vgic_its_send_mapd_cmd (struct vgic_its *its,u32 device_id, +vm_paddr_t itt_base,size_t itt_size,bool valid) +{ +struct its_cmd_block cmd = {}; + +its_encode_cmd (&cmd,GITS_CMD_MAPD ); +its_encode_devid (&cmd, device_id); +its_encode_size (&cmd,ilog2 (itt_size) -1 ); +its_encode_itt (&cmd, itt_base); +its_encode_valid (&cmd, valid); + +vgic_its_send_cmd (its, &cmd); +} + +void vgic_its_send_mapc_cmd (struct vgic_its *its,struct kvm_vcpu *vcpu, +u32 collection_id,bool valid) +{ +struct its_cmd_block cmd = {}; + +its_encode_cmd (&cmd,GITS_CMD_MAPC ); +its_encode_collection (&cmd, collection_id); +its_encode_target (&cmd, vcpu->id); +its_encode_valid (&cmd, valid); + +vgic_its_send_cmd (its, &cmd); +} + +void vgic_its_send_mapti_cmd (struct vgic_its *its,u32 device_id, +u32 event_id,u32 collection_id,u32 intid) +{ +struct its_cmd_block cmd = {}; + +its_encode_cmd (&cmd,GITS_CMD_MAPTI ); +its_encode_devid (&cmd, device_id); +its_encode_event_id (&cmd, event_id); +its_encode_phys_id (&cmd, intid); +its_encode_collection (&cmd, collection_id); + +vgic_its_send_cmd (its, &cmd); +}
tools/testing/selftests/kvm/lib/aarch64/vgic.c void kvm_irq_write_isactiver(int gic_fd, uint32_t intid, struct kvm_vcpu *vcpu) · line 409
Quoted 10 lines
tools/testing/selftests/kvm/lib/aarch64/vgic.c +10
-0+ +void vgic_its_send_invall_cmd (struct vgic_its *its,u32 collection_id) +{ +struct its_cmd_block cmd = {}; + +its_encode_cmd (&cmd,GITS_CMD_INVALL ); +its_encode_collection (&cmd, collection_id); + +vgic_its_send_cmd (its, &cmd); +}
Holy crap, that's a whole ITS driver in loserspace. *mindblown*.
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Re: [PATCH v2 19/23] KVM: selftests: Add a minimal library for interacting with an ITS
Quoted 13 lines · Marc Zyngier
Quoted 10 lines · Oliver Upton
A prerequisite of testing LPI injection performance is of course instantiating an ITS for the guest. Add a small library for creating an ITS and interacting with it *from userspace*.
Yep, you read that right. KVM unintentionally allows userspace to send commands to the virtual ITS via the command queue. Besides adding test coverage for an elusive UAPI, interacting with the ITS in userspace simplifies the handling of commands that need to allocate memory, like a MAPD command with an ITT.
Quoted 4 lines
I don't mean to derail the party, but I really think we should plug this hole. Either that, or we make it an official interface for state restore. And don't we all love to have multiple interfaces to do the same thing?
Ok, I've thought about it a bit more and I'm fully convinced we need to shut the door on this stupidity.
We expect CREADR == CWRITER at the time userspace saves the ITS registers, but we have a *hideous* ordering issue on the restore path.
If the order of restore from userspace is CBASER, CWRITER, CREADR then we **wind up replaying the entire command queue**. While insane, I'm pretty sure it is legal for the guest to write garbage after the read pointer has moved past a particular command index.
Fsck!!!
So, how about we do this:
- Provide a uaccess hook for CWRITER that changes the write-pointer without processing any commands
- Assert an invariant that at any time CWRITER or CREADR are read from userspace that CREADR == CWRITER. Fail the ioctl and scream if that isn't the case, so that way we never need to worry about processing 'in-flight' commands at the destination.
Signature · 2 lines
Thanks, Oliver
Marc Zyngier
maintainerRe: [PATCH v2 19/23] KVM: selftests: Add a minimal library for interacting with an ITS
Quoted 26 lines · Oliver Upton
Quoted 13 lines · Marc Zyngier
Quoted 10 lines · Oliver Upton
A prerequisite of testing LPI injection performance is of course instantiating an ITS for the guest. Add a small library for creating an ITS and interacting with it *from userspace*.
Yep, you read that right. KVM unintentionally allows userspace to send commands to the virtual ITS via the command queue. Besides adding test coverage for an elusive UAPI, interacting with the ITS in userspace simplifies the handling of commands that need to allocate memory, like a MAPD command with an ITT.
Quoted 4 lines
I don't mean to derail the party, but I really think we should plug this hole. Either that, or we make it an official interface for state restore. And don't we all love to have multiple interfaces to do the same thing?
Ok, I've thought about it a bit more and I'm fully convinced we need to shut the door on this stupidity.
We expect CREADR == CWRITER at the time userspace saves the ITS registers, but we have a *hideous* ordering issue on the restore path.
Quoted 6 lines
If the order of restore from userspace is CBASER, CWRITER, CREADR then we **wind up replaying the entire command queue**. While insane, I'm pretty sure it is legal for the guest to write garbage after the read pointer has moved past a particular command index.
Fsck!!!
This is documented Documentation/virt/kvm/devices/arm-vgic-its.rst to some extent, and it is allowed for the guest to crap itself on behalf of userspace if the ordering isn't respected.
Quoted 9 lines
So, how about we do this:
- Provide a uaccess hook for CWRITER that changes the write-pointer without processing any commands
- Assert an invariant that at any time CWRITER or CREADR are read from userspace that CREADR == CWRITER. Fail the ioctl and scream if that isn't the case, so that way we never need to worry about processing 'in-flight' commands at the destination.
Are we guaranteed that we cannot ever see CWRITER != CREADR at VM dumping time? I'm not convinced that we cannot preempt the vcpu thread at the right spot, specially given that you can have an arbitrary large batch of commands to execute.
Just add a page-fault to the mix, and a signal pending. Pronto, you see a guest exit and you should be able to start dumping things without the ITS having processed much. I haven't tried, but that doesn't seem totally unlikely.
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Re: [PATCH v2 19/23] KVM: selftests: Add a minimal library for interacting with an ITS
Quoted 10 lines · Marc Zyngier
Quoted 6 lines
If the order of restore from userspace is CBASER, CWRITER, CREADR then we **wind up replaying the entire command queue**. While insane, I'm pretty sure it is legal for the guest to write garbage after the read pointer has moved past a particular command index.
Fsck!!!
This is documented
Documentation/virt/kvm/devices/arm-vgic-its.rstto some extent, and it is allowed for the guest to crap itself on behalf of userspace if the ordering isn't respected.
Ah, fair, I missed the documentation here. If we require userspace to write CTLR last then we _should_ be fine, but damn is this a tricky set of expectations.
Quoted 15 lines
Quoted 9 lines
So, how about we do this:
- Provide a uaccess hook for CWRITER that changes the write-pointer without processing any commands
- Assert an invariant that at any time CWRITER or CREADR are read from userspace that CREADR == CWRITER. Fail the ioctl and scream if that isn't the case, so that way we never need to worry about processing 'in-flight' commands at the destination.
Are we guaranteed that we cannot ever see CWRITER != CREADR at VM dumping time? I'm not convinced that we cannot preempt the vcpu thread at the right spot, specially given that you can have an arbitrary large batch of commands to execute.
Quoted 4 lines
Just add a page-fault to the mix, and a signal pending. Pronto, you see a guest exit and you should be able to start dumping things without the ITS having processed much. I haven't tried, but that doesn't seem totally unlikely.
Well, we would need to run all userspace reads and writes through the cmd_lock in this case, which is what we already do for the CREADR uaccess hook. To me the 'racy' queue accessors only make sense for guest accesses, since the driver is expecting to poll for completion in that case.
Otherwise we decide the existing rules for restoring the ITS are fine and I get to keep my funky driver :)
Signature · 2 lines
Thanks, Oliver
Re: [PATCH v2 19/23] KVM: selftests: Add a minimal library for interacting with an ITS
Quoted 36 lines · Oliver Upton
Quoted 10 lines · Marc Zyngier
Quoted 6 lines
If the order of restore from userspace is CBASER, CWRITER, CREADR then we **wind up replaying the entire command queue**. While insane, I'm pretty sure it is legal for the guest to write garbage after the read pointer has moved past a particular command index.
Fsck!!!
This is documented
Documentation/virt/kvm/devices/arm-vgic-its.rstto some extent, and it is allowed for the guest to crap itself on behalf of userspace if the ordering isn't respected.Ah, fair, I missed the documentation here. If we require userspace to write CTLR last then we _should_ be fine, but damn is this a tricky set of expectations.
Quoted 15 lines
Quoted 9 lines
So, how about we do this:
- Provide a uaccess hook for CWRITER that changes the write-pointer without processing any commands
- Assert an invariant that at any time CWRITER or CREADR are read from userspace that CREADR == CWRITER. Fail the ioctl and scream if that isn't the case, so that way we never need to worry about processing 'in-flight' commands at the destination.
Are we guaranteed that we cannot ever see CWRITER != CREADR at VM dumping time? I'm not convinced that we cannot preempt the vcpu thread at the right spot, specially given that you can have an arbitrary large batch of commands to execute.
Quoted 4 lines
Just add a page-fault to the mix, and a signal pending. Pronto, you see a guest exit and you should be able to start dumping things without the ITS having processed much. I haven't tried, but that doesn't seem totally unlikely.
Quoted 5 lines
Well, we would need to run all userspace reads and writes through the
cmd_lockin this case, which is what we already do for the CREADR uaccess hook. To me the 'racy' queue accessors only make sense for guest accesses, since the driver is expecting to poll for completion in that case.
My proposed invariant cannot be maintained, of course, since userspace can do whatever it pleases on the cmdq pointers.
Quoted 7 lines
Otherwise we decide the existing rules for restoring the ITS are fine and I get to keep my funky driver :)
Signature · 2 lines
Thanks, Oliver
PATCH v2 20/23 KVM: selftests: Add helper for enabling LPIs on a redistributor
In reply to Oliver UptonThe selftests GIC library presently does not support LPIs. Add a userspace helper for configuring a redistributor for LPIs, installing an LPI configuration table and LPI pending table.
--- .../selftests/kvm/include/aarch64/vgic.h | 4 + .../testing/selftests/kvm/lib/aarch64/vgic.c | 78 +++++++++++++++++++ 2 files changed, 82 insertions(+)
tools/testing/selftests/kvm/include/aarch64/vgic.h +4 -0
@@ -32,6 +32,10 @@ void kvm_irq_write_isactiver(int gic_fd, uint32_t intid, struct kvm_vcpu *vcpu);#define KVM_IRQCHIP_NUM_PINS (1020 - 32) +void vgic_rdist_enable_lpis (int gic_fd,struct kvm_vcpu *vcpu, +vm_paddr_t cfg_table,size_t cfg_table_size, +vm_paddr_t pend_table); +struct vgic_its {int its_fd;void *cmdq_hva;
tools/testing/selftests/kvm/lib/aarch64/vgic.c +78 -0
92 lines not on this page · raw message
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 21/23 KVM: selftests: Use MPIDR_HWID_BITMASK from cputype.h
In reply to Oliver UptonNo need for a home-rolled definition, just rely on the common header.
--- tools/testing/selftests/kvm/aarch64/psci_test.c | 2 ++ tools/testing/selftests/kvm/include/aarch64/processor.h | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-)
tools/testing/selftests/kvm/aarch64/psci_test.c +2 -0
@@ -13,7 +13,9 @@#define _GNU_SOURCE +#include <linux/kernel.h> #include <linux/psci.h> +#include <asm/cputype.h> #include "kvm_util.h" #include "processor.h"
tools/testing/selftests/kvm/include/aarch64/processor.h +0 -2
@@ -58,8 +58,6 @@ MAIR_ATTRIDX(MAIR_ATTR_NORMAL, MT_NORMAL) | \ MAIR_ATTRIDX(MAIR_ATTR_NORMAL_WT, MT_NORMAL_WT)) -#define MPIDR_HWID_BITMASK (0xff00fffffful) -void aarch64_vcpu_setup (struct kvm_vcpu *vcpu,struct kvm_vcpu_init *init);struct kvm_vcpu *aarch64_vcpu_add (struct kvm_vm *vm,uint32_t vcpu_id,struct kvm_vcpu_init *init,void *guest_code);
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 22/23 KVM: selftests: Hack in support for aligned page allocations
In reply to Oliver UptonMany of the memory allocations handed off to a GIC ITS require 64K alignment. One would think that requesting 64K of pages would follow natural alignment in selftests (much like alloc_pages() in the kernel), however that is unfortunately not the case.
Add a new helper to allow a caller to decide if they want naturally-aligned page allocations. Deliberately avoid making this the default in case this subtly breaks assumptions or memory overheads in other selftests.
--- .../selftests/kvm/include/kvm_util_base.h | 2 ++ tools/testing/selftests/kvm/lib/kvm_util.c | 27 ++++++++++++++++--- 2 files changed, 25 insertions(+), 4 deletions(-)
tools/testing/selftests/kvm/include/kvm_util_base.h +2 -0
@@ -830,6 +830,8 @@ vm_paddr_t vm_phy_page_alloc(struct kvm_vm *vm, vm_paddr_t paddr_min,vm_paddr_t vm_phy_pages_alloc (struct kvm_vm *vm,size_t num,vm_paddr_t paddr_min,uint32_t memslot);vm_paddr_t vm_alloc_page_table (struct kvm_vm *vm); +vm_paddr_t vm_phy_pages_alloc_aligned (struct kvm_vm *vm,size_t num, +vm_paddr_t paddr_min,uint32_t memslot);/* * ____vm_create() does KVM_CREATE_VM and little else. __vm_create() also
tools/testing/selftests/kvm/lib/kvm_util.c +23 -4
@@ -1984,8 +1984,9 @@ const char *exit_reason_str(unsigned int exit_reason)* and their base address is returned. A TEST_ASSERT failure occurs if * not enough pages are available at or above paddr_min. */ -vm_paddr_t vm_phy_pages_alloc (struct kvm_vm *vm,size_t num, -vm_paddr_t paddr_min,uint32_t memslot) +static vm_paddr_t __vm_phy_pages_alloc (struct kvm_vm *vm,size_t num, +vm_paddr_t paddr_min,uint32_t memslot, +bool aligned) {struct userspace_mem_region *region;sparsebit_idx_t pg, base;
@@ -1997,14 +1998,20 @@ vm_paddr_t vm_phy_pages_alloc(struct kvm_vm *vm, size_t num," paddr_min: 0x%lx page_size: 0x%x" , paddr_min, vm->page_size); +TEST_ASSERT (!aligned || (paddr_min % vm->page_size * num) ==0 , +"Min physical address isn't naturally aligned.\n" +" paddr_min: 0x%lx page_size: 0x%x num: %lu" , + paddr_min, vm->page_size, num); + region =memslot2region (vm, memslot); base = pg = paddr_min >> vm->page_shift;do {for (; pg < base + num; ++pg) {if (!sparsebit_is_set (region->unused_phy_pages, pg)) { - base = pg =sparsebit_next_set (region->unused_phy_pages, pg); -break ; +do { + base = pg =sparsebit_next_set (region->unused_phy_pages, pg); + }while (aligned && ((pg % num) !=0 )); } } }while (pg && pg != base + num);
@@ -2024,6 +2031,12 @@ vm_paddr_t vm_phy_pages_alloc(struct kvm_vm *vm, size_t num,return base * vm->page_size; } +vm_paddr_t vm_phy_pages_alloc (struct kvm_vm *vm,size_t num, +vm_paddr_t paddr_min,uint32_t memslot) +{ +return __vm_phy_pages_alloc (vm, num, paddr_min, memslot, false); +} +vm_paddr_t vm_phy_page_alloc (struct kvm_vm *vm,vm_paddr_t paddr_min,uint32_t memslot) {
@@ -2036,6 +2049,12 @@ vm_paddr_t vm_alloc_page_table(struct kvm_vm *vm) vm->memslots[MEM_REGION_PT]); } +vm_paddr_t vm_phy_pages_alloc_aligned (struct kvm_vm *vm,size_t num, +vm_paddr_t paddr_min,uint32_t memslot) +{ +return __vm_phy_pages_alloc (vm, num, paddr_min, memslot, true); +} +/* * Address Guest Virtual to Host Virtual *
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
PATCH v2 23/23 KVM: selftests: Add stress test for LPI injection
In reply to Oliver UptonNow that all the infrastructure is in place, add a test to stress KVM's LPI injection. Keep a 1:1 mapping of device IDs to signalling threads, allowing the user to scale up/down the sender side of an LPI. Make use of the new VM stats for the translation cache to estimate the translation hit rate.
Since the primary focus of the test is on performance, you'll notice that the guest code is not pedantic about the LPIs it receives. Counting the number of LPIs would require synchronization between the device and vCPU threads to avoid coalescing and would get in the way of performance numbers.
--- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/aarch64/vgic_lpi_stress.c | 388 ++++++++++++++++++ 2 files changed, 389 insertions(+) create mode 100644 tools/testing/selftests/kvm/aarch64/vgic_lpi_stress.c
tools/testing/selftests/kvm/Makefile +1 -0
@@ -153,6 +153,7 @@ TEST_GEN_PROGS_aarch64 += aarch64/smccc_filter TEST_GEN_PROGS_aarch64 += aarch64/vcpu_width_config TEST_GEN_PROGS_aarch64 += aarch64/vgic_init TEST_GEN_PROGS_aarch64 += aarch64/vgic_irq +TEST_GEN_PROGS_aarch64 += aarch64/vgic_lpi_stress TEST_GEN_PROGS_aarch64 += aarch64/vpmu_counter_access TEST_GEN_PROGS_aarch64 += access_tracking_perf_test TEST_GEN_PROGS_aarch64 += demand_paging_test
tools/testing/selftests/kvm/aarch64/vgic_lpi_stress.c new file +388 -0
388 lines not on this page · raw message
Signature · 1 line
2.43.0.687.g38aa6559b0-goog
On Tue, Feb 13, 2024 at 09:32:37AM +0000, Oliver Upton wrote:
[...]
Quoted 30 lines
Clearly the RCU synchronization is a bounding issue in this case. I think other scenarios where the cache is overcommitted (16 vCPUs, 16 devices, 17 events / device) are able to hide effects somewhat, as other threads can make forward progress while others are stuck waiting on RCU.
A few ideas on next steps:
1) Rework the
lpi_list_lockas an rwlock. This would obviate the need for RCU protection in the LPI cache as well as memory allocations on the injection path. This is actually what I had in the internal version of the series, although it was very incomplete.I'd expect this to nullify the improvement on the slightly-overcommitted case and 'fix' the pathological case.
2)
call_rcu()and move on. This feels somewhat abusive of the API, as the guest can flood the host with RCU callbacks, but I wasn't able to make my machine fall over in any mean configuration of the test.I haven't studied the degree to which such a malicious VM could adversely affect neighboring workloads.
3) Redo the whole ITS representation with xarrays and allow RCU readers outside of the ITS lock. I haven't fully thought this out, and if we pursue this option then we will need a secondary data structure to track where ITSes have been placed in guest memory to avoid taking the SRCU lock. We can then stick RCU synchronization in ITS command processing, which feels right to me, and dump the translation cache altogether.
Quoted 2 lines
I'd expect slightly worse average case performance in favor of more consistent performance.
Marc and I had an off-list conversation about this and agreed on option 4!
It is somewhat similar in spirit to (3), in that KVM will maintain an xarray translation cache per ITS, indexed by (device_id, event_id). This will be a perfect cache that can fit the entire range addressed by the ITS. The memory overheads of the xarray are not anticipated to be consequential, as the ITS memory footprint already scales linearly with the number of entries in the ITS.
Separately the DB -> ITS translation will be resolved by walking the ITSes present in the VM.
The existing invalidation calls will be scoped to an ITS besides the case where the guest disables LPIs on a redistributor.
Signature · 2 lines
Thanks, Oliver
Marc Zyngier
maintainerQuoted 6 lines · Oliver Upton
For full details on the what/why, please see the cover letter in v1.
Apologies for the delay on v2, I wanted to spend some time to get a microbenchmark in place to slam the ITS code pretty hard, and based on the results I'm glad I did.
[...]
Buglets and potential improvements aside, I like the smell of this. At least the first handful of patches could easily be taken as a separate improvement series.
Let me know how you'd like to play this.
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Quoted 9 lines · Marc Zyngier
Quoted 6 lines · Oliver Upton
For full details on the what/why, please see the cover letter in v1.
Apologies for the delay on v2, I wanted to spend some time to get a microbenchmark in place to slam the ITS code pretty hard, and based on the results I'm glad I did.
Quoted 7 lines
[...]
Buglets and potential improvements aside, I like the smell of this. At least the first handful of patches could easily be taken as a separate improvement series.
Let me know how you'd like to play this.
Yeah, I think there's 3 independent series here if we want to take the initial improvements:
- Address contention around vgic_get_irq() / vgic_put_irq() with the first 10 patches. Appears there is violent agreement these are good to go.
- Changing out the translation cache into a per-ITS xarray
- A final series cleaning up a lot of the warts we have in LPI management, like vgic_copy_lpi_list(). I believe we can get rid of the lpi_list_lock as well, but this needs to be ordered after the first 2.
I'd really like to de-risk the performance changes from the cleanups, as I'm convinced they're going to have their own respective piles of bugs.
How does that sound?
Signature · 2 lines
Thanks, Oliver
Marc Zyngier
maintainerQuoted 33 lines · Oliver Upton
Quoted 9 lines · Marc Zyngier
Quoted 6 lines · Oliver Upton
For full details on the what/why, please see the cover letter in v1.
Apologies for the delay on v2, I wanted to spend some time to get a microbenchmark in place to slam the ITS code pretty hard, and based on the results I'm glad I did.
Quoted 7 lines
[...]
Buglets and potential improvements aside, I like the smell of this. At least the first handful of patches could easily be taken as a separate improvement series.
Let me know how you'd like to play this.
Yeah, I think there's 3 independent series here if we want to take the initial improvements:
- Address contention around
vgic_get_irq()/vgic_put_irq()with the first 10 patches. Appears there is violent agreement these are good to go.- Changing out the translation cache into a per-ITS xarray
- A final series cleaning up a lot of the warts we have in LPI management, like
vgic_copy_lpi_list(). I believe we can get rid of thelpi_list_lockas well, but this needs to be ordered after the first 2.
Quoted 4 lines
I'd really like to de-risk the performance changes from the cleanups, as I'm convinced they're going to have their own respective piles of bugs.
How does that sound?
Yup, I'd be on board with that. If you can respin the first part with bugs fixed and without the stats, that'd be great. We can further bikeshed on the rest in the 6.10 time frame.
Also please Cc: Eric Auger, as he dealt with a lot of the ITS save/restore stuff.
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Quoted 8 lines · Marc Zyngier
Quoted 4 lines
I'd really like to de-risk the performance changes from the cleanups, as I'm convinced they're going to have their own respective piles of bugs.
How does that sound?
Yup, I'd be on board with that. If you can respin the first part with bugs fixed and without the stats, that'd be great. We can further bikeshed on the rest in the 6.10 time frame.
Cool. That makes things much easier to manage. I'll respin the get / put improvements shortly, and hopefully this time I actually fix the stupid lock imbalance :)
Quoted 2 lines
Also please Cc: Eric Auger, as he dealt with a lot of the ITS save/restore stuff.
Always happy to designate another victim to review my crap.
Signature · 2 lines
Thanks, Oliver