KVM: arm64: Transition to a per-ITS translation cache
cover KVM: arm64: Transition to a per-ITS translation cache
As discussed [*], here is the new take on the LPI translation cache, migrating to an xarray indexed by (devid, eventid) per ITS.
The end result is quite satisfying, as it becomes possible to rip out other nasties such as the lpi_list_lock. To that end, patches 2-6 aren't _directly_ related to the translation cache cleanup, but instead are done to enable the cleanups at the end of the series.
I changed out my test machine from the last time so the baseline has moved a bit, but here are the results from the vgic_lpi_stress test:
+----------------------------+------------+-------------------+ | Configuration | v6.8-rc1 | v6.8-rc1 + series | +----------------------------+------------+-------------------+ | -v 1 -d 1 -e 1 -i 1000000 | 2063296.81 | 1362602.35 | | -v 16 -d 16 -e 16 -i 10000 | 610678.33 | 5200910.01 | | -v 16 -d 16 -e 17 -i 10000 | 678361.53 | 5890675.51 | | -v 32 -d 32 -e 1 -i 100000 | 580918.96 | 8304552.67 | | -v 1 -d 1 -e 17 -i 1000 | 1512443.94 | 1425953.8 | +----------------------------+------------+-------------------+
Unlike last time, no dramatic regressions at any performance point. The regression on a single interrupt stream is to be expected, as the overheads of SRCU and two tree traversals (kvm_io_bus_get_dev(), translation cache xarray) are likely greater than that of a linked-list with a single node.
Keep in mind this is a microbenchmark, and the added overhead shouldn't lead to real-world performance issues.
Applies to kvmarm/next, due to the dependency on the prior LPI rework going into 6.9.
[*] https://lore.kernel.org/kvmarm/[email protected]/
Oliver Upton (20):
KVM: Treat the device list as an rculist
KVM: arm64: vgic-its: Walk LPI xarray in its_sync_lpi_pending_table()
KVM: arm64: vgic-its: Walk LPI xarray in vgic_its_invall()
KVM: arm64: vgic-its: Walk LPI xarray in vgic_its_cmd_handle_movall()
KVM: arm64: vgic-debug: Use an xarray mark for debug iterator
KVM: arm64: vgic-its: Get rid of vgic_copy_lpi_list()
KVM: arm64: vgic-its: Scope translation cache invalidations to an ITS
KVM: arm64: vgic-its: Spin off helper for finding ITS by doorbell addr
KVM: arm64: vgic-its: Maintain a translation cache per ITS
KVM: arm64: vgic-its: Use the per-ITS translation cache for injection
KVM: arm64: vgic-its: Rip out the global translation cache
KVM: arm64: vgic-its: Get rid of the lpi_list_lock
KVM: selftests: Align with kernel's GIC definitions
KVM: selftests: Standardise layout of GIC frames
KVM: selftests: Add quadword MMIO accessors
KVM: selftests: Add a minimal library for interacting with an ITS
KVM: selftests: Add helper for enabling LPIs on a redistributor
KVM: selftests: Use MPIDR_HWID_BITMASK from cputype.h
KVM: selftests: Hack in support for aligned page allocations
KVM: selftests: Add stress test for LPI injection
arch/arm64/kvm/vgic/vgic-debug.c | 82 ++- arch/arm64/kvm/vgic/vgic-init.c | 8 - arch/arm64/kvm/vgic/vgic-its.c | 335 +++------- arch/arm64/kvm/vgic/vgic-mmio-v3.c | 2 +- arch/arm64/kvm/vgic/vgic.c | 6 +- arch/arm64/kvm/vgic/vgic.h | 6 +- include/kvm/arm_vgic.h | 13 +- tools/testing/selftests/kvm/Makefile | 2 + .../selftests/kvm/aarch64/arch_timer.c | 8 +- .../testing/selftests/kvm/aarch64/psci_test.c | 2 + .../testing/selftests/kvm/aarch64/vgic_irq.c | 15 +- .../selftests/kvm/aarch64/vgic_lpi_stress.c | 413 ++++++++++++ .../kvm/aarch64/vpmu_counter_access.c | 6 +- .../selftests/kvm/dirty_log_perf_test.c | 5 +- .../selftests/kvm/include/aarch64/gic.h | 21 +- .../selftests/kvm/include/aarch64/gic_v3.h | 586 +++++++++++++++++- .../kvm/include/aarch64/gic_v3_its.h | 19 + .../selftests/kvm/include/aarch64/processor.h | 19 +- .../selftests/kvm/include/aarch64/vgic.h | 5 +- .../selftests/kvm/include/kvm_util_base.h | 2 + tools/testing/selftests/kvm/lib/aarch64/gic.c | 18 +- .../selftests/kvm/lib/aarch64/gic_private.h | 4 +- .../selftests/kvm/lib/aarch64/gic_v3.c | 99 +-- .../selftests/kvm/lib/aarch64/gic_v3_its.c | 248 ++++++++ .../testing/selftests/kvm/lib/aarch64/vgic.c | 38 +- tools/testing/selftests/kvm/lib/kvm_util.c | 27 +- virt/kvm/kvm_main.c | 8 +- 27 files changed, 1590 insertions(+), 407 deletions(-) create mode 100644 tools/testing/selftests/kvm/aarch64/vgic_lpi_stress.c create mode 100644 tools/testing/selftests/kvm/include/aarch64/gic_v3_its.h create mode 100644 tools/testing/selftests/kvm/lib/aarch64/gic_v3_its.c
base-commit: 8b177908c5fe6e38911f46c82dfa0fec7644d6dc
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 1/20 KVM: Treat the device list as an rculist
A subsequent change to KVM/arm64 will necessitate walking the device list outside of the kvm->lock. Prepare by converting to an rculist. Note that this has zero effect the destruction path, as every reader should be protected by a valid reference on the KVM struct.
--- virt/kvm/kvm_main.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
virt/kvm/kvm_main.c +5 -3
@@ -1326,10 +1326,12 @@ static void kvm_destroy_devices(struct kvm *kvm)/* * We do not need to take the kvm->lock here, because nobody else * has a reference to the struct kvm at this point and therefore -* cannot access the devices list anyhow. +* cannot access the devices list anyhow. Furthermore, there are no RCU +* readers after the last reference is dropped, so there is no need to +* defer destruction until a grace period has elapsed. */ list_for_each_entry_safe (dev, tmp, &kvm->devices, vm_node) { -list_del (&dev->vm_node); +list_del_rcu (&dev->vm_node); dev->ops->destroy (dev); } }
@@ -4781,7 +4783,7 @@ static int kvm_ioctl_create_device(struct kvm *kvm,kfree (dev);return ret; } -list_add (&dev->vm_node, &kvm->devices); +list_add_rcu (&dev->vm_node, &kvm->devices);mutex_unlock (&kvm->lock);if (ops->init)
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
Re: [PATCH 01/20] KVM: Treat the device list as an rculist
Quoted 8 lines · Oliver Upton
A subsequent change to KVM/arm64 will necessitate walking the device list outside of the
kvm->lock. Prepare by converting to an rculist. Note that this has zero effect the destruction path, as every reader should be protected by a valid reference on the KVM struct.Cc Paolo Bonzini <[email protected]>Cc Sean Christopherson <[email protected]>Signed-off-by Oliver Upton <[email protected]>
Quoted 35 lines
--- virt/kvm/kvm_main.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)virt/kvm/kvm_main.c +5
-4@@ -1326,10 +1326,12 @@ static void kvm_destroy_devices(struct kvm *kvm)/* * We do not need to take the kvm->lock here, because nobody else * has a reference to the struct kvm at this point and therefore -* cannot access the devices list anyhow. +* cannot access the devices list anyhow. Furthermore, there are no RCU +* readers after the last reference is dropped, so there is no need to +* defer destruction until a grace period has elapsed. */ list_for_each_entry_safe (dev, tmp, &kvm->devices, vm_node) { -list_del (&dev->vm_node); +list_del_rcu (&dev->vm_node); dev->ops->destroy (dev); } }@@ -4781,7 +4783,7 @@ static int kvm_ioctl_create_device(struct kvm *kvm,kfree (dev);return ret; } -list_add (&dev->vm_node, &kvm->devices); +list_add_rcu (&dev->vm_node, &kvm->devices);mutex_unlock (&kvm->lock);if (ops->init) --2.44.0.rc1.240.g4c46232300-goog
Sean Christopherson
reviewerRe: [PATCH 01/20] KVM: Treat the device list as an rculist
In reply to Oliver UptonQuoted 4 lines · Oliver Upton
A subsequent change to KVM/arm64 will necessitate walking the device list outside of the
kvm->lock. Prepare by converting to an rculist. Note that this has zero effect the destruction path, as every reader should be protected by a valid reference on the KVM struct.
This should leave the destruction path alone then. Simply using list_del_rcu() doesn't convert this to an rculist, you need to actually synchronize against RCU for this to have any meaning/protection. The above alludes to this, and the comment in kvm_destroy_devices() helps a little, but the code itself is actively misleading.
I don't know how I feel about using list_add_rcu() in combination with list_del(), but I like it a lot better than using list_del_rcu() in a way that is blatantly wrong.
So if we don't have a better option, I would much rather do only the list_add_rcu(), and add a comment _there_ explaining why KVM inserts with RCU protection, but frees using regular ol' list_del().
Quoted 38 lines
Cc Paolo Bonzini <[email protected]>Cc Sean Christopherson <[email protected]>Signed-off-by Oliver Upton <[email protected]>--- virt/kvm/kvm_main.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)virt/kvm/kvm_main.c +5
-3@@ -1326,10 +1326,12 @@ static void kvm_destroy_devices(struct kvm *kvm)/* * We do not need to take the kvm->lock here, because nobody else * has a reference to the struct kvm at this point and therefore -* cannot access the devices list anyhow. +* cannot access the devices list anyhow. Furthermore, there are no RCU +* readers after the last reference is dropped, so there is no need to +* defer destruction until a grace period has elapsed. */ list_for_each_entry_safe (dev, tmp, &kvm->devices, vm_node) { -list_del (&dev->vm_node); +list_del_rcu (&dev->vm_node); dev->ops->destroy (dev); } }@@ -4781,7 +4783,7 @@ static int kvm_ioctl_create_device(struct kvm *kvm,kfree (dev);return ret; } -list_add (&dev->vm_node, &kvm->devices); +list_add_rcu (&dev->vm_node, &kvm->devices);mutex_unlock (&kvm->lock);if (ops->init)Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
Re: [PATCH 01/20] KVM: Treat the device list as an rculist
Quoted 9 lines · Sean Christopherson
Quoted 4 lines · Oliver Upton
A subsequent change to KVM/arm64 will necessitate walking the device list outside of the
kvm->lock. Prepare by converting to an rculist. Note that this has zero effect the destruction path, as every reader should be protected by a valid reference on the KVM struct.This should leave the destruction path alone then. Simply using
list_del_rcu()doesn't convert this to an rculist, you need to actually synchronize against RCU for this to have any meaning/protection.
Did anything about this diff give you the impression this wasn't thoroughly understood?
Quoted 3 lines
The above alludes to this, and the
comment inkvm_destroy_devices()helps a little, but the code itself is actively misleading.
Which is what comments are for. I deliberately chose to use a consistent programming model (albeit unnecessary to anyone who actually understands how this works) with the hope that _if_ someone comes along and needs to delete from the list later on they consider the requirements of RCU protection.
list_del() or list_del_rcu() will fail in an equally-miserable manner if the previously stated expectation of readers is violated. Poisoning the forward pointer would be nice from a debugging POV, but readers could still hit a use-after-free.
Quoted 7 lines
I don't know how I feel about using
list_add_rcu()in combination withlist_del(), but I like it a lot better than usinglist_del_rcu()in a way that is blatantly wrong.So if we don't have a better option, I would much rather do only the
list_add_rcu(), and add a comment _there_ explaining why KVM inserts with RCU protection, but frees using regular ol'list_del().
Describing our destruction rules on the insertion path? I feel like that borders on contempt for the reader. I'm happy to leave list_del() as-is but I'm going to insist on documenting destructor behavior in the destructor.
Signature · 2 lines
Thanks, Oliver
Sean Christopherson
reviewerRe: [PATCH 01/20] KVM: Treat the device list as an rculist
Quoted 7 lines · Oliver Upton
Quoted 6 lines · Sean Christopherson
Quoted 4 lines · Oliver Upton
A subsequent change to KVM/arm64 will necessitate walking the device list outside of the
kvm->lock. Prepare by converting to an rculist. Note that this has zero effect the destruction path, as every reader should be protected by a valid reference on the KVM struct.
Quoted 6 lines
Quoted 3 lines
This should leave the destruction path alone then. Simply using
list_del_rcu()doesn't convert this to an rculist, you need to actually synchronize against RCU for this to have any meaning/protection.Did anything about this diff give you the impression this wasn't thoroughly understood?
I'm taking exception to the "Prepare by converting to an rculist." statement. This is not an RCU-protected list, it's a list that abuses rcu_list_add() and list_for_each_rcu() to allow readers to run concurrently with insertion. E.g. IIUC, if it weren't for PROVE_RCU, the rcu_read_(un)lock() in the reader could be omitted and everything would work just fine.
Ah, but it's a moot point, because kvm_device_release() does delete from the list, and does not do so in an RCU-safe manner. So that needs to be fixed, and then this is indeed an RCU-protected list.
static int kvm_device_release(struct inode *inode, struct file *filp)
{
struct kvm_device *dev = filp->private_data;
struct kvm *kvm = dev->kvm;
if (dev->ops->release) {
mutex_lock(&kvm->lock);
list_del(&dev->vm_node);
dev->ops->release(dev);
mutex_unlock(&kvm->lock);
}
kvm_put_kvm(kvm);
return 0;}
Quoted 9 lines
Quoted 3 lines
The above alludes to this, and the
comment inkvm_destroy_devices()helps a little, but the code itself is actively misleading.Which is what comments are for. I deliberately chose to use a consistent programming model (albeit unnecessary to anyone who actually understands how this works) with the hope that _if_ someone comes along and needs to delete from the list later on they consider the requirements of RCU protection.
Ya, I see where you're coming from.
Quoted 4 lines
list_del()orlist_del_rcu()will fail in an equally-miserable manner if the previously stated expectation of readers is violated. Poisoning the forward pointer would be nice from a debugging POV, but readers could still hit a use-after-free.
My primary concern is not what happens on failure, I'm concerned about misleading readers by implying that this is a proper RCU-protected list. But as above, that's a moot point.
As far as the failure mode, my preference is to poison the forward pointer. It's not just debug friendly; hitting a fault (#GP on x86) is a "safer" failure mode than UAF, e.g. UAF could result in data corruption if the freed memory is re-allocated before the rogue write happens.
Quoted 9 lines
Quoted 7 lines
I don't know how I feel about using
list_add_rcu()in combination withlist_del(), but I like it a lot better than usinglist_del_rcu()in a way that is blatantly wrong.So if we don't have a better option, I would much rather do only the
list_add_rcu(), and add a comment _there_ explaining why KVM inserts with RCU protection, but frees using regular ol'list_del().Describing our destruction rules on the insertion path?
I was advocating we document the insertion rules. The destruction rules for devices aren't novel, they follow all of KVM's existing rules for destroying a VM.
But again, moot point, because I was looking at this from the perspetive of a list that isn't truly RCU-protected. With this being a true RCU-protected list, I agree that adding a comment to the insertion path would be redundant/unnecessary.
Quoted 1 line
I feel like that borders on contempt for the reader.
IMO, RCU is one of the most difficult things to use _safely_. (Ab)Using RCU in a way that _looks_ unsafe is setting people up to fail.
Bug once again, moot point :-)
Quoted 2 lines
I'm happy to leave
list_del()as-is but I'm going to insist on documenting destructor behavior in the destructor.
Ya.
Re: [PATCH 01/20] KVM: Treat the device list as an rculist
Quoted 5 lines · Sean Christopherson
I'm taking exception to the "Prepare by converting to an rculist." statement. This is not an RCU-protected list, it's a list that abuses
rcu_list_add()andlist_for_each_rcu()to allow readers to run concurrently with insertion. E.g. IIUC, if it weren't forPROVE_RCU, the rcu_read_(un)lock() in the reader could be omitted and everything would work just fine.
Fair, if we wanted to be exact about it this change relies on the pointer assignment properties of RCU, but not necessarily full-blown RCU protection. But...
Quoted 3 lines
Ah, but it's a moot point, because
kvm_device_release()does delete from the list, and does not do so in an RCU-safe manner. So that needs to be fixed, and then this is indeed an RCU-protected list.
Shows how much thought I threw at the problem :D Nice catch, I'll fold in a fix next time around.
Quoted 5 lines
Quoted 4 lines
list_del()orlist_del_rcu()will fail in an equally-miserable manner if the previously stated expectation of readers is violated. Poisoning the forward pointer would be nice from a debugging POV, but readers could still hit a use-after-free.
Quoted 8 lines
My primary concern is not what happens on failure, I'm concerned about misleading readers by implying that this is a proper RCU-protected list. But as above, that's a moot point.
As far as the failure mode, my preference is to poison the forward pointer. It's not just debug friendly; hitting a fault (#GP on x86) is a "safer" failure mode than UAF, e.g. UAF could result in data corruption if the freed memory is re-allocated before the rogue write happens.
Same -- what I was getting at here is that the moment our house of cards falls apart we cannot guarantee all readers have seen the poison (instead of the stale ->next pointer), so failure patterns are not predictable.
Quoted 4 lines
Quoted 1 line
I feel like that borders on contempt for the reader.
IMO, RCU is one of the most difficult things to use _safely_. (Ab)Using RCU in a way that _looks_ unsafe is setting people up to fail.
Fair enough. Coming along and saying "I know better, let's do X" is equally (if not more) contemptuous :)
Signature · 2 lines
Thanks, Oliver
Paolo Bonzini
maintainerRe: [PATCH 01/20] KVM: Treat the device list as an rculist
Quoted 12 lines · Oliver Upton
Quoted 5 lines · Sean Christopherson
I'm taking exception to the "Prepare by converting to an rculist." statement. This is not an RCU-protected list, it's a list that abuses
rcu_list_add()andlist_for_each_rcu()to allow readers to run concurrently with insertion. E.g. IIUC, if it weren't forPROVE_RCU, the rcu_read_(un)lock() in the reader could be omitted and everything would work just fine.Fair, if we wanted to be exact about it this change relies on the pointer assignment properties of RCU, but not necessarily full-blown RCU protection. But...
Quoted 6 lines
Quoted 3 lines
Ah, but it's a moot point, because
kvm_device_release()does delete from the list, and does not do so in an RCU-safe manner. So that needs to be fixed, and then this is indeed an RCU-protected list.Shows how much thought I threw at the problem :D Nice catch, I'll fold in a fix next time around.
You also need to adjust the error case of kvm_ioctl_create_device() (not a thorough review, I did).
Also double check that it's okay for your concurrent reader to find the device while ops->init runs (or, better, ensure that it only does what it has to do after ops->init is finished, with the appropriate store-release/load-acquire operations on the two sides).
Paolo
Re: [PATCH 01/20] KVM: Treat the device list as an rculist
Quoted 20 lines · Paolo Bonzini
Quoted 12 lines · Oliver Upton
Quoted 5 lines · Sean Christopherson
I'm taking exception to the "Prepare by converting to an rculist." statement. This is not an RCU-protected list, it's a list that abuses
rcu_list_add()andlist_for_each_rcu()to allow readers to run concurrently with insertion. E.g. IIUC, if it weren't forPROVE_RCU, the rcu_read_(un)lock() in the reader could be omitted and everything would work just fine.Fair, if we wanted to be exact about it this change relies on the pointer assignment properties of RCU, but not necessarily full-blown RCU protection. But...
Quoted 6 lines
Quoted 3 lines
Ah, but it's a moot point, because
kvm_device_release()does delete from the list, and does not do so in an RCU-safe manner. So that needs to be fixed, and then this is indeed an RCU-protected list.Shows how much thought I threw at the problem :D Nice catch, I'll fold in a fix next time around.
Quoted 7 lines
You also need to adjust the error case of
kvm_ioctl_create_device()(not a thorough review, I did).Also double check that it's okay for your concurrent reader to find the device while
ops->initruns (or, better, ensure that it only does what it has to do afterops->initis finished, with the appropriate store-release/load-acquire operations on the two sides).
The reader on the GIC side of things is protected against this by construction (no ->init() op), but I'd expect the reader & device to establish their own ordering / synchronization for the relevant data. Feel like the generic code is only responsible for protecting the structure of the liest.
TBH, the issue I was working around isn't lock serialization but rather lock ordering (I need to get at this while holding the vcpu->mutex). Zero performance expectations for the reader I added over in the vgic side of things.
So, with that in mind, I'm equally satisfied doing the less-complicated thing of protecting the structure of the list with its own lock.
Signature · 2 lines
Thanks, Oliver
PATCH 2/20 KVM: arm64: vgic-its: Walk LPI xarray in its_sync_lpi_pending_table()
In reply to Oliver UptonThe new LPI xarray makes it possible to walk the VM's LPIs without holding a lock, meaning that vgic_copy_lpi_list() is no longer necessary. Prepare for the deletion by walking the LPI xarray directly in its_sync_lpi_pending_table().
--- arch/arm64/kvm/vgic/vgic-its.c | 28 +++++++++++----------------- 1 file changed, 11 insertions(+), 17 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +11 -17
@@ -446,23 +446,18 @@ static u32 max_lpis_propbaser(u64 propbaser)static int its_sync_lpi_pending_table (struct kvm_vcpu *vcpu) {gpa_t pendbase =GICR_PENDBASER_ADDRESS (vcpu->arch.vgic_cpu.pendbaser); +struct vgic_dist *dist = &vcpu->kvm->arch.vgic; +unsigned long intid, flags;struct vgic_irq *irq;int last_byte_offset =-1 ;int ret =0 ; -u32 *intids; -int nr_irqs, i; -unsigned long flags;u8 pendmask; - nr_irqs =vgic_copy_lpi_list (vcpu->kvm, vcpu, &intids); -if (nr_irqs <0 ) -return nr_irqs; - -for (i =0 ; i < nr_irqs; i++) { +xa_for_each (&dist->lpi_xa, intid, irq) {int byte_offset, bit_nr; - byte_offset = intids[i] /BITS_PER_BYTE ; - bit_nr = intids[i] %BITS_PER_BYTE ; + byte_offset = intid /BITS_PER_BYTE ; + bit_nr = intid %BITS_PER_BYTE ;/* * For contiguously allocated LPIs chances are we just read
@@ -472,22 +467,21 @@ static int its_sync_lpi_pending_table(struct kvm_vcpu *vcpu) ret =kvm_read_guest_lock (vcpu->kvm, pendbase + byte_offset, &pendmask,1 ); -if (ret) { -kfree (intids); +if (ret)return ret; - } + last_byte_offset = byte_offset; } - irq =vgic_get_irq (vcpu->kvm,NULL , intids[i]); + irq =vgic_get_irq (vcpu->kvm,NULL , intid); +raw_spin_lock_irqsave (&irq->irq_lock, flags); - irq->pending_latch = pendmask & (1U << bit_nr); +if (irq->target_vcpu == vcpu) + irq->pending_latch = pendmask & (1U << bit_nr);vgic_queue_irq_unlock (vcpu->kvm, irq, flags);vgic_put_irq (vcpu->kvm, irq); } -kfree (intids); -return ret; }
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 3/20 KVM: arm64: vgic-its: Walk LPI xarray in vgic_its_invall()
In reply to Oliver UptonThe new LPI xarray makes it possible to walk the VM's LPIs without holding a lock, meaning that vgic_copy_lpi_list() is no longer necessary. Prepare for the deletion by walking the LPI xarray directly in vgic_its_invall().
--- arch/arm64/kvm/vgic/vgic-its.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +6 -10
@@ -1363,23 +1363,19 @@ static int vgic_its_cmd_handle_inv(struct kvm *kvm, struct vgic_its *its,int vgic_its_invall (struct kvm_vcpu *vcpu) {struct kvm *kvm = vcpu->kvm; -int irq_count, i =0 ; -u32 *intids; - - irq_count =vgic_copy_lpi_list (kvm, vcpu, &intids); -if (irq_count <0 ) -return irq_count; +struct vgic_dist *dist = &kvm->arch.vgic; +struct vgic_irq *irq; +unsigned long intid; -for (i =0 ; i < irq_count; i++) { -struct vgic_irq *irq =vgic_get_irq (kvm,NULL , intids[i]); +xa_for_each (&dist->lpi_xa, intid, irq) { + irq =vgic_get_irq (kvm,NULL , intid);if (!irq)continue ; +update_lpi_config (kvm, irq, vcpu, false);vgic_put_irq (kvm, irq); } -kfree (intids); -if (vcpu->arch.vgic_cpu.vgic_v3.its_vpe.its_vm)its_invall_vpe (&vcpu->arch.vgic_cpu.vgic_v3.its_vpe);
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 4/20 KVM: arm64: vgic-its: Walk LPI xarray in vgic_its_cmd_handle_movall()
In reply to Oliver UptonThe new LPI xarray makes it possible to walk the VM's LPIs without holding a lock, meaning that vgic_copy_lpi_list() is no longer necessary. Prepare for the deletion by walking the LPI xarray directly in vgic_its_cmd_handle_movall().
--- arch/arm64/kvm/vgic/vgic-its.c | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +4 -9
@@ -1418,10 +1418,10 @@ static int vgic_its_cmd_handle_invall(struct kvm *kvm, struct vgic_its *its,static int vgic_its_cmd_handle_movall (struct kvm *kvm,struct vgic_its *its,u64 *its_cmd) { +struct vgic_dist *dist = &kvm->arch.vgic;struct kvm_vcpu *vcpu1, *vcpu2;struct vgic_irq *irq; -u32 *intids; -int irq_count, i; +unsigned long intid;/* We advertise GITS_TYPER.PTA==0, making the address the vcpu ID */ vcpu1 =kvm_get_vcpu_by_id (kvm,its_cmd_get_target_addr (its_cmd));
@@ -1433,12 +1433,8 @@ static int vgic_its_cmd_handle_movall(struct kvm *kvm, struct vgic_its *its,if (vcpu1 == vcpu2)return 0 ; - irq_count =vgic_copy_lpi_list (kvm, vcpu1, &intids); -if (irq_count <0 ) -return irq_count; - -for (i =0 ; i < irq_count; i++) { - irq =vgic_get_irq (kvm,NULL , intids[i]); +xa_for_each (&dist->lpi_xa, intid, irq) { + irq =vgic_get_irq (kvm,NULL , intid);update_affinity (irq, vcpu2);
@@ -1447,7 +1443,6 @@ static int vgic_its_cmd_handle_movall(struct kvm *kvm, struct vgic_its *its,vgic_its_invalidate_cache (kvm); -kfree (intids);return 0 ; }
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 5/20 KVM: arm64: vgic-debug: Use an xarray mark for debug iterator
In reply to Oliver UptonThe vgic debug iterator is the final user of vgic_copy_lpi_list(), but is a bit more complicated to transition to something else. Use a mark in the LPI xarray to record the indices 'known' to the debug iterator. Protect against the LPIs from being freed by associating an additional reference with the xarray mark.
Rework iter_next() to let the xarray walk 'drive' the iteration after visiting all of the SGIs, PPIs, and SPIs.
--- arch/arm64/kvm/vgic/vgic-debug.c | 82 +++++++++++++++++++++++--------- arch/arm64/kvm/vgic/vgic-its.c | 4 +- arch/arm64/kvm/vgic/vgic.h | 1 + include/kvm/arm_vgic.h | 2 + 4 files changed, 64 insertions(+), 25 deletions(-)
arch/arm64/kvm/vgic/vgic-debug.c +60 -22
@@ -28,27 +28,65 @@ struct vgic_state_iter {int nr_lpis;int dist_id;int vcpu_id; -int intid; +unsigned long intid;int lpi_idx; -u32 *lpi_array; }; -static void iter_next (struct vgic_state_iter *iter) +static void iter_next (struct kvm *kvm,struct vgic_state_iter *iter) { +struct vgic_dist *dist = &kvm->arch.vgic; +if (iter->dist_id ==0 ) { iter->dist_id++;return ; } +/* +* Let the xarray drive the iterator after the last SPI, as the iterator +* has exhausted the sequentially-allocated INTID space. +*/ +if (iter->intid >= (iter->nr_spis +VGIC_NR_PRIVATE_IRQS -1 )) { +if (iter->lpi_idx < iter->nr_lpis) +xa_find_after (&dist->lpi_xa, &iter->intid, +VGIC_LPI_MAX_INTID , +LPI_XA_MARK_DEBUG_ITER ); + iter->lpi_idx++; +return ; + } + iter->intid++;if (iter->intid ==VGIC_NR_PRIVATE_IRQS && ++iter->vcpu_id < iter->nr_cpus) iter->intid =0 ; +} -if (iter->intid >= (iter->nr_spis +VGIC_NR_PRIVATE_IRQS )) { -if (iter->lpi_idx < iter->nr_lpis) - iter->intid = iter->lpi_array[iter->lpi_idx]; - iter->lpi_idx++; +static int iter_mark_lpis (struct kvm *kvm) +{ +struct vgic_dist *dist = &kvm->arch.vgic; +struct vgic_irq *irq; +unsigned long intid; +int nr_lpis =0 ; + +xa_for_each (&dist->lpi_xa, intid, irq) { +if (!vgic_try_get_irq_kref (irq)) +continue ; + +xa_set_mark (&dist->lpi_xa, intid,LPI_XA_MARK_DEBUG_ITER ); + nr_lpis++; + } + +return nr_lpis; +} + +static void iter_unmark_lpis (struct kvm *kvm) +{ +struct vgic_dist *dist = &kvm->arch.vgic; +struct vgic_irq *irq; +unsigned long intid; + +xa_for_each (&dist->lpi_xa, intid, irq) { +xa_clear_mark (&dist->lpi_xa, intid,LPI_XA_MARK_DEBUG_ITER ); +vgic_put_irq (kvm, irq); } }
@@ -61,15 +99,12 @@ static void iter_init(struct kvm *kvm, struct vgic_state_iter *iter, iter->nr_cpus = nr_cpus; iter->nr_spis = kvm->arch.vgic.nr_spis; -if (kvm->arch.vgic.vgic_model ==KVM_DEV_TYPE_ARM_VGIC_V3 ) { - iter->nr_lpis =vgic_copy_lpi_list (kvm,NULL , &iter->lpi_array); -if (iter->nr_lpis <0 ) - iter->nr_lpis =0 ; - } +if (kvm->arch.vgic.vgic_model ==KVM_DEV_TYPE_ARM_VGIC_V3 ) + iter->nr_lpis =iter_mark_lpis (kvm);/* Fast forward to the right position if needed */ while (pos--) -iter_next (iter); +iter_next (kvm, iter); }static bool end_of_vgic (struct vgic_state_iter *iter)
@@ -114,7 +149,7 @@ static void *vgic_debug_next(struct seq_file *s, void *v, loff_t *pos)struct vgic_state_iter *iter = kvm->arch.vgic.iter; ++*pos; -iter_next (iter); +iter_next (kvm, iter);if (end_of_vgic (iter)) iter =NULL ;return iter;
@@ -134,13 +169,14 @@ static void vgic_debug_stop(struct seq_file *s, void *v)mutex_lock (&kvm->arch.config_lock); iter = kvm->arch.vgic.iter; -kfree (iter->lpi_array); +iter_unmark_lpis (kvm);kfree (iter); kvm->arch.vgic.iter =NULL ;mutex_unlock (&kvm->arch.config_lock); } -static void print_dist_state (struct seq_file *s,struct vgic_dist *dist) +static void print_dist_state (struct seq_file *s,struct vgic_dist *dist, +struct vgic_state_iter *iter) {bool v3 = dist->vgic_model ==KVM_DEV_TYPE_ARM_VGIC_V3 ;
@@ -149,7 +185,7 @@ static void print_dist_state(struct seq_file *s, struct vgic_dist *dist)seq_printf (s,"vgic_model:\t%s\n" , v3 ?"GICv3" :"GICv2" );seq_printf (s,"nr_spis:\t%d\n" , dist->nr_spis);if (v3) -seq_printf (s,"nr_lpis:\t%d\n" ,atomic_read (&dist->lpi_count)); +seq_printf (s,"nr_lpis:\t%d\n" , iter->nr_lpis);seq_printf (s,"enabled:\t%d\n" , dist->enabled);seq_printf (s,"\n" );
@@ -236,7 +272,7 @@ static int vgic_debug_show(struct seq_file *s, void *v)unsigned long flags;if (iter->dist_id ==0 ) { -print_dist_state (s, &kvm->arch.vgic); +print_dist_state (s, &kvm->arch.vgic, iter);return 0 ; }
@@ -246,11 +282,13 @@ static int vgic_debug_show(struct seq_file *s, void *v)if (iter->vcpu_id < iter->nr_cpus) vcpu =kvm_get_vcpu (kvm, iter->vcpu_id); +/* +* Expect this to succeed, as iter_mark_lpis() takes a reference on +* every LPI to be visited. +*/ irq =vgic_get_irq (kvm, vcpu, iter->intid); -if (!irq) { -seq_printf (s," LPI %4d freed\n" , iter->intid); -return 0 ; - } +if (WARN_ON_ONCE (!irq)) +return -EINVAL ;raw_spin_lock_irqsave (&irq->irq_lock, flags);print_irq_state (s, irq, vcpu);
arch/arm64/kvm/vgic/vgic-its.c +1 -3
@@ -316,8 +316,6 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,return 0 ; } -#define GIC_LPI_MAX_INTID ((1 << INTERRUPT_ID_BITS_ITS) - 1) -/* * Create a snapshot of the current LPIs targeting @vcpu, so that we can * enumerate those LPIs without holding any lock.
@@ -347,7 +345,7 @@ int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr)raw_spin_lock_irqsave (&dist->lpi_list_lock, flags);rcu_read_lock (); -xas_for_each (&xas, irq,GIC_LPI_MAX_INTID ) { +xas_for_each (&xas, irq,VGIC_LPI_MAX_INTID ) {if (i == irq_count)break ;/* We don't need to "get" the IRQ, as we hold the list lock. */
arch/arm64/kvm/vgic/vgic.h +1 -0
@@ -16,6 +16,7 @@#define INTERRUPT_ID_BITS_SPIS 10#define INTERRUPT_ID_BITS_ITS 16 +#define VGIC_LPI_MAX_INTID ((1 << INTERRUPT_ID_BITS_ITS) - 1)#define VGIC_PRI_BITS 5#define vgic_irq_is_sgi (intid) ((intid) < VGIC_NR_SGIS)
include/kvm/arm_vgic.h +2 -0
@@ -276,6 +276,8 @@ struct vgic_dist {/* Protects the lpi_list. */ raw_spinlock_t lpi_list_lock; + +#define LPI_XA_MARK_DEBUG_ITER XA_MARK_0struct xarray lpi_xa;atomic_t lpi_count;
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 6/20 KVM: arm64: vgic-its: Get rid of vgic_copy_lpi_list()
In reply to Oliver UptonThe last user has been transitioned to walking the LPI xarray directly. Cut the wart off, and get rid of the now unneeded lpi_count while doing so.
--- arch/arm64/kvm/vgic/vgic-its.c | 48 ---------------------------------- arch/arm64/kvm/vgic/vgic.c | 1 - arch/arm64/kvm/vgic/vgic.h | 1 - include/kvm/arm_vgic.h | 1 - 4 files changed, 51 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +0 -48
@@ -86,11 +86,8 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,if (ret) {xa_release (&dist->lpi_xa, intid);kfree (irq); - goto out_unlock; } -atomic_inc (&dist->lpi_count); - out_unlock:raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags);
@@ -316,51 +313,6 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,return 0 ; } -/* -* Create a snapshot of the current LPIs targeting @vcpu, so that we can -* enumerate those LPIs without holding any lock. -* Returns their number and puts the kmalloc'ed array into intid_ptr. -*/ -int vgic_copy_lpi_list (struct kvm *kvm,struct kvm_vcpu *vcpu,u32 **intid_ptr) -{ -struct vgic_dist *dist = &kvm->arch.vgic; -XA_STATE (xas, &dist->lpi_xa,GIC_LPI_OFFSET ); -struct vgic_irq *irq; -unsigned long flags; -u32 *intids; -int irq_count, i =0 ; - -/* -* There is an obvious race between allocating the array and LPIs -* being mapped/unmapped. If we ended up here as a result of a -* command, we're safe (locks are held, preventing another -* command). If coming from another path (such as enabling LPIs), -* we must be careful not to overrun the array. -*/ - irq_count =atomic_read (&dist->lpi_count); - intids =kmalloc_array (irq_count,sizeof (intids[0 ]),GFP_KERNEL_ACCOUNT ); -if (!intids) -return -ENOMEM ; - -raw_spin_lock_irqsave (&dist->lpi_list_lock, flags); -rcu_read_lock (); - -xas_for_each (&xas, irq,VGIC_LPI_MAX_INTID ) { -if (i == irq_count) -break ; -/* We don't need to "get" the IRQ, as we hold the list lock. */ -if (vcpu && irq->target_vcpu != vcpu) -continue ; - intids[i++] = irq->intid; - } - -rcu_read_unlock (); -raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); - - *intid_ptr = intids; -return i; -} -static int update_affinity (struct vgic_irq *irq,struct kvm_vcpu *vcpu) {int ret =0 ;
arch/arm64/kvm/vgic/vgic.c +0 -1
@@ -126,7 +126,6 @@ void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq)__xa_erase (&dist->lpi_xa, irq->intid);xa_unlock_irqrestore (&dist->lpi_xa, flags); -atomic_dec (&dist->lpi_count);kfree_rcu (irq, rcu); }
arch/arm64/kvm/vgic/vgic.h +0 -1
@@ -331,7 +331,6 @@ static inline bool vgic_dist_overlap(struct kvm *kvm, gpa_t base, size_t size) }bool vgic_lpis_enabled (struct kvm_vcpu *vcpu); -int vgic_copy_lpi_list (struct kvm *kvm,struct kvm_vcpu *vcpu,u32 **intid_ptr);int vgic_its_resolve_lpi (struct kvm *kvm,struct vgic_its *its,u32 devid,u32 eventid,struct vgic_irq **irq);struct vgic_its *vgic_msi_to_its (struct kvm *kvm,struct kvm_msi *msi);
include/kvm/arm_vgic.h +0 -1
@@ -279,7 +279,6 @@ struct vgic_dist {#define LPI_XA_MARK_DEBUG_ITER XA_MARK_0struct xarray lpi_xa; -atomic_t lpi_count;/* LPI translation cache */ struct list_head lpi_translation_cache;
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 7/20 KVM: arm64: vgic-its: Scope translation cache invalidations to an ITS
In reply to Oliver UptonAs the current LPI translation cache is global, the corresponding invalidation helpers are also globally-scoped. In anticipation of constructing a translation cache per ITS, add a helper for scoped cache invalidations.
We still need to support global invalidations when LPIs are toggled on a redistributor, as a property of the translation cache is that all stored LPIs are known to be delieverable.
--- arch/arm64/kvm/vgic/vgic-its.c | 45 ++++++++++++++++++++++-------- arch/arm64/kvm/vgic/vgic-mmio-v3.c | 2 +- arch/arm64/kvm/vgic/vgic.h | 2 +- 3 files changed, 35 insertions(+), 14 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +33 -12
@@ -23,6 +23,8 @@#include "vgic.h" #include "vgic-mmio.h" +static struct kvm_device_ops kvm_arm_vgic_its_ops; +static int vgic_its_save_tables_v0 (struct vgic_its *its);static int vgic_its_restore_tables_v0 (struct vgic_its *its);static int vgic_its_commit_v0 (struct vgic_its *its);
@@ -614,7 +616,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); } -void vgic_its_invalidate_cache (struct kvm *kvm) +static void vgic_its_invalidate_cache (struct vgic_its *its) {struct vgic_dist *dist = &kvm->arch.vgic;struct vgic_translation_cache_entry *cte;
@@ -637,6 +639,24 @@ void vgic_its_invalidate_cache(struct kvm *kvm)raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); } +void vgic_its_invalidate_all_caches (struct kvm *kvm) +{ +struct kvm_device *dev; +struct vgic_its *its; + +rcu_read_lock (); + +list_for_each_entry_rcu (dev, &kvm->devices, vm_node) { +if (dev->ops != &kvm_arm_vgic_its_ops) +continue ; + + its = dev->private; +vgic_its_invalidate_cache (its); + } + +rcu_read_unlock (); +} +int vgic_its_resolve_lpi (struct kvm *kvm,struct vgic_its *its,u32 devid,u32 eventid,struct vgic_irq **irq) {
@@ -824,7 +844,7 @@ static int vgic_its_cmd_handle_discard(struct kvm *kvm, struct vgic_its *its,* don't bother here since we clear the ITTE anyway and the * pending state is a property of the ITTE struct. */ -vgic_its_invalidate_cache (kvm); +vgic_its_invalidate_cache (its);its_free_ite (kvm, ite);return 0 ;
@@ -861,7 +881,7 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its, ite->collection = collection; vcpu =collection_to_vcpu (kvm, collection); -vgic_its_invalidate_cache (kvm); +vgic_its_invalidate_cache (its);return update_affinity (ite->irq, vcpu); }
@@ -1108,7 +1128,8 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its, }/* Requires the its_lock to be held. */ -static void vgic_its_free_device (struct kvm *kvm,struct its_device *device) +static void vgic_its_free_device (struct kvm *kvm,struct vgic_its *its, +struct its_device *device) {struct its_ite *ite, *temp;
@@ -1120,7 +1141,7 @@ static void vgic_its_free_device(struct kvm *kvm, struct its_device *device)list_for_each_entry_safe (ite, temp, &device->itt_head, ite_list)its_free_ite (kvm, ite); -vgic_its_invalidate_cache (kvm); +vgic_its_invalidate_cache (its);list_del (&device->dev_list);kfree (device);
@@ -1132,7 +1153,7 @@ static void vgic_its_free_device_list(struct kvm *kvm, struct vgic_its *its)struct its_device *cur, *temp;list_for_each_entry_safe (cur, temp, &its->device_list, dev_list) -vgic_its_free_device (kvm, cur); +vgic_its_free_device (kvm, its, cur); }/* its lock must be held */
@@ -1191,7 +1212,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,* by removing the mapping and re-establishing it. */ if (device) -vgic_its_free_device (kvm, device); +vgic_its_free_device (kvm, its, device);/* * The spec does not say whether unmapping a not-mapped device
@@ -1222,7 +1243,7 @@ static int vgic_its_cmd_handle_mapc(struct kvm *kvm, struct vgic_its *its,if (!valid) {vgic_its_free_collection (its, coll_id); -vgic_its_invalidate_cache (kvm); +vgic_its_invalidate_cache (its); }else {struct kvm_vcpu *vcpu;
@@ -1391,7 +1412,7 @@ static int vgic_its_cmd_handle_movall(struct kvm *kvm, struct vgic_its *its,vgic_put_irq (kvm, irq); } -vgic_its_invalidate_cache (kvm); +vgic_its_invalidate_cache (its);return 0 ; }
@@ -1743,7 +1764,7 @@ static void vgic_mmio_write_its_ctlr(struct kvm *kvm, struct vgic_its *its, its->enabled = !!(val &GITS_CTLR_ENABLE );if (!its->enabled) -vgic_its_invalidate_cache (kvm); +vgic_its_invalidate_cache (its);/* * Try to process any pending commands. This function bails out early
@@ -1876,7 +1897,7 @@ void vgic_lpi_translation_cache_destroy(struct kvm *kvm)struct vgic_dist *dist = &kvm->arch.vgic;struct vgic_translation_cache_entry *cte, *tmp; -vgic_its_invalidate_cache (kvm); +vgic_its_invalidate_all_caches (kvm);list_for_each_entry_safe (cte, tmp, &dist->lpi_translation_cache, entry) {
@@ -2368,7 +2389,7 @@ static int vgic_its_restore_dte(struct vgic_its *its, u32 id, ret =vgic_its_restore_itt (its, dev);if (ret) { -vgic_its_free_device (its->dev->kvm, dev); +vgic_its_free_device (its->dev->kvm, its, dev);return ret; }
arch/arm64/kvm/vgic/vgic-mmio-v3.c +1 -1
@@ -277,7 +277,7 @@ static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu,return ;vgic_flush_pending_lpis (vcpu); -vgic_its_invalidate_cache (vcpu->kvm); +vgic_its_invalidate_all_caches (vcpu->kvm);atomic_set_release (&vgic_cpu->ctlr,0 ); }else { ctlr = atomic_cmpxchg_acquire(&vgic_cpu->ctlr, 0,
arch/arm64/kvm/vgic/vgic.h +1 -1
@@ -337,7 +337,7 @@ struct vgic_its *vgic_msi_to_its(struct kvm *kvm, struct kvm_msi *msi);int vgic_its_inject_cached_translation (struct kvm *kvm,struct kvm_msi *msi);void vgic_lpi_translation_cache_init (struct kvm *kvm);void vgic_lpi_translation_cache_destroy (struct kvm *kvm); -void vgic_its_invalidate_cache (struct kvm *kvm); +void vgic_its_invalidate_all_caches (struct kvm *kvm);/* GICv4.1 MMIO interface */ int vgic_its_inv_lpi (struct kvm *kvm,struct vgic_irq *irq);
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 8/20 KVM: arm64: vgic-its: Spin off helper for finding ITS by doorbell addr
In reply to Oliver UptonThe fast path will soon need to find an ITS by doorbell address, as the translation caches will become local to an ITS. Spin off a helper to do just that.
--- arch/arm64/kvm/vgic/vgic-its.c | 34 ++++++++++++++++++++-------------- 1 file changed, 20 insertions(+), 14 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +20 -14
@@ -509,6 +509,25 @@ static unsigned long vgic_mmio_read_its_idregs(struct kvm *kvm,return 0 ; } +static struct vgic_its *__vgic_doorbell_to_its (struct kvm *kvm,gpa_t db) +{ +struct kvm_io_device *kvm_io_dev; +struct vgic_io_device *iodev; + + kvm_io_dev =kvm_io_bus_get_dev (kvm,KVM_MMIO_BUS , db); +if (!kvm_io_dev) +return ERR_PTR (-EINVAL ); + +if (kvm_io_dev->ops != &kvm_io_gic_ops) +return ERR_PTR (-EINVAL ); + + iodev = container_of(kvm_io_dev, struct vgic_io_device, dev); +if (iodev->iodev_type !=IODEV_ITS ) +return ERR_PTR (-EINVAL ); + +return iodev->its; +} +static struct vgic_irq *__vgic_its_check_cache (struct vgic_dist *dist,phys_addr_t db,u32 devid,u32 eventid)
@@ -686,8 +705,6 @@ int vgic_its_resolve_lpi(struct kvm *kvm, struct vgic_its *its,struct vgic_its *vgic_msi_to_its (struct kvm *kvm,struct kvm_msi *msi) {u64 address; -struct kvm_io_device *kvm_io_dev; -struct vgic_io_device *iodev;if (!vgic_has_its (kvm))return ERR_PTR (-ENODEV );
@@ -697,18 +714,7 @@ struct vgic_its *vgic_msi_to_its(struct kvm *kvm, struct kvm_msi *msi) address = (u64 )msi->address_hi <<32 | msi->address_lo; - kvm_io_dev =kvm_io_bus_get_dev (kvm,KVM_MMIO_BUS , address); -if (!kvm_io_dev) -return ERR_PTR (-EINVAL ); - -if (kvm_io_dev->ops != &kvm_io_gic_ops) -return ERR_PTR (-EINVAL ); - - iodev = container_of(kvm_io_dev, struct vgic_io_device, dev); -if (iodev->iodev_type !=IODEV_ITS ) -return ERR_PTR (-EINVAL ); - -return iodev->its; +return __vgic_doorbell_to_its (kvm, address); }/*
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 9/20 KVM: arm64: vgic-its: Maintain a translation cache per ITS
In reply to Oliver UptonWithin the context of a single ITS, it is possible to use an xarray to cache the device ID & event ID translation to a particular irq descriptor. Take advantage of this to build a translation cache capable of fitting all valid translations for a given ITS.
--- arch/arm64/kvm/vgic/vgic-its.c | 26 +++++++++++++++++++++++++- include/kvm/arm_vgic.h | 6 ++++++ 2 files changed, 31 insertions(+), 1 deletion(-)
arch/arm64/kvm/vgic/vgic-its.c +25 -1
@@ -528,6 +528,11 @@ static struct vgic_its *__vgic_doorbell_to_its(struct kvm *kvm, gpa_t db)return iodev->its; } +static unsigned long vgic_its_cache_key (u32 devid,u32 eventid) +{ +return (((unsigned long )devid) <<32 ) | eventid; +} +static struct vgic_irq *__vgic_its_check_cache (struct vgic_dist *dist,phys_addr_t db,u32 devid,u32 eventid)
@@ -581,6 +586,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, u32 devid, u32 eventid, struct vgic_irq *irq) { + unsigned long cache_key = vgic_its_cache_key(devid, eventid);struct vgic_dist *dist = &kvm->arch.vgic;struct vgic_translation_cache_entry *cte;unsigned long flags;
@@ -590,6 +596,9 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,if (irq->hw)return ; +if (xa_reserve_irq (&its->translation_cache, cache_key,GFP_KERNEL_ACCOUNT )) +return ; +raw_spin_lock_irqsave (&dist->lpi_list_lock, flags);if (unlikely (list_empty (&dist->lpi_translation_cache)))
@@ -622,6 +631,11 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,*/ lockdep_assert_held (&its->its_lock);vgic_get_irq_kref (irq); +/* +* Get a second ref for the ITS' translation cache. This will +* disappear. +*/ +vgic_get_irq_kref (irq); cte->db = db; cte->devid = devid;
@@ -631,6 +645,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,/* Move the new translation to the head of the list */ list_move (&cte->entry, &dist->lpi_translation_cache); +xa_store (&its->translation_cache, cache_key, irq,0 ); out:raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); }
@@ -639,7 +654,8 @@ static void vgic_its_invalidate_cache(struct vgic_its *its) {struct vgic_dist *dist = &kvm->arch.vgic;struct vgic_translation_cache_entry *cte; -unsigned long flags; +unsigned long flags, idx; +struct vgic_irq *irq;raw_spin_lock_irqsave (&dist->lpi_list_lock, flags);
@@ -656,6 +672,11 @@ static void vgic_its_invalidate_cache(struct vgic_its *its) }raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); + +xa_for_each (&its->translation_cache, idx, irq) { +xa_erase (&its->translation_cache, idx); +vgic_put_irq (its->dev->kvm, irq); + } }void vgic_its_invalidate_all_caches (struct kvm *kvm)
@@ -1963,6 +1984,7 @@ static int vgic_its_create(struct kvm_device *dev, u32 type)INIT_LIST_HEAD (&its->device_list);INIT_LIST_HEAD (&its->collection_list); +xa_init_flags (&its->translation_cache,XA_FLAGS_LOCK_IRQ ); dev->kvm->arch.vgic.msis_require_devid = true; dev->kvm->arch.vgic.has_its = true;
@@ -1993,6 +2015,8 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)vgic_its_free_device_list (kvm, its);vgic_its_free_collection_list (kvm, its); +vgic_its_invalidate_cache (its); +xa_destroy (&its->translation_cache);mutex_unlock (&its->its_lock);kfree (its);
include/kvm/arm_vgic.h +6 -0
@@ -210,6 +210,12 @@ struct vgic_its {struct mutex its_lock;struct list_head device_list;struct list_head collection_list; + +/* +* Caches the (device_id, event_id) -> vgic_irq translation for +* *deliverable* LPIs. +*/ +struct xarray translation_cache; };struct vgic_state_iter ;
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 10/20 KVM: arm64: vgic-its: Use the per-ITS translation cache for injection
In reply to Oliver UptonEverything is in place to switch to per-ITS translation caches. Start using the per-ITS cache to avoid the lock serialization related to the global translation cache.
--- arch/arm64/kvm/vgic/vgic-its.c | 40 +++++++++++----------------------- 1 file changed, 13 insertions(+), 27 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +13 -27
@@ -533,35 +533,17 @@ static unsigned long vgic_its_cache_key(u32 devid, u32 eventid)return (((unsigned long )devid) <<32 ) | eventid; } -static struct vgic_irq *__vgic_its_check_cache (struct vgic_dist *dist, -phys_addr_t db, +static struct vgic_irq *__vgic_its_check_cache (struct kvm *kvm,phys_addr_t db,u32 devid,u32 eventid) { -struct vgic_translation_cache_entry *cte; - -list_for_each_entry (cte, &dist->lpi_translation_cache, entry) { -/* -* If we hit a NULL entry, there is nothing after this -* point. -*/ -if (!cte->irq) -break ; - -if (cte->db != db || cte->devid != devid || - cte->eventid != eventid) -continue ; - -/* -* Move this entry to the head, as it is the most -* recently used. -*/ -if (!list_is_first (&cte->entry, &dist->lpi_translation_cache)) -list_move (&cte->entry, &dist->lpi_translation_cache); +unsigned long cache_key =vgic_its_cache_key (devid, eventid); +struct vgic_its *its; -return cte->irq; - } + its =__vgic_doorbell_to_its (kvm, db); +if (IS_ERR (its)) +return NULL ; -return NULL ; +return xa_load (&its->translation_cache, cache_key); } static struct vgic_irq *vgic_its_check_cache(struct kvm *kvm, phys_addr_t db,
@@ -572,11 +554,13 @@ static struct vgic_irq *vgic_its_check_cache(struct kvm *kvm, phys_addr_t db,unsigned long flags;raw_spin_lock_irqsave (&dist->lpi_list_lock, flags); +rcu_read_lock (); - irq =__vgic_its_check_cache (dist, db, devid, eventid); + irq =__vgic_its_check_cache (kvm, db, devid, eventid);if (!vgic_try_get_irq_kref (irq)) irq =NULL ; +rcu_read_unlock ();raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags);return irq;
@@ -600,6 +584,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,return ;raw_spin_lock_irqsave (&dist->lpi_list_lock, flags); +rcu_read_lock ();if (unlikely (list_empty (&dist->lpi_translation_cache))) goto out;
@@ -610,7 +595,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,* already */ db = its->vgic_its_base +GITS_TRANSLATER ; -if (__vgic_its_check_cache (dist, db, devid, eventid)) +if (__vgic_its_check_cache (kvm, db, devid, eventid)) goto out;/* Always reuse the last entry (LRU policy) */
@@ -647,6 +632,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,xa_store (&its->translation_cache, cache_key, irq,0 ); out: +rcu_read_unlock ();raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); }
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 11/20 KVM: arm64: vgic-its: Rip out the global translation cache
In reply to Oliver UptonThe MSI injection fast path has been transitioned away from the global translation cache. Rip it out.
--- arch/arm64/kvm/vgic/vgic-init.c | 7 --- arch/arm64/kvm/vgic/vgic-its.c | 100 +------------------------------- arch/arm64/kvm/vgic/vgic.h | 2 - include/kvm/arm_vgic.h | 3 - 4 files changed, 1 insertion(+), 111 deletions(-)
arch/arm64/kvm/vgic/vgic-init.c +0 -7
@@ -53,7 +53,6 @@ void kvm_vgic_early_init(struct kvm *kvm) {struct vgic_dist *dist = &kvm->arch.vgic; -INIT_LIST_HEAD (&dist->lpi_translation_cache);raw_spin_lock_init (&dist->lpi_list_lock);xa_init_flags (&dist->lpi_xa,XA_FLAGS_LOCK_IRQ ); }
@@ -305,9 +304,6 @@ int vgic_init(struct kvm *kvm) } } -if (vgic_has_its (kvm)) -vgic_lpi_translation_cache_init (kvm); -/* * If we have GICv4.1 enabled, unconditionally request enable the * v4 support so that we get HW-accelerated vSGIs. Otherwise, only
@@ -361,9 +357,6 @@ static void kvm_vgic_dist_destroy(struct kvm *kvm) dist->vgic_cpu_base =VGIC_ADDR_UNDEF ; } -if (vgic_has_its (kvm)) -vgic_lpi_translation_cache_destroy (kvm); -if (vgic_supports_direct_msis (kvm))vgic_v4_teardown (kvm);
arch/arm64/kvm/vgic/vgic-its.c +1 -99
150 lines not on this page · raw message
arch/arm64/kvm/vgic/vgic.h +0 -2
@@ -335,8 +335,6 @@ int vgic_its_resolve_lpi(struct kvm *kvm, struct vgic_its *its, u32 devid, u32 eventid, struct vgic_irq **irq);struct vgic_its *vgic_msi_to_its (struct kvm *kvm,struct kvm_msi *msi);int vgic_its_inject_cached_translation (struct kvm *kvm,struct kvm_msi *msi); -void vgic_lpi_translation_cache_init (struct kvm *kvm); -void vgic_lpi_translation_cache_destroy (struct kvm *kvm);void vgic_its_invalidate_all_caches (struct kvm *kvm);/* GICv4.1 MMIO interface */
include/kvm/arm_vgic.h +0 -3
@@ -286,9 +286,6 @@ struct vgic_dist {#define LPI_XA_MARK_DEBUG_ITER XA_MARK_0struct xarray lpi_xa; -/* LPI translation cache */ -struct list_head lpi_translation_cache; -/* used by vgic-debug */ struct vgic_state_iter *iter;
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 12/20 KVM: arm64: vgic-its: Get rid of the lpi_list_lock
In reply to Oliver UptonThe last genuine use case for the lpi_list_lock was the global LPI translation cache, which has been removed in favor of a per-ITS xarray. Remove a layer from the locking puzzle by getting rid of it.
vgic_add_lpi() still has a critical section that needs to protect against the insertion of other LPIs; change it to take the LPI xarray's xa_lock to retain this property.
--- arch/arm64/kvm/vgic/vgic-init.c | 1 - arch/arm64/kvm/vgic/vgic-its.c | 17 ++++++----------- arch/arm64/kvm/vgic/vgic.c | 5 ++--- include/kvm/arm_vgic.h | 3 --- 4 files changed, 8 insertions(+), 18 deletions(-)
arch/arm64/kvm/vgic/vgic-init.c +0 -1
@@ -53,7 +53,6 @@ void kvm_vgic_early_init(struct kvm *kvm) {struct vgic_dist *dist = &kvm->arch.vgic; -raw_spin_lock_init (&dist->lpi_list_lock);xa_init_flags (&dist->lpi_xa,XA_FLAGS_LOCK_IRQ ); }
arch/arm64/kvm/vgic/vgic-its.c +6 -11
@@ -69,7 +69,7 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, irq->target_vcpu = vcpu; irq->group =1 ; -raw_spin_lock_irqsave (&dist->lpi_list_lock, flags); +xa_lock_irqsave (&dist->lpi_xa, flags);/* * There could be a race with another vgic_add_lpi(), so we need to
@@ -84,14 +84,14 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, goto out_unlock; } - ret =xa_err (xa_store (&dist->lpi_xa, intid, irq,0 )); + ret =xa_err (__xa_store (&dist->lpi_xa, intid, irq,0 ));if (ret) {xa_release (&dist->lpi_xa, intid);kfree (irq); } out_unlock: -raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); +xa_unlock_irqrestore (&dist->lpi_xa, flags);if (ret)return ERR_PTR (ret);
@@ -541,11 +541,8 @@ static struct vgic_irq *__vgic_its_check_cache(struct kvm *kvm, phys_addr_t db,static struct vgic_irq *vgic_its_check_cache (struct kvm *kvm,phys_addr_t db,u32 devid,u32 eventid) { -struct vgic_dist *dist = &kvm->arch.vgic;struct vgic_irq *irq; -unsigned long flags; -raw_spin_lock_irqsave (&dist->lpi_list_lock, flags);rcu_read_lock (); irq =__vgic_its_check_cache (kvm, db, devid, eventid);
@@ -553,7 +550,6 @@ static struct vgic_irq *vgic_its_check_cache(struct kvm *kvm, phys_addr_t db, irq =NULL ;rcu_read_unlock (); -raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags);return irq; }
@@ -563,7 +559,6 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, struct vgic_irq *irq) {unsigned long cache_key =vgic_its_cache_key (devid, eventid); -struct vgic_dist *dist = &kvm->arch.vgic;unsigned long flags;phys_addr_t db;
@@ -574,7 +569,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,if (xa_reserve_irq (&its->translation_cache, cache_key,GFP_KERNEL_ACCOUNT ))return ; -raw_spin_lock_irqsave (&dist->lpi_list_lock, flags); +xa_lock_irqsave (&its->translation_cache, flags);rcu_read_lock ();/*
@@ -593,10 +588,10 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,lockdep_assert_held (&its->its_lock);vgic_get_irq_kref (irq); -xa_store (&its->translation_cache, cache_key, irq,0 ); +__xa_store (&its->translation_cache, cache_key, irq,0 ); out:rcu_read_unlock (); -raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); +xa_unlock_irqrestore (&its->translation_cache, flags); }static void vgic_its_invalidate_cache (struct vgic_its *its)
arch/arm64/kvm/vgic/vgic.c +2 -3
@@ -29,9 +29,8 @@ struct vgic_global kvm_vgic_global_state __ro_after_init = {* its->cmd_lock (mutex) * its->its_lock (mutex) * vgic_cpu->ap_list_lock must be taken with IRQs disabled -* kvm->lpi_list_lock must be taken with IRQs disabled -* vgic_dist->lpi_xa.xa_lock must be taken with IRQs disabled -* vgic_irq->irq_lock must be taken with IRQs disabled +* vgic_dist->lpi_xa.xa_lock must be taken with IRQs disabled +* vgic_irq->irq_lock must be taken with IRQs disabled * * As the ap_list_lock might be taken from the timer interrupt handler, * we have to disable IRQs before taking this lock and everything lower
include/kvm/arm_vgic.h +0 -3
@@ -280,9 +280,6 @@ struct vgic_dist {*/ u64 propbaser; -/* Protects the lpi_list. */ -raw_spinlock_t lpi_list_lock; -#define LPI_XA_MARK_DEBUG_ITER XA_MARK_0struct xarray lpi_xa;
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 13/20 KVM: selftests: Align with kernel's GIC definitions
In reply to Oliver UptonThere are a few subtle incongruencies between the GIC definitions used by the kernel and selftests. Furthermore, the selftests header blends implementation detail (e.g. default priority) with the architectural definitions.
This is all rather annoying, since bulk imports of the kernel header is not possible. Move selftests-specific definitions out of the offending header and realign tests on the canonical definitions for things like sysregs. Finally, haul in a fresh copy of the gicv3 header to enable a forthcoming ITS selftest.
--- .../testing/selftests/kvm/aarch64/vgic_irq.c | 4 +- .../selftests/kvm/include/aarch64/gic_v3.h | 586 +++++++++++++++++- .../selftests/kvm/lib/aarch64/gic_v3.c | 13 +- 3 files changed, 568 insertions(+), 35 deletions(-)
tools/testing/selftests/kvm/aarch64/vgic_irq.c +2 -2
@@ -152,7 +152,7 @@ static void reset_stats(void)static uint64_t gic_read_ap1r0 (void ) { -uint64_t reg =read_sysreg_s (SYS_ICV_AP1R0_EL1 ); +uint64_t reg =read_sysreg_s (SYS_ICC_AP1R0_EL1 );dsb (sy);return reg;
@@ -160,7 +160,7 @@ static uint64_t gic_read_ap1r0(void)static void gic_write_ap1r0 (uint64_t val) { -write_sysreg_s (val,SYS_ICV_AP1R0_EL1 ); +write_sysreg_s (val,SYS_ICC_AP1R0_EL1 );isb (); }
tools/testing/selftests/kvm/include/aarch64/gic_v3.h +554 -32
636 lines not on this page · raw message
tools/testing/selftests/kvm/lib/aarch64/gic_v3.c +12 -1
@@ -9,9 +9,20 @@#include "processor.h" #include "delay.h" +#include "gic.h" #include "gic_v3.h" #include "gic_private.h" +#define GICV3_MAX_CPUS 512 + +#define GICD_INT_DEF_PRI 0xa0 +#define GICD_INT_DEF_PRI_X4 ((GICD_INT_DEF_PRI << 24) |\ + (GICD_INT_DEF_PRI << 16) |\ + (GICD_INT_DEF_PRI << 8) |\ + GICD_INT_DEF_PRI) + +#define ICC_PMR_DEF_PRIO 0xf0 +struct gicv3_data {void *dist_base;void *redist_base[GICV3_MAX_CPUS ];
@@ -320,7 +331,7 @@ static void gicv3_cpu_init(unsigned int cpu, void *redist_base)write_sysreg_s (ICC_PMR_DEF_PRIO ,SYS_ICC_PMR_EL1 );/* Enable non-secure Group-1 interrupts */ -write_sysreg_s (ICC_IGRPEN1_EL1_ENABLE ,SYS_ICC_GRPEN1_EL1 ); +write_sysreg_s (ICC_IGRPEN1_EL1_MASK ,SYS_ICC_IGRPEN1_EL1 ); gicv3_data.redist_base[cpu] = redist_base_cpu; }
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 14/20 KVM: selftests: Standardise layout of GIC frames
In reply to Oliver UptonIt would appear that all of the selftests are using the same exact layout for the GIC frames. Fold this back into the library implementation to avoid defining magic values all over the selftests.
This is an extension of Colton's change, ripping out parameterization of from the library internals in addition to the public interfaces.
--- .../selftests/kvm/aarch64/arch_timer.c | 8 +-- .../testing/selftests/kvm/aarch64/vgic_irq.c | 11 +--- .../kvm/aarch64/vpmu_counter_access.c | 6 +- .../selftests/kvm/dirty_log_perf_test.c | 5 +- .../selftests/kvm/include/aarch64/gic.h | 12 +++- .../selftests/kvm/include/aarch64/vgic.h | 3 +- tools/testing/selftests/kvm/lib/aarch64/gic.c | 18 +++--- .../selftests/kvm/lib/aarch64/gic_private.h | 4 +- .../selftests/kvm/lib/aarch64/gic_v3.c | 62 +++++++++---------- .../testing/selftests/kvm/lib/aarch64/vgic.c | 18 +++--- 10 files changed, 62 insertions(+), 85 deletions(-)
tools/testing/selftests/kvm/aarch64/arch_timer.c +2 -6
@@ -59,9 +59,6 @@ static struct test_args test_args = {#define msecs_to_usecs (msec) ((msec) * 1000LL) -#define GICD_BASE_GPA 0x8000000ULL -#define GICR_BASE_GPA 0x80A0000ULL -enum guest_stage {GUEST_STAGE_VTIMER_CVAL =1 ,GUEST_STAGE_VTIMER_TVAL ,
@@ -204,8 +201,7 @@ static void guest_code(void)local_irq_disable (); -gic_init (GIC_V3 , test_args.nr_vcpus, - (void *)GICD_BASE_GPA , (void *)GICR_BASE_GPA ); +gic_init (GIC_V3 , test_args.nr_vcpus);timer_set_ctl (VIRTUAL ,CTL_IMASK );timer_set_ctl (PHYSICAL ,CTL_IMASK );
@@ -391,7 +387,7 @@ static struct kvm_vm *test_vm_create(void)vcpu_init_descriptor_tables (vcpus[i]);test_init_timer_irq (vm); - gic_fd =vgic_v3_setup (vm, nr_vcpus,64 ,GICD_BASE_GPA ,GICR_BASE_GPA ); + gic_fd =vgic_v3_setup (vm, nr_vcpus,64 );__TEST_REQUIRE (gic_fd >=0 ,"Failed to create vgic-v3" );/* Make all the test's cmdline args visible to the guest */
tools/testing/selftests/kvm/aarch64/vgic_irq.c +2 -9
@@ -19,9 +19,6 @@#include "gic_v3.h" #include "vgic.h" -#define GICD_BASE_GPA 0x08000000ULL -#define GICR_BASE_GPA 0x080A0000ULL -/* * Stores the user specified args; it's passed to the guest and to every test * function.
@@ -49,9 +46,6 @@ struct test_args {#define IRQ_DEFAULT_PRIO (LOWEST_PRIO - 1)#define IRQ_DEFAULT_PRIO_REG (IRQ_DEFAULT_PRIO << KVM_PRIO_SHIFT)/* 0xf0 */ -static void *dist = (void *)GICD_BASE_GPA ; -static void *redist = (void *)GICR_BASE_GPA ; -/* * The kvm_inject_* utilities are used by the guest to ask the host to inject * interrupts (e.g., using the KVM_IRQ_LINE ioctl).
@@ -478,7 +472,7 @@ static void guest_code(struct test_args *args)bool level_sensitive = args->level_sensitive;struct kvm_inject_desc *f, *inject_fns; -gic_init (GIC_V3 ,1 , dist, redist); +gic_init (GIC_V3 ,1 );for (i =0 ; i < nr_irqs; i++)gic_irq_enable (i);
@@ -764,8 +758,7 @@ static void test_vgic(uint32_t nr_irqs, bool level_sensitive, bool eoi_split)memcpy (addr_gva2hva (vm, args_gva), &args,sizeof (args));vcpu_args_set (vcpu,1 , args_gva); - gic_fd =vgic_v3_setup (vm,1 , nr_irqs, -GICD_BASE_GPA ,GICR_BASE_GPA ); + gic_fd =vgic_v3_setup (vm,1 , nr_irqs);__TEST_REQUIRE (gic_fd >=0 ,"Failed to create vgic-v3, skipping" ); vm_install_exception_handler(vm, VECTOR_IRQ_CURRENT,
tools/testing/selftests/kvm/aarch64/vpmu_counter_access.c +1 -5
@@ -420,9 +420,6 @@ static void guest_code(uint64_t expected_pmcr_n)GUEST_DONE (); } -#define GICD_BASE_GPA 0x8000000ULL -#define GICR_BASE_GPA 0x80A0000ULL -/* Create a VM that has one vCPU with PMUv3 configured. */ static void create_vpmu_vm (void *guest_code) {
@@ -454,8 +451,7 @@ static void create_vpmu_vm(void *guest_code) init.features[0 ] |= (1 <<KVM_ARM_VCPU_PMU_V3 ); vpmu_vm.vcpu =aarch64_vcpu_add (vpmu_vm.vm,0 , &init, guest_code);vcpu_init_descriptor_tables (vpmu_vm.vcpu); - vpmu_vm.gic_fd =vgic_v3_setup (vpmu_vm.vm,1 ,64 , -GICD_BASE_GPA ,GICR_BASE_GPA ); + vpmu_vm.gic_fd =vgic_v3_setup (vpmu_vm.vm,1 ,64 );__TEST_REQUIRE (vpmu_vm.gic_fd >=0 ,"Failed to create vgic-v3, skipping" );
tools/testing/selftests/kvm/dirty_log_perf_test.c +1 -4
@@ -22,9 +22,6 @@#ifdef __aarch64__#include "aarch64/vgic.h" -#define GICD_BASE_GPA 0x8000000ULL -#define GICR_BASE_GPA 0x80A0000ULL -static int gic_fd;static void arch_setup_vm (struct kvm_vm *vm,unsigned int nr_vcpus)
@@ -33,7 +30,7 @@ static void arch_setup_vm(struct kvm_vm *vm, unsigned int nr_vcpus)* The test can still run even if hardware does not support GICv3, as it * is only an optimization to reduce guest exits. */ - gic_fd =vgic_v3_setup (vm, nr_vcpus,64 ,GICD_BASE_GPA ,GICR_BASE_GPA ); + gic_fd =vgic_v3_setup (vm, nr_vcpus,64 ); }static void arch_cleanup_vm (struct kvm_vm *vm)
tools/testing/selftests/kvm/include/aarch64/gic.h +10 -2
@@ -6,11 +6,20 @@#ifndef SELFTEST_KVM_GIC_H #define SELFTEST_KVM_GIC_H +#include <asm/kvm.h> +enum gic_type {GIC_V3 ,GIC_TYPE_MAX , }; +#define GICD_BASE_GPA 0x8000000ULL +#define GICR_BASE_GPA (GICD_BASE_GPA + KVM_VGIC_V3_DIST_SIZE) + +/* The GIC is identity-mapped into the guest at the time of setup. */ +#define GICD_BASE_GVA ((volatile void *)GICD_BASE_GPA) +#define GICR_BASE_GVA ((volatile void *)GICR_BASE_GPA) +#define MIN_SGI 0#define MIN_PPI 16#define MIN_SPI 32
@@ -21,8 +30,7 @@ enum gic_type {#define INTID_IS_PPI (intid) (MIN_PPI <= (intid) && (intid) < MIN_SPI)#define INTID_IS_SPI (intid) (MIN_SPI <= (intid) && (intid) <= MAX_SPI) -void gic_init (enum gic_type type,unsigned int nr_cpus, -void *dist_base,void *redist_base); +void gic_init (enum gic_type type,unsigned int nr_cpus);void gic_irq_enable (unsigned int intid);void gic_irq_disable (unsigned int intid);unsigned int gic_get_and_ack_irq (void );
tools/testing/selftests/kvm/include/aarch64/vgic.h +1 -2
@@ -16,8 +16,7 @@ ((uint64_t)(flags) << 12) | \ index) -int vgic_v3_setup (struct kvm_vm *vm,unsigned int nr_vcpus,uint32_t nr_irqs, -uint64_t gicd_base_gpa,uint64_t gicr_base_gpa); +int vgic_v3_setup (struct kvm_vm *vm,unsigned int nr_vcpus,uint32_t nr_irqs);#define VGIC_MAX_RESERVED 1023
tools/testing/selftests/kvm/lib/aarch64/gic.c +7 -11
@@ -17,13 +17,12 @@static const struct gic_common_ops *gic_common_ops;static struct spinlock gic_lock; -static void gic_cpu_init (unsigned int cpu,void *redist_base) +static void gic_cpu_init (unsigned int cpu) { - gic_common_ops->gic_cpu_init (cpu, redist_base); + gic_common_ops->gic_cpu_init (cpu); } -static void -gic_dist_init (enum gic_type type,unsigned int nr_cpus,void *dist_base) +static void gic_dist_init (enum gic_type type,unsigned int nr_cpus) {const struct gic_common_ops *gic_ops =NULL ;
@@ -40,7 +39,7 @@ gic_dist_init(enum gic_type type, unsigned int nr_cpus, void *dist_base)GUEST_ASSERT (gic_ops); - gic_ops->gic_init (nr_cpus, dist_base); + gic_ops->gic_init (nr_cpus); gic_common_ops = gic_ops;/* Make sure that the initialized data is visible to all the vCPUs */
@@ -49,18 +48,15 @@ gic_dist_init(enum gic_type type, unsigned int nr_cpus, void *dist_base)spin_unlock (&gic_lock); } -void gic_init (enum gic_type type,unsigned int nr_cpus, -void *dist_base,void *redist_base) +void gic_init (enum gic_type type,unsigned int nr_cpus) {uint32_t cpu =guest_get_vcpuid ();GUEST_ASSERT (type <GIC_TYPE_MAX ); -GUEST_ASSERT (dist_base); -GUEST_ASSERT (redist_base);GUEST_ASSERT (nr_cpus); -gic_dist_init (type, nr_cpus, dist_base); -gic_cpu_init (cpu, redist_base); +gic_dist_init (type, nr_cpus); +gic_cpu_init (cpu); }void gic_irq_enable (unsigned int intid)
tools/testing/selftests/kvm/lib/aarch64/gic_private.h +2 -2
@@ -8,8 +8,8 @@#define SELFTEST_KVM_GIC_PRIVATE_H struct gic_common_ops { -void (*gic_init)(unsigned int nr_cpus,void *dist_base); -void (*gic_cpu_init)(unsigned int cpu,void *redist_base); +void (*gic_init)(unsigned int nr_cpus); +void (*gic_cpu_init)(unsigned int cpu);void (*gic_irq_enable)(unsigned int intid);void (*gic_irq_disable)(unsigned int intid);uint64_t (*gic_read_iar)(void );
tools/testing/selftests/kvm/lib/aarch64/gic_v3.c +28 -34
160 lines not on this page · raw message
tools/testing/selftests/kvm/lib/aarch64/vgic.c +8 -10
@@ -19,8 +19,6 @@* Input args: * vm - KVM VM * nr_vcpus - Number of vCPUs supported by this VM -* gicd_base_gpa - Guest Physical Address of the Distributor region -* gicr_base_gpa - Guest Physical Address of the Redistributor region * * Output args: None *
@@ -30,11 +28,10 @@* redistributor regions of the guest. Since it depends on the number of * vCPUs for the VM, it must be called after all the vCPUs have been created. */ -int vgic_v3_setup (struct kvm_vm *vm,unsigned int nr_vcpus,uint32_t nr_irqs, -uint64_t gicd_base_gpa,uint64_t gicr_base_gpa) +int vgic_v3_setup (struct kvm_vm *vm,unsigned int nr_vcpus,uint32_t nr_irqs) {int gic_fd; -uint64_t redist_attr; +uint64_t attr;struct list_head *iter;unsigned int nr_gic_pages, nr_vcpus_created =0 ;
@@ -60,18 +57,19 @@ int vgic_v3_setup(struct kvm_vm *vm, unsigned int nr_vcpus, uint32_t nr_irqs,kvm_device_attr_set (gic_fd,KVM_DEV_ARM_VGIC_GRP_CTRL ,KVM_DEV_ARM_VGIC_CTRL_INIT ,NULL ); + attr =GICD_BASE_GPA ;kvm_device_attr_set (gic_fd,KVM_DEV_ARM_VGIC_GRP_ADDR , -KVM_VGIC_V3_ADDR_TYPE_DIST , &gicd_base_gpa); +KVM_VGIC_V3_ADDR_TYPE_DIST , &attr); nr_gic_pages =vm_calc_num_guest_pages (vm->mode,KVM_VGIC_V3_DIST_SIZE ); -virt_map (vm, gicd_base_gpa, gicd_base_gpa, nr_gic_pages); +virt_map (vm,GICD_BASE_GPA ,GICD_BASE_GPA , nr_gic_pages);/* Redistributor setup */ - redist_attr =REDIST_REGION_ATTR_ADDR (nr_vcpus, gicr_base_gpa,0 ,0 ); + attr =REDIST_REGION_ATTR_ADDR (nr_vcpus,GICR_BASE_GPA ,0 ,0 );kvm_device_attr_set (gic_fd,KVM_DEV_ARM_VGIC_GRP_ADDR , -KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION , &redist_attr); +KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION , &attr); nr_gic_pages =vm_calc_num_guest_pages (vm->mode,KVM_VGIC_V3_REDIST_SIZE * nr_vcpus); -virt_map (vm, gicr_base_gpa, gicr_base_gpa, nr_gic_pages); +virt_map (vm,GICR_BASE_GPA ,GICR_BASE_GPA , nr_gic_pages);kvm_device_attr_set (gic_fd,KVM_DEV_ARM_VGIC_GRP_CTRL ,KVM_DEV_ARM_VGIC_CTRL_INIT ,NULL );
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 15/20 KVM: selftests: Add quadword MMIO accessors
In reply to Oliver UptonThe base registers in the GIC ITS and redistributor for LPIs are 64 bits wide. Add quadword accessors to poke at them.
--- .../selftests/kvm/include/aarch64/processor.h | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
tools/testing/selftests/kvm/include/aarch64/processor.h +17 -0
@@ -177,11 +177,28 @@ static __always_inline u32 __raw_readl(const volatile void *addr)return val; } +static __always_inline void __raw_writeq(u64 val, volatile void *addr) +{ + asmvolatile ("str %0, [%1]" : :"rZ" (val),"r" (addr)); +} + +static __always_inline u64 __raw_readq(const volatile void *addr) +{ +u64 val; + asmvolatile ("ldr %0, [%1]" :"=r" (val) :"r" (addr)); +return val; +} +#define writel_relaxed (v,c) ((void)__raw_writel((__force u32)cpu_to_le32(v),(c)))#define readl_relaxed (c) ({ u32 __r = le32_to_cpu((__force __le32)__raw_readl(c)); __r; }) +#define writeq_relaxed (v,c) ((void)__raw_writeq((__force u64)cpu_to_le64(v),(c))) +#define readq_relaxed (c) ({ u64 __r = le64_to_cpu((__force __le64)__raw_readq(c)); __r; })#define writel (v,c) ({ __iowmb(); writel_relaxed((v),(c));})#define readl (c) ({ u32 __v = readl_relaxed(c); __iormb(__v); __v; }) +#define writeq (v,c) ({ __iowmb(); writeq_relaxed((v),(c));}) +#define readq (c) ({ u64 __v = readq_relaxed(c); __iormb(__v); __v; }) +static inline void local_irq_enable (void ) {
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 16/20 KVM: selftests: Add a minimal library for interacting with an ITS
In reply to Oliver UptonA prerequisite of testing LPI injection performance is of course instantiating an ITS for the guest. Add a small library for creating an ITS and interacting with it from the guest.
--- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/include/aarch64/gic.h | 8 +- .../kvm/include/aarch64/gic_v3_its.h | 19 ++ .../selftests/kvm/include/aarch64/vgic.h | 2 + .../selftests/kvm/lib/aarch64/gic_v3_its.c | 248 ++++++++++++++++++ .../testing/selftests/kvm/lib/aarch64/vgic.c | 18 ++ 6 files changed, 295 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/kvm/include/aarch64/gic_v3_its.h create mode 100644 tools/testing/selftests/kvm/lib/aarch64/gic_v3_its.c
tools/testing/selftests/kvm/Makefile +1 -0
@@ -43,6 +43,7 @@ LIBKVM_x86_64 += lib/x86_64/vmx.c LIBKVM_aarch64 += lib/aarch64/gic.c LIBKVM_aarch64 += lib/aarch64/gic_v3.c +LIBKVM_aarch64 += lib/aarch64/gic_v3_its.c LIBKVM_aarch64 += lib/aarch64/handlers.S LIBKVM_aarch64 += lib/aarch64/processor.c LIBKVM_aarch64 += lib/aarch64/spinlock.c
tools/testing/selftests/kvm/include/aarch64/gic.h +7 -1
@@ -13,10 +13,16 @@ enum gic_type { GIC_TYPE_MAX, }; -#define GICD_BASE_GPA 0x8000000ULL +/* +* Note that the redistributor frames are at the end, as the range scales +* with the number of vCPUs in the VM. +*/ +#define GITS_BASE_GPA 0x8000000ULL +#define GICD_BASE_GPA (GITS_BASE_GPA + KVM_VGIC_V3_ITS_SIZE)#define GICR_BASE_GPA (GICD_BASE_GPA + KVM_VGIC_V3_DIST_SIZE)/* The GIC is identity-mapped into the guest at the time of setup. */ +#define GITS_BASE_GVA ((volatile void *)GITS_BASE_GPA)#define GICD_BASE_GVA ((volatile void *)GICD_BASE_GPA)#define GICR_BASE_GVA ((volatile void *)GICR_BASE_GPA)
tools/testing/selftests/kvm/include/aarch64/gic_v3_its.h new file +19 -0
@@ -0,0 +1,19 @@ +// SPDX-License-Identifier: GPL-2.0 + +#ifndef __SELFTESTS_GIC_V3_ITS_H__ +#define __SELFTESTS_GIC_V3_ITS_H__ + +#include <linux/sizes.h> + +void its_init (vm_paddr_t coll_tbl,size_t coll_tbl_sz, +vm_paddr_t device_tbl,size_t device_tbl_sz, +vm_paddr_t cmdq,size_t cmdq_size); + +void its_send_mapd_cmd (void *cmdq_base,u32 device_id,vm_paddr_t itt_base, +size_t itt_size,bool valid); +void its_send_mapc_cmd (void *cmdq_base,u32 vcpu_id,u32 collection_id,bool valid); +void its_send_mapti_cmd (void *cmdq_base,u32 device_id,u32 event_id, +u32 collection_id,u32 intid); +void its_send_invall_cmd (void *cmdq_base,u32 collection_id); + +#endif // __SELFTESTS_GIC_V3_ITS_H__
tools/testing/selftests/kvm/include/aarch64/vgic.h +2 -0
@@ -32,4 +32,6 @@ void kvm_irq_write_isactiver(int gic_fd, uint32_t intid, struct kvm_vcpu *vcpu);#define KVM_IRQCHIP_NUM_PINS (1020 - 32) +int vgic_its_setup (struct kvm_vm *vm); +#endif // SELFTEST_KVM_VGIC_H
tools/testing/selftests/kvm/lib/aarch64/gic_v3_its.c new file +248 -0
248 lines not on this page · raw message
tools/testing/selftests/kvm/lib/aarch64/vgic.c +18 -0
@@ -166,3 +166,21 @@ void kvm_irq_write_isactiver(int gic_fd, uint32_t intid, struct kvm_vcpu *vcpu) {vgic_poke_irq (gic_fd, intid, vcpu,GICD_ISACTIVER ); } + +int vgic_its_setup (struct kvm_vm *vm) +{ +int its_fd =kvm_create_device (vm,KVM_DEV_TYPE_ARM_VGIC_ITS ); +u64 attr; + + attr =GITS_BASE_GPA ; +kvm_device_attr_set (its_fd,KVM_DEV_ARM_VGIC_GRP_ADDR , +KVM_VGIC_ITS_ADDR_TYPE , &attr); + +kvm_device_attr_set (its_fd,KVM_DEV_ARM_VGIC_GRP_CTRL , +KVM_DEV_ARM_VGIC_CTRL_INIT ,NULL ); + +virt_map (vm,GITS_BASE_GPA ,GITS_BASE_GPA , +vm_calc_num_guest_pages (vm->mode,KVM_VGIC_V3_ITS_SIZE )); + +return its_fd; +}
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 17/20 KVM: selftests: Add helper for enabling LPIs on a redistributor
In reply to Oliver UptonThe selftests GIC library presently does not support LPIs. Add a userspace helper for configuring a redistributor for LPIs, installing an LPI configuration table and LPI pending table.
--- .../selftests/kvm/include/aarch64/gic.h | 3 +++ .../selftests/kvm/lib/aarch64/gic_v3.c | 24 +++++++++++++++++++ .../testing/selftests/kvm/lib/aarch64/vgic.c | 2 ++ 3 files changed, 29 insertions(+)
tools/testing/selftests/kvm/include/aarch64/gic.h +3 -0
@@ -58,4 +58,7 @@ void gic_irq_clear_pending(unsigned int intid);bool gic_irq_get_pending (unsigned int intid);void gic_irq_set_config (unsigned int intid,bool is_edge); +void gic_rdist_enable_lpis (vm_paddr_t cfg_table,size_t cfg_table_size, +vm_paddr_t pend_table); +#endif /* SELFTEST_KVM_GIC_H */
tools/testing/selftests/kvm/lib/aarch64/gic_v3.c +24 -0
@@ -401,3 +401,27 @@ const struct gic_common_ops gicv3_ops = { .gic_irq_get_pending = gicv3_irq_get_pending, .gic_irq_set_config = gicv3_irq_set_config, }; + +void gic_rdist_enable_lpis (vm_paddr_t cfg_table,size_t cfg_table_size, +vm_paddr_t pend_table) +{ +volatile void *rdist_base =gicr_base_cpu (guest_get_vcpuid ()); + +u32 ctlr; +u64 val; + + val = (cfg_table | + GICR_PROPBASER_InnerShareable | + GICR_PROPBASER_RaWaWb | + ((ilog2 (cfg_table_size) -1 ) &GICR_PROPBASER_IDBITS_MASK )); +writeq_relaxed (val, rdist_base +GICR_PROPBASER ); + + val = (pend_table | + GICR_PENDBASER_InnerShareable | + GICR_PENDBASER_RaWaWb); +writeq_relaxed (val, rdist_base +GICR_PENDBASER ); + + ctlr =readl_relaxed (rdist_base +GICR_CTLR ); + ctlr |=GICR_CTLR_ENABLE_LPIS ; +writel_relaxed (ctlr, rdist_base +GICR_CTLR ); +}
tools/testing/selftests/kvm/lib/aarch64/vgic.c +2 -0
@@ -3,8 +3,10 @@* ARM Generic Interrupt Controller (GIC) v3 host support */ +#include <linux/kernel.h> #include <linux/kvm.h> #include <linux/sizes.h> +#include <asm/cputype.h> #include <asm/kvm_para.h> #include <asm/kvm.h>
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 18/20 KVM: selftests: Use MPIDR_HWID_BITMASK from cputype.h
In reply to Oliver UptonNo need for a home-rolled definition, just rely on the common header.
--- tools/testing/selftests/kvm/aarch64/psci_test.c | 2 ++ tools/testing/selftests/kvm/include/aarch64/processor.h | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-)
tools/testing/selftests/kvm/aarch64/psci_test.c +2 -0
@@ -13,7 +13,9 @@#define _GNU_SOURCE +#include <linux/kernel.h> #include <linux/psci.h> +#include <asm/cputype.h> #include "kvm_util.h" #include "processor.h"
tools/testing/selftests/kvm/include/aarch64/processor.h +0 -2
@@ -58,8 +58,6 @@ MAIR_ATTRIDX(MAIR_ATTR_NORMAL, MT_NORMAL) | \ MAIR_ATTRIDX(MAIR_ATTR_NORMAL_WT, MT_NORMAL_WT)) -#define MPIDR_HWID_BITMASK (0xff00fffffful) -void aarch64_vcpu_setup (struct kvm_vcpu *vcpu,struct kvm_vcpu_init *init);struct kvm_vcpu *aarch64_vcpu_add (struct kvm_vm *vm,uint32_t vcpu_id,struct kvm_vcpu_init *init,void *guest_code);
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 19/20 KVM: selftests: Hack in support for aligned page allocations
In reply to Oliver UptonMany of the memory allocations handed off to a GIC ITS require 64K alignment. One would think that requesting 64K of pages would follow natural alignment in selftests (much like alloc_pages() in the kernel), however that is unfortunately not the case.
Add a new helper to allow a caller to decide if they want naturally-aligned page allocations. Deliberately avoid making this the default in case this subtly breaks assumptions or memory overheads in other selftests.
--- .../selftests/kvm/include/kvm_util_base.h | 2 ++ tools/testing/selftests/kvm/lib/kvm_util.c | 27 ++++++++++++++++--- 2 files changed, 25 insertions(+), 4 deletions(-)
tools/testing/selftests/kvm/include/kvm_util_base.h +2 -0
@@ -830,6 +830,8 @@ vm_paddr_t vm_phy_page_alloc(struct kvm_vm *vm, vm_paddr_t paddr_min,vm_paddr_t vm_phy_pages_alloc (struct kvm_vm *vm,size_t num,vm_paddr_t paddr_min,uint32_t memslot);vm_paddr_t vm_alloc_page_table (struct kvm_vm *vm); +vm_paddr_t vm_phy_pages_alloc_aligned (struct kvm_vm *vm,size_t num, +vm_paddr_t paddr_min,uint32_t memslot);/* * ____vm_create() does KVM_CREATE_VM and little else. __vm_create() also
tools/testing/selftests/kvm/lib/kvm_util.c +23 -4
@@ -1984,8 +1984,9 @@ const char *exit_reason_str(unsigned int exit_reason)* and their base address is returned. A TEST_ASSERT failure occurs if * not enough pages are available at or above paddr_min. */ -vm_paddr_t vm_phy_pages_alloc (struct kvm_vm *vm,size_t num, -vm_paddr_t paddr_min,uint32_t memslot) +static vm_paddr_t __vm_phy_pages_alloc (struct kvm_vm *vm,size_t num, +vm_paddr_t paddr_min,uint32_t memslot, +bool aligned) {struct userspace_mem_region *region;sparsebit_idx_t pg, base;
@@ -1997,14 +1998,20 @@ vm_paddr_t vm_phy_pages_alloc(struct kvm_vm *vm, size_t num," paddr_min: 0x%lx page_size: 0x%x" , paddr_min, vm->page_size); +TEST_ASSERT (!aligned || (paddr_min % vm->page_size * num) ==0 , +"Min physical address isn't naturally aligned.\n" +" paddr_min: 0x%lx page_size: 0x%x num: %lu" , + paddr_min, vm->page_size, num); + region =memslot2region (vm, memslot); base = pg = paddr_min >> vm->page_shift;do {for (; pg < base + num; ++pg) {if (!sparsebit_is_set (region->unused_phy_pages, pg)) { - base = pg =sparsebit_next_set (region->unused_phy_pages, pg); -break ; +do { + base = pg =sparsebit_next_set (region->unused_phy_pages, pg); + }while (aligned && ((pg % num) !=0 )); } } }while (pg && pg != base + num);
@@ -2024,6 +2031,12 @@ vm_paddr_t vm_phy_pages_alloc(struct kvm_vm *vm, size_t num,return base * vm->page_size; } +vm_paddr_t vm_phy_pages_alloc (struct kvm_vm *vm,size_t num, +vm_paddr_t paddr_min,uint32_t memslot) +{ +return __vm_phy_pages_alloc (vm, num, paddr_min, memslot, false); +} +vm_paddr_t vm_phy_page_alloc (struct kvm_vm *vm,vm_paddr_t paddr_min,uint32_t memslot) {
@@ -2036,6 +2049,12 @@ vm_paddr_t vm_alloc_page_table(struct kvm_vm *vm) vm->memslots[MEM_REGION_PT]); } +vm_paddr_t vm_phy_pages_alloc_aligned (struct kvm_vm *vm,size_t num, +vm_paddr_t paddr_min,uint32_t memslot) +{ +return __vm_phy_pages_alloc (vm, num, paddr_min, memslot, true); +} +/* * Address Guest Virtual to Host Virtual *
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog
PATCH 20/20 KVM: selftests: Add stress test for LPI injection
In reply to Oliver UptonNow that all the infrastructure is in place, add a test to stress KVM's LPI injection. Keep a 1:1 mapping of device IDs to signalling threads, allowing the user to scale up/down the sender side of an LPI. Make use of the new VM stats for the translation cache to estimate the translation hit rate.
Since the primary focus of the test is on performance, you'll notice that the guest code is not pedantic about the LPIs it receives. Counting the number of LPIs would require synchronization between the device and vCPU threads to avoid coalescing and would get in the way of performance numbers.
--- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/aarch64/vgic_lpi_stress.c | 413 ++++++++++++++++++ 2 files changed, 414 insertions(+) create mode 100644 tools/testing/selftests/kvm/aarch64/vgic_lpi_stress.c
tools/testing/selftests/kvm/Makefile +1 -0
@@ -154,6 +154,7 @@ TEST_GEN_PROGS_aarch64 += aarch64/smccc_filter TEST_GEN_PROGS_aarch64 += aarch64/vcpu_width_config TEST_GEN_PROGS_aarch64 += aarch64/vgic_init TEST_GEN_PROGS_aarch64 += aarch64/vgic_irq +TEST_GEN_PROGS_aarch64 += aarch64/vgic_lpi_stress TEST_GEN_PROGS_aarch64 += aarch64/vpmu_counter_access TEST_GEN_PROGS_aarch64 += access_tracking_perf_test TEST_GEN_PROGS_aarch64 += demand_paging_test
tools/testing/selftests/kvm/aarch64/vgic_lpi_stress.c new file +413 -0
413 lines not on this page · raw message
Signature · 1 line
2.44.0.rc1.240.g4c46232300-goog