KVM: arm64: Transition to a per-ITS translation cache
v3 cover KVM: arm64: Transition to a per-ITS translation cache
v2: https://lore.kernel.org/kvmarm/[email protected]/
v2 -> v3:
- Add lockdep assertion to kvm_vfio_create() (Sean)
- Address intermediate compilation issue in vgic_its_invalidate_cache() (Marc)
- Use a more compact cache index based on the DID/EID range of the ITS (Marc)
- Comment improvements (Marc)
- Avoid explicit acquisition of the translation cache's xa_lock() (Marc)
- Eliminate the need to disable IRQs to acquire the translation cache's xa_lock()
- Collapse internal helper into vgic_its_check_cache()
Oliver Upton (19):
KVM: Treat the device list as an rculist
KVM: arm64: vgic-its: Walk LPI xarray in its_sync_lpi_pending_table()
KVM: arm64: vgic-its: Walk LPI xarray in vgic_its_invall()
KVM: arm64: vgic-its: Walk LPI xarray in vgic_its_cmd_handle_movall()
KVM: arm64: vgic-debug: Use an xarray mark for debug iterator
KVM: arm64: vgic-its: Get rid of vgic_copy_lpi_list()
KVM: arm64: vgic-its: Scope translation cache invalidations to an ITS
KVM: arm64: vgic-its: Maintain a translation cache per ITS
KVM: arm64: vgic-its: Spin off helper for finding ITS by doorbell addr
KVM: arm64: vgic-its: Use the per-ITS translation cache for injection
KVM: arm64: vgic-its: Rip out the global translation cache
KVM: arm64: vgic-its: Get rid of the lpi_list_lock
KVM: selftests: Align with kernel's GIC definitions
KVM: selftests: Standardise layout of GIC frames
KVM: selftests: Add quadword MMIO accessors
KVM: selftests: Add a minimal library for interacting with an ITS
KVM: selftests: Add helper for enabling LPIs on a redistributor
KVM: selftests: Use MPIDR_HWID_BITMASK from cputype.h
KVM: selftests: Add stress test for LPI injection
arch/arm64/kvm/vgic/vgic-debug.c | 82 ++- arch/arm64/kvm/vgic/vgic-init.c | 8 - arch/arm64/kvm/vgic/vgic-its.c | 352 ++++------- arch/arm64/kvm/vgic/vgic-mmio-v3.c | 2 +- arch/arm64/kvm/vgic/vgic.c | 6 +- arch/arm64/kvm/vgic/vgic.h | 6 +- include/kvm/arm_vgic.h | 13 +- tools/testing/selftests/kvm/Makefile | 2 + .../selftests/kvm/aarch64/arch_timer.c | 8 +- .../testing/selftests/kvm/aarch64/psci_test.c | 2 + .../testing/selftests/kvm/aarch64/vgic_irq.c | 15 +- .../selftests/kvm/aarch64/vgic_lpi_stress.c | 410 ++++++++++++ .../kvm/aarch64/vpmu_counter_access.c | 6 +- .../selftests/kvm/dirty_log_perf_test.c | 5 +- .../selftests/kvm/include/aarch64/gic.h | 21 +- .../selftests/kvm/include/aarch64/gic_v3.h | 586 +++++++++++++++++- .../kvm/include/aarch64/gic_v3_its.h | 19 + .../selftests/kvm/include/aarch64/processor.h | 19 +- .../selftests/kvm/include/aarch64/vgic.h | 5 +- tools/testing/selftests/kvm/lib/aarch64/gic.c | 18 +- .../selftests/kvm/lib/aarch64/gic_private.h | 4 +- .../selftests/kvm/lib/aarch64/gic_v3.c | 99 +-- .../selftests/kvm/lib/aarch64/gic_v3_its.c | 248 ++++++++ .../testing/selftests/kvm/lib/aarch64/vgic.c | 38 +- virt/kvm/kvm_main.c | 14 +- virt/kvm/vfio.c | 2 + 26 files changed, 1572 insertions(+), 418 deletions(-) create mode 100644 tools/testing/selftests/kvm/aarch64/vgic_lpi_stress.c create mode 100644 tools/testing/selftests/kvm/include/aarch64/gic_v3_its.h create mode 100644 tools/testing/selftests/kvm/lib/aarch64/gic_v3_its.c
base-commit: fec50db7033ea478773b159e0e2efb135270e3b7
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 1/19 KVM: Treat the device list as an rculist
A subsequent change to KVM/arm64 will necessitate walking the device list outside of the kvm->lock. Prepare by converting to an rculist. This has zero effect on the VM destruction path, as it is expected every reader is backed by a reference on the kvm struct.
On the other hand, ensure a given device is completely destroyed before dropping the kvm->lock in the release() path, as certain devices expect to be a singleton (e.g. the vfio-kvm device).
--- virt/kvm/kvm_main.c | 14 +++++++++++--- virt/kvm/vfio.c | 2 ++ 2 files changed, 13 insertions(+), 3 deletions(-)
virt/kvm/kvm_main.c +11 -3
@@ -1329,6 +1329,12 @@ static void kvm_destroy_devices(struct kvm *kvm)* We do not need to take the kvm->lock here, because nobody else * has a reference to the struct kvm at this point and therefore * cannot access the devices list anyhow. +* +* The device list is generally managed as an rculist, but list_del() +* is used intentionally here. If a bug in KVM introduced a reader that +* was not backed by a reference on the kvm struct, the hope is that +* it'd consume the poisoned forward pointer instead of suffering a +* use-after-free, even though this cannot be guaranteed. */ list_for_each_entry_safe (dev, tmp, &kvm->devices, vm_node) {list_del (&dev->vm_node);
@@ -4725,7 +4731,8 @@ static int kvm_device_release(struct inode *inode, struct file *filp)if (dev->ops->release) {mutex_lock (&kvm->lock); -list_del (&dev->vm_node); +list_del_rcu (&dev->vm_node); +synchronize_rcu (); dev->ops->release (dev);mutex_unlock (&kvm->lock); }
@@ -4808,7 +4815,7 @@ static int kvm_ioctl_create_device(struct kvm *kvm,kfree (dev);return ret; } -list_add (&dev->vm_node, &kvm->devices); +list_add_rcu (&dev->vm_node, &kvm->devices);mutex_unlock (&kvm->lock);if (ops->init)
@@ -4819,7 +4826,8 @@ static int kvm_ioctl_create_device(struct kvm *kvm,if (ret <0 ) {kvm_put_kvm_no_destroy (kvm);mutex_lock (&kvm->lock); -list_del (&dev->vm_node); +list_del_rcu (&dev->vm_node); +synchronize_rcu ();if (ops->release) ops->release (dev);mutex_unlock (&kvm->lock);
virt/kvm/vfio.c +2 -0
@@ -366,6 +366,8 @@ static int kvm_vfio_create(struct kvm_device *dev, u32 type)struct kvm_device *tmp;struct kvm_vfio *kv; +lockdep_assert_held (&dev->kvm->lock); +/* Only one VFIO "device" per VM */ list_for_each_entry (tmp, &dev->kvm->devices, vm_node)if (tmp->ops == &kvm_vfio_ops)
Signature · 1 line
2.44.0.769.g3c40516874-goog
Re: [PATCH v3 01/19] KVM: Treat the device list as an rculist
Quoted 5 lines · Oliver Upton
A subsequent change to KVM/arm64 will necessitate walking the device list outside of the
kvm->lock. Prepare by converting to an rculist. This has zero effect on the VM destruction path, as it is expected every reader is backed by a reference on the kvm struct.
Quoted 8 lines
On the other hand, ensure a given device is completely destroyed before dropping the
kvm->lockin therelease()path, as certain devices expect to be a singleton (e.g. the vfio-kvm device).Cc Paolo Bonzini <[email protected]>Cc Sean Christopherson <[email protected]>Signed-off-by Oliver Upton <[email protected]>---
PATCH v3 2/19 KVM: arm64: vgic-its: Walk LPI xarray in its_sync_lpi_pending_table()
In reply to Oliver UptonThe new LPI xarray makes it possible to walk the VM's LPIs without holding a lock, meaning that vgic_copy_lpi_list() is no longer necessary. Prepare for the deletion by walking the LPI xarray directly in its_sync_lpi_pending_table().
--- arch/arm64/kvm/vgic/vgic-its.c | 27 ++++++++++----------------- 1 file changed, 10 insertions(+), 17 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +10 -17
@@ -446,23 +446,18 @@ static u32 max_lpis_propbaser(u64 propbaser)static int its_sync_lpi_pending_table (struct kvm_vcpu *vcpu) {gpa_t pendbase =GICR_PENDBASER_ADDRESS (vcpu->arch.vgic_cpu.pendbaser); +struct vgic_dist *dist = &vcpu->kvm->arch.vgic; +unsigned long intid, flags;struct vgic_irq *irq;int last_byte_offset =-1 ;int ret =0 ; -u32 *intids; -int nr_irqs, i; -unsigned long flags;u8 pendmask; - nr_irqs =vgic_copy_lpi_list (vcpu->kvm, vcpu, &intids); -if (nr_irqs <0 ) -return nr_irqs; - -for (i =0 ; i < nr_irqs; i++) { +xa_for_each (&dist->lpi_xa, intid, irq) {int byte_offset, bit_nr; - byte_offset = intids[i] /BITS_PER_BYTE ; - bit_nr = intids[i] %BITS_PER_BYTE ; + byte_offset = intid /BITS_PER_BYTE ; + bit_nr = intid %BITS_PER_BYTE ;/* * For contiguously allocated LPIs chances are we just read
@@ -472,25 +467,23 @@ static int its_sync_lpi_pending_table(struct kvm_vcpu *vcpu) ret =kvm_read_guest_lock (vcpu->kvm, pendbase + byte_offset, &pendmask,1 ); -if (ret) { -kfree (intids); +if (ret)return ret; - } + last_byte_offset = byte_offset; } - irq =vgic_get_irq (vcpu->kvm,NULL , intids[i]); + irq =vgic_get_irq (vcpu->kvm,NULL , intid);if (!irq)continue ;raw_spin_lock_irqsave (&irq->irq_lock, flags); - irq->pending_latch = pendmask & (1U << bit_nr); +if (irq->target_vcpu == vcpu) + irq->pending_latch = pendmask & (1U << bit_nr);vgic_queue_irq_unlock (vcpu->kvm, irq, flags);vgic_put_irq (vcpu->kvm, irq); } -kfree (intids); -return ret; }
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 3/19 KVM: arm64: vgic-its: Walk LPI xarray in vgic_its_invall()
In reply to Oliver UptonThe new LPI xarray makes it possible to walk the VM's LPIs without holding a lock, meaning that vgic_copy_lpi_list() is no longer necessary. Prepare for the deletion by walking the LPI xarray directly in vgic_its_invall().
--- arch/arm64/kvm/vgic/vgic-its.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +6 -10
@@ -1365,23 +1365,19 @@ static int vgic_its_cmd_handle_inv(struct kvm *kvm, struct vgic_its *its,int vgic_its_invall (struct kvm_vcpu *vcpu) {struct kvm *kvm = vcpu->kvm; -int irq_count, i =0 ; -u32 *intids; - - irq_count =vgic_copy_lpi_list (kvm, vcpu, &intids); -if (irq_count <0 ) -return irq_count; +struct vgic_dist *dist = &kvm->arch.vgic; +struct vgic_irq *irq; +unsigned long intid; -for (i =0 ; i < irq_count; i++) { -struct vgic_irq *irq =vgic_get_irq (kvm,NULL , intids[i]); +xa_for_each (&dist->lpi_xa, intid, irq) { + irq =vgic_get_irq (kvm,NULL , intid);if (!irq)continue ; +update_lpi_config (kvm, irq, vcpu, false);vgic_put_irq (kvm, irq); } -kfree (intids); -if (vcpu->arch.vgic_cpu.vgic_v3.its_vpe.its_vm)its_invall_vpe (&vcpu->arch.vgic_cpu.vgic_v3.its_vpe);
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 4/19 KVM: arm64: vgic-its: Walk LPI xarray in vgic_its_cmd_handle_movall()
In reply to Oliver UptonThe new LPI xarray makes it possible to walk the VM's LPIs without holding a lock, meaning that vgic_copy_lpi_list() is no longer necessary. Prepare for the deletion by walking the LPI xarray directly in vgic_its_cmd_handle_movall().
--- arch/arm64/kvm/vgic/vgic-its.c | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +4 -9
@@ -1420,10 +1420,10 @@ static int vgic_its_cmd_handle_invall(struct kvm *kvm, struct vgic_its *its,static int vgic_its_cmd_handle_movall (struct kvm *kvm,struct vgic_its *its,u64 *its_cmd) { +struct vgic_dist *dist = &kvm->arch.vgic;struct kvm_vcpu *vcpu1, *vcpu2;struct vgic_irq *irq; -u32 *intids; -int irq_count, i; +unsigned long intid;/* We advertise GITS_TYPER.PTA==0, making the address the vcpu ID */ vcpu1 =kvm_get_vcpu_by_id (kvm,its_cmd_get_target_addr (its_cmd));
@@ -1435,12 +1435,8 @@ static int vgic_its_cmd_handle_movall(struct kvm *kvm, struct vgic_its *its,if (vcpu1 == vcpu2)return 0 ; - irq_count =vgic_copy_lpi_list (kvm, vcpu1, &intids); -if (irq_count <0 ) -return irq_count; - -for (i =0 ; i < irq_count; i++) { - irq =vgic_get_irq (kvm,NULL , intids[i]); +xa_for_each (&dist->lpi_xa, intid, irq) { + irq =vgic_get_irq (kvm,NULL , intid);if (!irq)continue ;
@@ -1451,7 +1447,6 @@ static int vgic_its_cmd_handle_movall(struct kvm *kvm, struct vgic_its *its,vgic_its_invalidate_cache (kvm); -kfree (intids);return 0 ; }
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 5/19 KVM: arm64: vgic-debug: Use an xarray mark for debug iterator
In reply to Oliver UptonThe vgic debug iterator is the final user of vgic_copy_lpi_list(), but is a bit more complicated to transition to something else. Use a mark in the LPI xarray to record the indices 'known' to the debug iterator. Protect against the LPIs from being freed by associating an additional reference with the xarray mark.
Rework iter_next() to let the xarray walk 'drive' the iteration after visiting all of the SGIs, PPIs, and SPIs.
--- arch/arm64/kvm/vgic/vgic-debug.c | 82 +++++++++++++++++++++++--------- arch/arm64/kvm/vgic/vgic-its.c | 4 +- arch/arm64/kvm/vgic/vgic.h | 1 + include/kvm/arm_vgic.h | 2 + 4 files changed, 64 insertions(+), 25 deletions(-)
arch/arm64/kvm/vgic/vgic-debug.c +60 -22
@@ -28,27 +28,65 @@ struct vgic_state_iter {int nr_lpis;int dist_id;int vcpu_id; -int intid; +unsigned long intid;int lpi_idx; -u32 *lpi_array; }; -static void iter_next (struct vgic_state_iter *iter) +static void iter_next (struct kvm *kvm,struct vgic_state_iter *iter) { +struct vgic_dist *dist = &kvm->arch.vgic; +if (iter->dist_id ==0 ) { iter->dist_id++;return ; } +/* +* Let the xarray drive the iterator after the last SPI, as the iterator +* has exhausted the sequentially-allocated INTID space. +*/ +if (iter->intid >= (iter->nr_spis +VGIC_NR_PRIVATE_IRQS -1 )) { +if (iter->lpi_idx < iter->nr_lpis) +xa_find_after (&dist->lpi_xa, &iter->intid, +VGIC_LPI_MAX_INTID , +LPI_XA_MARK_DEBUG_ITER ); + iter->lpi_idx++; +return ; + } + iter->intid++;if (iter->intid ==VGIC_NR_PRIVATE_IRQS && ++iter->vcpu_id < iter->nr_cpus) iter->intid =0 ; +} -if (iter->intid >= (iter->nr_spis +VGIC_NR_PRIVATE_IRQS )) { -if (iter->lpi_idx < iter->nr_lpis) - iter->intid = iter->lpi_array[iter->lpi_idx]; - iter->lpi_idx++; +static int iter_mark_lpis (struct kvm *kvm) +{ +struct vgic_dist *dist = &kvm->arch.vgic; +struct vgic_irq *irq; +unsigned long intid; +int nr_lpis =0 ; + +xa_for_each (&dist->lpi_xa, intid, irq) { +if (!vgic_try_get_irq_kref (irq)) +continue ; + +xa_set_mark (&dist->lpi_xa, intid,LPI_XA_MARK_DEBUG_ITER ); + nr_lpis++; + } + +return nr_lpis; +} + +static void iter_unmark_lpis (struct kvm *kvm) +{ +struct vgic_dist *dist = &kvm->arch.vgic; +struct vgic_irq *irq; +unsigned long intid; + +xa_for_each (&dist->lpi_xa, intid, irq) { +xa_clear_mark (&dist->lpi_xa, intid,LPI_XA_MARK_DEBUG_ITER ); +vgic_put_irq (kvm, irq); } }
@@ -61,15 +99,12 @@ static void iter_init(struct kvm *kvm, struct vgic_state_iter *iter, iter->nr_cpus = nr_cpus; iter->nr_spis = kvm->arch.vgic.nr_spis; -if (kvm->arch.vgic.vgic_model ==KVM_DEV_TYPE_ARM_VGIC_V3 ) { - iter->nr_lpis =vgic_copy_lpi_list (kvm,NULL , &iter->lpi_array); -if (iter->nr_lpis <0 ) - iter->nr_lpis =0 ; - } +if (kvm->arch.vgic.vgic_model ==KVM_DEV_TYPE_ARM_VGIC_V3 ) + iter->nr_lpis =iter_mark_lpis (kvm);/* Fast forward to the right position if needed */ while (pos--) -iter_next (iter); +iter_next (kvm, iter); }static bool end_of_vgic (struct vgic_state_iter *iter)
@@ -114,7 +149,7 @@ static void *vgic_debug_next(struct seq_file *s, void *v, loff_t *pos)struct vgic_state_iter *iter = kvm->arch.vgic.iter; ++*pos; -iter_next (iter); +iter_next (kvm, iter);if (end_of_vgic (iter)) iter =NULL ;return iter;
@@ -134,13 +169,14 @@ static void vgic_debug_stop(struct seq_file *s, void *v)mutex_lock (&kvm->arch.config_lock); iter = kvm->arch.vgic.iter; -kfree (iter->lpi_array); +iter_unmark_lpis (kvm);kfree (iter); kvm->arch.vgic.iter =NULL ;mutex_unlock (&kvm->arch.config_lock); } -static void print_dist_state (struct seq_file *s,struct vgic_dist *dist) +static void print_dist_state (struct seq_file *s,struct vgic_dist *dist, +struct vgic_state_iter *iter) {bool v3 = dist->vgic_model ==KVM_DEV_TYPE_ARM_VGIC_V3 ;
@@ -149,7 +185,7 @@ static void print_dist_state(struct seq_file *s, struct vgic_dist *dist)seq_printf (s,"vgic_model:\t%s\n" , v3 ?"GICv3" :"GICv2" );seq_printf (s,"nr_spis:\t%d\n" , dist->nr_spis);if (v3) -seq_printf (s,"nr_lpis:\t%d\n" ,atomic_read (&dist->lpi_count)); +seq_printf (s,"nr_lpis:\t%d\n" , iter->nr_lpis);seq_printf (s,"enabled:\t%d\n" , dist->enabled);seq_printf (s,"\n" );
@@ -236,7 +272,7 @@ static int vgic_debug_show(struct seq_file *s, void *v)unsigned long flags;if (iter->dist_id ==0 ) { -print_dist_state (s, &kvm->arch.vgic); +print_dist_state (s, &kvm->arch.vgic, iter);return 0 ; }
@@ -246,11 +282,13 @@ static int vgic_debug_show(struct seq_file *s, void *v)if (iter->vcpu_id < iter->nr_cpus) vcpu =kvm_get_vcpu (kvm, iter->vcpu_id); +/* +* Expect this to succeed, as iter_mark_lpis() takes a reference on +* every LPI to be visited. +*/ irq =vgic_get_irq (kvm, vcpu, iter->intid); -if (!irq) { -seq_printf (s," LPI %4d freed\n" , iter->intid); -return 0 ; - } +if (WARN_ON_ONCE (!irq)) +return -EINVAL ;raw_spin_lock_irqsave (&irq->irq_lock, flags);print_irq_state (s, irq, vcpu);
arch/arm64/kvm/vgic/vgic-its.c +1 -3
@@ -316,8 +316,6 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,return 0 ; } -#define GIC_LPI_MAX_INTID ((1 << INTERRUPT_ID_BITS_ITS) - 1) -/* * Create a snapshot of the current LPIs targeting @vcpu, so that we can * enumerate those LPIs without holding any lock.
@@ -347,7 +345,7 @@ int vgic_copy_lpi_list(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 **intid_ptr)raw_spin_lock_irqsave (&dist->lpi_list_lock, flags);rcu_read_lock (); -xas_for_each (&xas, irq,GIC_LPI_MAX_INTID ) { +xas_for_each (&xas, irq,VGIC_LPI_MAX_INTID ) {if (i == irq_count)break ;/* We don't need to "get" the IRQ, as we hold the list lock. */
arch/arm64/kvm/vgic/vgic.h +1 -0
@@ -16,6 +16,7 @@#define INTERRUPT_ID_BITS_SPIS 10#define INTERRUPT_ID_BITS_ITS 16 +#define VGIC_LPI_MAX_INTID ((1 << INTERRUPT_ID_BITS_ITS) - 1)#define VGIC_PRI_BITS 5#define vgic_irq_is_sgi (intid) ((intid) < VGIC_NR_SGIS)
include/kvm/arm_vgic.h +2 -0
@@ -276,6 +276,8 @@ struct vgic_dist {/* Protects the lpi_list. */ raw_spinlock_t lpi_list_lock; + +#define LPI_XA_MARK_DEBUG_ITER XA_MARK_0struct xarray lpi_xa;atomic_t lpi_count;
Signature · 1 line
2.44.0.769.g3c40516874-goog
Zenghui Yu
reviewerRe: [PATCH v3 05/19] KVM: arm64: vgic-debug: Use an xarray mark for debug iterator
Hi Oliver,
Quoted 43 lines · Oliver Upton
The vgic debug iterator is the final user of
vgic_copy_lpi_list(), but is a bit more complicated to transition to something else. Use a mark in the LPI xarray to record the indices 'known' to the debug iterator. Protect against the LPIs from being freed by associating an additional reference with the xarray mark.Rework
iter_next()to let the xarray walk 'drive' the iteration after visiting all of the SGIs, PPIs, and SPIs.Signed-off-by Oliver Upton <[email protected]>--- arch/arm64/kvm/vgic/vgic-debug.c | 82 +++++++++++++++++++++++--------- arch/arm64/kvm/vgic/vgic-its.c | 4 +- arch/arm64/kvm/vgic/vgic.h | 1 + include/kvm/arm_vgic.h | 2 + 4 files changed, 64 insertions(+), 25 deletions(-)arch/arm64/kvm/vgic/vgic-debug.c +6
-3@@ -28,27 +28,65 @@ struct vgic_state_iter {int nr_lpis;int dist_id;int vcpu_id; -int intid; +unsigned long intid;int lpi_idx; -u32 *lpi_array; }; -static void iter_next (struct vgic_state_iter *iter) +static void iter_next (struct kvm *kvm,struct vgic_state_iter *iter) { +struct vgic_dist *dist = &kvm->arch.vgic; +if (iter->dist_id ==0 ) { iter->dist_id++;return ; } +/* +* Let the xarray drive the iterator after the last SPI, as the iterator
arch/arm64/kvm/vgic/vgic-debug.c struct vgic_state_iter { · line 55
Quoted 10 lines
arch/arm64/kvm/vgic/vgic-debug.c +10
-0+* has exhausted the sequentially-allocated INTID space. +*/ +if (iter->intid >= (iter->nr_spis +VGIC_NR_PRIVATE_IRQS -1 )) { +if (iter->lpi_idx < iter->nr_lpis) +xa_find_after (&dist->lpi_xa, &iter->intid, +VGIC_LPI_MAX_INTID , +LPI_XA_MARK_DEBUG_ITER ); + iter->lpi_idx++; +return ; + }
What's the purpose of moving the LPI handling up here?
arch/arm64/kvm/vgic/vgic-debug.c struct vgic_state_iter { · line 60
Quoted 5 lines
arch/arm64/kvm/vgic/vgic-debug.c +1
-0+ iter->intid++;if (iter->intid ==VGIC_NR_PRIVATE_IRQS && ++iter->vcpu_id < iter->nr_cpus) iter->intid =0 ;
In case the guest *doesn't* have any LPI, we previously relied on the iterator setting
'intid = nr_spis + VGIC_NR_PRIVATE_IRQS' && 'lpi_idx = 1'
to exit the iterator. But it was broken with this refactor -- the intid remains at 'nr_spis + VGIC_NR_PRIVATE_IRQS - 1', and vgic_debug_show() endlessly prints the last SPI's state.
The following diff seems work for me.
arch/arm64/kvm/vgic/vgic-debug.c +6 -6
@@ -41,11 +41,16 @@ static void iter_next(struct kvm *kvm, struct vgic_state_iter *iter)return ; } + iter->intid++; +if (iter->intid ==VGIC_NR_PRIVATE_IRQS && + ++iter->vcpu_id < iter->nr_cpus) + iter->intid =0 ; +/* * Let the xarray drive the iterator after the last SPI, as the iterator * has exhausted the sequentially-allocated INTID space. */ -if (iter->intid >= (iter->nr_spis +VGIC_NR_PRIVATE_IRQS -1 )) { +if (iter->intid >= (iter->nr_spis +VGIC_NR_PRIVATE_IRQS )) {if (iter->lpi_idx < iter->nr_lpis) xa_find_after(&dist->lpi_xa, &iter->intid, VGIC_LPI_MAX_INTID,
@@ -53,11 +58,6 @@ static void iter_next(struct kvm *kvm, struct vgic_state_iter *iter) iter->lpi_idx++;return ; } - - iter->intid++; -if (iter->intid ==VGIC_NR_PRIVATE_IRQS && - ++iter->vcpu_id < iter->nr_cpus) - iter->intid =0 ; }static int iter_mark_lpis (struct kvm *kvm)
Zenghui Yu
reviewerRe: [PATCH v3 05/19] KVM: arm64: vgic-debug: Use an xarray mark for debug iterator
arch/arm64/kvm/vgic/vgic-debug.c static void iter_next(struct kvm *kvm, struct vgic_state_iter *iter) · line 44
Quoted 11 lines · Zenghui Yu
The following diff seems work for me.
arch/arm64/kvm/vgic/vgic-debug.c +1
-0@@ -41,11 +41,16 @@ static void iter_next(struct kvm *kvm, struct vgic_state_iter *iter)return ; } + iter->intid++;
[*]
arch/arm64/kvm/vgic/vgic-debug.c static void iter_next(struct kvm *kvm, struct vgic_state_iter *iter) · line 55
Quoted 12 lines
arch/arm64/kvm/vgic/vgic-debug.c +5
-1+if (iter->intid ==VGIC_NR_PRIVATE_IRQS && + ++iter->vcpu_id < iter->nr_cpus) + iter->intid =0 ; +/* * Let the xarray drive the iterator after the last SPI, as the iterator * has exhausted the sequentially-allocated INTID space. */ -if (iter->intid >= (iter->nr_spis +VGIC_NR_PRIVATE_IRQS -1 )) { +if (iter->intid >= (iter->nr_spis +VGIC_NR_PRIVATE_IRQS )) {if (iter->lpi_idx < iter->nr_lpis) xa_find_after(&dist->lpi_xa, &iter->intid,
Just noticed that it's wrong to increase intid before xa_find_after(), which would break the LPI case. Let me have a think...
Zenghui
Zenghui Yu
reviewerRe: [PATCH v3 05/19] KVM: arm64: vgic-debug: Use an xarray mark for debug iterator
Quoted 15 lines · Zenghui Yu
arch/arm64/kvm/vgic/vgic-debug.c static void iter_next(struct kvm *kvm, struct vgic_state_iter *iter) · line 44
Quoted 11 lines · Zenghui Yu
The following diff seems work for me.
arch/arm64/kvm/vgic/vgic-debug.c +1
-0@@ -41,11 +41,16 @@ static void iter_next(struct kvm *kvm, struct vgic_state_iter *iter)return ; } + iter->intid++;[*]
Quoted 15 lines
arch/arm64/kvm/vgic/vgic-debug.c static void iter_next(struct kvm *kvm, struct vgic_state_iter *iter) · line 55
Quoted 12 lines
arch/arm64/kvm/vgic/vgic-debug.c +5
-1+if (iter->intid ==VGIC_NR_PRIVATE_IRQS && + ++iter->vcpu_id < iter->nr_cpus) + iter->intid =0 ; +/* * Let the xarray drive the iterator after the last SPI, as the iterator * has exhausted the sequentially-allocated INTID space. */ -if (iter->intid >= (iter->nr_spis +VGIC_NR_PRIVATE_IRQS -1 )) { +if (iter->intid >= (iter->nr_spis +VGIC_NR_PRIVATE_IRQS )) {if (iter->lpi_idx < iter->nr_lpis) xa_find_after(&dist->lpi_xa, &iter->intid,Just noticed that it's wrong to increase intid before
xa_find_after(), which would break the LPI case. Let me have a think...
So searching the LPI xarray and populating lpi_idx when the guest doesn't have LPI is pointless. We can fix the reported issue by dealing with the 'nr_lpis == 0' case directly, which might be the easiest approach. Let me know what do you think.
arch/arm64/kvm/vgic/vgic-debug.c +3 -2
@@ -45,7 +45,8 @@ static void iter_next(struct kvm *kvm, struct vgic_state_iter *iter)* Let the xarray drive the iterator after the last SPI, as the iterator * has exhausted the sequentially-allocated INTID space. */ -if (iter->intid >= (iter->nr_spis +VGIC_NR_PRIVATE_IRQS -1 )) { +if (iter->intid >= (iter->nr_spis +VGIC_NR_PRIVATE_IRQS -1 ) && + iter->nr_lpis) {if (iter->lpi_idx < iter->nr_lpis) xa_find_after(&dist->lpi_xa, &iter->intid, VGIC_LPI_MAX_INTID,
@@ -112,7 +113,7 @@ static bool end_of_vgic(struct vgic_state_iter *iter)return iter->dist_id >0 && iter->vcpu_id == iter->nr_cpus && iter->intid >= (iter->nr_spis +VGIC_NR_PRIVATE_IRQS ) && - iter->lpi_idx > iter->nr_lpis; + (iter->lpi_idx > iter->nr_lpis || !iter->nr_lpis); }static void *vgic_debug_start (struct seq_file *s,loff_t *pos)
Thanks,
Zenghui
Re: [PATCH v3 05/19] KVM: arm64: vgic-debug: Use an xarray mark for debug iterator
arch/arm64/kvm/vgic/vgic-debug.c static bool end_of_vgic(struct vgic_state_iter *iter) · line 116
Quoted 6 lines · Zenghui Yu
arch/arm64/kvm/vgic/vgic-debug.c +1
-1@@ -112,7 +113,7 @@ static bool end_of_vgic(struct vgic_state_iter *iter)return iter->dist_id >0 && iter->vcpu_id == iter->nr_cpus && iter->intid >= (iter->nr_spis +VGIC_NR_PRIVATE_IRQS ) && - iter->lpi_idx > iter->nr_lpis; + (iter->lpi_idx > iter->nr_lpis || !iter->nr_lpis);
And this should actually be written as:
iter->lpi_idx >= iter->nr_lpis
even in the first commit adding the LPI status in debugfs (e294cb3a6d1a) if I understand it correctly. I will give it a bit more tests tomorrow..
Marc Zyngier
maintainerRe: [PATCH v3 05/19] KVM: arm64: vgic-debug: Use an xarray mark for debug iterator
Quoted 11 lines · Zenghui Yu
arch/arm64/kvm/vgic/vgic-debug.c static bool end_of_vgic(struct vgic_state_iter *iter) · line 116
Quoted 6 lines · Zenghui Yu
arch/arm64/kvm/vgic/vgic-debug.c +1
-1@@ -112,7 +113,7 @@ static bool end_of_vgic(struct vgic_state_iter *iter)return iter->dist_id >0 && iter->vcpu_id == iter->nr_cpus && iter->intid >= (iter->nr_spis +VGIC_NR_PRIVATE_IRQS ) && - iter->lpi_idx > iter->nr_lpis; + (iter->lpi_idx > iter->nr_lpis || !iter->nr_lpis);And this should actually be written as:
Quoted 4 lines
iter->lpi_idx>=iter->nr_lpiseven in the first commit adding the LPI status in debugfs (
e294cb3a6d1a) if I understand it correctly. I will give it a bit more tests tomorrow..
Yup, this looks like a long-standing bug (/me pleads guilty).
Maybe worth fixing them independently in order to facilitate the inevitable backports?
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Zenghui Yu
reviewerRe: [PATCH v3 05/19] KVM: arm64: vgic-debug: Use an xarray mark for debug iterator
Quoted 15 lines · Marc Zyngier
Quoted 13 lines · Zenghui Yu
arch/arm64/kvm/vgic/vgic-debug.c static bool end_of_vgic(struct vgic_state_iter *iter) · line 116
Quoted 6 lines · Zenghui Yu
arch/arm64/kvm/vgic/vgic-debug.c +1
-1@@ -112,7 +113,7 @@ static bool end_of_vgic(struct vgic_state_iter *iter)return iter->dist_id >0 && iter->vcpu_id == iter->nr_cpus && iter->intid >= (iter->nr_spis +VGIC_NR_PRIVATE_IRQS ) && - iter->lpi_idx > iter->nr_lpis; + (iter->lpi_idx > iter->nr_lpis || !iter->nr_lpis);And this should actually be written as:
iter->lpi_idx>=iter->nr_lpis
Quoted 7 lines
Quoted 2 lines
even in the first commit adding the LPI status in debugfs (
e294cb3a6d1a) if I understand it correctly. I will give it a bit more tests tomorrow..Yup, this looks like a long-standing bug (/me pleads guilty).
Maybe worth fixing them independently in order to facilitate the inevitable backports?
I'm sorry that I misread the code again (shouldn't have sent spam late at night :-( ).
Consider the last LPI:
|vgic_debug_next() {
| iter_next() // get the last valid LPI intid
| end_of_vgic() // lpi_idx == nr_lpis
|}We need to go ahead to print this LPI's state and go through one more vgic_debug_next() to exit the iterator. So there's no problem in the current implementation for LPI, it's just that the code is a bit hard to follow.
I've sent the "easiest approach" [*] out now.
Thanks,
Zenghui
PATCH v3 6/19 KVM: arm64: vgic-its: Get rid of vgic_copy_lpi_list()
In reply to Oliver UptonThe last user has been transitioned to walking the LPI xarray directly. Cut the wart off, and get rid of the now unneeded lpi_count while doing so.
--- arch/arm64/kvm/vgic/vgic-its.c | 48 ---------------------------------- arch/arm64/kvm/vgic/vgic.c | 1 - arch/arm64/kvm/vgic/vgic.h | 1 - include/kvm/arm_vgic.h | 1 - 4 files changed, 51 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +0 -48
@@ -86,11 +86,8 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,if (ret) {xa_release (&dist->lpi_xa, intid);kfree (irq); - goto out_unlock; } -atomic_inc (&dist->lpi_count); - out_unlock:raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags);
@@ -316,51 +313,6 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,return 0 ; } -/* -* Create a snapshot of the current LPIs targeting @vcpu, so that we can -* enumerate those LPIs without holding any lock. -* Returns their number and puts the kmalloc'ed array into intid_ptr. -*/ -int vgic_copy_lpi_list (struct kvm *kvm,struct kvm_vcpu *vcpu,u32 **intid_ptr) -{ -struct vgic_dist *dist = &kvm->arch.vgic; -XA_STATE (xas, &dist->lpi_xa,GIC_LPI_OFFSET ); -struct vgic_irq *irq; -unsigned long flags; -u32 *intids; -int irq_count, i =0 ; - -/* -* There is an obvious race between allocating the array and LPIs -* being mapped/unmapped. If we ended up here as a result of a -* command, we're safe (locks are held, preventing another -* command). If coming from another path (such as enabling LPIs), -* we must be careful not to overrun the array. -*/ - irq_count =atomic_read (&dist->lpi_count); - intids =kmalloc_array (irq_count,sizeof (intids[0 ]),GFP_KERNEL_ACCOUNT ); -if (!intids) -return -ENOMEM ; - -raw_spin_lock_irqsave (&dist->lpi_list_lock, flags); -rcu_read_lock (); - -xas_for_each (&xas, irq,VGIC_LPI_MAX_INTID ) { -if (i == irq_count) -break ; -/* We don't need to "get" the IRQ, as we hold the list lock. */ -if (vcpu && irq->target_vcpu != vcpu) -continue ; - intids[i++] = irq->intid; - } - -rcu_read_unlock (); -raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); - - *intid_ptr = intids; -return i; -} -static int update_affinity (struct vgic_irq *irq,struct kvm_vcpu *vcpu) {int ret =0 ;
arch/arm64/kvm/vgic/vgic.c +0 -1
@@ -126,7 +126,6 @@ void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq)__xa_erase (&dist->lpi_xa, irq->intid);xa_unlock_irqrestore (&dist->lpi_xa, flags); -atomic_dec (&dist->lpi_count);kfree_rcu (irq, rcu); }
arch/arm64/kvm/vgic/vgic.h +0 -1
@@ -331,7 +331,6 @@ static inline bool vgic_dist_overlap(struct kvm *kvm, gpa_t base, size_t size) }bool vgic_lpis_enabled (struct kvm_vcpu *vcpu); -int vgic_copy_lpi_list (struct kvm *kvm,struct kvm_vcpu *vcpu,u32 **intid_ptr);int vgic_its_resolve_lpi (struct kvm *kvm,struct vgic_its *its,u32 devid,u32 eventid,struct vgic_irq **irq);struct vgic_its *vgic_msi_to_its (struct kvm *kvm,struct kvm_msi *msi);
include/kvm/arm_vgic.h +0 -1
@@ -279,7 +279,6 @@ struct vgic_dist {#define LPI_XA_MARK_DEBUG_ITER XA_MARK_0struct xarray lpi_xa; -atomic_t lpi_count;/* LPI translation cache */ struct list_head lpi_translation_cache;
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 7/19 KVM: arm64: vgic-its: Scope translation cache invalidations to an ITS
In reply to Oliver UptonAs the current LPI translation cache is global, the corresponding invalidation helpers are also globally-scoped. In anticipation of constructing a translation cache per ITS, add a helper for scoped cache invalidations.
We still need to support global invalidations when LPIs are toggled on a redistributor, as a property of the translation cache is that all stored LPIs are known to be delieverable.
--- arch/arm64/kvm/vgic/vgic-its.c | 46 ++++++++++++++++++++++-------- arch/arm64/kvm/vgic/vgic-mmio-v3.c | 2 +- arch/arm64/kvm/vgic/vgic.h | 2 +- 3 files changed, 36 insertions(+), 14 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +34 -12
@@ -23,6 +23,8 @@#include "vgic.h" #include "vgic-mmio.h" +static struct kvm_device_ops kvm_arm_vgic_its_ops; +static int vgic_its_save_tables_v0 (struct vgic_its *its);static int vgic_its_restore_tables_v0 (struct vgic_its *its);static int vgic_its_commit_v0 (struct vgic_its *its);
@@ -616,8 +618,9 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); } -void vgic_its_invalidate_cache (struct kvm *kvm) +static void vgic_its_invalidate_cache (struct vgic_its *its) { +struct kvm *kvm = its->dev->kvm;struct vgic_dist *dist = &kvm->arch.vgic;struct vgic_translation_cache_entry *cte;unsigned long flags;
@@ -639,6 +642,24 @@ void vgic_its_invalidate_cache(struct kvm *kvm)raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); } +void vgic_its_invalidate_all_caches (struct kvm *kvm) +{ +struct kvm_device *dev; +struct vgic_its *its; + +rcu_read_lock (); + +list_for_each_entry_rcu (dev, &kvm->devices, vm_node) { +if (dev->ops != &kvm_arm_vgic_its_ops) +continue ; + + its = dev->private; +vgic_its_invalidate_cache (its); + } + +rcu_read_unlock (); +} +int vgic_its_resolve_lpi (struct kvm *kvm,struct vgic_its *its,u32 devid,u32 eventid,struct vgic_irq **irq) {
@@ -826,7 +847,7 @@ static int vgic_its_cmd_handle_discard(struct kvm *kvm, struct vgic_its *its,* don't bother here since we clear the ITTE anyway and the * pending state is a property of the ITTE struct. */ -vgic_its_invalidate_cache (kvm); +vgic_its_invalidate_cache (its);its_free_ite (kvm, ite);return 0 ;
@@ -863,7 +884,7 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its, ite->collection = collection; vcpu =collection_to_vcpu (kvm, collection); -vgic_its_invalidate_cache (kvm); +vgic_its_invalidate_cache (its);return update_affinity (ite->irq, vcpu); }
@@ -1110,7 +1131,8 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its, }/* Requires the its_lock to be held. */ -static void vgic_its_free_device (struct kvm *kvm,struct its_device *device) +static void vgic_its_free_device (struct kvm *kvm,struct vgic_its *its, +struct its_device *device) {struct its_ite *ite, *temp;
@@ -1122,7 +1144,7 @@ static void vgic_its_free_device(struct kvm *kvm, struct its_device *device)list_for_each_entry_safe (ite, temp, &device->itt_head, ite_list)its_free_ite (kvm, ite); -vgic_its_invalidate_cache (kvm); +vgic_its_invalidate_cache (its);list_del (&device->dev_list);kfree (device);
@@ -1134,7 +1156,7 @@ static void vgic_its_free_device_list(struct kvm *kvm, struct vgic_its *its)struct its_device *cur, *temp;list_for_each_entry_safe (cur, temp, &its->device_list, dev_list) -vgic_its_free_device (kvm, cur); +vgic_its_free_device (kvm, its, cur); }/* its lock must be held */
@@ -1193,7 +1215,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,* by removing the mapping and re-establishing it. */ if (device) -vgic_its_free_device (kvm, device); +vgic_its_free_device (kvm, its, device);/* * The spec does not say whether unmapping a not-mapped device
@@ -1224,7 +1246,7 @@ static int vgic_its_cmd_handle_mapc(struct kvm *kvm, struct vgic_its *its,if (!valid) {vgic_its_free_collection (its, coll_id); -vgic_its_invalidate_cache (kvm); +vgic_its_invalidate_cache (its); }else {struct kvm_vcpu *vcpu;
@@ -1395,7 +1417,7 @@ static int vgic_its_cmd_handle_movall(struct kvm *kvm, struct vgic_its *its,vgic_put_irq (kvm, irq); } -vgic_its_invalidate_cache (kvm); +vgic_its_invalidate_cache (its);return 0 ; }
@@ -1747,7 +1769,7 @@ static void vgic_mmio_write_its_ctlr(struct kvm *kvm, struct vgic_its *its, its->enabled = !!(val &GITS_CTLR_ENABLE );if (!its->enabled) -vgic_its_invalidate_cache (kvm); +vgic_its_invalidate_cache (its);/* * Try to process any pending commands. This function bails out early
@@ -1880,7 +1902,7 @@ void vgic_lpi_translation_cache_destroy(struct kvm *kvm)struct vgic_dist *dist = &kvm->arch.vgic;struct vgic_translation_cache_entry *cte, *tmp; -vgic_its_invalidate_cache (kvm); +vgic_its_invalidate_all_caches (kvm);list_for_each_entry_safe (cte, tmp, &dist->lpi_translation_cache, entry) {
@@ -2372,7 +2394,7 @@ static int vgic_its_restore_dte(struct vgic_its *its, u32 id, ret =vgic_its_restore_itt (its, dev);if (ret) { -vgic_its_free_device (its->dev->kvm, dev); +vgic_its_free_device (its->dev->kvm, its, dev);return ret; }
arch/arm64/kvm/vgic/vgic-mmio-v3.c +1 -1
@@ -277,7 +277,7 @@ static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu,return ;vgic_flush_pending_lpis (vcpu); -vgic_its_invalidate_cache (vcpu->kvm); +vgic_its_invalidate_all_caches (vcpu->kvm);atomic_set_release (&vgic_cpu->ctlr,0 ); }else { ctlr = atomic_cmpxchg_acquire(&vgic_cpu->ctlr, 0,
arch/arm64/kvm/vgic/vgic.h +1 -1
@@ -337,7 +337,7 @@ struct vgic_its *vgic_msi_to_its(struct kvm *kvm, struct kvm_msi *msi);int vgic_its_inject_cached_translation (struct kvm *kvm,struct kvm_msi *msi);void vgic_lpi_translation_cache_init (struct kvm *kvm);void vgic_lpi_translation_cache_destroy (struct kvm *kvm); -void vgic_its_invalidate_cache (struct kvm *kvm); +void vgic_its_invalidate_all_caches (struct kvm *kvm);/* GICv4.1 MMIO interface */ int vgic_its_inv_lpi (struct kvm *kvm,struct vgic_irq *irq);
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 8/19 KVM: arm64: vgic-its: Maintain a translation cache per ITS
In reply to Oliver UptonWithin the context of a single ITS, it is possible to use an xarray to cache the device ID & event ID translation to a particular irq descriptor. Take advantage of this to build a translation cache capable of fitting all valid translations for a given ITS.
--- arch/arm64/kvm/vgic/vgic-its.c | 37 +++++++++++++++++++++++++++++++++- include/kvm/arm_vgic.h | 6 ++++++ 2 files changed, 42 insertions(+), 1 deletion(-)
arch/arm64/kvm/vgic/vgic-its.c +36 -1
@@ -511,6 +511,11 @@ static unsigned long vgic_mmio_read_its_idregs(struct kvm *kvm,return 0 ; } +static unsigned long vgic_its_cache_key (u32 devid,u32 eventid) +{ +return (((unsigned long )devid) <<VITS_TYPER_IDBITS ) | eventid; +} +static struct vgic_irq *__vgic_its_check_cache (struct vgic_dist *dist,phys_addr_t db,u32 devid,u32 eventid)
@@ -564,8 +569,10 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, u32 devid, u32 eventid, struct vgic_irq *irq) { + unsigned long cache_key = vgic_its_cache_key(devid, eventid);struct vgic_dist *dist = &kvm->arch.vgic;struct vgic_translation_cache_entry *cte; +struct vgic_irq *old;unsigned long flags;phys_addr_t db;
@@ -604,6 +611,15 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,* its_lock, as the ITE (and the reference it holds) cannot be freed. */ lockdep_assert_held (&its->its_lock); + +/* +* Yes, two references are necessary at the moment: +* - One for the global LPI translation cache +* - Another for the translation cache belonging to @its +* +* This will soon disappear. +*/ +vgic_get_irq_kref (irq);vgic_get_irq_kref (irq); cte->db = db;
@@ -613,6 +629,16 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,/* Move the new translation to the head of the list */ list_move (&cte->entry, &dist->lpi_translation_cache); +raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); + +/* +* The per-ITS cache is a perfect cache, so it may already have an +* identical translation even if it were missing from the global +* cache. Ensure we don't leak a reference if that is the case. +*/ + old =xa_store (&its->translation_cache, cache_key, irq,GFP_KERNEL_ACCOUNT ); +if (old) +vgic_put_irq (kvm, old); out:raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags);
@@ -623,7 +649,8 @@ static void vgic_its_invalidate_cache(struct vgic_its *its)struct kvm *kvm = its->dev->kvm;struct vgic_dist *dist = &kvm->arch.vgic;struct vgic_translation_cache_entry *cte; -unsigned long flags; +unsigned long flags, idx; +struct vgic_irq *irq;raw_spin_lock_irqsave (&dist->lpi_list_lock, flags);
@@ -640,6 +667,11 @@ static void vgic_its_invalidate_cache(struct vgic_its *its) }raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); + +xa_for_each (&its->translation_cache, idx, irq) { +xa_erase (&its->translation_cache, idx); +vgic_put_irq (kvm, irq); + } }void vgic_its_invalidate_all_caches (struct kvm *kvm)
@@ -1962,6 +1994,7 @@ static int vgic_its_create(struct kvm_device *dev, u32 type)INIT_LIST_HEAD (&its->device_list);INIT_LIST_HEAD (&its->collection_list); +xa_init (&its->translation_cache); dev->kvm->arch.vgic.msis_require_devid = true; dev->kvm->arch.vgic.has_its = true;
@@ -1992,6 +2025,8 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)vgic_its_free_device_list (kvm, its);vgic_its_free_collection_list (kvm, its); +vgic_its_invalidate_cache (its); +xa_destroy (&its->translation_cache);mutex_unlock (&its->its_lock);kfree (its);
include/kvm/arm_vgic.h +6 -0
@@ -210,6 +210,12 @@ struct vgic_its {struct mutex its_lock;struct list_head device_list;struct list_head collection_list; + +/* +* Caches the (device_id, event_id) -> vgic_irq translation for +* LPIs that are mapped and enabled. +*/ +struct xarray translation_cache; };struct vgic_state_iter ;
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 9/19 KVM: arm64: vgic-its: Spin off helper for finding ITS by doorbell addr
In reply to Oliver UptonThe fast path will soon need to find an ITS by doorbell address, as the translation caches will become local to an ITS. Spin off a helper to do just that.
--- arch/arm64/kvm/vgic/vgic-its.c | 35 ++++++++++++++++++++-------------- 1 file changed, 21 insertions(+), 14 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +21 -14
@@ -511,9 +511,29 @@ static unsigned long vgic_mmio_read_its_idregs(struct kvm *kvm,return 0 ; } +static struct vgic_its *__vgic_doorbell_to_its (struct kvm *kvm,gpa_t db) +{ +struct kvm_io_device *kvm_io_dev; +struct vgic_io_device *iodev; + + kvm_io_dev =kvm_io_bus_get_dev (kvm,KVM_MMIO_BUS , db); +if (!kvm_io_dev) +return ERR_PTR (-EINVAL ); + +if (kvm_io_dev->ops != &kvm_io_gic_ops) +return ERR_PTR (-EINVAL ); + + iodev = container_of(kvm_io_dev, struct vgic_io_device, dev); +if (iodev->iodev_type !=IODEV_ITS ) +return ERR_PTR (-EINVAL ); + +return iodev->its; +} +static unsigned long vgic_its_cache_key (u32 devid,u32 eventid) {return (((unsigned long )devid) <<VITS_TYPER_IDBITS ) | eventid; + } static struct vgic_irq *__vgic_its_check_cache(struct vgic_dist *dist,
@@ -721,8 +741,6 @@ int vgic_its_resolve_lpi(struct kvm *kvm, struct vgic_its *its,struct vgic_its *vgic_msi_to_its (struct kvm *kvm,struct kvm_msi *msi) {u64 address; -struct kvm_io_device *kvm_io_dev; -struct vgic_io_device *iodev;if (!vgic_has_its (kvm))return ERR_PTR (-ENODEV );
@@ -732,18 +750,7 @@ struct vgic_its *vgic_msi_to_its(struct kvm *kvm, struct kvm_msi *msi) address = (u64 )msi->address_hi <<32 | msi->address_lo; - kvm_io_dev =kvm_io_bus_get_dev (kvm,KVM_MMIO_BUS , address); -if (!kvm_io_dev) -return ERR_PTR (-EINVAL ); - -if (kvm_io_dev->ops != &kvm_io_gic_ops) -return ERR_PTR (-EINVAL ); - - iodev = container_of(kvm_io_dev, struct vgic_io_device, dev); -if (iodev->iodev_type !=IODEV_ITS ) -return ERR_PTR (-EINVAL ); - -return iodev->its; +return __vgic_doorbell_to_its (kvm, address); }/*
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 10/19 KVM: arm64: vgic-its: Use the per-ITS translation cache for injection
In reply to Oliver UptonEverything is in place to switch to per-ITS translation caches. Start using the per-ITS cache to avoid the lock serialization related to the global translation cache. Explicitly check for out-of-range device and event IDs as the cache index is packed based on the range the ITS actually supports.
Take the RCU read lock to protect against the returned descriptor being freed while trying to take a reference on it, as it is no longer necessary to acquire the lpi_list_lock.
--- arch/arm64/kvm/vgic/vgic-its.c | 63 +++++++++------------------------- 1 file changed, 17 insertions(+), 46 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +17 -46
@@ -251,8 +251,10 @@ static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,#define GIC_LPI_OFFSET 8192 -#define VITS_TYPER_IDBITS 16 -#define VITS_TYPER_DEVBITS 16 +#define VITS_TYPER_IDBITS 16 +#define VITS_MAX_EVENTID (BIT(VITS_TYPER_IDBITS) - 1) +#define VITS_TYPER_DEVBITS 16 +#define VITS_MAX_DEVID (BIT(VITS_TYPER_DEVBITS) - 1)#define VITS_DTE_MAX_DEVID_OFFSET (BIT(14) - 1)#define VITS_ITE_MAX_EVENTID_OFFSET (BIT(16) - 1)
@@ -536,51 +538,27 @@ static unsigned long vgic_its_cache_key(u32 devid, u32 eventid) } -static struct vgic_irq *__vgic_its_check_cache (struct vgic_dist *dist, -phys_addr_t db, -u32 devid,u32 eventid) -{ -struct vgic_translation_cache_entry *cte; - -list_for_each_entry (cte, &dist->lpi_translation_cache, entry) { -/* -* If we hit a NULL entry, there is nothing after this -* point. -*/ -if (!cte->irq) -break ; - -if (cte->db != db || cte->devid != devid || - cte->eventid != eventid) -continue ; - -/* -* Move this entry to the head, as it is the most -* recently used. -*/ -if (!list_is_first (&cte->entry, &dist->lpi_translation_cache)) -list_move (&cte->entry, &dist->lpi_translation_cache); - -return cte->irq; - } - -return NULL ; -} -static struct vgic_irq *vgic_its_check_cache (struct kvm *kvm,phys_addr_t db,u32 devid,u32 eventid) { -struct vgic_dist *dist = &kvm->arch.vgic; +unsigned long cache_key =vgic_its_cache_key (devid, eventid); +struct vgic_its *its;struct vgic_irq *irq; -unsigned long flags; -raw_spin_lock_irqsave (&dist->lpi_list_lock, flags); +if (devid >VITS_MAX_DEVID || eventid >VITS_MAX_EVENTID ) +return NULL ; + + its =__vgic_doorbell_to_its (kvm, db); +if (IS_ERR (its)) +return NULL ; - irq =__vgic_its_check_cache (dist, db, devid, eventid); +rcu_read_lock (); + + irq =xa_load (&its->translation_cache, cache_key);if (!vgic_try_get_irq_kref (irq)) irq =NULL ; -raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); +rcu_read_unlock ();return irq; }
@@ -605,14 +583,7 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its,if (unlikely (list_empty (&dist->lpi_translation_cache))) goto out; -/* -* We could have raced with another CPU caching the same -* translation behind our back, so let's check it is not in -* already -*/ db = its->vgic_its_base +GITS_TRANSLATER ; -if (__vgic_its_check_cache (dist, db, devid, eventid)) - goto out;/* Always reuse the last entry (LRU policy) */ cte = list_last_entry(&dist->lpi_translation_cache,
@@ -958,7 +929,7 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id,switch (type) {case GITS_BASER_TYPE_DEVICE : -if (id >=BIT_ULL (VITS_TYPER_DEVBITS )) +if (id >VITS_MAX_DEVID )return false;break ;case GITS_BASER_TYPE_COLLECTION :
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 11/19 KVM: arm64: vgic-its: Rip out the global translation cache
In reply to Oliver UptonThe MSI injection fast path has been transitioned away from the global translation cache. Rip it out.
--- arch/arm64/kvm/vgic/vgic-init.c | 7 -- arch/arm64/kvm/vgic/vgic-its.c | 121 ++------------------------------ arch/arm64/kvm/vgic/vgic.h | 2 - include/kvm/arm_vgic.h | 3 - 4 files changed, 4 insertions(+), 129 deletions(-)
arch/arm64/kvm/vgic/vgic-init.c +0 -7
@@ -53,7 +53,6 @@ void kvm_vgic_early_init(struct kvm *kvm) {struct vgic_dist *dist = &kvm->arch.vgic; -INIT_LIST_HEAD (&dist->lpi_translation_cache);raw_spin_lock_init (&dist->lpi_list_lock);xa_init_flags (&dist->lpi_xa,XA_FLAGS_LOCK_IRQ ); }
@@ -305,9 +304,6 @@ int vgic_init(struct kvm *kvm) } } -if (vgic_has_its (kvm)) -vgic_lpi_translation_cache_init (kvm); -/* * If we have GICv4.1 enabled, unconditionally request enable the * v4 support so that we get HW-accelerated vSGIs. Otherwise, only
@@ -361,9 +357,6 @@ static void kvm_vgic_dist_destroy(struct kvm *kvm) dist->vgic_cpu_base =VGIC_ADDR_UNDEF ; } -if (vgic_has_its (kvm)) -vgic_lpi_translation_cache_destroy (kvm); -if (vgic_supports_direct_msis (kvm))vgic_v4_teardown (kvm);
arch/arm64/kvm/vgic/vgic-its.c +4 -117
169 lines not on this page · raw message
arch/arm64/kvm/vgic/vgic.h +0 -2
@@ -335,8 +335,6 @@ int vgic_its_resolve_lpi(struct kvm *kvm, struct vgic_its *its, u32 devid, u32 eventid, struct vgic_irq **irq);struct vgic_its *vgic_msi_to_its (struct kvm *kvm,struct kvm_msi *msi);int vgic_its_inject_cached_translation (struct kvm *kvm,struct kvm_msi *msi); -void vgic_lpi_translation_cache_init (struct kvm *kvm); -void vgic_lpi_translation_cache_destroy (struct kvm *kvm);void vgic_its_invalidate_all_caches (struct kvm *kvm);/* GICv4.1 MMIO interface */
include/kvm/arm_vgic.h +0 -3
@@ -286,9 +286,6 @@ struct vgic_dist {#define LPI_XA_MARK_DEBUG_ITER XA_MARK_0struct xarray lpi_xa; -/* LPI translation cache */ -struct list_head lpi_translation_cache; -/* used by vgic-debug */ struct vgic_state_iter *iter;
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 12/19 KVM: arm64: vgic-its: Get rid of the lpi_list_lock
In reply to Oliver UptonThe last genuine use case for the lpi_list_lock was the global LPI translation cache, which has been removed in favor of a per-ITS xarray. Remove a layer from the locking puzzle by getting rid of it.
vgic_add_lpi() still has a critical section that needs to protect against the insertion of other LPIs; change it to take the LPI xarray's xa_lock to retain this property.
--- arch/arm64/kvm/vgic/vgic-init.c | 1 - arch/arm64/kvm/vgic/vgic-its.c | 6 +++--- arch/arm64/kvm/vgic/vgic.c | 5 ++--- include/kvm/arm_vgic.h | 3 --- 4 files changed, 5 insertions(+), 10 deletions(-)
arch/arm64/kvm/vgic/vgic-init.c +0 -1
@@ -53,7 +53,6 @@ void kvm_vgic_early_init(struct kvm *kvm) {struct vgic_dist *dist = &kvm->arch.vgic; -raw_spin_lock_init (&dist->lpi_list_lock);xa_init_flags (&dist->lpi_xa,XA_FLAGS_LOCK_IRQ ); }
arch/arm64/kvm/vgic/vgic-its.c +3 -3
@@ -69,7 +69,7 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, irq->target_vcpu = vcpu; irq->group =1 ; -raw_spin_lock_irqsave (&dist->lpi_list_lock, flags); +xa_lock_irqsave (&dist->lpi_xa, flags);/* * There could be a race with another vgic_add_lpi(), so we need to
@@ -84,14 +84,14 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid, goto out_unlock; } - ret =xa_err (xa_store (&dist->lpi_xa, intid, irq,0 )); + ret =xa_err (__xa_store (&dist->lpi_xa, intid, irq,0 ));if (ret) {xa_release (&dist->lpi_xa, intid);kfree (irq); } out_unlock: -raw_spin_unlock_irqrestore (&dist->lpi_list_lock, flags); +xa_unlock_irqrestore (&dist->lpi_xa, flags);if (ret)return ERR_PTR (ret);
arch/arm64/kvm/vgic/vgic.c +2 -3
@@ -29,9 +29,8 @@ struct vgic_global kvm_vgic_global_state __ro_after_init = {* its->cmd_lock (mutex) * its->its_lock (mutex) * vgic_cpu->ap_list_lock must be taken with IRQs disabled -* kvm->lpi_list_lock must be taken with IRQs disabled -* vgic_dist->lpi_xa.xa_lock must be taken with IRQs disabled -* vgic_irq->irq_lock must be taken with IRQs disabled +* vgic_dist->lpi_xa.xa_lock must be taken with IRQs disabled +* vgic_irq->irq_lock must be taken with IRQs disabled * * As the ap_list_lock might be taken from the timer interrupt handler, * we have to disable IRQs before taking this lock and everything lower
include/kvm/arm_vgic.h +0 -3
@@ -280,9 +280,6 @@ struct vgic_dist {*/ u64 propbaser; -/* Protects the lpi_list. */ -raw_spinlock_t lpi_list_lock; -#define LPI_XA_MARK_DEBUG_ITER XA_MARK_0struct xarray lpi_xa;
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 13/19 KVM: selftests: Align with kernel's GIC definitions
In reply to Oliver UptonThere are a few subtle incongruencies between the GIC definitions used by the kernel and selftests. Furthermore, the selftests header blends implementation detail (e.g. default priority) with the architectural definitions.
This is all rather annoying, since bulk imports of the kernel header is not possible. Move selftests-specific definitions out of the offending header and realign tests on the canonical definitions for things like sysregs. Finally, haul in a fresh copy of the gicv3 header to enable a forthcoming ITS selftest.
--- .../testing/selftests/kvm/aarch64/vgic_irq.c | 4 +- .../selftests/kvm/include/aarch64/gic_v3.h | 586 +++++++++++++++++- .../selftests/kvm/lib/aarch64/gic_v3.c | 13 +- 3 files changed, 568 insertions(+), 35 deletions(-)
tools/testing/selftests/kvm/aarch64/vgic_irq.c +2 -2
@@ -152,7 +152,7 @@ static void reset_stats(void)static uint64_t gic_read_ap1r0 (void ) { -uint64_t reg =read_sysreg_s (SYS_ICV_AP1R0_EL1 ); +uint64_t reg =read_sysreg_s (SYS_ICC_AP1R0_EL1 );dsb (sy);return reg;
@@ -160,7 +160,7 @@ static uint64_t gic_read_ap1r0(void)static void gic_write_ap1r0 (uint64_t val) { -write_sysreg_s (val,SYS_ICV_AP1R0_EL1 ); +write_sysreg_s (val,SYS_ICC_AP1R0_EL1 );isb (); }
tools/testing/selftests/kvm/include/aarch64/gic_v3.h +554 -32
636 lines not on this page · raw message
tools/testing/selftests/kvm/lib/aarch64/gic_v3.c +12 -1
@@ -9,9 +9,20 @@#include "processor.h" #include "delay.h" +#include "gic.h" #include "gic_v3.h" #include "gic_private.h" +#define GICV3_MAX_CPUS 512 + +#define GICD_INT_DEF_PRI 0xa0 +#define GICD_INT_DEF_PRI_X4 ((GICD_INT_DEF_PRI << 24) |\ + (GICD_INT_DEF_PRI << 16) |\ + (GICD_INT_DEF_PRI << 8) |\ + GICD_INT_DEF_PRI) + +#define ICC_PMR_DEF_PRIO 0xf0 +struct gicv3_data {void *dist_base;void *redist_base[GICV3_MAX_CPUS ];
@@ -320,7 +331,7 @@ static void gicv3_cpu_init(unsigned int cpu, void *redist_base)write_sysreg_s (ICC_PMR_DEF_PRIO ,SYS_ICC_PMR_EL1 );/* Enable non-secure Group-1 interrupts */ -write_sysreg_s (ICC_IGRPEN1_EL1_ENABLE ,SYS_ICC_GRPEN1_EL1 ); +write_sysreg_s (ICC_IGRPEN1_EL1_MASK ,SYS_ICC_IGRPEN1_EL1 ); gicv3_data.redist_base[cpu] = redist_base_cpu; }
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 14/19 KVM: selftests: Standardise layout of GIC frames
In reply to Oliver UptonIt would appear that all of the selftests are using the same exact layout for the GIC frames. Fold this back into the library implementation to avoid defining magic values all over the selftests.
This is an extension of Colton's change, ripping out parameterization of from the library internals in addition to the public interfaces.
--- .../selftests/kvm/aarch64/arch_timer.c | 8 +-- .../testing/selftests/kvm/aarch64/vgic_irq.c | 11 +--- .../kvm/aarch64/vpmu_counter_access.c | 6 +- .../selftests/kvm/dirty_log_perf_test.c | 5 +- .../selftests/kvm/include/aarch64/gic.h | 12 +++- .../selftests/kvm/include/aarch64/vgic.h | 3 +- tools/testing/selftests/kvm/lib/aarch64/gic.c | 18 +++--- .../selftests/kvm/lib/aarch64/gic_private.h | 4 +- .../selftests/kvm/lib/aarch64/gic_v3.c | 62 +++++++++---------- .../testing/selftests/kvm/lib/aarch64/vgic.c | 18 +++--- 10 files changed, 62 insertions(+), 85 deletions(-)
tools/testing/selftests/kvm/aarch64/arch_timer.c +2 -6
@@ -14,9 +14,6 @@#include "timer_test.h" #include "vgic.h" -#define GICD_BASE_GPA 0x8000000ULL -#define GICR_BASE_GPA 0x80A0000ULL -enum guest_stage {GUEST_STAGE_VTIMER_CVAL =1 ,GUEST_STAGE_VTIMER_TVAL ,
@@ -149,8 +146,7 @@ static void guest_code(void)local_irq_disable (); -gic_init (GIC_V3 , test_args.nr_vcpus, - (void *)GICD_BASE_GPA , (void *)GICR_BASE_GPA ); +gic_init (GIC_V3 , test_args.nr_vcpus);timer_set_ctl (VIRTUAL ,CTL_IMASK );timer_set_ctl (PHYSICAL ,CTL_IMASK );
@@ -209,7 +205,7 @@ struct kvm_vm *test_vm_create(void)vcpu_init_descriptor_tables (vcpus[i]);test_init_timer_irq (vm); - gic_fd =vgic_v3_setup (vm, nr_vcpus,64 ,GICD_BASE_GPA ,GICR_BASE_GPA ); + gic_fd =vgic_v3_setup (vm, nr_vcpus,64 );__TEST_REQUIRE (gic_fd >=0 ,"Failed to create vgic-v3" );/* Make all the test's cmdline args visible to the guest */
tools/testing/selftests/kvm/aarch64/vgic_irq.c +2 -9
@@ -19,9 +19,6 @@#include "gic_v3.h" #include "vgic.h" -#define GICD_BASE_GPA 0x08000000ULL -#define GICR_BASE_GPA 0x080A0000ULL -/* * Stores the user specified args; it's passed to the guest and to every test * function.
@@ -49,9 +46,6 @@ struct test_args {#define IRQ_DEFAULT_PRIO (LOWEST_PRIO - 1)#define IRQ_DEFAULT_PRIO_REG (IRQ_DEFAULT_PRIO << KVM_PRIO_SHIFT)/* 0xf0 */ -static void *dist = (void *)GICD_BASE_GPA ; -static void *redist = (void *)GICR_BASE_GPA ; -/* * The kvm_inject_* utilities are used by the guest to ask the host to inject * interrupts (e.g., using the KVM_IRQ_LINE ioctl).
@@ -478,7 +472,7 @@ static void guest_code(struct test_args *args)bool level_sensitive = args->level_sensitive;struct kvm_inject_desc *f, *inject_fns; -gic_init (GIC_V3 ,1 , dist, redist); +gic_init (GIC_V3 ,1 );for (i =0 ; i < nr_irqs; i++)gic_irq_enable (i);
@@ -764,8 +758,7 @@ static void test_vgic(uint32_t nr_irqs, bool level_sensitive, bool eoi_split)memcpy (addr_gva2hva (vm, args_gva), &args,sizeof (args));vcpu_args_set (vcpu,1 , args_gva); - gic_fd =vgic_v3_setup (vm,1 , nr_irqs, -GICD_BASE_GPA ,GICR_BASE_GPA ); + gic_fd =vgic_v3_setup (vm,1 , nr_irqs);__TEST_REQUIRE (gic_fd >=0 ,"Failed to create vgic-v3, skipping" ); vm_install_exception_handler(vm, VECTOR_IRQ_CURRENT,
tools/testing/selftests/kvm/aarch64/vpmu_counter_access.c +1 -5
@@ -404,9 +404,6 @@ static void guest_code(uint64_t expected_pmcr_n)GUEST_DONE (); } -#define GICD_BASE_GPA 0x8000000ULL -#define GICR_BASE_GPA 0x80A0000ULL -/* Create a VM that has one vCPU with PMUv3 configured. */ static void create_vpmu_vm (void *guest_code) {
@@ -438,8 +435,7 @@ static void create_vpmu_vm(void *guest_code) init.features[0 ] |= (1 <<KVM_ARM_VCPU_PMU_V3 ); vpmu_vm.vcpu =aarch64_vcpu_add (vpmu_vm.vm,0 , &init, guest_code);vcpu_init_descriptor_tables (vpmu_vm.vcpu); - vpmu_vm.gic_fd =vgic_v3_setup (vpmu_vm.vm,1 ,64 , -GICD_BASE_GPA ,GICR_BASE_GPA ); + vpmu_vm.gic_fd =vgic_v3_setup (vpmu_vm.vm,1 ,64 );__TEST_REQUIRE (vpmu_vm.gic_fd >=0 ,"Failed to create vgic-v3, skipping" );
tools/testing/selftests/kvm/dirty_log_perf_test.c +1 -4
@@ -22,9 +22,6 @@#ifdef __aarch64__#include "aarch64/vgic.h" -#define GICD_BASE_GPA 0x8000000ULL -#define GICR_BASE_GPA 0x80A0000ULL -static int gic_fd;static void arch_setup_vm (struct kvm_vm *vm,unsigned int nr_vcpus)
@@ -33,7 +30,7 @@ static void arch_setup_vm(struct kvm_vm *vm, unsigned int nr_vcpus)* The test can still run even if hardware does not support GICv3, as it * is only an optimization to reduce guest exits. */ - gic_fd =vgic_v3_setup (vm, nr_vcpus,64 ,GICD_BASE_GPA ,GICR_BASE_GPA ); + gic_fd =vgic_v3_setup (vm, nr_vcpus,64 ); }static void arch_cleanup_vm (struct kvm_vm *vm)
tools/testing/selftests/kvm/include/aarch64/gic.h +10 -2
@@ -6,11 +6,20 @@#ifndef SELFTEST_KVM_GIC_H #define SELFTEST_KVM_GIC_H +#include <asm/kvm.h> +enum gic_type {GIC_V3 ,GIC_TYPE_MAX , }; +#define GICD_BASE_GPA 0x8000000ULL +#define GICR_BASE_GPA (GICD_BASE_GPA + KVM_VGIC_V3_DIST_SIZE) + +/* The GIC is identity-mapped into the guest at the time of setup. */ +#define GICD_BASE_GVA ((volatile void *)GICD_BASE_GPA) +#define GICR_BASE_GVA ((volatile void *)GICR_BASE_GPA) +#define MIN_SGI 0#define MIN_PPI 16#define MIN_SPI 32
@@ -21,8 +30,7 @@ enum gic_type {#define INTID_IS_PPI (intid) (MIN_PPI <= (intid) && (intid) < MIN_SPI)#define INTID_IS_SPI (intid) (MIN_SPI <= (intid) && (intid) <= MAX_SPI) -void gic_init (enum gic_type type,unsigned int nr_cpus, -void *dist_base,void *redist_base); +void gic_init (enum gic_type type,unsigned int nr_cpus);void gic_irq_enable (unsigned int intid);void gic_irq_disable (unsigned int intid);unsigned int gic_get_and_ack_irq (void );
tools/testing/selftests/kvm/include/aarch64/vgic.h +1 -2
@@ -16,8 +16,7 @@ ((uint64_t)(flags) << 12) | \ index) -int vgic_v3_setup (struct kvm_vm *vm,unsigned int nr_vcpus,uint32_t nr_irqs, -uint64_t gicd_base_gpa,uint64_t gicr_base_gpa); +int vgic_v3_setup (struct kvm_vm *vm,unsigned int nr_vcpus,uint32_t nr_irqs);#define VGIC_MAX_RESERVED 1023
tools/testing/selftests/kvm/lib/aarch64/gic.c +7 -11
@@ -17,13 +17,12 @@static const struct gic_common_ops *gic_common_ops;static struct spinlock gic_lock; -static void gic_cpu_init (unsigned int cpu,void *redist_base) +static void gic_cpu_init (unsigned int cpu) { - gic_common_ops->gic_cpu_init (cpu, redist_base); + gic_common_ops->gic_cpu_init (cpu); } -static void -gic_dist_init (enum gic_type type,unsigned int nr_cpus,void *dist_base) +static void gic_dist_init (enum gic_type type,unsigned int nr_cpus) {const struct gic_common_ops *gic_ops =NULL ;
@@ -40,7 +39,7 @@ gic_dist_init(enum gic_type type, unsigned int nr_cpus, void *dist_base)GUEST_ASSERT (gic_ops); - gic_ops->gic_init (nr_cpus, dist_base); + gic_ops->gic_init (nr_cpus); gic_common_ops = gic_ops;/* Make sure that the initialized data is visible to all the vCPUs */
@@ -49,18 +48,15 @@ gic_dist_init(enum gic_type type, unsigned int nr_cpus, void *dist_base)spin_unlock (&gic_lock); } -void gic_init (enum gic_type type,unsigned int nr_cpus, -void *dist_base,void *redist_base) +void gic_init (enum gic_type type,unsigned int nr_cpus) {uint32_t cpu =guest_get_vcpuid ();GUEST_ASSERT (type <GIC_TYPE_MAX ); -GUEST_ASSERT (dist_base); -GUEST_ASSERT (redist_base);GUEST_ASSERT (nr_cpus); -gic_dist_init (type, nr_cpus, dist_base); -gic_cpu_init (cpu, redist_base); +gic_dist_init (type, nr_cpus); +gic_cpu_init (cpu); }void gic_irq_enable (unsigned int intid)
tools/testing/selftests/kvm/lib/aarch64/gic_private.h +2 -2
@@ -8,8 +8,8 @@#define SELFTEST_KVM_GIC_PRIVATE_H struct gic_common_ops { -void (*gic_init)(unsigned int nr_cpus,void *dist_base); -void (*gic_cpu_init)(unsigned int cpu,void *redist_base); +void (*gic_init)(unsigned int nr_cpus); +void (*gic_cpu_init)(unsigned int cpu);void (*gic_irq_enable)(unsigned int intid);void (*gic_irq_disable)(unsigned int intid);uint64_t (*gic_read_iar)(void );
tools/testing/selftests/kvm/lib/aarch64/gic_v3.c +28 -34
160 lines not on this page · raw message
tools/testing/selftests/kvm/lib/aarch64/vgic.c +8 -10
@@ -19,8 +19,6 @@* Input args: * vm - KVM VM * nr_vcpus - Number of vCPUs supported by this VM -* gicd_base_gpa - Guest Physical Address of the Distributor region -* gicr_base_gpa - Guest Physical Address of the Redistributor region * * Output args: None *
@@ -30,11 +28,10 @@* redistributor regions of the guest. Since it depends on the number of * vCPUs for the VM, it must be called after all the vCPUs have been created. */ -int vgic_v3_setup (struct kvm_vm *vm,unsigned int nr_vcpus,uint32_t nr_irqs, -uint64_t gicd_base_gpa,uint64_t gicr_base_gpa) +int vgic_v3_setup (struct kvm_vm *vm,unsigned int nr_vcpus,uint32_t nr_irqs) {int gic_fd; -uint64_t redist_attr; +uint64_t attr;struct list_head *iter;unsigned int nr_gic_pages, nr_vcpus_created =0 ;
@@ -60,18 +57,19 @@ int vgic_v3_setup(struct kvm_vm *vm, unsigned int nr_vcpus, uint32_t nr_irqs,kvm_device_attr_set (gic_fd,KVM_DEV_ARM_VGIC_GRP_CTRL ,KVM_DEV_ARM_VGIC_CTRL_INIT ,NULL ); + attr =GICD_BASE_GPA ;kvm_device_attr_set (gic_fd,KVM_DEV_ARM_VGIC_GRP_ADDR , -KVM_VGIC_V3_ADDR_TYPE_DIST , &gicd_base_gpa); +KVM_VGIC_V3_ADDR_TYPE_DIST , &attr); nr_gic_pages =vm_calc_num_guest_pages (vm->mode,KVM_VGIC_V3_DIST_SIZE ); -virt_map (vm, gicd_base_gpa, gicd_base_gpa, nr_gic_pages); +virt_map (vm,GICD_BASE_GPA ,GICD_BASE_GPA , nr_gic_pages);/* Redistributor setup */ - redist_attr =REDIST_REGION_ATTR_ADDR (nr_vcpus, gicr_base_gpa,0 ,0 ); + attr =REDIST_REGION_ATTR_ADDR (nr_vcpus,GICR_BASE_GPA ,0 ,0 );kvm_device_attr_set (gic_fd,KVM_DEV_ARM_VGIC_GRP_ADDR , -KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION , &redist_attr); +KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION , &attr); nr_gic_pages =vm_calc_num_guest_pages (vm->mode,KVM_VGIC_V3_REDIST_SIZE * nr_vcpus); -virt_map (vm, gicr_base_gpa, gicr_base_gpa, nr_gic_pages); +virt_map (vm,GICR_BASE_GPA ,GICR_BASE_GPA , nr_gic_pages);kvm_device_attr_set (gic_fd,KVM_DEV_ARM_VGIC_GRP_CTRL ,KVM_DEV_ARM_VGIC_CTRL_INIT ,NULL );
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 15/19 KVM: selftests: Add quadword MMIO accessors
In reply to Oliver UptonThe base registers in the GIC ITS and redistributor for LPIs are 64 bits wide. Add quadword accessors to poke at them.
--- .../selftests/kvm/include/aarch64/processor.h | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
tools/testing/selftests/kvm/include/aarch64/processor.h +17 -0
@@ -177,11 +177,28 @@ static __always_inline u32 __raw_readl(const volatile void *addr)return val; } +static __always_inline void __raw_writeq(u64 val, volatile void *addr) +{ + asmvolatile ("str %0, [%1]" : :"rZ" (val),"r" (addr)); +} + +static __always_inline u64 __raw_readq(const volatile void *addr) +{ +u64 val; + asmvolatile ("ldr %0, [%1]" :"=r" (val) :"r" (addr)); +return val; +} +#define writel_relaxed (v,c) ((void)__raw_writel((__force u32)cpu_to_le32(v),(c)))#define readl_relaxed (c) ({ u32 __r = le32_to_cpu((__force __le32)__raw_readl(c)); __r; }) +#define writeq_relaxed (v,c) ((void)__raw_writeq((__force u64)cpu_to_le64(v),(c))) +#define readq_relaxed (c) ({ u64 __r = le64_to_cpu((__force __le64)__raw_readq(c)); __r; })#define writel (v,c) ({ __iowmb(); writel_relaxed((v),(c));})#define readl (c) ({ u32 __v = readl_relaxed(c); __iormb(__v); __v; }) +#define writeq (v,c) ({ __iowmb(); writeq_relaxed((v),(c));}) +#define readq (c) ({ u64 __v = readq_relaxed(c); __iormb(__v); __v; }) +static inline void local_irq_enable (void ) {
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 16/19 KVM: selftests: Add a minimal library for interacting with an ITS
In reply to Oliver UptonA prerequisite of testing LPI injection performance is of course instantiating an ITS for the guest. Add a small library for creating an ITS and interacting with it from the guest.
--- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/include/aarch64/gic.h | 8 +- .../kvm/include/aarch64/gic_v3_its.h | 19 ++ .../selftests/kvm/include/aarch64/vgic.h | 2 + .../selftests/kvm/lib/aarch64/gic_v3_its.c | 248 ++++++++++++++++++ .../testing/selftests/kvm/lib/aarch64/vgic.c | 18 ++ 6 files changed, 295 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/kvm/include/aarch64/gic_v3_its.h create mode 100644 tools/testing/selftests/kvm/lib/aarch64/gic_v3_its.c
tools/testing/selftests/kvm/Makefile +1 -0
@@ -45,6 +45,7 @@ LIBKVM_x86_64 += lib/x86_64/vmx.c LIBKVM_aarch64 += lib/aarch64/gic.c LIBKVM_aarch64 += lib/aarch64/gic_v3.c +LIBKVM_aarch64 += lib/aarch64/gic_v3_its.c LIBKVM_aarch64 += lib/aarch64/handlers.S LIBKVM_aarch64 += lib/aarch64/processor.c LIBKVM_aarch64 += lib/aarch64/spinlock.c
tools/testing/selftests/kvm/include/aarch64/gic.h +7 -1
@@ -13,10 +13,16 @@ enum gic_type { GIC_TYPE_MAX, }; -#define GICD_BASE_GPA 0x8000000ULL +/* +* Note that the redistributor frames are at the end, as the range scales +* with the number of vCPUs in the VM. +*/ +#define GITS_BASE_GPA 0x8000000ULL +#define GICD_BASE_GPA (GITS_BASE_GPA + KVM_VGIC_V3_ITS_SIZE)#define GICR_BASE_GPA (GICD_BASE_GPA + KVM_VGIC_V3_DIST_SIZE)/* The GIC is identity-mapped into the guest at the time of setup. */ +#define GITS_BASE_GVA ((volatile void *)GITS_BASE_GPA)#define GICD_BASE_GVA ((volatile void *)GICD_BASE_GPA)#define GICR_BASE_GVA ((volatile void *)GICR_BASE_GPA)
tools/testing/selftests/kvm/include/aarch64/gic_v3_its.h new file +19 -0
@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __SELFTESTS_GIC_V3_ITS_H__ +#define __SELFTESTS_GIC_V3_ITS_H__ + +#include <linux/sizes.h> + +void its_init (vm_paddr_t coll_tbl,size_t coll_tbl_sz, +vm_paddr_t device_tbl,size_t device_tbl_sz, +vm_paddr_t cmdq,size_t cmdq_size); + +void its_send_mapd_cmd (void *cmdq_base,u32 device_id,vm_paddr_t itt_base, +size_t itt_size,bool valid); +void its_send_mapc_cmd (void *cmdq_base,u32 vcpu_id,u32 collection_id,bool valid); +void its_send_mapti_cmd (void *cmdq_base,u32 device_id,u32 event_id, +u32 collection_id,u32 intid); +void its_send_invall_cmd (void *cmdq_base,u32 collection_id); + +#endif // __SELFTESTS_GIC_V3_ITS_H__
tools/testing/selftests/kvm/include/aarch64/vgic.h +2 -0
@@ -32,4 +32,6 @@ void kvm_irq_write_isactiver(int gic_fd, uint32_t intid, struct kvm_vcpu *vcpu);#define KVM_IRQCHIP_NUM_PINS (1020 - 32) +int vgic_its_setup (struct kvm_vm *vm); +#endif // SELFTEST_KVM_VGIC_H
tools/testing/selftests/kvm/lib/aarch64/gic_v3_its.c new file +248 -0
248 lines not on this page · raw message
tools/testing/selftests/kvm/lib/aarch64/vgic.c +18 -0
@@ -166,3 +166,21 @@ void kvm_irq_write_isactiver(int gic_fd, uint32_t intid, struct kvm_vcpu *vcpu) {vgic_poke_irq (gic_fd, intid, vcpu,GICD_ISACTIVER ); } + +int vgic_its_setup (struct kvm_vm *vm) +{ +int its_fd =kvm_create_device (vm,KVM_DEV_TYPE_ARM_VGIC_ITS ); +u64 attr; + + attr =GITS_BASE_GPA ; +kvm_device_attr_set (its_fd,KVM_DEV_ARM_VGIC_GRP_ADDR , +KVM_VGIC_ITS_ADDR_TYPE , &attr); + +kvm_device_attr_set (its_fd,KVM_DEV_ARM_VGIC_GRP_CTRL , +KVM_DEV_ARM_VGIC_CTRL_INIT ,NULL ); + +virt_map (vm,GITS_BASE_GPA ,GITS_BASE_GPA , +vm_calc_num_guest_pages (vm->mode,KVM_VGIC_V3_ITS_SIZE )); + +return its_fd; +}
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 17/19 KVM: selftests: Add helper for enabling LPIs on a redistributor
In reply to Oliver UptonThe selftests GIC library presently does not support LPIs. Add a userspace helper for configuring a redistributor for LPIs, installing an LPI configuration table and LPI pending table.
--- .../selftests/kvm/include/aarch64/gic.h | 3 +++ .../selftests/kvm/lib/aarch64/gic_v3.c | 24 +++++++++++++++++++ .../testing/selftests/kvm/lib/aarch64/vgic.c | 2 ++ 3 files changed, 29 insertions(+)
tools/testing/selftests/kvm/include/aarch64/gic.h +3 -0
@@ -58,4 +58,7 @@ void gic_irq_clear_pending(unsigned int intid);bool gic_irq_get_pending (unsigned int intid);void gic_irq_set_config (unsigned int intid,bool is_edge); +void gic_rdist_enable_lpis (vm_paddr_t cfg_table,size_t cfg_table_size, +vm_paddr_t pend_table); +#endif /* SELFTEST_KVM_GIC_H */
tools/testing/selftests/kvm/lib/aarch64/gic_v3.c +24 -0
@@ -401,3 +401,27 @@ const struct gic_common_ops gicv3_ops = { .gic_irq_get_pending = gicv3_irq_get_pending, .gic_irq_set_config = gicv3_irq_set_config, }; + +void gic_rdist_enable_lpis (vm_paddr_t cfg_table,size_t cfg_table_size, +vm_paddr_t pend_table) +{ +volatile void *rdist_base =gicr_base_cpu (guest_get_vcpuid ()); + +u32 ctlr; +u64 val; + + val = (cfg_table | + GICR_PROPBASER_InnerShareable | + GICR_PROPBASER_RaWaWb | + ((ilog2 (cfg_table_size) -1 ) &GICR_PROPBASER_IDBITS_MASK )); +writeq_relaxed (val, rdist_base +GICR_PROPBASER ); + + val = (pend_table | + GICR_PENDBASER_InnerShareable | + GICR_PENDBASER_RaWaWb); +writeq_relaxed (val, rdist_base +GICR_PENDBASER ); + + ctlr =readl_relaxed (rdist_base +GICR_CTLR ); + ctlr |=GICR_CTLR_ENABLE_LPIS ; +writel_relaxed (ctlr, rdist_base +GICR_CTLR ); +}
tools/testing/selftests/kvm/lib/aarch64/vgic.c +2 -0
@@ -3,8 +3,10 @@* ARM Generic Interrupt Controller (GIC) v3 host support */ +#include <linux/kernel.h> #include <linux/kvm.h> #include <linux/sizes.h> +#include <asm/cputype.h> #include <asm/kvm_para.h> #include <asm/kvm.h>
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 18/19 KVM: selftests: Use MPIDR_HWID_BITMASK from cputype.h
In reply to Oliver UptonNo need for a home-rolled definition, just rely on the common header.
--- tools/testing/selftests/kvm/aarch64/psci_test.c | 2 ++ tools/testing/selftests/kvm/include/aarch64/processor.h | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-)
tools/testing/selftests/kvm/aarch64/psci_test.c +2 -0
@@ -13,7 +13,9 @@#define _GNU_SOURCE +#include <linux/kernel.h> #include <linux/psci.h> +#include <asm/cputype.h> #include "kvm_util.h" #include "processor.h"
tools/testing/selftests/kvm/include/aarch64/processor.h +0 -2
@@ -58,8 +58,6 @@ MAIR_ATTRIDX(MAIR_ATTR_NORMAL, MT_NORMAL) | \ MAIR_ATTRIDX(MAIR_ATTR_NORMAL_WT, MT_NORMAL_WT)) -#define MPIDR_HWID_BITMASK (0xff00fffffful) -void aarch64_vcpu_setup (struct kvm_vcpu *vcpu,struct kvm_vcpu_init *init);struct kvm_vcpu *aarch64_vcpu_add (struct kvm_vm *vm,uint32_t vcpu_id,struct kvm_vcpu_init *init,void *guest_code);
Signature · 1 line
2.44.0.769.g3c40516874-goog
PATCH v3 19/19 KVM: selftests: Add stress test for LPI injection
In reply to Oliver UptonNow that all the infrastructure is in place, add a test to stress KVM's LPI injection. Keep a 1:1 mapping of device IDs to signalling threads, allowing the user to scale up/down the sender side of an LPI. Make use of the new VM stats for the translation cache to estimate the translation hit rate.
Since the primary focus of the test is on performance, you'll notice that the guest code is not pedantic about the LPIs it receives. Counting the number of LPIs would require synchronization between the device and vCPU threads to avoid coalescing and would get in the way of performance numbers.
--- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/aarch64/vgic_lpi_stress.c | 410 ++++++++++++++++++ 2 files changed, 411 insertions(+) create mode 100644 tools/testing/selftests/kvm/aarch64/vgic_lpi_stress.c
tools/testing/selftests/kvm/Makefile +1 -0
@@ -158,6 +158,7 @@ TEST_GEN_PROGS_aarch64 += aarch64/smccc_filter TEST_GEN_PROGS_aarch64 += aarch64/vcpu_width_config TEST_GEN_PROGS_aarch64 += aarch64/vgic_init TEST_GEN_PROGS_aarch64 += aarch64/vgic_irq +TEST_GEN_PROGS_aarch64 += aarch64/vgic_lpi_stress TEST_GEN_PROGS_aarch64 += aarch64/vpmu_counter_access TEST_GEN_PROGS_aarch64 += access_tracking_perf_test TEST_GEN_PROGS_aarch64 += arch_timer
tools/testing/selftests/kvm/aarch64/vgic_lpi_stress.c new file +410 -0
410 lines not on this page · raw message
Signature · 1 line
2.44.0.769.g3c40516874-goog
Quoted 15 lines · Oliver Upton
v2: https://lore.kernel.org/kvmarm/[email protected]/
v2 -> v3:
- Add lockdep assertion tokvm_vfio_create()(Sean)
- Address intermediate compilation issue invgic_its_invalidate_cache()(Marc)
- Use a more compact cache index based on the DID/EID range of the ITS (Marc)
- Comment improvements (Marc)
- Avoid explicit acquisition of the translation cache'sxa_lock()(Marc)
- Eliminate the need to disable IRQs to acquire the translation cache'sxa_lock()
- Collapse internal helper intovgic_its_check_cache()
Quoted 1 line
[...]
Applied to next, thanks!
[01/19] KVM: Treat the device list as an rculist
commit: ea54dd374232cc3b6d0ac0a89d715d61ebb04bf6
[02/19] KVM: arm64: vgic-its: Walk LPI xarray in its_sync_lpi_pending_table()
commit: 720f73b750e66ca753c56c29801009c28bb484ac
[03/19] KVM: arm64: vgic-its: Walk LPI xarray in vgic_its_invall()
commit: c64115c80fc8abacfb89c36d650b7021ebb3d739
[04/19] KVM: arm64: vgic-its: Walk LPI xarray in vgic_its_cmd_handle_movall()
commit: 11f4f8f3e6e0697fb640d5c6c79b27c2233bc3da
[05/19] KVM: arm64: vgic-debug: Use an xarray mark for debug iterator
commit: 85d3ccc8b75bb5a443edb3c42fa22e97da2e60ec
[06/19] KVM: arm64: vgic-its: Get rid of vgic_copy_lpi_list()
commit: 30a0ce9c4928640efd3112d01d432d0778878b7e
[07/19] KVM: arm64: vgic-its: Scope translation cache invalidations to an ITS
commit: c09c8ab99a8afb24f6b4a6bc7c2767fec348ae81
[08/19] KVM: arm64: vgic-its: Maintain a translation cache per ITS
commit: 8201d1028caa4fae88e222c4e8cf541fdf45b821
[09/19] KVM: arm64: vgic-its: Spin off helper for finding ITS by doorbell addr
commit: dedfcd17faf8718f4842e7fbfcd2e7026854d7f5
[10/19] KVM: arm64: vgic-its: Use the per-ITS translation cache for injection
commit: e64f2918c6e7a2c2cbf310d1b571d1a886b91475
[11/19] KVM: arm64: vgic-its: Rip out the global translation cache
commit: ec39bbfd55d07de2e2d4111f35c7ad9523c89ec3
[12/19] KVM: arm64: vgic-its: Get rid of the lpi_list_lock
commit: 481c9ee846d27c72acc0c3bb23025c7fdad8c171
[13/19] KVM: selftests: Align with kernel's GIC definitions
commit: d82689bdd828833bd582c2bf7a85071cacb52990
[14/19] KVM: selftests: Standardise layout of GIC frames
commit: 1505bc70f80df9824e9d68d15a7452856df7488c
[15/19] KVM: selftests: Add quadword MMIO accessors
commit: 232269eb7dd5242877abfab1d47a1eb049a44b95
[16/19] KVM: selftests: Add a minimal library for interacting with an ITS
commit: be26db61e880b3892f189e9ef54b7b80599245bf
[17/19] KVM: selftests: Add helper for enabling LPIs on a redistributor
commit: 03e560ab539009856266b0cf8c100c9f7d1f8fee
[18/19] KVM: selftests: Use MPIDR_HWID_BITMASK from cputype.h
commit: c3c369b508d9a447436b7abb2fded9aec18953ff
[19/19] KVM: selftests: Add stress test for LPI injection
commit: 96d36ad95b03c89857d405b3317efb0188ac59cbCheers,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.