KVM: arm64: Fix vcpu->mutex v. kvm->lock inversion
Subsystems (3)
v3 cover KVM: arm64: Fix vcpu->mutex v. kvm->lock inversion
As it so happens, lock ordering in KVM/arm64 is completely backwards. There's a significant amount of VM-wide state that needs to be accessed from the context of a vCPU. Until now, this was accomplished by acquiring the kvm->lock, but that cannot be nested within vcpu->mutex.
This series fixes the issue with some fine-grained locking for MP state and a new, dedicated mutex that can nest with both kvm->lock and vcpu->mutex.
Tested with kvmtool and QEMU scaled up to 64 vCPUs on a kernel w/ lockdep enabled. Applies to kvmarm-fixes-6.3-2.
Please note that these changes will most likely be taken in the 6.4 merge window and not as a fixup during 6.3. After discussing with Marc we agreed that letting these patches marinade in -next for a while is probably best as it is quite an overhaul to our locking. Additionally, there is no evidence of actual deadlocks occurring in the wild, likely because we _always_ ordered the locks backwards, hence the only breakage at the moment is lockdep.
Also, Jeremy, I chose to omit your Tested-by tag on the last patch as it was rather significantly changed from when you last took it for a spin.
v1: http://lore.kernel.org/kvmarm/[email protected]
v2: https://lore.kernel.org/kvmarm/[email protected]/
v2 -> v3:
- Continue to acquire the kvm->lock where we must protect against a concurrent vCPU creation (Marc)
- Plug a few missing WRITE_ONCE() promotions for mp_state (Marc)
- Hold the mp_state_lock when reading reset_state (Marc)
- Fix the unguarded write to mp_state in kvm_prepare_system_event()
- Collect Jeremy's Tested-by tags (thanks!)
v1 -> v2:
- Add a dedicated lock for serializing writes to MP state
- Inform lockdep of acquisition order at time of VM/vCPU creation
- Plug a race with GIC creation (Sean)
- Use the config_lock in GIC ITS flows as well
Oliver Upton (4):
KVM: arm64: Avoid vcpu->mutex v. kvm->lock inversion in CPU_ON
KVM: arm64: Avoid lock inversion when setting the VM register width
KVM: arm64: Use config_lock to protect data ordered against KVM_RUN
KVM: arm64: Use config_lock to protect vgic state
arch/arm64/include/asm/kvm_host.h | 4 ++ arch/arm64/kvm/arm.c | 53 +++++++++++++++++++++------ arch/arm64/kvm/guest.c | 2 + arch/arm64/kvm/hypercalls.c | 4 +- arch/arm64/kvm/pmu-emul.c | 23 +++--------- arch/arm64/kvm/psci.c | 28 ++++++++------ arch/arm64/kvm/reset.c | 15 ++++---- arch/arm64/kvm/vgic/vgic-debug.c | 8 ++-- arch/arm64/kvm/vgic/vgic-init.c | 36 +++++++++++------- arch/arm64/kvm/vgic/vgic-its.c | 18 ++++++--- arch/arm64/kvm/vgic/vgic-kvm-device.c | 47 ++++++++++++++---------- arch/arm64/kvm/vgic/vgic-mmio-v3.c | 4 +- arch/arm64/kvm/vgic/vgic-mmio.c | 12 +++--- arch/arm64/kvm/vgic/vgic-v4.c | 11 +++--- arch/arm64/kvm/vgic/vgic.c | 12 +++--- 15 files changed, 168 insertions(+), 109 deletions(-)
base-commit: 8c2e8ac8ad4be68409e806ce1cc78fc7a04539f3
Signature · 1 line
2.40.0.348.gf938b09366-goog
PATCH v3 1/4 KVM: arm64: Avoid vcpu->mutex v. kvm->lock inversion in CPU_ON
KVM/arm64 had the lock ordering backwards on vcpu->mutex and kvm->lock from the very beginning. One such example is the way vCPU resets are
handled: the kvm->lock is acquired while handling a guest CPU_ON PSCI call.
Add a dedicated lock to serialize writes to kvm_vcpu_arch::{mp_state, reset_state}. Promote all accessors of mp_state to {READ,WRITE}_ONCE() as readers do not acquire the mp_state_lock. While at it, plug yet another race by taking the mp_state_lock in the KVM_SET_MP_STATE ioctl handler.
As changes to MP state are now guarded with a dedicated lock, drop the kvm->lock acquisition from the PSCI CPU_ON path. Similarly, move the reader of reset_state outside of the kvm->lock and instead protect it with the mp_state_lock. Note that writes to reset_state::reset have been demoted to regular stores as both readers and writers acquire the mp_state_lock.
While the kvm->lock inversion still exists in kvm_reset_vcpu(), at least now PSCI CPU_ON no longer depends on it for serializing vCPU reset.
--- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/arm.c | 31 ++++++++++++++++++++++--------- arch/arm64/kvm/psci.c | 28 ++++++++++++++++------------ arch/arm64/kvm/reset.c | 9 +++++---- 4 files changed, 44 insertions(+), 25 deletions(-)
arch/arm64/include/asm/kvm_host.h +1 -0
@@ -522,6 +522,7 @@ struct kvm_vcpu_arch {/* vcpu power state */ struct kvm_mp_state mp_state; +spinlock_t mp_state_lock;/* Cache some mmu pages needed inside spinlock regions */ struct kvm_mmu_memory_cache mmu_page_cache;
arch/arm64/kvm/arm.c +22 -9
@@ -326,6 +326,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) {int err; +spin_lock_init (&vcpu->arch.mp_state_lock); +/* Force users to call KVM_ARM_VCPU_INIT */ vcpu->arch.target =-1 ;bitmap_zero (vcpu->arch.features,KVM_VCPU_MAX_FEATURES );
@@ -443,34 +445,41 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) vcpu->cpu =-1 ; } -void kvm_arm_vcpu_power_off (struct kvm_vcpu *vcpu) +static void __kvm_arm_vcpu_power_off (struct kvm_vcpu *vcpu) { - vcpu->arch.mp_state.mp_state =KVM_MP_STATE_STOPPED ; +WRITE_ONCE (vcpu->arch.mp_state.mp_state,KVM_MP_STATE_STOPPED );kvm_make_request (KVM_REQ_SLEEP , vcpu);kvm_vcpu_kick (vcpu); } +void kvm_arm_vcpu_power_off (struct kvm_vcpu *vcpu) +{ +spin_lock (&vcpu->arch.mp_state_lock); +__kvm_arm_vcpu_power_off (vcpu); +spin_unlock (&vcpu->arch.mp_state_lock); +} +bool kvm_arm_vcpu_stopped (struct kvm_vcpu *vcpu) { -return vcpu->arch.mp_state.mp_state ==KVM_MP_STATE_STOPPED ; +return READ_ONCE (vcpu->arch.mp_state.mp_state) ==KVM_MP_STATE_STOPPED ; }static void kvm_arm_vcpu_suspend (struct kvm_vcpu *vcpu) { - vcpu->arch.mp_state.mp_state =KVM_MP_STATE_SUSPENDED ; +WRITE_ONCE (vcpu->arch.mp_state.mp_state,KVM_MP_STATE_SUSPENDED );kvm_make_request (KVM_REQ_SUSPEND , vcpu);kvm_vcpu_kick (vcpu); }static bool kvm_arm_vcpu_suspended (struct kvm_vcpu *vcpu) { -return vcpu->arch.mp_state.mp_state ==KVM_MP_STATE_SUSPENDED ; +return READ_ONCE (vcpu->arch.mp_state.mp_state) ==KVM_MP_STATE_SUSPENDED ; }int kvm_arch_vcpu_ioctl_get_mpstate (struct kvm_vcpu *vcpu,struct kvm_mp_state *mp_state) { - *mp_state = vcpu->arch.mp_state; + *mp_state =READ_ONCE (vcpu->arch.mp_state);return 0 ; }
@@ -480,12 +489,14 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, {int ret =0 ; +spin_lock (&vcpu->arch.mp_state_lock); +switch (mp_state->mp_state) {case KVM_MP_STATE_RUNNABLE : - vcpu->arch.mp_state = *mp_state; +WRITE_ONCE (vcpu->arch.mp_state, *mp_state);break ;case KVM_MP_STATE_STOPPED : -kvm_arm_vcpu_power_off (vcpu); +__kvm_arm_vcpu_power_off (vcpu);break ;case KVM_MP_STATE_SUSPENDED :kvm_arm_vcpu_suspend (vcpu);
@@ -494,6 +505,8 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, ret = -EINVAL ; } +spin_unlock (&vcpu->arch.mp_state_lock); +return ret; }
@@ -1213,7 +1226,7 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,if (test_bit (KVM_ARM_VCPU_POWER_OFF , vcpu->arch.features))kvm_arm_vcpu_power_off (vcpu);else - vcpu->arch.mp_state.mp_state =KVM_MP_STATE_RUNNABLE ; +WRITE_ONCE (vcpu->arch.mp_state.mp_state,KVM_MP_STATE_RUNNABLE );return 0 ; }
arch/arm64/kvm/psci.c +16 -12
@@ -62,6 +62,7 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)struct vcpu_reset_state *reset_state;struct kvm *kvm = source_vcpu->kvm;struct kvm_vcpu *vcpu =NULL ; +int ret =PSCI_RET_SUCCESS ;unsigned long cpu_id; cpu_id =smccc_get_arg1 (source_vcpu);
@@ -76,11 +77,15 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)*/ if (!vcpu)return PSCI_RET_INVALID_PARAMS ; + +spin_lock (&vcpu->arch.mp_state_lock);if (!kvm_arm_vcpu_stopped (vcpu)) {if (kvm_psci_version (source_vcpu) !=KVM_ARM_PSCI_0_1 ) -return PSCI_RET_ALREADY_ON ; + ret =PSCI_RET_ALREADY_ON ;else -return PSCI_RET_INVALID_PARAMS ; + ret =PSCI_RET_INVALID_PARAMS ; + + goto out_unlock; } reset_state = &vcpu->arch.reset_state;
@@ -96,7 +101,7 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)*/ reset_state->r0 =smccc_get_arg3 (source_vcpu); -WRITE_ONCE (reset_state->reset, true); + reset_state->reset = true;kvm_make_request (KVM_REQ_VCPU_RESET , vcpu);/*
@@ -108,7 +113,9 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu) vcpu->arch.mp_state.mp_state =KVM_MP_STATE_RUNNABLE ;kvm_vcpu_wake_up (vcpu); -return PSCI_RET_SUCCESS ; +out_unlock: +spin_unlock (&vcpu->arch.mp_state_lock); +return ret; }static unsigned long kvm_psci_vcpu_affinity_info (struct kvm_vcpu *vcpu)
@@ -168,8 +175,11 @@ static void kvm_prepare_system_event(struct kvm_vcpu *vcpu, u32 type, u64 flags)* after this call is handled and before the VCPUs have been * re-initialized. */ -kvm_for_each_vcpu (i, tmp, vcpu->kvm) - tmp->arch.mp_state.mp_state =KVM_MP_STATE_STOPPED ; +kvm_for_each_vcpu (i, tmp, vcpu->kvm) { +spin_lock (&tmp->arch.mp_state_lock); +WRITE_ONCE (tmp->arch.mp_state.mp_state,KVM_MP_STATE_STOPPED ); +spin_unlock (&tmp->arch.mp_state_lock); + }kvm_make_all_cpus_request (vcpu->kvm,KVM_REQ_SLEEP );memset (&vcpu->run->system_event,0 ,sizeof (vcpu->run->system_event));
@@ -229,7 +239,6 @@ static unsigned long kvm_psci_check_allowed_function(struct kvm_vcpu *vcpu, u32static int kvm_psci_0_2_call (struct kvm_vcpu *vcpu) { -struct kvm *kvm = vcpu->kvm;u32 psci_fn =smccc_get_function (vcpu);unsigned long val;int ret =1 ;
@@ -254,9 +263,7 @@ static int kvm_psci_0_2_call(struct kvm_vcpu *vcpu)kvm_psci_narrow_to_32bit (vcpu); fallthrough;case PSCI_0_2_FN64_CPU_ON : -mutex_lock (&kvm->lock); val =kvm_psci_vcpu_on (vcpu); -mutex_unlock (&kvm->lock);break ;case PSCI_0_2_FN_AFFINITY_INFO :kvm_psci_narrow_to_32bit (vcpu);
@@ -395,7 +402,6 @@ static int kvm_psci_1_x_call(struct kvm_vcpu *vcpu, u32 minor)static int kvm_psci_0_1_call (struct kvm_vcpu *vcpu) { -struct kvm *kvm = vcpu->kvm;u32 psci_fn =smccc_get_function (vcpu);unsigned long val;
@@ -405,9 +411,7 @@ static int kvm_psci_0_1_call(struct kvm_vcpu *vcpu) val =PSCI_RET_SUCCESS ;break ;case KVM_PSCI_FN_CPU_ON : -mutex_lock (&kvm->lock); val =kvm_psci_vcpu_on (vcpu); -mutex_unlock (&kvm->lock);break ;default : val =PSCI_RET_NOT_SUPPORTED ;
arch/arm64/kvm/reset.c +5 -4
@@ -264,15 +264,16 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)mutex_lock (&vcpu->kvm->lock); ret =kvm_set_vm_width (vcpu); -if (!ret) { - reset_state = vcpu->arch.reset_state; -WRITE_ONCE (vcpu->arch.reset_state.reset, false); - }mutex_unlock (&vcpu->kvm->lock);if (ret)return ret; +spin_lock (&vcpu->arch.mp_state_lock); + reset_state = vcpu->arch.reset_state; + vcpu->arch.reset_state.reset = false; +spin_unlock (&vcpu->arch.mp_state_lock); +/* Reset PMU outside of the non-preemptible section */ kvm_pmu_vcpu_reset (vcpu);
Signature · 1 line
2.40.0.348.gf938b09366-goog
PATCH v3 2/4 KVM: arm64: Avoid lock inversion when setting the VM register width
In reply to Oliver Uptonkvm->lock must be taken outside of the vcpu->mutex. Of course, the locking documentation for KVM makes this abundantly clear. Nonetheless, the locking order in KVM/arm64 has been wrong for quite a while; we acquire the kvm->lock while holding the vcpu->mutex all over the shop.
All was seemingly fine until commit 42a90008f890 ("KVM: Ensure lockdep knows about kvm->lock vs. vcpu->mutex ordering rule") caught us with our pants down, leading to lockdep barfing:
======================================================
WARNING: possible circular locking dependency detected 6.2.0-rc7+ #19 Not tainted
------------------------------------------------------
qemu-system-aar/859 is trying to acquire lock:
ffff5aa69269eba0 (&host_kvm->lock){+.+.}-{3:3}, at: kvm_reset_vcpu+0x34/0x274
but task is already holding lock:
ffff5aa68768c0b8 (&vcpu->mutex){+.+.}-{3:3}, at: kvm_vcpu_ioctl+0x8c/0xba0
which lock already depends on the new lock.
Add a dedicated lock to serialize writes to VM-scoped configuration from the context of a vCPU. Protect the register width flags with the new lock, thus avoiding the need to grab the kvm->lock while holding vcpu->mutex in kvm_reset_vcpu().
--- arch/arm64/include/asm/kvm_host.h | 3 +++ arch/arm64/kvm/arm.c | 18 ++++++++++++++++++ arch/arm64/kvm/reset.c | 6 +++--- 3 files changed, 24 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_host.h +3 -0
@@ -199,6 +199,9 @@ struct kvm_arch {/* Mandated version of PSCI */ u32 psci_version; +/* Protects VM-scoped configuration data */ +struct mutex config_lock; +/* * If we encounter a data abort without valid instruction syndrome * information, report this to user space. User space can (and
arch/arm64/kvm/arm.c +18 -0
@@ -128,6 +128,16 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) {int ret; +mutex_init (&kvm->arch.config_lock); + +#ifdef CONFIG_LOCKDEP +/* Clue in lockdep that the config_lock must be taken inside kvm->lock */ +mutex_lock (&kvm->lock); +mutex_lock (&kvm->arch.config_lock); +mutex_unlock (&kvm->arch.config_lock); +mutex_unlock (&kvm->lock); +#endif + ret =kvm_share_hyp (kvm, kvm +1 );if (ret)return ret;
@@ -328,6 +338,14 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)spin_lock_init (&vcpu->arch.mp_state_lock); +#ifdef CONFIG_LOCKDEP +/* Inform lockdep that the config_lock is acquired after vcpu->mutex */ +mutex_lock (&vcpu->mutex); +mutex_lock (&vcpu->kvm->arch.config_lock); +mutex_unlock (&vcpu->kvm->arch.config_lock); +mutex_unlock (&vcpu->mutex); +#endif +/* Force users to call KVM_ARM_VCPU_INIT */ vcpu->arch.target =-1 ;bitmap_zero (vcpu->arch.features,KVM_VCPU_MAX_FEATURES );
arch/arm64/kvm/reset.c +3 -3
@@ -205,7 +205,7 @@ static int kvm_set_vm_width(struct kvm_vcpu *vcpu) is32bit =vcpu_has_feature (vcpu,KVM_ARM_VCPU_EL1_32BIT ); -lockdep_assert_held (&kvm->lock); +lockdep_assert_held (&kvm->arch.config_lock);if (test_bit (KVM_ARCH_FLAG_REG_WIDTH_CONFIGURED , &kvm->arch.flags)) {/*
@@ -262,9 +262,9 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)bool loaded;u32 pstate; -mutex_lock (&vcpu->kvm->lock); +mutex_lock (&vcpu->kvm->arch.config_lock); ret =kvm_set_vm_width (vcpu); -mutex_unlock (&vcpu->kvm->lock); +mutex_unlock (&vcpu->kvm->arch.config_lock);if (ret)return ret;
Signature · 1 line
2.40.0.348.gf938b09366-goog
PATCH v3 3/4 KVM: arm64: Use config_lock to protect data ordered against KVM_RUN
In reply to Oliver UptonThere are various bits of VM-scoped data that can only be configured before the first call to KVM_RUN, such as the hypercall bitmaps and the PMU. As these fields are protected by the kvm->lock and accessed while holding vcpu->mutex, this is yet another example of lock inversion.
Change out the kvm->lock for kvm->arch.config_lock in all of these instances. Opportunistically simplify the locking mechanics of the PMU configuration by holding the config_lock for the entirety of kvm_arm_pmu_v3_set_attr().
Note that this also addresses a couple of bugs. There is an unguarded read of the PMU version in KVM_ARM_VCPU_PMU_V3_FILTER which could race with KVM_ARM_VCPU_PMU_V3_SET_PMU. Additionally, until now writes to the per-vCPU vPMU irq were not serialized VM-wide, meaning concurrent calls to KVM_ARM_VCPU_PMU_V3_IRQ could lead to a false positive in pmu_irq_is_valid().
--- arch/arm64/kvm/arm.c | 4 ++-- arch/arm64/kvm/guest.c | 2 ++ arch/arm64/kvm/hypercalls.c | 4 ++-- arch/arm64/kvm/pmu-emul.c | 23 ++++++----------------- 4 files changed, 12 insertions(+), 21 deletions(-)
arch/arm64/kvm/arm.c +2 -2
@@ -624,9 +624,9 @@ int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)if (kvm_vm_is_protected (kvm))kvm_call_hyp_nvhe (__pkvm_vcpu_init_traps, vcpu); -mutex_lock (&kvm->lock); +mutex_lock (&kvm->arch.config_lock);set_bit (KVM_ARCH_FLAG_HAS_RAN_ONCE , &kvm->arch.flags); -mutex_unlock (&kvm->lock); +mutex_unlock (&kvm->arch.config_lock);return ret; }
arch/arm64/kvm/guest.c +2 -0
@@ -957,7 +957,9 @@ int kvm_arm_vcpu_arch_set_attr(struct kvm_vcpu *vcpu,switch (attr->group) {case KVM_ARM_VCPU_PMU_V3_CTRL : +mutex_lock (&vcpu->kvm->arch.config_lock); ret =kvm_arm_pmu_v3_set_attr (vcpu, attr); +mutex_unlock (&vcpu->kvm->arch.config_lock);break ;case KVM_ARM_VCPU_TIMER_CTRL : ret =kvm_arm_timer_set_attr (vcpu, attr);
arch/arm64/kvm/hypercalls.c +2 -2
@@ -377,7 +377,7 @@ static int kvm_arm_set_fw_reg_bmap(struct kvm_vcpu *vcpu, u64 reg_id, u64 val)if (val & ~fw_reg_features)return -EINVAL ; -mutex_lock (&kvm->lock); +mutex_lock (&kvm->arch.config_lock);if (test_bit (KVM_ARCH_FLAG_HAS_RAN_ONCE , &kvm->arch.flags) && val != *fw_reg_bmap) {
@@ -387,7 +387,7 @@ static int kvm_arm_set_fw_reg_bmap(struct kvm_vcpu *vcpu, u64 reg_id, u64 val)WRITE_ONCE (*fw_reg_bmap, val); out: -mutex_unlock (&kvm->lock); +mutex_unlock (&kvm->arch.config_lock);return ret; }
arch/arm64/kvm/pmu-emul.c +6 -17
@@ -875,7 +875,7 @@ static int kvm_arm_pmu_v3_set_pmu(struct kvm_vcpu *vcpu, int pmu_id)struct arm_pmu *arm_pmu;int ret = -ENXIO ; -mutex_lock (&kvm->lock); +lockdep_assert_held (&kvm->arch.config_lock);mutex_lock (&arm_pmus_lock);list_for_each_entry (entry, &arm_pmus, entry) {
@@ -895,7 +895,6 @@ static int kvm_arm_pmu_v3_set_pmu(struct kvm_vcpu *vcpu, int pmu_id) }mutex_unlock (&arm_pmus_lock); -mutex_unlock (&kvm->lock);return ret; }
@@ -903,22 +902,20 @@ int kvm_arm_pmu_v3_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr) {struct kvm *kvm = vcpu->kvm; +lockdep_assert_held (&kvm->arch.config_lock); +if (!kvm_vcpu_has_pmu (vcpu))return -ENODEV ;if (vcpu->arch.pmu.created)return -EBUSY ; -mutex_lock (&kvm->lock);if (!kvm->arch.arm_pmu) {/* No PMU set, get the default one */ kvm->arch.arm_pmu =kvm_pmu_probe_armpmu (); -if (!kvm->arch.arm_pmu) { -mutex_unlock (&kvm->lock); +if (!kvm->arch.arm_pmu)return -ENODEV ; - } } -mutex_unlock (&kvm->lock);switch (attr->attr) {case KVM_ARM_VCPU_PMU_V3_IRQ : {
@@ -962,19 +959,13 @@ int kvm_arm_pmu_v3_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr) filter.action != KVM_PMU_EVENT_DENY))return -EINVAL ; -mutex_lock (&kvm->lock); - -if (test_bit (KVM_ARCH_FLAG_HAS_RAN_ONCE , &kvm->arch.flags)) { -mutex_unlock (&kvm->lock); +if (test_bit (KVM_ARCH_FLAG_HAS_RAN_ONCE , &kvm->arch.flags))return -EBUSY ; - }if (!kvm->arch.pmu_filter) { kvm->arch.pmu_filter =bitmap_alloc (nr_events,GFP_KERNEL_ACCOUNT ); -if (!kvm->arch.pmu_filter) { -mutex_unlock (&kvm->lock); +if (!kvm->arch.pmu_filter)return -ENOMEM ; - }/* * The default depends on the first applied filter.
@@ -993,8 +984,6 @@ int kvm_arm_pmu_v3_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)else bitmap_clear (kvm->arch.pmu_filter, filter.base_event, filter.nevents); -mutex_unlock (&kvm->lock); -return 0 ; }case KVM_ARM_VCPU_PMU_V3_SET_PMU : {
Signature · 1 line
2.40.0.348.gf938b09366-goog
PATCH v3 4/4 KVM: arm64: Use config_lock to protect vgic state
In reply to Oliver UptonAlmost all of the vgic state is VM-scoped but accessed from the context of a vCPU. These accesses were serialized on the kvm->lock which cannot be nested within a vcpu->mutex critical section.
Move over the vgic state to using the config_lock. Tweak the lock ordering where necessary to ensure that the config_lock is acquired after the vcpu->mutex. Acquire the config_lock in kvm_vgic_create() to avoid a race between the converted flows and GIC creation. Where necessary, continue to acquire kvm->lock to avoid a race with vCPU creation (i.e. flows that use lock_all_vcpus()).
Finally, promote the locking expectations in comments to lockdep assertions and update the locking documentation for the config_lock as well as vcpu->mutex.
--- arch/arm64/kvm/vgic/vgic-debug.c | 8 ++--- arch/arm64/kvm/vgic/vgic-init.c | 36 ++++++++++++-------- arch/arm64/kvm/vgic/vgic-its.c | 18 ++++++---- arch/arm64/kvm/vgic/vgic-kvm-device.c | 47 ++++++++++++++++----------- arch/arm64/kvm/vgic/vgic-mmio-v3.c | 4 +-- arch/arm64/kvm/vgic/vgic-mmio.c | 12 +++---- arch/arm64/kvm/vgic/vgic-v4.c | 11 ++++--- arch/arm64/kvm/vgic/vgic.c | 12 ++++--- 8 files changed, 88 insertions(+), 60 deletions(-)
arch/arm64/kvm/vgic/vgic-debug.c +4 -4
@@ -85,7 +85,7 @@ static void *vgic_debug_start(struct seq_file *s, loff_t *pos)struct kvm *kvm = s->private;struct vgic_state_iter *iter; -mutex_lock (&kvm->lock); +mutex_lock (&kvm->arch.config_lock); iter = kvm->arch.vgic.iter;if (iter) { iter =ERR_PTR (-EBUSY );
@@ -104,7 +104,7 @@ static void *vgic_debug_start(struct seq_file *s, loff_t *pos)if (end_of_vgic (iter)) iter =NULL ; out: -mutex_unlock (&kvm->lock); +mutex_unlock (&kvm->arch.config_lock);return iter; }
@@ -132,12 +132,12 @@ static void vgic_debug_stop(struct seq_file *s, void *v)if (IS_ERR (v))return ; -mutex_lock (&kvm->lock); +mutex_lock (&kvm->arch.config_lock); iter = kvm->arch.vgic.iter;kfree (iter->lpi_array);kfree (iter); kvm->arch.vgic.iter =NULL ; -mutex_unlock (&kvm->lock); +mutex_unlock (&kvm->arch.config_lock); }static void print_dist_state (struct seq_file *s,struct vgic_dist *dist)
arch/arm64/kvm/vgic/vgic-init.c +23 -13
@@ -74,9 +74,6 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)unsigned long i;int ret; -if (irqchip_in_kernel (kvm)) -return -EEXIST ; -/* * This function is also called by the KVM_CREATE_IRQCHIP handler, * which had no chance yet to check the availability of the GICv2
@@ -87,10 +84,20 @@ int kvm_vgic_create(struct kvm *kvm, u32 type) !kvm_vgic_global_state.can_emulate_gicv2)return -ENODEV ; +/* Must be held to avoid race with vCPU creation */ +lockdep_assert_held (&kvm->lock); + ret = -EBUSY ;if (!lock_all_vcpus (kvm))return ret; +mutex_lock (&kvm->arch.config_lock); + +if (irqchip_in_kernel (kvm)) { + ret = -EEXIST ; + goto out_unlock; + } +kvm_for_each_vcpu (i, vcpu, kvm) {if (vcpu_has_run_once (vcpu)) goto out_unlock;
@@ -118,6 +125,7 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)INIT_LIST_HEAD (&kvm->arch.vgic.rd_regions); out_unlock: +mutex_unlock (&kvm->arch.config_lock);unlock_all_vcpus (kvm);return ret; }
@@ -227,9 +235,9 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu)* KVM io device for the redistributor that belongs to this VCPU. */ if (dist->vgic_model ==KVM_DEV_TYPE_ARM_VGIC_V3 ) { -mutex_lock (&vcpu->kvm->lock); +mutex_lock (&vcpu->kvm->arch.config_lock); ret =vgic_register_redist_iodev (vcpu); -mutex_unlock (&vcpu->kvm->lock); +mutex_unlock (&vcpu->kvm->arch.config_lock); }return ret; }
@@ -250,7 +258,6 @@ static void kvm_vgic_vcpu_enable(struct kvm_vcpu *vcpu)* The function is generally called when nr_spis has been explicitly set * by the guest through the KVM DEVICE API. If not nr_spis is set to 256. * vgic_initialized() returns true when this function has succeeded. -* Must be called with kvm->lock held! */ int vgic_init (struct kvm *kvm) {
@@ -259,6 +266,8 @@ int vgic_init(struct kvm *kvm)int ret =0 , i;unsigned long idx; +lockdep_assert_held (&kvm->arch.config_lock); +if (vgic_initialized (kvm))return 0 ;
@@ -373,12 +382,13 @@ void kvm_vgic_vcpu_destroy(struct kvm_vcpu *vcpu) vgic_cpu->rd_iodev.base_addr =VGIC_ADDR_UNDEF ; } -/* To be called with kvm->lock held */ static void __kvm_vgic_destroy (struct kvm *kvm) {struct kvm_vcpu *vcpu;unsigned long i; +lockdep_assert_held (&kvm->arch.config_lock); +vgic_debug_destroy (kvm);kvm_for_each_vcpu (i, vcpu, kvm)
@@ -389,9 +399,9 @@ static void __kvm_vgic_destroy(struct kvm *kvm)void kvm_vgic_destroy (struct kvm *kvm) { -mutex_lock (&kvm->lock); +mutex_lock (&kvm->arch.config_lock);__kvm_vgic_destroy (kvm); -mutex_unlock (&kvm->lock); +mutex_unlock (&kvm->arch.config_lock); }/**
@@ -414,9 +424,9 @@ int vgic_lazy_init(struct kvm *kvm)if (kvm->arch.vgic.vgic_model !=KVM_DEV_TYPE_ARM_VGIC_V2 )return -EBUSY ; -mutex_lock (&kvm->lock); +mutex_lock (&kvm->arch.config_lock); ret =vgic_init (kvm); -mutex_unlock (&kvm->lock); +mutex_unlock (&kvm->arch.config_lock); }return ret;
@@ -441,7 +451,7 @@ int kvm_vgic_map_resources(struct kvm *kvm)if (likely (vgic_ready (kvm)))return 0 ; -mutex_lock (&kvm->lock); +mutex_lock (&kvm->arch.config_lock);if (vgic_ready (kvm)) goto out;
@@ -459,7 +469,7 @@ int kvm_vgic_map_resources(struct kvm *kvm) dist->ready = true; out: -mutex_unlock (&kvm->lock); +mutex_unlock (&kvm->arch.config_lock);return ret; }
arch/arm64/kvm/vgic/vgic-its.c +12 -6
@@ -2045,6 +2045,13 @@ static int vgic_its_attr_regs_access(struct kvm_device *dev,mutex_lock (&dev->kvm->lock); +if (!lock_all_vcpus (dev->kvm)) { +mutex_unlock (&dev->kvm->lock); +return -EBUSY ; + } + +mutex_lock (&dev->kvm->arch.config_lock); +if (IS_VGIC_ADDR_UNDEF (its->vgic_its_base)) { ret = -ENXIO ; goto out;
@@ -2058,11 +2065,6 @@ static int vgic_its_attr_regs_access(struct kvm_device *dev, goto out; } -if (!lock_all_vcpus (dev->kvm)) { - ret = -EBUSY ; - goto out; - } - addr = its->vgic_its_base + offset; len = region->access_flags & VGIC_ACCESS_64bit ?8 :4 ;
@@ -2076,8 +2078,9 @@ static int vgic_its_attr_regs_access(struct kvm_device *dev, }else { *reg = region->its_read (dev->kvm, its, addr, len); } -unlock_all_vcpus (dev->kvm); out: +mutex_unlock (&dev->kvm->arch.config_lock); +unlock_all_vcpus (dev->kvm);mutex_unlock (&dev->kvm->lock);return ret; }
@@ -2757,6 +2760,8 @@ static int vgic_its_ctrl(struct kvm *kvm, struct vgic_its *its, u64 attr)return -EBUSY ; } +mutex_lock (&kvm->arch.config_lock); +switch (attr) {case KVM_DEV_ARM_ITS_CTRL_RESET :vgic_its_reset (kvm, its);
@@ -2769,6 +2774,7 @@ static int vgic_its_ctrl(struct kvm *kvm, struct vgic_its *its, u64 attr)break ; } +mutex_unlock (&kvm->arch.config_lock);unlock_all_vcpus (kvm);mutex_unlock (&its->its_lock);mutex_unlock (&kvm->lock);
arch/arm64/kvm/vgic/vgic-kvm-device.c +28 -19
129 lines not on this page · raw message
arch/arm64/kvm/vgic/vgic-mmio-v3.c +2 -2
@@ -111,7 +111,7 @@ static void vgic_mmio_write_v3_misc(struct kvm_vcpu *vcpu,case GICD_CTLR : {bool was_enabled, is_hwsgi; -mutex_lock (&vcpu->kvm->lock); +mutex_lock (&vcpu->kvm->arch.config_lock); was_enabled = dist->enabled; is_hwsgi = dist->nassgireq;
@@ -139,7 +139,7 @@ static void vgic_mmio_write_v3_misc(struct kvm_vcpu *vcpu,else if (!was_enabled && dist->enabled)vgic_kick_vcpus (vcpu->kvm); -mutex_unlock (&vcpu->kvm->lock); +mutex_unlock (&vcpu->kvm->arch.config_lock);break ; }case GICD_TYPER :
arch/arm64/kvm/vgic/vgic-mmio.c +6 -6
@@ -530,13 +530,13 @@ unsigned long vgic_mmio_read_active(struct kvm_vcpu *vcpu,u32 intid =VGIC_ADDR_TO_INTID (addr,1 );u32 val; -mutex_lock (&vcpu->kvm->lock); +mutex_lock (&vcpu->kvm->arch.config_lock);vgic_access_active_prepare (vcpu, intid); val =__vgic_mmio_read_active (vcpu, addr, len);vgic_access_active_finish (vcpu, intid); -mutex_unlock (&vcpu->kvm->lock); +mutex_unlock (&vcpu->kvm->arch.config_lock);return val; }
@@ -625,13 +625,13 @@ void vgic_mmio_write_cactive(struct kvm_vcpu *vcpu, {u32 intid =VGIC_ADDR_TO_INTID (addr,1 ); -mutex_lock (&vcpu->kvm->lock); +mutex_lock (&vcpu->kvm->arch.config_lock);vgic_access_active_prepare (vcpu, intid);__vgic_mmio_write_cactive (vcpu, addr, len, val);vgic_access_active_finish (vcpu, intid); -mutex_unlock (&vcpu->kvm->lock); +mutex_unlock (&vcpu->kvm->arch.config_lock); } int vgic_mmio_uaccess_write_cactive(struct kvm_vcpu *vcpu,
@@ -662,13 +662,13 @@ void vgic_mmio_write_sactive(struct kvm_vcpu *vcpu, {u32 intid =VGIC_ADDR_TO_INTID (addr,1 ); -mutex_lock (&vcpu->kvm->lock); +mutex_lock (&vcpu->kvm->arch.config_lock);vgic_access_active_prepare (vcpu, intid);__vgic_mmio_write_sactive (vcpu, addr, len, val);vgic_access_active_finish (vcpu, intid); -mutex_unlock (&vcpu->kvm->lock); +mutex_unlock (&vcpu->kvm->arch.config_lock); } int vgic_mmio_uaccess_write_sactive(struct kvm_vcpu *vcpu,
arch/arm64/kvm/vgic/vgic-v4.c +6 -5
@@ -232,9 +232,8 @@ int vgic_v4_request_vpe_irq(struct kvm_vcpu *vcpu, int irq)* @kvm: Pointer to the VM being initialized * * We may be called each time a vITS is created, or when the -* vgic is initialized. This relies on kvm->lock to be -* held. In both cases, the number of vcpus should now be -* fixed. +* vgic is initialized. In both cases, the number of vcpus +* should now be fixed. */ int vgic_v4_init (struct kvm *kvm) {
@@ -243,6 +242,8 @@ int vgic_v4_init(struct kvm *kvm)int nr_vcpus, ret;unsigned long i; +lockdep_assert_held (&kvm->arch.config_lock); +if (!kvm_vgic_global_state.has_gicv4)return 0 ;/* Nothing to see here... move along. */
@@ -309,14 +310,14 @@ int vgic_v4_init(struct kvm *kvm)/** * vgic_v4_teardown - Free the GICv4 data structures * @kvm: Pointer to the VM being destroyed -* -* Relies on kvm->lock to be held. */ void vgic_v4_teardown (struct kvm *kvm) {struct its_vm *its_vm = &kvm->arch.vgic.its_vm;int i; +lockdep_assert_held (&kvm->arch.config_lock); +if (!its_vm->vpes)return ;
arch/arm64/kvm/vgic/vgic.c +7 -5
@@ -24,11 +24,13 @@ struct vgic_global kvm_vgic_global_state __ro_after_init = {/* * Locking order is always: * kvm->lock (mutex) -* its->cmd_lock (mutex) -* its->its_lock (mutex) -* vgic_cpu->ap_list_lock must be taken with IRQs disabled -* kvm->lpi_list_lock must be taken with IRQs disabled -* vgic_irq->irq_lock must be taken with IRQs disabled +* vcpu->mutex (mutex) +* kvm->arch.config_lock (mutex) +* its->cmd_lock (mutex) +* its->its_lock (mutex) +* vgic_cpu->ap_list_lock must be taken with IRQs disabled +* kvm->lpi_list_lock must be taken with IRQs disabled +* vgic_irq->irq_lock must be taken with IRQs disabled * * As the ap_list_lock might be taken from the timer interrupt handler, * we have to disable IRQs before taking this lock and everything lower
Signature · 1 line
2.40.0.348.gf938b09366-goog
Quoted 10 lines · Oliver Upton
As it so happens, lock ordering in KVM/arm64 is completely backwards. There's a significant amount of VM-wide state that needs to be accessed from the context of a vCPU. Until now, this was accomplished by acquiring the
kvm->lock, but that cannot be nested withinvcpu->mutex.This series fixes the issue with some fine-grained locking for MP state and a new, dedicated mutex that can nest with both
kvm->lockandvcpu->mutex.[...]
Applied to next, thanks!
[1/4] KVM: arm64: Avoid vcpu->mutex v. kvm->lock inversion in CPU_ON
commit: 0acc7239c20a8401b8968c2adace8f7c9b0295ae
[2/4] KVM: arm64: Avoid lock inversion when setting the VM register width
commit: c43120afb5c66a3465c7468f5cf9806a26484cde
[3/4] KVM: arm64: Use config_lock to protect data ordered against KVM_RUN
commit: 4bba7f7def6f278266dadf845da472cfbfed784e
[4/4] KVM: arm64: Use config_lock to protect vgic state
commit: f00327731131d1b5aa6a1aa9f50bcf8d620ace4cCheers,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.