Skip to contentSkip to conversation

KVM: SEV: Mark nested locking of vcpu->lock

PATCH Superseded Peter Gonda 2 participants 3 messages Apr 4, 2022

Subsystems (3)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 KVM: SEV: Mark nested locking of vcpu->lock

svm_vm_migrate_from() uses sev_lock_vcpus_for_migration() to lock all source and target vcpu->locks. Mark the nested subclasses to avoid false positives from lockdep.

Fixes b56639318bb2b ("KVM: SEV: Add support for SEV intra host migration")
Cc Paolo Bonzini <[email protected]>
Cc Sean Christopherson <[email protected]>

---

Tested by running sev_migrate_tests with lockdep enabled. Before we see a warning from sev_lock_vcpus_for_migration(). After we get no warnings.

---
 arch/x86/kvm/svm/sev.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
arch/x86/kvm/svm/sev.c +7 -5
index 75fa6dd268f0..8f77421c1c4b 100644
@@ -1591,15 +1591,16 @@ static void sev_unlock_two_vms(struct kvm *dst_kvm, struct kvm *src_kvm)
 	atomic_set_release(&src_sev->migration_in_progress, 0);
 }
 
-
-static int sev_lock_vcpus_for_migration(struct kvm *kvm)
+static int sev_lock_vcpus_for_migration(struct kvm *kvm, unsigned int *subclass)
 {
 	struct kvm_vcpu *vcpu;
 	unsigned long i, j;
 
 	kvm_for_each_vcpu(i, vcpu, kvm) {
-		if (mutex_lock_killable(&vcpu->mutex))
+		if (mutex_lock_killable_nested(&vcpu->mutex, *subclass))
 			goto out_unlock;
+
+		++(*subclass);
 	}
 
 	return 0;
@@ -1717,6 +1718,7 @@ int sev_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd)
 	struct kvm *source_kvm;
 	bool charged = false;
 	int ret;
+	unsigned int vcpu_mutex_subclass = 0;
 
 	source_kvm_file = fget(source_fd);
 	if (!file_is_kvm(source_kvm_file)) {
@@ -1745,10 +1747,10 @@ int sev_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd)
 		charged = true;
 	}
 
-	ret = sev_lock_vcpus_for_migration(kvm);
+	ret = sev_lock_vcpus_for_migration(kvm, &vcpu_mutex_subclass);
 	if (ret)
 		goto out_dst_cgroup;
-	ret = sev_lock_vcpus_for_migration(source_kvm);
+	ret = sev_lock_vcpus_for_migration(source_kvm, &vcpu_mutex_subclass);
 	if (ret)
 		goto out_dst_vcpu;
 
Signature · 1 line
2.35.1.1094.g7c7d902a7c-goog
#1 lore raw
Quoted 32 lines · Peter Gonda

svm_vm_migrate_from() uses sev_lock_vcpus_for_migration() to lock all source and target vcpu->locks. Mark the nested subclasses to avoid false positives from lockdep.

Fixes b56639318bb2b ("KVM: SEV: Add support for SEV intra host migration")
Cc Paolo Bonzini <[email protected]>
Cc Sean Christopherson <[email protected]>

---

Tested by running sev_migrate_tests with lockdep enabled. Before we see a warning from sev_lock_vcpus_for_migration(). After we get no warnings.

---
 arch/x86/kvm/svm/sev.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
arch/x86/kvm/svm/sev.c +1 -2
index 75fa6dd268f0..8f77421c1c4b 100644
@@ -1591,15 +1591,16 @@ static void sev_unlock_two_vms(struct kvm *dst_kvm, struct kvm *src_kvm)
 	atomic_set_release(&src_sev->migration_in_progress, 0);
 }
 
-
-static int sev_lock_vcpus_for_migration(struct kvm *kvm)
+static int sev_lock_vcpus_for_migration(struct kvm *kvm, unsigned int *subclass)

arch/x86/kvm/svm/sev.c static void sev_unlock_two_vms(struct kvm *dst_kvm, struct kvm *src_kvm) · line 1603

Quoted 10 lines
arch/x86/kvm/svm/sev.c +3 -1
 {
 	struct kvm_vcpu *vcpu;
 	unsigned long i, j;
 
 	kvm_for_each_vcpu(i, vcpu, kvm) {
-		if (mutex_lock_killable(&vcpu->mutex))
+		if (mutex_lock_killable_nested(&vcpu->mutex, *subclass))
 			goto out_unlock;
+
+		++(*subclass);

This is rather gross, and I'm guessing it adds extra work for the non-lockdep case, assuming the compiler isn't so clever that it can figure out that the result is never used. Not that this is a hot path...

Does each lock actually need a separate subclass? If so, why don't the other paths that lock all vCPUs complain?

If differentiating the two VMs is sufficient, then we can pass in SINGLE_DEPTH_NESTING for the second round of locks. If a per-vCPU subclass is required, we can use the vCPU index and assign evens to one and odds to the other, e.g. this should work and compiles to a nop when LOCKDEP is disabled (compile tested only). It's still gross, but we could pretty it up, e.g. add defines for the 0/1 param.

arch/x86/kvm/svm/sev.c +4 -5
index 75fa6dd268f0..9be35902b809 100644
@@ -1591,14 +1591,13 @@ static void sev_unlock_two_vms(struct kvm *dst_kvm, struct kvm *src_kvm)
        atomic_set_release(&src_sev->migration_in_progress, 0);
 }
 
-
-static int sev_lock_vcpus_for_migration(struct kvm *kvm)
+static int sev_lock_vcpus_for_migration(struct kvm *kvm, int mod)
 {
        struct kvm_vcpu *vcpu;
        unsigned long i, j;
 
        kvm_for_each_vcpu(i, vcpu, kvm) {
-               if (mutex_lock_killable(&vcpu->mutex))
+               if (mutex_lock_killable_nested(&vcpu->mutex, i * 2 + mod))
                        goto out_unlock;
        }
 
@@ -1745,10 +1744,10 @@ int sev_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd)
                charged = true;
        }
 
-       ret = sev_lock_vcpus_for_migration(kvm);
+       ret = sev_lock_vcpus_for_migration(kvm, 0);
        if (ret)
                goto out_dst_cgroup;
-       ret = sev_lock_vcpus_for_migration(source_kvm);
+       ret = sev_lock_vcpus_for_migration(source_kvm, 1);
        if (ret)
                goto out_dst_vcpu;
 
#2 lore raw
Quoted 5 lines

This is rather gross, and I'm guessing it adds extra work for the non-lockdep case, assuming the compiler isn't so clever that it can figure out that the result is never used. Not that this is a hot path...

Quoted 8 lines

Does each lock actually need a separate subclass? If so, why don't the other paths that lock all vCPUs complain?

If differentiating the two VMs is sufficient, then we can pass in SINGLE_DEPTH_NESTING for the second round of locks. If a per-vCPU subclass is required, we can use the vCPU index and assign evens to one and odds to the other, e.g. this should work and compiles to a nop when LOCKDEP is disabled (compile tested only). It's still gross, but we could pretty it up, e.g. add defines for the 0/1 param.

I checked and the perf vCPU subclassing is required. If I just only use a SINGLE_DEPTH_NESTING on the second VM's vCPUs I still see the warning.

This odds and evens approach seems much better. I'll update to use that in the V2 unless there is a better idea.