Skip to contentSkip to conversation

KVM: arm64: Fix bugs related to mp_state updates

PATCH Applied Reiji Watanabe 2 participants 8 messages Apr 19, 2023 – Apr 21, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover KVM: arm64: Fix bugs related to mp_state updates

This series adds fixes that were missing in the patch [1].

The patch [1] added the mp_state_lock to serialize writes to kvm_vcpu_arch::{mp_state, reset_state}, and promoted all accessors of mp_state to {READ,WRITE}_ONCE() as readers do not acquire the mp_state_lock.

Since the patch [1] didn't fix all the relevant code, fix the code that weren't addressed yet.

This series is based on v6.3-rc7 with the series [2] applied.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

Reiji Watanabe (2):
KVM: arm64: Acquire mp_state_lock in kvm_arch_vcpu_ioctl_vcpu_init()
KVM: arm64: Have kvm_psci_vcpu_on() use WRITE_ONCE() to update
mp_state

 arch/arm64/kvm/arm.c  | 5 ++++-
 arch/arm64/kvm/psci.c | 2 +-
 2 files changed, 5 insertions(+), 2 deletions(-)
Signature · 1 line
2.40.0.396.gfff15efe05-goog
#1 lore raw

PATCH 1/2 KVM: arm64: Acquire mp_state_lock in kvm_arch_vcpu_ioctl_vcpu_init()

kvm_arch_vcpu_ioctl_vcpu_init() doesn't acquire mp_state_lock when setting the mp_state to KVM_MP_STATE_RUNNABLE. Fix the code to acquire the lock.

---
 arch/arm64/kvm/arm.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
arch/arm64/kvm/arm.c +4 -1
index fbafcbbcc463..388aa4f18f21 100644
@@ -1244,8 +1244,11 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
 	 */
 	if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
 		kvm_arm_vcpu_power_off(vcpu);
-	else
+	else {
+		spin_lock(&vcpu->arch.mp_state_lock);
 		WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
+		spin_unlock(&vcpu->arch.mp_state_lock);
+	}
 
 	return 0;
 }
Signature · 1 line
2.40.0.396.gfff15efe05-goog
maintainer
#2 lore raw

Re: [PATCH v1 1/2] KVM: arm64: Acquire mp_state_lock in kvm_arch_vcpu_ioctl_vcpu_init()

Quoted 17 lines · Reiji Watanabe

kvm_arch_vcpu_ioctl_vcpu_init() doesn't acquire mp_state_lock when setting the mp_state to KVM_MP_STATE_RUNNABLE. Fix the code to acquire the lock.

---
 arch/arm64/kvm/arm.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
arch/arm64/kvm/arm.c +0 -0
index fbafcbbcc463..388aa4f18f21 100644
@@ -1244,8 +1244,11 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
 	 */
 	if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))

arch/arm64/kvm/arm.c static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu, · line 1254

Quoted 10 lines
arch/arm64/kvm/arm.c +4 -1
 		kvm_arm_vcpu_power_off(vcpu);
-	else
+	else {
+		spin_lock(&vcpu->arch.mp_state_lock);
 		WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
+		spin_unlock(&vcpu->arch.mp_state_lock);
+	}
 
 	return 0;
 }

I'm not entirely convinced that this fixes anything. What does the lock hazard against given that the write is atomic? But maybe a slightly more readable of this would be to expand the critical section this way:

arch/arm64/kvm/arm.c +5 -1
index 4ec888fdd4f7..bb21d0c25de7 100644
@@ -1246,11 +1246,15 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
 	/*
 	 * Handle the "start in power-off" case.
 	 */
+	spin_lock(&vcpu->arch.mp_state_lock);
+
 	if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
-		kvm_arm_vcpu_power_off(vcpu);
+		__kvm_arm_vcpu_power_off(vcpu);
 	else
 		WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
 
+	spin_unlock(&vcpu->arch.mp_state_lock);
+
 	return 0;
 }
 

Thoughts?

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#3 lore raw

Re: [PATCH v1 1/2] KVM: arm64: Acquire mp_state_lock in kvm_arch_vcpu_ioctl_vcpu_init()

Hi Marc,

Quoted 26 lines · Marc Zyngier
Quoted 24 lines · Reiji Watanabe

kvm_arch_vcpu_ioctl_vcpu_init() doesn't acquire mp_state_lock when setting the mp_state to KVM_MP_STATE_RUNNABLE. Fix the code to acquire the lock.

---
 arch/arm64/kvm/arm.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
arch/arm64/kvm/arm.c +4 -1
index fbafcbbcc463..388aa4f18f21 100644
@@ -1244,8 +1244,11 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
 	 */
 	if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
 		kvm_arm_vcpu_power_off(vcpu);
-	else
+	else {
+		spin_lock(&vcpu->arch.mp_state_lock);
 		WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
+		spin_unlock(&vcpu->arch.mp_state_lock);
+	}
 
Quoted 5 lines

arch/arm64/kvm/arm.c static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu, · line 1259

Quoted 2 lines
arch/arm64/kvm/arm.c +0 -0
 	return 0;
 }

I'm not entirely convinced that this fixes anything. What does the lock hazard against given that the write is atomic? But maybe a

It appears that kvm_psci_vcpu_on() expects the vCPU's mp_state to not be changed by holding the lock. Although I don't think this code practically causes any real issues now, I am a little concerned about leaving one instance that updates mpstate without acquiring the lock, in terms of future maintenance, as holding the lock won't prevent mp_state from being updated.

What do you think ?

Quoted 24 lines

slightly more readable of this would be to expand the critical section this way:

arch/arm64/kvm/arm.c +5 -1
index 4ec888fdd4f7..bb21d0c25de7 100644
@@ -1246,11 +1246,15 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
 	/*
 	 * Handle the "start in power-off" case.
 	 */
+	spin_lock(&vcpu->arch.mp_state_lock);
+
 	if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
-		kvm_arm_vcpu_power_off(vcpu);
+		__kvm_arm_vcpu_power_off(vcpu);
 	else
 		WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
 
+	spin_unlock(&vcpu->arch.mp_state_lock);
+
 	return 0;
 }
 
Quoted 1 line

Thoughts?

Yes, it looks better!

Thank you,
Reiji

maintainer
#4 lore raw

Re: [PATCH v1 1/2] KVM: arm64: Acquire mp_state_lock in kvm_arch_vcpu_ioctl_vcpu_init()

Quoted 36 lines · Reiji Watanabe

Hi Marc,

Quoted 26 lines · Marc Zyngier
Quoted 24 lines · Reiji Watanabe

kvm_arch_vcpu_ioctl_vcpu_init() doesn't acquire mp_state_lock when setting the mp_state to KVM_MP_STATE_RUNNABLE. Fix the code to acquire the lock.

---
 arch/arm64/kvm/arm.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
arch/arm64/kvm/arm.c +4 -1
index fbafcbbcc463..388aa4f18f21 100644
@@ -1244,8 +1244,11 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
 	 */
 	if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
 		kvm_arm_vcpu_power_off(vcpu);
-	else
+	else {
+		spin_lock(&vcpu->arch.mp_state_lock);
 		WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
+		spin_unlock(&vcpu->arch.mp_state_lock);
+	}
 
Quoted 5 lines

arch/arm64/kvm/arm.c static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu, · line 1259

Quoted 2 lines
arch/arm64/kvm/arm.c +0 -0
 	return 0;
 }

I'm not entirely convinced that this fixes anything. What does the lock hazard against given that the write is atomic? But maybe a

Quoted 8 lines

It appears that kvm_psci_vcpu_on() expects the vCPU's mp_state to not be changed by holding the lock. Although I don't think this code practically causes any real issues now, I am a little concerned about leaving one instance that updates mpstate without acquiring the lock, in terms of future maintenance, as holding the lock won't prevent mp_state from being updated.

What do you think ?

Right, fair enough. It is probably better to take the lock and not have to think of this sort of things... I'm becoming more lazy by the minute!

Quoted 20 lines
Quoted 20 lines

slightly more readable of this would be to expand the critical section this way:

arch/arm64/kvm/arm.c +3 -1
index 4ec888fdd4f7..bb21d0c25de7 100644
@@ -1246,11 +1246,15 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
 	/*
 	 * Handle the "start in power-off" case.
 	 */
+	spin_lock(&vcpu->arch.mp_state_lock);
+
 	if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
-		kvm_arm_vcpu_power_off(vcpu);
+		__kvm_arm_vcpu_power_off(vcpu);
 	else
 		WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
 
Quoted 8 lines
Quoted 6 lines
arch/arm64/kvm/arm.c +2 -0
+	spin_unlock(&vcpu->arch.mp_state_lock);
+
 	return 0;
 }
 

Thoughts?

Yes, it looks better!

Cool. I've applied this change to your patch, applied the series to the lock inversion branch, and remerged the branch in -next.

We're getting there! ;-)

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#5 lore raw

Re: [PATCH v1 1/2] KVM: arm64: Acquire mp_state_lock in kvm_arch_vcpu_ioctl_vcpu_init()

Quoted 77 lines · Marc Zyngier
Quoted 36 lines · Reiji Watanabe

Hi Marc,

Quoted 26 lines · Marc Zyngier
Quoted 24 lines · Reiji Watanabe

kvm_arch_vcpu_ioctl_vcpu_init() doesn't acquire mp_state_lock when setting the mp_state to KVM_MP_STATE_RUNNABLE. Fix the code to acquire the lock.

---
 arch/arm64/kvm/arm.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
arch/arm64/kvm/arm.c +4 -1
index fbafcbbcc463..388aa4f18f21 100644
@@ -1244,8 +1244,11 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
   */
  if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
          kvm_arm_vcpu_power_off(vcpu);
- else
+ else {
+         spin_lock(&vcpu->arch.mp_state_lock);
          WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
+         spin_unlock(&vcpu->arch.mp_state_lock);
+ }
 
Quoted 5 lines

arch/arm64/kvm/arm.c static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu, · line 1259

Quoted 2 lines
arch/arm64/kvm/arm.c +0 -0
  return 0;
 }

I'm not entirely convinced that this fixes anything. What does the lock hazard against given that the write is atomic? But maybe a

Quoted 8 lines

It appears that kvm_psci_vcpu_on() expects the vCPU's mp_state to not be changed by holding the lock. Although I don't think this code practically causes any real issues now, I am a little concerned about leaving one instance that updates mpstate without acquiring the lock, in terms of future maintenance, as holding the lock won't prevent mp_state from being updated.

What do you think ?

Right, fair enough. It is probably better to take the lock and not have to think of this sort of things... I'm becoming more lazy by the minute!

Quoted 25 lines
Quoted 25 lines

slightly more readable of this would be to expand the critical section this way:

arch/arm64/kvm/arm.c +5 -1
index 4ec888fdd4f7..bb21d0c25de7 100644
@@ -1246,11 +1246,15 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
    /*
     * Handle the "start in power-off" case.
     */
+   spin_lock(&vcpu->arch.mp_state_lock);
+
    if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
-           kvm_arm_vcpu_power_off(vcpu);
+           __kvm_arm_vcpu_power_off(vcpu);
    else
            WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
 
+   spin_unlock(&vcpu->arch.mp_state_lock);
+
    return 0;
 }
 
Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Thoughts?

Yes, it looks better!

Cool. I've applied this change to your patch, applied the series to the lock inversion branch, and remerged the branch in -next.

We're getting there! ;-)

Thank you, Marc!
Reiji

#6 lore raw

PATCH 2/2 KVM: arm64: Have kvm_psci_vcpu_on() use WRITE_ONCE() to update mp_state

In reply to Reiji Watanabe

All accessors of kvm_vcpu_arch::mp_state should be {READ,WRITE}_ONCE(), since readers of the mp_state don't acquire the mp_state_lock. Nonetheless, kvm_psci_vcpu_on() updates the mp_state without using WRITE_ONCE(). So, fix the code to update the mp_state using WRITE_ONCE.

---
 arch/arm64/kvm/psci.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kvm/psci.c +1 -1
index 5767e6baa61a..d046e82e3723 100644
@@ -110,7 +110,7 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)
 	 */
 	smp_wmb();
 
-	vcpu->arch.mp_state.mp_state = KVM_MP_STATE_RUNNABLE;
+	WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
 	kvm_vcpu_wake_up(vcpu);
 
 out_unlock:
Signature · 1 line
2.40.0.396.gfff15efe05-goog
maintainer Applied
#7 lore raw
In reply to Reiji Watanabe
Quoted 8 lines · Reiji Watanabe

This series adds fixes that were missing in the patch [1].

The patch [1] added the mp_state_lock to serialize writes to kvm_vcpu_arch::{mp_state, reset_state}, and promoted all accessors of mp_state to {READ,WRITE}_ONCE() as readers do not acquire the mp_state_lock.

[...]

Applied to next, thanks!

[1/2] KVM: arm64: Acquire mp_state_lock in kvm_arch_vcpu_ioctl_vcpu_init()

      commit: 4ff910be01c0ca28c2ea8b354dd47a3a17524489
[2/2] KVM: arm64: Have kvm_psci_vcpu_on() use WRITE_ONCE() to update mp_state
      commit: a189884bdc9238aeba941c50f02e25eb584fafed

Cheers,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.