Skip to contentSkip to conversation

KVM: s390: index kvm->arch.idle_mask by vcpu_idx

PATCH In review Halil Pasic 5 participants 12 messages Aug 27, 2021 – Jan 31, 2022

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 KVM: s390: index kvm->arch.idle_mask by vcpu_idx

While in practice vcpu->vcpu_idx == vcpu->vcp_id is often true, it may not always be, and we must not rely on this.

Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies that code like

for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
                vcpu = kvm_get_vcpu(kvm, vcpu_id);
		do_stuff(vcpu);
}

is not legit. The trouble is, we do actually use kvm->arch.idle_mask like this. To fix this problem we have two options. Either use kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id, or switch to indexing via vcpu_idx. The latter is preferable for obvious reasons.

Let us make switch from indexing kvm->arch.idle_mask by vcpu_id to indexing it by vcpu_idx. To keep gisa_int.kicked_mask indexed by the same index as idle_mask lets make the same change for it as well.

Fixes 1ee0bc559dc3 ("KVM: s390: get rid of local_int array")
Cc <[email protected]> # 3.15+
---
 arch/s390/include/asm/kvm_host.h |  1 +
 arch/s390/kvm/interrupt.c        | 12 ++++++------
 arch/s390/kvm/kvm-s390.c         |  2 +-
 arch/s390/kvm/kvm-s390.h         |  2 +-
 4 files changed, 9 insertions(+), 8 deletions(-)
arch/s390/include/asm/kvm_host.h +1 -0
index 161a9e12bfb8..630eab0fa176 100644
@@ -957,6 +957,7 @@ struct kvm_arch{
 	atomic64_t cmma_dirty_pages;
 	/* subset of available cpu features enabled by user space */
 	DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS);
+	/* indexed by vcpu_idx */
 	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
 	struct kvm_s390_gisa_interrupt gisa_int;
 	struct kvm_s390_pv pv;
arch/s390/kvm/interrupt.c +6 -6
index d548d60caed2..16256e17a544 100644
@@ -419,13 +419,13 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
 static void __set_cpu_idle(struct kvm_vcpu *vcpu)
 {
 	kvm_s390_set_cpuflags(vcpu, CPUSTAT_WAIT);
-	set_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	set_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
 }
 
 static void __unset_cpu_idle(struct kvm_vcpu *vcpu)
 {
 	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_WAIT);
-	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
 }
 
 static void __reset_intercept_indicators(struct kvm_vcpu *vcpu)
@@ -3050,18 +3050,18 @@ int kvm_s390_get_irq_state(struct kvm_vcpu *vcpu, __u8 __user *buf, int len)
 
 static void __airqs_kick_single_vcpu(struct kvm *kvm, u8 deliverable_mask)
 {
-	int vcpu_id, online_vcpus = atomic_read(&kvm->online_vcpus);
+	int vcpu_idx, online_vcpus = atomic_read(&kvm->online_vcpus);
 	struct kvm_s390_gisa_interrupt *gi = &kvm->arch.gisa_int;
 	struct kvm_vcpu *vcpu;
 
-	for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
-		vcpu = kvm_get_vcpu(kvm, vcpu_id);
+	for_each_set_bit(vcpu_idx, kvm->arch.idle_mask, online_vcpus) {
+		vcpu = kvm_get_vcpu(kvm, vcpu_idx);
 		if (psw_ioint_disabled(vcpu))
 			continue;
 		deliverable_mask &= (u8)(vcpu->arch.sie_block->gcr[6] >> 24);
 		if (deliverable_mask) {
 			/* lately kicked but not yet running */
-			if (test_and_set_bit(vcpu_id, gi->kicked_mask))
+			if (test_and_set_bit(vcpu_idx, gi->kicked_mask))
 				return;
 			kvm_s390_vcpu_wakeup(vcpu);
 			return;
arch/s390/kvm/kvm-s390.c +1 -1
index 4527ac7b5961..8580543c5bc3 100644
@@ -4044,7 +4044,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu)
 		kvm_s390_patch_guest_per_regs(vcpu);
 	}
 
-	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.gisa_int.kicked_mask);
+	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.gisa_int.kicked_mask);
 
 	vcpu->arch.sie_block->icptcode = 0;
 	cpuflags = atomic_read(&vcpu->arch.sie_block->cpuflags);
arch/s390/kvm/kvm-s390.h +1 -1
index 9fad25109b0d..ecd741ee3276 100644
@@ -79,7 +79,7 @@ static inline int is_vcpu_stopped(struct kvm_vcpu *vcpu)
 
 static inline int is_vcpu_idle(struct kvm_vcpu *vcpu)
 {
-	return test_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	return test_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
 }
 
 static inline int kvm_is_ucontrol(struct kvm *kvm)

base-commit: 77dd11439b86e3f7990e4c0c9e0b67dca82750ba

Signature · 1 line
2.25.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 15 lines · Halil Pasic

While in practice vcpu->vcpu_idx == vcpu->vcp_id is often true, it may not always be, and we must not rely on this.

Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies that code like

for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
                 vcpu = kvm_get_vcpu(kvm, vcpu_id);
		do_stuff(vcpu);
}

is not legit. The trouble is, we do actually use kvm->arch.idle_mask like this. To fix this problem we have two options. Either use kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id, or switch to indexing via vcpu_idx. The latter is preferable for obvious reasons.

Quoted 7 lines

Let us make switch from indexing kvm->arch.idle_mask by vcpu_id to indexing it by vcpu_idx. To keep gisa_int.kicked_mask indexed by the same index as idle_mask lets make the same change for it as well.

Fixes 1ee0bc559dc3 ("KVM: s390: get rid of local_int array")
Cc <[email protected]> # 3.15+

Will apply and wait for the nightly run.

Quoted 91 lines
---
  arch/s390/include/asm/kvm_host.h |  1 +
  arch/s390/kvm/interrupt.c        | 12 ++++++------
  arch/s390/kvm/kvm-s390.c         |  2 +-
  arch/s390/kvm/kvm-s390.h         |  2 +-
  4 files changed, 9 insertions(+), 8 deletions(-)
arch/s390/include/asm/kvm_host.h +1 -0
index 161a9e12bfb8..630eab0fa176 100644
@@ -957,6 +957,7 @@ struct kvm_arch{
  	atomic64_t cmma_dirty_pages;
  	/* subset of available cpu features enabled by user space */
  	DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS);
+	/* indexed by vcpu_idx */
  	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
  	struct kvm_s390_gisa_interrupt gisa_int;
  	struct kvm_s390_pv pv;
arch/s390/kvm/interrupt.c +6 -6
index d548d60caed2..16256e17a544 100644
@@ -419,13 +419,13 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
  static void __set_cpu_idle(struct kvm_vcpu *vcpu)
  {
  	kvm_s390_set_cpuflags(vcpu, CPUSTAT_WAIT);
-	set_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	set_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
  }
 
  static void __unset_cpu_idle(struct kvm_vcpu *vcpu)
  {
  	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_WAIT);
-	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
  }
 
  static void __reset_intercept_indicators(struct kvm_vcpu *vcpu)
@@ -3050,18 +3050,18 @@ int kvm_s390_get_irq_state(struct kvm_vcpu *vcpu, __u8 __user *buf, int len)
 
  static void __airqs_kick_single_vcpu(struct kvm *kvm, u8 deliverable_mask)
  {
-	int vcpu_id, online_vcpus = atomic_read(&kvm->online_vcpus);
+	int vcpu_idx, online_vcpus = atomic_read(&kvm->online_vcpus);
  	struct kvm_s390_gisa_interrupt *gi = &kvm->arch.gisa_int;
  	struct kvm_vcpu *vcpu;
 
-	for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
-		vcpu = kvm_get_vcpu(kvm, vcpu_id);
+	for_each_set_bit(vcpu_idx, kvm->arch.idle_mask, online_vcpus) {
+		vcpu = kvm_get_vcpu(kvm, vcpu_idx);
  		if (psw_ioint_disabled(vcpu))
  			continue;
  		deliverable_mask &= (u8)(vcpu->arch.sie_block->gcr[6] >> 24);
  		if (deliverable_mask) {
  			/* lately kicked but not yet running */
-			if (test_and_set_bit(vcpu_id, gi->kicked_mask))
+			if (test_and_set_bit(vcpu_idx, gi->kicked_mask))
  				return;
  			kvm_s390_vcpu_wakeup(vcpu);
  			return;
arch/s390/kvm/kvm-s390.c +1 -1
index 4527ac7b5961..8580543c5bc3 100644
@@ -4044,7 +4044,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu)
  		kvm_s390_patch_guest_per_regs(vcpu);
  	}
 
-	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.gisa_int.kicked_mask);
+	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.gisa_int.kicked_mask);
 
  	vcpu->arch.sie_block->icptcode = 0;
  	cpuflags = atomic_read(&vcpu->arch.sie_block->cpuflags);
arch/s390/kvm/kvm-s390.h +1 -1
index 9fad25109b0d..ecd741ee3276 100644
@@ -79,7 +79,7 @@ static inline int is_vcpu_stopped(struct kvm_vcpu *vcpu)
 
  static inline int is_vcpu_idle(struct kvm_vcpu *vcpu)
  {
-	return test_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	return test_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
  }
 
  static inline int kvm_is_ucontrol(struct kvm *kvm)

base-commit: 77dd11439b86e3f7990e4c0c9e0b67dca82750ba

maintainer Reviewed-by on v1 1/1
#2 lore raw
In reply to Halil Pasic
Quoted 2 lines · On Fri, 27 Aug 2021 14:54:29 +0200 Halil Pasic <[email protected]> wrote:

While in practice vcpu->vcpu_idx == vcpu->vcp_id is often true, it may not always be, and we must not rely on this.

why?

maybe add a simple explanation of why vcpu_idx and vcpu_id can be different, namely:
KVM decides the vcpu_idx, userspace decides the vcpu_id, thus the two might not match

Quoted 5 lines

Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies that code like

for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
                vcpu = kvm_get_vcpu(kvm, vcpu_id);

you can also add a sentence to clarify that kvm_get_vcpu expects an vcpu_idx, not an vcpu_id.

Quoted 8 lines
		do_stuff(vcpu);
}

is not legit. The trouble is, we do actually use kvm->arch.idle_mask like this. To fix this problem we have two options. Either use kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id, or switch to indexing via vcpu_idx. The latter is preferable for obvious reasons.

Quoted 6 lines

Let us make switch from indexing kvm->arch.idle_mask by vcpu_id to indexing it by vcpu_idx. To keep gisa_int.kicked_mask indexed by the same index as idle_mask lets make the same change for it as well.

Fixes 1ee0bc559dc3 ("KVM: s390: get rid of local_int array")

otherwise looks good to me.

Quoted 91 lines
Cc <[email protected]> # 3.15+
---
 arch/s390/include/asm/kvm_host.h |  1 +
 arch/s390/kvm/interrupt.c        | 12 ++++++------
 arch/s390/kvm/kvm-s390.c         |  2 +-
 arch/s390/kvm/kvm-s390.h         |  2 +-
 4 files changed, 9 insertions(+), 8 deletions(-)
arch/s390/include/asm/kvm_host.h +1 -0
index 161a9e12bfb8..630eab0fa176 100644
@@ -957,6 +957,7 @@ struct kvm_arch{
 	atomic64_t cmma_dirty_pages;
 	/* subset of available cpu features enabled by user space */
 	DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS);
+	/* indexed by vcpu_idx */
 	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
 	struct kvm_s390_gisa_interrupt gisa_int;
 	struct kvm_s390_pv pv;
arch/s390/kvm/interrupt.c +6 -6
index d548d60caed2..16256e17a544 100644
@@ -419,13 +419,13 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
 static void __set_cpu_idle(struct kvm_vcpu *vcpu)
 {
 	kvm_s390_set_cpuflags(vcpu, CPUSTAT_WAIT);
-	set_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	set_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
 }
 
 static void __unset_cpu_idle(struct kvm_vcpu *vcpu)
 {
 	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_WAIT);
-	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
 }
 
 static void __reset_intercept_indicators(struct kvm_vcpu *vcpu)
@@ -3050,18 +3050,18 @@ int kvm_s390_get_irq_state(struct kvm_vcpu *vcpu, __u8 __user *buf, int len)
 
 static void __airqs_kick_single_vcpu(struct kvm *kvm, u8 deliverable_mask)
 {
-	int vcpu_id, online_vcpus = atomic_read(&kvm->online_vcpus);
+	int vcpu_idx, online_vcpus = atomic_read(&kvm->online_vcpus);
 	struct kvm_s390_gisa_interrupt *gi = &kvm->arch.gisa_int;
 	struct kvm_vcpu *vcpu;
 
-	for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
-		vcpu = kvm_get_vcpu(kvm, vcpu_id);
+	for_each_set_bit(vcpu_idx, kvm->arch.idle_mask, online_vcpus) {
+		vcpu = kvm_get_vcpu(kvm, vcpu_idx);
 		if (psw_ioint_disabled(vcpu))
 			continue;
 		deliverable_mask &= (u8)(vcpu->arch.sie_block->gcr[6] >> 24);
 		if (deliverable_mask) {
 			/* lately kicked but not yet running */
-			if (test_and_set_bit(vcpu_id, gi->kicked_mask))
+			if (test_and_set_bit(vcpu_idx, gi->kicked_mask))
 				return;
 			kvm_s390_vcpu_wakeup(vcpu);
 			return;
arch/s390/kvm/kvm-s390.c +1 -1
index 4527ac7b5961..8580543c5bc3 100644
@@ -4044,7 +4044,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu)
 		kvm_s390_patch_guest_per_regs(vcpu);
 	}
 
-	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.gisa_int.kicked_mask);
+	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.gisa_int.kicked_mask);
 
 	vcpu->arch.sie_block->icptcode = 0;
 	cpuflags = atomic_read(&vcpu->arch.sie_block->cpuflags);
arch/s390/kvm/kvm-s390.h +1 -1
index 9fad25109b0d..ecd741ee3276 100644
@@ -79,7 +79,7 @@ static inline int is_vcpu_stopped(struct kvm_vcpu *vcpu)
 
 static inline int is_vcpu_idle(struct kvm_vcpu *vcpu)
 {
-	return test_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	return test_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
 }
 
 static inline int kvm_is_ucontrol(struct kvm *kvm)

base-commit: 77dd11439b86e3f7990e4c0c9e0b67dca82750ba

#3 lore raw
Quoted 14 lines · Claudio Imbrenda
Quoted 2 lines · On Fri, 27 Aug 2021 14:54:29 +0200 Halil Pasic <[email protected]> wrote:

While in practice vcpu->vcpu_idx == vcpu->vcp_id is often true, it may not always be, and we must not rely on this.

why?

maybe add a simple explanation of why vcpu_idx and vcpu_id can be different, namely:
KVM decides the vcpu_idx, userspace decides the vcpu_id, thus the two might not match

Quoted 1 line
Quoted 9 lines
Quoted 4 lines

Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies that code like

for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
                 vcpu = kvm_get_vcpu(kvm, vcpu_id);

you can also add a sentence to clarify that kvm_get_vcpu expects an vcpu_idx, not an vcpu_id.

Quoted 1 line
		do_stuff(vcpu);

I will modify the patch description accordingly before sending to Paolo. Thanks for noticing.

Quoted 110 lines
Quoted 7 lines

}
is not legit. The trouble is, we do actually use kvm->arch.idle_mask like this. To fix this problem we have two options. Either use kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id, or switch to indexing via vcpu_idx. The latter is preferable for obvious reasons.

Quoted 6 lines

Let us make switch from indexing kvm->arch.idle_mask by vcpu_id to indexing it by vcpu_idx. To keep gisa_int.kicked_mask indexed by the same index as idle_mask lets make the same change for it as well.

Fixes 1ee0bc559dc3 ("KVM: s390: get rid of local_int array")

otherwise looks good to me.

Quoted 91 lines
Cc <[email protected]> # 3.15+
---
  arch/s390/include/asm/kvm_host.h |  1 +
  arch/s390/kvm/interrupt.c        | 12 ++++++------
  arch/s390/kvm/kvm-s390.c         |  2 +-
  arch/s390/kvm/kvm-s390.h         |  2 +-
  4 files changed, 9 insertions(+), 8 deletions(-)
arch/s390/include/asm/kvm_host.h +1 -0
index 161a9e12bfb8..630eab0fa176 100644
@@ -957,6 +957,7 @@ struct kvm_arch{
  	atomic64_t cmma_dirty_pages;
  	/* subset of available cpu features enabled by user space */
  	DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS);
+	/* indexed by vcpu_idx */
  	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
  	struct kvm_s390_gisa_interrupt gisa_int;
  	struct kvm_s390_pv pv;
arch/s390/kvm/interrupt.c +6 -6
index d548d60caed2..16256e17a544 100644
@@ -419,13 +419,13 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
  static void __set_cpu_idle(struct kvm_vcpu *vcpu)
  {
  	kvm_s390_set_cpuflags(vcpu, CPUSTAT_WAIT);
-	set_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	set_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
  }
 
  static void __unset_cpu_idle(struct kvm_vcpu *vcpu)
  {
  	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_WAIT);
-	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
  }
 
  static void __reset_intercept_indicators(struct kvm_vcpu *vcpu)
@@ -3050,18 +3050,18 @@ int kvm_s390_get_irq_state(struct kvm_vcpu *vcpu, __u8 __user *buf, int len)
 
  static void __airqs_kick_single_vcpu(struct kvm *kvm, u8 deliverable_mask)
  {
-	int vcpu_id, online_vcpus = atomic_read(&kvm->online_vcpus);
+	int vcpu_idx, online_vcpus = atomic_read(&kvm->online_vcpus);
  	struct kvm_s390_gisa_interrupt *gi = &kvm->arch.gisa_int;
  	struct kvm_vcpu *vcpu;
 
-	for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
-		vcpu = kvm_get_vcpu(kvm, vcpu_id);
+	for_each_set_bit(vcpu_idx, kvm->arch.idle_mask, online_vcpus) {
+		vcpu = kvm_get_vcpu(kvm, vcpu_idx);
  		if (psw_ioint_disabled(vcpu))
  			continue;
  		deliverable_mask &= (u8)(vcpu->arch.sie_block->gcr[6] >> 24);
  		if (deliverable_mask) {
  			/* lately kicked but not yet running */
-			if (test_and_set_bit(vcpu_id, gi->kicked_mask))
+			if (test_and_set_bit(vcpu_idx, gi->kicked_mask))
  				return;
  			kvm_s390_vcpu_wakeup(vcpu);
  			return;
arch/s390/kvm/kvm-s390.c +1 -1
index 4527ac7b5961..8580543c5bc3 100644
@@ -4044,7 +4044,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu)
  		kvm_s390_patch_guest_per_regs(vcpu);
  	}
 
-	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.gisa_int.kicked_mask);
+	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.gisa_int.kicked_mask);
 
  	vcpu->arch.sie_block->icptcode = 0;
  	cpuflags = atomic_read(&vcpu->arch.sie_block->cpuflags);
arch/s390/kvm/kvm-s390.h +1 -1
index 9fad25109b0d..ecd741ee3276 100644
@@ -79,7 +79,7 @@ static inline int is_vcpu_stopped(struct kvm_vcpu *vcpu)
 
  static inline int is_vcpu_idle(struct kvm_vcpu *vcpu)
  {
-	return test_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	return test_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
  }
 
  static inline int kvm_is_ucontrol(struct kvm *kvm)

base-commit: 77dd11439b86e3f7990e4c0c9e0b67dca82750ba

#4 lore raw
Quoted 5 lines · On Fri, 27 Aug 2021 18:36:48 +0200 Christian Borntraeger…
Quoted 4 lines · Claudio Imbrenda
Quoted 1 line · On Fri, 27 Aug 2021 14:54:29 +0200 Halil Pasic <[email protected]> wrote:

While in practice vcpu->vcpu_idx == vcpu->vcp_id is often true,

s/vcp_id/vcpu_id/

Quoted 15 lines
Quoted 15 lines
Quoted 1 line

it may not always be, and we must not rely on this.

why?

maybe add a simple explanation of why vcpu_idx and vcpu_id can be different, namely:
KVM decides the vcpu_idx, userspace decides the vcpu_id, thus the two might not match

Quoted 5 lines

Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies that code like

for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
                 vcpu = kvm_get_vcpu(kvm, vcpu_id);  
Quoted 7 lines
Quoted 4 lines

you can also add a sentence to clarify that kvm_get_vcpu expects an vcpu_idx, not an vcpu_id.

Quoted 1 line
		do_stuff(vcpu);  

I will modify the patch description accordingly before sending to Paolo. Thanks for noticing.

Can you also please fix the typo I pointed out above (in the first line of the long description).

Thanks!
Halil

#5 lore raw
Quoted 29 lines · Halil Pasic
Quoted 5 lines · On Fri, 27 Aug 2021 18:36:48 +0200 Christian Borntraeger…
Quoted 4 lines · Claudio Imbrenda
Quoted 1 line · On Fri, 27 Aug 2021 14:54:29 +0200 Halil Pasic <[email protected]> wrote:

While in practice vcpu->vcpu_idx == vcpu->vcp_id is often true,

s/vcp_id/vcpu_id/

Quoted 18 lines
Quoted 18 lines
Quoted 1 line

it may not always be, and we must not rely on this.

why?

maybe add a simple explanation of why vcpu_idx and vcpu_id can be different, namely:
KVM decides the vcpu_idx, userspace decides the vcpu_id, thus the two might not match

Quoted 5 lines

Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies that code like

for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
                  vcpu = kvm_get_vcpu(kvm, vcpu_id);

you can also add a sentence to clarify that kvm_get_vcpu expects an vcpu_idx, not an vcpu_id.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
		do_stuff(vcpu);

I will modify the patch description accordingly before sending to Paolo. Thanks for noticing.

Can you also please fix the typo I pointed out above (in the first line of the long description).

I already queued, but I think this is not a big deal.

#6 lore raw
In reply to Claudio Imbrenda
Quoted 4 lines · On Fri, 27 Aug 2021 16:06:16 +0200 Claudio Imbrenda <[email protected]>…
Quoted 1 line · On Fri, 27 Aug 2021 14:54:29 +0200 Halil Pasic <[email protected]> wrote:

While in practice vcpu->vcpu_idx == vcpu->vcp_id is often true,

s/vcp_id/vcpu_id/

Quoted 8 lines
Quoted 1 line

it may not always be, and we must not rely on this.

why?

maybe add a simple explanation of why vcpu_idx and vcpu_id can be different, namely:
KVM decides the vcpu_idx, userspace decides the vcpu_id, thus the two might not match

Not sure that is a good explanation. A quote from
Documentation/virt/kvm/api.rst:

"""
4.7 KVM_CREATE_VCPU
-------------------

:Capability: basic
:Architectures: all
:Type: vm ioctl
:Parameters: vcpu id (apic id on x86)
:Returns: vcpu fd on success, -1 on error

This API adds a vcpu to a virtual machine. No more than max_vcpus may be added. The vcpu id is an integer in the range [0, max_vcpu_id).

The recommended max_vcpus value can be retrieved using the KVM_CAP_NR_VCPUS of the KVM_CHECK_EXTENSION ioctl() at run-time.
The maximum possible value for max_vcpus can be retrieved using the KVM_CAP_MAX_VCPUS of the KVM_CHECK_EXTENSION ioctl() at run-time. """

Based on that and a quick look at the code, it looks to me like the set of valid vcpu_id values are a subset of the range of vcpu_idx-es, i.e. that kvm could decide to choose vcpu_id for the value of vcpu_idx. I don't think it should, but it could. Were the set of valid vcpu_id values not a subset of the set of supported vcpu_idx values, then one could argue that this is why.

I didn't want to get into explaining the why, I just wanted to state the fact.

Quoted 9 lines
Quoted 6 lines

Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies that code like

for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
                vcpu = kvm_get_vcpu(kvm, vcpu_id);  

you can also add a sentence to clarify that kvm_get_vcpu expects an vcpu_idx, not an vcpu_id.

maybe ...

Quoted 4 lines
Quoted 4 lines
		do_stuff(vcpu);
}

is not legit. The trouble is, we do actually use kvm->arch.idle_mask

... s/legit\./legit, because kvm_get_vcpu() expects a vcpu_idx and not a vcpu_id.

But I agree kvm_get_vcpu(kvm, vcpu_id); does not scream BUG at me while kvm_get_vcpu_by_idx(kvm, vcpu_id) would look much more suspicious.

[..]

Quoted 4 lines

otherwise looks good to me.

Thanks for your reveiew!

Halil

[..]

Reviewed-by on v1 1/1
#7 lore raw
In reply to Halil Pasic
Quoted 111 lines · Halil Pasic

While in practice vcpu->vcpu_idx == vcpu->vcp_id is often true, it may not always be, and we must not rely on this.

Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies that code like

for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
                 vcpu = kvm_get_vcpu(kvm, vcpu_id);
		do_stuff(vcpu);
}

is not legit. The trouble is, we do actually use kvm->arch.idle_mask like this. To fix this problem we have two options. Either use kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id, or switch to indexing via vcpu_idx. The latter is preferable for obvious reasons.

Let us make switch from indexing kvm->arch.idle_mask by vcpu_id to indexing it by vcpu_idx. To keep gisa_int.kicked_mask indexed by the same index as idle_mask lets make the same change for it as well.

Fixes 1ee0bc559dc3 ("KVM: s390: get rid of local_int array")
Cc <[email protected]> # 3.15+
---
  arch/s390/include/asm/kvm_host.h |  1 +
  arch/s390/kvm/interrupt.c        | 12 ++++++------
  arch/s390/kvm/kvm-s390.c         |  2 +-
  arch/s390/kvm/kvm-s390.h         |  2 +-
  4 files changed, 9 insertions(+), 8 deletions(-)
arch/s390/include/asm/kvm_host.h +1 -0
index 161a9e12bfb8..630eab0fa176 100644
@@ -957,6 +957,7 @@ struct kvm_arch{
  	atomic64_t cmma_dirty_pages;
  	/* subset of available cpu features enabled by user space */
  	DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS);
+	/* indexed by vcpu_idx */
  	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
  	struct kvm_s390_gisa_interrupt gisa_int;
  	struct kvm_s390_pv pv;
arch/s390/kvm/interrupt.c +6 -6
index d548d60caed2..16256e17a544 100644
@@ -419,13 +419,13 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
  static void __set_cpu_idle(struct kvm_vcpu *vcpu)
  {
  	kvm_s390_set_cpuflags(vcpu, CPUSTAT_WAIT);
-	set_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	set_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
  }
 
  static void __unset_cpu_idle(struct kvm_vcpu *vcpu)
  {
  	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_WAIT);
-	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
  }
 
  static void __reset_intercept_indicators(struct kvm_vcpu *vcpu)
@@ -3050,18 +3050,18 @@ int kvm_s390_get_irq_state(struct kvm_vcpu *vcpu, __u8 __user *buf, int len)
 
  static void __airqs_kick_single_vcpu(struct kvm *kvm, u8 deliverable_mask)
  {
-	int vcpu_id, online_vcpus = atomic_read(&kvm->online_vcpus);
+	int vcpu_idx, online_vcpus = atomic_read(&kvm->online_vcpus);
  	struct kvm_s390_gisa_interrupt *gi = &kvm->arch.gisa_int;
  	struct kvm_vcpu *vcpu;
 
-	for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
-		vcpu = kvm_get_vcpu(kvm, vcpu_id);
+	for_each_set_bit(vcpu_idx, kvm->arch.idle_mask, online_vcpus) {
+		vcpu = kvm_get_vcpu(kvm, vcpu_idx);
  		if (psw_ioint_disabled(vcpu))
  			continue;
  		deliverable_mask &= (u8)(vcpu->arch.sie_block->gcr[6] >> 24);
  		if (deliverable_mask) {
  			/* lately kicked but not yet running */
-			if (test_and_set_bit(vcpu_id, gi->kicked_mask))
+			if (test_and_set_bit(vcpu_idx, gi->kicked_mask))
  				return;
  			kvm_s390_vcpu_wakeup(vcpu);
  			return;
arch/s390/kvm/kvm-s390.c +1 -1
index 4527ac7b5961..8580543c5bc3 100644
@@ -4044,7 +4044,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu)
  		kvm_s390_patch_guest_per_regs(vcpu);
  	}
 
-	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.gisa_int.kicked_mask);
+	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.gisa_int.kicked_mask);
 
  	vcpu->arch.sie_block->icptcode = 0;
  	cpuflags = atomic_read(&vcpu->arch.sie_block->cpuflags);
arch/s390/kvm/kvm-s390.h +1 -1
index 9fad25109b0d..ecd741ee3276 100644
@@ -79,7 +79,7 @@ static inline int is_vcpu_stopped(struct kvm_vcpu *vcpu)
 
  static inline int is_vcpu_idle(struct kvm_vcpu *vcpu)
  {
-	return test_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	return test_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
  }
 
  static inline int kvm_is_ucontrol(struct kvm *kvm)
Quoted 1 line

base-commit: 77dd11439b86e3f7990e4c0c9e0b67dca82750ba

I did also run some tests and have prepared a qemu that will choose vcpu_id's that are not equal to the index in the kernel.

Michael

Quoted 1 line
#8 lore raw
In reply to Halil Pasic

Hi Halil,

Dne 27. 08. 21 v 14:54 Halil Pasic napsal(a):

Quoted 9 lines

While in practice vcpu->vcpu_idx == vcpu->vcp_id is often true, it may not always be, and we must not rely on this.

Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies that code like

for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
                 vcpu = kvm_get_vcpu(kvm, vcpu_id);
		do_stuff(vcpu);
}
Quoted 5 lines

is not legit. The trouble is, we do actually use kvm->arch.idle_mask like this. To fix this problem we have two options. Either use kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id, or switch to indexing via vcpu_idx. The latter is preferable for obvious reasons.

I'm just backporting this fix to SLES 12 SP5, and I've noticed that there is still this code in __floating_irq_kick():

	/* find idle VCPUs first, then round robin */
	sigcpu = find_first_bit(fi->idle_mask, online_vcpus);
/* ... round robin loop removed ...
	dst_vcpu = kvm_get_vcpu(kvm, sigcpu);

It seems to me that this does exactly the thing that is not legit, but I'm no expert. Did I miss something?

Petr T

Quoted 97 lines

Let us make switch from indexing kvm->arch.idle_mask by vcpu_id to indexing it by vcpu_idx. To keep gisa_int.kicked_mask indexed by the same index as idle_mask lets make the same change for it as well.

Fixes 1ee0bc559dc3 ("KVM: s390: get rid of local_int array")
Cc <[email protected]> # 3.15+
---
  arch/s390/include/asm/kvm_host.h |  1 +
  arch/s390/kvm/interrupt.c        | 12 ++++++------
  arch/s390/kvm/kvm-s390.c         |  2 +-
  arch/s390/kvm/kvm-s390.h         |  2 +-
  4 files changed, 9 insertions(+), 8 deletions(-)
arch/s390/include/asm/kvm_host.h +1 -0
index 161a9e12bfb8..630eab0fa176 100644
@@ -957,6 +957,7 @@ struct kvm_arch{
  	atomic64_t cmma_dirty_pages;
  	/* subset of available cpu features enabled by user space */
  	DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS);
+	/* indexed by vcpu_idx */
  	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
  	struct kvm_s390_gisa_interrupt gisa_int;
  	struct kvm_s390_pv pv;
arch/s390/kvm/interrupt.c +6 -6
index d548d60caed2..16256e17a544 100644
@@ -419,13 +419,13 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
  static void __set_cpu_idle(struct kvm_vcpu *vcpu)
  {
  	kvm_s390_set_cpuflags(vcpu, CPUSTAT_WAIT);
-	set_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	set_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
  }
 
  static void __unset_cpu_idle(struct kvm_vcpu *vcpu)
  {
  	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_WAIT);
-	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
  }
 
  static void __reset_intercept_indicators(struct kvm_vcpu *vcpu)
@@ -3050,18 +3050,18 @@ int kvm_s390_get_irq_state(struct kvm_vcpu *vcpu, __u8 __user *buf, int len)
 
  static void __airqs_kick_single_vcpu(struct kvm *kvm, u8 deliverable_mask)
  {
-	int vcpu_id, online_vcpus = atomic_read(&kvm->online_vcpus);
+	int vcpu_idx, online_vcpus = atomic_read(&kvm->online_vcpus);
  	struct kvm_s390_gisa_interrupt *gi = &kvm->arch.gisa_int;
  	struct kvm_vcpu *vcpu;
 
-	for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
-		vcpu = kvm_get_vcpu(kvm, vcpu_id);
+	for_each_set_bit(vcpu_idx, kvm->arch.idle_mask, online_vcpus) {
+		vcpu = kvm_get_vcpu(kvm, vcpu_idx);
  		if (psw_ioint_disabled(vcpu))
  			continue;
  		deliverable_mask &= (u8)(vcpu->arch.sie_block->gcr[6] >> 24);
  		if (deliverable_mask) {
  			/* lately kicked but not yet running */
-			if (test_and_set_bit(vcpu_id, gi->kicked_mask))
+			if (test_and_set_bit(vcpu_idx, gi->kicked_mask))
  				return;
  			kvm_s390_vcpu_wakeup(vcpu);
  			return;
arch/s390/kvm/kvm-s390.c +1 -1
index 4527ac7b5961..8580543c5bc3 100644
@@ -4044,7 +4044,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu)
  		kvm_s390_patch_guest_per_regs(vcpu);
  	}
 
-	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.gisa_int.kicked_mask);
+	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.gisa_int.kicked_mask);
 
  	vcpu->arch.sie_block->icptcode = 0;
  	cpuflags = atomic_read(&vcpu->arch.sie_block->cpuflags);
arch/s390/kvm/kvm-s390.h +1 -1
index 9fad25109b0d..ecd741ee3276 100644
@@ -79,7 +79,7 @@ static inline int is_vcpu_stopped(struct kvm_vcpu *vcpu)
 
  static inline int is_vcpu_idle(struct kvm_vcpu *vcpu)
  {
-	return test_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	return test_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
  }
 
  static inline int kvm_is_ucontrol(struct kvm *kvm)

base-commit: 77dd11439b86e3f7990e4c0c9e0b67dca82750ba

#9 lore raw
Quoted 21 lines · On Mon, 31 Jan 2022 11:13:18 +0100 Petr Tesařík <[email protected]> wrote:

Hi Halil,

Dne 27. 08. 21 v 14:54 Halil Pasic napsal(a):

Quoted 9 lines

While in practice vcpu->vcpu_idx == vcpu->vcp_id is often true, it may not always be, and we must not rely on this.

Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies that code like

for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
                 vcpu = kvm_get_vcpu(kvm, vcpu_id);
		do_stuff(vcpu);
}
Quoted 5 lines

is not legit. The trouble is, we do actually use kvm->arch.idle_mask like this. To fix this problem we have two options. Either use kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id, or switch to indexing via vcpu_idx. The latter is preferable for obvious reasons.

I'm just backporting this fix to SLES 12 SP5, and I've noticed that there is still this code in __floating_irq_kick():

Quoted 8 lines
	/* find idle VCPUs first, then round robin */
	sigcpu = find_first_bit(fi->idle_mask, online_vcpus);
/* ... round robin loop removed ...
	dst_vcpu = kvm_get_vcpu(kvm, sigcpu);

It seems to me that this does exactly the thing that is not legit, but I'm no expert. Did I miss something?

We made that legit by making the N-th bit in idle_mask correspond to the vcpu whose vcpu_idx == N. The second argument of kvm_get_vcpu() is the vcpu_idx. IMHO that ain't super-intuitive because it ain't spelled out.

So before this was a mismatch (with a vcpu_id based bitmap we would have to use kvm_get_vcpu_by_id()), and with this patch applied this code becomes legit because both idle_mask and kvm_get_vcpu() operate with vcpu_idx.

Does that make sense?

I'm sorry the commit message did not convey this clearly enough...

Regards,
Halil

Quoted 99 lines

Petr T

Quoted 97 lines

Let us make switch from indexing kvm->arch.idle_mask by vcpu_id to indexing it by vcpu_idx. To keep gisa_int.kicked_mask indexed by the same index as idle_mask lets make the same change for it as well.

Fixes 1ee0bc559dc3 ("KVM: s390: get rid of local_int array")
Cc <[email protected]> # 3.15+
---
  arch/s390/include/asm/kvm_host.h |  1 +
  arch/s390/kvm/interrupt.c        | 12 ++++++------
  arch/s390/kvm/kvm-s390.c         |  2 +-
  arch/s390/kvm/kvm-s390.h         |  2 +-
  4 files changed, 9 insertions(+), 8 deletions(-)
arch/s390/include/asm/kvm_host.h +1 -0
index 161a9e12bfb8..630eab0fa176 100644
@@ -957,6 +957,7 @@ struct kvm_arch{
  	atomic64_t cmma_dirty_pages;
  	/* subset of available cpu features enabled by user space */
  	DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS);
+	/* indexed by vcpu_idx */
  	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
  	struct kvm_s390_gisa_interrupt gisa_int;
  	struct kvm_s390_pv pv;
arch/s390/kvm/interrupt.c +6 -6
index d548d60caed2..16256e17a544 100644
@@ -419,13 +419,13 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
  static void __set_cpu_idle(struct kvm_vcpu *vcpu)
  {
  	kvm_s390_set_cpuflags(vcpu, CPUSTAT_WAIT);
-	set_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	set_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
  }
 
  static void __unset_cpu_idle(struct kvm_vcpu *vcpu)
  {
  	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_WAIT);
-	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
  }
 
  static void __reset_intercept_indicators(struct kvm_vcpu *vcpu)
@@ -3050,18 +3050,18 @@ int kvm_s390_get_irq_state(struct kvm_vcpu *vcpu, __u8 __user *buf, int len)
 
  static void __airqs_kick_single_vcpu(struct kvm *kvm, u8 deliverable_mask)
  {
-	int vcpu_id, online_vcpus = atomic_read(&kvm->online_vcpus);
+	int vcpu_idx, online_vcpus = atomic_read(&kvm->online_vcpus);
  	struct kvm_s390_gisa_interrupt *gi = &kvm->arch.gisa_int;
  	struct kvm_vcpu *vcpu;
 
-	for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
-		vcpu = kvm_get_vcpu(kvm, vcpu_id);
+	for_each_set_bit(vcpu_idx, kvm->arch.idle_mask, online_vcpus) {
+		vcpu = kvm_get_vcpu(kvm, vcpu_idx);
  		if (psw_ioint_disabled(vcpu))
  			continue;
  		deliverable_mask &= (u8)(vcpu->arch.sie_block->gcr[6] >> 24);
  		if (deliverable_mask) {
  			/* lately kicked but not yet running */
-			if (test_and_set_bit(vcpu_id, gi->kicked_mask))
+			if (test_and_set_bit(vcpu_idx, gi->kicked_mask))
  				return;
  			kvm_s390_vcpu_wakeup(vcpu);
  			return;
arch/s390/kvm/kvm-s390.c +1 -1
index 4527ac7b5961..8580543c5bc3 100644
@@ -4044,7 +4044,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu)
  		kvm_s390_patch_guest_per_regs(vcpu);
  	}
 
-	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.gisa_int.kicked_mask);
+	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.gisa_int.kicked_mask);
 
  	vcpu->arch.sie_block->icptcode = 0;
  	cpuflags = atomic_read(&vcpu->arch.sie_block->cpuflags);
arch/s390/kvm/kvm-s390.h +1 -1
index 9fad25109b0d..ecd741ee3276 100644
@@ -79,7 +79,7 @@ static inline int is_vcpu_stopped(struct kvm_vcpu *vcpu)
 
  static inline int is_vcpu_idle(struct kvm_vcpu *vcpu)
  {
-	return test_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	return test_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
  }
 
  static inline int kvm_is_ucontrol(struct kvm *kvm)

base-commit: 77dd11439b86e3f7990e4c0c9e0b67dca82750ba

#10 lore raw

Hi Halil,

Dne 31. 01. 22 v 12:53 Halil Pasic napsal(a):

Quoted 37 lines
Quoted 21 lines · On Mon, 31 Jan 2022 11:13:18 +0100 Petr Tesařík <[email protected]> wrote:

Hi Halil,

Dne 27. 08. 21 v 14:54 Halil Pasic napsal(a):

Quoted 9 lines

While in practice vcpu->vcpu_idx == vcpu->vcp_id is often true, it may not always be, and we must not rely on this.

Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies that code like

for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
                  vcpu = kvm_get_vcpu(kvm, vcpu_id);
		do_stuff(vcpu);
}
Quoted 5 lines

is not legit. The trouble is, we do actually use kvm->arch.idle_mask like this. To fix this problem we have two options. Either use kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id, or switch to indexing via vcpu_idx. The latter is preferable for obvious reasons.

I'm just backporting this fix to SLES 12 SP5, and I've noticed that there is still this code in __floating_irq_kick():

Quoted 8 lines
	/* find idle VCPUs first, then round robin */
	sigcpu = find_first_bit(fi->idle_mask, online_vcpus);
/* ... round robin loop removed ...
	dst_vcpu = kvm_get_vcpu(kvm, sigcpu);

It seems to me that this does exactly the thing that is not legit, but I'm no expert. Did I miss something?

We made that legit by making the N-th bit in idle_mask correspond to the vcpu whose vcpu_idx == N. The second argument of kvm_get_vcpu() is the vcpu_idx. IMHO that ain't super-intuitive because it ain't spelled out.

Quoted 6 lines

So before this was a mismatch (with a vcpu_id based bitmap we would have to use kvm_get_vcpu_by_id()), and with this patch applied this code becomes legit because both idle_mask and kvm_get_vcpu() operate with vcpu_idx.

Does that make sense?

Yes!

Quoted 1 line

I'm sorry the commit message did not convey this clearly enough...

No, it's not your fault. I didn't pay enough attention to the change, and with vcpu_id and vcpu_idx being so similar I got confused.

In short, there's no bug now, indeed. Thanks for your patience.

Petr T

#11 lore raw
Quoted 45 lines · On Mon, 31 Jan 2022 13:09:26 +0100 Petr Tesařík <[email protected]> wrote:

Hi Halil,

Dne 31. 01. 22 v 12:53 Halil Pasic napsal(a):

Quoted 42 lines
Quoted 21 lines · On Mon, 31 Jan 2022 11:13:18 +0100 Petr Tesařík <[email protected]> wrote:

Hi Halil,

Dne 27. 08. 21 v 14:54 Halil Pasic napsal(a):

Quoted 9 lines

While in practice vcpu->vcpu_idx == vcpu->vcp_id is often true, it may not always be, and we must not rely on this.

Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies that code like

for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
                  vcpu = kvm_get_vcpu(kvm, vcpu_id);
		do_stuff(vcpu);
}
Quoted 5 lines

is not legit. The trouble is, we do actually use kvm->arch.idle_mask like this. To fix this problem we have two options. Either use kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id, or switch to indexing via vcpu_idx. The latter is preferable for obvious reasons.

I'm just backporting this fix to SLES 12 SP5, and I've noticed that there is still this code in __floating_irq_kick():

Quoted 8 lines
	/* find idle VCPUs first, then round robin */
	sigcpu = find_first_bit(fi->idle_mask, online_vcpus);
/* ... round robin loop removed ...
	dst_vcpu = kvm_get_vcpu(kvm, sigcpu);

It seems to me that this does exactly the thing that is not legit, but I'm no expert. Did I miss something?

We made that legit by making the N-th bit in idle_mask correspond to the vcpu whose vcpu_idx == N. The second argument of kvm_get_vcpu() is the vcpu_idx. IMHO that ain't super-intuitive because it ain't spelled out.

So before this was a mismatch (with a vcpu_id based bitmap we would have to use kvm_get_vcpu_by_id()), and with this patch applied this code becomes legit because both idle_mask and kvm_get_vcpu() operate with vcpu_idx.

Quoted 8 lines
Quoted 1 line

Does that make sense?

Yes!

Quoted 1 line

I'm sorry the commit message did not convey this clearly enough...

No, it's not your fault. I didn't pay enough attention to the change, and with vcpu_id and vcpu_idx being so similar I got confused.

No problem at all!

Quoted 3 lines

In short, there's no bug now, indeed. Thanks for your patience.

Thank you for being mindful when backporting!

Regards,
Halil