Skip to contentSkip to conversation

s390x: KVM: CPU Topology

PATCH In review Pierre Morel 3 participants 13 messages Jul 14, 2021 – Jul 15, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover s390x: KVM: CPU Topology

To provide Topology information to the guest through the STSI instruction, we need to forward STSI with Function Code 15 to QEMU which will take care to provide the right information to the guest.

To let the guest use the PTF instruction and ask if a topology change occured we add a new KVM capability to enable the topology facility.

Pierre Morel (2):
s390x: KVM: accept STSI for CPU topology information
KVM: s390: Topology expose TOPOLOGY facility

 arch/s390/kvm/priv.c             | 11 ++++++++++-
 arch/s390/tools/gen_facilities.c |  1 +
 2 files changed, 11 insertions(+), 1 deletion(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/2 s390x: KVM: accept STSI for CPU topology information

STSI(15.1.x) gives information on the CPU configuration topology. Let's accept the interception of STSI with the function code 15 and let the userland part of the hypervisor handle it.

---
 arch/s390/kvm/priv.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
arch/s390/kvm/priv.c +10 -1
index 9928f785c677..4ab5f8b7780e 100644
@@ -856,7 +856,7 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
 	if (vcpu->arch.sie_block->gpsw.mask & PSW_MASK_PSTATE)
 		return kvm_s390_inject_program_int(vcpu, PGM_PRIVILEGED_OP);
 
-	if (fc > 3) {
+	if (fc > 3 && fc != 15) {
 		kvm_s390_set_psw_cc(vcpu, 3);
 		return 0;
 	}
@@ -893,6 +893,15 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
 			goto out_no_data;
 		handle_stsi_3_2_2(vcpu, (void *) mem);
 		break;
+	case 15:
+		if (sel1 != 1 || sel2 < 2 || sel2 > 6)
+			goto out_no_data;
+		if (vcpu->kvm->arch.user_stsi) {
+			insert_stsi_usr_data(vcpu, operand2, ar, fc, sel1, sel2);
+			return -EREMOTE;
+		}
+		kvm_s390_set_psw_cc(vcpu, 3);
+		return 0;
 	}
 	if (kvm_s390_pv_cpu_is_protected(vcpu)) {
 		memcpy((void *)sida_origin(vcpu->arch.sie_block), (void *)mem,
Signature · 1 line
2.25.1
#2 lore raw

Re: [PATCH v1 1/2] s390x: KVM: accept STSI for CPU topology information

Quoted 29 lines · Pierre Morel

STSI(15.1.x) gives information on the CPU configuration topology. Let's accept the interception of STSI with the function code 15 and let the userland part of the hypervisor handle it.

---
  arch/s390/kvm/priv.c | 11 ++++++++++-
  1 file changed, 10 insertions(+), 1 deletion(-)
arch/s390/kvm/priv.c +3 -1
index 9928f785c677..4ab5f8b7780e 100644
@@ -856,7 +856,7 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
  	if (vcpu->arch.sie_block->gpsw.mask & PSW_MASK_PSTATE)
  		return kvm_s390_inject_program_int(vcpu, PGM_PRIVILEGED_OP);
 
-	if (fc > 3) {
+	if (fc > 3 && fc != 15) {
  		kvm_s390_set_psw_cc(vcpu, 3);
  		return 0;
  	}
@@ -893,6 +893,15 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
  			goto out_no_data;
  		handle_stsi_3_2_2(vcpu, (void *) mem);
  		break;
+	case 15:
+		if (sel1 != 1 || sel2 < 2 || sel2 > 6)

arch/s390/kvm/priv.c static int handle_stsi(struct kvm_vcpu *vcpu) · line 907

Quoted 10 lines
arch/s390/kvm/priv.c +7 -0
+			goto out_no_data;
+		if (vcpu->kvm->arch.user_stsi) {
+			insert_stsi_usr_data(vcpu, operand2, ar, fc, sel1, sel2);
+			return -EREMOTE;
+		}
+		kvm_s390_set_psw_cc(vcpu, 3);
+		return 0;
  	}
  	if (kvm_s390_pv_cpu_is_protected(vcpu)) {
  		memcpy((void *)sida_origin(vcpu->arch.sie_block), (void *)mem,

1. Setting GPRS to 0

I was wondering why we have the "vcpu->run->s.regs.gprs[0] = 0;" for existing fc 1,2,3 in case we set cc=0.

Looking at the doc, all I find is:

"CC 0: Requested configuration-level number placed in general register 0 or requested SYSIB informa- tion stored"

But I don't find where it states that we are supposed to set general register 0 to 0. Wouldn't we also have to do it for fc=15 or for none?

If fc 1,2,3 and 15 are to be handled equally, I suggest the following:

arch/s390/kvm/priv.c +16 -10
index 9928f785c677..6eb86fa58b0b 100644
@@ -893,17 +893,23 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
                        goto out_no_data;
                handle_stsi_3_2_2(vcpu, (void *) mem);
                break;
+       case 15:
+               if (sel1 != 1 || sel2 < 2 || sel2 > 6)
+                       goto out_no_data;
+               break;
        }
-       if (kvm_s390_pv_cpu_is_protected(vcpu)) {
-               memcpy((void *)sida_origin(vcpu->arch.sie_block), (void *)mem,
-                      PAGE_SIZE);
-               rc = 0;
-       } else {
-               rc = write_guest(vcpu, operand2, ar, (void *)mem, PAGE_SIZE);
-       }
-       if (rc) {
-               rc = kvm_s390_inject_prog_cond(vcpu, rc);
-               goto out;
+       if (mem) {
+               if (kvm_s390_pv_cpu_is_protected(vcpu)) {
+                       memcpy((void *)sida_origin(vcpu->arch.sie_block),
+                              (void *)mem, PAGE_SIZE);
+               } else {
+                       rc = write_guest(vcpu, operand2, ar, (void *)mem,
+                                        PAGE_SIZE);
+                       if (rc) {
+                               rc = kvm_s390_inject_prog_cond(vcpu, rc);
+                               goto out;
+                       }
+               }
        }
        if (vcpu->kvm->arch.user_stsi) {
                insert_stsi_usr_data(vcpu, operand2, ar, fc, sel1, sel2);

2. maximum-MNest facility

"
1. If the maximum-MNest facility is installed and
selector 2 exceeds the nonzero model-depen-
dent maximum-selector-2 value."

2. If the maximum-MNest facility is not installed and selector 2 is not specified as two.
"

We will we be handling the presence/absence of the maximum-MNest facility (for our guest?) in QEMU, corect?

I do wonder if we should just let any fc=15 go to user space let the whole sel1 / sel2 checking be handled there. I don't think it's a fast path after all. But no strong opinion.

How do we identify availability of maximum-MNest facility?

3. User space awareness

How can user space identify that we actually forward these intercepts? How can it enable them? The old KVM_CAP_S390_USER_STSI capability is not sufficient.

I do wonder if we want KVM_CAP_S390_USER_STSI_15 or sth like that to change the behavior once enabled by user space.

4. Without vcpu->kvm->arch.user_stsi, we indicate cc=0 to our guest, also for fc 1,2,3. Is that actually what we want? (or do we simply not care because the guest is not supposed to use stsi?)

Signature · 3 lines
Thanks,

David / dhildenb
#3 lore raw

Re: [PATCH v1 1/2] s390x: KVM: accept STSI for CPU topology information

Quoted 18 lines · David Hildenbrand
Quoted 17 lines · Pierre Morel

STSI(15.1.x) gives information on the CPU configuration topology. Let's accept the interception of STSI with the function code 15 and let the userland part of the hypervisor handle it.

---
  arch/s390/kvm/priv.c | 11 ++++++++++-
  1 file changed, 10 insertions(+), 1 deletion(-)
arch/s390/kvm/priv.c +0 -0
index 9928f785c677..4ab5f8b7780e 100644
@@ -856,7 +856,7 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
  	if (vcpu->arch.sie_block->gpsw.mask & PSW_MASK_PSTATE)
  		return kvm_s390_inject_program_int(vcpu, PGM_PRIVILEGED_OP);
 
Quoted 15 lines
Quoted 6 lines
arch/s390/kvm/priv.c +1 -1
-	if (fc > 3) {
+	if (fc > 3 && fc != 15) {
  		kvm_s390_set_psw_cc(vcpu, 3);
  		return 0;
  	}

arch/s390/kvm/priv.c static int handle_stsi(struct kvm_vcpu *vcpu) · line 901

Quoted 9 lines
arch/s390/kvm/priv.c +6 -0
@@ -893,6 +893,15 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
  			goto out_no_data;
  		handle_stsi_3_2_2(vcpu, (void *) mem);
  		break;
+	case 15:
+		if (sel1 != 1 || sel2 < 2 || sel2 > 6)
+			goto out_no_data;
+		if (vcpu->kvm->arch.user_stsi) {
+			insert_stsi_usr_data(vcpu, operand2, ar, fc, sel1, sel2);
+			return -EREMOTE;

This bypasses the trace event further down.

Quoted 53 lines

arch/s390/kvm/priv.c static int handle_stsi(struct kvm_vcpu *vcpu) · line 907

Quoted 7 lines
arch/s390/kvm/priv.c +3 -0
+		}
+		kvm_s390_set_psw_cc(vcpu, 3);
+		return 0;
  	}
  	if (kvm_s390_pv_cpu_is_protected(vcpu)) {
  		memcpy((void *)sida_origin(vcpu->arch.sie_block), (void *)mem,

1. Setting GPRS to 0

I was wondering why we have the "vcpu->run->s.regs.gprs[0] = 0;" for existing fc 1,2,3 in case we set cc=0.

Looking at the doc, all I find is:

"CC 0: Requested configuration-level number placed in general register 0 or requested SYSIB informa- tion stored"

But I don't find where it states that we are supposed to set general register 0 to 0. Wouldn't we also have to do it for fc=15 or for none?

If fc 1,2,3 and 15 are to be handled equally, I suggest the following:

arch/s390/kvm/priv.c +9 -10
index 9928f785c677..6eb86fa58b0b 100644
@@ -893,17 +893,23 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
                         goto out_no_data;
                 handle_stsi_3_2_2(vcpu, (void *) mem);
                 break;
+       case 15:
+               if (sel1 != 1 || sel2 < 2 || sel2 > 6)
+                       goto out_no_data;
+               break;
         }
-       if (kvm_s390_pv_cpu_is_protected(vcpu)) {
-               memcpy((void *)sida_origin(vcpu->arch.sie_block), (void *)mem,
-                      PAGE_SIZE);
-               rc = 0;
-       } else {
-               rc = write_guest(vcpu, operand2, ar, (void *)mem, PAGE_SIZE);
-       }
-       if (rc) {
-               rc = kvm_s390_inject_prog_cond(vcpu, rc);
-               goto out;
+       if (mem) {
+               if (kvm_s390_pv_cpu_is_protected(vcpu)) {
+                       memcpy((void *)sida_origin(vcpu->arch.sie_block),
+                              (void *)mem, PAGE_SIZE);
+               } else {

arch/s390/kvm/priv.c static int handle_stsi(struct kvm_vcpu *vcpu) · line 915

Quoted 10 lines
arch/s390/kvm/priv.c +7 -0
+                       rc = write_guest(vcpu, operand2, ar, (void *)mem,
+                                        PAGE_SIZE);
+                       if (rc) {
+                               rc = kvm_s390_inject_prog_cond(vcpu, rc);
+                               goto out;
+                       }
+               }
         }
         if (vcpu->kvm->arch.user_stsi) {
                 insert_stsi_usr_data(vcpu, operand2, ar, fc, sel1, sel2);

Something like that sounds good, the code is getting a bit convoluted.

Quoted 13 lines

2. maximum-MNest facility

"
1. If the maximum-MNest facility is installed and
selector 2 exceeds the nonzero model-depen-
dent maximum-selector-2 value."

2. If the maximum-MNest facility is not installed and selector 2 is not specified as two.
"

Quoted 6 lines

We will we be handling the presence/absence of the maximum-MNest facility (for our guest?) in QEMU, corect?

I do wonder if we should just let any fc=15 go to user space let the whole sel1 / sel2 checking be handled there. I don't think it's a fast path after all. But no strong opinion.

If that makes handling easier, I think it would be a good idea.

Quoted 9 lines

How do we identify availability of maximum-MNest facility?

3. User space awareness

How can user space identify that we actually forward these intercepts? How can it enable them? The old KVM_CAP_S390_USER_STSI capability is not sufficient.

Why do you think that it is not sufficient? USER_STSI basically says "you may get an exit that tells you about a buffer to fill in some more data for a stsi command, and we also tell you which call". If userspace does not know what to add for a certain call, it is free to just do nothing, and if it does not get some calls it would support, that should not be a problem, either?

Quoted 8 lines

I do wonder if we want KVM_CAP_S390_USER_STSI_15 or sth like that to change the behavior once enabled by user space.

4. Without vcpu->kvm->arch.user_stsi, we indicate cc=0 to our guest, also for fc 1,2,3. Is that actually what we want? (or do we simply not care because the guest is not supposed to use stsi?)

If returning an empty buffer is ok, it should not be a problem, I guess. (I have not looked yet at the actual definitions.)

#4 lore raw

Re: [PATCH v1 1/2] s390x: KVM: accept STSI for CPU topology information

Quoted 136 lines · Cornelia Huck
Quoted 18 lines · David Hildenbrand
Quoted 17 lines · Pierre Morel

STSI(15.1.x) gives information on the CPU configuration topology. Let's accept the interception of STSI with the function code 15 and let the userland part of the hypervisor handle it.

---
   arch/s390/kvm/priv.c | 11 ++++++++++-
   1 file changed, 10 insertions(+), 1 deletion(-)
arch/s390/kvm/priv.c +0 -0
index 9928f785c677..4ab5f8b7780e 100644
@@ -856,7 +856,7 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
   	if (vcpu->arch.sie_block->gpsw.mask & PSW_MASK_PSTATE)
   		return kvm_s390_inject_program_int(vcpu, PGM_PRIVILEGED_OP);
 
Quoted 15 lines
Quoted 6 lines
arch/s390/kvm/priv.c +1 -1
-	if (fc > 3) {
+	if (fc > 3 && fc != 15) {
   		kvm_s390_set_psw_cc(vcpu, 3);
   		return 0;
   	}

arch/s390/kvm/priv.c static int handle_stsi(struct kvm_vcpu *vcpu) · line 901

Quoted 9 lines
arch/s390/kvm/priv.c +6 -0
@@ -893,6 +893,15 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
   			goto out_no_data;
   		handle_stsi_3_2_2(vcpu, (void *) mem);
   		break;
+	case 15:
+		if (sel1 != 1 || sel2 < 2 || sel2 > 6)
+			goto out_no_data;
+		if (vcpu->kvm->arch.user_stsi) {
+			insert_stsi_usr_data(vcpu, operand2, ar, fc, sel1, sel2);
+			return -EREMOTE;

This bypasses the trace event further down.

Quoted 53 lines

arch/s390/kvm/priv.c static int handle_stsi(struct kvm_vcpu *vcpu) · line 907

Quoted 7 lines
arch/s390/kvm/priv.c +3 -0
+		}
+		kvm_s390_set_psw_cc(vcpu, 3);
+		return 0;
   	}
   	if (kvm_s390_pv_cpu_is_protected(vcpu)) {
   		memcpy((void *)sida_origin(vcpu->arch.sie_block), (void *)mem,

1. Setting GPRS to 0

I was wondering why we have the "vcpu->run->s.regs.gprs[0] = 0;" for existing fc 1,2,3 in case we set cc=0.

Looking at the doc, all I find is:

"CC 0: Requested configuration-level number placed in general register 0 or requested SYSIB informa- tion stored"

But I don't find where it states that we are supposed to set general register 0 to 0. Wouldn't we also have to do it for fc=15 or for none?

If fc 1,2,3 and 15 are to be handled equally, I suggest the following:

arch/s390/kvm/priv.c +9 -10
index 9928f785c677..6eb86fa58b0b 100644
@@ -893,17 +893,23 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
                          goto out_no_data;
                  handle_stsi_3_2_2(vcpu, (void *) mem);
                  break;
+       case 15:
+               if (sel1 != 1 || sel2 < 2 || sel2 > 6)
+                       goto out_no_data;
+               break;
          }
-       if (kvm_s390_pv_cpu_is_protected(vcpu)) {
-               memcpy((void *)sida_origin(vcpu->arch.sie_block), (void *)mem,
-                      PAGE_SIZE);
-               rc = 0;
-       } else {
-               rc = write_guest(vcpu, operand2, ar, (void *)mem, PAGE_SIZE);
-       }
-       if (rc) {
-               rc = kvm_s390_inject_prog_cond(vcpu, rc);
-               goto out;
+       if (mem) {
+               if (kvm_s390_pv_cpu_is_protected(vcpu)) {
+                       memcpy((void *)sida_origin(vcpu->arch.sie_block),
+                              (void *)mem, PAGE_SIZE);
+               } else {

arch/s390/kvm/priv.c static int handle_stsi(struct kvm_vcpu *vcpu) · line 915

Quoted 10 lines
arch/s390/kvm/priv.c +7 -0
+                       rc = write_guest(vcpu, operand2, ar, (void *)mem,
+                                        PAGE_SIZE);
+                       if (rc) {
+                               rc = kvm_s390_inject_prog_cond(vcpu, rc);
+                               goto out;
+                       }
+               }
          }
          if (vcpu->kvm->arch.user_stsi) {
                  insert_stsi_usr_data(vcpu, operand2, ar, fc, sel1, sel2);

Something like that sounds good, the code is getting a bit convoluted.

Quoted 13 lines

2. maximum-MNest facility

"
1. If the maximum-MNest facility is installed and
selector 2 exceeds the nonzero model-depen-
dent maximum-selector-2 value."

2. If the maximum-MNest facility is not installed and selector 2 is not specified as two.
"

Quoted 6 lines

We will we be handling the presence/absence of the maximum-MNest facility (for our guest?) in QEMU, corect?

I do wonder if we should just let any fc=15 go to user space let the whole sel1 / sel2 checking be handled there. I don't think it's a fast path after all. But no strong opinion.

If that makes handling easier, I think it would be a good idea.

Quoted 9 lines

How do we identify availability of maximum-MNest facility?

3. User space awareness

How can user space identify that we actually forward these intercepts? How can it enable them? The old KVM_CAP_S390_USER_STSI capability is not sufficient.

Quoted 6 lines

Why do you think that it is not sufficient? USER_STSI basically says "you may get an exit that tells you about a buffer to fill in some more data for a stsi command, and we also tell you which call". If userspace does not know what to add for a certain call, it is free to just do nothing, and if it does not get some calls it would support, that should not be a problem, either?

If you migrate your VM from machine a to machine b, from kernel a to kernel b, and kernel b does not trigger exits to user space for fc=15, how could QEMU spot and catch the different capabilities to make sure the guest can continue using the feature?

Signature · 3 lines
Thanks,

David / dhildenb
#5 lore raw

Re: [PATCH v1 1/2] s390x: KVM: accept STSI for CPU topology information

Quoted 58 lines · David Hildenbrand
Quoted 50 lines · Cornelia Huck
Quoted 18 lines · David Hildenbrand
Quoted 17 lines · Pierre Morel

STSI(15.1.x) gives information on the CPU configuration topology. Let's accept the interception of STSI with the function code 15 and let the userland part of the hypervisor handle it.

---
   arch/s390/kvm/priv.c | 11 ++++++++++-
   1 file changed, 10 insertions(+), 1 deletion(-)
arch/s390/kvm/priv.c +0 -0
index 9928f785c677..4ab5f8b7780e 100644
@@ -856,7 +856,7 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
   	if (vcpu->arch.sie_block->gpsw.mask & PSW_MASK_PSTATE)
   		return kvm_s390_inject_program_int(vcpu, PGM_PRIVILEGED_OP);
 
Quoted 15 lines
Quoted 6 lines
arch/s390/kvm/priv.c +1 -1
-	if (fc > 3) {
+	if (fc > 3 && fc != 15) {
   		kvm_s390_set_psw_cc(vcpu, 3);
   		return 0;
   	}

arch/s390/kvm/priv.c static int handle_stsi(struct kvm_vcpu *vcpu) · line 901

Quoted 9 lines
arch/s390/kvm/priv.c +6 -0
@@ -893,6 +893,15 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
   			goto out_no_data;
   		handle_stsi_3_2_2(vcpu, (void *) mem);
   		break;
+	case 15:
+		if (sel1 != 1 || sel2 < 2 || sel2 > 6)
+			goto out_no_data;
+		if (vcpu->kvm->arch.user_stsi) {
+			insert_stsi_usr_data(vcpu, operand2, ar, fc, sel1, sel2);
+			return -EREMOTE;

This bypasses the trace event further down.

Quoted 11 lines

arch/s390/kvm/priv.c static int handle_stsi(struct kvm_vcpu *vcpu) · line 907

Quoted 6 lines
arch/s390/kvm/priv.c +3 -0
+		}
+		kvm_s390_set_psw_cc(vcpu, 3);
+		return 0;
   	}
   	if (kvm_s390_pv_cpu_is_protected(vcpu)) {
   		memcpy((void *)sida_origin(vcpu->arch.sie_block), (void *)mem,

3. User space awareness

How can user space identify that we actually forward these intercepts? How can it enable them? The old KVM_CAP_S390_USER_STSI capability is not sufficient.

Quoted 6 lines

Why do you think that it is not sufficient? USER_STSI basically says "you may get an exit that tells you about a buffer to fill in some more data for a stsi command, and we also tell you which call". If userspace does not know what to add for a certain call, it is free to just do nothing, and if it does not get some calls it would support, that should not be a problem, either?

Quoted 4 lines

If you migrate your VM from machine a to machine b, from kernel a to kernel b, and kernel b does not trigger exits to user space for fc=15, how could QEMU spot and catch the different capabilities to make sure the guest can continue using the feature?

Wouldn't that imply that the USER_STSI feature, in the function-agnostic way it is documented, was broken from the start?

Hm. Maybe we need some kind of facility where userspace can query the kernel and gets a list of the stsi subcodes it may get exits for, and possibly fail to start the migration. Having a new capability to be enabled for every new subcode feels like overkill. I don't think we can pass a payload ("enable these subfunctions") to a cap.

Or can we tie a subcode to another feature, like the mystery cap that was mentioned in the description, but does not seem to appear in the code?

#6 lore raw

Re: [PATCH v1 1/2] s390x: KVM: accept STSI for CPU topology information

Quoted 60 lines · Cornelia Huck
Quoted 58 lines · David Hildenbrand
Quoted 56 lines · Cornelia Huck
Quoted 18 lines · David Hildenbrand
Quoted 17 lines · Pierre Morel

STSI(15.1.x) gives information on the CPU configuration topology. Let's accept the interception of STSI with the function code 15 and let the userland part of the hypervisor handle it.

---
    arch/s390/kvm/priv.c | 11 ++++++++++-
    1 file changed, 10 insertions(+), 1 deletion(-)
arch/s390/kvm/priv.c +0 -0
index 9928f785c677..4ab5f8b7780e 100644
@@ -856,7 +856,7 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
    	if (vcpu->arch.sie_block->gpsw.mask & PSW_MASK_PSTATE)
    		return kvm_s390_inject_program_int(vcpu, PGM_PRIVILEGED_OP);
 
Quoted 15 lines
Quoted 6 lines
arch/s390/kvm/priv.c +1 -1
-	if (fc > 3) {
+	if (fc > 3 && fc != 15) {
    		kvm_s390_set_psw_cc(vcpu, 3);
    		return 0;
    	}

arch/s390/kvm/priv.c static int handle_stsi(struct kvm_vcpu *vcpu) · line 901

Quoted 9 lines
arch/s390/kvm/priv.c +6 -0
@@ -893,6 +893,15 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
    			goto out_no_data;
    		handle_stsi_3_2_2(vcpu, (void *) mem);
    		break;
+	case 15:
+		if (sel1 != 1 || sel2 < 2 || sel2 > 6)
+			goto out_no_data;
+		if (vcpu->kvm->arch.user_stsi) {
+			insert_stsi_usr_data(vcpu, operand2, ar, fc, sel1, sel2);
+			return -EREMOTE;

This bypasses the trace event further down.

Quoted 11 lines

arch/s390/kvm/priv.c static int handle_stsi(struct kvm_vcpu *vcpu) · line 907

Quoted 6 lines
arch/s390/kvm/priv.c +3 -0
+		}
+		kvm_s390_set_psw_cc(vcpu, 3);
+		return 0;
    	}
    	if (kvm_s390_pv_cpu_is_protected(vcpu)) {
    		memcpy((void *)sida_origin(vcpu->arch.sie_block), (void *)mem,

3. User space awareness

How can user space identify that we actually forward these intercepts? How can it enable them? The old KVM_CAP_S390_USER_STSI capability is not sufficient.

Why do you think that it is not sufficient? USER_STSI basically says "you may get an exit that tells you about a buffer to fill in some more data for a stsi command, and we also tell you which call". If userspace does not know what to add for a certain call, it is free to just do nothing, and if it does not get some calls it would support, that should not be a problem, either?

Quoted 7 lines
Quoted 4 lines

If you migrate your VM from machine a to machine b, from kernel a to kernel b, and kernel b does not trigger exits to user space for fc=15, how could QEMU spot and catch the different capabilities to make sure the guest can continue using the feature?

Wouldn't that imply that the USER_STSI feature, in the function-agnostic way it is documented, was broken from the start?

Likely. We should have forwarded everything to user space most probably and not try being smart in the kernel.

Quoted 6 lines

Hm. Maybe we need some kind of facility where userspace can query the kernel and gets a list of the stsi subcodes it may get exits for, and possibly fail to start the migration. Having a new capability to be enabled for every new subcode feels like overkill. I don't think we can pass a payload ("enable these subfunctions") to a cap.

Maybe a new capability that forwards everything to user space when enabled, and lets user space handle errors.

Or a specialized one only to unlock fc=15.

Signature · 3 lines
Thanks,

David / dhildenb
#7 lore raw

Re: [PATCH v1 1/2] s390x: KVM: accept STSI for CPU topology information

Quoted 78 lines · David Hildenbrand
Quoted 60 lines · Cornelia Huck
Quoted 58 lines · David Hildenbrand
Quoted 56 lines · Cornelia Huck
Quoted 25 lines · David Hildenbrand
Quoted 24 lines · Pierre Morel

STSI(15.1.x) gives information on the CPU configuration topology. Let's accept the interception of STSI with the function code 15 and let the userland part of the hypervisor handle it.

---
    arch/s390/kvm/priv.c | 11 ++++++++++-

    1 file changed, 10 insertions(+), 1 deletion(-)
arch/s390/kvm/priv.c +1 -1
index 9928f785c677..4ab5f8b7780e 100644
@@ -856,7 +856,7 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
         if (vcpu->arch.sie_block->gpsw.mask & PSW_MASK_PSTATE)
             return kvm_s390_inject_program_int(vcpu, PGM_PRIVILEGED_OP);
-    if (fc > 3) {
+    if (fc > 3 && fc != 15) {
             kvm_s390_set_psw_cc(vcpu, 3);
             return 0;
         }
@@ -893,6 +893,15 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
                 goto out_no_data;
             handle_stsi_3_2_2(vcpu, (void *) mem);
Quoted 8 lines

arch/s390/kvm/priv.c static int handle_stsi(struct kvm_vcpu *vcpu)

Quoted 8 lines
arch/s390/kvm/priv.c +6 -0
             break;
+    case 15:
+        if (sel1 != 1 || sel2 < 2 || sel2 > 6)
+            goto out_no_data;
+        if (vcpu->kvm->arch.user_stsi) {
+            insert_stsi_usr_data(vcpu, operand2, ar, fc, sel1, 
 sel2);
+            return -EREMOTE;

This bypasses the trace event further down.

Quoted 11 lines
Quoted 6 lines
arch/s390/kvm/priv.c +3 -0
+        }
+        kvm_s390_set_psw_cc(vcpu, 3);
+        return 0;
         }
         if (kvm_s390_pv_cpu_is_protected(vcpu)) {

            memcpy((void *)sida_origin(vcpu->arch.sie_block), (void *)mem,

3. User space awareness

How can user space identify that we actually forward these intercepts? How can it enable them? The old KVM_CAP_S390_USER_STSI capability is not sufficient.

Why do you think that it is not sufficient? USER_STSI basically says "you may get an exit that tells you about a buffer to fill in some more data for a stsi command, and we also tell you which call". If userspace does not know what to add for a certain call, it is free to just do nothing, and if it does not get some calls it would support, that should not be a problem, either?

Quoted 7 lines
Quoted 4 lines

If you migrate your VM from machine a to machine b, from kernel a to kernel b, and kernel b does not trigger exits to user space for fc=15, how could QEMU spot and catch the different capabilities to make sure the guest can continue using the feature?

Wouldn't that imply that the USER_STSI feature, in the function-agnostic way it is documented, was broken from the start?

Likely. We should have forwarded everything to user space most probably and not try being smart in the kernel.

Quoted 6 lines

Hm. Maybe we need some kind of facility where userspace can query the kernel and gets a list of the stsi subcodes it may get exits for, and possibly fail to start the migration. Having a new capability to be enabled for every new subcode feels like overkill. I don't think we can pass a payload ("enable these subfunctions") to a cap.

Quoted 4 lines

Maybe a new capability that forwards everything to user space when enabled, and lets user space handle errors.

Or a specialized one only to unlock fc=15.

I think the lack of a good comment in patch 2/2 is the problem. Facility 11 belong to CPU model and enables both the STSI 15 and the PFT instructions

Sorry about that.

Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
#8 lore raw

Re: [PATCH v1 1/2] s390x: KVM: accept STSI for CPU topology information

In reply to Cornelia Huck
Quoted 29 lines · Cornelia Huck
Quoted 27 lines · David Hildenbrand
Quoted 26 lines · Pierre Morel

STSI(15.1.x) gives information on the CPU configuration topology. Let's accept the interception of STSI with the function code 15 and let the userland part of the hypervisor handle it.

---
   arch/s390/kvm/priv.c | 11 ++++++++++-
   1 file changed, 10 insertions(+), 1 deletion(-)
arch/s390/kvm/priv.c +1 -1
index 9928f785c677..4ab5f8b7780e 100644
@@ -856,7 +856,7 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
   	if (vcpu->arch.sie_block->gpsw.mask & PSW_MASK_PSTATE)
   		return kvm_s390_inject_program_int(vcpu, PGM_PRIVILEGED_OP);
 
-	if (fc > 3) {
+	if (fc > 3 && fc != 15) {
   		kvm_s390_set_psw_cc(vcpu, 3);
   		return 0;
   	}
@@ -893,6 +893,15 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
   			goto out_no_data;
   		handle_stsi_3_2_2(vcpu, (void *) mem);
   		break;
Quoted 9 lines
Quoted 6 lines

arch/s390/kvm/priv.c static int handle_stsi(struct kvm_vcpu *vcpu)

Quoted 6 lines
arch/s390/kvm/priv.c +6 -0
+	case 15:
+		if (sel1 != 1 || sel2 < 2 || sel2 > 6)
+			goto out_no_data;
+		if (vcpu->kvm->arch.user_stsi) {
+			insert_stsi_usr_data(vcpu, operand2, ar, fc, sel1, sel2);
+			return -EREMOTE;

This bypasses the trace event further down.

Right, I can add a trace.
Note that we had no trace in the past.

Quoted 57 lines
Quoted 57 lines

arch/s390/kvm/priv.c static int handle_stsi(struct kvm_vcpu *vcpu) · line 907

Quoted 7 lines
arch/s390/kvm/priv.c +3 -0
+		}
+		kvm_s390_set_psw_cc(vcpu, 3);
+		return 0;
   	}
   	if (kvm_s390_pv_cpu_is_protected(vcpu)) {
   		memcpy((void *)sida_origin(vcpu->arch.sie_block), (void *)mem,

1. Setting GPRS to 0

I was wondering why we have the "vcpu->run->s.regs.gprs[0] = 0;" for existing fc 1,2,3 in case we set cc=0.

Looking at the doc, all I find is:

"CC 0: Requested configuration-level number placed in general register 0 or requested SYSIB informa- tion stored"

But I don't find where it states that we are supposed to set general register 0 to 0. Wouldn't we also have to do it for fc=15 or for none?

If fc 1,2,3 and 15 are to be handled equally, I suggest the following:

arch/s390/kvm/priv.c +13 -10
index 9928f785c677..6eb86fa58b0b 100644
@@ -893,17 +893,23 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
                          goto out_no_data;
                  handle_stsi_3_2_2(vcpu, (void *) mem);
                  break;
+       case 15:
+               if (sel1 != 1 || sel2 < 2 || sel2 > 6)
+                       goto out_no_data;
+               break;
          }
-       if (kvm_s390_pv_cpu_is_protected(vcpu)) {
-               memcpy((void *)sida_origin(vcpu->arch.sie_block), (void *)mem,
-                      PAGE_SIZE);
-               rc = 0;
-       } else {
-               rc = write_guest(vcpu, operand2, ar, (void *)mem, PAGE_SIZE);
-       }
-       if (rc) {
-               rc = kvm_s390_inject_prog_cond(vcpu, rc);
-               goto out;
+       if (mem) {
+               if (kvm_s390_pv_cpu_is_protected(vcpu)) {
+                       memcpy((void *)sida_origin(vcpu->arch.sie_block),
+                              (void *)mem, PAGE_SIZE);
+               } else {
+                       rc = write_guest(vcpu, operand2, ar, (void *)mem,
+                                        PAGE_SIZE);
+                       if (rc) {
+                               rc = kvm_s390_inject_prog_cond(vcpu, rc);
Quoted 9 lines

arch/s390/kvm/priv.c static int handle_stsi(struct kvm_vcpu *vcpu) · line 915

Quoted 6 lines
arch/s390/kvm/priv.c +3 -0
+                               goto out;
+                       }
+               }
          }
          if (vcpu->kvm->arch.user_stsi) {
                  insert_stsi_usr_data(vcpu, operand2, ar, fc, sel1, sel2);

Something like that sounds good, the code is getting a bit convoluted.

OK for me, in that case we can also suppress the check for FC=15 and let that to user space as it was suggested in a previous comment

Quoted 13 lines
Quoted 13 lines

2. maximum-MNest facility

"
1. If the maximum-MNest facility is installed and
selector 2 exceeds the nonzero model-depen-
dent maximum-selector-2 value."

2. If the maximum-MNest facility is not installed and selector 2 is not specified as two.
"

Quoted 8 lines
Quoted 6 lines

We will we be handling the presence/absence of the maximum-MNest facility (for our guest?) in QEMU, corect?

I do wonder if we should just let any fc=15 go to user space let the whole sel1 / sel2 checking be handled there. I don't think it's a fast path after all. But no strong opinion.

If that makes handling easier, I think it would be a good idea.

OK too

Quoted 23 lines
Quoted 10 lines

How do we identify availability of maximum-MNest facility?

3. User space awareness

How can user space identify that we actually forward these intercepts? How can it enable them? The old KVM_CAP_S390_USER_STSI capability is not sufficient.

Why do you think that it is not sufficient? USER_STSI basically says "you may get an exit that tells you about a buffer to fill in some more data for a stsi command, and we also tell you which call". If userspace does not know what to add for a certain call, it is free to just do nothing, and if it does not get some calls it would support, that should not be a problem, either?

Quoted 5 lines

I do wonder if we want KVM_CAP_S390_USER_STSI_15 or sth like that to change the behavior once enabled by user space.

Quoted 7 lines
Quoted 3 lines

4. Without vcpu->kvm->arch.user_stsi, we indicate cc=0 to our guest, also for fc 1,2,3. Is that actually what we want? (or do we simply not care because the guest is not supposed to use stsi?)

If returning an empty buffer is ok, it should not be a problem, I guess. (I have not looked yet at the actual definitions.)

When user_stsi is 0 for fc 1,2,3 the buffer is filled in the kernel, for 15 the kernel can not do this.

Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
#9 lore raw

Re: [PATCH v1 1/2] s390x: KVM: accept STSI for CPU topology information

In reply to David Hildenbrand
Quoted 47 lines · David Hildenbrand
Quoted 28 lines · Pierre Morel

STSI(15.1.x) gives information on the CPU configuration topology. Let's accept the interception of STSI with the function code 15 and let the userland part of the hypervisor handle it.

---
  arch/s390/kvm/priv.c | 11 ++++++++++-

  1 file changed, 10 insertions(+), 1 deletion(-)
arch/s390/kvm/priv.c +3 -1
index 9928f785c677..4ab5f8b7780e 100644
@@ -856,7 +856,7 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
       if (vcpu->arch.sie_block->gpsw.mask & PSW_MASK_PSTATE)
           return kvm_s390_inject_program_int(vcpu, PGM_PRIVILEGED_OP);
-    if (fc > 3) {
+    if (fc > 3 && fc != 15) {
           kvm_s390_set_psw_cc(vcpu, 3);
           return 0;
       }
@@ -893,6 +893,15 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
               goto out_no_data;
           handle_stsi_3_2_2(vcpu, (void *) mem);
           break;
+    case 15:
+        if (sel1 != 1 || sel2 < 2 || sel2 > 6)

arch/s390/kvm/priv.c static int handle_stsi(struct kvm_vcpu *vcpu) · line 907

Quoted 10 lines
arch/s390/kvm/priv.c +7 -0
+            goto out_no_data;
+        if (vcpu->kvm->arch.user_stsi) {
+            insert_stsi_usr_data(vcpu, operand2, ar, fc, sel1, sel2);
+            return -EREMOTE;
+        }
+        kvm_s390_set_psw_cc(vcpu, 3);
+        return 0;
       }
       if (kvm_s390_pv_cpu_is_protected(vcpu)) {
           memcpy((void *)sida_origin(vcpu->arch.sie_block), (void *)mem,

1. Setting GPRS to 0

I was wondering why we have the "vcpu->run->s.regs.gprs[0] = 0;" for existing fc 1,2,3 in case we set cc=0.

Looking at the doc, all I find is:

Quoted 7 lines

"CC 0: Requested configuration-level number placed in general register 0 or requested SYSIB informa- tion stored"

But I don't find where it states that we are supposed to set general register 0 to 0. Wouldn't we also have to do it for fc=15 or for none?

It is only needed for fc 0, I do not see any reference to register 0 being set on output for other function code.

Quoted 52 lines

If fc 1,2,3 and 15 are to be handled equally, I suggest the following:

arch/s390/kvm/priv.c +16 -10
index 9928f785c677..6eb86fa58b0b 100644
@@ -893,17 +893,23 @@ static int handle_stsi(struct kvm_vcpu *vcpu)
                         goto out_no_data;
                 handle_stsi_3_2_2(vcpu, (void *) mem);
                 break;
+       case 15:
+               if (sel1 != 1 || sel2 < 2 || sel2 > 6)
+                       goto out_no_data;
+               break;
         }
-       if (kvm_s390_pv_cpu_is_protected(vcpu)) {
-               memcpy((void *)sida_origin(vcpu->arch.sie_block), (void 
 *)mem,
-                      PAGE_SIZE);
-               rc = 0;
-       } else {
-               rc = write_guest(vcpu, operand2, ar, (void *)mem, 
 PAGE_SIZE);
-       }
-       if (rc) {
-               rc = kvm_s390_inject_prog_cond(vcpu, rc);
-               goto out;
+       if (mem) {
+               if (kvm_s390_pv_cpu_is_protected(vcpu)) {
+                       memcpy((void *)sida_origin(vcpu->arch.sie_block),
+                              (void *)mem, PAGE_SIZE);
+               } else {
+                       rc = write_guest(vcpu, operand2, ar, (void *)mem,
+                                        PAGE_SIZE);
+                       if (rc) {
+                               rc = kvm_s390_inject_prog_cond(vcpu, rc);
+                               goto out;
+                       }
+               }
         }

        if (vcpu->kvm->arch.user_stsi) { insert_stsi_usr_data(vcpu, operand2, ar, fc, sel1, sel2);

2. maximum-MNest facility

"
1. If the maximum-MNest facility is installed and
selector 2 exceeds the nonzero model-depen-
dent maximum-selector-2 value."

Quoted 6 lines

2. If the maximum-MNest facility is not installed and selector 2 is not specified as two.
"

We will we be handling the presence/absence of the maximum-MNest facility (for our guest?) in QEMU, corect?

Yes

Quoted 4 lines

I do wonder if we should just let any fc=15 go to user space let the whole sel1 / sel2 checking be handled there. I don't think it's a fast path after all. But no strong opinion.

We can, here it is only tested about values that are always false and not depending on MNEST. sel 1 is always 1
sel 2 can only be valid between 2 and 6

Quoted 3 lines

How do we identify availability of maximum-MNest facility?

With SCLP.
Here is no test on MNEST

Quoted 10 lines

3. User space awareness

How can user space identify that we actually forward these intercepts? How can it enable them? The old KVM_CAP_S390_USER_STSI capability is not sufficient.

I do wonder if we want KVM_CAP_S390_USER_STSI_15 or sth like that to change the behavior once enabled by user space.

Userland can check the configuration toplogy facility to see if the STSI-15 is available. Sorry about the old commit message in the next patch it does not make things clear.

Configuration topology facility, 11, enables function code 15 in the STSI instruction.

Quoted 5 lines

4. Without vcpu->kvm->arch.user_stsi, we indicate cc=0 to our guest, also for fc 1,2,3. Is that actually what we want? (or do we simply not care because the guest is not supposed to use stsi?)

Without the user_stsi, the information is gathered in the kernel.
For what can be gathered.
Patch e44fc8c9dab215ac0e from Ekaterina Tumanova introduced post handlers for STSI in user space.

Only what need to be done in userspace is checking the user_stsi.

Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
#10 lore raw

PATCH 2/2 KVM: s390: Topology expose TOPOLOGY facility

In reply to Pierre Morel

We add the KVM extension KVM_CAP_S390_CPU_TOPOLOGY, this will allow the userland hypervisor to handle the interception of the PTF (Perform topology Function) instruction.

---
 arch/s390/tools/gen_facilities.c | 1 +
 1 file changed, 1 insertion(+)
arch/s390/tools/gen_facilities.c +1 -0
index 606324e56e4e..2c260eb22bae 100644
@@ -112,6 +112,7 @@ static struct facility_def facility_defs[] = {
 
 		.name = "FACILITIES_KVM_CPUMODEL",
 		.bits = (int[]){
+			11, /* configuration topology facility */
 			12, /* AP Query Configuration Information */
 			15, /* AP Facilities Test */
 			156, /* etoken facility */
Signature · 1 line
2.25.1
#11 lore raw

Re: [PATCH v1 2/2] KVM: s390: Topology expose TOPOLOGY facility

Quoted 3 lines · Pierre Morel

We add the KVM extension KVM_CAP_S390_CPU_TOPOLOGY, this will allow the userland hypervisor to handle the interception of the PTF (Perform topology Function) instruction.

Ehm, no you don't add any new capability. Or my eyes are too tired to spot it :)

Quoted 19 lines
---
  arch/s390/tools/gen_facilities.c | 1 +
  1 file changed, 1 insertion(+)
arch/s390/tools/gen_facilities.c +1 -0
index 606324e56e4e..2c260eb22bae 100644
@@ -112,6 +112,7 @@ static struct facility_def facility_defs[] = {
 
  		.name = "FACILITIES_KVM_CPUMODEL",
  		.bits = (int[]){
+			11, /* configuration topology facility */
  			12, /* AP Query Configuration Information */
  			15, /* AP Facilities Test */
  			156, /* etoken facility */
Signature · 3 lines
Thanks,

David / dhildenb
#12 lore raw

Re: [PATCH v1 2/2] KVM: s390: Topology expose TOPOLOGY facility

Quoted 6 lines · David Hildenbrand
Quoted 3 lines · Pierre Morel

We add the KVM extension KVM_CAP_S390_CPU_TOPOLOGY, this will allow the userland hypervisor to handle the interception of the PTF (Perform topology Function) instruction.

Ehm, no you don't add any new capability. Or my eyes are too tired to spot it :)

hum, yes, sorry, seems I kept my old commit message as I let fall the capability after internal reviews.

Quoted 21 lines
Quoted 19 lines

---
  arch/s390/tools/gen_facilities.c | 1 +

  1 file changed, 1 insertion(+)
arch/s390/tools/gen_facilities.c +1 -0
index 606324e56e4e..2c260eb22bae 100644
@@ -112,6 +112,7 @@ static struct facility_def facility_defs[] = {
           .name = "FACILITIES_KVM_CPUMODEL",
           .bits = (int[]){
+            11, /* configuration topology facility */

              12, /* AP Query Configuration Information */ 15, /* AP Facilities Test */
              156, /* etoken facility */

Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen