Skip to contentSkip to conversation

s390: Enable AP instructions for pv-guests

PATCH In review Steffen Eiden 5 participants 11 messages Aug 23, 2023 – Sep 6, 2023

lore
Share & tools
lore
#0 lore raw

v4 cover s390: Enable AP instructions for pv-guests

This series enables general QEMU support for AP pass-through for Secure Execution guests (pv-guests).

To enable AP-PT on pv-guests QEMU has to turn on the corresponding bits in the KVM CPU-model[1] if the CPU firmware supports it. However, it only makes sense to turn on AP-PT if the QEMU user enabled (general) AP for that guest.

See: https://lore.kernel.org/linux-s390/[email protected]/T/#t

The series consists of five patches:
1/2) fixes from Janosch for AP handling
3) update kvm-s390 header for this series (NOTFORMERGE)
4) small cleanup for kvm_s390_set_attr() refactor code to add ap_available() and ap_enabled()
5) Add UV_CALL CPU model enablement

since v3:
- nitfixes (Thomas)
- r-b from Michael&Thomas

since v2:
- add fixes for AP from Janosch
- rename *UV_CALL* to UV_FEAT_GUEST (Janosch)
- early return on some functions (Janosch)
- add r-b from Michael (Patch 4)
- mark linux header update as NOTFORMERGE

since v1:

- removed the new features from the default gen16 model
- updated KVM-headers to match KVM series v3 [1]
- applied review comments from Thomas

Steffen

Janosch Frank (2):
s390x/ap: fix missing subsystem reset registration
s390x: switch pv and subsystem reset ordering on reboot

Steffen Eiden (3):
NOTFORMERGE update linux-headers/asm-s390/kvm.h
target/s390x/kvm: Refactor AP functionalities
target/s390x: AP-passthrough for PV guests

 hw/s390x/s390-virtio-ccw.c          |  7 ++-
 linux-headers/asm-s390/kvm.h        | 16 +++++
 target/s390x/cpu_features.h         |  1 +
 target/s390x/cpu_features_def.h.inc |  4 ++
 target/s390x/cpu_models.c           |  2 +
 target/s390x/gen-features.c         |  2 +
 target/s390x/kvm/kvm.c              | 94 ++++++++++++++++++++++++++---
 7 files changed, 116 insertions(+), 10 deletions(-)
Signature · 1 line
2.41.0
#1 lore raw

PATCH v4 1/5 s390x/ap: fix missing subsystem reset registration

From: Janosch Frank <[email protected]>

A subsystem reset contains a reset of AP resources which has been missing. Adding the AP bridge to the list of device types that need reset fixes this issue.

Fixes a51b3153 ("s390x/ap: base Adjunct Processor (AP) object model")
---
 hw/s390x/s390-virtio-ccw.c | 1 +
 1 file changed, 1 insertion(+)
hw/s390x/s390-virtio-ccw.c +1 -0
index 4516d73ff5..4b36c9970e 100644
@@ -109,6 +109,7 @@ static const char *const reset_dev_types[] = {
     "s390-flic",
     "diag288",
     TYPE_S390_PCI_HOST_BRIDGE,
+    TYPE_AP_BRIDGE,
 };
 
 static void subsystem_reset(void)
Signature · 1 line
2.41.0
#2 lore raw

PATCH v4 2/5 s390x: switch pv and subsystem reset ordering on reboot

In reply to Steffen Eiden

From: Janosch Frank <[email protected]>

Bound APQNs have to be reset before tearing down the secure config via s390_machine_unprotect(). Otherwise the Ultravisor will return a error code.

So let's switch the ordering around to make that happen.

---
 hw/s390x/s390-virtio-ccw.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
hw/s390x/s390-virtio-ccw.c +3 -3
index 4b36c9970e..795dd53d68 100644
@@ -442,13 +442,13 @@ static void s390_machine_reset(MachineState *machine, ShutdownCause reason)
     switch (reset_type) {
     case S390_RESET_EXTERNAL:
     case S390_RESET_REIPL:
+        qemu_devices_reset(reason);
+        s390_crypto_reset();
+
         if (s390_is_pv()) {
             s390_machine_unprotect(ms);
         }
 
-        qemu_devices_reset(reason);
-        s390_crypto_reset();
-
         /* configure and start the ipl CPU only */
         run_on_cpu(cs, s390_do_cpu_ipl, RUN_ON_CPU_NULL);
         break;
Signature · 1 line
2.41.0
#3 lore raw

Re: [PATCH v4 2/5] s390x: switch pv and subsystem reset ordering on reboot

Quoted 38 lines · Steffen Eiden

From: Janosch Frank <[email protected]>

Bound APQNs have to be reset before tearing down the secure config via s390_machine_unprotect(). Otherwise the Ultravisor will return a error code.

So let's switch the ordering around to make that happen.

---
 hw/s390x/s390-virtio-ccw.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
hw/s390x/s390-virtio-ccw.c +3 -3
index 4b36c9970e..795dd53d68 100644
@@ -442,13 +442,13 @@ static void s390_machine_reset(MachineState *machine, ShutdownCause reason)
     switch (reset_type) {
     case S390_RESET_EXTERNAL:
     case S390_RESET_REIPL:
+        qemu_devices_reset(reason);
+        s390_crypto_reset();
+
         if (s390_is_pv()) {
             s390_machine_unprotect(ms);
         }
 
-        qemu_devices_reset(reason);
-        s390_crypto_reset();
-
         /* configure and start the ipl CPU only */
         run_on_cpu(cs, s390_do_cpu_ipl, RUN_ON_CPU_NULL);
         break;
Signature · 1 line
2.41.0

Unfortunately, this breaks things for me. You can reproduce the problem easily… Start an SE guest via direct kernel boot and reboot the guest after the guest has booted.

e.g.

$ sudo qemu-system-s390x -smp 2 --machine accel=kvm,aes-key-wrap=off -kernel /var/lib/libvirt/images/hades/vmlinux-s390x.se.img -m 2048 -nographic -nodefaults -chardev stdio,id=c1,mux=on  -device sclpconsole,chardev=c1 -append "earlyprintk"
…
#  reboot  # (console in the guest)
…
[    3.966496] systemd-shutdown[1]: Syncing filesystems and block devices.
[    3.966606] systemd-shutdown[1]: Rebooting.
qemu-system-s390x: CPU reset failed on CPU 0 type aec4
qemu-system-s390x: CPU reset failed on CPU 1 type aec4
qemu-system-s390x: KVM PV command 4 (KVM_PV_VERIFY) failed: header rc 102 rrc 1b IOCTL rc: -22
Protected boot has failed: 0xa02
Signature · 8 lines
Kind regards / Beste Grüße
   Marc Hartmayer

IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: David Faller
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
#4 lore raw

Re: [PATCH v4 2/5] s390x: switch pv and subsystem reset ordering on reboot

Quoted 30 lines · Marc Hartmayer
Quoted 29 lines · Steffen Eiden

From: Janosch Frank <[email protected]>

Bound APQNs have to be reset before tearing down the secure config via s390_machine_unprotect(). Otherwise the Ultravisor will return a error code.

So let's switch the ordering around to make that happen.

---
  hw/s390x/s390-virtio-ccw.c | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)
hw/s390x/s390-virtio-ccw.c +3 -0
index 4b36c9970e..795dd53d68 100644
@@ -442,13 +442,13 @@ static void s390_machine_reset(MachineState *machine, ShutdownCause reason)
      switch (reset_type) {
      case S390_RESET_EXTERNAL:
      case S390_RESET_REIPL:
+        qemu_devices_reset(reason);
+        s390_crypto_reset();
+
          if (s390_is_pv()) {
              s390_machine_unprotect(ms);
          }
 
Quoted 13 lines
Quoted 9 lines
hw/s390x/s390-virtio-ccw.c +0 -3
-        qemu_devices_reset(reason);
-        s390_crypto_reset();
-
          /* configure and start the ipl CPU only */
          run_on_cpu(cs, s390_do_cpu_ipl, RUN_ON_CPU_NULL);
          break;
Signature · 1 line
2.41.0

Unfortunately, this breaks things for me. You can reproduce the problem easily… Start an SE guest via direct kernel boot and reboot the guest after the guest has booted.

Seems like we didn't test a direct kernel boot reboot... I'm looking into it.

#5 lore raw

PATCH 1/1 s390x: do a subsystem reset before the unprotect on reboot

Bound APQNs have to be reset before tearing down the secure config via s390_machine_unprotect(). Otherwise the Ultravisor will return a error code.

So let's do a subsystem_reset() which includes a AP reset before the unprotect call. We'll do a full device_reset() afterwards which will reset some devices twice. That's ok since we can't move the device_reset() before the unprotect as it includes a CPU clear reset which the Ultravisor does not expect at that point in time.

---

I'm not able to test this for the PV AP case right new, that has to wait to early next week. However Marc told me that the non-AP PV test works fine now.

---
 hw/s390x/s390-virtio-ccw.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
hw/s390x/s390-virtio-ccw.c +10 -0
index 3dd0b2372d..2d75f2131f 100644
@@ -438,10 +438,20 @@ static void s390_machine_reset(MachineState *machine, ShutdownCause reason)
     switch (reset_type) {
     case S390_RESET_EXTERNAL:
     case S390_RESET_REIPL:
+        /*
+         * Reset the subsystem which includes a AP reset. If a PV
+         * guest had APQNs attached the AP reset is a prerequisite to
+         * unprotecting since the UV checks if all APQNs are reset.
+         */
+        subsystem_reset();
         if (s390_is_pv()) {
             s390_machine_unprotect(ms);
         }
 
+        /*
+         * Device reset includes CPU clear resets so this has to be
+         * done AFTER the unprotect call above.
+         */
         qemu_devices_reset(reason);
         s390_crypto_reset();
 
Signature · 1 line
2.34.1
Tested-by on v1 1/1
#6 lore raw

Re: [PATCH] s390x: do a subsystem reset before the unprotect on reboot

Quoted 36 lines · Janosch Frank

Bound APQNs have to be reset before tearing down the secure config via s390_machine_unprotect(). Otherwise the Ultravisor will return a error code.

So let's do a subsystem_reset() which includes a AP reset before the unprotect call. We'll do a full device_reset() afterwards which will reset some devices twice. That's ok since we can't move the device_reset() before the unprotect as it includes a CPU clear reset which the Ultravisor does not expect at that point in time.

---

I'm not able to test this for the PV AP case right new, that has to wait to early next week. However Marc told me that the non-AP PV test works fine now.

---
 hw/s390x/s390-virtio-ccw.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
hw/s390x/s390-virtio-ccw.c +6 -0
index 3dd0b2372d..2d75f2131f 100644
@@ -438,10 +438,20 @@ static void s390_machine_reset(MachineState *machine, ShutdownCause reason)
     switch (reset_type) {
     case S390_RESET_EXTERNAL:
     case S390_RESET_REIPL:
+        /*
+         * Reset the subsystem which includes a AP reset. If a PV
+         * guest had APQNs attached the AP reset is a prerequisite to
+         * unprotecting since the UV checks if all APQNs are reset.
+         */
+        subsystem_reset();
         if (s390_is_pv()) {

hw/s390x/s390-virtio-ccw.c static void s390_machine_reset(MachineState *machine, ShutdownCause reason) · line 457

Quoted 10 lines
hw/s390x/s390-virtio-ccw.c +4 -0
             s390_machine_unprotect(ms);
         }
 
+        /*
+         * Device reset includes CPU clear resets so this has to be
+         * done AFTER the unprotect call above.
+         */
         qemu_devices_reset(reason);
         s390_crypto_reset();
 

I tested this with and without bound/associated APQNs both with booting from disk and with direct kernel boot. Subsequent reboots are correctly resetting the APQNs. I also successfully tested the case direct kernel boot -> chreipl -> disk boot.

If you want you can add

Acked-by on v1 1/1
#7 lore raw

Re: [PATCH] s390x: do a subsystem reset before the unprotect on reboot

In reply to Janosch Frank

Am 01.09.23 um 13:48 schrieb Janosch Frank:

Quoted 11 lines

Bound APQNs have to be reset before tearing down the secure config via s390_machine_unprotect(). Otherwise the Ultravisor will return a error code.

So let's do a subsystem_reset() which includes a AP reset before the unprotect call. We'll do a full device_reset() afterwards which will reset some devices twice. That's ok since we can't move the device_reset() before the unprotect as it includes a CPU clear reset which the Ultravisor does not expect at that point in time.

Makes sense.

Quoted 35 lines

---

I'm not able to test this for the PV AP case right new, that has to wait to early next week. However Marc told me that the non-AP PV test works fine now.

---
  hw/s390x/s390-virtio-ccw.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
hw/s390x/s390-virtio-ccw.c +10 -0
index 3dd0b2372d..2d75f2131f 100644
@@ -438,10 +438,20 @@ static void s390_machine_reset(MachineState *machine, ShutdownCause reason)
      switch (reset_type) {
      case S390_RESET_EXTERNAL:
      case S390_RESET_REIPL:
+        /*
+         * Reset the subsystem which includes a AP reset. If a PV
+         * guest had APQNs attached the AP reset is a prerequisite to
+         * unprotecting since the UV checks if all APQNs are reset.
+         */
+        subsystem_reset();
          if (s390_is_pv()) {
              s390_machine_unprotect(ms);
          }
 
+        /*
+         * Device reset includes CPU clear resets so this has to be
+         * done AFTER the unprotect call above.
+         */
          qemu_devices_reset(reason);
          s390_crypto_reset();
 
#8 lore raw

PATCH v4 3/5 NOTFORMERGE update linux-headers/asm-s390/kvm.h

In reply to Steffen Eiden

Likely to be included in Linux 6.{6,7}

---
 linux-headers/asm-s390/kvm.h | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
linux-headers/asm-s390/kvm.h +16 -0
index e2afd95420..023a2763a9 100644
@@ -159,6 +159,22 @@ struct kvm_s390_vm_cpu_subfunc {
 	__u8 reserved[1728];
 };
 
+#define KVM_S390_VM_CPU_PROCESSOR_UV_FEAT_GUEST	6
+#define KVM_S390_VM_CPU_MACHINE_UV_FEAT_GUEST	7
+
+#define KVM_S390_VM_CPU_UV_FEAT_NR_BITS	64
+struct kvm_s390_vm_cpu_uv_feat {
+	union {
+		struct {
+			__u64 : 4;
+			__u64 ap : 1;		/* bit 4 */
+			__u64 ap_intr : 1;	/* bit 5 */
+			__u64 : 58;
+		};
+		__u64 feat;
+	};
+};
+
 /* kvm attributes for crypto */
 #define KVM_S390_VM_CRYPTO_ENABLE_AES_KW	0
 #define KVM_S390_VM_CRYPTO_ENABLE_DEA_KW	1
Signature · 1 line
2.41.0
#9 lore raw

PATCH v4 4/5 target/s390x/kvm: Refactor AP functionalities

In reply to Steffen Eiden

kvm_s390_set_attr() is a misleading name as it only sets attributes for the KVM_S390_VM_CRYPTO group. Therefore, rename it to kvm_s390_set_crypto_attr().

Add new functions ap_available() and ap_enabled() to avoid code duplication later.

---
 target/s390x/kvm/kvm.c | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)
target/s390x/kvm/kvm.c +17 -7
index a9e5880349..a7e2cdf668 100644
@@ -250,7 +250,7 @@ static void kvm_s390_enable_cmma(void)
     trace_kvm_enable_cmma(rc);
 }
 
-static void kvm_s390_set_attr(uint64_t attr)
+static void kvm_s390_set_crypto_attr(uint64_t attr)
 {
     struct kvm_device_attr attribute = {
         .group = KVM_S390_VM_CRYPTO,
@@ -275,7 +275,7 @@ static void kvm_s390_init_aes_kw(void)
     }
 
     if (kvm_vm_check_attr(kvm_state, KVM_S390_VM_CRYPTO, attr)) {
-            kvm_s390_set_attr(attr);
+            kvm_s390_set_crypto_attr(attr);
     }
 }
 
@@ -289,7 +289,7 @@ static void kvm_s390_init_dea_kw(void)
     }
 
     if (kvm_vm_check_attr(kvm_state, KVM_S390_VM_CRYPTO, attr)) {
-            kvm_s390_set_attr(attr);
+            kvm_s390_set_crypto_attr(attr);
     }
 }
 
@@ -2296,6 +2296,17 @@ static int configure_cpu_subfunc(const S390FeatBitmap features)
     return kvm_vm_ioctl(kvm_state, KVM_SET_DEVICE_ATTR, &attr);
 }
 
+static bool ap_available(void)
+{
+    return kvm_vm_check_attr(kvm_state, KVM_S390_VM_CRYPTO,
+                             KVM_S390_VM_CRYPTO_ENABLE_APIE);
+}
+
+static bool ap_enabled(const S390FeatBitmap features)
+{
+    return test_bit(S390_FEAT_AP, features);
+}
+
 static int kvm_to_feat[][2] = {
     { KVM_S390_VM_CPU_FEAT_ESOP, S390_FEAT_ESOP },
     { KVM_S390_VM_CPU_FEAT_SIEF2, S390_FEAT_SIE_F2 },
@@ -2475,8 +2486,7 @@ void kvm_s390_get_host_cpu_model(S390CPUModel *model, Error **errp)
         return;
     }
     /* for now, we can only provide the AP feature with HW support */
-    if (kvm_vm_check_attr(kvm_state, KVM_S390_VM_CRYPTO,
-        KVM_S390_VM_CRYPTO_ENABLE_APIE)) {
+    if (ap_available()) {
         set_bit(S390_FEAT_AP, model->features);
     }
 
@@ -2502,7 +2512,7 @@ static void kvm_s390_configure_apie(bool interpret)
                                 KVM_S390_VM_CRYPTO_DISABLE_APIE;
 
     if (kvm_vm_check_attr(kvm_state, KVM_S390_VM_CRYPTO, attr)) {
-        kvm_s390_set_attr(attr);
+        kvm_s390_set_crypto_attr(attr);
     }
 }
 
@@ -2556,7 +2566,7 @@ void kvm_s390_apply_cpu_model(const S390CPUModel *model, Error **errp)
         kvm_s390_enable_cmma();
     }
 
-    if (test_bit(S390_FEAT_AP, model->features)) {
+    if (ap_enabled(model->features)) {
         kvm_s390_configure_apie(true);
     }
 }
Signature · 1 line
2.41.0
#10 lore raw

PATCH v4 5/5 target/s390x: AP-passthrough for PV guests

In reply to Steffen Eiden

Enabling AP-passthrough(AP-pt) for PV-guest by using the new CPU features for PV-AP-pt of KVM.

As usual QEMU first checks which CPU features are available and then sets them if available and selected by user. An additional check is done to verify that PV-AP can only be enabled if "regular" AP-pt is enabled as well. Note that KVM itself does not enforce this restriction.

---
 target/s390x/cpu_features.h         |  1 +
 target/s390x/cpu_features_def.h.inc |  4 ++
 target/s390x/cpu_models.c           |  2 +
 target/s390x/gen-features.c         |  2 +
 target/s390x/kvm/kvm.c              | 70 +++++++++++++++++++++++++++++
 5 files changed, 79 insertions(+)
target/s390x/cpu_features.h +1 -0
index 87463f064d..a9bd68a2e1 100644
@@ -43,6 +43,7 @@ typedef enum {
     S390_FEAT_TYPE_KDSA,
     S390_FEAT_TYPE_SORTL,
     S390_FEAT_TYPE_DFLTCC,
+    S390_FEAT_TYPE_UV_FEAT_GUEST,
 } S390FeatType;
 
 /* Definition of a CPU feature */
target/s390x/cpu_features_def.h.inc +4 -0
index e3cfe63735..e68da9b8ff 100644
@@ -379,3 +379,7 @@ DEF_FEAT(DEFLATE_GHDT, "dfltcc-gdht", DFLTCC, 1, "DFLTCC GDHT")
 DEF_FEAT(DEFLATE_CMPR, "dfltcc-cmpr", DFLTCC, 2, "DFLTCC CMPR")
 DEF_FEAT(DEFLATE_XPND, "dfltcc-xpnd", DFLTCC, 4, "DFLTCC XPND")
 DEF_FEAT(DEFLATE_F0, "dfltcc-f0", DFLTCC, 192, "DFLTCC format 0 parameter-block")
+
+/* Features exposed via the UV-CALL instruction */
+DEF_FEAT(UV_FEAT_AP, "appv", UV_FEAT_GUEST, 4, "AP instructions installed for secure guests")
+DEF_FEAT(UV_FEAT_AP_INTR, "appvi", UV_FEAT_GUEST, 5, "AP instructions interruption support for secure guests")
target/s390x/cpu_models.c +2 -0
index 42b52afdb4..65331c37a3 100644
@@ -483,6 +483,8 @@ static void check_consistency(const S390CPUModel *model)
         { S390_FEAT_DIAG_318, S390_FEAT_EXTENDED_LENGTH_SCCB },
         { S390_FEAT_NNPA, S390_FEAT_VECTOR },
         { S390_FEAT_RDP, S390_FEAT_LOCAL_TLB_CLEARING },
+        { S390_FEAT_UV_FEAT_AP, S390_FEAT_AP },
+        { S390_FEAT_UV_FEAT_AP_INTR, S390_FEAT_UV_FEAT_AP },
     };
     int i;
 
target/s390x/gen-features.c +2 -0
index 1e3b7c0dc9..2b2bfc3736 100644
@@ -576,6 +576,8 @@ static uint16_t full_GEN16_GA1[] = {
     S390_FEAT_RDP,
     S390_FEAT_PAI,
     S390_FEAT_PAIE,
+    S390_FEAT_UV_FEAT_AP,
+    S390_FEAT_UV_FEAT_AP_INTR,
 };
 
 
target/s390x/kvm/kvm.c +70 -0

93 lines not on this page · raw message

Signature · 1 line
2.41.0