s390: Enable AP instructions for pv-guests
v4 cover s390: Enable AP instructions for pv-guests
This series enables general QEMU support for AP pass-through for Secure Execution guests (pv-guests).
To enable AP-PT on pv-guests QEMU has to turn on the corresponding bits in the KVM CPU-model[1] if the CPU firmware supports it. However, it only makes sense to turn on AP-PT if the QEMU user enabled (general) AP for that guest.
See: https://lore.kernel.org/linux-s390/[email protected]/T/#t
The series consists of five patches:
1/2) fixes from Janosch for AP handling
3) update kvm-s390 header for this series (NOTFORMERGE)
4) small cleanup for kvm_s390_set_attr() refactor code to add ap_available() and ap_enabled()
5) Add UV_CALL CPU model enablement
since v3:
- nitfixes (Thomas)
- r-b from Michael&Thomas
since v2:
- add fixes for AP from Janosch
- rename *UV_CALL* to UV_FEAT_GUEST (Janosch)
- early return on some functions (Janosch)
- add r-b from Michael (Patch 4)
- mark linux header update as NOTFORMERGE
since v1:
- removed the new features from the default gen16 model
- updated KVM-headers to match KVM series v3 [1]
- applied review comments from Thomas
Steffen
Janosch Frank (2):
s390x/ap: fix missing subsystem reset registration
s390x: switch pv and subsystem reset ordering on reboot
Steffen Eiden (3):
NOTFORMERGE update linux-headers/asm-s390/kvm.h
target/s390x/kvm: Refactor AP functionalities
target/s390x: AP-passthrough for PV guests
hw/s390x/s390-virtio-ccw.c | 7 ++- linux-headers/asm-s390/kvm.h | 16 +++++ target/s390x/cpu_features.h | 1 + target/s390x/cpu_features_def.h.inc | 4 ++ target/s390x/cpu_models.c | 2 + target/s390x/gen-features.c | 2 + target/s390x/kvm/kvm.c | 94 ++++++++++++++++++++++++++--- 7 files changed, 116 insertions(+), 10 deletions(-)
Signature · 1 line
2.41.0
PATCH v4 1/5 s390x/ap: fix missing subsystem reset registration
From: Janosch Frank <[email protected]>
A subsystem reset contains a reset of AP resources which has been missing. Adding the AP bridge to the list of device types that need reset fixes this issue.
--- hw/s390x/s390-virtio-ccw.c | 1 + 1 file changed, 1 insertion(+)
hw/s390x/s390-virtio-ccw.c +1 -0
@@ -109,6 +109,7 @@ static const char *const reset_dev_types[] = {"s390-flic" ,"diag288" ,TYPE_S390_PCI_HOST_BRIDGE , + TYPE_AP_BRIDGE, };static void subsystem_reset (void )
Signature · 1 line
2.41.0
PATCH v4 2/5 s390x: switch pv and subsystem reset ordering on reboot
In reply to Steffen EidenFrom: Janosch Frank <[email protected]>
Bound APQNs have to be reset before tearing down the secure config via s390_machine_unprotect(). Otherwise the Ultravisor will return a error code.
So let's switch the ordering around to make that happen.
--- hw/s390x/s390-virtio-ccw.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
hw/s390x/s390-virtio-ccw.c +3 -3
@@ -442,13 +442,13 @@ static void s390_machine_reset(MachineState *machine, ShutdownCause reason)switch (reset_type) {case S390_RESET_EXTERNAL :case S390_RESET_REIPL : +qemu_devices_reset (reason); +s390_crypto_reset (); +if (s390_is_pv ()) {s390_machine_unprotect (ms); } -qemu_devices_reset (reason); -s390_crypto_reset (); -/* configure and start the ipl CPU only */ run_on_cpu (cs, s390_do_cpu_ipl,RUN_ON_CPU_NULL );break ;
Signature · 1 line
2.41.0
Re: [PATCH v4 2/5] s390x: switch pv and subsystem reset ordering on reboot
Quoted 38 lines · Steffen Eiden
From: Janosch Frank <[email protected]>
Bound APQNs have to be reset before tearing down the secure config via
s390_machine_unprotect(). Otherwise the Ultravisor will return a error code.So let's switch the ordering around to make that happen.
Reviewed-by Christian Borntraeger <[email protected]>Signed-off-by Janosch Frank <[email protected]>--- hw/s390x/s390-virtio-ccw.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)hw/s390x/s390-virtio-ccw.c +3
-3@@ -442,13 +442,13 @@ static void s390_machine_reset(MachineState *machine, ShutdownCause reason)switch (reset_type) {case S390_RESET_EXTERNAL :case S390_RESET_REIPL : +qemu_devices_reset (reason); +s390_crypto_reset (); +if (s390_is_pv ()) {s390_machine_unprotect (ms); } -qemu_devices_reset (reason); -s390_crypto_reset (); -/* configure and start the ipl CPU only */ run_on_cpu (cs, s390_do_cpu_ipl,RUN_ON_CPU_NULL );break ;Signature · 1 line
2.41.0
Unfortunately, this breaks things for me. You can reproduce the problem easily… Start an SE guest via direct kernel boot and reboot the guest after the guest has booted.
e.g.
$ sudo qemu-system-s390x -smp 2 --machine accel=kvm,aes-key-wrap=off -kernel /var/lib/libvirt/images/hades/vmlinux-s390x.se.img -m 2048 -nographic -nodefaults -chardev stdio,id=c1,mux=on -device sclpconsole,chardev=c1 -append "earlyprintk" … # reboot # (console in the guest) … [ 3.966496] systemd-shutdown[1]: Syncing filesystems and block devices. [ 3.966606] systemd-shutdown[1]: Rebooting. qemu-system-s390x: CPU reset failed on CPU 0 type aec4 qemu-system-s390x: CPU reset failed on CPU 1 type aec4 qemu-system-s390x: KVM PV command 4 (KVM_PV_VERIFY) failed: header rc 102 rrc 1b IOCTL rc: -22 Protected boot has failed: 0xa02
Signature · 8 lines
Kind regards / Beste Grüße Marc Hartmayer IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Gregor Pillen Geschäftsführung: David Faller Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
Re: [PATCH v4 2/5] s390x: switch pv and subsystem reset ordering on reboot
Quoted 30 lines · Marc Hartmayer
Quoted 29 lines · Steffen Eiden
From: Janosch Frank <[email protected]>
Bound APQNs have to be reset before tearing down the secure config via
s390_machine_unprotect(). Otherwise the Ultravisor will return a error code.So let's switch the ordering around to make that happen.
Reviewed-by Christian Borntraeger <[email protected]>Signed-off-by Janosch Frank <[email protected]>--- hw/s390x/s390-virtio-ccw.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)hw/s390x/s390-virtio-ccw.c +3
-0@@ -442,13 +442,13 @@ static void s390_machine_reset(MachineState *machine, ShutdownCause reason)switch (reset_type) {case S390_RESET_EXTERNAL :case S390_RESET_REIPL : +qemu_devices_reset (reason); +s390_crypto_reset (); +if (s390_is_pv ()) {s390_machine_unprotect (ms); }
Quoted 13 lines
Quoted 9 lines
hw/s390x/s390-virtio-ccw.c +0
-3-qemu_devices_reset (reason); -s390_crypto_reset (); -/* configure and start the ipl CPU only */ run_on_cpu (cs, s390_do_cpu_ipl,RUN_ON_CPU_NULL );break ;Signature · 1 line
2.41.0Unfortunately, this breaks things for me. You can reproduce the problem easily… Start an SE guest via direct kernel boot and reboot the guest after the guest has booted.
Seems like we didn't test a direct kernel boot reboot... I'm looking into it.
PATCH 1/1 s390x: do a subsystem reset before the unprotect on reboot
Bound APQNs have to be reset before tearing down the secure config via s390_machine_unprotect(). Otherwise the Ultravisor will return a error code.
So let's do a subsystem_reset() which includes a AP reset before the unprotect call. We'll do a full device_reset() afterwards which will reset some devices twice. That's ok since we can't move the device_reset() before the unprotect as it includes a CPU clear reset which the Ultravisor does not expect at that point in time.
---
I'm not able to test this for the PV AP case right new, that has to wait to early next week. However Marc told me that the non-AP PV test works fine now.
--- hw/s390x/s390-virtio-ccw.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
hw/s390x/s390-virtio-ccw.c +10 -0
@@ -438,10 +438,20 @@ static void s390_machine_reset(MachineState *machine, ShutdownCause reason)switch (reset_type) {case S390_RESET_EXTERNAL :case S390_RESET_REIPL : +/* +* Reset the subsystem which includes a AP reset. If a PV +* guest had APQNs attached the AP reset is a prerequisite to +* unprotecting since the UV checks if all APQNs are reset. +*/ +subsystem_reset ();if (s390_is_pv ()) {s390_machine_unprotect (ms); } +/* +* Device reset includes CPU clear resets so this has to be +* done AFTER the unprotect call above. +*/ qemu_devices_reset (reason);s390_crypto_reset ();
Signature · 1 line
2.34.1
Re: [PATCH] s390x: do a subsystem reset before the unprotect on reboot
Quoted 36 lines · Janosch Frank
Bound APQNs have to be reset before tearing down the secure config via
s390_machine_unprotect(). Otherwise the Ultravisor will return a error code.So let's do a
subsystem_reset()which includes a AP reset before the unprotect call. We'll do a fulldevice_reset()afterwards which will reset some devices twice. That's ok since we can't move thedevice_reset()before the unprotect as it includes a CPU clear reset which the Ultravisor does not expect at that point in time.Signed-off-by Janosch Frank <[email protected]>---
I'm not able to test this for the PV AP case right new, that has to wait to early next week. However Marc told me that the non-AP PV test works fine now.
--- hw/s390x/s390-virtio-ccw.c | 10 ++++++++++ 1 file changed, 10 insertions(+)hw/s390x/s390-virtio-ccw.c +6
-0@@ -438,10 +438,20 @@ static void s390_machine_reset(MachineState *machine, ShutdownCause reason)switch (reset_type) {case S390_RESET_EXTERNAL :case S390_RESET_REIPL : +/* +* Reset the subsystem which includes a AP reset. If a PV +* guest had APQNs attached the AP reset is a prerequisite to +* unprotecting since the UV checks if all APQNs are reset. +*/ +subsystem_reset ();if (s390_is_pv ()) {
hw/s390x/s390-virtio-ccw.c static void s390_machine_reset(MachineState *machine, ShutdownCause reason) · line 457
Quoted 10 lines
hw/s390x/s390-virtio-ccw.c +4
-0s390_machine_unprotect (ms); } +/* +* Device reset includes CPU clear resets so this has to be +* done AFTER the unprotect call above. +*/ qemu_devices_reset (reason);s390_crypto_reset ();
I tested this with and without bound/associated APQNs both with booting from disk and with direct kernel boot. Subsequent reboots are correctly resetting the APQNs. I also successfully tested the case direct kernel boot -> chreipl -> disk boot.
If you want you can add
Re: [PATCH] s390x: do a subsystem reset before the unprotect on reboot
In reply to Janosch FrankAm 01.09.23 um 13:48 schrieb Janosch Frank:
Quoted 11 lines
Bound APQNs have to be reset before tearing down the secure config via
s390_machine_unprotect(). Otherwise the Ultravisor will return a error code.So let's do a
subsystem_reset()which includes a AP reset before the unprotect call. We'll do a fulldevice_reset()afterwards which will reset some devices twice. That's ok since we can't move thedevice_reset()before the unprotect as it includes a CPU clear reset which the Ultravisor does not expect at that point in time.Signed-off-by Janosch Frank <[email protected]>
Makes sense.
Quoted 35 lines
---
I'm not able to test this for the PV AP case right new, that has to wait to early next week. However Marc told me that the non-AP PV test works fine now.
--- hw/s390x/s390-virtio-ccw.c | 10 ++++++++++ 1 file changed, 10 insertions(+)hw/s390x/s390-virtio-ccw.c +10
-0@@ -438,10 +438,20 @@ static void s390_machine_reset(MachineState *machine, ShutdownCause reason)switch (reset_type) {case S390_RESET_EXTERNAL :case S390_RESET_REIPL : +/* +* Reset the subsystem which includes a AP reset. If a PV +* guest had APQNs attached the AP reset is a prerequisite to +* unprotecting since the UV checks if all APQNs are reset. +*/ +subsystem_reset ();if (s390_is_pv ()) {s390_machine_unprotect (ms); } +/* +* Device reset includes CPU clear resets so this has to be +* done AFTER the unprotect call above. +*/ qemu_devices_reset (reason);s390_crypto_reset ();
PATCH v4 3/5 NOTFORMERGE update linux-headers/asm-s390/kvm.h
In reply to Steffen EidenLikely to be included in Linux 6.{6,7}
--- linux-headers/asm-s390/kvm.h | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
linux-headers/asm-s390/kvm.h +16 -0
@@ -159,6 +159,22 @@ struct kvm_s390_vm_cpu_subfunc {__u8 reserved[1728 ]; }; +#define KVM_S390_VM_CPU_PROCESSOR_UV_FEAT_GUEST 6 +#define KVM_S390_VM_CPU_MACHINE_UV_FEAT_GUEST 7 + +#define KVM_S390_VM_CPU_UV_FEAT_NR_BITS 64 +struct kvm_s390_vm_cpu_uv_feat { +union { +struct { + __u64 : 4; +__u64 ap :1 ;/* bit 4 */ +__u64 ap_intr :1 ;/* bit 5 */ + __u64 : 58; + }; +__u64 feat; + }; +}; +/* kvm attributes for crypto */ #define KVM_S390_VM_CRYPTO_ENABLE_AES_KW 0#define KVM_S390_VM_CRYPTO_ENABLE_DEA_KW 1
Signature · 1 line
2.41.0
PATCH v4 4/5 target/s390x/kvm: Refactor AP functionalities
In reply to Steffen Eidenkvm_s390_set_attr() is a misleading name as it only sets attributes for the KVM_S390_VM_CRYPTO group. Therefore, rename it to kvm_s390_set_crypto_attr().
Add new functions ap_available() and ap_enabled() to avoid code duplication later.
--- target/s390x/kvm/kvm.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-)
target/s390x/kvm/kvm.c +17 -7
@@ -250,7 +250,7 @@ static void kvm_s390_enable_cmma(void)trace_kvm_enable_cmma (rc); } -static void kvm_s390_set_attr (uint64_t attr) +static void kvm_s390_set_crypto_attr (uint64_t attr) {struct kvm_device_attr attribute = { .group =KVM_S390_VM_CRYPTO ,
@@ -275,7 +275,7 @@ static void kvm_s390_init_aes_kw(void) }if (kvm_vm_check_attr (kvm_state,KVM_S390_VM_CRYPTO , attr)) { -kvm_s390_set_attr (attr); +kvm_s390_set_crypto_attr (attr); } }
@@ -289,7 +289,7 @@ static void kvm_s390_init_dea_kw(void) }if (kvm_vm_check_attr (kvm_state,KVM_S390_VM_CRYPTO , attr)) { -kvm_s390_set_attr (attr); +kvm_s390_set_crypto_attr (attr); } }
@@ -2296,6 +2296,17 @@ static int configure_cpu_subfunc(const S390FeatBitmap features)return kvm_vm_ioctl (kvm_state,KVM_SET_DEVICE_ATTR , &attr); } +static bool ap_available (void ) +{ +return kvm_vm_check_attr (kvm_state,KVM_S390_VM_CRYPTO , +KVM_S390_VM_CRYPTO_ENABLE_APIE ); +} + +static bool ap_enabled (const S390FeatBitmap features) +{ +return test_bit (S390_FEAT_AP , features); +} +static int kvm_to_feat[][2 ] = { {KVM_S390_VM_CPU_FEAT_ESOP ,S390_FEAT_ESOP }, {KVM_S390_VM_CPU_FEAT_SIEF2 ,S390_FEAT_SIE_F2 },
@@ -2475,8 +2486,7 @@ void kvm_s390_get_host_cpu_model(S390CPUModel *model, Error **errp)return ; }/* for now, we can only provide the AP feature with HW support */ -if (kvm_vm_check_attr (kvm_state,KVM_S390_VM_CRYPTO , -KVM_S390_VM_CRYPTO_ENABLE_APIE )) { +if (ap_available ()) {set_bit (S390_FEAT_AP , model->features); }
@@ -2502,7 +2512,7 @@ static void kvm_s390_configure_apie(bool interpret)KVM_S390_VM_CRYPTO_DISABLE_APIE ;if (kvm_vm_check_attr (kvm_state,KVM_S390_VM_CRYPTO , attr)) { -kvm_s390_set_attr (attr); +kvm_s390_set_crypto_attr (attr); } }
@@ -2556,7 +2566,7 @@ void kvm_s390_apply_cpu_model(const S390CPUModel *model, Error **errp)kvm_s390_enable_cmma (); } -if (test_bit (S390_FEAT_AP , model->features)) { +if (ap_enabled (model->features)) {kvm_s390_configure_apie (true); } }
Signature · 1 line
2.41.0
PATCH v4 5/5 target/s390x: AP-passthrough for PV guests
In reply to Steffen EidenEnabling AP-passthrough(AP-pt) for PV-guest by using the new CPU features for PV-AP-pt of KVM.
As usual QEMU first checks which CPU features are available and then sets them if available and selected by user. An additional check is done to verify that PV-AP can only be enabled if "regular" AP-pt is enabled as well. Note that KVM itself does not enforce this restriction.
--- target/s390x/cpu_features.h | 1 + target/s390x/cpu_features_def.h.inc | 4 ++ target/s390x/cpu_models.c | 2 + target/s390x/gen-features.c | 2 + target/s390x/kvm/kvm.c | 70 +++++++++++++++++++++++++++++ 5 files changed, 79 insertions(+)
target/s390x/cpu_features.h +1 -0
@@ -43,6 +43,7 @@ typedef enum {S390_FEAT_TYPE_KDSA ,S390_FEAT_TYPE_SORTL ,S390_FEAT_TYPE_DFLTCC , +S390_FEAT_TYPE_UV_FEAT_GUEST , } S390FeatType;/* Definition of a CPU feature */
target/s390x/cpu_features_def.h.inc +4 -0
@@ -379,3 +379,7 @@ DEF_FEAT(DEFLATE_GHDT, "dfltcc-gdht", DFLTCC, 1, "DFLTCC GDHT") DEF_FEAT(DEFLATE_CMPR, "dfltcc-cmpr", DFLTCC, 2, "DFLTCC CMPR") DEF_FEAT(DEFLATE_XPND, "dfltcc-xpnd", DFLTCC, 4, "DFLTCC XPND") DEF_FEAT(DEFLATE_F0, "dfltcc-f0", DFLTCC, 192, "DFLTCC format 0 parameter-block") + +/* Features exposed via the UV-CALL instruction */ +DEF_FEAT(UV_FEAT_AP, "appv", UV_FEAT_GUEST, 4, "AP instructions installed for secure guests") +DEF_FEAT(UV_FEAT_AP_INTR, "appvi", UV_FEAT_GUEST, 5, "AP instructions interruption support for secure guests")
target/s390x/cpu_models.c +2 -0
@@ -483,6 +483,8 @@ static void check_consistency(const S390CPUModel *model) { S390_FEAT_DIAG_318, S390_FEAT_EXTENDED_LENGTH_SCCB }, { S390_FEAT_NNPA, S390_FEAT_VECTOR }, { S390_FEAT_RDP, S390_FEAT_LOCAL_TLB_CLEARING }, + { S390_FEAT_UV_FEAT_AP, S390_FEAT_AP }, + { S390_FEAT_UV_FEAT_AP_INTR, S390_FEAT_UV_FEAT_AP }, };int i;
target/s390x/gen-features.c +2 -0
@@ -576,6 +576,8 @@ static uint16_t full_GEN16_GA1[] = {S390_FEAT_RDP ,S390_FEAT_PAI ,S390_FEAT_PAIE , +S390_FEAT_UV_FEAT_AP , + S390_FEAT_UV_FEAT_AP_INTR, };
target/s390x/kvm/kvm.c +70 -0
93 lines not on this page · raw message
Signature · 1 line
2.41.0