kvm: s390: Add PV dump support
Subsystems (4)
Janosch Frank
maintainerv2 cover kvm: s390: Add PV dump support
Sometimes dumping inside of a VM fails, is unavailable or doesn't yield the required data. For these occasions we dump the VM from the outside, writing memory and cpu data to a file.
Up to now PV guests only supported dumping from the inside of the guest through dumpers like KDUMP. A PV guest can be dumped from the hypervisor but the data will be stale and / or encrypted.
To get the actual state of the PV VM we need the help of the Ultravisor who safeguards the VM state. New UV calls have been added to initialize the dump, dump storage state data, dump cpu data and complete the dump process.
Notes:
I chose not to document the dump data provided by the Ultravisor since KVM doesn't interprete it in any way. We're currently searching for a location and enough cycles to make it available to all.
v2:
* Added vcpu SIE blocking to avoid validities
* Moved the KVM CAP to patch #7
* Renamed len to len_max and introduced len_written for extendability
* Added Rev-bys
Janosch Frank (9):
s390x: Add SE hdr query information
s390: uv: Add dump fields to query
KVM: s390: pv: Add query interface
KVM: s390: pv: Add dump support definitions
KVM: s390: pv: Add query dump information
kvm: s390: Add configuration dump functionality
kvm: s390: Add CPU dump functionality
Documentation: virt: Protected virtual machine dumps Documentation/virt/kvm/api.rst: Add protvirt dump/info api
descriptions
Documentation/virt/kvm/api.rst | 150 +++++++++++- Documentation/virt/kvm/index.rst | 1 + Documentation/virt/kvm/s390-pv-dump.rst | 60 +++++ arch/s390/boot/uv.c | 4 + arch/s390/include/asm/kvm_host.h | 1 + arch/s390/include/asm/uv.h | 45 +++- arch/s390/kernel/uv.c | 53 +++++ arch/s390/kvm/kvm-s390.c | 297 ++++++++++++++++++++++++ arch/s390/kvm/kvm-s390.h | 3 + arch/s390/kvm/pv.c | 131 +++++++++++ include/uapi/linux/kvm.h | 55 +++++ 11 files changed, 797 insertions(+), 3 deletions(-) create mode 100644 Documentation/virt/kvm/s390-pv-dump.rst
Signature · 1 line
2.32.0
Janosch Frank
maintainerPATCH v2 1/9 s390x: Add SE hdr query information
We have information about the supported se header version and pcf bits so let's expose it via the sysfs files.
--- arch/s390/boot/uv.c | 2 ++ arch/s390/include/asm/uv.h | 7 ++++++- arch/s390/kernel/uv.c | 20 ++++++++++++++++++++ 3 files changed, 28 insertions(+), 1 deletion(-)
arch/s390/boot/uv.c +2 -0
@@ -41,6 +41,8 @@ void uv_query_info(void) uv_info.max_num_sec_conf = uvcb.max_num_sec_conf; uv_info.max_guest_cpu_id = uvcb.max_guest_cpu_id; uv_info.uv_feature_indications = uvcb.uv_feature_indications; + uv_info.supp_se_hdr_ver = uvcb.supp_se_hdr_versions; + uv_info.supp_se_hdr_pcf = uvcb.supp_se_hdr_pcf; }#ifdef CONFIG_PROTECTED_VIRTUALIZATION_GUEST
arch/s390/include/asm/uv.h +6 -1
@@ -108,7 +108,10 @@ struct uv_cb_qui {u8 reserved88[158 -136 ];/* 0x0088 */ u16 max_guest_cpu_id;/* 0x009e */ u64 uv_feature_indications;/* 0x00a0 */ -u8 reserveda8[200 -168 ];/* 0x00a8 */ +u64 reserveda8;/* 0x00a8 */ +u64 supp_se_hdr_versions;/* 0x00b0 */ +u64 supp_se_hdr_pcf;/* 0x00b8 */ +u64 reservedc0;/* 0x00c0 */ } __packed__aligned(8) ;/* Initialize Ultravisor */
@@ -286,6 +289,8 @@ struct uv_info {unsigned int max_num_sec_conf;unsigned short max_guest_cpu_id;unsigned long uv_feature_indications; +unsigned long supp_se_hdr_ver; +unsigned long supp_se_hdr_pcf; };extern struct uv_info uv_info;
arch/s390/kernel/uv.c +20 -0
@@ -392,6 +392,24 @@ static ssize_t uv_query_facilities(struct kobject *kobj,static struct kobj_attribute uv_query_facilities_attr =__ATTR (facilities,0444 , uv_query_facilities,NULL ); +static ssize_t uv_query_supp_se_hdr_ver (struct kobject *kobj, +struct kobj_attribute *attr,char *buf) +{ +return sysfs_emit (buf,"%lx\n" , uv_info.supp_se_hdr_ver); +} + +static struct kobj_attribute uv_query_supp_se_hdr_ver_attr = +__ATTR (supp_se_hdr_ver,0444 , uv_query_supp_se_hdr_ver,NULL ); + +static ssize_t uv_query_supp_se_hdr_pcf (struct kobject *kobj, +struct kobj_attribute *attr,char *buf) +{ +return sysfs_emit (buf,"%lx\n" , uv_info.supp_se_hdr_pcf); +} + +static struct kobj_attribute uv_query_supp_se_hdr_pcf_attr = +__ATTR (supp_se_hdr_pcf,0444 , uv_query_supp_se_hdr_pcf,NULL ); +static ssize_t uv_query_feature_indications (struct kobject *kobj,struct kobj_attribute *attr,char *buf) {
@@ -437,6 +455,8 @@ static struct attribute *uv_query_attrs[] = { &uv_query_max_guest_cpus_attr.attr, &uv_query_max_guest_vms_attr.attr, &uv_query_max_guest_addr_attr.attr, + &uv_query_supp_se_hdr_ver_attr.attr, + &uv_query_supp_se_hdr_pcf_attr.attr, NULL, };
Signature · 1 line
2.32.0
Janosch Frank
maintainerPATCH v2 2/9 s390: uv: Add dump fields to query
In reply to Janosch FrankThe new dump feature requires us to know how much memory is needed for the "dump storage state" and "dump finalize" ultravisor call. These values are reported via the UV query call.
--- arch/s390/boot/uv.c | 2 ++ arch/s390/include/asm/uv.h | 5 +++++ arch/s390/kernel/uv.c | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 40 insertions(+)
arch/s390/boot/uv.c +2 -0
@@ -43,6 +43,8 @@ void uv_query_info(void) uv_info.uv_feature_indications = uvcb.uv_feature_indications; uv_info.supp_se_hdr_ver = uvcb.supp_se_hdr_versions; uv_info.supp_se_hdr_pcf = uvcb.supp_se_hdr_pcf; + uv_info.conf_dump_storage_state_len = uvcb.conf_dump_storage_state_len; + uv_info.conf_dump_finalize_len = uvcb.conf_dump_finalize_len; }#ifdef CONFIG_PROTECTED_VIRTUALIZATION_GUEST
arch/s390/include/asm/uv.h +5 -0
@@ -112,6 +112,9 @@ struct uv_cb_qui {u64 supp_se_hdr_versions;/* 0x00b0 */ u64 supp_se_hdr_pcf;/* 0x00b8 */ u64 reservedc0;/* 0x00c0 */ +u64 conf_dump_storage_state_len;/* 0x00c8 */ +u64 conf_dump_finalize_len;/* 0x00d0 */ +u8 reservedd8[256 -216 ];/* 0x00d8 */ } __packed__aligned(8) ;/* Initialize Ultravisor */
@@ -291,6 +294,8 @@ struct uv_info {unsigned long uv_feature_indications;unsigned long supp_se_hdr_ver;unsigned long supp_se_hdr_pcf; +unsigned long conf_dump_storage_state_len; +unsigned long conf_dump_finalize_len; };extern struct uv_info uv_info;
arch/s390/kernel/uv.c +33 -0
@@ -410,6 +410,36 @@ static ssize_t uv_query_supp_se_hdr_pcf(struct kobject *kobj,static struct kobj_attribute uv_query_supp_se_hdr_pcf_attr =__ATTR (supp_se_hdr_pcf,0444 , uv_query_supp_se_hdr_pcf,NULL ); +static ssize_t uv_query_dump_cpu_len (struct kobject *kobj, +struct kobj_attribute *attr,char *page) +{ +return scnprintf (page,PAGE_SIZE ,"%lx\n" , + uv_info.guest_cpu_stor_len); +} + +static struct kobj_attribute uv_query_dump_cpu_len_attr = +__ATTR (uv_query_dump_cpu_len,0444 , uv_query_dump_cpu_len,NULL ); + +static ssize_t uv_query_dump_storage_state_len (struct kobject *kobj, +struct kobj_attribute *attr,char *page) +{ +return scnprintf (page,PAGE_SIZE ,"%lx\n" , + uv_info.conf_dump_storage_state_len); +} + +static struct kobj_attribute uv_query_dump_storage_state_len_attr = +__ATTR (dump_storage_state_len,0444 , uv_query_dump_storage_state_len,NULL ); + +static ssize_t uv_query_dump_finalize_len (struct kobject *kobj, +struct kobj_attribute *attr,char *page) +{ +return scnprintf (page,PAGE_SIZE ,"%lx\n" , + uv_info.conf_dump_finalize_len); +} + +static struct kobj_attribute uv_query_dump_finalize_len_attr = +__ATTR (dump_finalize_len,0444 , uv_query_dump_finalize_len,NULL ); +static ssize_t uv_query_feature_indications (struct kobject *kobj,struct kobj_attribute *attr,char *buf) {
@@ -457,6 +487,9 @@ static struct attribute *uv_query_attrs[] = { &uv_query_max_guest_addr_attr.attr, &uv_query_supp_se_hdr_ver_attr.attr, &uv_query_supp_se_hdr_pcf_attr.attr, + &uv_query_dump_storage_state_len_attr.attr, + &uv_query_dump_finalize_len_attr.attr, + &uv_query_dump_cpu_len_attr.attr, NULL, };
Signature · 1 line
2.32.0
Janosch Frank
maintainerPATCH v2 3/9 KVM: s390: pv: Add query interface
In reply to Janosch FrankSome of the query information is already available via sysfs but having a IOCTL makes the information easier to retrieve.
--- arch/s390/kvm/kvm-s390.c | 76 ++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 25 +++++++++++++ 2 files changed, 101 insertions(+)
arch/s390/kvm/kvm-s390.c +76 -0
@@ -2224,6 +2224,42 @@ static int kvm_s390_cpus_to_pv(struct kvm *kvm, u16 *rc, u16 *rrc)return r; } +/* +* Here we provide user space with a direct interface to query UV +* related data like UV maxima and available features as well as +* feature specific data. +* +* To facilitate future extension of the data structures we'll try to +* write data up to the maximum requested length. +*/ +static ssize_t kvm_s390_handle_pv_info (struct kvm_s390_pv_info *info) +{ +ssize_t len; + +switch (info->header.id) { +case KVM_PV_INFO_VM : { + len =sizeof (info->header) +sizeof (info->vm); + +if (info->header.len_max < len) +return -EINVAL ; + +memcpy (info->vm.inst_calls_list, + uv_info.inst_calls_list, +sizeof (uv_info.inst_calls_list)); + +/* It's max cpuidm not max cpus so it's off by one */ + info->vm.max_cpus = uv_info.max_guest_cpu_id +1 ; + info->vm.max_guests = uv_info.max_num_sec_conf; + info->vm.max_guest_addr = uv_info.max_sec_stor_addr; + info->vm.feature_indication = uv_info.uv_feature_indications; + +return len; + } +default : +return -EINVAL ; + } +} +static int kvm_s390_handle_pv (struct kvm *kvm,struct kvm_pv_cmd *cmd) {int r =0 ;
@@ -2360,6 +2396,46 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) cmd->rc, cmd->rrc);break ; } +case KVM_PV_INFO : { +struct kvm_s390_pv_info info = {}; +ssize_t data_len; + +/* +* No need to check the VM protection here. +* +* Maybe user space wants to query some of the data +* when the VM is still unprotected. If we see the +* need to fence a new data command we can still +* return an error in the info handler. +*/ + + r = -EFAULT ; +if (copy_from_user (&info, argp,sizeof (info.header))) +break ; + + r = -EINVAL ; +if (info.header.len_max <sizeof (info.header)) +break ; + + data_len =kvm_s390_handle_pv_info (&info); +if (data_len <0 ) { + r = data_len; +break ; + } +/* +* If a data command struct is extended (multiple +* times) this can be used to determine how much of it +* is valid. +*/ + info.header.len_written = data_len; + + r = -EFAULT ; +if (copy_to_user (argp, &info, data_len)) +break ; + + r =0 ; +break ; + }default : r = -ENOTTY ; }
include/uapi/linux/kvm.h +25 -0
@@ -1643,6 +1643,30 @@ struct kvm_s390_pv_unp {__u64 tweak; }; +enum pv_cmd_info_id { +KVM_PV_INFO_VM , +}; + +struct kvm_s390_pv_info_vm { +__u64 inst_calls_list[4 ]; +__u64 max_cpus; +__u64 max_guests; +__u64 max_guest_addr; +__u64 feature_indication; +}; + +struct kvm_s390_pv_info_header { +__u32 id; +__u32 len_max; +__u32 len_written; +__u32 reserved; +}; + +struct kvm_s390_pv_info { +struct kvm_s390_pv_info_header header; +struct kvm_s390_pv_info_vm vm; +}; +enum pv_cmd_id {KVM_PV_ENABLE ,KVM_PV_DISABLE ,
@@ -1651,6 +1675,7 @@ enum pv_cmd_id {KVM_PV_VERIFY ,KVM_PV_PREP_RESET ,KVM_PV_UNSHARE_ALL , + KVM_PV_INFO, }; struct kvm_pv_cmd {
Signature · 1 line
2.32.0
Claudio Imbrenda
maintainerRe: [PATCH v2 3/9] KVM: s390: pv: Add query interface
Quoted 2 lines · On Thu, 10 Mar 2022 10:31:06 +0000 Janosch Frank <[email protected]> wrote:
Some of the query information is already available via sysfs but having a IOCTL makes the information easier to retrieve.
if I understand correctly, this will be forward-compatible but not backwards compatible.
you return the amount of bytes written into the buffer, but only if the buffer was already big enough.
a newer userspace will work with an older kernel, but an older userspace will not work with a newer kernel.
a solution would be to return the size of the struct, so userspace can know how much of the buffer was written (if it was bigger than the struct), or that there are unwritten bits (if the buffer was smaller).
and even if the buffer was too small, write back as much of it as possible to userspace.
this way, an older userspace will get the information it expects.
I am also not a big fan of writing the size in the input struct (I think returning it would be cleaner), but I do not have a strong opinion
Quoted 144 lines
Signed-off-by Janosch Frank <[email protected]>--- arch/s390/kvm/kvm-s390.c | 76 ++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 25 +++++++++++++ 2 files changed, 101 insertions(+)arch/s390/kvm/kvm-s390.c +76
-0@@ -2224,6 +2224,42 @@ static int kvm_s390_cpus_to_pv(struct kvm *kvm, u16 *rc, u16 *rrc)return r; } +/* +* Here we provide user space with a direct interface to query UV +* related data like UV maxima and available features as well as +* feature specific data. +* +* To facilitate future extension of the data structures we'll try to +* write data up to the maximum requested length. +*/ +static ssize_t kvm_s390_handle_pv_info (struct kvm_s390_pv_info *info) +{ +ssize_t len; + +switch (info->header.id) { +case KVM_PV_INFO_VM : { + len =sizeof (info->header) +sizeof (info->vm); + +if (info->header.len_max < len) +return -EINVAL ; + +memcpy (info->vm.inst_calls_list, + uv_info.inst_calls_list, +sizeof (uv_info.inst_calls_list)); + +/* It's max cpuidm not max cpus so it's off by one */ + info->vm.max_cpus = uv_info.max_guest_cpu_id +1 ; + info->vm.max_guests = uv_info.max_num_sec_conf; + info->vm.max_guest_addr = uv_info.max_sec_stor_addr; + info->vm.feature_indication = uv_info.uv_feature_indications; + +return len; + } +default : +return -EINVAL ; + } +} +static int kvm_s390_handle_pv (struct kvm *kvm,struct kvm_pv_cmd *cmd) {int r =0 ;@@ -2360,6 +2396,46 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) cmd->rc, cmd->rrc);break ; } +case KVM_PV_INFO : { +struct kvm_s390_pv_info info = {}; +ssize_t data_len; + +/* +* No need to check the VM protection here. +* +* Maybe user space wants to query some of the data +* when the VM is still unprotected. If we see the +* need to fence a new data command we can still +* return an error in the info handler. +*/ + + r = -EFAULT ; +if (copy_from_user (&info, argp,sizeof (info.header))) +break ; + + r = -EINVAL ; +if (info.header.len_max <sizeof (info.header)) +break ; + + data_len =kvm_s390_handle_pv_info (&info); +if (data_len <0 ) { + r = data_len; +break ; + } +/* +* If a data command struct is extended (multiple +* times) this can be used to determine how much of it +* is valid. +*/ + info.header.len_written = data_len; + + r = -EFAULT ; +if (copy_to_user (argp, &info, data_len)) +break ; + + r =0 ; +break ; + }default : r = -ENOTTY ; }include/uapi/linux/kvm.h +25
-0@@ -1643,6 +1643,30 @@ struct kvm_s390_pv_unp {__u64 tweak; }; +enum pv_cmd_info_id { +KVM_PV_INFO_VM , +}; + +struct kvm_s390_pv_info_vm { +__u64 inst_calls_list[4 ]; +__u64 max_cpus; +__u64 max_guests; +__u64 max_guest_addr; +__u64 feature_indication; +}; + +struct kvm_s390_pv_info_header { +__u32 id; +__u32 len_max; +__u32 len_written; +__u32 reserved; +}; + +struct kvm_s390_pv_info { +struct kvm_s390_pv_info_header header; +struct kvm_s390_pv_info_vm vm; +}; +enum pv_cmd_id {KVM_PV_ENABLE ,KVM_PV_DISABLE ,@@ -1651,6 +1675,7 @@ enum pv_cmd_id {KVM_PV_VERIFY ,KVM_PV_PREP_RESET ,KVM_PV_UNSHARE_ALL , + KVM_PV_INFO, }; struct kvm_pv_cmd {
Janosch Frank
maintainerRe: [PATCH v2 3/9] KVM: s390: pv: Add query interface
Quoted 6 lines · Claudio Imbrenda
Quoted 2 lines · On Thu, 10 Mar 2022 10:31:06 +0000 Janosch Frank <[email protected]> wrote:
Some of the query information is already available via sysfs but having a IOCTL makes the information easier to retrieve.
Quoted 8 lines
if I understand correctly, this will be forward-compatible but not backwards compatible.
you return the amount of bytes written into the buffer, but only if the buffer was already big enough.
a newer userspace will work with an older kernel, but an older userspace will not work with a newer kernel.
I expect the first version of userspace to request a minimum length hence I return -EINVAL if less space is given.
In the future the minimum will be a constant and we'll write between the min and the new data length.
IMHO there's no sense in allowing to request less data than the v1 of the interface will provide.
Quoted 159 lines
a solution would be to return the size of the struct, so userspace can know how much of the buffer was written (if it was bigger than the struct), or that there are unwritten bits (if the buffer was smaller).
and even if the buffer was too small, write back as much of it as possible to userspace.
this way, an older userspace will get the information it expects.
I am also not a big fan of writing the size in the input struct (I think returning it would be cleaner), but I do not have a strong opinion
Quoted 144 lines
Signed-off-by Janosch Frank <[email protected]>--- arch/s390/kvm/kvm-s390.c | 76 ++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 25 +++++++++++++ 2 files changed, 101 insertions(+)arch/s390/kvm/kvm-s390.c +76
-0@@ -2224,6 +2224,42 @@ static int kvm_s390_cpus_to_pv(struct kvm *kvm, u16 *rc, u16 *rrc)return r; } +/* +* Here we provide user space with a direct interface to query UV +* related data like UV maxima and available features as well as +* feature specific data. +* +* To facilitate future extension of the data structures we'll try to +* write data up to the maximum requested length. +*/ +static ssize_t kvm_s390_handle_pv_info (struct kvm_s390_pv_info *info) +{ +ssize_t len; + +switch (info->header.id) { +case KVM_PV_INFO_VM : { + len =sizeof (info->header) +sizeof (info->vm); + +if (info->header.len_max < len) +return -EINVAL ; + +memcpy (info->vm.inst_calls_list, + uv_info.inst_calls_list, +sizeof (uv_info.inst_calls_list)); + +/* It's max cpuidm not max cpus so it's off by one */ + info->vm.max_cpus = uv_info.max_guest_cpu_id +1 ; + info->vm.max_guests = uv_info.max_num_sec_conf; + info->vm.max_guest_addr = uv_info.max_sec_stor_addr; + info->vm.feature_indication = uv_info.uv_feature_indications; + +return len; + } +default : +return -EINVAL ; + } +} +static int kvm_s390_handle_pv (struct kvm *kvm,struct kvm_pv_cmd *cmd) {int r =0 ;@@ -2360,6 +2396,46 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) cmd->rc, cmd->rrc);break ; } +case KVM_PV_INFO : { +struct kvm_s390_pv_info info = {}; +ssize_t data_len; + +/* +* No need to check the VM protection here. +* +* Maybe user space wants to query some of the data +* when the VM is still unprotected. If we see the +* need to fence a new data command we can still +* return an error in the info handler. +*/ + + r = -EFAULT ; +if (copy_from_user (&info, argp,sizeof (info.header))) +break ; + + r = -EINVAL ; +if (info.header.len_max <sizeof (info.header)) +break ; + + data_len =kvm_s390_handle_pv_info (&info); +if (data_len <0 ) { + r = data_len; +break ; + } +/* +* If a data command struct is extended (multiple +* times) this can be used to determine how much of it +* is valid. +*/ + info.header.len_written = data_len; + + r = -EFAULT ; +if (copy_to_user (argp, &info, data_len)) +break ; + + r =0 ; +break ; + }default : r = -ENOTTY ; }include/uapi/linux/kvm.h +25
-0@@ -1643,6 +1643,30 @@ struct kvm_s390_pv_unp {__u64 tweak; }; +enum pv_cmd_info_id { +KVM_PV_INFO_VM , +}; + +struct kvm_s390_pv_info_vm { +__u64 inst_calls_list[4 ]; +__u64 max_cpus; +__u64 max_guests; +__u64 max_guest_addr; +__u64 feature_indication; +}; + +struct kvm_s390_pv_info_header { +__u32 id; +__u32 len_max; +__u32 len_written; +__u32 reserved; +}; + +struct kvm_s390_pv_info { +struct kvm_s390_pv_info_header header; +struct kvm_s390_pv_info_vm vm; +}; +enum pv_cmd_id {KVM_PV_ENABLE ,KVM_PV_DISABLE ,@@ -1651,6 +1675,7 @@ enum pv_cmd_id {KVM_PV_VERIFY ,KVM_PV_PREP_RESET ,KVM_PV_UNSHARE_ALL , + KVM_PV_INFO, }; struct kvm_pv_cmd {
Claudio Imbrenda
maintainerRe: [PATCH v2 3/9] KVM: s390: pv: Add query interface
Quoted 10 lines · On Mon, 14 Mar 2022 11:02:40 +0100 Janosch Frank <[email protected]> wrote:
Quoted 9 lines · Claudio Imbrenda
Quoted 2 lines · On Thu, 10 Mar 2022 10:31:06 +0000 Janosch Frank <[email protected]> wrote:
Some of the query information is already available via sysfs but having a IOCTL makes the information easier to retrieve.
if I understand correctly, this will be forward-compatible but not backwards compatible.
Quoted 8 lines
Quoted 5 lines
you return the amount of bytes written into the buffer, but only if the buffer was already big enough.
a newer userspace will work with an older kernel, but an older userspace will not work with a newer kernel.
I expect the first version of userspace to request a minimum length hence I return
-EINVALif less space is given.
this makes sense
Quoted 3 lines
In the future the minimum will be a constant and we'll write between the min and the new data length.
fair enough, but this means that future patches must be careful and not forget this.
it's probably easier to read, and more future proof, if you put a check for the minimum size (and perhaps also a comment to explain why), and not use sizeof(struct ...)
Quoted 3 lines
IMHO there's no sense in allowing to request less data than the v1 of the interface will provide.
of course
Quoted 162 lines
Quoted 161 lines
a solution would be to return the size of the struct, so userspace can know how much of the buffer was written (if it was bigger than the struct), or that there are unwritten bits (if the buffer was smaller).
and even if the buffer was too small, write back as much of it as possible to userspace.
this way, an older userspace will get the information it expects.
I am also not a big fan of writing the size in the input struct (I think returning it would be cleaner), but I do not have a strong opinion
Quoted 144 lines
Signed-off-by Janosch Frank <[email protected]>--- arch/s390/kvm/kvm-s390.c | 76 ++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 25 +++++++++++++ 2 files changed, 101 insertions(+)arch/s390/kvm/kvm-s390.c +76
-0@@ -2224,6 +2224,42 @@ static int kvm_s390_cpus_to_pv(struct kvm *kvm, u16 *rc, u16 *rrc)return r; } +/* +* Here we provide user space with a direct interface to query UV +* related data like UV maxima and available features as well as +* feature specific data. +* +* To facilitate future extension of the data structures we'll try to +* write data up to the maximum requested length. +*/ +static ssize_t kvm_s390_handle_pv_info (struct kvm_s390_pv_info *info) +{ +ssize_t len; + +switch (info->header.id) { +case KVM_PV_INFO_VM : { + len =sizeof (info->header) +sizeof (info->vm); + +if (info->header.len_max < len) +return -EINVAL ; + +memcpy (info->vm.inst_calls_list, + uv_info.inst_calls_list, +sizeof (uv_info.inst_calls_list)); + +/* It's max cpuidm not max cpus so it's off by one */ + info->vm.max_cpus = uv_info.max_guest_cpu_id +1 ; + info->vm.max_guests = uv_info.max_num_sec_conf; + info->vm.max_guest_addr = uv_info.max_sec_stor_addr; + info->vm.feature_indication = uv_info.uv_feature_indications; + +return len; + } +default : +return -EINVAL ; + } +} +static int kvm_s390_handle_pv (struct kvm *kvm,struct kvm_pv_cmd *cmd) {int r =0 ;@@ -2360,6 +2396,46 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) cmd->rc, cmd->rrc);break ; } +case KVM_PV_INFO : { +struct kvm_s390_pv_info info = {}; +ssize_t data_len; + +/* +* No need to check the VM protection here. +* +* Maybe user space wants to query some of the data +* when the VM is still unprotected. If we see the +* need to fence a new data command we can still +* return an error in the info handler. +*/ + + r = -EFAULT ; +if (copy_from_user (&info, argp,sizeof (info.header))) +break ; + + r = -EINVAL ; +if (info.header.len_max <sizeof (info.header)) +break ; + + data_len =kvm_s390_handle_pv_info (&info); +if (data_len <0 ) { + r = data_len; +break ; + } +/* +* If a data command struct is extended (multiple +* times) this can be used to determine how much of it +* is valid. +*/ + info.header.len_written = data_len; + + r = -EFAULT ; +if (copy_to_user (argp, &info, data_len)) +break ; + + r =0 ; +break ; + }default : r = -ENOTTY ; }include/uapi/linux/kvm.h +25
-0@@ -1643,6 +1643,30 @@ struct kvm_s390_pv_unp {__u64 tweak; }; +enum pv_cmd_info_id { +KVM_PV_INFO_VM , +}; + +struct kvm_s390_pv_info_vm { +__u64 inst_calls_list[4 ]; +__u64 max_cpus; +__u64 max_guests; +__u64 max_guest_addr; +__u64 feature_indication; +}; + +struct kvm_s390_pv_info_header { +__u32 id; +__u32 len_max; +__u32 len_written; +__u32 reserved; +}; + +struct kvm_s390_pv_info { +struct kvm_s390_pv_info_header header; +struct kvm_s390_pv_info_vm vm; +}; +enum pv_cmd_id {KVM_PV_ENABLE ,KVM_PV_DISABLE ,@@ -1651,6 +1675,7 @@ enum pv_cmd_id {KVM_PV_VERIFY ,KVM_PV_PREP_RESET ,KVM_PV_UNSHARE_ALL , + KVM_PV_INFO, }; struct kvm_pv_cmd {
Janosch Frank
maintainerRe: [PATCH v2 3/9] KVM: s390: pv: Add query interface
Quoted 22 lines · Claudio Imbrenda
Quoted 10 lines · On Mon, 14 Mar 2022 11:02:40 +0100 Janosch Frank <[email protected]> wrote:
Quoted 9 lines · Claudio Imbrenda
Quoted 2 lines · On Thu, 10 Mar 2022 10:31:06 +0000 Janosch Frank <[email protected]> wrote:
Some of the query information is already available via sysfs but having a IOCTL makes the information easier to retrieve.
if I understand correctly, this will be forward-compatible but not backwards compatible.
Quoted 8 lines
Quoted 5 lines
you return the amount of bytes written into the buffer, but only if the buffer was already big enough.
a newer userspace will work with an older kernel, but an older userspace will not work with a newer kernel.
I expect the first version of userspace to request a minimum length hence I return
-EINVALif less space is given.
Quoted 8 lines
this makes sense
Quoted 3 lines
In the future the minimum will be a constant and we'll write between the min and the new data length.
fair enough, but this means that future patches must be careful and not forget this.
Sure, I'll find a way to make this clearer.
Quoted 174 lines
it's probably easier to read, and more future proof, if you put a check for the minimum size (and perhaps also a comment to explain why), and not use sizeof(struct ...)
Quoted 3 lines
IMHO there's no sense in allowing to request less data than the v1 of the interface will provide.
of course
Quoted 162 lines
Quoted 161 lines
a solution would be to return the size of the struct, so userspace can know how much of the buffer was written (if it was bigger than the struct), or that there are unwritten bits (if the buffer was smaller).
and even if the buffer was too small, write back as much of it as possible to userspace.
this way, an older userspace will get the information it expects.
I am also not a big fan of writing the size in the input struct (I think returning it would be cleaner), but I do not have a strong opinion
Quoted 144 lines
Signed-off-by Janosch Frank <[email protected]>--- arch/s390/kvm/kvm-s390.c | 76 ++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 25 +++++++++++++ 2 files changed, 101 insertions(+)arch/s390/kvm/kvm-s390.c +76
-0@@ -2224,6 +2224,42 @@ static int kvm_s390_cpus_to_pv(struct kvm *kvm, u16 *rc, u16 *rrc)return r; } +/* +* Here we provide user space with a direct interface to query UV +* related data like UV maxima and available features as well as +* feature specific data. +* +* To facilitate future extension of the data structures we'll try to +* write data up to the maximum requested length. +*/ +static ssize_t kvm_s390_handle_pv_info (struct kvm_s390_pv_info *info) +{ +ssize_t len; + +switch (info->header.id) { +case KVM_PV_INFO_VM : { + len =sizeof (info->header) +sizeof (info->vm); + +if (info->header.len_max < len) +return -EINVAL ; + +memcpy (info->vm.inst_calls_list, + uv_info.inst_calls_list, +sizeof (uv_info.inst_calls_list)); + +/* It's max cpuidm not max cpus so it's off by one */ + info->vm.max_cpus = uv_info.max_guest_cpu_id +1 ; + info->vm.max_guests = uv_info.max_num_sec_conf; + info->vm.max_guest_addr = uv_info.max_sec_stor_addr; + info->vm.feature_indication = uv_info.uv_feature_indications; + +return len; + } +default : +return -EINVAL ; + } +} +static int kvm_s390_handle_pv (struct kvm *kvm,struct kvm_pv_cmd *cmd) {int r =0 ;@@ -2360,6 +2396,46 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) cmd->rc, cmd->rrc);break ; } +case KVM_PV_INFO : { +struct kvm_s390_pv_info info = {}; +ssize_t data_len; + +/* +* No need to check the VM protection here. +* +* Maybe user space wants to query some of the data +* when the VM is still unprotected. If we see the +* need to fence a new data command we can still +* return an error in the info handler. +*/ + + r = -EFAULT ; +if (copy_from_user (&info, argp,sizeof (info.header))) +break ; + + r = -EINVAL ; +if (info.header.len_max <sizeof (info.header)) +break ; + + data_len =kvm_s390_handle_pv_info (&info); +if (data_len <0 ) { + r = data_len; +break ; + } +/* +* If a data command struct is extended (multiple +* times) this can be used to determine how much of it +* is valid. +*/ + info.header.len_written = data_len; + + r = -EFAULT ; +if (copy_to_user (argp, &info, data_len)) +break ; + + r =0 ; +break ; + }default : r = -ENOTTY ; }include/uapi/linux/kvm.h +25
-0@@ -1643,6 +1643,30 @@ struct kvm_s390_pv_unp {__u64 tweak; }; +enum pv_cmd_info_id { +KVM_PV_INFO_VM , +}; + +struct kvm_s390_pv_info_vm { +__u64 inst_calls_list[4 ]; +__u64 max_cpus; +__u64 max_guests; +__u64 max_guest_addr; +__u64 feature_indication; +}; + +struct kvm_s390_pv_info_header { +__u32 id; +__u32 len_max; +__u32 len_written; +__u32 reserved; +}; + +struct kvm_s390_pv_info { +struct kvm_s390_pv_info_header header; +struct kvm_s390_pv_info_vm vm; +}; +enum pv_cmd_id {KVM_PV_ENABLE ,KVM_PV_DISABLE ,@@ -1651,6 +1675,7 @@ enum pv_cmd_id {KVM_PV_VERIFY ,KVM_PV_PREP_RESET ,KVM_PV_UNSHARE_ALL , + KVM_PV_INFO, }; struct kvm_pv_cmd {
Janosch Frank
maintainerPATCH v2 4/9 KVM: s390: pv: Add dump support definitions
In reply to Janosch FrankLets add the constants and structure definitions needed for the dump support.
--- arch/s390/include/asm/uv.h | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
arch/s390/include/asm/uv.h +33 -0
@@ -50,6 +50,10 @@#define UVC_CMD_SET_UNSHARE_ALL 0x0340#define UVC_CMD_PIN_PAGE_SHARED 0x0341#define UVC_CMD_UNPIN_PAGE_SHARED 0x0342 +#define UVC_CMD_DUMP_INIT 0x0400 +#define UVC_CMD_DUMP_CONF_STOR_STATE 0x0401 +#define UVC_CMD_DUMP_CPU 0x0402 +#define UVC_CMD_DUMP_COMPLETE 0x0403#define UVC_CMD_SET_SHARED_ACCESS 0x1000#define UVC_CMD_REMOVE_SHARED_ACCESS 0x1001
@@ -76,6 +80,10 @@ enum uv_cmds_inst {BIT_UVC_CMD_UNSHARE_ALL =20 ,BIT_UVC_CMD_PIN_PAGE_SHARED =21 ,BIT_UVC_CMD_UNPIN_PAGE_SHARED =22 , +BIT_UVC_CMD_DUMP_INIT =24 , +BIT_UVC_CMD_DUMP_CONFIG_STOR_STATE =25 , +BIT_UVC_CMD_DUMP_CPU =26 , + BIT_UVC_CMD_DUMP_COMPLETE = 27, }; enum uv_feat_ind {
@@ -225,6 +233,31 @@ struct uv_cb_share {u64 reserved28; } __packed__aligned(8) ; +struct uv_cb_dump_cpu { +struct uv_cb_header header; +u64 reserved08[2 ]; +u64 cpu_handle; +u64 dump_area_origin; +u64 reserved28[5 ]; +} __packed__aligned(8) ; + +struct uv_cb_dump_stor_state { +struct uv_cb_header header; +u64 reserved08[2 ]; +u64 config_handle; +u64 dump_area_origin; +u64 gaddr; +u64 reserved28[4 ]; +} __packed__aligned(8) ; + +struct uv_cb_dump_complete { +struct uv_cb_header header; +u64 reserved08[2 ]; +u64 config_handle; +u64 dump_area_origin; +u64 reserved30[5 ]; +} __packed__aligned(8) ; +static inline int __uv_call (unsigned long r1,unsigned long r2) {int cc;
Signature · 1 line
2.32.0
Re: [PATCH v2 4/9] KVM: s390: pv: Add dump support definitions
Quoted 4 lines · On Thu, 10 Mar 2022 10:31:07 +0000 Janosch Frank <[email protected]> wrote:
Lets add the constants and structure definitions needed for the dump support.
Signed-off-by Janosch Frank <[email protected]>
Quoted 62 lines
--- arch/s390/include/asm/uv.h | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)arch/s390/include/asm/uv.h +33
-0@@ -50,6 +50,10 @@#define UVC_CMD_SET_UNSHARE_ALL 0x0340#define UVC_CMD_PIN_PAGE_SHARED 0x0341#define UVC_CMD_UNPIN_PAGE_SHARED 0x0342 +#define UVC_CMD_DUMP_INIT 0x0400 +#define UVC_CMD_DUMP_CONF_STOR_STATE 0x0401 +#define UVC_CMD_DUMP_CPU 0x0402 +#define UVC_CMD_DUMP_COMPLETE 0x0403#define UVC_CMD_SET_SHARED_ACCESS 0x1000#define UVC_CMD_REMOVE_SHARED_ACCESS 0x1001@@ -76,6 +80,10 @@ enum uv_cmds_inst {BIT_UVC_CMD_UNSHARE_ALL =20 ,BIT_UVC_CMD_PIN_PAGE_SHARED =21 ,BIT_UVC_CMD_UNPIN_PAGE_SHARED =22 , +BIT_UVC_CMD_DUMP_INIT =24 , +BIT_UVC_CMD_DUMP_CONFIG_STOR_STATE =25 , +BIT_UVC_CMD_DUMP_CPU =26 , + BIT_UVC_CMD_DUMP_COMPLETE = 27, }; enum uv_feat_ind {@@ -225,6 +233,31 @@ struct uv_cb_share {u64 reserved28; } __packed__aligned(8) ; +struct uv_cb_dump_cpu { +struct uv_cb_header header; +u64 reserved08[2 ]; +u64 cpu_handle; +u64 dump_area_origin; +u64 reserved28[5 ]; +} __packed__aligned(8) ; + +struct uv_cb_dump_stor_state { +struct uv_cb_header header; +u64 reserved08[2 ]; +u64 config_handle; +u64 dump_area_origin; +u64 gaddr; +u64 reserved28[4 ]; +} __packed__aligned(8) ; + +struct uv_cb_dump_complete { +struct uv_cb_header header; +u64 reserved08[2 ]; +u64 config_handle; +u64 dump_area_origin; +u64 reserved30[5 ]; +} __packed__aligned(8) ; +static inline int __uv_call (unsigned long r1,unsigned long r2) {int cc;
Janosch Frank
maintainerPATCH v2 5/9 KVM: s390: pv: Add query dump information
In reply to Janosch FrankThe dump API requires userspace to provide buffers into which we will store data. The dump information added in this patch tells userspace how big those buffers need to be.
--- arch/s390/kvm/kvm-s390.c | 11 +++++++++++ include/uapi/linux/kvm.h | 12 +++++++++++- 2 files changed, 22 insertions(+), 1 deletion(-)
arch/s390/kvm/kvm-s390.c +11 -0
@@ -2255,6 +2255,17 @@ static ssize_t kvm_s390_handle_pv_info(struct kvm_s390_pv_info *info)return len; } +case KVM_PV_INFO_DUMP : { + len =sizeof (info->header) +sizeof (info->dump); + +if (info->header.len_max < len) +return -EINVAL ; + + info->dump.dump_cpu_buffer_len = uv_info.guest_cpu_stor_len; + info->dump.dump_config_mem_buffer_per_1m = uv_info.conf_dump_storage_state_len; + info->dump.dump_config_finalize_len = uv_info.conf_dump_finalize_len; +return len; + }default :return -EINVAL ; }
include/uapi/linux/kvm.h +11 -1
@@ -1645,6 +1645,13 @@ struct kvm_s390_pv_unp {enum pv_cmd_info_id {KVM_PV_INFO_VM , +KVM_PV_INFO_DUMP , +}; + +struct kvm_s390_pv_info_dump { +__u64 dump_cpu_buffer_len; +__u64 dump_config_mem_buffer_per_1m; +__u64 dump_config_finalize_len; };struct kvm_s390_pv_info_vm {
@@ -1664,7 +1671,10 @@ struct kvm_s390_pv_info_header {struct kvm_s390_pv_info {struct kvm_s390_pv_info_header header; -struct kvm_s390_pv_info_vm vm; +union { +struct kvm_s390_pv_info_dump dump; +struct kvm_s390_pv_info_vm vm; + }; };enum pv_cmd_id {
Signature · 1 line
2.32.0
Claudio Imbrenda
maintainerRe: [PATCH v2 5/9] KVM: s390: pv: Add query dump information
Quoted 3 lines · On Thu, 10 Mar 2022 10:31:08 +0000 Janosch Frank <[email protected]> wrote:
The dump API requires userspace to provide buffers into which we will store data. The dump information added in this patch tells userspace how big those buffers need to be.
it seems this has the same issue as the other patch.
if userspace gives a smaller buffer, as much data as possible should be written into it
Quoted 59 lines
Signed-off-by Janosch Frank <[email protected]>--- arch/s390/kvm/kvm-s390.c | 11 +++++++++++ include/uapi/linux/kvm.h | 12 +++++++++++- 2 files changed, 22 insertions(+), 1 deletion(-)arch/s390/kvm/kvm-s390.c +11
-0@@ -2255,6 +2255,17 @@ static ssize_t kvm_s390_handle_pv_info(struct kvm_s390_pv_info *info)return len; } +case KVM_PV_INFO_DUMP : { + len =sizeof (info->header) +sizeof (info->dump); + +if (info->header.len_max < len) +return -EINVAL ; + + info->dump.dump_cpu_buffer_len = uv_info.guest_cpu_stor_len; + info->dump.dump_config_mem_buffer_per_1m = uv_info.conf_dump_storage_state_len; + info->dump.dump_config_finalize_len = uv_info.conf_dump_finalize_len; +return len; + }default :return -EINVAL ; }include/uapi/linux/kvm.h +11
-1@@ -1645,6 +1645,13 @@ struct kvm_s390_pv_unp {enum pv_cmd_info_id {KVM_PV_INFO_VM , +KVM_PV_INFO_DUMP , +}; + +struct kvm_s390_pv_info_dump { +__u64 dump_cpu_buffer_len; +__u64 dump_config_mem_buffer_per_1m; +__u64 dump_config_finalize_len; };struct kvm_s390_pv_info_vm {@@ -1664,7 +1671,10 @@ struct kvm_s390_pv_info_header {struct kvm_s390_pv_info {struct kvm_s390_pv_info_header header; -struct kvm_s390_pv_info_vm vm; +union { +struct kvm_s390_pv_info_dump dump; +struct kvm_s390_pv_info_vm vm; + }; };enum pv_cmd_id {
Janosch Frank
maintainerPATCH v2 6/9 kvm: s390: Add configuration dump functionality
In reply to Janosch FrankSometimes dumping inside of a VM fails, is unavailable or doesn't yield the required data. For these occasions we dump the VM from the outside, writing memory and cpu data to a file.
Up to now PV guests only supported dumping from the inside of the guest through dumpers like KDUMP. A PV guest can be dumped from the hypervisor but the data will be stale and / or encrypted.
To get the actual state of the PV VM we need the help of the Ultravisor who safeguards the VM state. New UV calls have been added to initialize the dump, dump storage state data, dump cpu data and complete the dump process. We expose these calls in this patch via a new UV ioctl command.
The sensitive parts of the dump data are encrypted, the dump key is derived from the Customer Communication Key (CCK). This ensures that only the owner of the VM who has the CCK can decrypt the dump data.
The memory is dumped / read via a normal export call and a re-import after the dump initialization is not needed (no re-encryption with a dump key).
--- arch/s390/include/asm/kvm_host.h | 1 + arch/s390/kvm/kvm-s390.c | 137 +++++++++++++++++++++++++++++++ arch/s390/kvm/kvm-s390.h | 2 + arch/s390/kvm/pv.c | 115 ++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 15 ++++ 5 files changed, 270 insertions(+)
arch/s390/include/asm/kvm_host.h +1 -0
@@ -921,6 +921,7 @@ struct kvm_s390_pv {u64 guest_len;unsigned long stor_base;void *stor_var; +bool dumping; };struct kvm_arch {
arch/s390/kvm/kvm-s390.c +137 -0
161 lines not on this page · raw message
arch/s390/kvm/kvm-s390.h +2 -0
@@ -250,6 +250,8 @@ int kvm_s390_pv_set_sec_parms(struct kvm *kvm, void *hdr, u64 length, u16 *rc,int kvm_s390_pv_unpack (struct kvm *kvm,unsigned long addr,unsigned long size,unsigned long tweak,u16 *rc,u16 *rrc);int kvm_s390_pv_set_cpu_state (struct kvm_vcpu *vcpu,u8 state); +int kvm_s390_pv_dump_stor_state (struct kvm *kvm,void __user *buff_user, +u64 *gaddr,u64 buff_user_len,u16 *rc,u16 *rrc);static inline u64 kvm_s390_pv_get_handle (struct kvm *kvm) {
arch/s390/kvm/pv.c +115 -0
118 lines not on this page · raw message
include/uapi/linux/kvm.h +15 -0
@@ -1643,6 +1643,20 @@ struct kvm_s390_pv_unp {__u64 tweak; }; +enum pv_cmd_dmp_id { +KVM_PV_DUMP_INIT , +KVM_PV_DUMP_CONFIG_STOR_STATE , +KVM_PV_DUMP_COMPLETE , +}; + +struct kvm_s390_pv_dmp { +__u64 subcmd; +__u64 buff_addr; +__u64 buff_len; +__u64 gaddr;/* For dump storage state */ +__u64 reserved[4 ]; +}; +enum pv_cmd_info_id {KVM_PV_INFO_VM ,KVM_PV_INFO_DUMP ,
@@ -1686,6 +1700,7 @@ enum pv_cmd_id {KVM_PV_PREP_RESET ,KVM_PV_UNSHARE_ALL ,KVM_PV_INFO , + KVM_PV_DUMP, }; struct kvm_pv_cmd {
Signature · 1 line
2.32.0
Claudio Imbrenda
maintainerRe: [PATCH v2 6/9] kvm: s390: Add configuration dump functionality
Quoted 48 lines · On Thu, 10 Mar 2022 10:31:09 +0000 Janosch Frank <[email protected]> wrote:
Sometimes dumping inside of a VM fails, is unavailable or doesn't yield the required data. For these occasions we dump the VM from the outside, writing memory and cpu data to a file.
Up to now PV guests only supported dumping from the inside of the guest through dumpers like KDUMP. A PV guest can be dumped from the hypervisor but the data will be stale and / or encrypted.
To get the actual state of the PV VM we need the help of the Ultravisor who safeguards the VM state. New UV calls have been added to initialize the dump, dump storage state data, dump cpu data and complete the dump process. We expose these calls in this patch via a new UV ioctl command.
The sensitive parts of the dump data are encrypted, the dump key is derived from the Customer Communication Key (CCK). This ensures that only the owner of the VM who has the CCK can decrypt the dump data.
The memory is dumped / read via a normal export call and a re-import after the dump initialization is not needed (no re-encryption with a dump key).
Signed-off-by Janosch Frank <[email protected]>--- arch/s390/include/asm/kvm_host.h | 1 + arch/s390/kvm/kvm-s390.c | 137 +++++++++++++++++++++++++++++++ arch/s390/kvm/kvm-s390.h | 2 + arch/s390/kvm/pv.c | 115 ++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 15 ++++ 5 files changed, 270 insertions(+)arch/s390/include/asm/kvm_host.h +1
-0@@ -921,6 +921,7 @@ struct kvm_s390_pv {u64 guest_len;unsigned long stor_base;void *stor_var; +bool dumping; };struct kvm_arch {arch/s390/kvm/kvm-s390.c +0
-0
arch/s390/kvm/kvm-s390.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 613
Quoted 8 lines
arch/s390/kvm/kvm-s390.c +5
-0@@ -606,6 +606,26 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_S390_PROTECTED : r =is_prot_virt_host ();break ; +case KVM_CAP_S390_PROTECTED_DUMP : { +u64 pv_cmds_dump[] = { +BIT_UVC_CMD_DUMP_INIT , +BIT_UVC_CMD_DUMP_CONFIG_STOR_STATE , +BIT_UVC_CMD_DUMP_CONFIG_STOR_STATE ,
you have this twice ^
Quoted 19 lines
arch/s390/kvm/kvm-s390.c +15
-0+ BIT_UVC_CMD_DUMP_CPU, + }; +int i; + +if (!is_prot_virt_host ()) +return 0 ; + + r =1 ; +for (i =0 ; i <ARRAY_SIZE (pv_cmds_dump); i++) { +if (!test_bit_inv (pv_cmds_dump[i], + (unsigned long *)&uv_info.inst_calls_list)) +return 0 ; + } +break ; + }default : r =0 ; }
arch/s390/kvm/kvm-s390.c static ssize_t kvm_s390_handle_pv_info(struct kvm_s390_pv_info *info) · line 2294
Quoted 4 lines
arch/s390/kvm/kvm-s390.c +1
-0@@ -2271,6 +2291,92 @@ static ssize_t kvm_s390_handle_pv_info(struct kvm_s390_pv_info *info) } } +static int kvm_s390_pv_dmp(struct kvm *kvm, struct kvm_pv_cmd *cmd,
I am all for shortening long words, but dmp -> dump is only one extra byte
Quoted 6 lines
arch/s390/kvm/kvm-s390.c +6
-0+ struct kvm_s390_pv_dmp dmp) +{ +int r = -EINVAL ; +void __user *result_buff = (void __user *)dmp.buff_addr; + +switch (dmp.subcmd) {
arch/s390/kvm/kvm-s390.c static ssize_t kvm_s390_handle_pv_info(struct kvm_s390_pv_info *info) · line 2310
Quoted 10 lines
arch/s390/kvm/kvm-s390.c +10
-0+case KVM_PV_DUMP_INIT : { +if (kvm->arch.pv.dumping) +break ; + + r =uv_cmd_nodata (kvm_s390_pv_get_handle (kvm), +UVC_CMD_DUMP_INIT , &cmd->rc, &cmd->rrc); +KVM_UV_EVENT (kvm,3 ,"PROTVIRT DUMP INIT: rc %x rrc %x" , + cmd->rc, cmd->rrc); +if (!r) + kvm->arch.pv.dumping = true;
so at this point no guest CPUs should be running, right? below you add a check to prevent guest CPUs from being dispatched if we are in dump mode.
are there any checks to make sure all guest CPUs are stopped when issuing this command? could a guest CPU be running while this command is issued?
arch/s390/kvm/kvm-s390.c static ssize_t kvm_s390_handle_pv_info(struct kvm_s390_pv_info *info) · line 2320
Quoted 10 lines
arch/s390/kvm/kvm-s390.c +10
-0+else + r = -EINVAL ; +break ; + } +case KVM_PV_DUMP_CONFIG_STOR_STATE : { +if (!kvm->arch.pv.dumping) +break ; + +/* +* GADDR is an output parameter since we might stop
why uppercase?
Quoted 162 lines
arch/s390/kvm/kvm-s390.c +90
-0+* early. As dmp will be copied back in our caller, we +* don't need to do it ourselves. +*/ + r =kvm_s390_pv_dump_stor_state (kvm, result_buff, &dmp.gaddr, dmp.buff_len, + &cmd->rc, &cmd->rrc); +break ; + } +case KVM_PV_DUMP_COMPLETE : { +struct uv_cb_dump_complete complete = { + .header.len =sizeof (complete), + .header.cmd =UVC_CMD_DUMP_COMPLETE , + .config_handle =kvm_s390_pv_get_handle (kvm), + }; +u64 *compl_data; + + r = -EINVAL ; +if (!kvm->arch.pv.dumping) +break ; + +if (dmp.buff_len < uv_info.conf_dump_finalize_len) +break ; + +/* Allocate dump area */ + r = -ENOMEM ; + compl_data =vzalloc (uv_info.conf_dump_finalize_len); +if (!compl_data) +break ; + complete.dump_area_origin = (u64 )compl_data; + + r =uv_call (0 , (u64)&complete); + cmd->rc = complete.header.rc; + cmd->rrc = complete.header.rrc; +KVM_UV_EVENT (kvm,3 ,"PROTVIRT DUMP COMPLETE: rc %x rrc %x" , + complete.header.rc, complete.header.rrc); + +if (!r) { +/* +* kvm_s390_pv_dealloc_vm() will also (mem)set +* this to false on a reboot or other destroy +* operation for this vm. +*/ + kvm->arch.pv.dumping = false; + r =copy_to_user (result_buff, compl_data, uv_info.conf_dump_finalize_len); +if (r) + r = -EFAULT ; + } +vfree (compl_data); +if (r >0 ) + r = -EINVAL ; +break ; + } +default : + r = -ENOTTY ; +break ; + } + +return r; +} +static int kvm_s390_handle_pv (struct kvm *kvm,struct kvm_pv_cmd *cmd) {int r =0 ;@@ -2447,6 +2553,28 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) r =0 ;break ; } +case KVM_PV_DUMP : { +struct kvm_s390_pv_dmp dmp; + + r = -EINVAL ; +if (!kvm_s390_pv_is_protected (kvm)) +break ; + + r = -EFAULT ; +if (copy_from_user (&dmp, argp,sizeof (dmp))) +break ; + + r =kvm_s390_pv_dmp (kvm, cmd, dmp); +if (r) +break ; + +if (copy_to_user (argp, &dmp,sizeof (dmp))) { + r = -EFAULT ; +break ; + } + +break ; + }default : r = -ENOTTY ; }@@ -4542,6 +4670,15 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)struct kvm_run *kvm_run = vcpu->run;int rc; +/* +* Running a VM while dumping always has the potential to +* produce inconsistent dump data. But for PV vcpus a SIE +* entry while dumping could also lead to a validity which we +* absolutely want to avoid. +*/ +if (vcpu->kvm->arch.pv.dumping) +return -EINVAL ; +if (kvm_run->immediate_exit)return -EINTR ;arch/s390/kvm/kvm-s390.h +2
-0@@ -250,6 +250,8 @@ int kvm_s390_pv_set_sec_parms(struct kvm *kvm, void *hdr, u64 length, u16 *rc,int kvm_s390_pv_unpack (struct kvm *kvm,unsigned long addr,unsigned long size,unsigned long tweak,u16 *rc,u16 *rrc);int kvm_s390_pv_set_cpu_state (struct kvm_vcpu *vcpu,u8 state); +int kvm_s390_pv_dump_stor_state (struct kvm *kvm,void __user *buff_user, +u64 *gaddr,u64 buff_user_len,u16 *rc,u16 *rrc);static inline u64 kvm_s390_pv_get_handle (struct kvm *kvm) {arch/s390/kvm/pv.c +34
-0@@ -303,3 +303,118 @@ int kvm_s390_pv_set_cpu_state(struct kvm_vcpu *vcpu, u8 state)return -EINVAL ;return 0 ; } + +/* Size of the cache for the storage state dump data. 1MB for now */ +#define DUMP_BUFF_LEN HPAGE_SIZE + +/* +* kvm_s390_pv_dump_stor_state +* +* @kvm: pointer to the guest's KVM struct +* @buff_user: Userspace pointer where we will write the results to +* @gaddr: Starting absolute guest address for which the storage state +* is requested. This value will be updated with the last +* address for which data was written when returning to +* userspace. +* @buff_user_len: Length of the buff_user buffer +* @rc: Pointer to where the uvcb return code is stored +* @rrc: Pointer to where the uvcb return reason code is stored +* +* Return: +* 0 on success +* -ENOMEM if allocating the cache fails +* -EINVAL if gaddr is not aligned to 1MB +* -EINVAL if buff_user_len is not aligned to uv_info.conf_dump_storage_state_len +* -EINVAL if the UV call fails, rc and rrc will be set in this case +* -EFAULT if copying the result to buff_user failed +*/ +int kvm_s390_pv_dump_stor_state (struct kvm *kvm,void __user *buff_user, +u64 *gaddr,u64 buff_user_len,u16 *rc,u16 *rrc) +{ +struct uv_cb_dump_stor_state uvcb = { + .header.cmd =UVC_CMD_DUMP_CONF_STOR_STATE , + .header.len =sizeof (uvcb), + .config_handle = kvm->arch.pv.handle, + .gaddr = *gaddr, + .dump_area_origin =0 ,
arch/s390/kvm/pv.c int kvm_s390_pv_set_cpu_state(struct kvm_vcpu *vcpu, u8 state) · line 349
Quoted 10 lines
arch/s390/kvm/pv.c +10
-0+ }; +size_t buff_kvm_size; +size_t size_done =0 ; +u8 *buff_kvm =NULL ; +int cc, ret; + + ret = -EINVAL ; +/* UV call processes 1MB guest storage chunks at a time */ +if (*gaddr & ~HPAGE_MASK ) + goto out;
so we only allow userspace to use starting addresses that are 1M aligned...
Quoted 8 lines
arch/s390/kvm/pv.c +8
-0+ +/* +* We provide the storage state for 1MB chunks of guest +* storage. The buffer will need to be aligned to +* conf_dump_storage_state_len so we don't end on a partial +* chunk. +*/ + if (!buff_user_len ||
arch/s390/kvm/pv.c int kvm_s390_pv_set_cpu_state(struct kvm_vcpu *vcpu, u8 state) · line 367
Quoted 10 lines
arch/s390/kvm/pv.c +10
-0+ buff_user_len & (uv_info.conf_dump_storage_state_len - 1)) + goto out; + +/* +* Allocate a buffer from which we will later copy to the user process. +* +* We don't want userspace to dictate our buffer size so we limit it to DUMP_BUFF_LEN. +*/ + ret = -ENOMEM ; + buff_kvm_size = buff_user_len <=DUMP_BUFF_LEN ? buff_user_len :DUMP_BUFF_LEN ;
... and we allow the length to be less than 1M
meaning that userspace can request a small chunk, but won't be able to continue, it will have to request the chunk again with a larger buffer size if it wants to do any forward progess, is this correct?
Quoted 11 lines
arch/s390/kvm/pv.c +11
-0+ buff_kvm =vzalloc (buff_kvm_size); +if (!buff_kvm) + goto out; + + ret =0 ; + uvcb.dump_area_origin = (u64 )buff_kvm; +/* We will loop until the user buffer is filled or an error occurs */ + do { + /* Get a page of data */ + cc = uv_call_sched(0, (u64)&uvcb); +
arch/s390/kvm/pv.c int kvm_s390_pv_set_cpu_state(struct kvm_vcpu *vcpu, u8 state) · line 388
Quoted 10 lines
arch/s390/kvm/pv.c +10
-0+/* All or nothing */ +if (cc) { + ret = -EINVAL ; +break ; + } + + size_done += uv_info.conf_dump_storage_state_len; + uvcb.dump_area_origin += uv_info.conf_dump_storage_state_len; + uvcb.gaddr +=HPAGE_SIZE ; + buff_user_len -=PAGE_SIZE ;
shouldn't this be HPAGE_SIZE ^ ?
what happens if buff_user_len is > 1M but not 1M aligned?
arch/s390/kvm/pv.c int kvm_s390_pv_set_cpu_state(struct kvm_vcpu *vcpu, u8 state) · line 392
Quoted 4 lines
arch/s390/kvm/pv.c +4
-0+ +/* KVM Buffer full, time to copy to the process */ +if (!buff_user_len || + uvcb.dump_area_origin == (uintptr_t )buff_kvm + buff_kvm_size) {
can't you use size_done here ? ^
Quoted 61 lines
arch/s390/kvm/pv.c +28
-0+ +if (copy_to_user (buff_user, buff_kvm, + uvcb.dump_area_origin - (uintptr_t )buff_kvm)) { + ret = -EFAULT ; +break ; + } + + buff_user += size_done; + size_done =0 ; + uvcb.dump_area_origin = (u64 )buff_kvm; + } + }while (buff_user_len); + +/* Report back where we ended dumping */ + *gaddr = uvcb.gaddr; + +/* Lets only log errors, we don't want to spam */ +out: +if (ret) +KVM_UV_EVENT (kvm,3 , +"PROTVIRT DUMP STORAGE STATE: addr %llx ret %d, uvcb rc %x rrc %x" , + uvcb.gaddr, ret, uvcb.header.rc, uvcb.header.rrc); + *rc = uvcb.header.rc; + *rrc = uvcb.header.rrc; +vfree (buff_kvm); + +return ret; +}include/uapi/linux/kvm.h +15
-0@@ -1643,6 +1643,20 @@ struct kvm_s390_pv_unp {__u64 tweak; }; +enum pv_cmd_dmp_id { +KVM_PV_DUMP_INIT , +KVM_PV_DUMP_CONFIG_STOR_STATE , +KVM_PV_DUMP_COMPLETE , +}; + +struct kvm_s390_pv_dmp { +__u64 subcmd; +__u64 buff_addr; +__u64 buff_len; +__u64 gaddr;/* For dump storage state */ +__u64 reserved[4 ]; +}; +enum pv_cmd_info_id {KVM_PV_INFO_VM ,KVM_PV_INFO_DUMP ,@@ -1686,6 +1700,7 @@ enum pv_cmd_id {KVM_PV_PREP_RESET ,KVM_PV_UNSHARE_ALL ,KVM_PV_INFO , + KVM_PV_DUMP, }; struct kvm_pv_cmd {
Janosch Frank
maintainerRe: [PATCH v2 6/9] kvm: s390: Add configuration dump functionality
Quoted 45 lines · Claudio Imbrenda
Quoted 42 lines · On Thu, 10 Mar 2022 10:31:09 +0000 Janosch Frank <[email protected]> wrote:
Sometimes dumping inside of a VM fails, is unavailable or doesn't yield the required data. For these occasions we dump the VM from the outside, writing memory and cpu data to a file.
Up to now PV guests only supported dumping from the inside of the guest through dumpers like KDUMP. A PV guest can be dumped from the hypervisor but the data will be stale and / or encrypted.
To get the actual state of the PV VM we need the help of the Ultravisor who safeguards the VM state. New UV calls have been added to initialize the dump, dump storage state data, dump cpu data and complete the dump process. We expose these calls in this patch via a new UV ioctl command.
The sensitive parts of the dump data are encrypted, the dump key is derived from the Customer Communication Key (CCK). This ensures that only the owner of the VM who has the CCK can decrypt the dump data.
The memory is dumped / read via a normal export call and a re-import after the dump initialization is not needed (no re-encryption with a dump key).
Signed-off-by Janosch Frank <[email protected]>--- arch/s390/include/asm/kvm_host.h | 1 + arch/s390/kvm/kvm-s390.c | 137 +++++++++++++++++++++++++++++++ arch/s390/kvm/kvm-s390.h | 2 + arch/s390/kvm/pv.c | 115 ++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 15 ++++ 5 files changed, 270 insertions(+)arch/s390/include/asm/kvm_host.h +1
-0@@ -921,6 +921,7 @@ struct kvm_s390_pv {u64 guest_len;unsigned long stor_base;void *stor_var; +bool dumping; };
Quoted 16 lines
Quoted 6 lines
arch/s390/include/asm/kvm_host.h +0
-0struct kvm_arch {arch/s390/kvm/kvm-s390.c +0
-0arch/s390/kvm/kvm-s390.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 613
Quoted 8 lines
arch/s390/kvm/kvm-s390.c +5
-0@@ -606,6 +606,26 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_S390_PROTECTED : r =is_prot_virt_host ();break ; +case KVM_CAP_S390_PROTECTED_DUMP : { +u64 pv_cmds_dump[] = { +BIT_UVC_CMD_DUMP_INIT , +BIT_UVC_CMD_DUMP_CONFIG_STOR_STATE , +BIT_UVC_CMD_DUMP_CONFIG_STOR_STATE ,you have this twice ^
Yup, will fix
Quoted 23 lines
Quoted 23 lines
arch/s390/kvm/kvm-s390.c +15
-0+ BIT_UVC_CMD_DUMP_CPU, + }; +int i; + +if (!is_prot_virt_host ()) +return 0 ; + + r =1 ; +for (i =0 ; i <ARRAY_SIZE (pv_cmds_dump); i++) { +if (!test_bit_inv (pv_cmds_dump[i], + (unsigned long *)&uv_info.inst_calls_list)) +return 0 ; + } +break ; + }default : r =0 ; }@@ -2271,6 +2291,92 @@ static ssize_t kvm_s390_handle_pv_info(struct kvm_s390_pv_info *info) } }
Quoted 4 lines
arch/s390/kvm/kvm-s390.c static ssize_t kvm_s390_handle_pv_info(struct kvm_s390_pv_info *info) · line 2294
Quoted 1 line
arch/s390/kvm/kvm-s390.c +1
-0+static int kvm_s390_pv_dmp(struct kvm *kvm, struct kvm_pv_cmd *cmd,I am all for shortening long words, but dmp -> dump is only one extra byte
Sure, will fix
Quoted 22 lines
Quoted 7 lines
arch/s390/kvm/kvm-s390.c +6
-0+ struct kvm_s390_pv_dmp dmp) +{ +int r = -EINVAL ; +void __user *result_buff = (void __user *)dmp.buff_addr; + +switch (dmp.subcmd) {arch/s390/kvm/kvm-s390.c static ssize_t kvm_s390_handle_pv_info(struct kvm_s390_pv_info *info) · line 2310
Quoted 10 lines
arch/s390/kvm/kvm-s390.c +10
-0+case KVM_PV_DUMP_INIT : { +if (kvm->arch.pv.dumping) +break ; + + r =uv_cmd_nodata (kvm_s390_pv_get_handle (kvm), +UVC_CMD_DUMP_INIT , &cmd->rc, &cmd->rrc); +KVM_UV_EVENT (kvm,3 ,"PROTVIRT DUMP INIT: rc %x rrc %x" , + cmd->rc, cmd->rrc); +if (!r) + kvm->arch.pv.dumping = true;so at this point no guest CPUs should be running, right? below you add a check to prevent guest CPUs from being dispatched if we are in dump mode.
Quoted 3 lines
are there any checks to make sure all guest CPUs are stopped when issuing this command? could a guest CPU be running while this command is issued?
Seems like I have made changes to this but didn't send them yet. There'll be a vcpu block before the init uv call.
The unblock will be done after the dump completion.
Quoted 13 lines
arch/s390/kvm/kvm-s390.c static ssize_t kvm_s390_handle_pv_info(struct kvm_s390_pv_info *info) · line 2320
Quoted 11 lines
arch/s390/kvm/kvm-s390.c +10
-0+else + r = -EINVAL ; +break ; + } +case KVM_PV_DUMP_CONFIG_STOR_STATE : { +if (!kvm->arch.pv.dumping) +break ; + +/* +* GADDR is an output parameter since we might stop why uppercase?
Will fix
[...]
Quoted 35 lines
Quoted 35 lines
arch/s390/kvm/kvm-s390.c +35
-0+/* +* kvm_s390_pv_dump_stor_state +* +* @kvm: pointer to the guest's KVM struct +* @buff_user: Userspace pointer where we will write the results to +* @gaddr: Starting absolute guest address for which the storage state +* is requested. This value will be updated with the last +* address for which data was written when returning to +* userspace. +* @buff_user_len: Length of the buff_user buffer +* @rc: Pointer to where the uvcb return code is stored +* @rrc: Pointer to where the uvcb return reason code is stored +* +* Return: +* 0 on success +* -ENOMEM if allocating the cache fails +* -EINVAL if gaddr is not aligned to 1MB +* -EINVAL if buff_user_len is not aligned to uv_info.conf_dump_storage_state_len +* -EINVAL if the UV call fails, rc and rrc will be set in this case +* -EFAULT if copying the result to buff_user failed +*/ +int kvm_s390_pv_dump_stor_state (struct kvm *kvm,void __user *buff_user, +u64 *gaddr,u64 buff_user_len,u16 *rc,u16 *rrc) +{ +struct uv_cb_dump_stor_state uvcb = { + .header.cmd =UVC_CMD_DUMP_CONF_STOR_STATE , + .header.len =sizeof (uvcb), + .config_handle = kvm->arch.pv.handle, + .gaddr = *gaddr, + .dump_area_origin =0 , + }; +size_t buff_kvm_size; +size_t size_done =0 ; +u8 *buff_kvm =NULL ; +int cc, ret;
Quoted 8 lines
arch/s390/kvm/kvm-s390.c static ssize_t kvm_s390_handle_pv_info(struct kvm_s390_pv_info *info) · line 2360
Quoted 5 lines
arch/s390/kvm/kvm-s390.c +5
-0+ + ret = -EINVAL ; +/* UV call processes 1MB guest storage chunks at a time */ +if (*gaddr & ~HPAGE_MASK ) + goto out;so we only allow userspace to use starting addresses that are 1M aligned...
We're speaking about G2A addresses here...
Quoted 22 lines
Quoted 9 lines
arch/s390/kvm/kvm-s390.c +8
-0+ +/* +* We provide the storage state for 1MB chunks of guest +* storage. The buffer will need to be aligned to +* conf_dump_storage_state_len so we don't end on a partial +* chunk. +*/ + if (!buff_user_len ||arch/s390/kvm/kvm-s390.c static ssize_t kvm_s390_handle_pv_info(struct kvm_s390_pv_info *info) · line 2378
Quoted 10 lines
arch/s390/kvm/kvm-s390.c +10
-0+ buff_user_len & (uv_info.conf_dump_storage_state_len - 1)) + goto out; + +/* +* Allocate a buffer from which we will later copy to the user process. +* +* We don't want userspace to dictate our buffer size so we limit it to DUMP_BUFF_LEN. +*/ + ret = -ENOMEM ; + buff_kvm_size = buff_user_len <=DUMP_BUFF_LEN ? buff_user_len :DUMP_BUFF_LEN ;... and we allow the length to be less than 1M
Quoted 3 lines
meaning that userspace can request a small chunk, but won't be able to continue, it will have to request the chunk again with a larger buffer size if it wants to do any forward progess, is this correct?
...while buff_user and buff_user_len are QEMU G1V addresses.
1 page in the user buffer currently equals a megabyte of G2 storage state.
That's the UV API. You request the storage state (i.e. storage metadata needed for the decryption) for 1MB worth of guest 2 storage. The UV will write the metadata to a kernel buffer so we're basically only proxying.
Quoted 25 lines
Quoted 12 lines
arch/s390/kvm/kvm-s390.c +11
-0+ buff_kvm =vzalloc (buff_kvm_size); +if (!buff_kvm) + goto out; + + ret =0 ; + uvcb.dump_area_origin = (u64 )buff_kvm; +/* We will loop until the user buffer is filled or an error occurs */ + do { + /* Get a page of data */ + cc = uv_call_sched(0, (u64)&uvcb); +arch/s390/kvm/kvm-s390.c static ssize_t kvm_s390_handle_pv_info(struct kvm_s390_pv_info *info)
Quoted 10 lines
arch/s390/kvm/kvm-s390.c +10
-0+/* All or nothing */ +if (cc) { + ret = -EINVAL ; +break ; + } + + size_done += uv_info.conf_dump_storage_state_len; + uvcb.dump_area_origin += uv_info.conf_dump_storage_state_len; + uvcb.gaddr +=HPAGE_SIZE ; + buff_user_len -=PAGE_SIZE ;shouldn't this be
HPAGE_SIZE^ ?
Quoted 1 line
what happens if
buff_user_lenis > 1M but not 1M aligned?
See above
Quoted 70 lines
arch/s390/kvm/kvm-s390.c
Quoted 5 lines
arch/s390/kvm/kvm-s390.c +4
-0+ +/* KVM Buffer full, time to copy to the process */ +if (!buff_user_len || + uvcb.dump_area_origin == (uintptr_t )buff_kvm + buff_kvm_size) {can't you use
size_donehere ? ^Quoted 61 lines
arch/s390/kvm/kvm-s390.c +28
-0+ +if (copy_to_user (buff_user, buff_kvm, + uvcb.dump_area_origin - (uintptr_t )buff_kvm)) { + ret = -EFAULT ; +break ; + } + + buff_user += size_done; + size_done =0 ; + uvcb.dump_area_origin = (u64 )buff_kvm; + } + }while (buff_user_len); + +/* Report back where we ended dumping */ + *gaddr = uvcb.gaddr; + +/* Lets only log errors, we don't want to spam */ +out: +if (ret) +KVM_UV_EVENT (kvm,3 , +"PROTVIRT DUMP STORAGE STATE: addr %llx ret %d, uvcb rc %x rrc %x" , + uvcb.gaddr, ret, uvcb.header.rc, uvcb.header.rrc); + *rc = uvcb.header.rc; + *rrc = uvcb.header.rrc; +vfree (buff_kvm); + +return ret; +}include/uapi/linux/kvm.h +15
-0@@ -1643,6 +1643,20 @@ struct kvm_s390_pv_unp {__u64 tweak; }; +enum pv_cmd_dmp_id { +KVM_PV_DUMP_INIT , +KVM_PV_DUMP_CONFIG_STOR_STATE , +KVM_PV_DUMP_COMPLETE , +}; + +struct kvm_s390_pv_dmp { +__u64 subcmd; +__u64 buff_addr; +__u64 buff_len; +__u64 gaddr;/* For dump storage state */ +__u64 reserved[4 ]; +}; +enum pv_cmd_info_id {KVM_PV_INFO_VM ,KVM_PV_INFO_DUMP ,@@ -1686,6 +1700,7 @@ enum pv_cmd_id {KVM_PV_PREP_RESET ,KVM_PV_UNSHARE_ALL ,KVM_PV_INFO , + KVM_PV_DUMP, }; struct kvm_pv_cmd {
Janosch Frank
maintainerPATCH v2 7/9 kvm: s390: Add CPU dump functionality
In reply to Janosch FrankThe previous patch introduced the per-VM dump functions now let's focus on dumping the VCPU state via the newly introduced KVM_S390_PV_CPU_COMMAND ioctl which mirrors the VM UV ioctl and can be extended with new commands later.
--- arch/s390/kvm/kvm-s390.c | 73 ++++++++++++++++++++++++++++++++++++++++ arch/s390/kvm/kvm-s390.h | 1 + arch/s390/kvm/pv.c | 16 +++++++++ include/uapi/linux/kvm.h | 5 +++ 4 files changed, 95 insertions(+)
arch/s390/kvm/kvm-s390.c +73 -0
@@ -5118,6 +5118,52 @@ long kvm_arch_vcpu_async_ioctl(struct file *filp,return -ENOIOCTLCMD ; } +static int kvm_s390_handle_pv_vcpu_dump (struct kvm_vcpu *vcpu, +struct kvm_pv_cmd *cmd) +{ +struct kvm_s390_pv_dmp dmp; +void *data; +int ret; + +/* Dump initialization is a prerequisite */ +if (!vcpu->kvm->arch.pv.dumping) +return -EINVAL ; + +if (copy_from_user (&dmp, (__u8 __user *)cmd->data,sizeof (dmp))) +return -EFAULT ; + +/* We only handle this subcmd right now */ +if (dmp.subcmd !=KVM_PV_DUMP_CPU ) +return -EINVAL ; + +/* CPU dump length is the same as create cpu storage donation. */ +if (dmp.buff_len != uv_info.guest_cpu_stor_len) +return -EINVAL ; + + data =vzalloc (uv_info.guest_cpu_stor_len); +if (!data) +return -ENOMEM ; + + ret =kvm_s390_pv_dump_cpu (vcpu, data, &cmd->rc, &cmd->rrc); + +VCPU_EVENT (vcpu,3 ,"PROTVIRT DUMP CPU %d rc %x rrc %x" , + vcpu->vcpu_id, cmd->rc, cmd->rrc); + +if (ret) { +vfree (data); +return -EINVAL ; + } + +/* On success copy over the dump data */ +if (copy_to_user ((__u8 __user *)dmp.buff_addr, data, uv_info.guest_cpu_stor_len)) { +vfree (data); +return -EFAULT ; + } + +vfree (data); +return 0 ; +} +long kvm_arch_vcpu_ioctl (struct file *filp,unsigned int ioctl,unsigned long arg) {
@@ -5282,6 +5328,33 @@ long kvm_arch_vcpu_ioctl(struct file *filp, irq_state.len);break ; } +case KVM_S390_PV_CPU_COMMAND : { +struct kvm_pv_cmd cmd; + + r = -EINVAL ; +if (!is_prot_virt_host ()) +break ; + + r = -EFAULT ; +if (copy_from_user (&cmd, argp,sizeof (cmd))) +break ; + + r = -EINVAL ; +if (cmd.flags) +break ; + +/* We only handle this cmd right now */ +if (cmd.cmd !=KVM_PV_DUMP ) +break ; + + r =kvm_s390_handle_pv_vcpu_dump (vcpu, &cmd); + +/* Always copy over UV rc / rrc data */ +if (copy_to_user ((__u8 __user *)argp, &cmd.rc, +sizeof (cmd.rc) +sizeof (cmd.rrc))) + r = -EFAULT ; +break ; + }default : r = -ENOTTY ; }
arch/s390/kvm/kvm-s390.h +1 -0
@@ -250,6 +250,7 @@ int kvm_s390_pv_set_sec_parms(struct kvm *kvm, void *hdr, u64 length, u16 *rc,int kvm_s390_pv_unpack (struct kvm *kvm,unsigned long addr,unsigned long size,unsigned long tweak,u16 *rc,u16 *rrc);int kvm_s390_pv_set_cpu_state (struct kvm_vcpu *vcpu,u8 state); +int kvm_s390_pv_dump_cpu (struct kvm_vcpu *vcpu,void *buff,u16 *rc,u16 *rrc);int kvm_s390_pv_dump_stor_state (struct kvm *kvm,void __user *buff_user,u64 *gaddr,u64 buff_user_len,u16 *rc,u16 *rrc);
arch/s390/kvm/pv.c +16 -0
@@ -304,6 +304,22 @@ int kvm_s390_pv_set_cpu_state(struct kvm_vcpu *vcpu, u8 state)return 0 ; } +int kvm_s390_pv_dump_cpu (struct kvm_vcpu *vcpu,void *buff,u16 *rc,u16 *rrc) +{ +struct uv_cb_dump_cpu uvcb = { + .header.cmd =UVC_CMD_DUMP_CPU , + .header.len =sizeof (uvcb), + .cpu_handle = vcpu->arch.pv.handle, + .dump_area_origin = (u64 )buff, + }; +int cc; + + cc =uv_call_sched (0 , (u64)&uvcb); + *rc = uvcb.header.rc; + *rrc = uvcb.header.rrc; +return cc; +} +/* Size of the cache for the storage state dump data. 1MB for now */ #define DUMP_BUFF_LEN HPAGE_SIZE
include/uapi/linux/kvm.h +5 -0
@@ -1142,6 +1142,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_SYS_ATTRIBUTES 209#define KVM_CAP_PPC_AIL_MODE_3 210#define KVM_CAP_S390_MEM_OP_EXTENSION 211 +#define KVM_CAP_S390_PROTECTED_DUMP 212#ifdef KVM_CAP_IRQ_ROUTING
@@ -1647,6 +1648,7 @@ enum pv_cmd_dmp_id {KVM_PV_DUMP_INIT ,KVM_PV_DUMP_CONFIG_STOR_STATE ,KVM_PV_DUMP_COMPLETE , + KVM_PV_DUMP_CPU, }; struct kvm_s390_pv_dmp {
@@ -2106,4 +2108,7 @@ struct kvm_stats_desc {/* Available with KVM_CAP_XSAVE2 */ #define KVM_GET_XSAVE2 _IOR(KVMIO, 0xcf, struct kvm_xsave) +/* Available with KVM_CAP_S390_PROTECTED_DUMP */ +#define KVM_S390_PV_CPU_COMMAND _IOWR(KVMIO, 0xd0, struct kvm_pv_cmd) +#endif /* __LINUX_KVM_H */
Signature · 1 line
2.32.0
Janosch Frank
maintainerPATCH v2 8/9 Documentation: virt: Protected virtual machine dumps
In reply to Janosch FrankLet's add a documentation file which describes the dump process. Since we only copy the UV dump data from the UV to userspace we'll not go into detail here and let the party which processes the data describe its structure.
--- Documentation/virt/kvm/index.rst | 1 + Documentation/virt/kvm/s390-pv-dump.rst | 60 +++++++++++++++++++++++++ 2 files changed, 61 insertions(+) create mode 100644 Documentation/virt/kvm/s390-pv-dump.rst
Documentation/virt/kvm/index.rst +1 -0
@@ -20,6 +20,7 @@ KVM s390-diag s390-pv s390-pv-boot + s390-pv-dump timekeeping vcpu-requests
Documentation/virt/kvm/s390-pv-dump.rst new file +60 -0
60 lines not on this page · raw message
Signature · 1 line
2.32.0
Janosch Frank
maintainerPATCH v2 9/9 Documentation/virt/kvm/api.rst: Add protvirt dump/info api descriptions
In reply to Janosch FrankTime to add the dump API changes to the api documentation file. Also some minor cleanup.
--- Documentation/virt/kvm/api.rst | 150 ++++++++++++++++++++++++++++++++- 1 file changed, 148 insertions(+), 2 deletions(-)
Documentation/virt/kvm/api.rst +148 -2
@@ -5062,7 +5062,7 @@ into ESA mode. This reset is a superset of the initial reset. __u32 reserved[3]; }; -cmd values: +**cmd values:** KVM_PV_ENABLE Allocate memory and register the VM with the Ultravisor, thereby
@@ -5078,7 +5078,6 @@ KVM_PV_ENABLE ===== ============================= KVM_PV_DISABLE - Deregister the VM from the Ultravisor and reclaim the memory that had been donated to the Ultravisor, making it usable by the kernel again. All registered VCPUs are converted back to non-protected
@@ -5095,6 +5094,114 @@ KVM_PV_VM_VERIFY Verify the integrity of the unpacked image. Only if this succeeds, KVM is allowed to start protected VCPUs. +KVM_PV_INFO + :Capability: KVM_CAP_S390_PROTECTED_DUMP + + Presents an API that provides Ultravisor related data to userspace + via subcommands. len_max is the size of the user space buffer, + len_written is KVM's indication of how much bytes of that buffer + were actually written to. len_written can be used to determine the + valid fields if more response fields are added in the future. + + :: + enum pv_cmd_info_id { + KVM_PV_INFO_VM, + KVM_PV_INFO_DUMP, + }; + + struct kvm_s390_pv_info_header { + __u32 id; + __u32 len_max; + __u32 len_written; + __u32 reserved; + }; + + struct kvm_s390_pv_info { + struct kvm_s390_pv_info_header header; + struct kvm_s390_pv_info_dump dump; + struct kvm_s390_pv_info_vm vm; + }; + +**subcommands:** + + KVM_PV_INFO_VM + This subcommand provides basic Ultravisor information for PV + hosts. These values are likely also exported as files in the sysfs + firmware UV query interface but they are more easily available to + programs in this API. + + The installed calls and feature_indication members provide the + installed UV calls and the UV's other feature indications. + + The max_* members provide information about the maximum number of PV + vcpus, PV guests and PV guest memory size. + + :: + + struct kvm_s390_pv_info_vm { + __u64 inst_calls_list[4]; + __u64 max_cpus; + __u64 max_guests; + __u64 max_guest_addr; + __u64 feature_indication; + }; + + + KVM_PV_INFO_DUMP + This subcommand provides information related to dumping PV guests. + + :: + + struct kvm_s390_pv_info_dump { + __u64 dump_cpu_buffer_len; + __u64 dump_config_mem_buffer_per_1m; + __u64 dump_config_finalize_len; + }; + +KVM_PV_DUMP + :Capability: KVM_CAP_S390_PROTECTED_DUMP + + Presents an API that provides calls which facilitate dumping a + protected VM. + + :: + + struct kvm_s390_pv_dmp { + __u64 subcmd; + __u64 buff_addr; + __u64 buff_len; + __u64 gaddr; /* For dump storage state */ + }; + + **subcommands:** + + KVM_PV_DUMP_INIT + Initializes the dump process of a protected VM. If this call does + not succeed all other subcommands will fail with -EINVAL. This + subcommand will return -EINVAL if a dump process has not yet been + completed. + + Not all PV vms can be dumped, the owner needs to set `dump + allowed` PCF bit 34 in the SE header to allow dumping. + + KVM_PV_DUMP_CONFIG_STOR_STATE + Stores `buff_len` bytes of tweak component values starting with + the 1MB block specified by the absolute guest address + (`gaddr`). `buff_len` needs to be `conf_dump_storage_state_len` + aligned and at least >= the `conf_dump_storage_state_len` value + provided by the dump uv_info data. + + KVM_PV_DUMP_COMPLETE + If the subcommand succeeds it completes the dump process and lets + KVM_PV_DUMP_INIT be called again. + + On success `conf_dump_finalize_len` bytes of completion data will be + stored to the `buff_addr`. The completion data contains a key + derivation seed, IV, tweak nonce and encryption keys as well as an + authentication tag all of which are needed to decrypt the dump at a + later time. + + 4.126 KVM_X86_SET_MSR_FILTER ----------------------------
@@ -5643,6 +5750,32 @@ The offsets of the state save areas in struct kvm_xsave follow the contents of CPUID leaf 0xD on the host. +4.135 KVM_S390_PV_CPU_COMMAND +----------------------------- + +:Capability: KVM_CAP_S390_PROTECTED_DUMP +:Architectures: s390 +:Type: vcpu ioctl +:Parameters: none +:Returns: 0 on success, < 0 on error + +This ioctl closely mirrors `KVM_S390_PV_COMMAND` but handles requests +for vcpus. It re-uses the kvm_s390_pv_dmp struct and hence also shares +the command ids. + +**command:** + +KVM_PV_DUMP + Presents an API that provides calls which facilitate dumping a vcpu + of a protected VM. + +**subcommand:** + +KVM_PV_DUMP_CPU + Provides encrypted dump data like register values. + The length of the returned data is provided by uv_info.guest_cpu_stor_len. + + 5. The kvm_run structure ========================
@@ -7643,3 +7776,16 @@ The argument to KVM_ENABLE_CAP is also a bitmask, and must be a subset of the result of KVM_CHECK_EXTENSION. KVM will forward to userspace the hypercalls whose corresponding bit is in the argument, and return ENOSYS for the others. + +8.35 KVM_CAP_S390_PROTECTED_DUMP +-------------------------------- + +:Capability: KVM_CAP_S390_PROTECTED_DUMP +:Architectures: s390 +:Type: vm + +This capability indicates that KVM and the Ultravisor support dumping +PV guests. The `KVM_PV_DUMP` command is available for the +`KVM_S390_PV_COMMAND` ioctl and the `KVM_PV_INFO` command provides +dump related UV data. Also the vcpu ioctl `KVM_S390_PV_CPU_COMMAND` is +available and supports the `KVM_PV_DUMP_CPU` subcommand.
Signature · 1 line
2.32.0
Janosch Frank
maintainerQuoted 4 lines · Janosch Frank
Sometimes dumping inside of a VM fails, is unavailable or doesn't yield the required data. For these occasions we dump the VM from the outside, writing memory and cpu data to a file.
Quoted 8 lines
Up to now PV guests only supported dumping from the inside of the guest through dumpers like KDUMP. A PV guest can be dumped from the hypervisor but the data will be stale and / or encrypted.
To get the actual state of the PV VM we need the help of the Ultravisor who safeguards the VM state. New UV calls have been added to initialize the dump, dump storage state data, dump cpu data and complete the dump process.
Ping