SEV-SNP: Add support for SNP certificate fetching
v2 cover SEV-SNP: Add support for SNP certificate fetching
This patchset is also available at:
https://github.com/amdese/qemu/commits/snp-certs-rfc2
and is based on top of qemu master (7433709a1477)
Overview
--------
The GHCB 2.0 specification defines 2 GHCB request types to allow SNP guests to send encrypted messages/requests to firmware: SNP Guest Requests and SNP Extended Guest Requests. These encrypted messages are used for things like servicing attestation requests issued by the guest. Implementing support for these is required to be fully GHCB-compliant.
For the most part, KVM only needs to handle forwarding these requests to firmware (to be issued via the SNP_GUEST_REQUEST firmware command defined in the SEV-SNP Firmware ABI), and then forwarding the encrypted response to the guest.
However, in the case of SNP Extended Guest Requests, the host is also able to provide the certificate data corresponding to the endorsement key used by firmware to sign attestation report requests. This certificate data is provided by userspace because:
1) It allows for different keys/key types to be used for each particular guest with requiring any sort of KVM API to configure the certificate table in advance on a per-guest basis.
2) It provides additional flexibility with how attestation requests might be handled during live migration where the certificate data for source/dest might be different.
3) It allows all synchronization between certificates and firmware/signing key updates to be handled purely by userspace rather than requiring some in-kernel mechanism to facilitate it. [1]
To support fetching certificate data from userspace, a new KVM KVM_EXIT_SNP_REQ_CERTS exit type is used to fetch the data similarly to KVM_EXIT_MMIO/etc, with an associate KVM capability to detect/enable the exits depending on whether userspace has been configured to provide certificate data.
Add support for this exit in QEMU. Additionally, because some of the locking behavior relies on kvm_immediate_exit, add an some infrastructure to allow for callbacks to be registered and executed upon a kvm_immediate_exit-induced exit from KVM.
See the Documentation in the related KVM patches for more details on the expected usage and kvm_immediate_exit handling:
https://lore.kernel.org/kvm/[email protected]/#t
[1] https://lore.kernel.org/kvm/[email protected]/
Testing
-------
For testing this, use the following host kernel:
https://github.com/amdese/linux/commits/snp-certs-v3
For the guest, any 5.19 or newer kernel will do.
A basic command-line invocation for an SNP guest with certificate data supplied would be:
qemu-system-x86_64 -smp 32,maxcpus=255 -cpu EPYC-Milan-v2
-machine q35,confidential-guest-support=sev0,memory-backend=ram1
-object memory-backend-memfd,id=ram1,size=4G,share=true,reserve=false
-object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,id-auth=,certs-path=/home/mroth/cert.blob -bios OVMF.fd
Something like the following simple example can be used to simulate an exclusive lock being held on the certificate by management tools performing an update:
#include <stdlib.h>
#include <stdio.h>
#define __USE_GNU
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <stdbool.h>
#include <sys/types.h>
#include <sys/stat.h>
int main(int argc, void **argv)
{
int ret, fd, i = 0;
char *path = argv[1]; struct flock fl = {
.l_whence = SEEK_SET,
.l_start = 0,
.l_len = 0,
.l_type = F_WRLCK
};
fd = open(path, O_RDWR);
ret = fcntl(fd, F_OFD_SETLK, &fl);
if (ret) {
printf("error locking file, ret %d errno %d\n", ret, errno);
return ret;
}
while (true) {
i++;
printf("now holding lock (%d seconds elapsed)...\n", i);
usleep(1000 * 1000);
}
return 0;}
For requesting attestation with certificate data attached, you can use something like the following tool/workflow:
https://github.com/virtee/snpguest?tab=readme-ov-file#extended-attestation-workflow
The format of the certificate blob is defined in the GHCB 2.0 specification, but if it's not being parsed on the guest-side then random data will suffice for testing the KVM bits.
Any feedback/review is appreciated.
Thanks!
-Melody
Changes from v1 to v2:
- Recreation race problem: Add a while(1) loop using the libvirt example with fstat and stat for the locking certificate blob code in open_certs_locked() to fix the recreation race problem suggested by Daniel P. Berrangé.
- Fix the json format and filename suggested by Markus Armbruster.
- Some text formulation comments.
Michael Roth (3):
linux-headers: Update for 6.12 and SNP certificate support accel/kvm: Add kvm_immediate_exit callback infrastructure i386/sev: Add KVM_EXIT_SNP_REQ_CERTS support for certificate-fetching
accel/kvm/kvm-all.c | 43 ++++++ include/system/kvm.h | 3 + linux-headers/linux/kvm.h | 10 ++ qapi/qom.json | 21 ++- target/i386/kvm/kvm.c | 10 ++ target/i386/sev-system-stub.c | 5 + target/i386/sev.c | 277 ++++++++++++++++++++++++++++++++++ target/i386/sev.h | 2 +- 8 files changed, 369 insertions(+), 2 deletions(-)
Signature · 1 line
2.34.1
RFC PATCH v2 1/3 linux-headers: Update for 6.12 and SNP certificate support
From: Michael Roth <[email protected]>
Add defines for SNP certificate support.
--- linux-headers/linux/kvm.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
linux-headers/linux/kvm.h +10 -0
@@ -135,6 +135,12 @@ struct kvm_xen_exit { } u; }; +struct kvm_exit_snp_req_certs { +__u64 gfn; +__u32 npages; +__u32 ret; +}; +#define KVM_S390_GET_SKEYS_NONE 1#define KVM_S390_SKEYS_MAX 1048576
@@ -178,6 +184,7 @@ struct kvm_xen_exit {#define KVM_EXIT_NOTIFY 37#define KVM_EXIT_LOONGARCH_IOCSR 38#define KVM_EXIT_MEMORY_FAULT 39 +#define KVM_EXIT_SNP_REQ_CERTS 40/* For KVM_EXIT_INTERNAL_ERROR */ /* Emulate instruction failed. */
@@ -438,6 +445,8 @@ struct kvm_run {__u64 gpa;__u64 size; } memory_fault; +/* KVM_EXIT_SNP_REQ_CERTS */ +struct kvm_exit_snp_req_certs snp_req_certs;/* Fix the size of the union. */ char padding[256 ]; };
@@ -925,6 +934,7 @@ struct kvm_enable_cap {#define KVM_CAP_PRE_FAULT_MEMORY 236#define KVM_CAP_X86_APIC_BUS_CYCLES_NS 237#define KVM_CAP_X86_GUEST_MODE 238 +#define KVM_CAP_EXIT_SNP_REQ_CERTS 239struct kvm_irq_routing_irqchip {__u32 irqchip;
Signature · 1 line
2.34.1
RFC PATCH v2 2/3 accel/kvm: Add kvm_immediate_exit callback infrastructure
In reply to Melody WangFrom: Michael Roth <[email protected]>
Provide a generic mechanism so that users of struct kvm_run's 'immediate_exit' flag can register a callback that can be issued after returning from KVM.
--- accel/kvm/kvm-all.c | 43 +++++++++++++++++++++++++++++++++++++++++++ include/system/kvm.h | 3 +++ 2 files changed, 46 insertions(+)
accel/kvm/kvm-all.c +43 -0
@@ -3091,6 +3091,46 @@ out_unref:return ret; } +typedef struct ImmediateExitEntry ImmediateExitEntry ; + +struct ImmediateExitEntry { +ImmediateExitCb *cb; +void *opaque; +QTAILQ_ENTRY (ImmediateExitEntry) next; +}; + +static QTAILQ_HEAD(, ImmediateExitEntry) immediate_exit_list = \ +QTAILQ_HEAD_INITIALIZER (immediate_exit_list); + +void add_immediate_exit_callback (CPUState *cpu,ImmediateExitCb *cb,void *opaque) +{ +ImmediateExitEntry *entry =g_new (ImmediateExitEntry,1 ); + entry->cb = cb; + entry->opaque = opaque; + +bql_lock (); +QTAILQ_INSERT_TAIL (&immediate_exit_list, entry, next); +bql_unlock (); + +kvm_cpu_kick (cpu); +} + +static void process_immediate_exit_callbacks (void ) +{ +ImmediateExitEntry *entry, *tmp; + +bql_lock (); + +/* Handle any pending immediate-exit callbacks */ + QTAILQ_FOREACH_SAFE(entry, &immediate_exit_list, next, tmp) { + entry->cb (entry->opaque); +QTAILQ_REMOVE (&immediate_exit_list, entry, next); +g_free (entry); + } + +bql_unlock (); +} +int kvm_cpu_exec (CPUState *cpu) {struct kvm_run *run = cpu->kvm_run;
@@ -3160,6 +3200,9 @@ int kvm_cpu_exec(CPUState *cpu)if (run_ret == -EINTR || run_ret == -EAGAIN ) {trace_kvm_io_window_exit ();kvm_eat_signals (cpu); +if (kvm_immediate_exit) { +process_immediate_exit_callbacks (); + } ret =EXCP_INTERRUPT ;break ; }
include/system/kvm.h +3 -0
@@ -586,4 +586,7 @@ int kvm_set_memory_attributes_shared(hwaddr start, uint64_t size);int kvm_convert_memory (hwaddr start,hwaddr size,bool to_private); +typedef void ImmediateExitCb (void *opaque); +void add_immediate_exit_callback (CPUState *cpu,ImmediateExitCb *cb,void *opaque); +#endif
Signature · 1 line
2.34.1
RFC PATCH v2 3/3 i386/sev: Add KVM_EXIT_SNP_REQ_CERTS support for certificate-fetching
In reply to Melody WangFrom: Michael Roth <[email protected]>
The GHCB specification[1] defines a VMGEXIT-based Guest Request hypercall to allow an SNP guest to issue encrypted requests directly to SNP firmware to do things like query the attestation report for the guest. These are generally handled purely in the kernel.
In some some cases, it's useful for the host to be able to additionally supply the certificate chain for the signing key that SNP firmware uses to sign these attestation reports. To allow for this, the GHCB specification defines an Extended Guest Request where this certificate data can be provided in a special format described in the GHCB spec. This certificate data may be global or guest-specific depending on how the guest was configured. Rather than providing interfaces to manage these within the kernel, KVM provides a new KVM_EXIT_SNP_REQ_CERTS exit to request the certificate contents from userspace. Implement support for that in QEMU.
To synchronize delivery of the certificates to the guest in a way where they will not be rendered invalid by updates to SNP firmware or attestation singing/endorsement keys by management tools outside the purview of QEMU, it is expected by users of KVM_EXIT_SNP_REQ_CERTS to obtain a shared/read lock on the certificate file prior to delivering them back to KVM. Only after this will the attestation report be retrieved from firmware and bundled with the certificate data, so QEMU must continue to hold the file lock until KVM confirms that the attestation report has been retrieved/bundled. This confirmation is done by way of the kvm_immediate_exit callback infrastructure that was introduced in a previous patch.
[1] "Guest Hypervisor Communication Block (GHCB) Standardization",
https://www.amd.com/en/developer/sev.html
[Melody: Add a while(1) loop using the libvirt example with fstat and stat for the locking certificate blob code in open_certs_locked() to fix the recreation race problem suggested by Daniel P. Berrangé. And fix the json format and filename suggested by Markus Armbruster.]
--- qapi/qom.json | 21 ++- target/i386/kvm/kvm.c | 10 ++ target/i386/sev-system-stub.c | 5 + target/i386/sev.c | 277 ++++++++++++++++++++++++++++++++++ target/i386/sev.h | 2 +- 5 files changed, 313 insertions(+), 2 deletions(-)
qapi/qom.json +20 -1
@@ -1034,6 +1034,23 @@ # firmware. Set this to true to disable the use of VCEK. # (default: false) (since: 9.1) # +# @certs-filename: Certificate data that can be passed to guests via +# SNP Extended Guest Requests. File should be in the format +# described in the GHCB specification. (default: none) +# (since: 10.0) +# +# @certs-timeout: Maximum time in milliseconds to wait to obtain a read lock +# on the certificate file specified by @certs-filename. This is +# not a cumulative value and only affects how long QEMU waits +# before returning execution to the vCPU and informing the guest +# of the timeout, so the guest can still continue retrying for +# as long as it likes (which will be about 60 seconds for linux +# guests at the time of this writing). If the guest-side timeout +# is insufficient, set this higher to allow more time to fetch the +# certificate. If the guest-side timeout is sufficient, set this +# lower to reduce the likelihood of soft lockups in the guest. +# (default: 100) (since: 10.0) +# # Since: 9.1 ## { 'struct': 'SevSnpGuestProperties',
@@ -1045,7 +1062,9 @@ '*id-auth': 'str', '*author-key-enabled': 'bool', '*host-data': 'str', - '*vcek-disabled': 'bool' } } + '*vcek-disabled': 'bool', + '*certs-filename': 'str', + '*certs-timeout': 'uint32' } } ## # @ThreadContextProperties:
target/i386/kvm/kvm.c +10 -0
@@ -6122,6 +6122,16 @@ int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run)#endif case KVM_EXIT_HYPERCALL : ret =kvm_handle_hypercall (run); +break ; +case KVM_EXIT_SNP_REQ_CERTS : +if (!sev_snp_enabled ()) { +error_report ("KVM: Encountered a certificate request exit for a " +"non-SEV-SNP guest." ); + ret =-1 ; + }else { + ret =kvm_handle_snp_req_certs (cs, run); + } +break ;default :fprintf (stderr,"KVM: unknown exit reason %d\n" , run->exit_reason);
target/i386/sev-system-stub.c +5 -0
@@ -71,3 +71,8 @@ void hmp_info_sev(Monitor *mon, const QDict *qdict)void pc_system_parse_sev_metadata (uint8_t *flash_ptr,size_t flash_size) { } + +int kvm_handle_snp_req_certs (CPUState *cpu,struct kvm_run *run) +{ +g_assert_not_reached (); +}
target/i386/sev.c +277 -0
319 lines not on this page · raw message
target/i386/sev.h +1 -1
@@ -70,5 +70,5 @@ void pc_system_parse_sev_metadata(uint8_t *flash_ptr, size_t flash_size);uint32_t sev_get_cbit_position (void );uint32_t sev_get_reduced_phys_bits (void ); - +int kvm_handle_snp_req_certs (CPUState *cpu,struct kvm_run *run);#endif
Signature · 1 line
2.34.1
Re: [RFC PATCH v2 3/3] i386/sev: Add KVM_EXIT_SNP_REQ_CERTS support for certificate-fetching
Quoted 18 lines · Melody Wang
From: Michael Roth <[email protected]>
The GHCB specification[1] defines a VMGEXIT-based Guest Request hypercall to allow an SNP guest to issue encrypted requests directly to SNP firmware to do things like query the attestation report for the guest. These are generally handled purely in the kernel.
In some some cases, it's useful for the host to be able to additionally supply the certificate chain for the signing key that SNP firmware uses to sign these attestation reports. To allow for this, the GHCB specification defines an Extended Guest Request where this certificate data can be provided in a special format described in the GHCB spec. This certificate data may be global or guest-specific depending on how the guest was configured. Rather than providing interfaces to manage these within the kernel, KVM provides a new
KVM_EXIT_SNP_REQ_CERTSexit to request the certificate contents from userspace. Implement support for that in QEMU.
Quoted 11 lines
To synchronize delivery of the certificates to the guest in a way where they will not be rendered invalid by updates to SNP firmware or attestation singing/endorsement keys by management tools outside the purview of QEMU, it is expected by users of
KVM_EXIT_SNP_REQ_CERTSto obtain a shared/read lock on the certificate file prior to delivering them back to KVM. Only after this will the attestation report be retrieved from firmware and bundled with the certificate data, so QEMU must continue to hold the file lock until KVM confirms that the attestation report has been retrieved/bundled. This confirmation is done by way of thekvm_immediate_exitcallback infrastructure that was introduced in a previous patch.
In review of v1, I inquired about the locking, and Mike explained it some more. I wonder whether we should work some of that into our documentation.
Quoted 23 lines
[1] "Guest Hypervisor Communication Block (GHCB) Standardization",
https://www.amd.com/en/developer/sev.html[Melody: Add a while(1) loop using the libvirt example with fstat and stat for the locking certificate blob code in
open_certs_locked()to fix the recreation race problem suggested by Daniel P. Berrangé. And fix the json format and filename suggested by Markus Armbruster.]Signed-off-by Michael Roth <[email protected]>Signed-off-by Melody Wang <[email protected]>--- qapi/qom.json | 21 ++- target/i386/kvm/kvm.c | 10 ++ target/i386/sev-system-stub.c | 5 + target/i386/sev.c | 277 ++++++++++++++++++++++++++++++++++ target/i386/sev.h | 2 +- 5 files changed, 313 insertions(+), 2 deletions(-)qapi/qom.json +0
-0
qapi/qom.json line 1039
Quoted 6 lines
qapi/qom.json +3
-0@@ -1034,6 +1034,23 @@ # firmware. Set this to true to disable the use of VCEK. # (default: false) (since: 9.1) # +# @certs-filename: Certificate data that can be passed to guests via +# SNP Extended Guest Requests. File should be in the format +# described in the GHCB specification. (default: none)
Is this referring to the "Guest Hypervisor Communication Block (GHCB) Standardization" mentioned in the commit message?
That's 70 pages... Where exactly is the file format specified?
Quoted 4 lines
qapi/qom.json +4
-0+# (since: 10.0) +# +# @certs-timeout: Maximum time in milliseconds to wait to obtain a read lock +# on the certificate file specified by @certs-filename. This is
qapi/qom.json line 1053
Quoted 10 lines
qapi/qom.json +10
-0+# not a cumulative value and only affects how long QEMU waits +# before returning execution to the vCPU and informing the guest +# of the timeout, so the guest can still continue retrying for +# as long as it likes (which will be about 60 seconds for linux +# guests at the time of this writing). If the guest-side timeout +# is insufficient, set this higher to allow more time to fetch the +# certificate. If the guest-side timeout is sufficient, set this +# lower to reduce the likelihood of soft lockups in the guest. +# (default: 100) (since: 10.0) +#
docs/devel/qapi-code-gen.rst:
For legibility, wrap text paragraphs so every line is at most 70 characters long.
Separate sentences with two spaces.
Quoted 14 lines
qapi/qom.json +3
-1# Since: 9.1 ## { 'struct': 'SevSnpGuestProperties',@@ -1045,7 +1062,9 @@ '*id-auth': 'str', '*author-key-enabled': 'bool', '*host-data': 'str', - '*vcek-disabled': 'bool' } } + '*vcek-disabled': 'bool', + '*certs-filename': 'str', + '*certs-timeout': 'uint32' } } ## # @ThreadContextProperties: