Skip to contentSkip to conversation

SEV-SNP: Add support for SNP certificate fetching

RFC PATCH In review Melody Wang 2 participants 5 messages Jan 20, 2025 – Feb 5, 2025

lore
Share & tools
lore
#0 lore raw

v2 cover SEV-SNP: Add support for SNP certificate fetching

This patchset is also available at:

https://github.com/amdese/qemu/commits/snp-certs-rfc2

and is based on top of qemu master (7433709a1477)

Overview
--------

The GHCB 2.0 specification defines 2 GHCB request types to allow SNP guests to send encrypted messages/requests to firmware: SNP Guest Requests and SNP Extended Guest Requests. These encrypted messages are used for things like servicing attestation requests issued by the guest. Implementing support for these is required to be fully GHCB-compliant.

For the most part, KVM only needs to handle forwarding these requests to firmware (to be issued via the SNP_GUEST_REQUEST firmware command defined in the SEV-SNP Firmware ABI), and then forwarding the encrypted response to the guest.

However, in the case of SNP Extended Guest Requests, the host is also able to provide the certificate data corresponding to the endorsement key used by firmware to sign attestation report requests. This certificate data is provided by userspace because:

1) It allows for different keys/key types to be used for each particular guest with requiring any sort of KVM API to configure the certificate table in advance on a per-guest basis.

2) It provides additional flexibility with how attestation requests might be handled during live migration where the certificate data for source/dest might be different.

3) It allows all synchronization between certificates and firmware/signing key updates to be handled purely by userspace rather than requiring some in-kernel mechanism to facilitate it. [1]

To support fetching certificate data from userspace, a new KVM KVM_EXIT_SNP_REQ_CERTS exit type is used to fetch the data similarly to KVM_EXIT_MMIO/etc, with an associate KVM capability to detect/enable the exits depending on whether userspace has been configured to provide certificate data.

Add support for this exit in QEMU. Additionally, because some of the locking behavior relies on kvm_immediate_exit, add an some infrastructure to allow for callbacks to be registered and executed upon a kvm_immediate_exit-induced exit from KVM.

See the Documentation in the related KVM patches for more details on the expected usage and kvm_immediate_exit handling:

https://lore.kernel.org/kvm/[email protected]/#t

[1] https://lore.kernel.org/kvm/[email protected]/

Testing
-------

For testing this, use the following host kernel:

https://github.com/amdese/linux/commits/snp-certs-v3

For the guest, any 5.19 or newer kernel will do.

A basic command-line invocation for an SNP guest with certificate data supplied would be:

qemu-system-x86_64 -smp 32,maxcpus=255 -cpu EPYC-Milan-v2
-machine q35,confidential-guest-support=sev0,memory-backend=ram1
-object memory-backend-memfd,id=ram1,size=4G,share=true,reserve=false
-object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,id-auth=,certs-path=/home/mroth/cert.blob -bios OVMF.fd

Something like the following simple example can be used to simulate an exclusive lock being held on the certificate by management tools performing an update:

  #include <stdlib.h>
  #include <stdio.h>
  #define __USE_GNU
  #include <fcntl.h>
  #include <unistd.h>
  #include <errno.h>
  #include <stdbool.h>
  #include <sys/types.h>
  #include <sys/stat.h>
  
  int main(int argc, void **argv)
  {
      int ret, fd, i = 0;
      char *path = argv[1];
      struct flock fl = {
          .l_whence = SEEK_SET,
          .l_start = 0,
          .l_len = 0,
          .l_type = F_WRLCK
      };
  
      fd = open(path, O_RDWR);
      ret = fcntl(fd, F_OFD_SETLK, &fl);
      if (ret) {
          printf("error locking file, ret %d errno %d\n", ret, errno);
          return ret;
      }
  
      while (true) {
          i++;
          printf("now holding lock (%d seconds elapsed)...\n", i);
          usleep(1000 * 1000);
      }
  
      return 0;

}

For requesting attestation with certificate data attached, you can use something like the following tool/workflow:

https://github.com/virtee/snpguest?tab=readme-ov-file#extended-attestation-workflow

The format of the certificate blob is defined in the GHCB 2.0 specification, but if it's not being parsed on the guest-side then random data will suffice for testing the KVM bits.

Any feedback/review is appreciated.

Thanks!

-Melody

Changes from v1 to v2:

- Recreation race problem: Add a while(1) loop using the libvirt example with fstat and stat for the locking certificate blob code in open_certs_locked() to fix the recreation race problem suggested by Daniel P. Berrangé.
- Fix the json format and filename suggested by Markus Armbruster.
- Some text formulation comments.

Michael Roth (3):
linux-headers: Update for 6.12 and SNP certificate support accel/kvm: Add kvm_immediate_exit callback infrastructure i386/sev: Add KVM_EXIT_SNP_REQ_CERTS support for certificate-fetching

 accel/kvm/kvm-all.c           |  43 ++++++
 include/system/kvm.h          |   3 +
 linux-headers/linux/kvm.h     |  10 ++
 qapi/qom.json                 |  21 ++-
 target/i386/kvm/kvm.c         |  10 ++
 target/i386/sev-system-stub.c |   5 +
 target/i386/sev.c             | 277 ++++++++++++++++++++++++++++++++++
 target/i386/sev.h             |   2 +-
 8 files changed, 369 insertions(+), 2 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

RFC PATCH v2 1/3 linux-headers: Update for 6.12 and SNP certificate support

From: Michael Roth <[email protected]>

Add defines for SNP certificate support.

---
 linux-headers/linux/kvm.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
linux-headers/linux/kvm.h +10 -0
index 3bcd4eabe3..10f6c908d9 100644
@@ -135,6 +135,12 @@ struct kvm_xen_exit {
 	} u;
 };
 
+struct kvm_exit_snp_req_certs {
+	__u64 gfn;
+	__u32 npages;
+	__u32 ret;
+};
+
 #define KVM_S390_GET_SKEYS_NONE   1
 #define KVM_S390_SKEYS_MAX        1048576
 
@@ -178,6 +184,7 @@ struct kvm_xen_exit {
 #define KVM_EXIT_NOTIFY           37
 #define KVM_EXIT_LOONGARCH_IOCSR  38
 #define KVM_EXIT_MEMORY_FAULT     39
+#define KVM_EXIT_SNP_REQ_CERTS    40
 
 /* For KVM_EXIT_INTERNAL_ERROR */
 /* Emulate instruction failed. */
@@ -438,6 +445,8 @@ struct kvm_run {
 			__u64 gpa;
 			__u64 size;
 		} memory_fault;
+		/* KVM_EXIT_SNP_REQ_CERTS */
+		struct kvm_exit_snp_req_certs snp_req_certs;
 		/* Fix the size of the union. */
 		char padding[256];
 	};
@@ -925,6 +934,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_PRE_FAULT_MEMORY 236
 #define KVM_CAP_X86_APIC_BUS_CYCLES_NS 237
 #define KVM_CAP_X86_GUEST_MODE 238
+#define KVM_CAP_EXIT_SNP_REQ_CERTS 239
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
Signature · 1 line
2.34.1
#2 lore raw

RFC PATCH v2 2/3 accel/kvm: Add kvm_immediate_exit callback infrastructure

In reply to Melody Wang

From: Michael Roth <[email protected]>

Provide a generic mechanism so that users of struct kvm_run's 'immediate_exit' flag can register a callback that can be issued after returning from KVM.

---
 accel/kvm/kvm-all.c  | 43 +++++++++++++++++++++++++++++++++++++++++++
 include/system/kvm.h |  3 +++
 2 files changed, 46 insertions(+)
accel/kvm/kvm-all.c +43 -0
index c65b790433..c8516d6ba8 100644
@@ -3091,6 +3091,46 @@ out_unref:
     return ret;
 }
 
+typedef struct ImmediateExitEntry ImmediateExitEntry;
+
+struct ImmediateExitEntry {
+    ImmediateExitCb *cb;
+    void *opaque;
+    QTAILQ_ENTRY(ImmediateExitEntry) next;
+};
+
+static QTAILQ_HEAD(, ImmediateExitEntry) immediate_exit_list = \
+    QTAILQ_HEAD_INITIALIZER(immediate_exit_list);
+
+void add_immediate_exit_callback(CPUState *cpu, ImmediateExitCb *cb, void *opaque)
+{
+    ImmediateExitEntry *entry = g_new(ImmediateExitEntry, 1);
+    entry->cb = cb;
+    entry->opaque = opaque;
+
+    bql_lock();
+    QTAILQ_INSERT_TAIL(&immediate_exit_list, entry, next);
+    bql_unlock();
+
+    kvm_cpu_kick(cpu);
+}
+
+static void process_immediate_exit_callbacks(void)
+{
+    ImmediateExitEntry *entry, *tmp;
+
+    bql_lock();
+
+    /* Handle any pending immediate-exit callbacks */
+    QTAILQ_FOREACH_SAFE(entry, &immediate_exit_list, next, tmp) {
+        entry->cb(entry->opaque);
+        QTAILQ_REMOVE(&immediate_exit_list, entry, next);
+        g_free(entry);
+    }
+
+    bql_unlock();
+}
+
 int kvm_cpu_exec(CPUState *cpu)
 {
     struct kvm_run *run = cpu->kvm_run;
@@ -3160,6 +3200,9 @@ int kvm_cpu_exec(CPUState *cpu)
             if (run_ret == -EINTR || run_ret == -EAGAIN) {
                 trace_kvm_io_window_exit();
                 kvm_eat_signals(cpu);
+                if (kvm_immediate_exit) {
+                    process_immediate_exit_callbacks();
+                }
                 ret = EXCP_INTERRUPT;
                 break;
             }
include/system/kvm.h +3 -0
index ab17c09a55..fb21bdeede 100644
@@ -586,4 +586,7 @@ int kvm_set_memory_attributes_shared(hwaddr start, uint64_t size);
 
 int kvm_convert_memory(hwaddr start, hwaddr size, bool to_private);
 
+typedef void ImmediateExitCb(void *opaque);
+void add_immediate_exit_callback(CPUState *cpu, ImmediateExitCb *cb, void *opaque);
+
 #endif
Signature · 1 line
2.34.1
#3 lore raw

RFC PATCH v2 3/3 i386/sev: Add KVM_EXIT_SNP_REQ_CERTS support for certificate-fetching

In reply to Melody Wang

From: Michael Roth <[email protected]>

The GHCB specification[1] defines a VMGEXIT-based Guest Request hypercall to allow an SNP guest to issue encrypted requests directly to SNP firmware to do things like query the attestation report for the guest. These are generally handled purely in the kernel.

In some some cases, it's useful for the host to be able to additionally supply the certificate chain for the signing key that SNP firmware uses to sign these attestation reports. To allow for this, the GHCB specification defines an Extended Guest Request where this certificate data can be provided in a special format described in the GHCB spec. This certificate data may be global or guest-specific depending on how the guest was configured. Rather than providing interfaces to manage these within the kernel, KVM provides a new KVM_EXIT_SNP_REQ_CERTS exit to request the certificate contents from userspace. Implement support for that in QEMU.

To synchronize delivery of the certificates to the guest in a way where they will not be rendered invalid by updates to SNP firmware or attestation singing/endorsement keys by management tools outside the purview of QEMU, it is expected by users of KVM_EXIT_SNP_REQ_CERTS to obtain a shared/read lock on the certificate file prior to delivering them back to KVM. Only after this will the attestation report be retrieved from firmware and bundled with the certificate data, so QEMU must continue to hold the file lock until KVM confirms that the attestation report has been retrieved/bundled. This confirmation is done by way of the kvm_immediate_exit callback infrastructure that was introduced in a previous patch.

[1] "Guest Hypervisor Communication Block (GHCB) Standardization",
https://www.amd.com/en/developer/sev.html

[Melody: Add a while(1) loop using the libvirt example with fstat and stat for the locking certificate blob code in open_certs_locked() to fix the recreation race problem suggested by Daniel P. Berrangé. And fix the json format and filename suggested by Markus Armbruster.]

---
 qapi/qom.json                 |  21 ++-
 target/i386/kvm/kvm.c         |  10 ++
 target/i386/sev-system-stub.c |   5 +
 target/i386/sev.c             | 277 ++++++++++++++++++++++++++++++++++
 target/i386/sev.h             |   2 +-
 5 files changed, 313 insertions(+), 2 deletions(-)
qapi/qom.json +20 -1
index 28ce24cd8d..c6f9ad5d45 100644
@@ -1034,6 +1034,23 @@
 #     firmware.  Set this to true to disable the use of VCEK.
 #     (default: false) (since: 9.1)
 #
+# @certs-filename: Certificate data that can be passed to guests via
+#     SNP Extended Guest Requests. File should be in the format
+#     described in the GHCB specification. (default: none)
+#     (since: 10.0)
+#
+# @certs-timeout: Maximum time in milliseconds to wait to obtain a read lock
+#     on the certificate file specified by @certs-filename. This is
+#     not a cumulative value and only affects how long QEMU waits
+#     before returning execution to the vCPU and informing the guest
+#     of the timeout, so the guest can still continue retrying for
+#     as long as it likes (which will be about 60 seconds for linux
+#     guests at the time of this writing). If the guest-side timeout
+#     is insufficient, set this higher to allow more time to fetch the
+#     certificate. If the guest-side timeout is sufficient, set this
+#     lower to reduce the likelihood of soft lockups in the guest.
+#     (default: 100) (since: 10.0)
+#
 # Since: 9.1
 ##
 { 'struct': 'SevSnpGuestProperties',
@@ -1045,7 +1062,9 @@
             '*id-auth': 'str',
             '*author-key-enabled': 'bool',
             '*host-data': 'str',
-            '*vcek-disabled': 'bool' } }
+            '*vcek-disabled': 'bool',
+            '*certs-filename': 'str',
+            '*certs-timeout': 'uint32' } }
 
 ##
 # @ThreadContextProperties:
target/i386/kvm/kvm.c +10 -0
index 2f66e63b88..5df272d3d9 100644
@@ -6122,6 +6122,16 @@ int kvm_arch_handle_exit(CPUState *cs, struct kvm_run *run)
 #endif
     case KVM_EXIT_HYPERCALL:
         ret = kvm_handle_hypercall(run);
+        break;
+    case KVM_EXIT_SNP_REQ_CERTS:
+        if (!sev_snp_enabled()) {
+            error_report("KVM: Encountered a certificate request exit for a "
+                         "non-SEV-SNP guest.");
+            ret = -1;
+        } else {
+            ret = kvm_handle_snp_req_certs(cs, run);
+        }
+
         break;
     default:
         fprintf(stderr, "KVM: unknown exit reason %d\n", run->exit_reason);
target/i386/sev-system-stub.c +5 -0
index d5bf886e79..685c56f62c 100644
@@ -71,3 +71,8 @@ void hmp_info_sev(Monitor *mon, const QDict *qdict)
 void pc_system_parse_sev_metadata(uint8_t *flash_ptr, size_t flash_size)
 {
 }
+
+int kvm_handle_snp_req_certs(CPUState *cpu, struct kvm_run *run)
+{
+    g_assert_not_reached();
+}
target/i386/sev.c +277 -0

319 lines not on this page · raw message

target/i386/sev.h +1 -1
index 373669eaac..c85c585d9d 100644
@@ -70,5 +70,5 @@ void pc_system_parse_sev_metadata(uint8_t *flash_ptr, size_t flash_size);
 
 uint32_t sev_get_cbit_position(void);
 uint32_t sev_get_reduced_phys_bits(void);
-
+int kvm_handle_snp_req_certs(CPUState *cpu, struct kvm_run *run);
 #endif
Signature · 1 line
2.34.1
#4 lore raw

Re: [RFC PATCH v2 3/3] i386/sev: Add KVM_EXIT_SNP_REQ_CERTS support for certificate-fetching

Quoted 18 lines · Melody Wang

From: Michael Roth <[email protected]>

The GHCB specification[1] defines a VMGEXIT-based Guest Request hypercall to allow an SNP guest to issue encrypted requests directly to SNP firmware to do things like query the attestation report for the guest. These are generally handled purely in the kernel.

In some some cases, it's useful for the host to be able to additionally supply the certificate chain for the signing key that SNP firmware uses to sign these attestation reports. To allow for this, the GHCB specification defines an Extended Guest Request where this certificate data can be provided in a special format described in the GHCB spec. This certificate data may be global or guest-specific depending on how the guest was configured. Rather than providing interfaces to manage these within the kernel, KVM provides a new KVM_EXIT_SNP_REQ_CERTS exit to request the certificate contents from userspace. Implement support for that in QEMU.

Quoted 11 lines

To synchronize delivery of the certificates to the guest in a way where they will not be rendered invalid by updates to SNP firmware or attestation singing/endorsement keys by management tools outside the purview of QEMU, it is expected by users of KVM_EXIT_SNP_REQ_CERTS to obtain a shared/read lock on the certificate file prior to delivering them back to KVM. Only after this will the attestation report be retrieved from firmware and bundled with the certificate data, so QEMU must continue to hold the file lock until KVM confirms that the attestation report has been retrieved/bundled. This confirmation is done by way of the kvm_immediate_exit callback infrastructure that was introduced in a previous patch.

In review of v1, I inquired about the locking, and Mike explained it some more. I wonder whether we should work some of that into our documentation.

Quoted 23 lines

[1] "Guest Hypervisor Communication Block (GHCB) Standardization",
https://www.amd.com/en/developer/sev.html

[Melody: Add a while(1) loop using the libvirt example with fstat and stat for the locking certificate blob code in open_certs_locked() to fix the recreation race problem suggested by Daniel P. Berrangé. And fix the json format and filename suggested by Markus Armbruster.]

---
 qapi/qom.json                 |  21 ++-
 target/i386/kvm/kvm.c         |  10 ++
 target/i386/sev-system-stub.c |   5 +
 target/i386/sev.c             | 277 ++++++++++++++++++++++++++++++++++
 target/i386/sev.h             |   2 +-
 5 files changed, 313 insertions(+), 2 deletions(-)
qapi/qom.json +0 -0
index 28ce24cd8d..c6f9ad5d45 100644

qapi/qom.json line 1039

Quoted 6 lines
qapi/qom.json +3 -0
@@ -1034,6 +1034,23 @@
 #     firmware.  Set this to true to disable the use of VCEK.
 #     (default: false) (since: 9.1)
 #
+# @certs-filename: Certificate data that can be passed to guests via
+#     SNP Extended Guest Requests. File should be in the format
+#     described in the GHCB specification. (default: none)

Is this referring to the "Guest Hypervisor Communication Block (GHCB) Standardization" mentioned in the commit message?

That's 70 pages... Where exactly is the file format specified?

Quoted 4 lines
qapi/qom.json +4 -0
+#     (since: 10.0)
+#
+# @certs-timeout: Maximum time in milliseconds to wait to obtain a read lock
+#     on the certificate file specified by @certs-filename. This is

qapi/qom.json line 1053

Quoted 10 lines
qapi/qom.json +10 -0
+#     not a cumulative value and only affects how long QEMU waits
+#     before returning execution to the vCPU and informing the guest
+#     of the timeout, so the guest can still continue retrying for
+#     as long as it likes (which will be about 60 seconds for linux
+#     guests at the time of this writing). If the guest-side timeout
+#     is insufficient, set this higher to allow more time to fetch the
+#     certificate. If the guest-side timeout is sufficient, set this
+#     lower to reduce the likelihood of soft lockups in the guest.
+#     (default: 100) (since: 10.0)
+#

docs/devel/qapi-code-gen.rst:

For legibility, wrap text paragraphs so every line is at most 70 characters long.

Separate sentences with two spaces.

Quoted 14 lines
qapi/qom.json +3 -1
 # Since: 9.1
 ##
 { 'struct': 'SevSnpGuestProperties',
@@ -1045,7 +1062,9 @@
             '*id-auth': 'str',
             '*author-key-enabled': 'bool',
             '*host-data': 'str',
-            '*vcek-disabled': 'bool' } }
+            '*vcek-disabled': 'bool',
+            '*certs-filename': 'str',
+            '*certs-timeout': 'uint32' } }
 
 ##
 # @ThreadContextProperties: