Add AMD SEV guest live migration support

From: Brijesh Singh <[email protected]>

The command is used to finailize the encryption context created with KVM_SEV_SEND_START command.

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

Cc Paolo Bonzini <[email protected]>

Cc Joerg Roedel <[email protected]>

Cc Borislav Petkov <[email protected]>

Cc Tom Lendacky <[email protected]>

Cc [email protected]

Cc [email protected]

Cc [email protected]

Signed-off-by Brijesh Singh <[email protected]>

Signed-off-by Ashish Kalra <[email protected]>

---
 .../virt/kvm/amd-memory-encryption.rst        |  8 +++++++
 arch/x86/kvm/svm/sev.c                        | 23 +++++++++++++++++++
 2 files changed, 31 insertions(+)

Documentation/virt/kvm/amd-memory-encryption.rst +8 -0

index 3c5456e0268a..26c4e6c83f62 100644
@@ -335,6 +335,14 @@ Returns: 0 on success, -negative on error
                 __u32 trans_len;
         };
 
+12. KVM_SEV_SEND_FINISH
+------------------------
+
+After completion of the migration flow, the KVM_SEV_SEND_FINISH command can be
+issued by the hypervisor to delete the encryption context.
+
+Returns: 0 on success, -negative on error
+
 References
 ==========
 

arch/x86/kvm/svm/sev.c +23 -1

index 30527285a39a..92325d9527ce 100644
@@ -1350,6 +1350,26 @@ static int sev_send_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp)
        return ret;
 }
 
+static int sev_send_finish(struct kvm *kvm, struct kvm_sev_cmd *argp)
+{
+       struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+       struct sev_data_send_finish *data;
+       int ret;
+
+       if (!sev_guest(kvm))
+               return -ENOTTY;
+
+       data = kzalloc(sizeof(*data), GFP_KERNEL);
+       if (!data)
+               return -ENOMEM;
+
+       data->handle = sev->handle;
+       ret = sev_issue_cmd(kvm, SEV_CMD_SEND_FINISH, data, &argp->error);
+
+       kfree(data);
+       return ret;
+}
+
 int svm_mem_enc_op(struct kvm *kvm, void __user *argp)
 {
        struct kvm_sev_cmd sev_cmd;

@@ -1409,6 +1429,9 @@ int svm_mem_enc_op(struct kvm *kvm, void __user *argp)
        case KVM_SEV_SEND_UPDATE_DATA:
                r = sev_send_update_data(kvm, &sev_cmd);
                break;
+       case KVM_SEV_SEND_FINISH:
+               r = sev_send_finish(kvm, &sev_cmd);
+               break;
        default:
                r = -EINVAL;
                goto out;
--

2.17.1

From: Brijesh Singh <[email protected]>

The command is used to create the encryption context for an incoming SEV guest. The encryption context can be later used by the hypervisor to import the incoming data into the SEV guest memory space.

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

Cc Paolo Bonzini <[email protected]>

Cc Joerg Roedel <[email protected]>

Cc Borislav Petkov <[email protected]>

Cc Tom Lendacky <[email protected]>

Cc [email protected]

Cc [email protected]

Cc [email protected]

Signed-off-by Brijesh Singh <[email protected]>

Signed-off-by Ashish Kalra <[email protected]>

---
 .../virt/kvm/amd-memory-encryption.rst        | 29 +++++++
 arch/x86/kvm/svm/sev.c                        | 81 +++++++++++++++++++
 include/uapi/linux/kvm.h                      |  9 +++
 3 files changed, 119 insertions(+)

Documentation/virt/kvm/amd-memory-encryption.rst +29 -0

index 26c4e6c83f62..c86c1ded8dd8 100644
@@ -343,6 +343,35 @@ issued by the hypervisor to delete the encryption context.
 
 Returns: 0 on success, -negative on error
 
+13. KVM_SEV_RECEIVE_START
+------------------------
+
+The KVM_SEV_RECEIVE_START command is used for creating the memory encryption
+context for an incoming SEV guest. To create the encryption context, the user must
+provide a guest policy, the platform public Diffie-Hellman (PDH) key and session
+information.
+
+Parameters: struct  kvm_sev_receive_start (in/out)
+
+Returns: 0 on success, -negative on error
+
+::
+
+        struct kvm_sev_receive_start {
+                __u32 handle;           /* if zero then firmware creates a new handle */
+                __u32 policy;           /* guest's policy */
+
+                __u64 pdh_uaddr;        /* userspace address pointing to the PDH key */
+                __u32 pdh_len;
+
+                __u64 session_uaddr;    /* userspace address which points to the guest session information */
+                __u32 session_len;
+        };
+
+On success, the 'handle' field contains a new handle and on error, a negative value.
+
+For more details, see SEV spec Section 6.12.
+
 References
 ==========
 

arch/x86/kvm/svm/sev.c +81 -0

index 92325d9527ce..e530c2b34b5e 100644
@@ -1370,6 +1370,84 @@ static int sev_send_finish(struct kvm *kvm, struct kvm_sev_cmd *argp)
        return ret;
 }
 
+static int sev_receive_start(struct kvm *kvm, struct kvm_sev_cmd *argp)
+{
+       struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+       struct sev_data_receive_start *start;
+       struct kvm_sev_receive_start params;
+       int *error = &argp->error;
+       void *session_data;
+       void *pdh_data;
+       int ret;
+
+       if (!sev_guest(kvm))
+               return -ENOTTY;
+
+       /* Get parameter from the userspace */
+       if (copy_from_user(&params, (void __user *)(uintptr_t)argp->data,
+                       sizeof(struct kvm_sev_receive_start)))
+               return -EFAULT;
+
+       /* some sanity checks */
+       if (!params.pdh_uaddr || !params.pdh_len ||
+           !params.session_uaddr || !params.session_len)
+               return -EINVAL;
+
+       pdh_data = psp_copy_user_blob(params.pdh_uaddr, params.pdh_len);
+       if (IS_ERR(pdh_data))
+               return PTR_ERR(pdh_data);
+
+       session_data = psp_copy_user_blob(params.session_uaddr,
+                       params.session_len);
+       if (IS_ERR(session_data)) {
+               ret = PTR_ERR(session_data);
+               goto e_free_pdh;
+       }
+
+       ret = -ENOMEM;
+       start = kzalloc(sizeof(*start), GFP_KERNEL);
+       if (!start)
+               goto e_free_session;
+
+       start->handle = params.handle;
+       start->policy = params.policy;
+       start->pdh_cert_address = __psp_pa(pdh_data);
+       start->pdh_cert_len = params.pdh_len;
+       start->session_address = __psp_pa(session_data);
+       start->session_len = params.session_len;
+
+       /* create memory encryption context */
+       ret = __sev_issue_cmd(argp->sev_fd, SEV_CMD_RECEIVE_START, start,
+                               error);
+       if (ret)
+               goto e_free;
+
+       /* Bind ASID to this guest */
+       ret = sev_bind_asid(kvm, start->handle, error);
+       if (ret)
+               goto e_free;
+
+       params.handle = start->handle;
+       if (copy_to_user((void __user *)(uintptr_t)argp->data,
+                        &params, sizeof(struct kvm_sev_receive_start))) {
+               ret = -EFAULT;
+               sev_unbind_asid(kvm, start->handle);
+               goto e_free;
+       }
+
+       sev->handle = start->handle;
+       sev->fd = argp->sev_fd;
+
+e_free:
+       kfree(start);
+e_free_session:
+       kfree(session_data);
+e_free_pdh:
+       kfree(pdh_data);
+
+       return ret;
+}
+
 int svm_mem_enc_op(struct kvm *kvm, void __user *argp)
 {
        struct kvm_sev_cmd sev_cmd;

@@ -1432,6 +1510,9 @@ int svm_mem_enc_op(struct kvm *kvm, void __user *argp)
        case KVM_SEV_SEND_FINISH:
                r = sev_send_finish(kvm, &sev_cmd);
                break;
+       case KVM_SEV_RECEIVE_START:
+               r = sev_receive_start(kvm, &sev_cmd);
+               break;
        default:
                r = -EINVAL;
                goto out;

include/uapi/linux/kvm.h +9 -1

index d45af34c31be..29c25e641a0c 100644
@@ -1750,6 +1750,15 @@ struct kvm_sev_send_update_data {
        __u32 trans_len;
 };
 
+struct kvm_sev_receive_start {
+       __u32 handle;
+       __u32 policy;
+       __u64 pdh_uaddr;
+       __u32 pdh_len;
+       __u64 session_uaddr;
+       __u32 session_len;
+};
+
 #define KVM_DEV_ASSIGN_ENABLE_IOMMU    (1 << 0)
 #define KVM_DEV_ASSIGN_PCI_2_3         (1 << 1)
 #define KVM_DEV_ASSIGN_MASK_INTX       (1 << 2)
--

2.17.1

From: Brijesh Singh <[email protected]>

The command is used for copying the incoming buffer into the SEV guest memory space.

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

Cc Paolo Bonzini <[email protected]>

Cc Joerg Roedel <[email protected]>

Cc Borislav Petkov <[email protected]>

Cc Tom Lendacky <[email protected]>

Cc [email protected]

Cc [email protected]

Cc [email protected]

Signed-off-by Brijesh Singh <[email protected]>

Signed-off-by Ashish Kalra <[email protected]>

---
 .../virt/kvm/amd-memory-encryption.rst        | 24 ++++++
 arch/x86/kvm/svm/sev.c                        | 79 +++++++++++++++++++
 include/uapi/linux/kvm.h                      |  9 +++
 3 files changed, 112 insertions(+)

Documentation/virt/kvm/amd-memory-encryption.rst +24 -0

index c86c1ded8dd8..c6ed5b26d841 100644
@@ -372,6 +372,30 @@ On success, the 'handle' field contains a new handle and on error, a negative va
 
 For more details, see SEV spec Section 6.12.
 
+14. KVM_SEV_RECEIVE_UPDATE_DATA
+----------------------------
+
+The KVM_SEV_RECEIVE_UPDATE_DATA command can be used by the hypervisor to copy
+the incoming buffers into the guest memory region with encryption context
+created during the KVM_SEV_RECEIVE_START.
+
+Parameters (in): struct kvm_sev_receive_update_data
+
+Returns: 0 on success, -negative on error
+
+::
+
+        struct kvm_sev_launch_receive_update_data {
+                __u64 hdr_uaddr;        /* userspace address containing the packet header */
+                __u32 hdr_len;
+
+                __u64 guest_uaddr;      /* the destination guest memory region */
+                __u32 guest_len;
+
+                __u64 trans_uaddr;      /* the incoming buffer memory region  */
+                __u32 trans_len;
+        };
+
 References
 ==========
 

arch/x86/kvm/svm/sev.c +79 -0

index e530c2b34b5e..2c95657cc9bf 100644
@@ -1448,6 +1448,82 @@ static int sev_receive_start(struct kvm *kvm, struct kvm_sev_cmd *argp)
        return ret;
 }
 
+static int sev_receive_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp)
+{
+       struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+       struct kvm_sev_receive_update_data params;
+       struct sev_data_receive_update_data *data;
+       void *hdr = NULL, *trans = NULL;
+       struct page **guest_page;
+       unsigned long n;
+       int ret, offset;
+
+       if (!sev_guest(kvm))
+               return -EINVAL;
+
+       if (copy_from_user(&params, (void __user *)(uintptr_t)argp->data,
+                       sizeof(struct kvm_sev_receive_update_data)))
+               return -EFAULT;
+
+       if (!params.hdr_uaddr || !params.hdr_len ||
+           !params.guest_uaddr || !params.guest_len ||
+           !params.trans_uaddr || !params.trans_len)
+               return -EINVAL;
+
+       /* Check if we are crossing the page boundary */
+       offset = params.guest_uaddr & (PAGE_SIZE - 1);
+       if ((params.guest_len + offset > PAGE_SIZE))
+               return -EINVAL;
+
+       hdr = psp_copy_user_blob(params.hdr_uaddr, params.hdr_len);
+       if (IS_ERR(hdr))
+               return PTR_ERR(hdr);
+
+       trans = psp_copy_user_blob(params.trans_uaddr, params.trans_len);
+       if (IS_ERR(trans)) {
+               ret = PTR_ERR(trans);
+               goto e_free_hdr;
+       }
+
+       ret = -ENOMEM;
+       data = kzalloc(sizeof(*data), GFP_KERNEL);
+       if (!data)
+               goto e_free_trans;
+
+       data->hdr_address = __psp_pa(hdr);
+       data->hdr_len = params.hdr_len;
+       data->trans_address = __psp_pa(trans);
+       data->trans_len = params.trans_len;
+
+       /* Pin guest memory */
+       ret = -EFAULT;
+       guest_page = sev_pin_memory(kvm, params.guest_uaddr & PAGE_MASK,
+                                   PAGE_SIZE, &n, 0);
+       if (!guest_page)
+               goto e_free;
+
+       /* The RECEIVE_UPDATE_DATA command requires C-bit to be always set. */
+       data->guest_address = (page_to_pfn(guest_page[0]) << PAGE_SHIFT) +
+                               offset;
+       data->guest_address |= sev_me_mask;
+       data->guest_len = params.guest_len;
+       data->handle = sev->handle;
+
+       ret = sev_issue_cmd(kvm, SEV_CMD_RECEIVE_UPDATE_DATA, data,
+                               &argp->error);
+
+       sev_unpin_memory(kvm, guest_page, n);
+
+e_free:
+       kfree(data);
+e_free_trans:
+       kfree(trans);
+e_free_hdr:
+       kfree(hdr);
+
+       return ret;
+}
+
 int svm_mem_enc_op(struct kvm *kvm, void __user *argp)
 {
        struct kvm_sev_cmd sev_cmd;

@@ -1513,6 +1589,9 @@ int svm_mem_enc_op(struct kvm *kvm, void __user *argp)
        case KVM_SEV_RECEIVE_START:
                r = sev_receive_start(kvm, &sev_cmd);
                break;
+       case KVM_SEV_RECEIVE_UPDATE_DATA:
+               r = sev_receive_update_data(kvm, &sev_cmd);
+               break;
        default:
                r = -EINVAL;
                goto out;

include/uapi/linux/kvm.h +9 -1

index 29c25e641a0c..3a656d43fc6c 100644
@@ -1759,6 +1759,15 @@ struct kvm_sev_receive_start {
        __u32 session_len;
 };
 
+struct kvm_sev_receive_update_data {
+       __u64 hdr_uaddr;
+       __u32 hdr_len;
+       __u64 guest_uaddr;
+       __u32 guest_len;
+       __u64 trans_uaddr;
+       __u32 trans_len;
+};
+
 #define KVM_DEV_ASSIGN_ENABLE_IOMMU    (1 << 0)
 #define KVM_DEV_ASSIGN_PCI_2_3         (1 << 1)
 #define KVM_DEV_ASSIGN_MASK_INTX       (1 << 2)
--

2.17.1

From: Brijesh Singh <[email protected]>

The command finalize the guest receiving process and make the SEV guest ready for the execution.

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

Cc Paolo Bonzini <[email protected]>

Cc Joerg Roedel <[email protected]>

Cc Borislav Petkov <[email protected]>

Cc Tom Lendacky <[email protected]>

Cc [email protected]

Cc [email protected]

Cc [email protected]

Reviewed-by Steve Rutherford <[email protected]>

Signed-off-by Brijesh Singh <[email protected]>

Signed-off-by Ashish Kalra <[email protected]>

---
 .../virt/kvm/amd-memory-encryption.rst        |  8 +++++++
 arch/x86/kvm/svm/sev.c                        | 23 +++++++++++++++++++
 2 files changed, 31 insertions(+)

Documentation/virt/kvm/amd-memory-encryption.rst +8 -0

index c6ed5b26d841..0466c0febff9 100644
@@ -396,6 +396,14 @@ Returns: 0 on success, -negative on error
                 __u32 trans_len;
         };
 
+15. KVM_SEV_RECEIVE_FINISH
+------------------------
+
+After completion of the migration flow, the KVM_SEV_RECEIVE_FINISH command can be
+issued by the hypervisor to make the guest ready for execution.
+
+Returns: 0 on success, -negative on error
+
 References
 ==========
 

arch/x86/kvm/svm/sev.c +23 -1

index 2c95657cc9bf..c9795a22e502 100644
@@ -1524,6 +1524,26 @@ static int sev_receive_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp)
        return ret;
 }
 
+static int sev_receive_finish(struct kvm *kvm, struct kvm_sev_cmd *argp)
+{
+       struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+       struct sev_data_receive_finish *data;
+       int ret;
+
+       if (!sev_guest(kvm))
+               return -ENOTTY;
+
+       data = kzalloc(sizeof(*data), GFP_KERNEL);
+       if (!data)
+               return -ENOMEM;
+
+       data->handle = sev->handle;
+       ret = sev_issue_cmd(kvm, SEV_CMD_RECEIVE_FINISH, data, &argp->error);
+
+       kfree(data);
+       return ret;
+}
+
 int svm_mem_enc_op(struct kvm *kvm, void __user *argp)
 {
        struct kvm_sev_cmd sev_cmd;

@@ -1592,6 +1612,9 @@ int svm_mem_enc_op(struct kvm *kvm, void __user *argp)
        case KVM_SEV_RECEIVE_UPDATE_DATA:
                r = sev_receive_update_data(kvm, &sev_cmd);
                break;
+       case KVM_SEV_RECEIVE_FINISH:
+               r = sev_receive_finish(kvm, &sev_cmd);
+               break;
        default:
                r = -EINVAL;
                goto out;
--

2.17.1

From: Ashish Kalra <[email protected]>

This hypercall is used by the SEV guest to notify a change in the page encryption status to the hypervisor. The hypercall should be invoked only when the encryption attribute is changed from encrypted -> decrypted and vice versa. By default all guest pages are considered encrypted.

The hypercall exits to userspace to manage the guest shared regions and integrate with the userspace VMM's migration code.

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

Cc Paolo Bonzini <[email protected]>

Cc Joerg Roedel <[email protected]>

Cc Borislav Petkov <[email protected]>

Cc Tom Lendacky <[email protected]>

Cc [email protected]

Cc [email protected]

Cc [email protected]

Signed-off-by Brijesh Singh <[email protected]>

Signed-off-by Ashish Kalra <[email protected]>

Co-developed-by Sean Christopherson <[email protected]>

Signed-off-by Sean Christopherson <[email protected]>

---
 Documentation/virt/kvm/hypercalls.rst | 15 ++++++++++++++
 arch/x86/include/asm/kvm_host.h       |  2 ++
 arch/x86/kvm/svm/sev.c                |  1 +
 arch/x86/kvm/x86.c                    | 29 +++++++++++++++++++++++++++
 include/uapi/linux/kvm_para.h         |  1 +
 5 files changed, 48 insertions(+)

Documentation/virt/kvm/hypercalls.rst +15 -0

index ed4fddd364ea..7aff0cebab7c 100644
@@ -169,3 +169,18 @@ a0: destination APIC ID
 
 :Usage example: When sending a call-function IPI-many to vCPUs, yield if
                any of the IPI target vCPUs was preempted.
+
+
+8. KVM_HC_PAGE_ENC_STATUS
+-------------------------
+:Architecture: x86
+:Status: active
+:Purpose: Notify the encryption status changes in guest page table (SEV guest)
+
+a0: the guest physical address of the start page
+a1: the number of pages
+a2: encryption attribute
+
+   Where:
+       * 1: Encryption attribute is set
+       * 0: Encryption attribute is cleared

arch/x86/include/asm/kvm_host.h +2 -0

index 3768819693e5..42eb0fe3df5d 100644
@@ -1050,6 +1050,8 @@ struct kvm_arch {
 
        bool bus_lock_detection_enabled;
 
+       bool page_enc_hc_enable;
+
        /* Deflect RDMSR and WRMSR to user space when they trigger a #GP */
        u32 user_space_msr_mask;
        struct kvm_x86_msr_filter __rcu *msr_filter;

arch/x86/kvm/svm/sev.c +1 -0

index c9795a22e502..5184a0c0131a 100644
@@ -197,6 +197,7 @@ static int sev_guest_init(struct kvm *kvm, struct kvm_sev_cmd *argp)
        sev->active = true;
        sev->asid = asid;
        INIT_LIST_HEAD(&sev->regions_list);
+       kvm->arch.page_enc_hc_enable = true;
 
        return 0;
 

arch/x86/kvm/x86.c +29 -0

index f7d12fca397b..e8986478b653 100644
@@ -8208,6 +8208,13 @@ static void kvm_sched_yield(struct kvm *kvm, unsigned long dest_id)
                kvm_vcpu_yield_to(target);
 }
 
+static int complete_hypercall_exit(struct kvm_vcpu *vcpu)
+{
+       kvm_rax_write(vcpu, vcpu->run->hypercall.ret);
+       ++vcpu->stat.hypercalls;
+       return kvm_skip_emulated_instruction(vcpu);
+}
+
 int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
 {
        unsigned long nr, a0, a1, a2, a3, ret;

@@ -8273,6 +8280,28 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
                kvm_sched_yield(vcpu->kvm, a0);
                ret = 0;
                break;
+       case KVM_HC_PAGE_ENC_STATUS: {
+               u64 gpa = a0, npages = a1, enc = a2;
+
+               ret = -KVM_ENOSYS;
+               if (!vcpu->kvm->arch.page_enc_hc_enable)
+                       break;
+
+               if (!PAGE_ALIGNED(gpa) || !npages ||
+                   gpa_to_gfn(gpa) + npages <= gpa_to_gfn(gpa)) {
+                       ret = -EINVAL;
+                       break;
+               }
+
+               vcpu->run->exit_reason        = KVM_EXIT_HYPERCALL;
+               vcpu->run->hypercall.nr       = KVM_HC_PAGE_ENC_STATUS;
+               vcpu->run->hypercall.args[0]  = gpa;
+               vcpu->run->hypercall.args[1]  = npages;
+               vcpu->run->hypercall.args[2]  = enc;
+               vcpu->run->hypercall.longmode = op_64_bit;
+               vcpu->arch.complete_userspace_io = complete_hypercall_exit;
+               return 0;
+       }
        default:
                ret = -KVM_ENOSYS;
                break;

include/uapi/linux/kvm_para.h +1 -1

index 8b86609849b9..847b83b75dc8 100644
@@ -29,6 +29,7 @@
 #define KVM_HC_CLOCK_PAIRING           9
 #define KVM_HC_SEND_IPI                10
 #define KVM_HC_SCHED_YIELD             11
+#define KVM_HC_PAGE_ENC_STATUS         12
 
 /*
  * hypercalls use architecture specific
--

2.17.1

From: Brijesh Singh <[email protected]>

Invoke a hypercall when a memory region is changed from encrypted -> decrypted and vice versa. Hypervisor needs to know the page encryption status during the guest migration.

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

Cc Paolo Bonzini <[email protected]>

Cc Joerg Roedel <[email protected]>

Cc Borislav Petkov <[email protected]>

Cc Tom Lendacky <[email protected]>

Cc [email protected]

Cc [email protected]

Cc [email protected]

Reviewed-by Venu Busireddy <[email protected]>

Signed-off-by Brijesh Singh <[email protected]>

Signed-off-by Ashish Kalra <[email protected]>

---
 arch/x86/include/asm/paravirt.h       | 10 +++++
 arch/x86/include/asm/paravirt_types.h |  2 +
 arch/x86/kernel/paravirt.c            |  1 +
 arch/x86/mm/mem_encrypt.c             | 57 ++++++++++++++++++++++++++-
 arch/x86/mm/pat/set_memory.c          |  7 ++++
 5 files changed, 76 insertions(+), 1 deletion(-)

arch/x86/include/asm/paravirt.h +10 -0

index 4abf110e2243..efaa3e628967 100644
@@ -84,6 +84,12 @@ static inline void paravirt_arch_exit_mmap(struct mm_struct *mm)
        PVOP_VCALL1(mmu.exit_mmap, mm);
 }
 
+static inline void page_encryption_changed(unsigned long vaddr, int npages,
+                                               bool enc)
+{
+       PVOP_VCALL3(mmu.page_encryption_changed, vaddr, npages, enc);
+}
+
 #ifdef CONFIG_PARAVIRT_XXL
 static inline void load_sp0(unsigned long sp0)
 {

@@ -799,6 +805,10 @@ static inline void paravirt_arch_dup_mmap(struct mm_struct *oldmm,
 static inline void paravirt_arch_exit_mmap(struct mm_struct *mm)
 {
 }
+
+static inline void page_encryption_changed(unsigned long vaddr, int npages, bool enc)
+{
+}
 #endif
 #endif /* __ASSEMBLY__ */
 #endif /* _ASM_X86_PARAVIRT_H */

arch/x86/include/asm/paravirt_types.h +2 -0

index de87087d3bde..69ef9c207b38 100644
@@ -195,6 +195,8 @@ struct pv_mmu_ops {
 
        /* Hook for intercepting the destruction of an mm_struct. */
        void (*exit_mmap)(struct mm_struct *mm);
+       void (*page_encryption_changed)(unsigned long vaddr, int npages,
+                                       bool enc);
 
 #ifdef CONFIG_PARAVIRT_XXL
        struct paravirt_callee_save read_cr2;

arch/x86/kernel/paravirt.c +1 -0

index c60222ab8ab9..9f206e192f6b 100644
@@ -335,6 +335,7 @@ struct paravirt_patch_template pv_ops = {
                        (void (*)(struct mmu_gather *, void *))tlb_remove_page,
 
        .mmu.exit_mmap          = paravirt_nop,
+       .mmu.page_encryption_changed    = paravirt_nop,
 
 #ifdef CONFIG_PARAVIRT_XXL
        .mmu.read_cr2           = __PV_IS_CALLEE_SAVE(native_read_cr2),

arch/x86/mm/mem_encrypt.c +56 -1

index ae78cef79980..fae9ccbd0da7 100644
@@ -19,6 +19,7 @@
 #include <linux/kernel.h>
 #include <linux/bitops.h>
 #include <linux/dma-mapping.h>
+#include <linux/kvm_para.h>
 
 #include <asm/tlbflush.h>
 #include <asm/fixmap.h>

@@ -29,6 +30,7 @@
 #include <asm/processor-flags.h>
 #include <asm/msr.h>
 #include <asm/cmdline.h>
+#include <asm/kvm_para.h>
 
 #include "mm_internal.h"
 

@@ -229,6 +231,47 @@ void __init sev_setup_arch(void)
        swiotlb_adjust_size(size);
 }
 
+static void set_memory_enc_dec_hypercall(unsigned long vaddr, int npages,
+                                       bool enc)
+{
+       unsigned long sz = npages << PAGE_SHIFT;
+       unsigned long vaddr_end, vaddr_next;
+
+       vaddr_end = vaddr + sz;
+
+       for (; vaddr < vaddr_end; vaddr = vaddr_next) {
+               int psize, pmask, level;
+               unsigned long pfn;
+               pte_t *kpte;
+
+               kpte = lookup_address(vaddr, &level);
+               if (!kpte || pte_none(*kpte))
+                       return;
+
+               switch (level) {
+               case PG_LEVEL_4K:
+                       pfn = pte_pfn(*kpte);
+                       break;
+               case PG_LEVEL_2M:
+                       pfn = pmd_pfn(*(pmd_t *)kpte);
+                       break;
+               case PG_LEVEL_1G:
+                       pfn = pud_pfn(*(pud_t *)kpte);
+                       break;
+               default:
+                       return;
+               }
+
+               psize = page_level_size(level);
+               pmask = page_level_mask(level);
+
+               kvm_sev_hypercall3(KVM_HC_PAGE_ENC_STATUS,
+                                  pfn << PAGE_SHIFT, psize >> PAGE_SHIFT, enc);
+
+               vaddr_next = (vaddr & pmask) + psize;
+       }
+}
+
 static void __init __set_clr_pte_enc(pte_t *kpte, int level, bool enc)
 {
        pgprot_t old_prot, new_prot;

@@ -286,12 +329,13 @@ static void __init __set_clr_pte_enc(pte_t *kpte, int level, bool enc)
 static int __init early_set_memory_enc_dec(unsigned long vaddr,
                                           unsigned long size, bool enc)
 {
-       unsigned long vaddr_end, vaddr_next;
+       unsigned long vaddr_end, vaddr_next, start;
        unsigned long psize, pmask;
        int split_page_size_mask;
        int level, ret;
        pte_t *kpte;
 
+       start = vaddr;
        vaddr_next = vaddr;
        vaddr_end = vaddr + size;
 

@@ -346,6 +390,8 @@ static int __init early_set_memory_enc_dec(unsigned long vaddr,
 
        ret = 0;
 
+       set_memory_enc_dec_hypercall(start, PAGE_ALIGN(size) >> PAGE_SHIFT,
+                                       enc);
 out:
        __flush_tlb_all();
        return ret;

@@ -481,6 +527,15 @@ void __init mem_encrypt_init(void)
        if (sev_active() && !sev_es_active())
                static_branch_enable(&sev_enable_key);
 
+#ifdef CONFIG_PARAVIRT
+       /*
+        * With SEV, we need to make a hypercall when page encryption state is
+        * changed.
+        */
+       if (sev_active())
+               pv_ops.mmu.page_encryption_changed = set_memory_enc_dec_hypercall;
+#endif
+
        print_mem_encrypt_feature_info();
 }
 

arch/x86/mm/pat/set_memory.c +7 -1

index 16f878c26667..3576b583ac65 100644
@@ -27,6 +27,7 @@
 #include <asm/proto.h>
 #include <asm/memtype.h>
 #include <asm/set_memory.h>
+#include <asm/paravirt.h>
 
 #include "../mm_internal.h"
 

@@ -2012,6 +2013,12 @@ static int __set_memory_enc_dec(unsigned long addr, int numpages, bool enc)
         */
        cpa_flush(&cpa, 0);
 
+       /* Notify hypervisor that a given memory range is mapped encrypted
+        * or decrypted. The hypervisor will use this information during the
+        * VM migration.
+        */
+       page_encryption_changed(addr, numpages, enc);
+
        return ret;
 }
 
--

2.17.1

From: Ashish Kalra <[email protected]>

Add new KVM_FEATURE_SEV_LIVE_MIGRATION feature for guest to check for host-side support for SEV live migration. Also add a new custom MSR_KVM_SEV_LIVE_MIGRATION for guest to enable the SEV live migration feature.

MSR is handled by userspace using MSR filters.

Signed-off-by Ashish Kalra <[email protected]>

---
 Documentation/virt/kvm/cpuid.rst     |  5 +++++
 Documentation/virt/kvm/msr.rst       | 12 ++++++++++++
 arch/x86/include/uapi/asm/kvm_para.h |  4 ++++
 arch/x86/kvm/cpuid.c                 |  3 ++-
 4 files changed, 23 insertions(+), 1 deletion(-)

Documentation/virt/kvm/cpuid.rst +5 -0

index cf62162d4be2..0bdb6cdb12d3 100644
@@ -96,6 +96,11 @@ KVM_FEATURE_MSI_EXT_DEST_ID        15          guest checks this feature bit
                                                before using extended destination
                                                ID bits in MSI address bits 11-5.
 
+KVM_FEATURE_SEV_LIVE_MIGRATION     16          guest checks this feature bit before
+                                               using the page encryption state
+                                               hypercall to notify the page state
+                                               change
+
 KVM_FEATURE_CLOCKSOURCE_STABLE_BIT 24          host will warn if no guest-side
                                                per-cpu warps are expected in
                                                kvmclock

Documentation/virt/kvm/msr.rst +12 -0

index e37a14c323d2..020245d16087 100644
@@ -376,3 +376,15 @@ data:
        write '1' to bit 0 of the MSR, this causes the host to re-scan its queue
        and check if there are more notifications pending. The MSR is available
        if KVM_FEATURE_ASYNC_PF_INT is present in CPUID.
+
+MSR_KVM_SEV_LIVE_MIGRATION:
+        0x4b564d08
+
+       Control SEV Live Migration features.
+
+data:
+        Bit 0 enables (1) or disables (0) host-side SEV Live Migration feature,
+        in other words, this is guest->host communication that it's properly
+        handling the shared pages list.
+
+        All other bits are reserved.

arch/x86/include/uapi/asm/kvm_para.h +4 -0

index 950afebfba88..f6bfa138874f 100644
@@ -33,6 +33,7 @@
 #define KVM_FEATURE_PV_SCHED_YIELD     13
 #define KVM_FEATURE_ASYNC_PF_INT       14
 #define KVM_FEATURE_MSI_EXT_DEST_ID    15
+#define KVM_FEATURE_SEV_LIVE_MIGRATION 16
 
 #define KVM_HINTS_REALTIME      0
 

@@ -54,6 +55,7 @@
 #define MSR_KVM_POLL_CONTROL   0x4b564d05
 #define MSR_KVM_ASYNC_PF_INT   0x4b564d06
 #define MSR_KVM_ASYNC_PF_ACK   0x4b564d07
+#define MSR_KVM_SEV_LIVE_MIGRATION     0x4b564d08
 
 struct kvm_steal_time {
        __u64 steal;

@@ -136,4 +138,6 @@ struct kvm_vcpu_pv_apf_data {
 #define KVM_PV_EOI_ENABLED KVM_PV_EOI_MASK
 #define KVM_PV_EOI_DISABLED 0x0
 
+#define KVM_SEV_LIVE_MIGRATION_ENABLED BIT_ULL(0)
+
 #endif /* _UAPI_ASM_X86_KVM_PARA_H */

arch/x86/kvm/cpuid.c +2 -2

index 6bd2f8b830e4..4e2e69a692aa 100644
@@ -812,7 +812,8 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
                             (1 << KVM_FEATURE_PV_SEND_IPI) |
                             (1 << KVM_FEATURE_POLL_CONTROL) |
                             (1 << KVM_FEATURE_PV_SCHED_YIELD) |
-                            (1 << KVM_FEATURE_ASYNC_PF_INT);
+                            (1 << KVM_FEATURE_ASYNC_PF_INT) |
+                            (1 << KVM_FEATURE_SEV_LIVE_MIGRATION);
 
                if (sched_info_on())
                        entry->eax |= (1 << KVM_FEATURE_STEAL_TIME);
--

2.17.1

From: Ashish Kalra <[email protected]>

Introduce a new AMD Memory Encryption GUID which is currently used for defining a new UEFI environment variable which indicates UEFI/OVMF support for the SEV live migration feature. This variable is setup when UEFI/OVMF detects host/hypervisor support for SEV live migration and later this variable is read by the kernel using EFI runtime services to verify if OVMF supports the live migration feature.

Signed-off-by Ashish Kalra <[email protected]>

---
 include/linux/efi.h | 1 +
 1 file changed, 1 insertion(+)

include/linux/efi.h +1 -1

index 6b5d36babfcc..6f364ace82cb 100644
@@ -362,6 +362,7 @@ void efi_native_runtime_setup(void);
 
 /* OEM GUIDs */
 #define DELLEMC_EFI_RCI2_TABLE_GUID            EFI_GUID(0x2d9f28a2, 0xa886, 0x456a,  0x97, 0xa8, 0xf1, 0x1e, 0xf2, 0x4f, 0xf4, 0x55)
+#define MEM_ENCRYPT_GUID                       EFI_GUID(0x0cf29b71, 0x9e51, 0x433a,  0xa3, 0xb7, 0x81, 0xf3, 0xab, 0x16, 0xb8, 0x75)
 
 typedef struct {
        efi_guid_t guid;
--

2.17.1

From: Ashish Kalra <[email protected]>

The guest support for detecting and enabling SEV Live migration feature uses the following logic :

- kvm_init_plaform() invokes check_kvm_sev_migration() which checks if its booted under the EFI

- If not EFI,

i) check for the KVM_FEATURE_CPUID

ii) if CPUID reports that migration is supported, issue a wrmsrl()
to enable the SEV live migration support

- If EFI,

i) check for the KVM_FEATURE_CPUID

ii) If CPUID reports that migration is supported, read the UEFI variable which
indicates OVMF support for live migration

iii) the variable indicates live migration is supported, issue a wrmsrl() to
enable the SEV live migration support

The EFI live migration check is done using a late_initcall() callback.

Also, ensure that _bss_decrypted section is marked as decrypted in the shared pages list.

Signed-off-by Ashish Kalra <[email protected]>

---
 arch/x86/include/asm/mem_encrypt.h |  8 +++++
 arch/x86/kernel/kvm.c              | 52 ++++++++++++++++++++++++++++++
 arch/x86/mm/mem_encrypt.c          | 41 +++++++++++++++++++++++
 3 files changed, 101 insertions(+)

arch/x86/include/asm/mem_encrypt.h +8 -0

index 31c4df123aa0..19b77f3a62dc 100644
@@ -21,6 +21,7 @@
 extern u64 sme_me_mask;
 extern u64 sev_status;
 extern bool sev_enabled;
+extern bool sev_live_migration_enabled;
 
 void sme_encrypt_execute(unsigned long encrypted_kernel_vaddr,
                         unsigned long decrypted_kernel_vaddr,

@@ -44,8 +45,11 @@ void __init sme_enable(struct boot_params *bp);
 
 int __init early_set_memory_decrypted(unsigned long vaddr, unsigned long size);
 int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long size);
+void __init early_set_mem_enc_dec_hypercall(unsigned long vaddr, int npages,
+                                           bool enc);
 
 void __init mem_encrypt_free_decrypted_mem(void);
+void __init check_kvm_sev_migration(void);
 
 /* Architecture __weak replacement functions */
 void __init mem_encrypt_init(void);

@@ -60,6 +64,7 @@ bool sev_es_active(void);
 #else  /* !CONFIG_AMD_MEM_ENCRYPT */
 
 #define sme_me_mask    0ULL
+#define sev_live_migration_enabled     false
 
 static inline void __init sme_early_encrypt(resource_size_t paddr,
                                            unsigned long size) { }

@@ -84,8 +89,11 @@ static inline int __init
 early_set_memory_decrypted(unsigned long vaddr, unsigned long size) { return 0; }
 static inline int __init
 early_set_memory_encrypted(unsigned long vaddr, unsigned long size) { return 0; }
+static inline void __init
+early_set_mem_enc_dec_hypercall(unsigned long vaddr, int npages, bool enc) {}
 
 static inline void mem_encrypt_free_decrypted_mem(void) { }
+static inline void check_kvm_sev_migration(void) { }
 
 #define __bss_decrypted
 

arch/x86/kernel/kvm.c +52 -0

index 78bb0fae3982..bcc82e0c9779 100644
@@ -26,6 +26,7 @@
 #include <linux/kprobes.h>
 #include <linux/nmi.h>
 #include <linux/swait.h>
+#include <linux/efi.h>
 #include <asm/timer.h>
 #include <asm/cpu.h>
 #include <asm/traps.h>

@@ -429,6 +430,56 @@ static inline void __set_percpu_decrypted(void *ptr, unsigned long size)
        early_set_memory_decrypted((unsigned long) ptr, size);
 }
 
+static int __init setup_kvm_sev_migration(void)
+{
+       efi_char16_t efi_sev_live_migration_enabled[] = L"SevLiveMigrationEnabled";
+       efi_guid_t efi_variable_guid = MEM_ENCRYPT_GUID;
+       efi_status_t status;
+       unsigned long size;
+       bool enabled;
+
+       /*
+        * check_kvm_sev_migration() invoked via kvm_init_platform() before
+        * this callback would have setup the indicator that live migration
+        * feature is supported/enabled.
+        */
+       if (!sev_live_migration_enabled)
+               return 0;
+
+       if (!efi_enabled(EFI_RUNTIME_SERVICES)) {
+               pr_info("%s : EFI runtime services are not enabled\n", __func__);
+               return 0;
+       }
+
+       size = sizeof(enabled);
+
+       /* Get variable contents into buffer */
+       status = efi.get_variable(efi_sev_live_migration_enabled,
+                                 &efi_variable_guid, NULL, &size, &enabled);
+
+       if (status == EFI_NOT_FOUND) {
+               pr_info("%s : EFI live migration variable not found\n", __func__);
+               return 0;
+       }
+
+       if (status != EFI_SUCCESS) {
+               pr_info("%s : EFI variable retrieval failed\n", __func__);
+               return 0;
+       }
+
+       if (enabled == 0) {
+               pr_info("%s: live migration disabled in EFI\n", __func__);
+               return 0;
+       }
+
+       pr_info("%s : live migration enabled in EFI\n", __func__);
+       wrmsrl(MSR_KVM_SEV_LIVE_MIGRATION, KVM_SEV_LIVE_MIGRATION_ENABLED);
+
+       return true;
+}
+
+late_initcall(setup_kvm_sev_migration);
+
 /*
  * Iterate through all possible CPUs and map the memory region pointed
  * by apf_reason, steal_time and kvm_apic_eoi as decrypted at once.

@@ -747,6 +798,7 @@ static bool __init kvm_msi_ext_dest_id(void)
 
 static void __init kvm_init_platform(void)
 {
+       check_kvm_sev_migration();
        kvmclock_init();
        x86_platform.apic_post_init = kvm_apic_init;
 }

arch/x86/mm/mem_encrypt.c +28 -0

index fae9ccbd0da7..4de417333c09 100644
@@ -20,6 +20,7 @@
 #include <linux/bitops.h>
 #include <linux/dma-mapping.h>
 #include <linux/kvm_para.h>
+#include <linux/efi.h>
 
 #include <asm/tlbflush.h>
 #include <asm/fixmap.h>

@@ -48,6 +49,8 @@ EXPORT_SYMBOL_GPL(sev_enable_key);
 
 bool sev_enabled __section(".data");
 
+bool sev_live_migration_enabled __section(".data");
+
 /* Buffer used for early in-place encryption by BSP, no locking needed */
 static char sme_early_buffer[PAGE_SIZE] __initdata __aligned(PAGE_SIZE);
 

@@ -237,6 +240,9 @@ static void set_memory_enc_dec_hypercall(unsigned long vaddr, int npages,
        unsigned long sz = npages << PAGE_SHIFT;
        unsigned long vaddr_end, vaddr_next;
 
+       if (!sev_live_migration_enabled)
+               return;
+
        vaddr_end = vaddr + sz;
 
        for (; vaddr < vaddr_end; vaddr = vaddr_next) {

@@ -407,6 +413,12 @@ int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long size)
        return early_set_memory_enc_dec(vaddr, size, true);
 }
 
+void __init early_set_mem_enc_dec_hypercall(unsigned long vaddr, int npages,
+                                       bool enc)
+{
+       set_memory_enc_dec_hypercall(vaddr, npages, enc);
+}
+
 /*
  * SME and SEV are very similar but they are not the same, so there are
  * times that the kernel will need to distinguish between SME and SEV. The

@@ -462,6 +474,35 @@ bool force_dma_unencrypted(struct device *dev)
        return false;
 }
 
+void __init check_kvm_sev_migration(void)
+{
+       if (sev_active() &&
+           kvm_para_has_feature(KVM_FEATURE_SEV_LIVE_MIGRATION)) {
+               unsigned long nr_pages;
+
+               pr_info("KVM enable live migration\n");
+               sev_live_migration_enabled = true;
+
+               /*
+                * Ensure that _bss_decrypted section is marked as decrypted in the
+                * shared pages list.
+                */
+               nr_pages = DIV_ROUND_UP(__end_bss_decrypted - __start_bss_decrypted,
+                                       PAGE_SIZE);
+               early_set_mem_enc_dec_hypercall((unsigned long)__start_bss_decrypted,

arch/x86/mm/mem_encrypt.c +3 -1

+               }
+}
+
 void __init mem_encrypt_free_decrypted_mem(void)
 {
        unsigned long vaddr, vaddr_end, npages;
--

2.17.1

Quoted 228 lines · Ashish Kalra

From: Ashish Kalra <[email protected]>

The guest support for detecting and enabling SEV Live migration feature uses the following logic :

- kvm_init_plaform() invokes check_kvm_sev_migration() which checks if its booted under the EFI

- If not EFI,

i) check for the KVM_FEATURE_CPUID

ii) if CPUID reports that migration is supported, issue a wrmsrl()
to enable the SEV live migration support

- If EFI,

i) check for the KVM_FEATURE_CPUID

ii) If CPUID reports that migration is supported, read the UEFI variable which
indicates OVMF support for live migration

iii) the variable indicates live migration is supported, issue a wrmsrl() to
enable the SEV live migration support

The EFI live migration check is done using a late_initcall() callback.

Also, ensure that _bss_decrypted section is marked as decrypted in the shared pages list.

Signed-off-by Ashish Kalra <[email protected]>

---
 arch/x86/include/asm/mem_encrypt.h |  8 +++++
 arch/x86/kernel/kvm.c              | 52 ++++++++++++++++++++++++++++++
 arch/x86/mm/mem_encrypt.c          | 41 +++++++++++++++++++++++
 3 files changed, 101 insertions(+)

arch/x86/include/asm/mem_encrypt.h +8 -0

index 31c4df123aa0..19b77f3a62dc 100644
@@ -21,6 +21,7 @@
 extern u64 sme_me_mask;
 extern u64 sev_status;
 extern bool sev_enabled;
+extern bool sev_live_migration_enabled;
 
 void sme_encrypt_execute(unsigned long encrypted_kernel_vaddr,
                         unsigned long decrypted_kernel_vaddr,

@@ -44,8 +45,11 @@ void __init sme_enable(struct boot_params *bp);
 
 int __init early_set_memory_decrypted(unsigned long vaddr, unsigned long size);
 int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long size);
+void __init early_set_mem_enc_dec_hypercall(unsigned long vaddr, int npages,
+                                           bool enc);
 
 void __init mem_encrypt_free_decrypted_mem(void);
+void __init check_kvm_sev_migration(void);
 
 /* Architecture __weak replacement functions */
 void __init mem_encrypt_init(void);

@@ -60,6 +64,7 @@ bool sev_es_active(void);
 #else  /* !CONFIG_AMD_MEM_ENCRYPT */
 
 #define sme_me_mask    0ULL
+#define sev_live_migration_enabled     false
 
 static inline void __init sme_early_encrypt(resource_size_t paddr,
                                            unsigned long size) { }

@@ -84,8 +89,11 @@ static inline int __init
 early_set_memory_decrypted(unsigned long vaddr, unsigned long size) { return 0; }
 static inline int __init
 early_set_memory_encrypted(unsigned long vaddr, unsigned long size) { return 0; }
+static inline void __init
+early_set_mem_enc_dec_hypercall(unsigned long vaddr, int npages, bool enc) {}
 
 static inline void mem_encrypt_free_decrypted_mem(void) { }
+static inline void check_kvm_sev_migration(void) { }
 
 #define __bss_decrypted
 

arch/x86/kernel/kvm.c +52 -0

index 78bb0fae3982..bcc82e0c9779 100644
@@ -26,6 +26,7 @@
 #include <linux/kprobes.h>
 #include <linux/nmi.h>
 #include <linux/swait.h>
+#include <linux/efi.h>
 #include <asm/timer.h>
 #include <asm/cpu.h>
 #include <asm/traps.h>

@@ -429,6 +430,56 @@ static inline void __set_percpu_decrypted(void *ptr, unsigned long size)
        early_set_memory_decrypted((unsigned long) ptr, size);
 }
 
+static int __init setup_kvm_sev_migration(void)
+{
+       efi_char16_t efi_sev_live_migration_enabled[] = L"SevLiveMigrationEnabled";
+       efi_guid_t efi_variable_guid = MEM_ENCRYPT_GUID;
+       efi_status_t status;
+       unsigned long size;
+       bool enabled;
+
+       /*
+        * check_kvm_sev_migration() invoked via kvm_init_platform() before
+        * this callback would have setup the indicator that live migration
+        * feature is supported/enabled.
+        */
+       if (!sev_live_migration_enabled)
+               return 0;
+
+       if (!efi_enabled(EFI_RUNTIME_SERVICES)) {
+               pr_info("%s : EFI runtime services are not enabled\n", __func__);
+               return 0;
+       }
+
+       size = sizeof(enabled);
+
+       /* Get variable contents into buffer */
+       status = efi.get_variable(efi_sev_live_migration_enabled,
+                                 &efi_variable_guid, NULL, &size, &enabled);
+
+       if (status == EFI_NOT_FOUND) {
+               pr_info("%s : EFI live migration variable not found\n", __func__);
+               return 0;
+       }
+
+       if (status != EFI_SUCCESS) {
+               pr_info("%s : EFI variable retrieval failed\n", __func__);
+               return 0;
+       }
+
+       if (enabled == 0) {
+               pr_info("%s: live migration disabled in EFI\n", __func__);
+               return 0;
+       }
+
+       pr_info("%s : live migration enabled in EFI\n", __func__);
+       wrmsrl(MSR_KVM_SEV_LIVE_MIGRATION, KVM_SEV_LIVE_MIGRATION_ENABLED);
+
+       return true;
+}
+
+late_initcall(setup_kvm_sev_migration);
+
 /*
  * Iterate through all possible CPUs and map the memory region pointed
  * by apf_reason, steal_time and kvm_apic_eoi as decrypted at once.

@@ -747,6 +798,7 @@ static bool __init kvm_msi_ext_dest_id(void)
 
 static void __init kvm_init_platform(void)
 {
+       check_kvm_sev_migration();
        kvmclock_init();
        x86_platform.apic_post_init = kvm_apic_init;
 }

arch/x86/mm/mem_encrypt.c +33 -0

index fae9ccbd0da7..4de417333c09 100644
@@ -20,6 +20,7 @@
 #include <linux/bitops.h>
 #include <linux/dma-mapping.h>
 #include <linux/kvm_para.h>
+#include <linux/efi.h>
 
 #include <asm/tlbflush.h>
 #include <asm/fixmap.h>

@@ -48,6 +49,8 @@ EXPORT_SYMBOL_GPL(sev_enable_key);
 
 bool sev_enabled __section(".data");
 
+bool sev_live_migration_enabled __section(".data");
+
 /* Buffer used for early in-place encryption by BSP, no locking needed */
 static char sme_early_buffer[PAGE_SIZE] __initdata __aligned(PAGE_SIZE);
 

@@ -237,6 +240,9 @@ static void set_memory_enc_dec_hypercall(unsigned long vaddr, int npages,
        unsigned long sz = npages << PAGE_SHIFT;
        unsigned long vaddr_end, vaddr_next;
 
+       if (!sev_live_migration_enabled)
+               return;
+
        vaddr_end = vaddr + sz;
 
        for (; vaddr < vaddr_end; vaddr = vaddr_next) {

@@ -407,6 +413,12 @@ int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long size)
        return early_set_memory_enc_dec(vaddr, size, true);
 }
 
+void __init early_set_mem_enc_dec_hypercall(unsigned long vaddr, int npages,
+                                       bool enc)
+{
+       set_memory_enc_dec_hypercall(vaddr, npages, enc);
+}
+
 /*
  * SME and SEV are very similar but they are not the same, so there are
  * times that the kernel will need to distinguish between SME and SEV. The

@@ -462,6 +474,35 @@ bool force_dma_unencrypted(struct device *dev)
        return false;
 }
 
+void __init check_kvm_sev_migration(void)
+{
+       if (sev_active() &&
+           kvm_para_has_feature(KVM_FEATURE_SEV_LIVE_MIGRATION)) {
+               unsigned long nr_pages;
+
+               pr_info("KVM enable live migration\n");
+               sev_live_migration_enabled = true;
+
+               /*
+                * Ensure that _bss_decrypted section is marked as decrypted in the
+                * shared pages list.
+                */
+               nr_pages = DIV_ROUND_UP(__end_bss_decrypted - __start_bss_decrypted,
+                                       PAGE_SIZE);
+               early_set_mem_enc_dec_hypercall((unsigned long)__start_bss_decrypted,
+                                               nr_pages, 0);
+
+               /*
+                * If not booted using EFI, enable Live migration support.
+                */

arch/x86/mm/mem_encrypt.c bool force_dma_unencrypted(struct device *dev) · line 502

Quoted 5 lines

arch/x86/mm/mem_encrypt.c +5 -0

+               if (!efi_enabled(EFI_BOOT))
+                       wrmsrl(MSR_KVM_SEV_LIVE_MIGRATION,
+                              KVM_SEV_LIVE_MIGRATION_ENABLED);
+               } else {
+                       pr_info("KVM enable live migration feature unsupported\n");

I might be misunderstanding this, but I'm not sure this log message is
correct: isn't the intention that the late initcall will be the one to check if this should be enabled later in this case?

I have a similar question above about the log message after "!efi_enabled(EFI_RUNTIME_SERVICES)": shouldn't that avoid logging if !efi_enabled(EFI_BOOT) (since the wrmsl call already had been made here?)

Quoted 9 lines

arch/x86/mm/mem_encrypt.c +3 -1

+               }
+}
+
 void __init mem_encrypt_free_decrypted_mem(void)
 {
        unsigned long vaddr, vaddr_end, npages;
--

2.17.1

Other than these:

Reviewed-by Steve Rutherford <[email protected]>

From: Ashish Kalra <[email protected]>

Reset the host's shared pages list related to kernel specific page encryption status settings before we load a new kernel by kexec. We cannot reset the complete shared pages list here as we need to retain the UEFI/OVMF firmware specific settings.

The host's shared pages list is maintained for the guest to keep track of all unencrypted guest memory regions, therefore we need to explicitly mark all shared pages as encrypted again before rebooting into the new guest kernel.

Signed-off-by Ashish Kalra <[email protected]>

---
 arch/x86/kernel/kvm.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

arch/x86/kernel/kvm.c +1 -0

index bcc82e0c9779..4ad3ed547ff1 100644
@@ -39,6 +39,7 @@
 #include <asm/cpuidle_haltpoll.h>
 #include <asm/ptrace.h>
 #include <asm/svm.h>
+#include <asm/e820/api.h>
 
 DEFINE_STATIC_KEY_FALSE(kvm_async_pf_enabled);
 

@@ -384,6 +385,29 @@ static void kvm_pv_guest_cpu_reboot(void *unused)
         */

arch/x86/kernel/kvm.c +15 -1

+               int i;
+               unsigned long nr_pages;
+
+               for (i = 0; i < e820_table->nr_entries; i++) {
+                       struct e820_entry *entry = &e820_table->entries[i];
+
+                       if (entry->type != E820_TYPE_RAM)
+                               continue;
+
+                       nr_pages = DIV_ROUND_UP(entry->size, PAGE_SIZE);
+
+                       kvm_sev_hypercall3(KVM_HC_PAGE_ENC_STATUS,
+                                          entry->addr, nr_pages, 1);
+               }
+       }
        kvm_pv_disable_apf();
        kvm_disable_steal_time();
 }
--

2.17.1

Quoted 33 lines · Ashish Kalra

From: Ashish Kalra <[email protected]>

Reset the host's shared pages list related to kernel specific page encryption status settings before we load a new kernel by kexec. We cannot reset the complete shared pages list here as we need to retain the UEFI/OVMF firmware specific settings.

The host's shared pages list is maintained for the guest to keep track of all unencrypted guest memory regions, therefore we need to explicitly mark all shared pages as encrypted again before rebooting into the new guest kernel.

Signed-off-by Ashish Kalra <[email protected]>

---
 arch/x86/kernel/kvm.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

arch/x86/kernel/kvm.c +1 -0

index bcc82e0c9779..4ad3ed547ff1 100644
@@ -39,6 +39,7 @@
 #include <asm/cpuidle_haltpoll.h>
 #include <asm/ptrace.h>
 #include <asm/svm.h>
+#include <asm/e820/api.h>
 
 DEFINE_STATIC_KEY_FALSE(kvm_async_pf_enabled);
 

@@ -384,6 +385,29 @@ static void kvm_pv_guest_cpu_reboot(void *unused)
         */

arch/x86/kernel/kvm.c static void kvm_pv_guest_cpu_reboot(void *unused) · line 395

Quoted 10 lines

arch/x86/kernel/kvm.c +8 -0

        if (kvm_para_has_feature(KVM_FEATURE_PV_EOI))
                wrmsrl(MSR_KVM_PV_EOI_EN, 0);
+       /*
+        * Reset the host's shared pages list related to kernel
+        * specific page encryption status settings before we load a
+        * new kernel by kexec. NOTE: We cannot reset the complete
+        * shared pages list here as we need to retain the
+        * UEFI/OVMF firmware specific settings.
+        */
+       if (sev_live_migration_enabled & (smp_processor_id() == 0)) {

What happens if the reboot of CPU0 races with another CPU servicing a device request (while the reboot is pending for that CPU)? Seems like you could run into a scenario where you have hypercalls racing.

Calling this on every core isn't free, but it is an easy way to avoid this race. You could also count cores, and have only last core do the job, but that seems more complicated.

Steve

Quoted 13 lines

arch/x86/kernel/kvm.c +13 -0

+               int i;
+               unsigned long nr_pages;
+
+               for (i = 0; i < e820_table->nr_entries; i++) {
+                       struct e820_entry *entry = &e820_table->entries[i];
+
+                       if (entry->type != E820_TYPE_RAM)
+                               continue;
+
+                       nr_pages = DIV_ROUND_UP(entry->size, PAGE_SIZE);
+
+                       kvm_sev_hypercall3(KVM_HC_PAGE_ENC_STATUS,
+                                          entry->addr, nr_pages, 1);

Quoted 8 lines

arch/x86/kernel/kvm.c +2 -1

+               }
+       }
        kvm_pv_disable_apf();
        kvm_disable_steal_time();
 }
--

2.17.1

Quoted 49 lines · Steve Rutherford

Quoted 33 lines · Ashish Kalra

From: Ashish Kalra <[email protected]>

Reset the host's shared pages list related to kernel specific page encryption status settings before we load a new kernel by kexec. We cannot reset the complete shared pages list here as we need to retain the UEFI/OVMF firmware specific settings.

The host's shared pages list is maintained for the guest to keep track of all unencrypted guest memory regions, therefore we need to explicitly mark all shared pages as encrypted again before rebooting into the new guest kernel.

Signed-off-by Ashish Kalra <[email protected]>

---
 arch/x86/kernel/kvm.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

arch/x86/kernel/kvm.c +1 -0

index bcc82e0c9779..4ad3ed547ff1 100644
@@ -39,6 +39,7 @@
 #include <asm/cpuidle_haltpoll.h>
 #include <asm/ptrace.h>
 #include <asm/svm.h>
+#include <asm/e820/api.h>
 
 DEFINE_STATIC_KEY_FALSE(kvm_async_pf_enabled);
 

@@ -384,6 +385,29 @@ static void kvm_pv_guest_cpu_reboot(void *unused)
         */

arch/x86/kernel/kvm.c static void kvm_pv_guest_cpu_reboot(void *unused) · line 395

Quoted 10 lines

arch/x86/kernel/kvm.c +8 -0

        if (kvm_para_has_feature(KVM_FEATURE_PV_EOI))
                wrmsrl(MSR_KVM_PV_EOI_EN, 0);
+       /*
+        * Reset the host's shared pages list related to kernel
+        * specific page encryption status settings before we load a
+        * new kernel by kexec. NOTE: We cannot reset the complete
+        * shared pages list here as we need to retain the
+        * UEFI/OVMF firmware specific settings.
+        */
+       if (sev_live_migration_enabled & (smp_processor_id() == 0)) {

What happens if the reboot of CPU0 races with another CPU servicing a device request (while the reboot is pending for that CPU)? Seems like you could run into a scenario where you have hypercalls racing.

Quoted 3 lines

Calling this on every core isn't free, but it is an easy way to avoid this race. You could also count cores, and have only last core do the job, but that seems more complicated.

On second thought, I think this may be insufficient as a fix, since my read of kernel/reboot.c seems to imply that devices aren't shutdown until after these notifiers occur. As such, a single thread might be able to race with itself. I could be wrong here though.

The heavy hammer would be to disable migration through the MSR (which the subsequent boot will re-enable).

I'm curious if there is a less "blocking" way of handling kexecs (that strategy would block LM while the guest booted).

One option that comes to mind would be for the guest to "mute" the encryption status hypercall after the call to reset the encryption status. The problem would be that the encryption status for pages would be very temporarily inaccurate in the window between that call and the start of the next boot. That isn't ideal, but, on the other hand, the VM was about to reboot anyway, so a corrupted shared page for device communication probably isn't super important. Still, I'm not really a fan of that. This would avoid corrupting the next boot, which is clearly an improvement.

Each time the kernel boots it could also choose something like a generation ID, and pass that down each time it calls the hypercall. This would then let userspace identify which requests were coming from the subsequent boot.

Everything here (except, perhaps, disabling migration through the MSR) seems kind of complicated. I somewhat hope my interpretation of kernel/reboot.c is wrong and this race just is not possible in the first place.

Quoted 21 lines

Quoted 21 lines

arch/x86/kernel/kvm.c +15 -1

+               int i;
+               unsigned long nr_pages;
+
+               for (i = 0; i < e820_table->nr_entries; i++) {
+                       struct e820_entry *entry = &e820_table->entries[i];
+
+                       if (entry->type != E820_TYPE_RAM)
+                               continue;
+
+                       nr_pages = DIV_ROUND_UP(entry->size, PAGE_SIZE);
+
+                       kvm_sev_hypercall3(KVM_HC_PAGE_ENC_STATUS,
+                                          entry->addr, nr_pages, 1);
+               }
+       }
        kvm_pv_disable_apf();
        kvm_disable_steal_time();
 }
--

2.17.1

Quoted 79 lines · Steve Rutherford

Quoted 49 lines · Steve Rutherford

Quoted 33 lines · Ashish Kalra

From: Ashish Kalra <[email protected]>

Reset the host's shared pages list related to kernel specific page encryption status settings before we load a new kernel by kexec. We cannot reset the complete shared pages list here as we need to retain the UEFI/OVMF firmware specific settings.

The host's shared pages list is maintained for the guest to keep track of all unencrypted guest memory regions, therefore we need to explicitly mark all shared pages as encrypted again before rebooting into the new guest kernel.

Signed-off-by Ashish Kalra <[email protected]>

---
 arch/x86/kernel/kvm.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

arch/x86/kernel/kvm.c +1 -0

index bcc82e0c9779..4ad3ed547ff1 100644
@@ -39,6 +39,7 @@
 #include <asm/cpuidle_haltpoll.h>
 #include <asm/ptrace.h>
 #include <asm/svm.h>
+#include <asm/e820/api.h>
 
 DEFINE_STATIC_KEY_FALSE(kvm_async_pf_enabled);
 

@@ -384,6 +385,29 @@ static void kvm_pv_guest_cpu_reboot(void *unused)
         */

arch/x86/kernel/kvm.c static void kvm_pv_guest_cpu_reboot(void *unused) · line 395

Quoted 10 lines

arch/x86/kernel/kvm.c +8 -0

        if (kvm_para_has_feature(KVM_FEATURE_PV_EOI))
                wrmsrl(MSR_KVM_PV_EOI_EN, 0);
+       /*
+        * Reset the host's shared pages list related to kernel
+        * specific page encryption status settings before we load a
+        * new kernel by kexec. NOTE: We cannot reset the complete
+        * shared pages list here as we need to retain the
+        * UEFI/OVMF firmware specific settings.
+        */
+       if (sev_live_migration_enabled & (smp_processor_id() == 0)) {

What happens if the reboot of CPU0 races with another CPU servicing a device request (while the reboot is pending for that CPU)? Seems like you could run into a scenario where you have hypercalls racing.

Quoted 3 lines

Calling this on every core isn't free, but it is an easy way to avoid this race. You could also count cores, and have only last core do the job, but that seems more complicated.

On second thought, I think this may be insufficient as a fix, since my read of kernel/reboot.c seems to imply that devices aren't shutdown until after these notifiers occur. As such, a single thread might be able to race with itself. I could be wrong here though.

The heavy hammer would be to disable migration through the MSR (which the subsequent boot will re-enable).

I'm curious if there is a less "blocking" way of handling kexecs (that strategy would block LM while the guest booted).

One option that comes to mind would be for the guest to "mute" the encryption status hypercall after the call to reset the encryption status. The problem would be that the encryption status for pages would be very temporarily inaccurate in the window between that call and the start of the next boot. That isn't ideal, but, on the other hand, the VM was about to reboot anyway, so a corrupted shared page for device communication probably isn't super important. Still, I'm not really a fan of that. This would avoid corrupting the next boot, which is clearly an improvement.

Each time the kernel boots it could also choose something like a generation ID, and pass that down each time it calls the hypercall. This would then let userspace identify which requests were coming from the subsequent boot.

Quoted 5 lines

Everything here (except, perhaps, disabling migration through the MSR) seems kind of complicated. I somewhat hope my interpretation of kernel/reboot.c is wrong and this race just is not possible in the first place.

Disabling migration through the MSR after resetting the page encryption status is a reasonable approach. There is a similar window existing for normal VM boot during which LM is disabled, from the point where OVMF checks and adds support for SEV LM and the kernel boot checks for the same and enables LM using the MSR.

Thanks,
Ashish

Quoted 13 lines

Quoted 13 lines

Quoted 13 lines

arch/x86/kernel/kvm.c +13 -0

+               int i;
+               unsigned long nr_pages;
+
+               for (i = 0; i < e820_table->nr_entries; i++) {
+                       struct e820_entry *entry = &e820_table->entries[i];
+
+                       if (entry->type != E820_TYPE_RAM)
+                               continue;
+
+                       nr_pages = DIV_ROUND_UP(entry->size, PAGE_SIZE);
+
+                       kvm_sev_hypercall3(KVM_HC_PAGE_ENC_STATUS,
+                                          entry->addr, nr_pages, 1);

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

arch/x86/kernel/kvm.c +2 -1

+               }
+       }
        kvm_pv_disable_apf();
        kvm_disable_steal_time();
 }
--

2.17.1

Quoted 87 lines · Ashish Kalra

Quoted 79 lines · Steve Rutherford

Quoted 49 lines · Steve Rutherford

Quoted 33 lines · Ashish Kalra

From: Ashish Kalra <[email protected]>

Reset the host's shared pages list related to kernel specific page encryption status settings before we load a new kernel by kexec. We cannot reset the complete shared pages list here as we need to retain the UEFI/OVMF firmware specific settings.

The host's shared pages list is maintained for the guest to keep track of all unencrypted guest memory regions, therefore we need to explicitly mark all shared pages as encrypted again before rebooting into the new guest kernel.

Signed-off-by Ashish Kalra <[email protected]>

---
 arch/x86/kernel/kvm.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

arch/x86/kernel/kvm.c +1 -0

index bcc82e0c9779..4ad3ed547ff1 100644
@@ -39,6 +39,7 @@
 #include <asm/cpuidle_haltpoll.h>
 #include <asm/ptrace.h>
 #include <asm/svm.h>
+#include <asm/e820/api.h>
 
 DEFINE_STATIC_KEY_FALSE(kvm_async_pf_enabled);
 

@@ -384,6 +385,29 @@ static void kvm_pv_guest_cpu_reboot(void *unused)
         */

arch/x86/kernel/kvm.c static void kvm_pv_guest_cpu_reboot(void *unused) · line 395

Quoted 10 lines

arch/x86/kernel/kvm.c +8 -0

        if (kvm_para_has_feature(KVM_FEATURE_PV_EOI))
                wrmsrl(MSR_KVM_PV_EOI_EN, 0);
+       /*
+        * Reset the host's shared pages list related to kernel
+        * specific page encryption status settings before we load a
+        * new kernel by kexec. NOTE: We cannot reset the complete
+        * shared pages list here as we need to retain the
+        * UEFI/OVMF firmware specific settings.
+        */
+       if (sev_live_migration_enabled & (smp_processor_id() == 0)) {

What happens if the reboot of CPU0 races with another CPU servicing a device request (while the reboot is pending for that CPU)? Seems like you could run into a scenario where you have hypercalls racing.

Quoted 3 lines

Calling this on every core isn't free, but it is an easy way to avoid this race. You could also count cores, and have only last core do the job, but that seems more complicated.

On second thought, I think this may be insufficient as a fix, since my read of kernel/reboot.c seems to imply that devices aren't shutdown until after these notifiers occur. As such, a single thread might be able to race with itself. I could be wrong here though.

The heavy hammer would be to disable migration through the MSR (which the subsequent boot will re-enable).

I'm curious if there is a less "blocking" way of handling kexecs (that strategy would block LM while the guest booted).

One option that comes to mind would be for the guest to "mute" the encryption status hypercall after the call to reset the encryption status. The problem would be that the encryption status for pages would be very temporarily inaccurate in the window between that call and the start of the next boot. That isn't ideal, but, on the other hand, the VM was about to reboot anyway, so a corrupted shared page for device communication probably isn't super important. Still, I'm not really a fan of that. This would avoid corrupting the next boot, which is clearly an improvement.

Each time the kernel boots it could also choose something like a generation ID, and pass that down each time it calls the hypercall. This would then let userspace identify which requests were coming from the subsequent boot.

Quoted 5 lines

Everything here (except, perhaps, disabling migration through the MSR) seems kind of complicated. I somewhat hope my interpretation of kernel/reboot.c is wrong and this race just is not possible in the first place.

Quoted 5 lines

Disabling migration through the MSR after resetting the page encryption status is a reasonable approach. There is a similar window existing for normal VM boot during which LM is disabled, from the point where OVMF checks and adds support for SEV LM and the kernel boot checks for the same and enables LM using the MSR.

I'm not totally confident that disabling LM through the MSR is sufficient. I also think the newly booted kernel needs to reset the state itself, since nothing stops the hypercalls after the disable goes through. The host won't know the difference between early boot (pre-enablement) hypercalls and racy just-before-restart hypercalls. You might disable migration through the hypercall, get a late status change hypercall, reboot, then re-enable migration, but still have stale state.

I _believe_ that the kernel doesn't mark it's RAM as private on boot as an optimization (might be wrong about this), since it would have been expensive to mark all of ram as encrypted previously. I believe that is no longer a limitation given the KVM_EXIT, so we can reset this during early boot instead of just before the kexec.

Quoted 21 lines

Quoted 21 lines

Quoted 21 lines

Quoted 21 lines

arch/x86/kernel/kvm.c +15 -1

+               int i;
+               unsigned long nr_pages;
+
+               for (i = 0; i < e820_table->nr_entries; i++) {
+                       struct e820_entry *entry = &e820_table->entries[i];
+
+                       if (entry->type != E820_TYPE_RAM)
+                               continue;
+
+                       nr_pages = DIV_ROUND_UP(entry->size, PAGE_SIZE);
+
+                       kvm_sev_hypercall3(KVM_HC_PAGE_ENC_STATUS,
+                                          entry->addr, nr_pages, 1);
+               }
+       }
        kvm_pv_disable_apf();
        kvm_disable_steal_time();
 }
--

2.17.1

Quoted 103 lines · Steve Rutherford

Quoted 87 lines · Ashish Kalra

Quoted 79 lines · Steve Rutherford

Quoted 49 lines · Steve Rutherford

Quoted 33 lines · Ashish Kalra

From: Ashish Kalra <[email protected]>

Reset the host's shared pages list related to kernel specific page encryption status settings before we load a new kernel by kexec. We cannot reset the complete shared pages list here as we need to retain the UEFI/OVMF firmware specific settings.

The host's shared pages list is maintained for the guest to keep track of all unencrypted guest memory regions, therefore we need to explicitly mark all shared pages as encrypted again before rebooting into the new guest kernel.

Signed-off-by Ashish Kalra <[email protected]>

---
 arch/x86/kernel/kvm.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

arch/x86/kernel/kvm.c +1 -0

index bcc82e0c9779..4ad3ed547ff1 100644
@@ -39,6 +39,7 @@
 #include <asm/cpuidle_haltpoll.h>
 #include <asm/ptrace.h>
 #include <asm/svm.h>
+#include <asm/e820/api.h>
 
 DEFINE_STATIC_KEY_FALSE(kvm_async_pf_enabled);
 

@@ -384,6 +385,29 @@ static void kvm_pv_guest_cpu_reboot(void *unused)
         */

arch/x86/kernel/kvm.c static void kvm_pv_guest_cpu_reboot(void *unused) · line 395

Quoted 10 lines

arch/x86/kernel/kvm.c +8 -0

        if (kvm_para_has_feature(KVM_FEATURE_PV_EOI))
                wrmsrl(MSR_KVM_PV_EOI_EN, 0);
+       /*
+        * Reset the host's shared pages list related to kernel
+        * specific page encryption status settings before we load a
+        * new kernel by kexec. NOTE: We cannot reset the complete
+        * shared pages list here as we need to retain the
+        * UEFI/OVMF firmware specific settings.
+        */
+       if (sev_live_migration_enabled & (smp_processor_id() == 0)) {

What happens if the reboot of CPU0 races with another CPU servicing a device request (while the reboot is pending for that CPU)? Seems like you could run into a scenario where you have hypercalls racing.

Quoted 3 lines

Calling this on every core isn't free, but it is an easy way to avoid this race. You could also count cores, and have only last core do the job, but that seems more complicated.

On second thought, I think this may be insufficient as a fix, since my read of kernel/reboot.c seems to imply that devices aren't shutdown until after these notifiers occur. As such, a single thread might be able to race with itself. I could be wrong here though.

The heavy hammer would be to disable migration through the MSR (which the subsequent boot will re-enable).

I'm curious if there is a less "blocking" way of handling kexecs (that strategy would block LM while the guest booted).

One option that comes to mind would be for the guest to "mute" the encryption status hypercall after the call to reset the encryption status. The problem would be that the encryption status for pages would be very temporarily inaccurate in the window between that call and the start of the next boot. That isn't ideal, but, on the other hand, the VM was about to reboot anyway, so a corrupted shared page for device communication probably isn't super important. Still, I'm not really a fan of that. This would avoid corrupting the next boot, which is clearly an improvement.

Each time the kernel boots it could also choose something like a generation ID, and pass that down each time it calls the hypercall. This would then let userspace identify which requests were coming from the subsequent boot.

Quoted 5 lines

Everything here (except, perhaps, disabling migration through the MSR) seems kind of complicated. I somewhat hope my interpretation of kernel/reboot.c is wrong and this race just is not possible in the first place.

Quoted 5 lines

Disabling migration through the MSR after resetting the page encryption status is a reasonable approach. There is a similar window existing for normal VM boot during which LM is disabled, from the point where OVMF checks and adds support for SEV LM and the kernel boot checks for the same and enables LM using the MSR.

I'm not totally confident that disabling LM through the MSR is sufficient. I also think the newly booted kernel needs to reset the state itself, since nothing stops the hypercalls after the disable goes through. The host won't know the difference between early boot (pre-enablement) hypercalls and racy just-before-restart hypercalls. You might disable migration through the hypercall, get a late status change hypercall, reboot, then re-enable migration, but still have stale state.

Quoted 6 lines

I _believe_ that the kernel doesn't mark it's RAM as private on boot as an optimization (might be wrong about this), since it would have been expensive to mark all of ram as encrypted previously. I believe that is no longer a limitation given the KVM_EXIT, so we can reset this during early boot instead of just before the kexec.

I was wondering if disabling both migration (via the MSR) and "muting" the hypercall using the "sev_live_migration_enabled" variable after the page encryption status has been reset, will reset the page encryption status of the guest to the (last known/good) configuration available to the guest at boot time (i.e, all RAM pages marked as private and UEFI setup shared MMIO/device regions, etc).

But disabling migration and muting hypercalls after page encryption status reset is still "racy" with hypercalls on other vCPUS, and that can potentially mess-up the page encryption status available to guest after kexec.

So probably, as you mentioned above, resetting the page encryption status during early boot (immediately after detecting host support for migration and enabling the hypercalls) instead of just before the kexec is a good fix.

Thanks,
Ashish

Quoted 13 lines

Quoted 13 lines

Quoted 13 lines

Quoted 13 lines

Quoted 13 lines

arch/x86/kernel/kvm.c +13 -0

+               int i;
+               unsigned long nr_pages;
+
+               for (i = 0; i < e820_table->nr_entries; i++) {
+                       struct e820_entry *entry = &e820_table->entries[i];
+
+                       if (entry->type != E820_TYPE_RAM)
+                               continue;
+
+                       nr_pages = DIV_ROUND_UP(entry->size, PAGE_SIZE);
+
+                       kvm_sev_hypercall3(KVM_HC_PAGE_ENC_STATUS,
+                                          entry->addr, nr_pages, 1);

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

arch/x86/kernel/kvm.c +2 -1

+               }
+       }
        kvm_pv_disable_apf();
        kvm_disable_steal_time();
 }
--

2.17.1