Skip to contentSkip to conversation

Add AMD Secure Nested Paging (SEV-SNP) Hypervisor Support

RFC PATCH Superseded Michael Roth 22 participants 205 messages Dec 14, 2022 – May 15, 2023 Page 1 of 2

Subsystems (12)

v7 of 15 Superseded Latest is v15 →

lore
Share & tools
lore
#0 lore raw

v7 cover Add AMD Secure Nested Paging (SEV-SNP) Hypervisor Support

This patchset is based on top of the following patchset:

"[PATCH v10 0/9] KVM: mm: fd-based approach for supporting KVM"
https://lore.kernel.org/lkml/[email protected]/T/#me1dd3a4c295758b4e4ac8ff600f2db055bc5f987

and is also available at:

https://github.com/amdese/linux/commits/upmv10-host-snp-v7-rfc

== OVERVIEW ==

This version is being posted as an RFC due to fairly extensive changes relating to transitioning the SEV-SNP implementation to using restricted/private memslots (aka Unmapped Private Memory) to manage private guest pages instead of the legacy SEV memory registration ioctls.

Alongside that work we've also been investigating leveraging UPM to to implement lazy-pinning support for SEV guests, rather than the legacy SEV memory registration ioctls which rely on pinning everything in advance.

For both of these SEV and SEV-SNP use-cases we've needed to add a number of hooks in the restricted, so we thought it would be useful for this version at least to include both UPM-based SEV and SNP implementations so can see if these hooks might be needed for other archs/platforms and start consolidating around whether/how they should be defined for general usage. There are still some TODOs in this area, but we hope this implementation is complete enough to at least outline the required additions needed for using UPM for these use-cases.

Outside of UPM-related items, we've also included fairly extensive changes based on review feedback from v6 and would appreciate any feedback on those aspects as well.

== LAYOUT ==

PATCH 01-06: pre-patches that add the UPM hooks and KVM capability needed
to switch between UPM and legacy SEV memory registration.
PATCH 07-12: implement SEV lazy-pinning using UPM to manage private memory PATCH 13-32: general SNP detection/enablement for host and CCP driver PATCH 33-58: base KVM-specific support for running SNP guests PATCH 59-64: misc./documentation/IOMMU changes

== TESTING ==

Tested with the following QEMU tree, which is based on Chao Peng's UPM v10 QEMU tree:
https://github.com/mdroth/qemu/commits/upmv10-snpv3

SEV-SNP with UPM:

  qemu-system-x86_64 -cpu EPYC-Milan-v2 \
    -object memory-backend-memfd-private,id=ram1,size=1G,share=true \
    -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,upm-mode=on \
    -machine q35,confidential-guest-support=sev0,memory-backend=ram1 \
    ...

SEV with UPM (requires patched OVMF[1]):

  qemu-system-x86_64 -cpu EPYC-Milan-v2 \
    -object memory-backend-memfd-private,id=ram1,size=1G,share=true \
    -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,upm-mode=on \
    -machine q35,confidential-guest-support=sev0,memory-backend=ram1 \
    ...

[1] https://github.com/mdroth/edk2/commits/upmv8-seves-v1

== BACKGROUND ==

This part of the Secure Encrypted Paging (SEV-SNP) series focuses on the changes required in a host OS for SEV-SNP support. The series builds upon SEV-SNP Guest Support now part of mainline.

This series provides the basic building blocks to support booting the SEV-SNP VMs, it does not cover all the security enhancement introduced by the SEV-SNP such as interrupt protection.

The CCP driver is enhanced to provide new APIs that use the SEV-SNP specific commands defined in the SEV-SNP firmware specification. The KVM driver uses those APIs to create and managed the SEV-SNP guests.

The GHCB specification version 2 introduces new set of NAE's that is used by the SEV-SNP guest to communicate with the hypervisor. The series provides support to handle the following new NAE events:
- Register GHCB GPA
- Page State Change Request
- Hypevisor feature
- Guest message request

The RMP check is enforced as soon as SEV-SNP is enabled. Not every memory access requires an RMP check. In particular, the read accesses from the hypervisor do not require RMP checks because the data confidentiality is already protected via memory encryption. When hardware encounters an RMP checks failure, it raises a page-fault exception. If RMP check failure is due to the page-size mismatch, then split the large page to resolve the fault.

The series does not provide support for the interrupt security and migration and those feature will be added after the base support.

Changes since v6:

* Added support for restrictedmem/UPM, and removed SEV-specific implementation of private memory management. As a result of this rework the following patches were no longer needed so were dropped:
- KVM: SVM: Mark the private vma unmergable for SEV-SNP guests
- KVM: SVM: Disallow registering memory range from HugeTLB for SNP guest
- KVM: x86/mmu: Introduce kvm_mmu_map_tdp_page() for use by TDX and SNP
- KVM: x86: Introduce kvm_mmu_get_tdp_walk() for SEV-SNP use
* Moved RMP table entry structure definition (struct rmpentry) to sev.c, to not expose this non-architectural definition to rest of the kernel and making the structure private to SNP code. Also made RMP table entry accessors to be inline functions and removed all accessors which are not called more than once. Added a new function rmptable_entry() to index into the RMP table and return RMP table entry.
* Moved RMPUPDATE, PSMASH helper function declerations to x86 arch specific include namespace from linux namespace. Added comments for these helper functions.
* Introduce set_memory_p() to provide a way to change atributes of a memory range to be marked as present and added to the kernel directmap, and invalidating/restoring pages from directmap are now done using set_memory_np() and set_memory_p().
* Added detailed comments around user RMP #PF fault handling and simplified computation of the faulting pfn for large-pages.
* Added support to return pfn from dump_pagetable() to do SEV-specific fault handling, this is added a pre-patch. This support is now used to dump RMP entry in case of RMP #PF in show_fault_oops().
* Added a new generic SNP command params structure sev_data_snp_addr, which is used for all SNP firmware API commands requiring a single physical address parameter.
* Added support for new SNP_INIT_EX command with support for HV-Fixed page range list.
* Added support for new SNP_SHUTDOWN_EX command which allows disabling enforcement of SNP in the IOMMU. Also DF_FLUSH is done at SNP shutdown if it indicates DF_FLUSH is required.
* Make sev_do_cmd() a generic API interface for the hypervisor to issue commands to manage an SEV and SNP guest. Also removed the API wrappers used by the hypervisor to manage an SEV-SNP guest. All these APIs now invoke sev_do_cmd() directly.
* Introduce snp leaked pages list. If pages are unsafe to be released back to the page-allocator as they can't be reclaimed or transitioned back to hypervisor/shared state are now added
to this internal leaked pages list to prevent fatal page faults when accessing these pages. The function snp_leak_pages() is renamed to snp_mark_pages_offline() and is an external function available to both CCP driver and the SNP hypervisor code. Removed call to memory_failure() when leaking/marking pages offline.
* Remove snp_set_rmp_state() multiplexor code and add new separate helpers such as rmp_mark_pages_firmware() & rmp_mark_pages_shared(). The callers now issue snp_reclaim_pages() directly when needed as done by __snp_free_firmware_pages() and unmap_firmware_writeable(). All callers of snp_set_rmp_state() modified to call helpers rmp_mark_pages_firmware() or rmp_mark_pages_shared() as required.
* Change snp_reclaim_pages() to take physical address as an argument and clear C-bit from this physical address argument internally.
* Output parameter sev_user_data_ext_snp_config in sev_ioctl_snp_get_config() is memset to zero to avoid kernel memory leaking.
* Prevent race between sev_ioctl_snp_set_config() and snp_guest_ext_guest_request() for sev->snp_certs_data by acquiring sev->snp_certs_lock mutex.
* Zeroed out struct sev_user_data_snp_config in sev_ioctl_snp_set_config() to prevent leaking uninitialized kernel memory.
* Optimized snp_safe_alloc_page() by avoiding multiple calls to pfn_to_page() and checking for a hugepage using pfn instead of expanding to full physical address.
* Invoke host_rmp_make_shared() with leak parameter set to true if VMSA page cannot be transitioned back to shared state.
* Fix snp_launch_finish() to always sent the ID_AUTH struct to the firmware. Use params.auth_key_en indicator to set
if the ID_AUTH struct contains an author key or not.
* Cleanup snp_context_create() and allocate certs_data in this function using kzalloc() to prevent giving the guest uninitialized kernel memory.
* Remove the check for guest supplied buffer greater than the data provided by the hypervisor in snp_handle_ext_guest_request().
* Add check in sev_snp_ap_create() if a malicious guest can RMPADJUST a large page into VMSA which will hit the SNP erratum where the CPU will incorrectly signal an RMP violation #PF if a hugepage collides with the RMP entry of VMSA page, reject the AP CREATE request if VMSA address from guest is 2M aligned.
* Make VMSAVE target area memory allocation SNP safe, implemented workaround for an SNP erratum where the CPU will incorrectly signal an RMP violation #PF if a hugepage (2mb or 1gb) collides with the RMP entry of the VMSAVE target page.
* Fix handle_split_page_fault() to work with memfd backed pages.
* Add KVM commands for per-VM instance certificates.
* Add IOMMU_SNP_SHUTDOWN support, this adds support for Host kexec support with SNP.

----------------------------------------------------------------
Ashish Kalra (6):
x86/mm/pat: Introduce set_memory_p
x86/fault: Return pfn from dump_pagetable() for SEV-specific fault handling.
crypto: ccp: Introduce snp leaked pages list
KVM: SVM: Sync the GHCB scratch buffer using already mapped ghcb
KVM: SVM: Make VMSAVE target area memory allocation SNP safe iommu/amd: Add IOMMU_SNP_SHUTDOWN support

Brijesh Singh (36):
x86/cpufeatures: Add SEV-SNP CPU feature
x86/sev: Add the host SEV-SNP initialization support
x86/sev: Add RMP entry lookup helpers
x86/sev: Add helper functions for RMPUPDATE and PSMASH instruction x86/sev: Invalidate pages from the direct map when adding them to the RMP table x86/traps: Define RMP violation #PF error code
x86/fault: Add support to handle the RMP fault for user address x86/fault: Add support to dump RMP entry on fault
crypto:ccp: Define the SEV-SNP commands
crypto: ccp: Add support to initialize the AMD-SP for SEV-SNP crypto:ccp: Provide API to issue SEV and SNP commands
crypto: ccp: Handle the legacy TMR allocation when SNP is enabled
crypto: ccp: Handle the legacy SEV command when SNP is enabled
crypto: ccp: Add the SNP_PLATFORM_STATUS command
crypto: ccp: Add the SNP_{SET,GET}_EXT_CONFIG command
crypto: ccp: Provide APIs to query extended attestation report
KVM: SVM: Provide the Hypervisor Feature support VMGEXIT
KVM: SVM: Make AVIC backing, VMSA and VMCB memory allocation SNP safe
KVM: SVM: Add initial SEV-SNP support
KVM: SVM: Add KVM_SNP_INIT command
KVM: SVM: Add KVM_SEV_SNP_LAUNCH_START command
KVM: SVM: Add KVM_SEV_SNP_LAUNCH_UPDATE command
KVM: SVM: Add KVM_SEV_SNP_LAUNCH_FINISH command
KVM: X86: Keep the NPT and RMP page level in sync
KVM: x86: Define RMP page fault error bits for #NPF
KVM: SVM: Do not use long-lived GHCB map while setting scratch area
KVM: SVM: Remove the long-lived GHCB host map
KVM: SVM: Add support to handle GHCB GPA register VMGEXIT
KVM: SVM: Add support to handle MSR based Page State Change VMGEXIT
KVM: SVM: Add support to handle Page State Change VMGEXIT
KVM: SVM: Introduce ops for the post gfn map and unmap
KVM: x86: Export the kvm_zap_gfn_range() for the SNP use
KVM: SVM: Add support to handle the RMP nested page fault
KVM: SVM: Provide support for SNP_GUEST_REQUEST NAE event
KVM: SVM: Add module parameter to enable the SEV-SNP
ccp: Add support to decrypt the page

Dionna Glaze (2):
x86/sev: Add KVM commands for instance certs x86/sev: Document KVM_SEV_SNP_{G,S}ET_CERTS

Hugh Dickins (1):
x86/fault: fix handle_split_page_fault() to work with memfd backed pages

Michael Roth (9):
KVM: x86: Add KVM_CAP_UNMAPPED_PRIVATE_MEMORY
KVM: x86: Add 'fault_is_private' x86 op
KVM: x86: Add 'update_mem_attr' x86 op
KVM: x86: Add platform hooks for private memory invalidations
KVM: SEV: Implement .fault_is_private callback
KVM: SVM: Add KVM_EXIT_VMGEXIT
KVM: SVM: Add SNP-specific handling for memory attribute updates
KVM: x86/mmu: Generate KVM_EXIT_MEMORY_FAULT for implicit conversions for SNP
KVM: SEV: Handle restricted memory invalidations for SNP

Nikunj A Dadhania (5):
KVM: Fix memslot boundary condition for large page
KVM: SVM: Advertise private memory support to KVM
KVM: SEV: Handle KVM_HC_MAP_GPA_RANGE hypercall
KVM: Move kvm_for_each_memslot_in_hva_range() to be used in SVM
KVM: SEV: Support private pages in LAUNCH_UPDATE_DATA

Tom Lendacky (3):
KVM: SVM: Add support to handle AP reset MSR protocol
KVM: SVM: Use a VMSA physical address variable for populating VMCB
KVM: SVM: Support SEV-SNP AP Creation NAE event

Vishal Annapurve (2):
KVM: Add HVA range operator
KVM: SEV: Populate private memory fd during LAUNCH_UPDATE_DATA

 Documentation/virt/coco/sev-guest.rst              |   54 +
 .../virt/kvm/x86/amd-memory-encryption.rst         |  146 ++
 arch/x86/include/asm/cpufeatures.h                 |    1 +
 arch/x86/include/asm/disabled-features.h           |    8 +-
 arch/x86/include/asm/kvm-x86-ops.h                 |    6 +
 arch/x86/include/asm/kvm_host.h                    |   23 +
 arch/x86/include/asm/msr-index.h                   |   11 +-
 arch/x86/include/asm/set_memory.h                  |    3 +-
 arch/x86/include/asm/sev-common.h                  |   28 +
 arch/x86/include/asm/sev.h                         |   28 +
 arch/x86/include/asm/svm.h                         |    6 +
 arch/x86/include/asm/trap_pf.h                     |   18 +-
 arch/x86/kernel/cpu/amd.c                          |    5 +-
 arch/x86/kernel/sev.c                              |  437 ++++
 arch/x86/kvm/lapic.c                               |    5 +-
 arch/x86/kvm/mmu.h                                 |    2 -
 arch/x86/kvm/mmu/mmu.c                             |   34 +-
 arch/x86/kvm/mmu/mmu_internal.h                    |   40 +-
 arch/x86/kvm/svm/sev.c                             | 2217 +++++++++++++++++---
 arch/x86/kvm/svm/svm.c                             |   84 +-
 arch/x86/kvm/svm/svm.h                             |   75 +-
 arch/x86/kvm/trace.h                               |   34 +
 arch/x86/kvm/x86.c                                 |   36 +
 arch/x86/mm/fault.c                                |  118 +-
 arch/x86/mm/pat/set_memory.c                       |   12 +-
 drivers/crypto/ccp/sev-dev.c                       | 1055 +++++++++-
 drivers/crypto/ccp/sev-dev.h                       |   18 +
 drivers/iommu/amd/init.c                           |   53 +
 include/linux/amd-iommu.h                          |    1 +
 include/linux/kvm_host.h                           |   15 +
 include/linux/mm.h                                 |    3 +-
 include/linux/mm_types.h                           |    3 +
 include/linux/psp-sev.h                            |  352 +++-
 include/uapi/linux/kvm.h                           |   75 +
 include/uapi/linux/psp-sev.h                       |   60 +
 mm/memory.c                                        |   15 +
 mm/restrictedmem.c                                 |   16 +
 tools/arch/x86/include/asm/cpufeatures.h           |    1 +
 virt/kvm/kvm_main.c                                |   87 +-
 39 files changed, 4838 insertions(+), 347 deletions(-)
#1 lore raw

RFC PATCH v7 1/64 KVM: Fix memslot boundary condition for large page

From: Nikunj A Dadhania <[email protected]>

Aligned end boundary causes a kvm crash, handle the case.

---
 arch/x86/kvm/mmu/mmu.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kvm/mmu/mmu.c +3 -0
index b1953ebc012e..b3ffc61c668c 100644
@@ -7159,6 +7159,9 @@ static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm,
 		for (gfn = first + pages; gfn < last; gfn += pages)
 			linfo_set_mixed(gfn, slot, level, false);
 
+		if (gfn == last)
+			goto out;
+
 		gfn = last;
 		gfn_end = min(last + pages, slot->base_gfn + slot->npages);
 		mixed = mem_attrs_mixed(kvm, slot, level, attrs, gfn, gfn_end);
Signature · 1 line
2.25.1
maintainer
#2 lore raw

Re: [PATCH RFC v7 01/64] KVM: Fix memslot boundary condition for large page

Quoted 15 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

Aligned end boundary causes a kvm crash, handle the case.

---
 arch/x86/kvm/mmu/mmu.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kvm/mmu/mmu.c +0 -0
index b1953ebc012e..b3ffc61c668c 100644

arch/x86/kvm/mmu/mmu.c static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm, · line 7163

Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +2 -0
@@ -7159,6 +7159,9 @@ static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm,
 		for (gfn = first + pages; gfn < last; gfn += pages)
 			linfo_set_mixed(gfn, slot, level, false);
 
+		if (gfn == last)
+			goto out;

I'm guessing this was supposed to be "return;" here:

arch/x86/kvm/mmu/mmu.c: In function ‘kvm_update_lpage_private_shared_mixed’:
arch/x86/kvm/mmu/mmu.c:7090:25: error: label ‘out’ used but not defined

 7090 |                         goto out;
      |                         ^~~~

/me goes and digs deeper.

Aha, it was a "return" but you reordered the patches and the one adding the out label:

KVM: x86: Add 'update_mem_attr' x86 op

went further down and this became the first but it didn't have the label anymore.

Yeah, each patch needs to build successfully for bisection reasons, ofc.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#3 lore raw

Re: [PATCH RFC v7 01/64] KVM: Fix memslot boundary condition for large page

Quoted 19 lines · Borislav Petkov
Quoted 18 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

Aligned end boundary causes a kvm crash, handle the case.

---
 arch/x86/kvm/mmu/mmu.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kvm/mmu/mmu.c +0 -0
index b1953ebc012e..b3ffc61c668c 100644
@@ -7159,6 +7159,9 @@ static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm,
 		for (gfn = first + pages; gfn < last; gfn += pages)
 			linfo_set_mixed(gfn, slot, level, false);
 
Quoted 4 lines

arch/x86/kvm/mmu/mmu.c static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm, · line 7163

Quoted 2 lines
arch/x86/kvm/mmu/mmu.c +2 -0
+		if (gfn == last)
+			goto out;

I'm guessing this was supposed to be "return;" here:

If we finally need this, this should be "continue;", we can't skip the remaining huge page levels.

Thanks,
Chao

Quoted 25 lines

arch/x86/kvm/mmu/mmu.c: In function ‘kvm_update_lpage_private_shared_mixed’:
arch/x86/kvm/mmu/mmu.c:7090:25: error: label ‘out’ used but not defined

 7090 |                         goto out;
      |                         ^~~~

/me goes and digs deeper.

Aha, it was a "return" but you reordered the patches and the one adding the out label:

KVM: x86: Add 'update_mem_attr' x86 op

went further down and this became the first but it didn't have the label anymore.

Yeah, each patch needs to build successfully for bisection reasons, ofc.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#4 lore raw

Re: [PATCH RFC v7 01/64] KVM: Fix memslot boundary condition for large page

In reply to Michael Roth
Quoted 4 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

Aligned end boundary causes a kvm crash, handle the case.

Chao, are you aware of this issue already?

Quoted 23 lines
---
 arch/x86/kvm/mmu/mmu.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kvm/mmu/mmu.c +3 -0
index b1953ebc012e..b3ffc61c668c 100644
@@ -7159,6 +7159,9 @@ static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm,
 		for (gfn = first + pages; gfn < last; gfn += pages)
 			linfo_set_mixed(gfn, slot, level, false);
 
+		if (gfn == last)
+			goto out;
+
 		gfn = last;
 		gfn_end = min(last + pages, slot->base_gfn + slot->npages);
 		mixed = mem_attrs_mixed(kvm, slot, level, attrs, gfn, gfn_end);
Signature · 1 line
2.25.1

BR, Jarkko

#5 lore raw

Re: [PATCH RFC v7 01/64] KVM: Fix memslot boundary condition for large page

Quoted 9 lines · Jarkko Sakkinen
Quoted 4 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

Aligned end boundary causes a kvm crash, handle the case.

Chao, are you aware of this issue already?

Thanks Jarkko adding me. I'm not aware of there is a fix.

Quoted 15 lines
Quoted 15 lines
---
 arch/x86/kvm/mmu/mmu.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kvm/mmu/mmu.c +0 -0
index b1953ebc012e..b3ffc61c668c 100644
@@ -7159,6 +7159,9 @@ static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm,
 		for (gfn = first + pages; gfn < last; gfn += pages)
 			linfo_set_mixed(gfn, slot, level, false);
 
Quoted 3 lines

arch/x86/kvm/mmu/mmu.c static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm, · line 7164

Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3 -0
+		if (gfn == last)
+			goto out;
+

Nikunj or Michael, could you help me understand in which case it causes a KVM crash? To me, even the end is aligned to huge page boundary, but:

    last = (end - 1) & mask;

so 'last' is the base address for the last effective huage page. Even when gfn == last, it should still a valid page and needs to be updated for mem_attrs, correct?

Thanks,
Chao

Quoted 9 lines
Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +0 -0
 		gfn = last;
 		gfn_end = min(last + pages, slot->base_gfn + slot->npages);
 		mixed = mem_attrs_mixed(kvm, slot, level, attrs, gfn, gfn_end);
Signature · 1 line
2.25.1

BR, Jarkko

#6 lore raw

Re: [PATCH RFC v7 01/64] KVM: Fix memslot boundary condition for large page

Quoted 4 lines · Chao Peng
Quoted 3 lines · Jarkko Sakkinen
Quoted 2 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Aligned end boundary causes a kvm crash, handle the case.

Chao, are you aware of this issue already?

Thanks Jarkko adding me. I'm not aware of there is a fix.

It was discussed here: https://lore.kernel.org/all/[email protected]/

I was hitting this with one of the selftests case.

Quoted 22 lines
Quoted 16 lines
Quoted 16 lines
---
 arch/x86/kvm/mmu/mmu.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kvm/mmu/mmu.c +0 -0
index b1953ebc012e..b3ffc61c668c 100644
@@ -7159,6 +7159,9 @@ static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm,
 		for (gfn = first + pages; gfn < last; gfn += pages)
 			linfo_set_mixed(gfn, slot, level, false);
 
Quoted 3 lines

arch/x86/kvm/mmu/mmu.c static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm, · line 7164

Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3 -0
+		if (gfn == last)
+			goto out;
+

Nikunj or Michael, could you help me understand in which case it causes a KVM crash? To me, even the end is aligned to huge page boundary, but:

Quoted 4 lines
    last = (end - 1) & mask;

so 'last' is the base address for the last effective huage page. Even when gfn == last, it should still a valid page and needs to be updated for mem_attrs, correct?

Yes, that is correct with: last = (end - 1) & mask;

We can drop this patch from SNP series.

Regards
Nikunj

#7 lore raw

Re: [PATCH RFC v7 01/64] KVM: Fix memslot boundary condition for large page

Quoted 12 lines · Nikunj A. Dadhania
Quoted 9 lines · Chao Peng
Quoted 8 lines · Jarkko Sakkinen
Quoted 4 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

Aligned end boundary causes a kvm crash, handle the case.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Chao, are you aware of this issue already?

Thanks Jarkko adding me. I'm not aware of there is a fix.

It was discussed here: https://lore.kernel.org/all/[email protected]/

I was hitting this with one of the selftests case.

Yeah, I remember that discussion. With the new UPM code, this bug should be fixed. If you still hit the issue please let me know.

Thanks,
Chao

Quoted 34 lines
Quoted 23 lines
Quoted 17 lines
Quoted 17 lines
---
 arch/x86/kvm/mmu/mmu.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/kvm/mmu/mmu.c +0 -0
index b1953ebc012e..b3ffc61c668c 100644
@@ -7159,6 +7159,9 @@ static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm,
 		for (gfn = first + pages; gfn < last; gfn += pages)
 			linfo_set_mixed(gfn, slot, level, false);
 
Quoted 3 lines

arch/x86/kvm/mmu/mmu.c static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm, · line 7164

Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +3 -0
+		if (gfn == last)
+			goto out;
+

Nikunj or Michael, could you help me understand in which case it causes a KVM crash? To me, even the end is aligned to huge page boundary, but:

Quoted 4 lines
    last = (end - 1) & mask;

so 'last' is the base address for the last effective huage page. Even when gfn == last, it should still a valid page and needs to be updated for mem_attrs, correct?

Yes, that is correct with: last = (end - 1) & mask;

We can drop this patch from SNP series.

Regards
Nikunj

#8 lore raw

RFC PATCH v7 2/64 KVM: x86: Add KVM_CAP_UNMAPPED_PRIVATE_MEMORY

In reply to Michael Roth

This mainly indicates to KVM that it should expect all private guest memory to be backed by private memslots. Ideally this would work similarly for others archs, give or take a few additional flags, but for now it's a simple boolean indicator for x86.

---
 arch/x86/include/asm/kvm_host.h |  3 +++
 arch/x86/kvm/x86.c              | 10 ++++++++++
 include/uapi/linux/kvm.h        |  1 +
 3 files changed, 14 insertions(+)
arch/x86/include/asm/kvm_host.h +3 -0
index 27ef31133352..2b6244525107 100644
@@ -1438,6 +1438,9 @@ struct kvm_arch {
 	 */
 #define SPLIT_DESC_CACHE_MIN_NR_OBJECTS (SPTE_ENT_PER_PAGE + 1)
 	struct kvm_mmu_memory_cache split_desc_cache;
+
+	/* Use/enforce unmapped private memory. */
+	bool upm_mode;
 };
 
 struct kvm_vm_stat {
arch/x86/kvm/x86.c +10 -0
index c67e22f3e2ee..99ecf99bc4d2 100644
@@ -4421,6 +4421,11 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_EXIT_HYPERCALL:
 		r = KVM_EXIT_HYPERCALL_VALID_MASK;
 		break;
+#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES
+	case KVM_CAP_UNMAPPED_PRIVATE_MEM:
+		r = 1;
+		break;
+#endif
 	case KVM_CAP_SET_GUEST_DEBUG2:
 		return KVM_GUESTDBG_VALID_MASK;
 #ifdef CONFIG_KVM_XEN
@@ -6382,6 +6387,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		}
 		mutex_unlock(&kvm->lock);
 		break;
+	case KVM_CAP_UNMAPPED_PRIVATE_MEM:
+		kvm->arch.upm_mode = true;
+		r = 0;
+		break;
 	default:
 		r = -EINVAL;
 		break;
@@ -12128,6 +12137,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz;
 	kvm->arch.guest_can_read_msr_platform_info = true;
 	kvm->arch.enable_pmu = enable_pmu;
+	kvm->arch.upm_mode = false;
 
 #if IS_ENABLED(CONFIG_HYPERV)
 	spin_lock_init(&kvm->arch.hv_root_tdp_lock);
include/uapi/linux/kvm.h +1 -0
index c7e9d375a902..cc9424ccf9b2 100644
@@ -1219,6 +1219,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_DIRTY_LOG_RING_ACQ_REL 223
 #define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 224
 #define KVM_CAP_MEMORY_ATTRIBUTES 225
+#define KVM_CAP_UNMAPPED_PRIVATE_MEM 240
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
Signature · 1 line
2.25.1
maintainer
#9 lore raw

Re: [PATCH RFC v7 02/64] KVM: x86: Add KVM_CAP_UNMAPPED_PRIVATE_MEMORY

Quoted 4 lines · Michael Roth

This mainly indicates to KVM that it should expect all private guest memory to be backed by private memslots. Ideally this would work similarly for others archs, give or take a few additional flags, but for now it's a simple boolean indicator for x86.

...

include/uapi/linux/kvm.h struct kvm_ppc_resize_hpt { · line 1222

Quoted 9 lines
include/uapi/linux/kvm.h +1 -0
index c7e9d375a902..cc9424ccf9b2 100644
@@ -1219,6 +1219,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_DIRTY_LOG_RING_ACQ_REL 223
 #define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 224
 #define KVM_CAP_MEMORY_ATTRIBUTES 225
+#define KVM_CAP_UNMAPPED_PRIVATE_MEM 240

Isn't this new cap supposed to be documented somewhere in Documentation/virt/kvm/api.rst ?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#10 lore raw

Re: [PATCH RFC v7 02/64] KVM: x86: Add KVM_CAP_UNMAPPED_PRIVATE_MEMORY

Quoted 8 lines · Borislav Petkov
Quoted 4 lines · Michael Roth

This mainly indicates to KVM that it should expect all private guest memory to be backed by private memslots. Ideally this would work similarly for others archs, give or take a few additional flags, but for now it's a simple boolean indicator for x86.

...

Quoted 12 lines

include/uapi/linux/kvm.h struct kvm_ppc_resize_hpt { · line 1222

Quoted 9 lines
include/uapi/linux/kvm.h +1 -0
index c7e9d375a902..cc9424ccf9b2 100644
@@ -1219,6 +1219,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_DIRTY_LOG_RING_ACQ_REL 223
 #define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 224
 #define KVM_CAP_MEMORY_ATTRIBUTES 225
+#define KVM_CAP_UNMAPPED_PRIVATE_MEM 240

Isn't this new cap supposed to be documented somewhere in Documentation/virt/kvm/api.rst ?

It should, but this is sort of a placeholder for now. Ideally we'd re-use the capabilities introduced by UPM patchset rather than introduce a new one. Originally the UPM patchset had a KVM_CAP_PRIVATE_MEM which we planned to use to switch between legacy SEV and UPM-based SEV (for lazy-pinning support) by making it writeable, but that was removed in v10 in favor of KVM_CAP_MEMORY_ATTRIBUTES, which is tied to the new KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES/KVM_SET_MEMORY_ATTRIBUTES ioctls:

https://lore.kernel.org/lkml/CA+EHjTxXOdzcP25F57Mtmnb1NWyG5DcyqeDPqzjEOzRUrqH8FQ@mail.gmail.com/

It wasn't clear at the time if that was the right interface to use for this particular case, so we stuck with the more general 'use-upm/dont-use-upm' semantics originally provided by making KVM_CAP_UNMAPPED_PRIVATE_MEM/KVM_CAP_PRIVATE_MEM writeable.

But maybe it's okay to just make KVM_CAP_MEMORY_ATTRIBUTES writeable and require userspace to negotiate it rather than just tying it to CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES. Or maybe introducing a new KVM_SET_SUPPORTED_MEMORY_ATTRIBUTES ioctl to pair with KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES. It sort of makes sense, since userspace needs to be prepared to deal with KVM_EXIT_MEMORY_FAULTs relating to these attributes.

-Mike

Quoted 6 lines
Signature · 5 lines
-- 
Regards/Gruss,
    Boris.

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpeople.kernel.org%2Ftglx%2Fnotes-about-netiquette&data=05%7C01%7Cmichael.roth%40amd.com%7Cb019ddcb34a74fae1e3e08dae417c322%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638073087997837943%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QRyV96y3drt%2BqwxfifWzJ6UF6te8NOKWAFuGAYOdKcg%3D&reserved=0
maintainer
#11 lore raw

Re: [PATCH RFC v7 02/64] KVM: x86: Add KVM_CAP_UNMAPPED_PRIVATE_MEMORY

Quoted 7 lines · Michael Roth

But maybe it's okay to just make KVM_CAP_MEMORY_ATTRIBUTES writeable and require userspace to negotiate it rather than just tying it to CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES. Or maybe introducing a new KVM_SET_SUPPORTED_MEMORY_ATTRIBUTES ioctl to pair with KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES. It sort of makes sense, since userspace needs to be prepared to deal with KVM_EXIT_MEMORY_FAULTs relating to these attributes.

Makes sense.

AFAICT, ofc.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#12 lore raw

Re: [PATCH RFC v7 02/64] KVM: x86: Add KVM_CAP_UNMAPPED_PRIVATE_MEMORY

In reply to Michael Roth
Quoted 37 lines · Michael Roth
Quoted 8 lines · Borislav Petkov
Quoted 4 lines · Michael Roth

This mainly indicates to KVM that it should expect all private guest memory to be backed by private memslots. Ideally this would work similarly for others archs, give or take a few additional flags, but for now it's a simple boolean indicator for x86.

...

Quoted 12 lines

include/uapi/linux/kvm.h struct kvm_ppc_resize_hpt { · line 1222

Quoted 9 lines
include/uapi/linux/kvm.h +1 -0
index c7e9d375a902..cc9424ccf9b2 100644
@@ -1219,6 +1219,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_DIRTY_LOG_RING_ACQ_REL 223
 #define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 224
 #define KVM_CAP_MEMORY_ATTRIBUTES 225
+#define KVM_CAP_UNMAPPED_PRIVATE_MEM 240

Isn't this new cap supposed to be documented somewhere in Documentation/virt/kvm/api.rst ?

It should, but this is sort of a placeholder for now. Ideally we'd re-use the capabilities introduced by UPM patchset rather than introduce a new one. Originally the UPM patchset had a KVM_CAP_PRIVATE_MEM which we planned to use to switch between legacy SEV and UPM-based SEV (for lazy-pinning support) by making it writeable, but that was removed in v10 in favor of KVM_CAP_MEMORY_ATTRIBUTES, which is tied to the new KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES/KVM_SET_MEMORY_ATTRIBUTES ioctls:

https://lore.kernel.org/lkml/CA+EHjTxXOdzcP25F57Mtmnb1NWyG5DcyqeDPqzjEOzRUrqH8FQ@mail.gmail.com/

It wasn't clear at the time if that was the right interface to use for this particular case, so we stuck with the more general 'use-upm/dont-use-upm' semantics originally provided by making KVM_CAP_UNMAPPED_PRIVATE_MEM/KVM_CAP_PRIVATE_MEM writeable.

Quoted 7 lines

But maybe it's okay to just make KVM_CAP_MEMORY_ATTRIBUTES writeable and require userspace to negotiate it rather than just tying it to CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES. Or maybe introducing a new KVM_SET_SUPPORTED_MEMORY_ATTRIBUTES ioctl to pair with KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES. It sort of makes sense, since userspace needs to be prepared to deal with KVM_EXIT_MEMORY_FAULTs relating to these attributes.

Doesn't upm patch set imply that user space should negotiate the memory attributes with the ioctl?

For me it looks like that the problem is introduced by conflicting usage pattern in the SNP code [*].

Perhaps sev_launch_update_gfn_handler() should not set memory attributes but instead expect user space to do it before the call?

[*] https://lore.kernel.org/all/[email protected]/

BR, Jarkko

#13 lore raw

Re: [PATCH RFC v7 02/64] KVM: x86: Add KVM_CAP_UNMAPPED_PRIVATE_MEMORY

In reply to Michael Roth
Quoted 76 lines · Michael Roth

This mainly indicates to KVM that it should expect all private guest memory to be backed by private memslots. Ideally this would work similarly for others archs, give or take a few additional flags, but for now it's a simple boolean indicator for x86.

---
 arch/x86/include/asm/kvm_host.h |  3 +++
 arch/x86/kvm/x86.c              | 10 ++++++++++
 include/uapi/linux/kvm.h        |  1 +
 3 files changed, 14 insertions(+)
arch/x86/include/asm/kvm_host.h +3 -0
index 27ef31133352..2b6244525107 100644
@@ -1438,6 +1438,9 @@ struct kvm_arch {
 	 */
 #define SPLIT_DESC_CACHE_MIN_NR_OBJECTS (SPTE_ENT_PER_PAGE + 1)
 	struct kvm_mmu_memory_cache split_desc_cache;
+
+	/* Use/enforce unmapped private memory. */
+	bool upm_mode;
 };
 
 struct kvm_vm_stat {
arch/x86/kvm/x86.c +10 -0
index c67e22f3e2ee..99ecf99bc4d2 100644
@@ -4421,6 +4421,11 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_EXIT_HYPERCALL:
 		r = KVM_EXIT_HYPERCALL_VALID_MASK;
 		break;
+#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES
+	case KVM_CAP_UNMAPPED_PRIVATE_MEM:
+		r = 1;
+		break;
+#endif
 	case KVM_CAP_SET_GUEST_DEBUG2:
 		return KVM_GUESTDBG_VALID_MASK;
 #ifdef CONFIG_KVM_XEN
@@ -6382,6 +6387,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		}
 		mutex_unlock(&kvm->lock);
 		break;
+	case KVM_CAP_UNMAPPED_PRIVATE_MEM:
+		kvm->arch.upm_mode = true;
+		r = 0;
+		break;
 	default:
 		r = -EINVAL;
 		break;
@@ -12128,6 +12137,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz;
 	kvm->arch.guest_can_read_msr_platform_info = true;
 	kvm->arch.enable_pmu = enable_pmu;
+	kvm->arch.upm_mode = false;
 
 #if IS_ENABLED(CONFIG_HYPERV)
 	spin_lock_init(&kvm->arch.hv_root_tdp_lock);
include/uapi/linux/kvm.h +1 -0
index c7e9d375a902..cc9424ccf9b2 100644
@@ -1219,6 +1219,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_DIRTY_LOG_RING_ACQ_REL 223
 #define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 224
 #define KVM_CAP_MEMORY_ATTRIBUTES 225
+#define KVM_CAP_UNMAPPED_PRIVATE_MEM 240
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
Signature · 1 line
2.25.1

Why we want to carry non-UPM support still?

BR, Jarkko

#14 lore raw

Re: [PATCH RFC v7 02/64] KVM: x86: Add KVM_CAP_UNMAPPED_PRIVATE_MEMORY

Quoted 72 lines · Jarkko Sakkinen
Quoted 71 lines · Michael Roth

This mainly indicates to KVM that it should expect all private guest memory to be backed by private memslots. Ideally this would work similarly for others archs, give or take a few additional flags, but for now it's a simple boolean indicator for x86.

---
 arch/x86/include/asm/kvm_host.h |  3 +++
 arch/x86/kvm/x86.c              | 10 ++++++++++
 include/uapi/linux/kvm.h        |  1 +
 3 files changed, 14 insertions(+)
arch/x86/include/asm/kvm_host.h +3 -0
index 27ef31133352..2b6244525107 100644
@@ -1438,6 +1438,9 @@ struct kvm_arch {
 	 */
 #define SPLIT_DESC_CACHE_MIN_NR_OBJECTS (SPTE_ENT_PER_PAGE + 1)
 	struct kvm_mmu_memory_cache split_desc_cache;
+
+	/* Use/enforce unmapped private memory. */
+	bool upm_mode;
 };
 
 struct kvm_vm_stat {
arch/x86/kvm/x86.c +10 -0
index c67e22f3e2ee..99ecf99bc4d2 100644
@@ -4421,6 +4421,11 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_EXIT_HYPERCALL:
 		r = KVM_EXIT_HYPERCALL_VALID_MASK;
 		break;
+#ifdef CONFIG_HAVE_KVM_MEMORY_ATTRIBUTES
+	case KVM_CAP_UNMAPPED_PRIVATE_MEM:
+		r = 1;
+		break;
+#endif
 	case KVM_CAP_SET_GUEST_DEBUG2:
 		return KVM_GUESTDBG_VALID_MASK;
 #ifdef CONFIG_KVM_XEN
@@ -6382,6 +6387,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		}
 		mutex_unlock(&kvm->lock);
 		break;
+	case KVM_CAP_UNMAPPED_PRIVATE_MEM:
+		kvm->arch.upm_mode = true;
+		r = 0;
+		break;
 	default:
 		r = -EINVAL;
 		break;
@@ -12128,6 +12137,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz;
 	kvm->arch.guest_can_read_msr_platform_info = true;
 	kvm->arch.enable_pmu = enable_pmu;
+	kvm->arch.upm_mode = false;
 
 #if IS_ENABLED(CONFIG_HYPERV)
 	spin_lock_init(&kvm->arch.hv_root_tdp_lock);
include/uapi/linux/kvm.h +1 -0
index c7e9d375a902..cc9424ccf9b2 100644
@@ -1219,6 +1219,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_DIRTY_LOG_RING_ACQ_REL 223
 #define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 224
 #define KVM_CAP_MEMORY_ATTRIBUTES 225
+#define KVM_CAP_UNMAPPED_PRIVATE_MEM 240
 
Quoted 7 lines
Quoted 5 lines
include/uapi/linux/kvm.h +0 -0
 #ifdef KVM_CAP_IRQ_ROUTING
 
Signature · 1 line
2.25.1

Why we want to carry non-UPM support still?

For SNP, non-UPM support is no longer included in this patchset.

However, this patchset also adds support for UPM-based SEV (for lazy-pinning support). So we still need a way to let userspace switch between those 2 modes.

-Mike

Quoted 3 lines

BR, Jarkko

#15 lore raw

RFC PATCH v7 3/64 KVM: SVM: Advertise private memory support to KVM

In reply to Michael Roth

From: Nikunj A Dadhania <[email protected]>

KVM should use private memory for guests that have upm_mode flag set.

Add a kvm_x86_ops hook for determining UPM support that accounts for this situation by only enabling UPM test mode in the case of non-SEV guests.

[mdr: add x86 hook for determining restricted/private memory support]
---
 arch/x86/include/asm/kvm-x86-ops.h |  1 +
 arch/x86/include/asm/kvm_host.h    |  1 +
 arch/x86/kvm/svm/svm.c             | 10 ++++++++++
 arch/x86/kvm/x86.c                 |  8 ++++++++
 4 files changed, 20 insertions(+)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index abccd51dcfca..f530a550c092 100644
@@ -131,6 +131,7 @@ KVM_X86_OP(msr_filter_changed)
 KVM_X86_OP(complete_emulated_msr)
 KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
+KVM_X86_OP_OPTIONAL_RET0(private_mem_enabled);
 
 #undef KVM_X86_OP
 #undef KVM_X86_OP_OPTIONAL
arch/x86/include/asm/kvm_host.h +1 -0
index 2b6244525107..9317abffbf68 100644
@@ -1635,6 +1635,7 @@ struct kvm_x86_ops {
 
 	void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 			     int root_level);
+	int (*private_mem_enabled)(struct kvm *kvm);
 
 	bool (*has_wbinvd_exit)(void);
 
arch/x86/kvm/svm/svm.c +10 -0
index 91352d692845..7f3e4d91c0c6 100644
@@ -4694,6 +4694,14 @@ static int svm_vm_init(struct kvm *kvm)
 	return 0;
 }
 
+static int svm_private_mem_enabled(struct kvm *kvm)
+{
+	if (sev_guest(kvm))
+		return kvm->arch.upm_mode ? 1 : 0;
+
+	return IS_ENABLED(CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING) ? 1 : 0;
+}
+
 static struct kvm_x86_ops svm_x86_ops __initdata = {
 	.name = "kvm_amd",
 
@@ -4774,6 +4782,8 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
 
 	.vcpu_after_set_cpuid = svm_vcpu_after_set_cpuid,
 
+	.private_mem_enabled = svm_private_mem_enabled,
+
 	.has_wbinvd_exit = svm_has_wbinvd_exit,
 
 	.get_l2_tsc_offset = svm_get_l2_tsc_offset,
arch/x86/kvm/x86.c +8 -0
index 99ecf99bc4d2..bb6adb216054 100644
@@ -12266,6 +12266,14 @@ void __user * __x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa,
 }
 EXPORT_SYMBOL_GPL(__x86_set_memory_region);
 
+bool kvm_arch_has_private_mem(struct kvm *kvm)
+{
+	if (static_call(kvm_x86_private_mem_enabled)(kvm))
+		return true;
+
+	return false;
+}
+
 void kvm_arch_pre_destroy_vm(struct kvm *kvm)
 {
 	kvm_mmu_pre_destroy_vm(kvm);
Signature · 1 line
2.25.1
maintainer
#16 lore raw

Re: [PATCH RFC v7 03/64] KVM: SVM: Advertise private memory support to KVM

Quoted 1 line · Michael Roth

+ bool (*private_mem_enabled)(struct kvm *kvm);

This looks like a function returning boolean to me. IOW, you can simplify this to:

arch/x86/include/asm/kvm-x86-ops.h +1 -0
index 82ba4a564e58..4449aeff0dff 100644
@@ -129,6 +129,7 @@ KVM_X86_OP(msr_filter_changed)
 KVM_X86_OP(complete_emulated_msr)
 KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
+KVM_X86_OP_OPTIONAL_RET0(private_mem_enabled);
 
 #undef KVM_X86_OP
 #undef KVM_X86_OP_OPTIONAL
arch/x86/include/asm/kvm_host.h +1 -0
index 1da0474edb2d..1b4b89ddeb55 100644
@@ -1574,6 +1574,7 @@ struct kvm_x86_ops {
 
 	void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 			     int root_level);
+	bool (*private_mem_enabled)(struct kvm *kvm);
 
 	bool (*has_wbinvd_exit)(void);
 
arch/x86/kvm/svm/svm.c +10 -0
index ce362e88a567..73b780fa4653 100644
@@ -4680,6 +4680,14 @@ static int svm_vm_init(struct kvm *kvm)
 	return 0;
 }
 
+static bool svm_private_mem_enabled(struct kvm *kvm)
+{
+	if (sev_guest(kvm))
+		return kvm->arch.upm_mode;
+
+	return IS_ENABLED(CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING);
+}
+
 static struct kvm_x86_ops svm_x86_ops __initdata = {
 	.name = "kvm_amd",
 
@@ -4760,6 +4768,8 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
 
 	.vcpu_after_set_cpuid = svm_vcpu_after_set_cpuid,
 
+	.private_mem_enabled = svm_private_mem_enabled,
+
 	.has_wbinvd_exit = svm_has_wbinvd_exit,
 
 	.get_l2_tsc_offset = svm_get_l2_tsc_offset,
arch/x86/kvm/x86.c +5 -0
index 823646d601db..9a1ca59d36a4 100644
@@ -12556,6 +12556,11 @@ void __user * __x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa,
 }
 EXPORT_SYMBOL_GPL(__x86_set_memory_region);
 
+bool kvm_arch_has_private_mem(struct kvm *kvm)
+{
+	return static_call(kvm_x86_private_mem_enabled)(kvm);
+}
+
 void kvm_arch_pre_destroy_vm(struct kvm *kvm)
 {
 	kvm_mmu_pre_destroy_vm(kvm);
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#17 lore raw

Re: [PATCH RFC v7 03/64] KVM: SVM: Advertise private memory support to KVM

Quoted 5 lines · Borislav Petkov
Quoted 1 line · Michael Roth

+ bool (*private_mem_enabled)(struct kvm *kvm);

This looks like a function returning boolean to me. IOW, you can simplify this to:

The semantics and existing uses of KVM_X86_OP_OPTIONAL_RET0() gave me the impression it needed to return an integer value, since by default if a platform doesn't implement the op it would "return 0", and so could still be called unconditionally.

Maybe that's not actually enforced, by it seems awkward to try to use a bool return instead. At least for KVM_X86_OP_OPTIONAL_RET0().

However, we could just use KVM_X86_OP() to declare it so we can cleanly use a function that returns bool, and then we just need to do:

  bool kvm_arch_has_private_mem(struct kvm *kvm)
  {
          if (kvm_x86_ops.private_mem_enabled)
                  return static_call(kvm_x86_private_mem_enabled)(kvm);
  }

instead of relying on default return value. So I'll take that approach and adopt your other suggested changes.

...

On a separate topic though, at a high level, this hook is basically a way for platform-specific code to tell generic KVM code that private memslots are supported by overriding the kvm_arch_has_private_mem() weak reference. In this case the AMD platform is using using kvm->arch.upm_mode flag to convey that, which is in turn set by the KVM_CAP_UNMAPPED_PRIVATE_MEMORY introduced in this series.

But if, as I suggested in response to your PATCH 2 comments, we drop KVM_CAP_UNAMMPED_PRIVATE_MEMORY in favor of KVM_SET_SUPPORTED_MEMORY_ATTRIBUTES ioctl to enable "UPM mode" in SEV/SNP code, then we need to rethink things a bit, since KVM_SET_MEMORY_ATTRIBUTES in-part relies on kvm_arch_has_private_mem() to determine what flags are supported, whereas SEV/SNP code would be using what was set by KVM_SET_MEMORY_ATTRIBUTES to determine the return value in kvm_arch_has_private_mem().

So, for AMD, the return value of kvm_arch_has_private_mem() needs to rely on something else. Maybe the logic can just be:

  bool svm_private_mem_enabled(struct kvm *kvm)
  {
    return sev_enabled(kvm) || sev_snp_enabled(kvm)
  }

(at least in the context of this patchset where UPM support is added for both SEV and SNP).

So I'll plan to make that change as well.

-Mike

Quoted 75 lines
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index 82ba4a564e58..4449aeff0dff 100644
@@ -129,6 +129,7 @@ KVM_X86_OP(msr_filter_changed)
 KVM_X86_OP(complete_emulated_msr)
 KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
+KVM_X86_OP_OPTIONAL_RET0(private_mem_enabled);
 
 #undef KVM_X86_OP
 #undef KVM_X86_OP_OPTIONAL
arch/x86/include/asm/kvm_host.h +1 -0
index 1da0474edb2d..1b4b89ddeb55 100644
@@ -1574,6 +1574,7 @@ struct kvm_x86_ops {
 
 	void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 			     int root_level);
+	bool (*private_mem_enabled)(struct kvm *kvm);
 
 	bool (*has_wbinvd_exit)(void);
 
arch/x86/kvm/svm/svm.c +10 -0
index ce362e88a567..73b780fa4653 100644
@@ -4680,6 +4680,14 @@ static int svm_vm_init(struct kvm *kvm)
 	return 0;
 }
 
+static bool svm_private_mem_enabled(struct kvm *kvm)
+{
+	if (sev_guest(kvm))
+		return kvm->arch.upm_mode;
+
+	return IS_ENABLED(CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING);
+}
+
 static struct kvm_x86_ops svm_x86_ops __initdata = {
 	.name = "kvm_amd",
 
@@ -4760,6 +4768,8 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
 
 	.vcpu_after_set_cpuid = svm_vcpu_after_set_cpuid,
 
+	.private_mem_enabled = svm_private_mem_enabled,
+
 	.has_wbinvd_exit = svm_has_wbinvd_exit,
 
 	.get_l2_tsc_offset = svm_get_l2_tsc_offset,
arch/x86/kvm/x86.c +5 -0
index 823646d601db..9a1ca59d36a4 100644
@@ -12556,6 +12556,11 @@ void __user * __x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa,
 }
 EXPORT_SYMBOL_GPL(__x86_set_memory_region);
 
+bool kvm_arch_has_private_mem(struct kvm *kvm)
+{
+	return static_call(kvm_x86_private_mem_enabled)(kvm);
+}
+
 void kvm_arch_pre_destroy_vm(struct kvm *kvm)
 {
 	kvm_mmu_pre_destroy_vm(kvm);
Signature · 5 lines
-- 
Regards/Gruss,
    Boris.

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpeople.kernel.org%2Ftglx%2Fnotes-about-netiquette&data=05%7C01%7Cmichael.roth%40amd.com%7C319e89ce555a46eace4d08dae506b51a%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638074114318137471%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aG11K7va1BhemwlKCKKdcIXEwXGUzImYL%2BZ9%2FQ7XToI%3D&reserved=0
maintainer
#18 lore raw

Re: [PATCH RFC v7 03/64] KVM: SVM: Advertise private memory support to KVM

Quoted 2 lines · Michael Roth

Maybe that's not actually enforced, by it seems awkward to try to use a bool return instead. At least for KVM_X86_OP_OPTIONAL_RET0().

I don't see there being a problem/restriction for bool functions, see

5be2226f417d ("KVM: x86: allow defining return-0 static calls")

and __static_call_return0() returns a long which, if you wanna interpret as bool, works too as "false".

I still need to disassemble and single-step through a static_call to see what all that magic does in detail, to be sure.

Quoted 7 lines

However, we could just use KVM_X86_OP() to declare it so we can cleanly use a function that returns bool, and then we just need to do:

  bool kvm_arch_has_private_mem(struct kvm *kvm)
  {
          if (kvm_x86_ops.private_mem_enabled)
                  return static_call(kvm_x86_private_mem_enabled)(kvm);

That would be defeating the whole purpose of static calls, AFAICT, as you're testing the pointer. Might as well leave it be a normal function pointer then.

Quoted 16 lines

On a separate topic though, at a high level, this hook is basically a way for platform-specific code to tell generic KVM code that private memslots are supported by overriding the kvm_arch_has_private_mem() weak reference. In this case the AMD platform is using using kvm->arch.upm_mode flag to convey that, which is in turn set by the KVM_CAP_UNMAPPED_PRIVATE_MEMORY introduced in this series.

But if, as I suggested in response to your PATCH 2 comments, we drop KVM_CAP_UNAMMPED_PRIVATE_MEMORY in favor of KVM_SET_SUPPORTED_MEMORY_ATTRIBUTES ioctl to enable "UPM mode" in SEV/SNP code, then we need to rethink things a bit, since KVM_SET_MEMORY_ATTRIBUTES in-part relies on kvm_arch_has_private_mem() to determine what flags are supported, whereas SEV/SNP code would be using what was set by KVM_SET_MEMORY_ATTRIBUTES to determine the return value in kvm_arch_has_private_mem().

Quoted 6 lines

So, for AMD, the return value of kvm_arch_has_private_mem() needs to rely on something else. Maybe the logic can just be:

  bool svm_private_mem_enabled(struct kvm *kvm)
  {
    return sev_enabled(kvm) || sev_snp_enabled(kvm)

I haven't followed the whole discussion in detail but this means that SEV/SNP *means* UPM. I.e., no SEV/SNP without UPM, correct? I guess that's the final thing you guys decided to do ...

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#19 lore raw

Re: [PATCH RFC v7 03/64] KVM: SVM: Advertise private memory support to KVM

Quoted 6 lines · Borislav Petkov
Quoted 2 lines · Michael Roth

Maybe that's not actually enforced, by it seems awkward to try to use a bool return instead. At least for KVM_X86_OP_OPTIONAL_RET0().

I don't see there being a problem/restriction for bool functions, see

Quoted 7 lines

5be2226f417d ("KVM: x86: allow defining return-0 static calls")

and __static_call_return0() returns a long which, if you wanna interpret as bool, works too as "false".

I still need to disassemble and single-step through a static_call to see what all that magic does in detail, to be sure.

Hmm, yah, looking at that patch it seems pretty clear (at least at the time) that bool returns are expected to work fine and there are still existing cases that do things that way.

Quoted 11 lines
Quoted 8 lines

However, we could just use KVM_X86_OP() to declare it so we can cleanly use a function that returns bool, and then we just need to do:

  bool kvm_arch_has_private_mem(struct kvm *kvm)
  {
          if (kvm_x86_ops.private_mem_enabled)
                  return static_call(kvm_x86_private_mem_enabled)(kvm);

That would be defeating the whole purpose of static calls, AFAICT, as you're testing the pointer. Might as well leave it be a normal function pointer then.

That's true, we should just use the function pointer for those cases. But given the above maybe KVM_X86_OP_OPTIONAL_RET0() is fine after all so we can continue using static_call().

Quoted 20 lines
Quoted 20 lines

On a separate topic though, at a high level, this hook is basically a way for platform-specific code to tell generic KVM code that private memslots are supported by overriding the kvm_arch_has_private_mem() weak reference. In this case the AMD platform is using using kvm->arch.upm_mode flag to convey that, which is in turn set by the KVM_CAP_UNMAPPED_PRIVATE_MEMORY introduced in this series.

But if, as I suggested in response to your PATCH 2 comments, we drop KVM_CAP_UNAMMPED_PRIVATE_MEMORY in favor of KVM_SET_SUPPORTED_MEMORY_ATTRIBUTES ioctl to enable "UPM mode" in SEV/SNP code, then we need to rethink things a bit, since KVM_SET_MEMORY_ATTRIBUTES in-part relies on kvm_arch_has_private_mem() to determine what flags are supported, whereas SEV/SNP code would be using what was set by KVM_SET_MEMORY_ATTRIBUTES to determine the return value in kvm_arch_has_private_mem().

So, for AMD, the return value of kvm_arch_has_private_mem() needs to rely on something else. Maybe the logic can just be:

Quoted 7 lines
Quoted 3 lines
  bool svm_private_mem_enabled(struct kvm *kvm)
  {
    return sev_enabled(kvm) || sev_snp_enabled(kvm)

I haven't followed the whole discussion in detail but this means that SEV/SNP *means* UPM. I.e., no SEV/SNP without UPM, correct? I guess that's the final thing you guys decided to do ...

It's more about reporting whether UPM is *possible*. In the case of this patchset there is support for using UPM in conjunction with SEV or SEV-SNP, and that's all the above check is conveying. This is basically what KVM_SET_SUPPORTED_MEMORY_ATTRIBUTES ioctl will rely on to determine what attributes it reports to the guest as being supported, which is basically what tells userspace whether or not it can make use of UPM features.

In the case of SEV, it would still be up to userspace whether or not it actually wants to make use of UPM functionality like KVM_SET_MEMORY_ATTRIBUTES and private memslots. Otherwise, to maintain backward-compatibility, userspace can do things as it has always done and continue running SEV without relying on private memslots/KVM_SET_MEMORY_ATTRIBUTES or any of the new ioctls.

For SNP however it is required that userspace uses/implements UPM functionality.

-Mike

Quoted 8 lines

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpeople.kernel.org%2Ftglx%2Fnotes-about-netiquette&data=05%7C01%7Cmichael.roth%40amd.com%7C4bfcf0c3d1e54046703008daef2e35d4%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638085279100077750%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=oCjQx7LV1paTYdMA4JpFjIexf5UkiZEf2pYOWTvpkZ0%3D&reserved=0
maintainer
#20 lore raw

Re: [PATCH RFC v7 03/64] KVM: SVM: Advertise private memory support to KVM

Quoted 8 lines · Michael Roth

In the case of SEV, it would still be up to userspace whether or not it actually wants to make use of UPM functionality like KVM_SET_MEMORY_ATTRIBUTES and private memslots. Otherwise, to maintain backward-compatibility, userspace can do things as it has always done and continue running SEV without relying on private memslots/KVM_SET_MEMORY_ATTRIBUTES or any of the new ioctls.

For SNP however it is required that userspace uses/implements UPM functionality.

Makes sense to me.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#21 lore raw

Re: [PATCH RFC v7 03/64] KVM: SVM: Advertise private memory support to KVM

In reply to Michael Roth
Quoted 15 lines · Michael Roth
Quoted 5 lines · Borislav Petkov
Quoted 1 line · Michael Roth

+ bool (*private_mem_enabled)(struct kvm *kvm);

This looks like a function returning boolean to me. IOW, you can simplify this to:

The semantics and existing uses of KVM_X86_OP_OPTIONAL_RET0() gave me the impression it needed to return an integer value, since by default if a platform doesn't implement the op it would "return 0", and so could still be called unconditionally.

Maybe that's not actually enforced, by it seems awkward to try to use a bool return instead. At least for KVM_X86_OP_OPTIONAL_RET0().

Quoted 7 lines

However, we could just use KVM_X86_OP() to declare it so we can cleanly use a function that returns bool, and then we just need to do:

  bool kvm_arch_has_private_mem(struct kvm *kvm)
  {
          if (kvm_x86_ops.private_mem_enabled)
                  return static_call(kvm_x86_private_mem_enabled)(kvm);

I guess this is missing:

        return false;
Quoted 14 lines
  }

instead of relying on default return value. So I'll take that approach and adopt your other suggested changes.

...

On a separate topic though, at a high level, this hook is basically a way for platform-specific code to tell generic KVM code that private memslots are supported by overriding the kvm_arch_has_private_mem() weak reference. In this case the AMD platform is using using kvm->arch.upm_mode flag to convey that, which is in turn set by the KVM_CAP_UNMAPPED_PRIVATE_MEMORY introduced in this series.

Quoted 8 lines

But if, as I suggested in response to your PATCH 2 comments, we drop KVM_CAP_UNAMMPED_PRIVATE_MEMORY in favor of KVM_SET_SUPPORTED_MEMORY_ATTRIBUTES ioctl to enable "UPM mode" in SEV/SNP code, then we need to rethink things a bit, since KVM_SET_MEMORY_ATTRIBUTES in-part relies on kvm_arch_has_private_mem() to determine what flags are supported, whereas SEV/SNP code would be using what was set by KVM_SET_MEMORY_ATTRIBUTES to determine the return value in kvm_arch_has_private_mem().

Does this mean that internal calls to kvm_vm_set_region_attr() will cease to exist, and it will rely for user space to use the ioctl properly instead?

Quoted 85 lines

So, for AMD, the return value of kvm_arch_has_private_mem() needs to rely on something else. Maybe the logic can just be:

  bool svm_private_mem_enabled(struct kvm *kvm)
  {
    return sev_enabled(kvm) || sev_snp_enabled(kvm)
  }

(at least in the context of this patchset where UPM support is added for both SEV and SNP).

So I'll plan to make that change as well.

-Mike

Quoted 70 lines
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index 82ba4a564e58..4449aeff0dff 100644
@@ -129,6 +129,7 @@ KVM_X86_OP(msr_filter_changed)
 KVM_X86_OP(complete_emulated_msr)
 KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
+KVM_X86_OP_OPTIONAL_RET0(private_mem_enabled);
 
 #undef KVM_X86_OP
 #undef KVM_X86_OP_OPTIONAL
arch/x86/include/asm/kvm_host.h +1 -0
index 1da0474edb2d..1b4b89ddeb55 100644
@@ -1574,6 +1574,7 @@ struct kvm_x86_ops {
 
 	void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 			     int root_level);
+	bool (*private_mem_enabled)(struct kvm *kvm);
 
 	bool (*has_wbinvd_exit)(void);
 
arch/x86/kvm/svm/svm.c +10 -0
index ce362e88a567..73b780fa4653 100644
@@ -4680,6 +4680,14 @@ static int svm_vm_init(struct kvm *kvm)
 	return 0;
 }
 
+static bool svm_private_mem_enabled(struct kvm *kvm)
+{
+	if (sev_guest(kvm))
+		return kvm->arch.upm_mode;
+
+	return IS_ENABLED(CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING);
+}
+
 static struct kvm_x86_ops svm_x86_ops __initdata = {
 	.name = "kvm_amd",
 
@@ -4760,6 +4768,8 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
 
 	.vcpu_after_set_cpuid = svm_vcpu_after_set_cpuid,
 
+	.private_mem_enabled = svm_private_mem_enabled,
+
 	.has_wbinvd_exit = svm_has_wbinvd_exit,
 
 	.get_l2_tsc_offset = svm_get_l2_tsc_offset,
arch/x86/kvm/x86.c +5 -0
index 823646d601db..9a1ca59d36a4 100644
@@ -12556,6 +12556,11 @@ void __user * __x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa,
 }
 EXPORT_SYMBOL_GPL(__x86_set_memory_region);
 
+bool kvm_arch_has_private_mem(struct kvm *kvm)
+{
+	return static_call(kvm_x86_private_mem_enabled)(kvm);
+}
+
 void kvm_arch_pre_destroy_vm(struct kvm *kvm)
 {
 	kvm_mmu_pre_destroy_vm(kvm);
Quoted 5 lines
Quoted 5 lines
Signature · 4 lines
Regards/Gruss,
    Boris.

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpeople.kernel.org%2Ftglx%2Fnotes-about-netiquette&data=05%7C01%7Cmichael.roth%40amd.com%7C319e89ce555a46eace4d08dae506b51a%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638074114318137471%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aG11K7va1BhemwlKCKKdcIXEwXGUzImYL%2BZ9%2FQ7XToI%3D&reserved=0

BR, Jarkko

#22 lore raw

Re: [PATCH RFC v7 03/64] KVM: SVM: Advertise private memory support to KVM

Quoted 51 lines · Jarkko Sakkinen
Quoted 15 lines · Michael Roth
Quoted 5 lines · Borislav Petkov
Quoted 1 line · Michael Roth

+ bool (*private_mem_enabled)(struct kvm *kvm);

This looks like a function returning boolean to me. IOW, you can simplify this to:

The semantics and existing uses of KVM_X86_OP_OPTIONAL_RET0() gave me the impression it needed to return an integer value, since by default if a platform doesn't implement the op it would "return 0", and so could still be called unconditionally.

Maybe that's not actually enforced, by it seems awkward to try to use a bool return instead. At least for KVM_X86_OP_OPTIONAL_RET0().

Quoted 7 lines

However, we could just use KVM_X86_OP() to declare it so we can cleanly use a function that returns bool, and then we just need to do:

  bool kvm_arch_has_private_mem(struct kvm *kvm)
  {
          if (kvm_x86_ops.private_mem_enabled)
                  return static_call(kvm_x86_private_mem_enabled)(kvm);

I guess this is missing:

        return false;
Quoted 14 lines
  }

instead of relying on default return value. So I'll take that approach and adopt your other suggested changes.

...

On a separate topic though, at a high level, this hook is basically a way for platform-specific code to tell generic KVM code that private memslots are supported by overriding the kvm_arch_has_private_mem() weak reference. In this case the AMD platform is using using kvm->arch.upm_mode flag to convey that, which is in turn set by the KVM_CAP_UNMAPPED_PRIVATE_MEMORY introduced in this series.

Quoted 8 lines

But if, as I suggested in response to your PATCH 2 comments, we drop KVM_CAP_UNAMMPED_PRIVATE_MEMORY in favor of KVM_SET_SUPPORTED_MEMORY_ATTRIBUTES ioctl to enable "UPM mode" in SEV/SNP code, then we need to rethink things a bit, since KVM_SET_MEMORY_ATTRIBUTES in-part relies on kvm_arch_has_private_mem() to determine what flags are supported, whereas SEV/SNP code would be using what was set by KVM_SET_MEMORY_ATTRIBUTES to determine the return value in kvm_arch_has_private_mem().

Quoted 3 lines

Does this mean that internal calls to kvm_vm_set_region_attr() will cease to exist, and it will rely for user space to use the ioctl properly instead?

Patches 1-3 are no longer needed and have been dropped for v8, instead "UPM mode" is set via KVM_VM_CREATE vm_type arg, and SEV/SNP can simply call kvm_arch_has_private_mem() to query whether userspace has enabled UPM mode or not.

But even still, we call kvm_vm_set_region_attr() in sev_launch_update_data() and snp_launch_update() after copying initial payload into private memory.

I don't think there's much worth in having userspace have to do it via KVM_SET_MEMORY_ATTRIBUTES afterward. It could be done that way I suppose, but generally RMP update from shared->private happens as part of KVM_SET_MEMORY_ATTRIBUTES, whereas in this case it would necessarily happen *after* the RMP updates, since SNP_LAUNCH_UPDATE expects the pages to be marked private beforehand.

Just seems like more corner cases to deal with and more boilerplate code for userspace, which already needed to operate under the assumption that pages will be private after SNP_LAUNCH_UPDATE, so seems to make sense to just have the memory attributes also updated accordingly.

-Mike

Quoted 93 lines
Quoted 86 lines

So, for AMD, the return value of kvm_arch_has_private_mem() needs to rely on something else. Maybe the logic can just be:

  bool svm_private_mem_enabled(struct kvm *kvm)
  {
    return sev_enabled(kvm) || sev_snp_enabled(kvm)
  }

(at least in the context of this patchset where UPM support is added for both SEV and SNP).

So I'll plan to make that change as well.

-Mike

Quoted 70 lines
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index 82ba4a564e58..4449aeff0dff 100644
@@ -129,6 +129,7 @@ KVM_X86_OP(msr_filter_changed)
 KVM_X86_OP(complete_emulated_msr)
 KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
+KVM_X86_OP_OPTIONAL_RET0(private_mem_enabled);
 
 #undef KVM_X86_OP
 #undef KVM_X86_OP_OPTIONAL
arch/x86/include/asm/kvm_host.h +1 -0
index 1da0474edb2d..1b4b89ddeb55 100644
@@ -1574,6 +1574,7 @@ struct kvm_x86_ops {
 
 	void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 			     int root_level);
+	bool (*private_mem_enabled)(struct kvm *kvm);
 
 	bool (*has_wbinvd_exit)(void);
 
arch/x86/kvm/svm/svm.c +10 -0
index ce362e88a567..73b780fa4653 100644
@@ -4680,6 +4680,14 @@ static int svm_vm_init(struct kvm *kvm)
 	return 0;
 }
 
+static bool svm_private_mem_enabled(struct kvm *kvm)
+{
+	if (sev_guest(kvm))
+		return kvm->arch.upm_mode;
+
+	return IS_ENABLED(CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING);
+}
+
 static struct kvm_x86_ops svm_x86_ops __initdata = {
 	.name = "kvm_amd",
 
@@ -4760,6 +4768,8 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
 
 	.vcpu_after_set_cpuid = svm_vcpu_after_set_cpuid,
 
+	.private_mem_enabled = svm_private_mem_enabled,
+
 	.has_wbinvd_exit = svm_has_wbinvd_exit,
 
 	.get_l2_tsc_offset = svm_get_l2_tsc_offset,
arch/x86/kvm/x86.c +5 -0
index 823646d601db..9a1ca59d36a4 100644
@@ -12556,6 +12556,11 @@ void __user * __x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa,
 }
 EXPORT_SYMBOL_GPL(__x86_set_memory_region);
 
+bool kvm_arch_has_private_mem(struct kvm *kvm)
+{
+	return static_call(kvm_x86_private_mem_enabled)(kvm);
+}
+
 void kvm_arch_pre_destroy_vm(struct kvm *kvm)
 {
 	kvm_mmu_pre_destroy_vm(kvm);
Quoted 5 lines
Quoted 5 lines
Signature · 4 lines
Regards/Gruss,
    Boris.

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpeople.kernel.org%2Ftglx%2Fnotes-about-netiquette&data=05%7C01%7Cmichael.roth%40amd.com%7C319e89ce555a46eace4d08dae506b51a%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638074114318137471%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aG11K7va1BhemwlKCKKdcIXEwXGUzImYL%2BZ9%2FQ7XToI%3D&reserved=0

BR, Jarkko

#23 lore raw

Re: [PATCH RFC v7 03/64] KVM: SVM: Advertise private memory support to KVM

In reply to Michael Roth
Quoted 48 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

KVM should use private memory for guests that have upm_mode flag set.

Add a kvm_x86_ops hook for determining UPM support that accounts for this situation by only enabling UPM test mode in the case of non-SEV guests.

[mdr: add x86 hook for determining restricted/private memory support]
---
 arch/x86/include/asm/kvm-x86-ops.h |  1 +
 arch/x86/include/asm/kvm_host.h    |  1 +
 arch/x86/kvm/svm/svm.c             | 10 ++++++++++
 arch/x86/kvm/x86.c                 |  8 ++++++++
 4 files changed, 20 insertions(+)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index abccd51dcfca..f530a550c092 100644
@@ -131,6 +131,7 @@ KVM_X86_OP(msr_filter_changed)
 KVM_X86_OP(complete_emulated_msr)
 KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
+KVM_X86_OP_OPTIONAL_RET0(private_mem_enabled);
 
 #undef KVM_X86_OP
 #undef KVM_X86_OP_OPTIONAL
arch/x86/include/asm/kvm_host.h +1 -0
index 2b6244525107..9317abffbf68 100644
@@ -1635,6 +1635,7 @@ struct kvm_x86_ops {
 
 	void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 			     int root_level);
+	int (*private_mem_enabled)(struct kvm *kvm);
 
 	bool (*has_wbinvd_exit)(void);
 
arch/x86/kvm/svm/svm.c +0 -0
index 91352d692845..7f3e4d91c0c6 100644
@@ -4694,6 +4694,14 @@ static int svm_vm_init(struct kvm *kvm)
 	return 0;

arch/x86/kvm/svm/svm.c static int svm_vm_init(struct kvm *kvm) · line 4704

Quoted 10 lines
arch/x86/kvm/svm/svm.c +8 -0
 }
 
+static int svm_private_mem_enabled(struct kvm *kvm)
+{
+	if (sev_guest(kvm))
+		return kvm->arch.upm_mode ? 1 : 0;
+
+	return IS_ENABLED(CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING) ? 1 : 0;
+}
+

Is this new callback really needed? Shouldn't kvm->arch.upm_mode be sufficient enough to indicate whether the private memory will be used or not?

Probably the CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING is the concern here. But this Kconfig option is not even x86-specific, so shouldn't the handling of it be done in common code too?

For instance, can we explicitly set 'kvm->arch.upm_mode' to 'true' at some point of creating the VM if we see CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING is true?

[snip]

#24 lore raw

Re: [PATCH RFC v7 03/64] KVM: SVM: Advertise private memory support to KVM

Quoted 9 lines · Kai
Quoted 8 lines · Michael Roth
arch/x86/kvm/svm/svm.c +0 -0
index 91352d692845..7f3e4d91c0c6 100644
@@ -4694,6 +4694,14 @@ static int svm_vm_init(struct kvm *kvm)
 	return 0;
 }
 
Quoted 10 lines

arch/x86/kvm/svm/svm.c static int svm_vm_init(struct kvm *kvm) · line 4704

Quoted 8 lines
arch/x86/kvm/svm/svm.c +8 -0
+static int svm_private_mem_enabled(struct kvm *kvm)
+{
+	if (sev_guest(kvm))
+		return kvm->arch.upm_mode ? 1 : 0;
+
+	return IS_ENABLED(CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING) ? 1 : 0;
+}
+

Is this new callback really needed?

Probably not. For anything in this series that gets within spitting distance of CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING, my recommendation is to make a mental note but otherwise ignore things like this for now. I suspect it will be much, much more efficient to sort all of this out when I smush UPM+SNP+TDX together in a few weeks.

#25 lore raw

RFC PATCH v7 4/64 KVM: x86: Add 'fault_is_private' x86 op

In reply to Michael Roth

This callback is used by the KVM MMU to check whether a #NPF was or a private GPA or not.

---
 arch/x86/include/asm/kvm-x86-ops.h |  1 +
 arch/x86/include/asm/kvm_host.h    |  1 +
 arch/x86/kvm/mmu/mmu.c             |  3 +--
 arch/x86/kvm/mmu/mmu_internal.h    | 40 +++++++++++++++++++++++++++---
 4 files changed, 39 insertions(+), 6 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index f530a550c092..efae987cdce0 100644
@@ -132,6 +132,7 @@ KVM_X86_OP(complete_emulated_msr)
 KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
 KVM_X86_OP_OPTIONAL_RET0(private_mem_enabled);
+KVM_X86_OP_OPTIONAL_RET0(fault_is_private);
 
 #undef KVM_X86_OP
 #undef KVM_X86_OP_OPTIONAL
arch/x86/include/asm/kvm_host.h +1 -0
index 9317abffbf68..92539708f062 100644
@@ -1636,6 +1636,7 @@ struct kvm_x86_ops {
 	void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 			     int root_level);
 	int (*private_mem_enabled)(struct kvm *kvm);
+	int (*fault_is_private)(struct kvm *kvm, gpa_t gpa, u64 error_code, bool *private_fault);
 
 	bool (*has_wbinvd_exit)(void);
 
arch/x86/kvm/mmu/mmu.c +1 -2
index b3ffc61c668c..61a7c221b966 100644
@@ -5646,8 +5646,7 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err
 	}
 
 	if (r == RET_PF_INVALID) {
-		r = kvm_mmu_do_page_fault(vcpu, cr2_or_gpa,
-					  lower_32_bits(error_code), false);
+		r = kvm_mmu_do_page_fault(vcpu, cr2_or_gpa, error_code, false);
 		if (KVM_BUG_ON(r == RET_PF_INVALID, vcpu->kvm))
 			return -EIO;
 	}
arch/x86/kvm/mmu/mmu_internal.h +36 -4
index e2f508db0b6e..04ea8da86510 100644
@@ -230,6 +230,38 @@ struct kvm_page_fault {
 
 int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);
 
+static bool kvm_mmu_fault_is_private(struct kvm *kvm, gpa_t gpa, u64 err)
+{
+	struct kvm_memory_slot *slot;
+	bool private_fault = false;
+	gfn_t gfn = gpa_to_gfn(gpa);
+
+	slot = gfn_to_memslot(kvm, gfn);
+	if (!slot) {
+		pr_debug("%s: no slot, GFN: 0x%llx\n", __func__, gfn);
+		goto out;
+	}
+
+	if (!kvm_slot_can_be_private(slot)) {
+		pr_debug("%s: slot is not private, GFN: 0x%llx\n", __func__, gfn);
+		goto out;
+	}
+
+	if (static_call(kvm_x86_fault_is_private)(kvm, gpa, err, &private_fault) == 1)
+		goto out;
+
+	/*
+	 * Handling below is for UPM self-tests and guests that use
+	 * slot->shared_bitmap for encrypted access tracking.
+	 */
+	if (IS_ENABLED(CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING))
+		private_fault = kvm_mem_is_private(kvm, gpa >> PAGE_SHIFT);
+
+out:
+	pr_debug("%s: GFN: 0x%llx, private: %d\n", __func__, gfn, private_fault);
+	return private_fault;
+}
+
 /*
  * Return values of handle_mmio_page_fault(), mmu.page_fault(), fast_page_fault(),
  * and of course kvm_mmu_do_page_fault().
@@ -261,13 +293,13 @@ enum {
 };
 
 static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
-					u32 err, bool prefetch)
+					u64 err, bool prefetch)
 {
 	bool is_tdp = likely(vcpu->arch.mmu->page_fault == kvm_tdp_page_fault);
 
 	struct kvm_page_fault fault = {
 		.addr = cr2_or_gpa,
-		.error_code = err,
+		.error_code = lower_32_bits(err),
 		.exec = err & PFERR_FETCH_MASK,
 		.write = err & PFERR_WRITE_MASK,
 		.present = err & PFERR_PRESENT_MASK,
@@ -281,8 +313,8 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 		.max_level = KVM_MAX_HUGEPAGE_LEVEL,
 		.req_level = PG_LEVEL_4K,
 		.goal_level = PG_LEVEL_4K,
-		.is_private = IS_ENABLED(CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING) && is_tdp &&
-				kvm_mem_is_private(vcpu->kvm, cr2_or_gpa >> PAGE_SHIFT),
+		.is_private = is_tdp && kvm_mmu_fault_is_private(vcpu->kvm,
+								 cr2_or_gpa, err),
 	};
 	int r;
 
Signature · 1 line
2.25.1
maintainer
#26 lore raw

Re: [PATCH RFC v7 04/64] KVM: x86: Add 'fault_is_private' x86 op

Quoted 2 lines · Michael Roth

This callback is used by the KVM MMU to check whether a #NPF was or a private GPA or not.

s/or //

Quoted 26 lines
---
 arch/x86/include/asm/kvm-x86-ops.h |  1 +
 arch/x86/include/asm/kvm_host.h    |  1 +
 arch/x86/kvm/mmu/mmu.c             |  3 +--
 arch/x86/kvm/mmu/mmu_internal.h    | 40 +++++++++++++++++++++++++++---
 4 files changed, 39 insertions(+), 6 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index f530a550c092..efae987cdce0 100644
@@ -132,6 +132,7 @@ KVM_X86_OP(complete_emulated_msr)
 KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
 KVM_X86_OP_OPTIONAL_RET0(private_mem_enabled);
+KVM_X86_OP_OPTIONAL_RET0(fault_is_private);
 
 #undef KVM_X86_OP
 #undef KVM_X86_OP_OPTIONAL
arch/x86/include/asm/kvm_host.h +0 -0
index 9317abffbf68..92539708f062 100644

arch/x86/include/asm/kvm_host.h struct kvm_x86_ops { · line 1639

Quoted 4 lines
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1636,6 +1636,7 @@ struct kvm_x86_ops {
 	void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 			     int root_level);
 	int (*private_mem_enabled)(struct kvm *kvm);
+	int (*fault_is_private)(struct kvm *kvm, gpa_t gpa, u64 error_code, bool *private_fault);

bool

and then you don't need the silly "== 1" at the call site.

Quoted 2 lines
arch/x86/include/asm/kvm_host.h +0 -0
 	bool (*has_wbinvd_exit)(void);

...

arch/x86/include/asm/kvm_host.h enum { · line 296

Quoted 6 lines
arch/x86/include/asm/kvm_host.h +1 -1
@@ -261,13 +293,13 @@ enum {
 };
 
 static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
-					u32 err, bool prefetch)
+					u64 err, bool prefetch)

The u32 -> u64 change of err could use a sentence or two of clarification in the commit message...

Quoted 24 lines
arch/x86/include/asm/kvm_host.h +3 -3
 {
 	bool is_tdp = likely(vcpu->arch.mmu->page_fault == kvm_tdp_page_fault);
 
 	struct kvm_page_fault fault = {
 		.addr = cr2_or_gpa,
-		.error_code = err,
+		.error_code = lower_32_bits(err),
 		.exec = err & PFERR_FETCH_MASK,
 		.write = err & PFERR_WRITE_MASK,
 		.present = err & PFERR_PRESENT_MASK,
@@ -281,8 +313,8 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 		.max_level = KVM_MAX_HUGEPAGE_LEVEL,
 		.req_level = PG_LEVEL_4K,
 		.goal_level = PG_LEVEL_4K,
-		.is_private = IS_ENABLED(CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING) && is_tdp &&
-				kvm_mem_is_private(vcpu->kvm, cr2_or_gpa >> PAGE_SHIFT),
+		.is_private = is_tdp && kvm_mmu_fault_is_private(vcpu->kvm,
+								 cr2_or_gpa, err),
 	};
 	int r;
 
Signature · 1 line
2.25.1
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#27 lore raw

Re: [PATCH RFC v7 04/64] KVM: x86: Add 'fault_is_private' x86 op

Quoted 39 lines · Borislav Petkov
Quoted 2 lines · Michael Roth

This callback is used by the KVM MMU to check whether a #NPF was or a private GPA or not.

s/or //

Quoted 26 lines
---
 arch/x86/include/asm/kvm-x86-ops.h |  1 +
 arch/x86/include/asm/kvm_host.h    |  1 +
 arch/x86/kvm/mmu/mmu.c             |  3 +--
 arch/x86/kvm/mmu/mmu_internal.h    | 40 +++++++++++++++++++++++++++---
 4 files changed, 39 insertions(+), 6 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index f530a550c092..efae987cdce0 100644
@@ -132,6 +132,7 @@ KVM_X86_OP(complete_emulated_msr)
 KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
 KVM_X86_OP_OPTIONAL_RET0(private_mem_enabled);
+KVM_X86_OP_OPTIONAL_RET0(fault_is_private);
 
 #undef KVM_X86_OP
 #undef KVM_X86_OP_OPTIONAL
arch/x86/include/asm/kvm_host.h +0 -0
index 9317abffbf68..92539708f062 100644

arch/x86/include/asm/kvm_host.h struct kvm_x86_ops { · line 1639

Quoted 4 lines
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1636,6 +1636,7 @@ struct kvm_x86_ops {
 	void (*load_mmu_pgd)(struct kvm_vcpu *vcpu, hpa_t root_hpa,
 			     int root_level);
 	int (*private_mem_enabled)(struct kvm *kvm);
+	int (*fault_is_private)(struct kvm *kvm, gpa_t gpa, u64 error_code, bool *private_fault);

bool

Quoted 1 line

and then you don't need the silly "== 1" at the call site.

Obviously I need to add some proper documentation for this, but a 1 return basically means 'private_fault' pass-by-ref arg has been set with the appropriate value, whereas 0 means "there's no platform-specific handling for this, so if you have some generic way to determine this then use that instead".

This is mainly to handle CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING, which just parrots whatever kvm_mem_is_private() returns to support running KVM selftests without needed hardware/platform support. If we don't take care to skip this check where the above fault_is_private() hook returns 1, then it ends up breaking SNP in cases where the kernel has been compiled with CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING, since SNP relies on the page fault flags to make this determination, not kvm_mem_is_private(), which normally only tracks the memory attributes set by userspace via KVM_SET_MEMORY_ATTRIBUTES ioctl.

Quoted 9 lines
Quoted 3 lines
arch/x86/include/asm/kvm_host.h +0 -0
 	bool (*has_wbinvd_exit)(void);

...

Quoted 3 lines
arch/x86/include/asm/kvm_host.h +0 -0
@@ -261,13 +293,13 @@ enum {
 };
 
Quoted 6 lines

arch/x86/include/asm/kvm_host.h enum { · line 296

Quoted 3 lines
arch/x86/include/asm/kvm_host.h +1 -1
 static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
-					u32 err, bool prefetch)
+					u64 err, bool prefetch)

The u32 -> u64 change of err could use a sentence or two of clarification in the commit message...

Will do.

-Mike

Quoted 31 lines
Quoted 25 lines
arch/x86/include/asm/kvm_host.h +3 -3
 {
 	bool is_tdp = likely(vcpu->arch.mmu->page_fault == kvm_tdp_page_fault);
 
 	struct kvm_page_fault fault = {
 		.addr = cr2_or_gpa,
-		.error_code = err,
+		.error_code = lower_32_bits(err),
 		.exec = err & PFERR_FETCH_MASK,
 		.write = err & PFERR_WRITE_MASK,
 		.present = err & PFERR_PRESENT_MASK,
@@ -281,8 +313,8 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 		.max_level = KVM_MAX_HUGEPAGE_LEVEL,
 		.req_level = PG_LEVEL_4K,
 		.goal_level = PG_LEVEL_4K,
-		.is_private = IS_ENABLED(CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING) && is_tdp &&
-				kvm_mem_is_private(vcpu->kvm, cr2_or_gpa >> PAGE_SHIFT),
+		.is_private = is_tdp && kvm_mmu_fault_is_private(vcpu->kvm,
+								 cr2_or_gpa, err),
 	};
 	int r;
 
Signature · 1 line
2.25.1
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer
#28 lore raw

Re: [PATCH RFC v7 04/64] KVM: x86: Add 'fault_is_private' x86 op

Quoted 5 lines · Michael Roth

Obviously I need to add some proper documentation for this, but a 1 return basically means 'private_fault' pass-by-ref arg has been set with the appropriate value, whereas 0 means "there's no platform-specific handling for this, so if you have some generic way to determine this then use that instead".

Still binary, tho, and can be bool, right?

I.e., you can just as well do:

if (static_call(kvm_x86_fault_is_private)(kvm, gpa, err, &private_fault))
goto out;

at the call site.

Quoted 9 lines

This is mainly to handle CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING, which just parrots whatever kvm_mem_is_private() returns to support running KVM selftests without needed hardware/platform support. If we don't take care to skip this check where the above fault_is_private() hook returns 1, then it ends up breaking SNP in cases where the kernel has been compiled with CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING, since SNP relies on the page fault flags to make this determination, not kvm_mem_is_private(), which normally only tracks the memory attributes set by userspace via KVM_SET_MEMORY_ATTRIBUTES ioctl.

Some of that explanation belongs into the commit message, which is a bit lacking...

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#29 lore raw

Re: [PATCH RFC v7 04/64] KVM: x86: Add 'fault_is_private' x86 op

Quoted 7 lines · Borislav Petkov
Quoted 5 lines · Michael Roth

Obviously I need to add some proper documentation for this, but a 1 return basically means 'private_fault' pass-by-ref arg has been set with the appropriate value, whereas 0 means "there's no platform-specific handling for this, so if you have some generic way to determine this then use that instead".

Quoted 8 lines

Still binary, tho, and can be bool, right?

I.e., you can just as well do:

if (static_call(kvm_x86_fault_is_private)(kvm, gpa, err, &private_fault))
goto out;

at the call site.

Ya. Don't spend too much time trying to make this look super pretty though, there are subtle bugs inherited from the base UPM series that need to be sorted out and will impact this code. E.g. invoking kvm_mem_is_private() outside of the protection of mmu_invalidate_seq means changes to the attributes may not be reflected in the page tables.

I'm also hoping we can avoid a callback entirely, though that may prove to be more pain than gain. I'm poking at the UPM and testing series right now, will circle back to this and TDX in a few weeks to see if there's a sane way to communicate shared vs. private without having to resort to a callback, and without having races between page faults, KVM_SET_MEMORY_ATTRIBUTES, and KVM_SET_USER_MEMORY_REGION2.

Quoted 10 lines
Quoted 9 lines

This is mainly to handle CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING, which just parrots whatever kvm_mem_is_private() returns to support running KVM selftests without needed hardware/platform support. If we don't take care to skip this check where the above fault_is_private() hook returns 1, then it ends up breaking SNP in cases where the kernel has been compiled with CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING, since SNP relies on the page fault flags to make this determination, not kvm_mem_is_private(), which normally only tracks the memory attributes set by userspace via KVM_SET_MEMORY_ATTRIBUTES ioctl.

Quoted 2 lines

Some of that explanation belongs into the commit message, which is a bit lacking...

I'll circle back to this too when I give this series (and TDX) a proper look, there's got too be a better way to handle this.

maintainer
#30 lore raw

Re: [PATCH RFC v7 04/64] KVM: x86: Add 'fault_is_private' x86 op

Quoted 3 lines · Sean Christopherson

Ya. Don't spend too much time trying to make this look super pretty though, there are subtle bugs inherited from the base UPM series that need to be sorted out and will impact this code.

Yeah, I'm simply trying to find my way around the code and no better way than reviewing it. But thanks for the heads up.

Quoted 2 lines

I'll circle back to this too when I give this series (and TDX) a proper look, there's got too be a better way to handle this.

Good.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#31 lore raw

Re: [PATCH RFC v7 04/64] KVM: x86: Add 'fault_is_private' x86 op

In reply to Sean Christopherson
Quoted 23 lines · Sean Christopherson
Quoted 7 lines · Borislav Petkov
Quoted 5 lines · Michael Roth

Obviously I need to add some proper documentation for this, but a 1 return basically means 'private_fault' pass-by-ref arg has been set with the appropriate value, whereas 0 means "there's no platform-specific handling for this, so if you have some generic way to determine this then use that instead".

Quoted 8 lines

Still binary, tho, and can be bool, right?

I.e., you can just as well do:

if (static_call(kvm_x86_fault_is_private)(kvm, gpa, err, &private_fault))
goto out;

at the call site.

Ya. Don't spend too much time trying to make this look super pretty though, there are subtle bugs inherited from the base UPM series that need to be sorted out and will impact this code. E.g. invoking kvm_mem_is_private() outside of the protection of mmu_invalidate_seq means changes to the attributes may not be reflected in the page tables.

Quoted 5 lines

I'm also hoping we can avoid a callback entirely, though that may prove to be more pain than gain. I'm poking at the UPM and testing series right now, will circle back to this and TDX in a few weeks to see if there's a sane way to communicate shared vs. private without having to resort to a callback, and without having races between page faults, KVM_SET_MEMORY_ATTRIBUTES, and KVM_SET_USER_MEMORY_REGION2.

Can circle back on this, but for v8 at least I've kept the callback, but simplified SVM implementation of it so that it's only needed for SNP. For protected-SEV it will fall through to the same generic handling used by UPM self-tests.

It seems like it's safe to have a callback of that sort here for TDX/SNP (or whatever we end up replacing the callback with), since the #NPF flags themselves won't change based on attribute updates, and the subsequent comparison to kvm_mem_is_private() will happen after mmu_invalidate_seq is logged.

But for protected-SEV and UPM selftests the initial kvm_mem_is_private() can become stale vs. the one in __kvm_faultin_pfn(), but it seems like ATM it would only lead to a spurious KVM_EXIT_MEMORY_FAULT, which SEV at least should treat at an implicit page-state change and be able to recover from. But yah, not ideal, and maybe for self-tests that makes it difficult to tell if things are working as expected or not.

Maybe we should just skip setting fault->is_private here in the non-TDX/non-SNP cases, and just have some other indicator so it's initialized/ignored in kvm_mem_is_private() later. I think some iterations of UPM did it this way prior to 'is_private' becoming const.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

This is mainly to handle CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING, which just parrots whatever kvm_mem_is_private() returns to support running KVM selftests without needed hardware/platform support. If we don't take care to skip this check where the above fault_is_private() hook returns 1, then it ends up breaking SNP in cases where the kernel has been compiled with CONFIG_HAVE_KVM_PRIVATE_MEM_TESTING, since SNP relies on the page fault flags to make this determination, not kvm_mem_is_private(), which normally only tracks the memory attributes set by userspace via KVM_SET_MEMORY_ATTRIBUTES ioctl.

Quoted 6 lines
Quoted 2 lines

Some of that explanation belongs into the commit message, which is a bit lacking...

I'll circle back to this too when I give this series (and TDX) a proper look, there's got too be a better way to handle this.

It seems like for SNP/TDX we just need to register the shared/encrypted bits with KVM MMU and let it handle checking the #NPF flags, but can iterate on that for the next spin when we have a better idea what it should look like.

-Mike

#32 lore raw

RFC PATCH v7 5/64 KVM: x86: Add 'update_mem_attr' x86 op

In reply to Michael Roth

This callback will handle any platform-specific handling needed for converting pages between shared/private.

---
 arch/x86/include/asm/kvm-x86-ops.h |  1 +
 arch/x86/include/asm/kvm_host.h    |  2 ++
 arch/x86/kvm/mmu/mmu.c             | 10 ++++++++--
 3 files changed, 11 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index efae987cdce0..52f94a0ba5e9 100644
@@ -133,6 +133,7 @@ KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
 KVM_X86_OP_OPTIONAL_RET0(private_mem_enabled);
 KVM_X86_OP_OPTIONAL_RET0(fault_is_private);
+KVM_X86_OP_OPTIONAL_RET0(update_mem_attr)
 
 #undef KVM_X86_OP
 #undef KVM_X86_OP_OPTIONAL
arch/x86/include/asm/kvm_host.h +2 -0
index 92539708f062..13802389f0f9 100644
@@ -1637,6 +1637,8 @@ struct kvm_x86_ops {
 			     int root_level);
 	int (*private_mem_enabled)(struct kvm *kvm);
 	int (*fault_is_private)(struct kvm *kvm, gpa_t gpa, u64 error_code, bool *private_fault);
+	int (*update_mem_attr)(struct kvm_memory_slot *slot, unsigned int attr,
+			       gfn_t start, gfn_t end);
 
 	bool (*has_wbinvd_exit)(void);
 
arch/x86/kvm/mmu/mmu.c +8 -2
index 61a7c221b966..a0c41d391547 100644
@@ -7130,7 +7130,7 @@ static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm,
 {
 	unsigned long pages, mask;
 	gfn_t gfn, gfn_end, first, last;
-	int level;
+	int level, ret;
 	bool mixed;
 
 	/*
@@ -7153,7 +7153,7 @@ static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm,
 		linfo_set_mixed(gfn, slot, level, mixed);
 
 		if (first == last)
-			return;
+			goto out;
 
 		for (gfn = first + pages; gfn < last; gfn += pages)
 			linfo_set_mixed(gfn, slot, level, false);
@@ -7166,6 +7166,12 @@ static void kvm_update_lpage_private_shared_mixed(struct kvm *kvm,
 		mixed = mem_attrs_mixed(kvm, slot, level, attrs, gfn, gfn_end);
 		linfo_set_mixed(gfn, slot, level, mixed);
 	}
+
+out:
+	ret = static_call(kvm_x86_update_mem_attr)(slot, attrs, start, end);
+	if (ret)
+		pr_warn_ratelimited("Failed to update GFN range 0x%llx-0x%llx with attributes 0x%lx. Ret: %d\n",
+				    start, end, attrs, ret);
 }
 
 void kvm_arch_set_memory_attributes(struct kvm *kvm,
Signature · 1 line
2.25.1
maintainer
#33 lore raw

Re: [PATCH RFC v7 05/64] KVM: x86: Add 'update_mem_attr' x86 op

Quoted 1 line · Michael Roth

This callback will handle any platform-specific handling needed for

s/handle/do/

Quoted 26 lines

converting pages between shared/private.

---
 arch/x86/include/asm/kvm-x86-ops.h |  1 +
 arch/x86/include/asm/kvm_host.h    |  2 ++
 arch/x86/kvm/mmu/mmu.c             | 10 ++++++++--
 3 files changed, 11 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index efae987cdce0..52f94a0ba5e9 100644
@@ -133,6 +133,7 @@ KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
 KVM_X86_OP_OPTIONAL_RET0(private_mem_enabled);
 KVM_X86_OP_OPTIONAL_RET0(fault_is_private);
+KVM_X86_OP_OPTIONAL_RET0(update_mem_attr)
 
 #undef KVM_X86_OP
 #undef KVM_X86_OP_OPTIONAL
arch/x86/include/asm/kvm_host.h +0 -0
index 92539708f062..13802389f0f9 100644

arch/x86/include/asm/kvm_host.h struct kvm_x86_ops { · line 1641

Quoted 5 lines
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1637,6 +1637,8 @@ struct kvm_x86_ops {
 			     int root_level);
 	int (*private_mem_enabled)(struct kvm *kvm);
 	int (*fault_is_private)(struct kvm *kvm, gpa_t gpa, u64 error_code, bool *private_fault);
+	int (*update_mem_attr)(struct kvm_memory_slot *slot, unsigned int attr,
+			       gfn_t start, gfn_t end);

Does this one want to return bool too?

I guess I'll see later...

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#34 lore raw

RFC PATCH v7 6/64 KVM: x86: Add platform hooks for private memory invalidations

In reply to Michael Roth

In some cases, like with SEV-SNP, guest memory needs to be updated in a platform-specific manner before it can be safely freed back to the host. Add hooks to wire up handling of this sort to the invalidation notifiers for restricted memory.

Also issue invalidations of all allocated pages during notifier unregistration so that the pages are not left in an unusable state when they eventually get freed back to the host upon FD release.

---
 arch/x86/include/asm/kvm-x86-ops.h |  1 +
 arch/x86/include/asm/kvm_host.h    |  1 +
 arch/x86/kvm/mmu/mmu.c             |  5 +++++
 include/linux/kvm_host.h           |  2 ++
 mm/restrictedmem.c                 | 16 ++++++++++++++++
 virt/kvm/kvm_main.c                |  5 +++++
 6 files changed, 30 insertions(+)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index 52f94a0ba5e9..c71df44b0f02 100644
@@ -134,6 +134,7 @@ KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
 KVM_X86_OP_OPTIONAL_RET0(private_mem_enabled);
 KVM_X86_OP_OPTIONAL_RET0(fault_is_private);
 KVM_X86_OP_OPTIONAL_RET0(update_mem_attr)
+KVM_X86_OP_OPTIONAL(invalidate_restricted_mem)
 
 #undef KVM_X86_OP
 #undef KVM_X86_OP_OPTIONAL
arch/x86/include/asm/kvm_host.h +1 -0
index 13802389f0f9..9ef8d73455d9 100644
@@ -1639,6 +1639,7 @@ struct kvm_x86_ops {
 	int (*fault_is_private)(struct kvm *kvm, gpa_t gpa, u64 error_code, bool *private_fault);
 	int (*update_mem_attr)(struct kvm_memory_slot *slot, unsigned int attr,
 			       gfn_t start, gfn_t end);
+	void (*invalidate_restricted_mem)(struct kvm_memory_slot *slot, gfn_t start, gfn_t end);
 
 	bool (*has_wbinvd_exit)(void);
 
arch/x86/kvm/mmu/mmu.c +5 -0
index a0c41d391547..2713632e5061 100644
@@ -7183,3 +7183,8 @@ void kvm_arch_set_memory_attributes(struct kvm *kvm,
 		kvm_update_lpage_private_shared_mixed(kvm, slot, attrs,
 						      start, end);
 }
+
+void kvm_arch_invalidate_restricted_mem(struct kvm_memory_slot *slot, gfn_t start, gfn_t end)
+{
+	static_call_cond(kvm_x86_invalidate_restricted_mem)(slot, start, end);
+}
include/linux/kvm_host.h +2 -0
index f032d878e034..f72a2e0b8699 100644
@@ -2327,6 +2327,7 @@ void kvm_arch_set_memory_attributes(struct kvm *kvm,
 				    struct kvm_memory_slot *slot,
 				    unsigned long attrs,
 				    gfn_t start, gfn_t end);
+
 #else
 static inline void kvm_arch_set_memory_attributes(struct kvm *kvm,
 						  struct kvm_memory_slot *slot,
@@ -2366,6 +2367,7 @@ static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
 }
 
 void kvm_arch_memory_mce(struct kvm *kvm);
+void kvm_arch_invalidate_restricted_mem(struct kvm_memory_slot *slot, gfn_t start, gfn_t end);
 #endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
 
 #endif
mm/restrictedmem.c +16 -0
index 56953c204e5c..74fa2cfb8618 100644
@@ -54,6 +54,11 @@ static int restrictedmem_release(struct inode *inode, struct file *file)
 {
 	struct restrictedmem_data *data = inode->i_mapping->private_data;
 
+	pr_debug("%s: releasing memfd, invalidating page offsets 0x0-0x%llx\n",
+		 __func__, inode->i_size >> PAGE_SHIFT);
+	restrictedmem_invalidate_start(data, 0, inode->i_size >> PAGE_SHIFT);
+	restrictedmem_invalidate_end(data, 0, inode->i_size >> PAGE_SHIFT);
+
 	fput(data->memfd);
 	kfree(data);
 	return 0;
@@ -258,6 +263,17 @@ void restrictedmem_unregister_notifier(struct file *file,
 				       struct restrictedmem_notifier *notifier)
 {
 	struct restrictedmem_data *data = file->f_mapping->private_data;
+	struct inode *inode = file_inode(data->memfd);
+
+	/* TODO: this will issue notifications to all registered notifiers,
+	 * but it's only the one being unregistered that needs to process
+	 * invalidations for any ranges still allocated at this point in
+	 * time. For now this relies on KVM currently being the only notifier.
+	 */
+	pr_debug("%s: unregistering notifier, invalidating page offsets 0x0-0x%llx\n",
+		 __func__, inode->i_size >> PAGE_SHIFT);
+	restrictedmem_invalidate_start(data, 0, inode->i_size >> PAGE_SHIFT);
+	restrictedmem_invalidate_end(data, 0, inode->i_size >> PAGE_SHIFT);
 
 	mutex_lock(&data->lock);
 	list_del(&notifier->list);
virt/kvm/kvm_main.c +5 -0
index d2d829d23442..d2daa049e94a 100644
@@ -974,6 +974,9 @@ static void kvm_restrictedmem_invalidate_begin(struct restrictedmem_notifier *no
 					  &gfn_start, &gfn_end))
 		return;
 
+	pr_debug("%s: start: 0x%lx, end: 0x%lx, roffset: 0x%llx, gfn_start: 0x%llx, gfn_end: 0x%llx\n",
+		 __func__, start, end, slot->restricted_offset, gfn_start, gfn_end);
+
 	gfn_range.start = gfn_start;
 	gfn_range.end = gfn_end;
 	gfn_range.slot = slot;
@@ -988,6 +991,8 @@ static void kvm_restrictedmem_invalidate_begin(struct restrictedmem_notifier *no
 	if (kvm_unmap_gfn_range(kvm, &gfn_range))
 		kvm_flush_remote_tlbs(kvm);
 
+	kvm_arch_invalidate_restricted_mem(slot, gfn_start, gfn_end);
+
 	KVM_MMU_UNLOCK(kvm);
 	srcu_read_unlock(&kvm->srcu, idx);
 }
Signature · 1 line
2.25.1
maintainer
#35 lore raw

Re: [PATCH RFC v7 06/64] KVM: x86: Add platform hooks for private memory invalidations

arch/x86/kvm/mmu/mmu.c void kvm_arch_set_memory_attributes(struct kvm *kvm, · line 7189

Quoted 12 lines · Michael Roth
arch/x86/kvm/mmu/mmu.c +4 -0
index a0c41d391547..2713632e5061 100644
@@ -7183,3 +7183,8 @@ void kvm_arch_set_memory_attributes(struct kvm *kvm,
 		kvm_update_lpage_private_shared_mixed(kvm, slot, attrs,
 						      start, end);
 }
+
+void kvm_arch_invalidate_restricted_mem(struct kvm_memory_slot *slot, gfn_t start, gfn_t end)
+{
+	static_call_cond(kvm_x86_invalidate_restricted_mem)(slot, start, end);

Why _cond?

arch/x86/kvm/mmu/mmu.c void restrictedmem_unregister_notifier(struct file *file, · line 268

Quoted 7 lines
arch/x86/kvm/mmu/mmu.c +3 -0
@@ -258,6 +263,17 @@ void restrictedmem_unregister_notifier(struct file *file,
 				       struct restrictedmem_notifier *notifier)
 {
 	struct restrictedmem_data *data = file->f_mapping->private_data;
+	struct inode *inode = file_inode(data->memfd);
+
+	/* TODO: this will issue notifications to all registered notifiers,

First of all:

verify_comment_style: WARNING: Multi-line comment needs to start text on the second line:
[+ /* TODO: this will issue notifications to all registered notifiers,]

Then, if you only want to run the callbacks for the one going away only, why don't you simply do:

        mutex_lock(&data->lock);
        notifier->ops->invalidate_start(notifier, 0, inode->i_size >> PAGE_SHIFT);
        notifier->ops->invalidate_end(notifier, 0, inode->i_size >> PAGE_SHIFT);
        list_del(&notifier->list);
        mutex_unlock(&data->lock);

here?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#36 lore raw

Re: [PATCH RFC v7 06/64] KVM: x86: Add platform hooks for private memory invalidations

Quoted 15 lines · Borislav Petkov

arch/x86/kvm/mmu/mmu.c void kvm_arch_set_memory_attributes(struct kvm *kvm, · line 7189

Quoted 12 lines · Michael Roth
arch/x86/kvm/mmu/mmu.c +4 -0
index a0c41d391547..2713632e5061 100644
@@ -7183,3 +7183,8 @@ void kvm_arch_set_memory_attributes(struct kvm *kvm,
 		kvm_update_lpage_private_shared_mixed(kvm, slot, attrs,
 						      start, end);
 }
+
+void kvm_arch_invalidate_restricted_mem(struct kvm_memory_slot *slot, gfn_t start, gfn_t end)
+{
+	static_call_cond(kvm_x86_invalidate_restricted_mem)(slot, start, end);

Why _cond?

Since this hook is declared via KVM_X86_OP_OPTIONAL() (instead of KVM_X86_OP_OPTIONAL_RET0 like the previous hooks), the comment in kvm-x86-ops.h suggests this should be called via static_call_cond():

/*
* KVM_X86_OP() and KVM_X86_OP_OPTIONAL() are used to help generate
* both DECLARE/DEFINE_STATIC_CALL() invocations and
* "static_call_update()" calls.
*
* KVM_X86_OP_OPTIONAL() can be used for those functions that can have
* a NULL definition, for example if "static_call_cond()" will be used
* at the call sites. KVM_X86_OP_OPTIONAL_RET0() can be used likewise
* to make a definition optional, but in this case the default will
* be __static_call_return0.
*/

Quoted 17 lines

arch/x86/kvm/mmu/mmu.c void restrictedmem_unregister_notifier(struct file *file, · line 268

Quoted 8 lines
arch/x86/kvm/mmu/mmu.c +3 -0
@@ -258,6 +263,17 @@ void restrictedmem_unregister_notifier(struct file *file,
 				       struct restrictedmem_notifier *notifier)
 {
 	struct restrictedmem_data *data = file->f_mapping->private_data;
+	struct inode *inode = file_inode(data->memfd);
+
+	/* TODO: this will issue notifications to all registered notifiers,

First of all:

verify_comment_style: WARNING: Multi-line comment needs to start text on the second line:
[+ /* TODO: this will issue notifications to all registered notifiers,]

Then, if you only want to run the callbacks for the one going away only, why don't you simply do:

Quoted 7 lines
        mutex_lock(&data->lock);
        notifier->ops->invalidate_start(notifier, 0, inode->i_size >> PAGE_SHIFT);
        notifier->ops->invalidate_end(notifier, 0, inode->i_size >> PAGE_SHIFT);
        list_del(&notifier->list);
        mutex_unlock(&data->lock);

here?

That should do the trick. Thanks for the suggestion.

-Mike

Quoted 6 lines
Signature · 5 lines
-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#37 lore raw

Re: [PATCH RFC v7 06/64] KVM: x86: Add platform hooks for private memory invalidations

In reply to Michael Roth
Quoted 125 lines · Michael Roth

In some cases, like with SEV-SNP, guest memory needs to be updated in a platform-specific manner before it can be safely freed back to the host. Add hooks to wire up handling of this sort to the invalidation notifiers for restricted memory.

Also issue invalidations of all allocated pages during notifier unregistration so that the pages are not left in an unusable state when they eventually get freed back to the host upon FD release.

---
 arch/x86/include/asm/kvm-x86-ops.h |  1 +
 arch/x86/include/asm/kvm_host.h    |  1 +
 arch/x86/kvm/mmu/mmu.c             |  5 +++++
 include/linux/kvm_host.h           |  2 ++
 mm/restrictedmem.c                 | 16 ++++++++++++++++
 virt/kvm/kvm_main.c                |  5 +++++
 6 files changed, 30 insertions(+)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index 52f94a0ba5e9..c71df44b0f02 100644
@@ -134,6 +134,7 @@ KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
 KVM_X86_OP_OPTIONAL_RET0(private_mem_enabled);
 KVM_X86_OP_OPTIONAL_RET0(fault_is_private);
 KVM_X86_OP_OPTIONAL_RET0(update_mem_attr)
+KVM_X86_OP_OPTIONAL(invalidate_restricted_mem)
 
 #undef KVM_X86_OP
 #undef KVM_X86_OP_OPTIONAL
arch/x86/include/asm/kvm_host.h +1 -0
index 13802389f0f9..9ef8d73455d9 100644
@@ -1639,6 +1639,7 @@ struct kvm_x86_ops {
 	int (*fault_is_private)(struct kvm *kvm, gpa_t gpa, u64 error_code, bool *private_fault);
 	int (*update_mem_attr)(struct kvm_memory_slot *slot, unsigned int attr,
 			       gfn_t start, gfn_t end);
+	void (*invalidate_restricted_mem)(struct kvm_memory_slot *slot, gfn_t start, gfn_t end);
 
 	bool (*has_wbinvd_exit)(void);
 
arch/x86/kvm/mmu/mmu.c +5 -0
index a0c41d391547..2713632e5061 100644
@@ -7183,3 +7183,8 @@ void kvm_arch_set_memory_attributes(struct kvm *kvm,
 		kvm_update_lpage_private_shared_mixed(kvm, slot, attrs,
 						      start, end);
 }
+
+void kvm_arch_invalidate_restricted_mem(struct kvm_memory_slot *slot, gfn_t start, gfn_t end)
+{
+	static_call_cond(kvm_x86_invalidate_restricted_mem)(slot, start, end);
+}
include/linux/kvm_host.h +2 -0
index f032d878e034..f72a2e0b8699 100644
@@ -2327,6 +2327,7 @@ void kvm_arch_set_memory_attributes(struct kvm *kvm,
 				    struct kvm_memory_slot *slot,
 				    unsigned long attrs,
 				    gfn_t start, gfn_t end);
+
 #else
 static inline void kvm_arch_set_memory_attributes(struct kvm *kvm,
 						  struct kvm_memory_slot *slot,
@@ -2366,6 +2367,7 @@ static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
 }
 
 void kvm_arch_memory_mce(struct kvm *kvm);
+void kvm_arch_invalidate_restricted_mem(struct kvm_memory_slot *slot, gfn_t start, gfn_t end);
 #endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
 
 #endif
mm/restrictedmem.c +16 -0
index 56953c204e5c..74fa2cfb8618 100644
@@ -54,6 +54,11 @@ static int restrictedmem_release(struct inode *inode, struct file *file)
 {
 	struct restrictedmem_data *data = inode->i_mapping->private_data;
 
+	pr_debug("%s: releasing memfd, invalidating page offsets 0x0-0x%llx\n",
+		 __func__, inode->i_size >> PAGE_SHIFT);
+	restrictedmem_invalidate_start(data, 0, inode->i_size >> PAGE_SHIFT);
+	restrictedmem_invalidate_end(data, 0, inode->i_size >> PAGE_SHIFT);
+
 	fput(data->memfd);
 	kfree(data);
 	return 0;
@@ -258,6 +263,17 @@ void restrictedmem_unregister_notifier(struct file *file,
 				       struct restrictedmem_notifier *notifier)
 {
 	struct restrictedmem_data *data = file->f_mapping->private_data;
+	struct inode *inode = file_inode(data->memfd);
+
+	/* TODO: this will issue notifications to all registered notifiers,
+	 * but it's only the one being unregistered that needs to process
+	 * invalidations for any ranges still allocated at this point in
+	 * time. For now this relies on KVM currently being the only notifier.
+	 */
+	pr_debug("%s: unregistering notifier, invalidating page offsets 0x0-0x%llx\n",
+		 __func__, inode->i_size >> PAGE_SHIFT);
+	restrictedmem_invalidate_start(data, 0, inode->i_size >> PAGE_SHIFT);
+	restrictedmem_invalidate_end(data, 0, inode->i_size >> PAGE_SHIFT);
 
 	mutex_lock(&data->lock);
 	list_del(&notifier->list);
virt/kvm/kvm_main.c +3 -0
index d2d829d23442..d2daa049e94a 100644
@@ -974,6 +974,9 @@ static void kvm_restrictedmem_invalidate_begin(struct restrictedmem_notifier *no
 					  &gfn_start, &gfn_end))
 		return;
 
+	pr_debug("%s: start: 0x%lx, end: 0x%lx, roffset: 0x%llx, gfn_start: 0x%llx, gfn_end: 0x%llx\n",
+		 __func__, start, end, slot->restricted_offset, gfn_start, gfn_end);
+
 	gfn_range.start = gfn_start;
 	gfn_range.end = gfn_end;
 	gfn_range.slot = slot;

virt/kvm/kvm_main.c static void kvm_restrictedmem_invalidate_begin(struct restrictedmem_notifier *no · line 994

Quoted 4 lines
virt/kvm/kvm_main.c +1 -0
@@ -988,6 +991,8 @@ static void kvm_restrictedmem_invalidate_begin(struct restrictedmem_notifier *no
 	if (kvm_unmap_gfn_range(kvm, &gfn_range))
 		kvm_flush_remote_tlbs(kvm);
 
+	kvm_arch_invalidate_restricted_mem(slot, gfn_start, gfn_end);

Calling kvm_arch_invalidate_restricted_mem while the KVM MMU lock is taken causes problems, because taking said lock disables preemption. Within kvm_arch_invalidate_restricted_mem a few calls down, eventually vm_unmap_aliases is called which tries to lock a mutex, which shouldn't happen with preemption disabled. This causes a "scheduling while atomic" bug:

[  152.846596] BUG: scheduling while atomic: enarx/8302/0x00000002
[  152.846599] Modules linked in: nf_conntrack_netlink(E) xfrm_user(E) xfrm_algo(E) xt_addrtype(E) br_netfilter(E) xt_CHECKSUM(E) xt_MASQUERADE(E) xt_conntrack(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_tcpudp(E) nft_compat(E) nft_chain_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nf_tables(E) libcrc32c(E) nfnetlink(E) bridge(E) stp(E) llc(E) bonding(E) intel_rapl_msr(E) intel_rapl_common(E) amd64_edac(E) edac_mce_amd(E) kvm_amd(E) tun(E) ipmi_ssif(E) rfkill(E) overlay(E) ghash_clmulni_intel(E) sha512_ssse3(E) sha512_generic(E) aesni_intel(E) libaes(E) crypto_simd(E) cryptd(E) rapl(E) wmi_bmof(E) binfmt_misc(E) kvm(E) irqbypass(E) nls_ascii(E) nls_cp437(E) vfat(E) fat(E) snd_usb_audio(E) snd_usbmidi_lib(E) snd_hwdep(E) mc(E) snd_pcm(E) snd_seq_midi(E) snd_seq_midi_event(E) snd_rawmidi(E) snd_seq(E) ast(E) snd_seq_device(E) drm_vram_helper(E) drm_ttm_helper(E) snd_timer(E) ttm(E) joydev(E) snd(E) ccp(E) drm_kms_helper(E) soundcore(E) sg(E) i2c_algo_bit(E) rng_core(E)
[  152.846629]  k10temp(E) evdev(E) acpi_ipmi(E) ipmi_si(E) ipmi_devintf(E) ipmi_msghandler(E) acpi_cpufreq(E) button(E) squashfs(E) loop(E) sch_fq_codel(E) msr(E) parport_pc(E) ppdev(E) lp(E) ramoops(E) parport(E) reed_solomon(E) fuse(E) drm(E) efi_pstore(E) configfs(E) efivarfs(E) ip_tables(E) x_tables(E) autofs4(E) ext4(E) crc32c_generic(E) crc16(E) mbcache(E) jbd2(E) rndis_host(E) cdc_ether(E) usbnet(E) mii(E) hid_generic(E) usbhid(E) hid(E) sd_mod(E) t10_pi(E) crc64_rocksoft(E) crc64(E) crc_t10dif(E) crct10dif_generic(E) crct10dif_pclmul(E) crct10dif_common(E) crc32_pclmul(E) crc32c_intel(E) ahci(E) libahci(E) xhci_pci(E) libata(E) bnxt_en(E) xhci_hcd(E) scsi_mod(E) ptp(E) scsi_common(E) pps_core(E) usbcore(E) i2c_piix4(E) usb_common(E) wmi(E)
[  152.846657] Preemption disabled at:
[  152.846657] [<ffffffffc146a09a>] kvm_restrictedmem_invalidate_begin+0xba/0x1c0 [kvm]
[  152.846688] CPU: 108 PID: 8302 Comm: enarx Tainted: G        W   E      6.1.0-rc4+ #30
[  152.846690] Hardware name: Supermicro Super Server/H12SSL-NT, BIOS 2.4 04/14/2022
[  152.846691] Call Trace:
[  152.846692]  <TASK>
[  152.846694]  dump_stack_lvl+0x49/0x63
[  152.846695]  ? kvm_restrictedmem_invalidate_begin+0xba/0x1c0 [kvm]
[  152.846723]  dump_stack+0x10/0x16
[  152.846725]  __schedule_bug.cold+0x81/0x92
[  152.846727]  __schedule+0x809/0xa00
[  152.846729]  ? asm_sysvec_call_function+0x1b/0x20
[  152.846731]  schedule+0x6b/0xf0
[  152.846733]  schedule_preempt_disabled+0x18/0x30
[  152.846735]  __mutex_lock.constprop.0+0x723/0x750
[  152.846738]  ? smp_call_function_many_cond+0xc1/0x2e0
[  152.846740]  __mutex_lock_slowpath+0x13/0x20
[  152.846742]  mutex_lock+0x49/0x60
[  152.846744]  _vm_unmap_aliases+0x10e/0x160
[  152.846746]  vm_unmap_aliases+0x19/0x20
[  152.846748]  change_page_attr_set_clr+0xb7/0x1c0
[  152.846751]  set_memory_p+0x29/0x30
[  152.846753]  rmpupdate+0xd5/0x110
[  152.846756]  rmp_make_shared+0xb7/0xc0
[  152.846758]  snp_make_page_shared.constprop.0+0x4c/0x90 [kvm_amd]
[  152.846765]  sev_invalidate_private_range+0x156/0x330 [kvm_amd]
[  152.846770]  ? kvm_unmap_gfn_range+0xef/0x100 [kvm]
[  152.846801]  kvm_arch_invalidate_restricted_mem+0xe/0x20 [kvm]
[  152.846829]  kvm_restrictedmem_invalidate_begin+0x106/0x1c0 [kvm]
[  152.846856]  restrictedmem_unregister_notifier+0x74/0x150
[  152.846859]  kvm_free_memslot+0x6b/0x80 [kvm]
[  152.846885]  kvm_free_memslots.part.0+0x47/0x70 [kvm]
[  152.846911]  kvm_destroy_vm+0x222/0x320 [kvm]
[  152.846937]  kvm_put_kvm+0x2a/0x50 [kvm]
[  152.846964]  kvm_vm_release+0x22/0x30 [kvm]
[  152.846990]  __fput+0xa8/0x280
[  152.846992]  ____fput+0xe/0x20
[  152.846994]  task_work_run+0x61/0xb0
[  152.846996]  do_exit+0x362/0xb30
[  152.846998]  ? tomoyo_path_number_perm+0x6f/0x200
[  152.847001]  do_group_exit+0x38/0xa0
[  152.847003]  get_signal+0x999/0x9c0
[  152.847005]  arch_do_signal_or_restart+0x37/0x7e0
[  152.847008]  ? __might_fault+0x26/0x30
[  152.847010]  ? __rseq_handle_notify_resume+0xd5/0x4f0
[  152.847013]  exit_to_user_mode_prepare+0xd3/0x170
[  152.847016]  syscall_exit_to_user_mode+0x26/0x50
[  152.847019]  do_syscall_64+0x48/0x90
[  152.847020]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  152.847022] RIP: 0033:0x7fa345f1aaff
[  152.847023] Code: Unable to access opcode bytes at 0x7fa345f1aad5.
[  152.847024] RSP: 002b:00007fff99d6c050 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  152.847026] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 00007fa345f1aaff
[  152.847027] RDX: 00007fff99d6c188 RSI: 00000000c008aeba RDI: 0000000000000006
[  152.847028] RBP: 00007fff99576000 R08: 0000000000000000 R09: 0000000000000000
[  152.847029] R10: 0000000001680000 R11: 0000000000000246 R12: 00007fff99d752c0
[  152.847030] R13: 00007fff99d75270 R14: 0000000000000000 R15: 00007fff99577000
[  152.847032]  </TASK>

This bug can be triggered by destroying multiple SNP VMs at the same time.

Quoted 8 lines
virt/kvm/kvm_main.c +1 -0
+
 	KVM_MMU_UNLOCK(kvm);
 	srcu_read_unlock(&kvm->srcu, idx);
 }
Signature · 1 line
2.25.1

Regards, Tom

#38 lore raw

Re: [PATCH RFC v7 06/64] KVM: x86: Add platform hooks for private memory invalidations

Quoted 199 lines · Tom Dohrmann
Quoted 125 lines · Michael Roth

In some cases, like with SEV-SNP, guest memory needs to be updated in a platform-specific manner before it can be safely freed back to the host. Add hooks to wire up handling of this sort to the invalidation notifiers for restricted memory.

Also issue invalidations of all allocated pages during notifier unregistration so that the pages are not left in an unusable state when they eventually get freed back to the host upon FD release.

---
 arch/x86/include/asm/kvm-x86-ops.h |  1 +
 arch/x86/include/asm/kvm_host.h    |  1 +
 arch/x86/kvm/mmu/mmu.c             |  5 +++++
 include/linux/kvm_host.h           |  2 ++
 mm/restrictedmem.c                 | 16 ++++++++++++++++
 virt/kvm/kvm_main.c                |  5 +++++
 6 files changed, 30 insertions(+)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index 52f94a0ba5e9..c71df44b0f02 100644
@@ -134,6 +134,7 @@ KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
 KVM_X86_OP_OPTIONAL_RET0(private_mem_enabled);
 KVM_X86_OP_OPTIONAL_RET0(fault_is_private);
 KVM_X86_OP_OPTIONAL_RET0(update_mem_attr)
+KVM_X86_OP_OPTIONAL(invalidate_restricted_mem)
 
 #undef KVM_X86_OP
 #undef KVM_X86_OP_OPTIONAL
arch/x86/include/asm/kvm_host.h +1 -0
index 13802389f0f9..9ef8d73455d9 100644
@@ -1639,6 +1639,7 @@ struct kvm_x86_ops {
 	int (*fault_is_private)(struct kvm *kvm, gpa_t gpa, u64 error_code, bool *private_fault);
 	int (*update_mem_attr)(struct kvm_memory_slot *slot, unsigned int attr,
 			       gfn_t start, gfn_t end);
+	void (*invalidate_restricted_mem)(struct kvm_memory_slot *slot, gfn_t start, gfn_t end);
 
 	bool (*has_wbinvd_exit)(void);
 
arch/x86/kvm/mmu/mmu.c +5 -0
index a0c41d391547..2713632e5061 100644
@@ -7183,3 +7183,8 @@ void kvm_arch_set_memory_attributes(struct kvm *kvm,
 		kvm_update_lpage_private_shared_mixed(kvm, slot, attrs,
 						      start, end);
 }
+
+void kvm_arch_invalidate_restricted_mem(struct kvm_memory_slot *slot, gfn_t start, gfn_t end)
+{
+	static_call_cond(kvm_x86_invalidate_restricted_mem)(slot, start, end);
+}
include/linux/kvm_host.h +2 -0
index f032d878e034..f72a2e0b8699 100644
@@ -2327,6 +2327,7 @@ void kvm_arch_set_memory_attributes(struct kvm *kvm,
 				    struct kvm_memory_slot *slot,
 				    unsigned long attrs,
 				    gfn_t start, gfn_t end);
+
 #else
 static inline void kvm_arch_set_memory_attributes(struct kvm *kvm,
 						  struct kvm_memory_slot *slot,
@@ -2366,6 +2367,7 @@ static inline int kvm_restricted_mem_get_pfn(struct kvm_memory_slot *slot,
 }
 
 void kvm_arch_memory_mce(struct kvm *kvm);
+void kvm_arch_invalidate_restricted_mem(struct kvm_memory_slot *slot, gfn_t start, gfn_t end);
 #endif /* CONFIG_HAVE_KVM_RESTRICTED_MEM */
 
 #endif
mm/restrictedmem.c +16 -0
index 56953c204e5c..74fa2cfb8618 100644
@@ -54,6 +54,11 @@ static int restrictedmem_release(struct inode *inode, struct file *file)
 {
 	struct restrictedmem_data *data = inode->i_mapping->private_data;
 
+	pr_debug("%s: releasing memfd, invalidating page offsets 0x0-0x%llx\n",
+		 __func__, inode->i_size >> PAGE_SHIFT);
+	restrictedmem_invalidate_start(data, 0, inode->i_size >> PAGE_SHIFT);
+	restrictedmem_invalidate_end(data, 0, inode->i_size >> PAGE_SHIFT);
+
 	fput(data->memfd);
 	kfree(data);
 	return 0;
@@ -258,6 +263,17 @@ void restrictedmem_unregister_notifier(struct file *file,
 				       struct restrictedmem_notifier *notifier)
 {
 	struct restrictedmem_data *data = file->f_mapping->private_data;
+	struct inode *inode = file_inode(data->memfd);
+
+	/* TODO: this will issue notifications to all registered notifiers,
+	 * but it's only the one being unregistered that needs to process
+	 * invalidations for any ranges still allocated at this point in
+	 * time. For now this relies on KVM currently being the only notifier.
+	 */
+	pr_debug("%s: unregistering notifier, invalidating page offsets 0x0-0x%llx\n",
+		 __func__, inode->i_size >> PAGE_SHIFT);
+	restrictedmem_invalidate_start(data, 0, inode->i_size >> PAGE_SHIFT);
+	restrictedmem_invalidate_end(data, 0, inode->i_size >> PAGE_SHIFT);
 
 	mutex_lock(&data->lock);
 	list_del(&notifier->list);
virt/kvm/kvm_main.c +3 -0
index d2d829d23442..d2daa049e94a 100644
@@ -974,6 +974,9 @@ static void kvm_restrictedmem_invalidate_begin(struct restrictedmem_notifier *no
 					  &gfn_start, &gfn_end))
 		return;
 
+	pr_debug("%s: start: 0x%lx, end: 0x%lx, roffset: 0x%llx, gfn_start: 0x%llx, gfn_end: 0x%llx\n",
+		 __func__, start, end, slot->restricted_offset, gfn_start, gfn_end);
+
 	gfn_range.start = gfn_start;
 	gfn_range.end = gfn_end;
 	gfn_range.slot = slot;

virt/kvm/kvm_main.c static void kvm_restrictedmem_invalidate_begin(struct restrictedmem_notifier *no · line 994

Quoted 4 lines
virt/kvm/kvm_main.c +1 -0
@@ -988,6 +991,8 @@ static void kvm_restrictedmem_invalidate_begin(struct restrictedmem_notifier *no
 	if (kvm_unmap_gfn_range(kvm, &gfn_range))
 		kvm_flush_remote_tlbs(kvm);
 
+	kvm_arch_invalidate_restricted_mem(slot, gfn_start, gfn_end);

Calling kvm_arch_invalidate_restricted_mem while the KVM MMU lock is taken causes problems, because taking said lock disables preemption. Within kvm_arch_invalidate_restricted_mem a few calls down, eventually vm_unmap_aliases is called which tries to lock a mutex, which shouldn't happen with preemption disabled. This causes a "scheduling while atomic" bug:

[  152.846596] BUG: scheduling while atomic: enarx/8302/0x00000002
[  152.846599] Modules linked in: nf_conntrack_netlink(E) xfrm_user(E) xfrm_algo(E) xt_addrtype(E) br_netfilter(E) xt_CHECKSUM(E) xt_MASQUERADE(E) xt_conntrack(E) ipt_REJECT(E) nf_reject_ipv4(E) xt_tcpudp(E) nft_compat(E) nft_chain_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) nf_tables(E) libcrc32c(E) nfnetlink(E) bridge(E) stp(E) llc(E) bonding(E) intel_rapl_msr(E) intel_rapl_common(E) amd64_edac(E) edac_mce_amd(E) kvm_amd(E) tun(E) ipmi_ssif(E) rfkill(E) overlay(E) ghash_clmulni_intel(E) sha512_ssse3(E) sha512_generic(E) aesni_intel(E) libaes(E) crypto_simd(E) cryptd(E) rapl(E) wmi_bmof(E) binfmt_misc(E) kvm(E) irqbypass(E) nls_ascii(E) nls_cp437(E) vfat(E) fat(E) snd_usb_audio(E) snd_usbmidi_lib(E) snd_hwdep(E) mc(E) snd_pcm(E) snd_seq_midi(E) snd_seq_midi_event(E) snd_rawmidi(E) snd_seq(E) ast(E) snd_seq_device(E) drm_vram_helper(E) drm_ttm_helper(E) snd_timer(E) ttm(E) joydev(E) snd(E) ccp(E) drm_kms_helper(E) soundcore(E) sg(E) i2c_algo_bit(E) rng_core(E)
[  152.846629]  k10temp(E) evdev(E) acpi_ipmi(E) ipmi_si(E) ipmi_devintf(E) ipmi_msghandler(E) acpi_cpufreq(E) button(E) squashfs(E) loop(E) sch_fq_codel(E) msr(E) parport_pc(E) ppdev(E) lp(E) ramoops(E) parport(E) reed_solomon(E) fuse(E) drm(E) efi_pstore(E) configfs(E) efivarfs(E) ip_tables(E) x_tables(E) autofs4(E) ext4(E) crc32c_generic(E) crc16(E) mbcache(E) jbd2(E) rndis_host(E) cdc_ether(E) usbnet(E) mii(E) hid_generic(E) usbhid(E) hid(E) sd_mod(E) t10_pi(E) crc64_rocksoft(E) crc64(E) crc_t10dif(E) crct10dif_generic(E) crct10dif_pclmul(E) crct10dif_common(E) crc32_pclmul(E) crc32c_intel(E) ahci(E) libahci(E) xhci_pci(E) libata(E) bnxt_en(E) xhci_hcd(E) scsi_mod(E) ptp(E) scsi_common(E) pps_core(E) usbcore(E) i2c_piix4(E) usb_common(E) wmi(E)
[  152.846657] Preemption disabled at:
[  152.846657] [<ffffffffc146a09a>] kvm_restrictedmem_invalidate_begin+0xba/0x1c0 [kvm]
[  152.846688] CPU: 108 PID: 8302 Comm: enarx Tainted: G        W   E      6.1.0-rc4+ #30
[  152.846690] Hardware name: Supermicro Super Server/H12SSL-NT, BIOS 2.4 04/14/2022
[  152.846691] Call Trace:
[  152.846692]  <TASK>
[  152.846694]  dump_stack_lvl+0x49/0x63
[  152.846695]  ? kvm_restrictedmem_invalidate_begin+0xba/0x1c0 [kvm]
[  152.846723]  dump_stack+0x10/0x16
[  152.846725]  __schedule_bug.cold+0x81/0x92
[  152.846727]  __schedule+0x809/0xa00
[  152.846729]  ? asm_sysvec_call_function+0x1b/0x20
[  152.846731]  schedule+0x6b/0xf0
[  152.846733]  schedule_preempt_disabled+0x18/0x30
[  152.846735]  __mutex_lock.constprop.0+0x723/0x750
[  152.846738]  ? smp_call_function_many_cond+0xc1/0x2e0
[  152.846740]  __mutex_lock_slowpath+0x13/0x20
[  152.846742]  mutex_lock+0x49/0x60
[  152.846744]  _vm_unmap_aliases+0x10e/0x160
[  152.846746]  vm_unmap_aliases+0x19/0x20
[  152.846748]  change_page_attr_set_clr+0xb7/0x1c0
[  152.846751]  set_memory_p+0x29/0x30
[  152.846753]  rmpupdate+0xd5/0x110
[  152.846756]  rmp_make_shared+0xb7/0xc0
[  152.846758]  snp_make_page_shared.constprop.0+0x4c/0x90 [kvm_amd]
[  152.846765]  sev_invalidate_private_range+0x156/0x330 [kvm_amd]
[  152.846770]  ? kvm_unmap_gfn_range+0xef/0x100 [kvm]
[  152.846801]  kvm_arch_invalidate_restricted_mem+0xe/0x20 [kvm]
[  152.846829]  kvm_restrictedmem_invalidate_begin+0x106/0x1c0 [kvm]
[  152.846856]  restrictedmem_unregister_notifier+0x74/0x150
[  152.846859]  kvm_free_memslot+0x6b/0x80 [kvm]
[  152.846885]  kvm_free_memslots.part.0+0x47/0x70 [kvm]
[  152.846911]  kvm_destroy_vm+0x222/0x320 [kvm]
[  152.846937]  kvm_put_kvm+0x2a/0x50 [kvm]
[  152.846964]  kvm_vm_release+0x22/0x30 [kvm]
[  152.846990]  __fput+0xa8/0x280
[  152.846992]  ____fput+0xe/0x20
[  152.846994]  task_work_run+0x61/0xb0
[  152.846996]  do_exit+0x362/0xb30
[  152.846998]  ? tomoyo_path_number_perm+0x6f/0x200
[  152.847001]  do_group_exit+0x38/0xa0
[  152.847003]  get_signal+0x999/0x9c0
[  152.847005]  arch_do_signal_or_restart+0x37/0x7e0
[  152.847008]  ? __might_fault+0x26/0x30
[  152.847010]  ? __rseq_handle_notify_resume+0xd5/0x4f0
[  152.847013]  exit_to_user_mode_prepare+0xd3/0x170
[  152.847016]  syscall_exit_to_user_mode+0x26/0x50
[  152.847019]  do_syscall_64+0x48/0x90
[  152.847020]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  152.847022] RIP: 0033:0x7fa345f1aaff
[  152.847023] Code: Unable to access opcode bytes at 0x7fa345f1aad5.
[  152.847024] RSP: 002b:00007fff99d6c050 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  152.847026] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 00007fa345f1aaff
[  152.847027] RDX: 00007fff99d6c188 RSI: 00000000c008aeba RDI: 0000000000000006
[  152.847028] RBP: 00007fff99576000 R08: 0000000000000000 R09: 0000000000000000
[  152.847029] R10: 0000000001680000 R11: 0000000000000246 R12: 00007fff99d752c0
[  152.847030] R13: 00007fff99d75270 R14: 0000000000000000 R15: 00007fff99577000
[  152.847032]  </TASK>
Quoted 1 line

This bug can be triggered by destroying multiple SNP VMs at the same time.

I can also reproduce this one.

If I do "cargo test -- --test-threads=1", then this does not happen.

Even then I get this:

[  232.054359] SEV-SNP launch update failed, ret: 0xfffffffb, fw_error: 0x16
[  232.061466] ------------[ cut here ]------------
[  232.061467] WARNING: CPU: 18 PID: 2436 at arch/x86/kvm/../../../virt/kvm/kvm_main.c:3665 mark_page_dirty_in_slot+0x99/0xd0
[  232.061472] Modules linked in: af_packet irdma intel_rapl_msr i40e ib_uverbs ib_core dell_smbios wmi_bmof dell_wmi_descriptor evdev mac_hid dcdbas amd64_edac edac_mce_amd edac_core intel_rapl_common crc32_pclmul polyval_clmulni polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 sha512_generic aesni_intel libaes crypto_simd cryptd rapl deflate efi_pstore bonding tls cfg80211 rfkill ip6_tables xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp ip6t_rpfilter ipt_rpfilter xt_pkttype nft_compat nf_tables libcrc32c sch_fq_codel nfnetlink atkbd libps2 serio vivaldi_fmap loop tun tap macvlan bridge stp llc ipmi_ssif ipmi_watchdog dm_round_robin dm_multipath kvm_amd mgag200 drm_shmem_helper drm_kms_helper i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt ice ptp pps_core sp5100_tco watchdog k10temp i2c_piix4 ptdma virt_dma hed tpm_crb wmi acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler tpm_tis tpm_tis_core tpm acpi_power_meter nls_iso8859_1 nls_cp437 vfat fat
[  232.061525]  tiny_power_button button ccp rng_core drm pstore fuse backlight i2c_core configfs efivarfs ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache jbd2 sd_mod ahci libahci xhci_pci xhci_pci_renesas libata xhci_hcd nvme nvme_core scsi_mod usbcore t10_pi crc32c_intel crc64_rocksoft crc64 crc_t10dif scsi_common crct10dif_generic crct10dif_pclmul crct10dif_common usb_common rtc_cmos dm_mod dax
[  232.061547] CPU: 18 PID: 2436 Comm: enarx Not tainted 6.1.0-rc4 #1-NixOS
[  232.061549] Hardware name: Dell Inc. PowerEdge R6515/068NXX, BIOS 2.6.6 01/13/2022
[  232.061550] RIP: 0010:mark_page_dirty_in_slot+0x99/0xd0
[  232.061552] Code: 83 04 01 00 00 4c 2b a3 b0 00 00 00 85 d2 74 27 c1 e6 10 5b 49 8d bd f8 19 00 00 5d 4c 89 e2 09 c6 41 5c 41 5d e9 07 aa 00 00 <0f> 0b 5b 5d 41 5c 41 5d c3 cc cc cc cc 48 8b 83 c0 00 00 00 49 63
[  232.061553] RSP: 0018:ffffc169265978f0 EFLAGS: 00010246
[  232.061555] RAX: 0000000080000000 RBX: ffffa0c045c49600 RCX: 0000000000000000
[  232.061556] RDX: 0000000000000001 RSI: ffffffff92d076bd RDI: 00000000ffffffff
[  232.061557] RBP: ffffc1692177d000 R08: 0000000000000001 R09: 0000000000001000
[  232.061557] R10: 0000000000000001 R11: 0000000000000001 R12: 00000000000ffe02
[  232.061558] R13: 0000000000000000 R14: ffffc1692177d000 R15: 00007f6cacf89000
[  232.061559] FS:  00007f6cacfc3540(0000) GS:ffffa0cfffa80000(0000) knlGS:0000000000000000
[  232.061560] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  232.061561] CR2: 00007f6cacf89020 CR3: 000000208c9c6005 CR4: 0000000000770ee0
[  232.061561] PKRU: 55555554
[  232.061562] Call Trace:
[  232.061563]  <TASK>
[  232.061566]  __kvm_write_guest_page+0xac/0xf0
[  232.061569]  snp_launch_update_gfn_handler.cold+0x5e/0xfe [kvm_amd]
[  232.061578]  kvm_vm_do_hva_range_op+0x142/0x1c0
[  232.061579]  ? sev_launch_update_gfn_handler+0x470/0x470 [kvm_amd]
[  232.061585]  sev_mem_enc_ioctl+0x4dd/0x1270 [kvm_amd]
[  232.061591]  ? sev_pin_memory+0x159/0x1a0 [kvm_amd]
[  232.061595]  ? sev_mem_enc_register_region+0xe3/0x130 [kvm_amd]
[  232.061602]  kvm_arch_vm_ioctl+0x6a6/0xc20
[  232.061604]  ? __blk_flush_plug+0x102/0x160
[  232.061606]  ? _raw_spin_unlock_irqrestore+0x23/0x40
[  232.061609]  ? get_page_from_freelist+0x1451/0x15a0
[  232.061612]  ? __mod_node_page_state+0x7c/0xb0
[  232.061615]  ? try_charge_memcg+0x466/0x800
[  232.061618]  ? __mod_node_page_state+0x7c/0xb0
[  232.061619]  ? __mod_memcg_lruvec_state+0x6e/0xd0
[  232.061620]  kvm_vm_ioctl+0x7ba/0x1290
[  232.061622]  ? folio_add_lru+0x6e/0xa0
[  232.061624]  ? _raw_spin_unlock+0x15/0x30
[  232.061626]  ? __handle_mm_fault+0xace/0xc70
[  232.061629]  ? handle_mm_fault+0xb2/0x2a0
[  232.061630]  __x64_sys_ioctl+0x8a/0xc0
[  232.061634]  do_syscall_64+0x3b/0x90
[  232.061636]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  232.061638] RIP: 0033:0x7f6cad0c1e37
[  232.061659] Code: ff ff 48 89 d8 5b 5d 41 5c c3 66 0f 1f 84 00 00 00 00 00 48 89 e8 48 f7 d8 48 39 c3 0f 92 c0 eb c9 66 90 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d b1 0f 0f 00 f7 d8 64 89 01 48
[  232.061660] RSP: 002b:00007ffee9081668 EFLAGS: 00000216 ORIG_RAX: 0000000000000010
[  232.061661] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f6cad0c1e37
[  232.061662] RDX: 00007ffee9081748 RSI: 00000000c008aeba RDI: 0000000000000004
[  232.061663] RBP: 00007ffee888e000 R08: 0000000000000006 R09: 0000000000000000
[  232.061663] R10: 0000000000001000 R11: 0000000000000216 R12: 00007ffee908a260
[  232.061664] R13: 00007ffee908a210 R14: 0000000000000000 R15: 00007ffee888f000
[  232.061665]  </TASK>
[  232.061666] ---[ end trace 0000000000000000 ]---
[  232.759176] SEV-SNP launch update failed, ret: 0xfffffffb, fw_error: 0x16
[  233.449477] SEV-SNP launch update failed, ret: 0xfffffffb, fw_error: 0x16
[  235.284941] SEV-SNP launch update failed, ret: 0xfffffffb, fw_error: 0x16
[  237.129469] SEV-SNP launch update failed, ret: 0xfffffffb, fw_error: 0x16
[  237.978020] SEV-SNP launch update failed, ret: 0xfffffffb, fw_error: 0x16
[  239.034002] SEV-SNP launch update failed, ret: 0xfffffffb, fw_error: 0x16
[  239.598148] SEV-SNP launch update failed, ret: 0xfffffffb, fw_error: 0x16
[  240.152122] SEV-SNP launch update failed, ret: 0xfffffffb, fw_error: 0x16
[  240.718185] SEV-SNP launch update failed, ret: 0xfffffffb, fw_error: 0x16
[  241.271235] SEV-SNP launch update failed, ret: 0xfffffffb, fw_error: 0x16
[  241.836286] SEV-SNP launch update failed, ret: 0xfffffffb, fw_error: 0x16
[  242.390345] SEV-SNP launch update failed, ret: 0xfffffffb, fw_error: 0x16
[  242.962499] SEV-SNP launch update failed, ret: 0xfffffffb, fw_error: 0x16
[  243.501430] SEV-SNP launch update failed, ret: 0xfffffffb, fw_error: 0x16
[  244.093547] SEV-SNP launch update failed, ret: 0xfffffffb, fw_error: 0x16

There is no cumulative klog output, i.e. only first run of cargo test emits this. Also software runs without issues, including attestation and everything.

BR, Jarkko

#39 lore raw

RFC PATCH v7 7/64 KVM: SEV: Handle KVM_HC_MAP_GPA_RANGE hypercall

In reply to Michael Roth

From: Nikunj A Dadhania <[email protected]>

KVM_HC_MAP_GPA_RANGE hypercall is used by the SEV guest to notify a change in the page encryption status to the hypervisor.

The hypercall exits to userspace with KVM_EXIT_HYPERCALL exit code, currently this is used for explicit memory conversion between shared/private for memfd based private memory.

---
 arch/x86/kvm/x86.c  | 8 ++++++++
 virt/kvm/kvm_main.c | 1 +
 2 files changed, 9 insertions(+)
arch/x86/kvm/x86.c +8 -0
index bb6adb216054..732f9cbbadb5 100644
@@ -9649,6 +9649,7 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
 		break;
 	case KVM_HC_MAP_GPA_RANGE: {
 		u64 gpa = a0, npages = a1, attrs = a2;
+		struct kvm_memory_slot *slot;
 
 		ret = -KVM_ENOSYS;
 		if (!(vcpu->kvm->arch.hypercall_exit_enabled & (1 << KVM_HC_MAP_GPA_RANGE)))
@@ -9660,6 +9661,13 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
 			break;
 		}
 
+		slot = kvm_vcpu_gfn_to_memslot(vcpu, gpa_to_gfn(gpa));
+		if (!vcpu->kvm->arch.upm_mode ||
+		    !kvm_slot_can_be_private(slot)) {
+			ret = 0;
+			break;
+		}
+
 		vcpu->run->exit_reason        = KVM_EXIT_HYPERCALL;
 		vcpu->run->hypercall.nr       = KVM_HC_MAP_GPA_RANGE;
 		vcpu->run->hypercall.args[0]  = gpa;
virt/kvm/kvm_main.c +1 -0
index d2daa049e94a..73bf0bdedb59 100644
@@ -2646,6 +2646,7 @@ struct kvm_memory_slot *kvm_vcpu_gfn_to_memslot(struct kvm_vcpu *vcpu, gfn_t gfn
 
 	return NULL;
 }
+EXPORT_SYMBOL_GPL(kvm_vcpu_gfn_to_memslot);
 
 bool kvm_is_visible_gfn(struct kvm *kvm, gfn_t gfn)
 {
Signature · 1 line
2.25.1
maintainer
#40 lore raw

Re: [PATCH RFC v7 07/64] KVM: SEV: Handle KVM_HC_MAP_GPA_RANGE hypercall

Quoted 8 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

KVM_HC_MAP_GPA_RANGE hypercall is used by the SEV guest to notify a change in the page encryption status to the hypervisor.

The hypercall exits to userspace with KVM_EXIT_HYPERCALL exit code, currently this is used for explicit memory conversion between shared/private for memfd based private memory.

So Tom and I spent a while to figure out what this is doing...

Please explain in more detail what that is. Like the hypercall gets ignored for memslots which cannot be private...?

And what's the story with supporting UPM with SEV{,-ES} guests?

In general, this text needs more background and why this is being done.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#41 lore raw

Re: [PATCH RFC v7 07/64] KVM: SEV: Handle KVM_HC_MAP_GPA_RANGE hypercall

Quoted 6 lines · Borislav Petkov
Quoted 5 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

KVM_HC_MAP_GPA_RANGE hypercall is used by the SEV guest to notify a change in the page encryption status to the hypervisor.

Quoted 8 lines
Quoted 3 lines

The hypercall exits to userspace with KVM_EXIT_HYPERCALL exit code, currently this is used for explicit memory conversion between shared/private for memfd based private memory.

So Tom and I spent a while to figure out what this is doing...

Please explain in more detail what that is. Like the hypercall gets ignored for memslots which cannot be private...?

Don't bother, just drop the patch. It's perfectly legal for userspace to create the private memslot in response to a guest request.

#42 lore raw

Re: [PATCH RFC v7 07/64] KVM: SEV: Handle KVM_HC_MAP_GPA_RANGE hypercall

Quoted 7 lines · Sean Christopherson
Quoted 6 lines · Borislav Petkov
Quoted 5 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

KVM_HC_MAP_GPA_RANGE hypercall is used by the SEV guest to notify a change in the page encryption status to the hypervisor.

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

The hypercall exits to userspace with KVM_EXIT_HYPERCALL exit code, currently this is used for explicit memory conversion between shared/private for memfd based private memory.

So Tom and I spent a while to figure out what this is doing...

Please explain in more detail what that is. Like the hypercall gets ignored for memslots which cannot be private...?

This was required when we were using per memslot bitmap for storing the private information, mem_attr_array is not dependent on memslot anymore.

Quoted 2 lines

Don't bother, just drop the patch.

Agree, we can drop this. I have tested SEV without this patch.

Quoted 1 line

It's perfectly legal for userspace to create the private memslot in response > to a guest request.

Sean, did not understand this part, how could a memslot be created on a guest request?

Regards
Nikunj

#43 lore raw

Re: [PATCH RFC v7 07/64] KVM: SEV: Handle KVM_HC_MAP_GPA_RANGE hypercall

Quoted 5 lines · Nikunj A. Dadhania
Quoted 2 lines · Sean Christopherson

It's perfectly legal for userspace to create the private memslot in response to a guest request.

Sean, did not understand this part, how could a memslot be created on a guest request?

KVM_HC_MAP_GPA_RANGE gets routed to host userspace, at that point userspace can take any action it wants to satisfy the guest request. E.g. a userspace+guest setup could define memory as shared by default, and only create KVM_MEM_PRIVATE memslots for memory that the guest explicitly requests to be mapped private.

I don't anticipate any real world use cases actually doing something like that, but I also don't see any value in going out of our way to disallow it. Normally I like to be conservative when it comes to KVM's uAPI, e.g. allow the minimum needed to support known use cases, but restricting KVM_HC_MAP_GPA_RANGE doesn't actually achieve anything and just makes things more complex for KVM. E.g. the behavior is non-deterministic from KVM's perspective if a userspace memslots update is in-progress.

#44 lore raw

Re: [PATCH RFC v7 07/64] KVM: SEV: Handle KVM_HC_MAP_GPA_RANGE hypercall

In reply to Michael Roth

arch/x86/kvm/x86.c int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)

Quoted 21 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

KVM_HC_MAP_GPA_RANGE hypercall is used by the SEV guest to notify a change in the page encryption status to the hypervisor.

The hypercall exits to userspace with KVM_EXIT_HYPERCALL exit code, currently this is used for explicit memory conversion between shared/private for memfd based private memory.

---
 arch/x86/kvm/x86.c  | 8 ++++++++
 virt/kvm/kvm_main.c | 1 +
 2 files changed, 9 insertions(+)
arch/x86/kvm/x86.c +0 -0
index bb6adb216054..732f9cbbadb5 100644
@@ -9649,6 +9649,7 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)

Couldn't find a better commit to comment on:
when the guest has the ptp-kvm module, it will issue a KVM_HC_CLOCK_PAIRING hypercall. This will pass sev_es_validate_vmgexit validation and end up in this function where kvm_pv_clock_pairing() is called, and that calls kvm_write_guest(). This results in a CPU soft-lockup, at least in my testing.

Are there any emulated hypercalls that make sense for snp guests? We should block at least the ones that definitely don't work.

Jeremi

Quoted 36 lines
arch/x86/kvm/x86.c +8 -0
 		break;
 	case KVM_HC_MAP_GPA_RANGE: {
 		u64 gpa = a0, npages = a1, attrs = a2;
+		struct kvm_memory_slot *slot;
 
 		ret = -KVM_ENOSYS;
 		if (!(vcpu->kvm->arch.hypercall_exit_enabled & (1 << KVM_HC_MAP_GPA_RANGE)))
@@ -9660,6 +9661,13 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
 			break;
 		}
 
+		slot = kvm_vcpu_gfn_to_memslot(vcpu, gpa_to_gfn(gpa));
+		if (!vcpu->kvm->arch.upm_mode ||
+		    !kvm_slot_can_be_private(slot)) {
+			ret = 0;
+			break;
+		}
+
 		vcpu->run->exit_reason        = KVM_EXIT_HYPERCALL;
 		vcpu->run->hypercall.nr       = KVM_HC_MAP_GPA_RANGE;
 		vcpu->run->hypercall.args[0]  = gpa;
virt/kvm/kvm_main.c +1 -0
index d2daa049e94a..73bf0bdedb59 100644
@@ -2646,6 +2646,7 @@ struct kvm_memory_slot *kvm_vcpu_gfn_to_memslot(struct kvm_vcpu *vcpu, gfn_t gfn
 
 	return NULL;
 }
+EXPORT_SYMBOL_GPL(kvm_vcpu_gfn_to_memslot);
 
 bool kvm_is_visible_gfn(struct kvm *kvm, gfn_t gfn)
 {
Signature · 1 line
2.25.1
#45 lore raw

Re: [PATCH RFC v7 07/64] KVM: SEV: Handle KVM_HC_MAP_GPA_RANGE hypercall

Quoted 29 lines · Jeremi Piotrowski

arch/x86/kvm/x86.c int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)

Quoted 21 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

KVM_HC_MAP_GPA_RANGE hypercall is used by the SEV guest to notify a change in the page encryption status to the hypervisor.

The hypercall exits to userspace with KVM_EXIT_HYPERCALL exit code, currently this is used for explicit memory conversion between shared/private for memfd based private memory.

---
 arch/x86/kvm/x86.c  | 8 ++++++++
 virt/kvm/kvm_main.c | 1 +
 2 files changed, 9 insertions(+)
arch/x86/kvm/x86.c +0 -0
index bb6adb216054..732f9cbbadb5 100644
@@ -9649,6 +9649,7 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)

Couldn't find a better commit to comment on:
when the guest has the ptp-kvm module, it will issue a KVM_HC_CLOCK_PAIRING hypercall. This will pass sev_es_validate_vmgexit validation and end up in this function where kvm_pv_clock_pairing() is called, and that calls kvm_write_guest(). This results in a CPU soft-lockup, at least in my testing.

Quoted 4 lines

Are there any emulated hypercalls that make sense for snp guests? We should block at least the ones that definitely don't work.

Jeremi

So turns out the soft-lockup is a nested issue (details here for those
interested: [^1]), but the questions still stands, of whether we should block kvm_write_page (and similar) explicitly or rely on the rmp fault.

[^1]: https://github.com/jepio/linux/commit/6c3bdf552e93664ae172660e24ceceed60fd4df5

#46 lore raw

RFC PATCH v7 8/64 KVM: Move kvm_for_each_memslot_in_hva_range() to be used in SVM

In reply to Michael Roth

From: Nikunj A Dadhania <[email protected]>

Move the macro to kvm_host.h and make if visible for SVM to use.

No functional change intended.

---
 include/linux/kvm_host.h | 6 ++++++
 virt/kvm/kvm_main.c      | 6 ------
 2 files changed, 6 insertions(+), 6 deletions(-)
include/linux/kvm_host.h +6 -0
index f72a2e0b8699..43b5c5aa8e80 100644
@@ -1116,6 +1116,12 @@ static inline bool kvm_memslot_iter_is_valid(struct kvm_memslot_iter *iter, gfn_
 	     kvm_memslot_iter_is_valid(iter, end);			\
 	     kvm_memslot_iter_next(iter))
 
+/* Iterate over each memslot intersecting [start, last] (inclusive) range */
+#define kvm_for_each_memslot_in_hva_range(node, slots, start, last)	     \
+	for (node = interval_tree_iter_first(&slots->hva_tree, start, last); \
+	     node;							     \
+	     node = interval_tree_iter_next(node, start, last))
+
 /*
  * KVM_SET_USER_MEMORY_REGION ioctl allows the following operations:
  * - create a new memory slot
virt/kvm/kvm_main.c +0 -6
index 73bf0bdedb59..a2306ccf9ab1 100644
@@ -623,12 +623,6 @@ static void kvm_null_fn(void)
 }
 #define IS_KVM_NULL_FN(fn) ((fn) == (void *)kvm_null_fn)
 
-/* Iterate over each memslot intersecting [start, last] (inclusive) range */
-#define kvm_for_each_memslot_in_hva_range(node, slots, start, last)	     \
-	for (node = interval_tree_iter_first(&slots->hva_tree, start, last); \
-	     node;							     \
-	     node = interval_tree_iter_next(node, start, last))	     \
-
 static __always_inline int __kvm_handle_hva_range(struct kvm *kvm,
 						  const struct kvm_hva_range *range)
 {
Signature · 1 line
2.25.1
maintainer
#47 lore raw

Re: [PATCH RFC v7 08/64] KVM: Move kvm_for_each_memslot_in_hva_range() to be used in SVM

Quoted 3 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

Move the macro to kvm_host.h and make if visible for SVM to use.

s/if/it/

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#48 lore raw

RFC PATCH v7 9/64 KVM: Add HVA range operator

In reply to Michael Roth

From: Vishal Annapurve <[email protected]>

Introduce HVA range operator so that other KVM subsystems can operate on HVA range.

---
 include/linux/kvm_host.h |  6 +++++
 virt/kvm/kvm_main.c      | 48 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+)
include/linux/kvm_host.h +6 -0
index 43b5c5aa8e80..21a539ab17f6 100644
@@ -1407,6 +1407,12 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm);
 void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end);
 void kvm_mmu_invalidate_end(struct kvm *kvm);
 
+typedef int (*kvm_hva_range_op_t)(struct kvm *kvm,
+				struct kvm_gfn_range *range, void *data);
+
+int kvm_vm_do_hva_range_op(struct kvm *kvm, unsigned long hva_start,
+		unsigned long hva_end, kvm_hva_range_op_t handler, void *data);
+
 long kvm_arch_dev_ioctl(struct file *filp,
 			unsigned int ioctl, unsigned long arg);
 long kvm_arch_vcpu_ioctl(struct file *filp,
virt/kvm/kvm_main.c +48 -0

54 lines not on this page · raw message

Signature · 1 line
2.25.1
#49 lore raw

RFC PATCH v7 10/64 KVM: SEV: Populate private memory fd during LAUNCH_UPDATE_DATA

In reply to Michael Roth

From: Vishal Annapurve <[email protected]>

This change adds handling of HVA ranges to copy contents to private memory while doing sev launch update data.

mem_attr array is updated during LAUNCH_UPDATE_DATA to ensure that encrypted memory is marked as private.

[mdr: use gfn_to_hva_memslot_prot() for shared GFN handler to deal with
read-only slots for ROMs]

---
 arch/x86/kvm/svm/sev.c   | 99 ++++++++++++++++++++++++++++++++++++----
 include/linux/kvm_host.h |  1 +
 virt/kvm/kvm_main.c      | 27 ++++++++---
 3 files changed, 111 insertions(+), 16 deletions(-)
arch/x86/kvm/svm/sev.c +90 -9
index 69dbf17f0d6a..a7e4e3005786 100644
@@ -493,23 +493,26 @@ static unsigned long get_num_contig_pages(unsigned long idx,
 	return pages;
 }
 
-static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp)
+static int sev_launch_update_shared_gfn_handler(struct kvm *kvm,
+						struct kvm_gfn_range *range,
+						struct kvm_sev_cmd *argp)
 {
 	unsigned long vaddr, vaddr_end, next_vaddr, npages, pages, size, i;
 	struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
-	struct kvm_sev_launch_update_data params;
 	struct sev_data_launch_update_data data;
 	struct page **inpages;
 	int ret;
 
-	if (!sev_guest(kvm))
-		return -ENOTTY;
-
-	if (copy_from_user(&params, (void __user *)(uintptr_t)argp->data, sizeof(params)))
-		return -EFAULT;
+	vaddr = gfn_to_hva_memslot_prot(range->slot, range->start, NULL);
+	pr_debug("%s: shared GFN: %llx, slot.id: %d, slot.base_gfn: %llx, slot.userspace_addr: %lx, slot.flags: %x, vaddr: %lx\n",
+		 __func__, range->start, range->slot->id, range->slot->base_gfn,
+		 range->slot->userspace_addr, range->slot->flags, vaddr);
+	if (kvm_is_error_hva(vaddr)) {
+		pr_err("vaddr is erroneous 0x%lx\n", vaddr);
+		return -EINVAL;
+	}
 
-	vaddr = params.uaddr;
-	size = params.len;
+	size = (range->end - range->start) << PAGE_SHIFT;
 	vaddr_end = vaddr + size;
 
 	/* Lock the user memory. */
@@ -561,6 +564,84 @@ static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp)
 	return ret;
 }
 
+static int sev_launch_update_priv_gfn_handler(struct kvm *kvm,
+					      struct kvm_gfn_range *range,
+					      struct kvm_sev_cmd *argp)
+{
+	struct sev_data_launch_update_data data;
+	struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+	gfn_t gfn;
+	kvm_pfn_t pfn;
+	struct kvm_memory_slot *memslot = range->slot;
+	int ret = 0;
+
+	data.reserved = 0;
+	data.handle = sev->handle;
+
+	for (gfn = range->start; gfn < range->end; gfn++) {
+		int order;
+		void *kvaddr;
+
+		ret = kvm_restricted_mem_get_pfn(memslot, gfn, &pfn, &order);
+		if (ret)
+			return ret;
+
+		kvaddr = pfn_to_kaddr(pfn);
+		if (!virt_addr_valid(kvaddr)) {
+			pr_err("Invalid kvaddr 0x%llx\n", (uint64_t)kvaddr);
+			ret = -EINVAL;
+			goto e_ret;
+		}
+
+		ret = kvm_read_guest_page(kvm, gfn, kvaddr, 0, PAGE_SIZE);
+		if (ret) {
+			pr_err("guest read failed 0x%x\n", ret);
+			goto e_ret;
+		}
+
+		if (!this_cpu_has(X86_FEATURE_SME_COHERENT))
+			clflush_cache_range(kvaddr, PAGE_SIZE);
+
+		data.len = PAGE_SIZE;
+		data.address = __sme_set(pfn << PAGE_SHIFT);
+		ret = sev_issue_cmd(kvm, SEV_CMD_LAUNCH_UPDATE_DATA, &data, &argp->error);
+		if (ret)
+			goto e_ret;
+		kvm_release_pfn_clean(pfn);
+	}
+	kvm_vm_set_region_attr(kvm, range->start, range->end,
+		true /* priv_attr */);
+
+e_ret:
+	return ret;
+}
+
+static int sev_launch_update_gfn_handler(struct kvm *kvm, struct kvm_gfn_range *range,
+					 void *data)
+{
+	struct kvm_sev_cmd *argp = (struct kvm_sev_cmd *)data;
+
+	if (kvm_slot_can_be_private(range->slot))
+		return sev_launch_update_priv_gfn_handler(kvm, range, argp);
+
+	return sev_launch_update_shared_gfn_handler(kvm, range, argp);
+}
+
+static int sev_launch_update_data(struct kvm *kvm,
+		struct kvm_sev_cmd *argp)
+{
+	struct kvm_sev_launch_update_data params;
+
+	if (!sev_guest(kvm))
+		return -ENOTTY;
+
+	if (copy_from_user(&params, (void __user *)(uintptr_t)argp->data, sizeof(params)))
+		return -EFAULT;
+
+	return kvm_vm_do_hva_range_op(kvm, params.uaddr, params.uaddr + params.len,
+		sev_launch_update_gfn_handler, argp);
+}
+
 static int sev_es_sync_vmsa(struct vcpu_svm *svm)
 {
 	struct sev_es_save_area *save = svm->sev_es.vmsa;
include/linux/kvm_host.h +1 -0
index 21a539ab17f6..33fa0b1435d3 100644
@@ -973,6 +973,7 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
 void kvm_exit(void);
 
 void kvm_get_kvm(struct kvm *kvm);
+int kvm_vm_set_region_attr(struct kvm *kvm, gfn_t start, gfn_t end, u64 attributes);
 bool kvm_get_kvm_safe(struct kvm *kvm);
 void kvm_put_kvm(struct kvm *kvm);
 bool file_is_kvm(struct file *file);
virt/kvm/kvm_main.c +20 -7
index 4ff7adaf6c56..1343070657d1 100644
@@ -747,6 +747,7 @@ int kvm_vm_do_hva_range_op(struct kvm *kvm, unsigned long hva_start,
 
 	return ret;
 }
+EXPORT_SYMBOL_GPL(kvm_vm_do_hva_range_op);
 
 static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn,
 						unsigned long start,
@@ -2595,12 +2596,28 @@ static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end,
 		kvm_flush_remote_tlbs(kvm);
 }
 
+int kvm_vm_set_region_attr(struct kvm *kvm, gfn_t start, gfn_t end,
+			   u64 attributes)
+{
+	gfn_t index;
+	void *entry;
+
+	entry = attributes ? xa_mk_value(attributes) : NULL;
+
+	for (index = start; index < end; index++)
+		if (xa_err(xa_store(&kvm->mem_attr_array, index, entry,
+				    GFP_KERNEL_ACCOUNT)))
+			break;
+
+	return index;
+}
+EXPORT_SYMBOL_GPL(kvm_vm_set_region_attr);
+
 static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 					   struct kvm_memory_attributes *attrs)
 {
 	gfn_t start, end;
 	unsigned long i;
-	void *entry;
 	int idx;
 	u64 supported_attrs = kvm_supported_mem_attributes(kvm);
 
@@ -2617,8 +2634,6 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 	start = attrs->address >> PAGE_SHIFT;
 	end = (attrs->address + attrs->size - 1 + PAGE_SIZE) >> PAGE_SHIFT;
 
-	entry = attrs->attributes ? xa_mk_value(attrs->attributes) : NULL;
-
 	if (kvm_arch_has_private_mem(kvm)) {
 		KVM_MMU_LOCK(kvm);
 		kvm_mmu_invalidate_begin(kvm);
@@ -2627,10 +2642,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 	}
 
 	mutex_lock(&kvm->lock);
-	for (i = start; i < end; i++)
-		if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,
-				    GFP_KERNEL_ACCOUNT)))
-			break;
+	i = kvm_vm_set_region_attr(kvm, start, end, attrs->attributes);
 	mutex_unlock(&kvm->lock);
 
 	if (kvm_arch_has_private_mem(kvm)) {
@@ -2793,6 +2805,7 @@ unsigned long gfn_to_hva_memslot_prot(struct kvm_memory_slot *slot,
 
 	return hva;
 }
+EXPORT_SYMBOL_GPL(gfn_to_hva_memslot_prot);
 
 unsigned long gfn_to_hva_prot(struct kvm *kvm, gfn_t gfn, bool *writable)
 {
Signature · 1 line
2.25.1
maintainer
#50 lore raw

Re: [PATCH RFC v7 10/64] KVM: SEV: Populate private memory fd during LAUNCH_UPDATE_DATA

Quoted 3 lines · Michael Roth

From: Vishal Annapurve <[email protected]>

This change adds handling of HVA ranges to copy contents

s/This change adds handling of/Handle/

Quoted 15 lines
arch/x86/kvm/svm/sev.c +15 -0
+static int sev_launch_update_priv_gfn_handler(struct kvm *kvm,
+					      struct kvm_gfn_range *range,
+					      struct kvm_sev_cmd *argp)
+{
+	struct sev_data_launch_update_data data;
+	struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+	gfn_t gfn;
+	kvm_pfn_t pfn;
+	struct kvm_memory_slot *memslot = range->slot;
+	int ret = 0;
+
+	data.reserved = 0;
+	data.handle = sev->handle;
+
+	for (gfn = range->start; gfn < range->end; gfn++) {

arch/x86/kvm/svm/sev.c static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) · line 591

Quoted 10 lines
arch/x86/kvm/svm/sev.c +10 -0
+		int order;
+		void *kvaddr;
+
+		ret = kvm_restricted_mem_get_pfn(memslot, gfn, &pfn, &order);
+		if (ret)
+			return ret;
+
+		kvaddr = pfn_to_kaddr(pfn);
+		if (!virt_addr_valid(kvaddr)) {
+			pr_err("Invalid kvaddr 0x%llx\n", (uint64_t)kvaddr);

Is that some debugging help leftover or what is that printk issued for?

arch/x86/kvm/svm/sev.c static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) · line 602

Quoted 11 lines
arch/x86/kvm/svm/sev.c +11 -0
+			ret = -EINVAL;
+			goto e_ret;
+		}
+
+		ret = kvm_read_guest_page(kvm, gfn, kvaddr, 0, PAGE_SIZE);
+		if (ret) {
+			pr_err("guest read failed 0x%x\n", ret);
+			goto e_ret;
+		}
+
+		if (!this_cpu_has(X86_FEATURE_SME_COHERENT))

check_for_deprecated_apis: WARNING: arch/x86/kvm/svm/sev.c:602: Do not use this_cpu_has() - use cpu_feature_enabled() instead.

arch/x86/kvm/svm/sev.c static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) · line 613

Quoted 11 lines
arch/x86/kvm/svm/sev.c +11 -0
+			clflush_cache_range(kvaddr, PAGE_SIZE);
+
+		data.len = PAGE_SIZE;
+		data.address = __sme_set(pfn << PAGE_SHIFT);
+		ret = sev_issue_cmd(kvm, SEV_CMD_LAUNCH_UPDATE_DATA, &data, &argp->error);
+		if (ret)
+			goto e_ret;
+		kvm_release_pfn_clean(pfn);
+	}
+	kvm_vm_set_region_attr(kvm, range->start, range->end,
+		true /* priv_attr */);

No need to break that line.

Quoted 16 lines
arch/x86/kvm/svm/sev.c +16 -0
+
+e_ret:
+	return ret;
+}
+
+static int sev_launch_update_gfn_handler(struct kvm *kvm, struct kvm_gfn_range *range,
+					 void *data)
+{
+	struct kvm_sev_cmd *argp = (struct kvm_sev_cmd *)data;
+
+	if (kvm_slot_can_be_private(range->slot))
+		return sev_launch_update_priv_gfn_handler(kvm, range, argp);
+
+	return sev_launch_update_shared_gfn_handler(kvm, range, argp);
+}
+

arch/x86/kvm/svm/sev.c static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) · line 639

Quoted 10 lines
arch/x86/kvm/svm/sev.c +10 -0
+static int sev_launch_update_data(struct kvm *kvm,
+		struct kvm_sev_cmd *argp)
+{
+	struct kvm_sev_launch_update_data params;
+
+	if (!sev_guest(kvm))
+		return -ENOTTY;
+
+	if (copy_from_user(&params, (void __user *)(uintptr_t)argp->data, sizeof(params)))
+		return -EFAULT;

Not gonna check those user-supplied values for sanity?

Or is this check

        if (WARN_ON_ONCE(hva_end <= hva_start))
                return -EINVAL;

in kvm_vm_do_hva_range_op() enough?

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#51 lore raw

Re: [PATCH RFC v7 10/64] KVM: SEV: Populate private memory fd during LAUNCH_UPDATE_DATA

Quoted 26 lines · Borislav Petkov
Quoted 3 lines · Michael Roth

From: Vishal Annapurve <[email protected]>

This change adds handling of HVA ranges to copy contents

s/This change adds handling of/Handle/

Quoted 19 lines
arch/x86/kvm/svm/sev.c +19 -0
+static int sev_launch_update_priv_gfn_handler(struct kvm *kvm,
+					      struct kvm_gfn_range *range,
+					      struct kvm_sev_cmd *argp)
+{
+	struct sev_data_launch_update_data data;
+	struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+	gfn_t gfn;
+	kvm_pfn_t pfn;
+	struct kvm_memory_slot *memslot = range->slot;
+	int ret = 0;
+
+	data.reserved = 0;
+	data.handle = sev->handle;
+
+	for (gfn = range->start; gfn < range->end; gfn++) {
+		int order;
+		void *kvaddr;
+
+		ret = kvm_restricted_mem_get_pfn(memslot, gfn, &pfn, &order);
Quoted 8 lines

arch/x86/kvm/svm/sev.c static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) · line 591

Quoted 6 lines
arch/x86/kvm/svm/sev.c +6 -0
+		if (ret)
+			return ret;
+
+		kvaddr = pfn_to_kaddr(pfn);
+		if (!virt_addr_valid(kvaddr)) {
+			pr_err("Invalid kvaddr 0x%llx\n", (uint64_t)kvaddr);

Is that some debugging help leftover or what is that printk issued for?

A mix of error-reporting and debugging I think. I think the error message isn't needed since the error value will get plumbed straight to userspace and anything beyond that is kernel debugging, so I added some context and switched this to pr_debug().

Quoted 57 lines

arch/x86/kvm/svm/sev.c static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) · line 602

Quoted 11 lines
arch/x86/kvm/svm/sev.c +11 -0
+			ret = -EINVAL;
+			goto e_ret;
+		}
+
+		ret = kvm_read_guest_page(kvm, gfn, kvaddr, 0, PAGE_SIZE);
+		if (ret) {
+			pr_err("guest read failed 0x%x\n", ret);
+			goto e_ret;
+		}
+
+		if (!this_cpu_has(X86_FEATURE_SME_COHERENT))

check_for_deprecated_apis: WARNING: arch/x86/kvm/svm/sev.c:602: Do not use this_cpu_has() - use cpu_feature_enabled() instead.

arch/x86/kvm/svm/sev.c static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) · line 613

Quoted 11 lines
arch/x86/kvm/svm/sev.c +11 -0
+			clflush_cache_range(kvaddr, PAGE_SIZE);
+
+		data.len = PAGE_SIZE;
+		data.address = __sme_set(pfn << PAGE_SHIFT);
+		ret = sev_issue_cmd(kvm, SEV_CMD_LAUNCH_UPDATE_DATA, &data, &argp->error);
+		if (ret)
+			goto e_ret;
+		kvm_release_pfn_clean(pfn);
+	}
+	kvm_vm_set_region_attr(kvm, range->start, range->end,
+		true /* priv_attr */);

No need to break that line.

Quoted 16 lines
arch/x86/kvm/svm/sev.c +16 -0
+
+e_ret:
+	return ret;
+}
+
+static int sev_launch_update_gfn_handler(struct kvm *kvm, struct kvm_gfn_range *range,
+					 void *data)
+{
+	struct kvm_sev_cmd *argp = (struct kvm_sev_cmd *)data;
+
+	if (kvm_slot_can_be_private(range->slot))
+		return sev_launch_update_priv_gfn_handler(kvm, range, argp);
+
+	return sev_launch_update_shared_gfn_handler(kvm, range, argp);
+}
+

arch/x86/kvm/svm/sev.c static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) · line 639

Quoted 10 lines
arch/x86/kvm/svm/sev.c +10 -0
+static int sev_launch_update_data(struct kvm *kvm,
+		struct kvm_sev_cmd *argp)
+{
+	struct kvm_sev_launch_update_data params;
+
+	if (!sev_guest(kvm))
+		return -ENOTTY;
+
+	if (copy_from_user(&params, (void __user *)(uintptr_t)argp->data, sizeof(params)))
+		return -EFAULT;

Not gonna check those user-supplied values for sanity?

Quoted 6 lines

Or is this check

        if (WARN_ON_ONCE(hva_end <= hva_start))
                return -EINVAL;

in kvm_vm_do_hva_range_op() enough?

Only partially, but kvm_vm_do_hva_range_op() should sanitize the address range and only call sev_launch_update_gfn_handler() on valid GFNs within the range, so if userspace provides a bogus HVA range that doesn't actually correspond to a valid memslot it'll simply be treated as a no-op.

-Mike

Quoted 8 lines

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#52 lore raw

Re: [PATCH RFC v7 10/64] KVM: SEV: Populate private memory fd during LAUNCH_UPDATE_DATA

In reply to Michael Roth
Quoted 229 lines · Michael Roth

From: Vishal Annapurve <[email protected]>

This change adds handling of HVA ranges to copy contents to private memory while doing sev launch update data.

mem_attr array is updated during LAUNCH_UPDATE_DATA to ensure that encrypted memory is marked as private.

[mdr: use gfn_to_hva_memslot_prot() for shared GFN handler to deal with
read-only slots for ROMs]

---
 arch/x86/kvm/svm/sev.c   | 99 ++++++++++++++++++++++++++++++++++++----
 include/linux/kvm_host.h |  1 +
 virt/kvm/kvm_main.c      | 27 ++++++++---
 3 files changed, 111 insertions(+), 16 deletions(-)
arch/x86/kvm/svm/sev.c +90 -9
index 69dbf17f0d6a..a7e4e3005786 100644
@@ -493,23 +493,26 @@ static unsigned long get_num_contig_pages(unsigned long idx,
 	return pages;
 }
 
-static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp)
+static int sev_launch_update_shared_gfn_handler(struct kvm *kvm,
+						struct kvm_gfn_range *range,
+						struct kvm_sev_cmd *argp)
 {
 	unsigned long vaddr, vaddr_end, next_vaddr, npages, pages, size, i;
 	struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
-	struct kvm_sev_launch_update_data params;
 	struct sev_data_launch_update_data data;
 	struct page **inpages;
 	int ret;
 
-	if (!sev_guest(kvm))
-		return -ENOTTY;
-
-	if (copy_from_user(&params, (void __user *)(uintptr_t)argp->data, sizeof(params)))
-		return -EFAULT;
+	vaddr = gfn_to_hva_memslot_prot(range->slot, range->start, NULL);
+	pr_debug("%s: shared GFN: %llx, slot.id: %d, slot.base_gfn: %llx, slot.userspace_addr: %lx, slot.flags: %x, vaddr: %lx\n",
+		 __func__, range->start, range->slot->id, range->slot->base_gfn,
+		 range->slot->userspace_addr, range->slot->flags, vaddr);
+	if (kvm_is_error_hva(vaddr)) {
+		pr_err("vaddr is erroneous 0x%lx\n", vaddr);
+		return -EINVAL;
+	}
 
-	vaddr = params.uaddr;
-	size = params.len;
+	size = (range->end - range->start) << PAGE_SHIFT;
 	vaddr_end = vaddr + size;
 
 	/* Lock the user memory. */
@@ -561,6 +564,84 @@ static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp)
 	return ret;
 }
 
+static int sev_launch_update_priv_gfn_handler(struct kvm *kvm,
+					      struct kvm_gfn_range *range,
+					      struct kvm_sev_cmd *argp)
+{
+	struct sev_data_launch_update_data data;
+	struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+	gfn_t gfn;
+	kvm_pfn_t pfn;
+	struct kvm_memory_slot *memslot = range->slot;
+	int ret = 0;
+
+	data.reserved = 0;
+	data.handle = sev->handle;
+
+	for (gfn = range->start; gfn < range->end; gfn++) {
+		int order;
+		void *kvaddr;
+
+		ret = kvm_restricted_mem_get_pfn(memslot, gfn, &pfn, &order);
+		if (ret)
+			return ret;
+
+		kvaddr = pfn_to_kaddr(pfn);
+		if (!virt_addr_valid(kvaddr)) {
+			pr_err("Invalid kvaddr 0x%llx\n", (uint64_t)kvaddr);
+			ret = -EINVAL;
+			goto e_ret;
+		}
+
+		ret = kvm_read_guest_page(kvm, gfn, kvaddr, 0, PAGE_SIZE);
+		if (ret) {
+			pr_err("guest read failed 0x%x\n", ret);
+			goto e_ret;
+		}
+
+		if (!this_cpu_has(X86_FEATURE_SME_COHERENT))
+			clflush_cache_range(kvaddr, PAGE_SIZE);
+
+		data.len = PAGE_SIZE;
+		data.address = __sme_set(pfn << PAGE_SHIFT);
+		ret = sev_issue_cmd(kvm, SEV_CMD_LAUNCH_UPDATE_DATA, &data, &argp->error);
+		if (ret)
+			goto e_ret;
+		kvm_release_pfn_clean(pfn);
+	}
+	kvm_vm_set_region_attr(kvm, range->start, range->end,
+		true /* priv_attr */);
+
+e_ret:
+	return ret;
+}
+
+static int sev_launch_update_gfn_handler(struct kvm *kvm, struct kvm_gfn_range *range,
+					 void *data)
+{
+	struct kvm_sev_cmd *argp = (struct kvm_sev_cmd *)data;
+
+	if (kvm_slot_can_be_private(range->slot))
+		return sev_launch_update_priv_gfn_handler(kvm, range, argp);
+
+	return sev_launch_update_shared_gfn_handler(kvm, range, argp);
+}
+
+static int sev_launch_update_data(struct kvm *kvm,
+		struct kvm_sev_cmd *argp)
+{
+	struct kvm_sev_launch_update_data params;
+
+	if (!sev_guest(kvm))
+		return -ENOTTY;
+
+	if (copy_from_user(&params, (void __user *)(uintptr_t)argp->data, sizeof(params)))
+		return -EFAULT;
+
+	return kvm_vm_do_hva_range_op(kvm, params.uaddr, params.uaddr + params.len,
+		sev_launch_update_gfn_handler, argp);
+}
+
 static int sev_es_sync_vmsa(struct vcpu_svm *svm)
 {
 	struct sev_es_save_area *save = svm->sev_es.vmsa;
include/linux/kvm_host.h +1 -0
index 21a539ab17f6..33fa0b1435d3 100644
@@ -973,6 +973,7 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
 void kvm_exit(void);
 
 void kvm_get_kvm(struct kvm *kvm);
+int kvm_vm_set_region_attr(struct kvm *kvm, gfn_t start, gfn_t end, u64 attributes);
 bool kvm_get_kvm_safe(struct kvm *kvm);
 void kvm_put_kvm(struct kvm *kvm);
 bool file_is_kvm(struct file *file);
virt/kvm/kvm_main.c +20 -7
index 4ff7adaf6c56..1343070657d1 100644
@@ -747,6 +747,7 @@ int kvm_vm_do_hva_range_op(struct kvm *kvm, unsigned long hva_start,
 
 	return ret;
 }
+EXPORT_SYMBOL_GPL(kvm_vm_do_hva_range_op);
 
 static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn,
 						unsigned long start,
@@ -2595,12 +2596,28 @@ static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end,
 		kvm_flush_remote_tlbs(kvm);
 }
 
+int kvm_vm_set_region_attr(struct kvm *kvm, gfn_t start, gfn_t end,
+			   u64 attributes)
+{
+	gfn_t index;
+	void *entry;
+
+	entry = attributes ? xa_mk_value(attributes) : NULL;
+
+	for (index = start; index < end; index++)
+		if (xa_err(xa_store(&kvm->mem_attr_array, index, entry,
+				    GFP_KERNEL_ACCOUNT)))
+			break;
+
+	return index;
+}
+EXPORT_SYMBOL_GPL(kvm_vm_set_region_attr);
+
 static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 					   struct kvm_memory_attributes *attrs)
 {
 	gfn_t start, end;
 	unsigned long i;
-	void *entry;
 	int idx;
 	u64 supported_attrs = kvm_supported_mem_attributes(kvm);
 
@@ -2617,8 +2634,6 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 	start = attrs->address >> PAGE_SHIFT;
 	end = (attrs->address + attrs->size - 1 + PAGE_SIZE) >> PAGE_SHIFT;
 
-	entry = attrs->attributes ? xa_mk_value(attrs->attributes) : NULL;
-
 	if (kvm_arch_has_private_mem(kvm)) {
 		KVM_MMU_LOCK(kvm);
 		kvm_mmu_invalidate_begin(kvm);
@@ -2627,10 +2642,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 	}
 
 	mutex_lock(&kvm->lock);
-	for (i = start; i < end; i++)
-		if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,
-				    GFP_KERNEL_ACCOUNT)))
-			break;
+	i = kvm_vm_set_region_attr(kvm, start, end, attrs->attributes);
 	mutex_unlock(&kvm->lock);
 
 	if (kvm_arch_has_private_mem(kvm)) {
@@ -2793,6 +2805,7 @@ unsigned long gfn_to_hva_memslot_prot(struct kvm_memory_slot *slot,
 
 	return hva;
 }
+EXPORT_SYMBOL_GPL(gfn_to_hva_memslot_prot);
 
 unsigned long gfn_to_hva_prot(struct kvm *kvm, gfn_t gfn, bool *writable)
 {
Signature · 1 line
2.25.1

Hmm.. but user space is still allowed to call KVM_SET_MEMORY_ATTRIBUTES with KVM_MEMORY_ATTRIBUTE_PRIVATE set? How do these behaviours complement each other?

SEV specific changes and kvm_vm_set_region_attr() definition should really be separate patches.

BR, Jarkko

#53 lore raw

Re: [PATCH RFC v7 10/64] KVM: SEV: Populate private memory fd during LAUNCH_UPDATE_DATA

Quoted 225 lines · Jarkko Sakkinen
Quoted 224 lines · Michael Roth

From: Vishal Annapurve <[email protected]>

This change adds handling of HVA ranges to copy contents to private memory while doing sev launch update data.

mem_attr array is updated during LAUNCH_UPDATE_DATA to ensure that encrypted memory is marked as private.

[mdr: use gfn_to_hva_memslot_prot() for shared GFN handler to deal with
read-only slots for ROMs]

---
 arch/x86/kvm/svm/sev.c   | 99 ++++++++++++++++++++++++++++++++++++----
 include/linux/kvm_host.h |  1 +
 virt/kvm/kvm_main.c      | 27 ++++++++---
 3 files changed, 111 insertions(+), 16 deletions(-)
arch/x86/kvm/svm/sev.c +90 -9
index 69dbf17f0d6a..a7e4e3005786 100644
@@ -493,23 +493,26 @@ static unsigned long get_num_contig_pages(unsigned long idx,
 	return pages;
 }
 
-static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp)
+static int sev_launch_update_shared_gfn_handler(struct kvm *kvm,
+						struct kvm_gfn_range *range,
+						struct kvm_sev_cmd *argp)
 {
 	unsigned long vaddr, vaddr_end, next_vaddr, npages, pages, size, i;
 	struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
-	struct kvm_sev_launch_update_data params;
 	struct sev_data_launch_update_data data;
 	struct page **inpages;
 	int ret;
 
-	if (!sev_guest(kvm))
-		return -ENOTTY;
-
-	if (copy_from_user(&params, (void __user *)(uintptr_t)argp->data, sizeof(params)))
-		return -EFAULT;
+	vaddr = gfn_to_hva_memslot_prot(range->slot, range->start, NULL);
+	pr_debug("%s: shared GFN: %llx, slot.id: %d, slot.base_gfn: %llx, slot.userspace_addr: %lx, slot.flags: %x, vaddr: %lx\n",
+		 __func__, range->start, range->slot->id, range->slot->base_gfn,
+		 range->slot->userspace_addr, range->slot->flags, vaddr);
+	if (kvm_is_error_hva(vaddr)) {
+		pr_err("vaddr is erroneous 0x%lx\n", vaddr);
+		return -EINVAL;
+	}
 
-	vaddr = params.uaddr;
-	size = params.len;
+	size = (range->end - range->start) << PAGE_SHIFT;
 	vaddr_end = vaddr + size;
 
 	/* Lock the user memory. */
@@ -561,6 +564,84 @@ static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp)
 	return ret;
 }
 
+static int sev_launch_update_priv_gfn_handler(struct kvm *kvm,
+					      struct kvm_gfn_range *range,
+					      struct kvm_sev_cmd *argp)
+{
+	struct sev_data_launch_update_data data;
+	struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+	gfn_t gfn;
+	kvm_pfn_t pfn;
+	struct kvm_memory_slot *memslot = range->slot;
+	int ret = 0;
+
+	data.reserved = 0;
+	data.handle = sev->handle;
+
+	for (gfn = range->start; gfn < range->end; gfn++) {
+		int order;
+		void *kvaddr;
+
+		ret = kvm_restricted_mem_get_pfn(memslot, gfn, &pfn, &order);
+		if (ret)
+			return ret;
+
+		kvaddr = pfn_to_kaddr(pfn);
+		if (!virt_addr_valid(kvaddr)) {
+			pr_err("Invalid kvaddr 0x%llx\n", (uint64_t)kvaddr);
+			ret = -EINVAL;
+			goto e_ret;
+		}
+
+		ret = kvm_read_guest_page(kvm, gfn, kvaddr, 0, PAGE_SIZE);
+		if (ret) {
+			pr_err("guest read failed 0x%x\n", ret);
+			goto e_ret;
+		}
+
+		if (!this_cpu_has(X86_FEATURE_SME_COHERENT))
+			clflush_cache_range(kvaddr, PAGE_SIZE);
+
+		data.len = PAGE_SIZE;
+		data.address = __sme_set(pfn << PAGE_SHIFT);
+		ret = sev_issue_cmd(kvm, SEV_CMD_LAUNCH_UPDATE_DATA, &data, &argp->error);
+		if (ret)
+			goto e_ret;
+		kvm_release_pfn_clean(pfn);
+	}
+	kvm_vm_set_region_attr(kvm, range->start, range->end,
+		true /* priv_attr */);
+
+e_ret:
+	return ret;
+}
+
+static int sev_launch_update_gfn_handler(struct kvm *kvm, struct kvm_gfn_range *range,
+					 void *data)
+{
+	struct kvm_sev_cmd *argp = (struct kvm_sev_cmd *)data;
+
+	if (kvm_slot_can_be_private(range->slot))
+		return sev_launch_update_priv_gfn_handler(kvm, range, argp);
+
+	return sev_launch_update_shared_gfn_handler(kvm, range, argp);
+}
+
+static int sev_launch_update_data(struct kvm *kvm,
+		struct kvm_sev_cmd *argp)
+{
+	struct kvm_sev_launch_update_data params;
+
+	if (!sev_guest(kvm))
+		return -ENOTTY;
+
+	if (copy_from_user(&params, (void __user *)(uintptr_t)argp->data, sizeof(params)))
+		return -EFAULT;
+
+	return kvm_vm_do_hva_range_op(kvm, params.uaddr, params.uaddr + params.len,
+		sev_launch_update_gfn_handler, argp);
+}
+
 static int sev_es_sync_vmsa(struct vcpu_svm *svm)
 {
 	struct sev_es_save_area *save = svm->sev_es.vmsa;
include/linux/kvm_host.h +1 -0
index 21a539ab17f6..33fa0b1435d3 100644
@@ -973,6 +973,7 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
 void kvm_exit(void);
 
 void kvm_get_kvm(struct kvm *kvm);
+int kvm_vm_set_region_attr(struct kvm *kvm, gfn_t start, gfn_t end, u64 attributes);
 bool kvm_get_kvm_safe(struct kvm *kvm);
 void kvm_put_kvm(struct kvm *kvm);
 bool file_is_kvm(struct file *file);
virt/kvm/kvm_main.c +20 -7
index 4ff7adaf6c56..1343070657d1 100644
@@ -747,6 +747,7 @@ int kvm_vm_do_hva_range_op(struct kvm *kvm, unsigned long hva_start,
 
 	return ret;
 }
+EXPORT_SYMBOL_GPL(kvm_vm_do_hva_range_op);
 
 static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn,
 						unsigned long start,
@@ -2595,12 +2596,28 @@ static void kvm_unmap_mem_range(struct kvm *kvm, gfn_t start, gfn_t end,
 		kvm_flush_remote_tlbs(kvm);
 }
 
+int kvm_vm_set_region_attr(struct kvm *kvm, gfn_t start, gfn_t end,
+			   u64 attributes)
+{
+	gfn_t index;
+	void *entry;
+
+	entry = attributes ? xa_mk_value(attributes) : NULL;
+
+	for (index = start; index < end; index++)
+		if (xa_err(xa_store(&kvm->mem_attr_array, index, entry,
+				    GFP_KERNEL_ACCOUNT)))
+			break;
+
+	return index;
+}
+EXPORT_SYMBOL_GPL(kvm_vm_set_region_attr);
+
 static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 					   struct kvm_memory_attributes *attrs)
 {
 	gfn_t start, end;
 	unsigned long i;
-	void *entry;
 	int idx;
 	u64 supported_attrs = kvm_supported_mem_attributes(kvm);
 
@@ -2617,8 +2634,6 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 	start = attrs->address >> PAGE_SHIFT;
 	end = (attrs->address + attrs->size - 1 + PAGE_SIZE) >> PAGE_SHIFT;
 
-	entry = attrs->attributes ? xa_mk_value(attrs->attributes) : NULL;
-
 	if (kvm_arch_has_private_mem(kvm)) {
 		KVM_MMU_LOCK(kvm);
 		kvm_mmu_invalidate_begin(kvm);
@@ -2627,10 +2642,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 	}
 
 	mutex_lock(&kvm->lock);
-	for (i = start; i < end; i++)
-		if (xa_err(xa_store(&kvm->mem_attr_array, i, entry,
-				    GFP_KERNEL_ACCOUNT)))
-			break;
+	i = kvm_vm_set_region_attr(kvm, start, end, attrs->attributes);
 	mutex_unlock(&kvm->lock);
 
 	if (kvm_arch_has_private_mem(kvm)) {
@@ -2793,6 +2805,7 @@ unsigned long gfn_to_hva_memslot_prot(struct kvm_memory_slot *slot,
 
 	return hva;
 }
+EXPORT_SYMBOL_GPL(gfn_to_hva_memslot_prot);
 
Quoted 9 lines
Quoted 5 lines
virt/kvm/kvm_main.c +0 -0
 unsigned long gfn_to_hva_prot(struct kvm *kvm, gfn_t gfn, bool *writable)
 {
Signature · 1 line
2.25.1

Hmm.. but user space is still allowed to call KVM_SET_MEMORY_ATTRIBUTES with KVM_MEMORY_ATTRIBUTE_PRIVATE set? How do these behaviours complement each other?

In the current series it's allowed. It reduces some headaches since an SNP guest can issue duplicate page-state changes without violating the spec, and it would complicate things further if kernel/userspace attempted to enforce something stricter.

Quoted 3 lines

SEV specific changes and kvm_vm_set_region_attr() definition should really be separate patches.

Done.

-Mike

Quoted 2 lines

BR, Jarkko

#54 lore raw

RFC PATCH v7 11/64 KVM: SEV: Support private pages in LAUNCH_UPDATE_DATA

In reply to Michael Roth

From: Nikunj A Dadhania <[email protected]>

Pre-boot guest payload needs to be encrypted and VMM has copied it over to the private-fd. Add support to get the pfn from the memfile fd for encrypting the payload in-place.

---
 arch/x86/kvm/svm/sev.c | 79 ++++++++++++++++++++++++++++++++++--------
 1 file changed, 64 insertions(+), 15 deletions(-)
arch/x86/kvm/svm/sev.c +64 -15
index a7e4e3005786..ae4920aeb281 100644
@@ -107,6 +107,11 @@ static inline bool is_mirroring_enc_context(struct kvm *kvm)
 	return !!to_kvm_svm(kvm)->sev_info.enc_context_owner;
 }
 
+static bool kvm_is_upm_enabled(struct kvm *kvm)
+{
+	return kvm->arch.upm_mode;
+}
+
 /* Must be called with the sev_bitmap_lock held */
 static bool __sev_recycle_asids(int min_asid, int max_asid)
 {
@@ -382,6 +387,38 @@ static int sev_launch_start(struct kvm *kvm, struct kvm_sev_cmd *argp)
 	return ret;
 }
 
+static int sev_get_memfile_pfn_handler(struct kvm *kvm, struct kvm_gfn_range *range, void *data)
+{
+	struct kvm_memory_slot *memslot = range->slot;
+	struct page **pages = data;
+	int ret = 0, i = 0;
+	kvm_pfn_t pfn;
+	gfn_t gfn;
+
+	for (gfn = range->start; gfn < range->end; gfn++) {
+		int order;
+
+		ret = kvm_restricted_mem_get_pfn(memslot, gfn, &pfn, &order);
+		if (ret)
+			return ret;
+
+		if (is_error_noslot_pfn(pfn))
+			return -EFAULT;
+
+		pages[i++] = pfn_to_page(pfn);
+	}
+
+	return ret;
+}
+
+static int sev_get_memfile_pfn(struct kvm *kvm, unsigned long addr,
+			       unsigned long size, unsigned long npages,
+			       struct page **pages)
+{
+	return kvm_vm_do_hva_range_op(kvm, addr, size,
+				      sev_get_memfile_pfn_handler, pages);
+}
+
 static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr,
 				    unsigned long ulen, unsigned long *n,
 				    int write)
@@ -424,16 +461,25 @@ static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr,
 	if (!pages)
 		return ERR_PTR(-ENOMEM);
 
-	/* Pin the user virtual address. */
-	npinned = pin_user_pages_fast(uaddr, npages, write ? FOLL_WRITE : 0, pages);
-	if (npinned != npages) {
-		pr_err("SEV: Failure locking %lu pages.\n", npages);
-		ret = -ENOMEM;
-		goto err;
+	if (kvm_is_upm_enabled(kvm)) {
+		/* Get the PFN from memfile */
+		if (sev_get_memfile_pfn(kvm, uaddr, ulen, npages, pages)) {
+			pr_err("%s: ERROR: unable to find slot for uaddr %lx", __func__, uaddr);
+			ret = -ENOMEM;
+			goto err;
+		}
+	} else {
+		/* Pin the user virtual address. */
+		npinned = pin_user_pages_fast(uaddr, npages, write ? FOLL_WRITE : 0, pages);
+		if (npinned != npages) {
+			pr_err("SEV: Failure locking %lu pages.\n", npages);
+			ret = -ENOMEM;
+			goto err;
+		}
+		sev->pages_locked = locked;
 	}
 
 	*n = npages;
-	sev->pages_locked = locked;
 
 	return pages;
 
@@ -514,6 +560,7 @@ static int sev_launch_update_shared_gfn_handler(struct kvm *kvm,
 
 	size = (range->end - range->start) << PAGE_SHIFT;
 	vaddr_end = vaddr + size;
+	WARN_ON(size < PAGE_SIZE);
 
 	/* Lock the user memory. */
 	inpages = sev_pin_memory(kvm, vaddr, size, &npages, 1);
@@ -554,13 +601,16 @@ static int sev_launch_update_shared_gfn_handler(struct kvm *kvm,
 	}
 
 e_unpin:
-	/* content of memory is updated, mark pages dirty */
-	for (i = 0; i < npages; i++) {
-		set_page_dirty_lock(inpages[i]);
-		mark_page_accessed(inpages[i]);
+	if (!kvm_is_upm_enabled(kvm)) {
+		/* content of memory is updated, mark pages dirty */
+		for (i = 0; i < npages; i++) {
+			set_page_dirty_lock(inpages[i]);
+			mark_page_accessed(inpages[i]);
+		}
+		/* unlock the user pages */
+		sev_unpin_memory(kvm, inpages, npages);
 	}
-	/* unlock the user pages */
-	sev_unpin_memory(kvm, inpages, npages);
+
 	return ret;
 }
 
@@ -609,9 +659,8 @@ static int sev_launch_update_priv_gfn_handler(struct kvm *kvm,
 			goto e_ret;
 		kvm_release_pfn_clean(pfn);
 	}
-	kvm_vm_set_region_attr(kvm, range->start, range->end,
-		true /* priv_attr */);
 
+	kvm_vm_set_region_attr(kvm, range->start, range->end, KVM_MEMORY_ATTRIBUTE_PRIVATE);
 e_ret:
 	return ret;
 }
Signature · 1 line
2.25.1
#55 lore raw

Re: [PATCH RFC v7 11/64] KVM: SEV: Support private pages in LAUNCH_UPDATE_DATA

Quoted 53 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

Pre-boot guest payload needs to be encrypted and VMM has copied it over to the private-fd. Add support to get the pfn from the memfile fd for encrypting the payload in-place.

---
 arch/x86/kvm/svm/sev.c | 79 ++++++++++++++++++++++++++++++++++--------
 1 file changed, 64 insertions(+), 15 deletions(-)
arch/x86/kvm/svm/sev.c +26 -0
index a7e4e3005786..ae4920aeb281 100644
@@ -107,6 +107,11 @@ static inline bool is_mirroring_enc_context(struct kvm *kvm)
 	return !!to_kvm_svm(kvm)->sev_info.enc_context_owner;
 }
 
+static bool kvm_is_upm_enabled(struct kvm *kvm)
+{
+	return kvm->arch.upm_mode;
+}
+
 /* Must be called with the sev_bitmap_lock held */
 static bool __sev_recycle_asids(int min_asid, int max_asid)
 {
@@ -382,6 +387,38 @@ static int sev_launch_start(struct kvm *kvm, struct kvm_sev_cmd *argp)
 	return ret;
 }
 
+static int sev_get_memfile_pfn_handler(struct kvm *kvm, struct kvm_gfn_range *range, void *data)
+{
+	struct kvm_memory_slot *memslot = range->slot;
+	struct page **pages = data;
+	int ret = 0, i = 0;
+	kvm_pfn_t pfn;
+	gfn_t gfn;
+
+	for (gfn = range->start; gfn < range->end; gfn++) {
+		int order;
+
+		ret = kvm_restricted_mem_get_pfn(memslot, gfn, &pfn, &order);
+		if (ret)
+			return ret;
+
+		if (is_error_noslot_pfn(pfn))
+			return -EFAULT;
+
+		pages[i++] = pfn_to_page(pfn);
+	}
+

arch/x86/kvm/svm/sev.c static int sev_launch_start(struct kvm *kvm, struct kvm_sev_cmd *argp) · line 420

Quoted 10 lines
arch/x86/kvm/svm/sev.c +10 -0
+	return ret;
+}
+
+static int sev_get_memfile_pfn(struct kvm *kvm, unsigned long addr,
+			       unsigned long size, unsigned long npages,
+			       struct page **pages)
+{
+	return kvm_vm_do_hva_range_op(kvm, addr, size,
+				      sev_get_memfile_pfn_handler, pages);
+}

The third argument for the kvm_vm_do_hva_range_op call should be addr+size; the function expects the end of the range not the size of the range.

Quoted 11 lines
arch/x86/kvm/svm/sev.c +1 -3
+
 static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr,
 				    unsigned long ulen, unsigned long *n,
 				    int write)
@@ -424,16 +461,25 @@ static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr,
 	if (!pages)
 		return ERR_PTR(-ENOMEM);
 
-	/* Pin the user virtual address. */
-	npinned = pin_user_pages_fast(uaddr, npages, write ? FOLL_WRITE : 0, pages);
-	if (npinned != npages) {

arch/x86/kvm/svm/sev.c static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, · line 470

Quoted 10 lines
arch/x86/kvm/svm/sev.c +7 -3
-		pr_err("SEV: Failure locking %lu pages.\n", npages);
-		ret = -ENOMEM;
-		goto err;
+	if (kvm_is_upm_enabled(kvm)) {
+		/* Get the PFN from memfile */
+		if (sev_get_memfile_pfn(kvm, uaddr, ulen, npages, pages)) {
+			pr_err("%s: ERROR: unable to find slot for uaddr %lx", __func__, uaddr);
+			ret = -ENOMEM;
+			goto err;
+		}

This branch doesn't initialize npinned. If sev_get_memfile_pfn fails, the code following the err label passes the uninitialized value to unpin_user_pages.

Quoted 59 lines
arch/x86/kvm/svm/sev.c +20 -10
+	} else {
+		/* Pin the user virtual address. */
+		npinned = pin_user_pages_fast(uaddr, npages, write ? FOLL_WRITE : 0, pages);
+		if (npinned != npages) {
+			pr_err("SEV: Failure locking %lu pages.\n", npages);
+			ret = -ENOMEM;
+			goto err;
+		}
+		sev->pages_locked = locked;
 	}
 
 	*n = npages;
-	sev->pages_locked = locked;
 
 	return pages;
 
@@ -514,6 +560,7 @@ static int sev_launch_update_shared_gfn_handler(struct kvm *kvm,
 
 	size = (range->end - range->start) << PAGE_SHIFT;
 	vaddr_end = vaddr + size;
+	WARN_ON(size < PAGE_SIZE);
 
 	/* Lock the user memory. */
 	inpages = sev_pin_memory(kvm, vaddr, size, &npages, 1);
@@ -554,13 +601,16 @@ static int sev_launch_update_shared_gfn_handler(struct kvm *kvm,
 	}
 
 e_unpin:
-	/* content of memory is updated, mark pages dirty */
-	for (i = 0; i < npages; i++) {
-		set_page_dirty_lock(inpages[i]);
-		mark_page_accessed(inpages[i]);
+	if (!kvm_is_upm_enabled(kvm)) {
+		/* content of memory is updated, mark pages dirty */
+		for (i = 0; i < npages; i++) {
+			set_page_dirty_lock(inpages[i]);
+			mark_page_accessed(inpages[i]);
+		}
+		/* unlock the user pages */
+		sev_unpin_memory(kvm, inpages, npages);
 	}
-	/* unlock the user pages */
-	sev_unpin_memory(kvm, inpages, npages);
+
 	return ret;
 }
 
@@ -609,9 +659,8 @@ static int sev_launch_update_priv_gfn_handler(struct kvm *kvm,
 			goto e_ret;
 		kvm_release_pfn_clean(pfn);
 	}
-	kvm_vm_set_region_attr(kvm, range->start, range->end,
-		true /* priv_attr */);
 
+	kvm_vm_set_region_attr(kvm, range->start, range->end, KVM_MEMORY_ATTRIBUTE_PRIVATE);
 e_ret:
 	return ret;
 }
--
Quoted 2 lines

2.25.1

Regards, Tom

#56 lore raw

Re: [PATCH RFC v7 11/64] KVM: SEV: Support private pages in LAUNCH_UPDATE_DATA

Quoted 59 lines · On 22/12/22 23:54, [email protected] wrote:
Quoted 58 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

Pre-boot guest payload needs to be encrypted and VMM has copied it over to the private-fd. Add support to get the pfn from the memfile fd for encrypting the payload in-place.

---
 arch/x86/kvm/svm/sev.c | 79 ++++++++++++++++++++++++++++++++++--------
 1 file changed, 64 insertions(+), 15 deletions(-)
arch/x86/kvm/svm/sev.c +31 -0
index a7e4e3005786..ae4920aeb281 100644
@@ -107,6 +107,11 @@ static inline bool is_mirroring_enc_context(struct kvm *kvm)
 	return !!to_kvm_svm(kvm)->sev_info.enc_context_owner;
 }
 
+static bool kvm_is_upm_enabled(struct kvm *kvm)
+{
+	return kvm->arch.upm_mode;
+}
+
 /* Must be called with the sev_bitmap_lock held */
 static bool __sev_recycle_asids(int min_asid, int max_asid)
 {
@@ -382,6 +387,38 @@ static int sev_launch_start(struct kvm *kvm, struct kvm_sev_cmd *argp)
 	return ret;
 }
 
+static int sev_get_memfile_pfn_handler(struct kvm *kvm, struct kvm_gfn_range *range, void *data)
+{
+	struct kvm_memory_slot *memslot = range->slot;
+	struct page **pages = data;
+	int ret = 0, i = 0;
+	kvm_pfn_t pfn;
+	gfn_t gfn;
+
+	for (gfn = range->start; gfn < range->end; gfn++) {
+		int order;
+
+		ret = kvm_restricted_mem_get_pfn(memslot, gfn, &pfn, &order);
+		if (ret)
+			return ret;
+
+		if (is_error_noslot_pfn(pfn))
+			return -EFAULT;
+
+		pages[i++] = pfn_to_page(pfn);
+	}
+
+	return ret;
+}
+
+static int sev_get_memfile_pfn(struct kvm *kvm, unsigned long addr,
+			       unsigned long size, unsigned long npages,
Quoted 8 lines

arch/x86/kvm/svm/sev.c static int sev_launch_start(struct kvm *kvm, struct kvm_sev_cmd *argp) · line 420

Quoted 5 lines
arch/x86/kvm/svm/sev.c +5 -0
+			       struct page **pages)
+{
+	return kvm_vm_do_hva_range_op(kvm, addr, size,
+				      sev_get_memfile_pfn_handler, pages);
+}

The third argument for the kvm_vm_do_hva_range_op call should be addr+size; the function expects the end of the range not the size of the range.

Good catch, will fix.

Quoted 8 lines
Quoted 8 lines
arch/x86/kvm/svm/sev.c +1 -0
+
 static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr,
 				    unsigned long ulen, unsigned long *n,
 				    int write)
@@ -424,16 +461,25 @@ static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr,
 	if (!pages)
 		return ERR_PTR(-ENOMEM);
 
Quoted 16 lines

arch/x86/kvm/svm/sev.c static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, · line 470

Quoted 13 lines
arch/x86/kvm/svm/sev.c +7 -6
-	/* Pin the user virtual address. */
-	npinned = pin_user_pages_fast(uaddr, npages, write ? FOLL_WRITE : 0, pages);
-	if (npinned != npages) {
-		pr_err("SEV: Failure locking %lu pages.\n", npages);
-		ret = -ENOMEM;
-		goto err;
+	if (kvm_is_upm_enabled(kvm)) {
+		/* Get the PFN from memfile */
+		if (sev_get_memfile_pfn(kvm, uaddr, ulen, npages, pages)) {
+			pr_err("%s: ERROR: unable to find slot for uaddr %lx", __func__, uaddr);
+			ret = -ENOMEM;
+			goto err;
+		}

This branch doesn't initialize npinned. If sev_get_memfile_pfn fails, the code following the err label passes the uninitialized value to unpin_user_pages.

Sure, will fix.

Quoted 55 lines
Quoted 55 lines
arch/x86/kvm/svm/sev.c +19 -9
+	} else {
+		/* Pin the user virtual address. */
+		npinned = pin_user_pages_fast(uaddr, npages, write ? FOLL_WRITE : 0, pages);
+		if (npinned != npages) {
+			pr_err("SEV: Failure locking %lu pages.\n", npages);
+			ret = -ENOMEM;
+			goto err;
+		}
+		sev->pages_locked = locked;
 	}
 
 	*n = npages;
-	sev->pages_locked = locked;
 
 	return pages;
 
@@ -514,6 +560,7 @@ static int sev_launch_update_shared_gfn_handler(struct kvm *kvm,
 
 	size = (range->end - range->start) << PAGE_SHIFT;
 	vaddr_end = vaddr + size;
+	WARN_ON(size < PAGE_SIZE);
 
 	/* Lock the user memory. */
 	inpages = sev_pin_memory(kvm, vaddr, size, &npages, 1);
@@ -554,13 +601,16 @@ static int sev_launch_update_shared_gfn_handler(struct kvm *kvm,
 	}
 
 e_unpin:
-	/* content of memory is updated, mark pages dirty */
-	for (i = 0; i < npages; i++) {
-		set_page_dirty_lock(inpages[i]);
-		mark_page_accessed(inpages[i]);
+	if (!kvm_is_upm_enabled(kvm)) {
+		/* content of memory is updated, mark pages dirty */
+		for (i = 0; i < npages; i++) {
+			set_page_dirty_lock(inpages[i]);
+			mark_page_accessed(inpages[i]);
+		}
+		/* unlock the user pages */
+		sev_unpin_memory(kvm, inpages, npages);
 	}
-	/* unlock the user pages */
-	sev_unpin_memory(kvm, inpages, npages);
+
 	return ret;
 }
 
@@ -609,9 +659,8 @@ static int sev_launch_update_priv_gfn_handler(struct kvm *kvm,
 			goto e_ret;
 		kvm_release_pfn_clean(pfn);
 	}
-	kvm_vm_set_region_attr(kvm, range->start, range->end,
-		true /* priv_attr */);
 
Quoted 9 lines
Quoted 7 lines
arch/x86/kvm/svm/sev.c +1 -1
+	kvm_vm_set_region_attr(kvm, range->start, range->end, KVM_MEMORY_ATTRIBUTE_PRIVATE);
 e_ret:
 	return ret;
 }
--

2.25.1

Regards, Tom

Thanks
Nikunj

#57 lore raw

Re: [PATCH RFC v7 11/64] KVM: SEV: Support private pages in LAUNCH_UPDATE_DATA

In reply to Michael Roth
Quoted 145 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

Pre-boot guest payload needs to be encrypted and VMM has copied it over to the private-fd. Add support to get the pfn from the memfile fd for encrypting the payload in-place.

---
 arch/x86/kvm/svm/sev.c | 79 ++++++++++++++++++++++++++++++++++--------
 1 file changed, 64 insertions(+), 15 deletions(-)
arch/x86/kvm/svm/sev.c +64 -15
index a7e4e3005786..ae4920aeb281 100644
@@ -107,6 +107,11 @@ static inline bool is_mirroring_enc_context(struct kvm *kvm)
 	return !!to_kvm_svm(kvm)->sev_info.enc_context_owner;
 }
 
+static bool kvm_is_upm_enabled(struct kvm *kvm)
+{
+	return kvm->arch.upm_mode;
+}
+
 /* Must be called with the sev_bitmap_lock held */
 static bool __sev_recycle_asids(int min_asid, int max_asid)
 {
@@ -382,6 +387,38 @@ static int sev_launch_start(struct kvm *kvm, struct kvm_sev_cmd *argp)
 	return ret;
 }
 
+static int sev_get_memfile_pfn_handler(struct kvm *kvm, struct kvm_gfn_range *range, void *data)
+{
+	struct kvm_memory_slot *memslot = range->slot;
+	struct page **pages = data;
+	int ret = 0, i = 0;
+	kvm_pfn_t pfn;
+	gfn_t gfn;
+
+	for (gfn = range->start; gfn < range->end; gfn++) {
+		int order;
+
+		ret = kvm_restricted_mem_get_pfn(memslot, gfn, &pfn, &order);
+		if (ret)
+			return ret;
+
+		if (is_error_noslot_pfn(pfn))
+			return -EFAULT;
+
+		pages[i++] = pfn_to_page(pfn);
+	}
+
+	return ret;
+}
+
+static int sev_get_memfile_pfn(struct kvm *kvm, unsigned long addr,
+			       unsigned long size, unsigned long npages,
+			       struct page **pages)
+{
+	return kvm_vm_do_hva_range_op(kvm, addr, size,
+				      sev_get_memfile_pfn_handler, pages);
+}
+
 static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr,
 				    unsigned long ulen, unsigned long *n,
 				    int write)
@@ -424,16 +461,25 @@ static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr,
 	if (!pages)
 		return ERR_PTR(-ENOMEM);
 
-	/* Pin the user virtual address. */
-	npinned = pin_user_pages_fast(uaddr, npages, write ? FOLL_WRITE : 0, pages);
-	if (npinned != npages) {
-		pr_err("SEV: Failure locking %lu pages.\n", npages);
-		ret = -ENOMEM;
-		goto err;
+	if (kvm_is_upm_enabled(kvm)) {
+		/* Get the PFN from memfile */
+		if (sev_get_memfile_pfn(kvm, uaddr, ulen, npages, pages)) {
+			pr_err("%s: ERROR: unable to find slot for uaddr %lx", __func__, uaddr);
+			ret = -ENOMEM;
+			goto err;
+		}
+	} else {
+		/* Pin the user virtual address. */
+		npinned = pin_user_pages_fast(uaddr, npages, write ? FOLL_WRITE : 0, pages);
+		if (npinned != npages) {
+			pr_err("SEV: Failure locking %lu pages.\n", npages);
+			ret = -ENOMEM;
+			goto err;
+		}
+		sev->pages_locked = locked;
 	}
 
 	*n = npages;
-	sev->pages_locked = locked;
 
 	return pages;
 
@@ -514,6 +560,7 @@ static int sev_launch_update_shared_gfn_handler(struct kvm *kvm,
 
 	size = (range->end - range->start) << PAGE_SHIFT;
 	vaddr_end = vaddr + size;
+	WARN_ON(size < PAGE_SIZE);
 
 	/* Lock the user memory. */
 	inpages = sev_pin_memory(kvm, vaddr, size, &npages, 1);
@@ -554,13 +601,16 @@ static int sev_launch_update_shared_gfn_handler(struct kvm *kvm,
 	}
 
 e_unpin:
-	/* content of memory is updated, mark pages dirty */
-	for (i = 0; i < npages; i++) {
-		set_page_dirty_lock(inpages[i]);
-		mark_page_accessed(inpages[i]);
+	if (!kvm_is_upm_enabled(kvm)) {
+		/* content of memory is updated, mark pages dirty */
+		for (i = 0; i < npages; i++) {
+			set_page_dirty_lock(inpages[i]);
+			mark_page_accessed(inpages[i]);
+		}
+		/* unlock the user pages */
+		sev_unpin_memory(kvm, inpages, npages);
 	}
-	/* unlock the user pages */
-	sev_unpin_memory(kvm, inpages, npages);
+
 	return ret;
 }
 
@@ -609,9 +659,8 @@ static int sev_launch_update_priv_gfn_handler(struct kvm *kvm,
 			goto e_ret;
 		kvm_release_pfn_clean(pfn);
 	}
-	kvm_vm_set_region_attr(kvm, range->start, range->end,
-		true /* priv_attr */);
 
+	kvm_vm_set_region_attr(kvm, range->start, range->end, KVM_MEMORY_ATTRIBUTE_PRIVATE);
 e_ret:
 	return ret;
 }
Signature · 1 line
2.25.1

kvm_vm_set_region_attr() should be fixed already in:

https://lore.kernel.org/all/[email protected]/

BR, Jarkko

#58 lore raw

Re: [PATCH RFC v7 11/64] KVM: SEV: Support private pages in LAUNCH_UPDATE_DATA

Quoted 11 lines · Jarkko Sakkinen

arch/x86/kvm/svm/sev.c static int sev_launch_update_priv_gfn_handler(struct kvm *kvm, · line 663

Quoted 10 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

arch/x86/kvm/svm/sev.c +1 -2
@@ -609,9 +659,8 @@ static int sev_launch_update_priv_gfn_handler(struct kvm *kvm,
 			goto e_ret;
 		kvm_release_pfn_clean(pfn);
 	}
-	kvm_vm_set_region_attr(kvm, range->start, range->end,
-		true /* priv_attr */);
 
+	kvm_vm_set_region_attr(kvm, range->start, range->end, KVM_MEMORY_ATTRIBUTE_PRIVATE);

As the memory attribute is no more a boolean in the UPM series, I had this change.

Quoted 9 lines
Quoted 6 lines
arch/x86/kvm/svm/sev.c +0 -0
 e_ret:
 	return ret;
 }
Signature · 1 line
2.25.1

kvm_vm_set_region_attr() should be fixed already in:

Quoted 1 line

https://lore.kernel.org/all/[email protected]/

Will discuss with Mike and move this hunk to above patch.

Regards
Nikunj

maintainer
#59 lore raw

Re: [PATCH RFC v7 11/64] KVM: SEV: Support private pages in LAUNCH_UPDATE_DATA

In reply to Michael Roth
Quoted 3 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

Pre-boot guest payload needs to be encrypted and VMM has copied it

"has to have copied it over" I presume?

Quoted 2 lines

over to the private-fd. Add support to get the pfn from the memfile fd for encrypting the payload in-place.

Why is that a good thing?

I guess with UPM you're supposed to get the PFN of that encrypted guest payload from that memslot.

IOW, such commit messages are too laconic for my taste and you could try to explain more why this is happening instead of me having to "reverse-deduce" what you're doing from the code...

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#60 lore raw

Re: [PATCH RFC v7 11/64] KVM: SEV: Support private pages in LAUNCH_UPDATE_DATA

Quoted 6 lines · Borislav Petkov
Quoted 3 lines · Michael Roth

From: Nikunj A Dadhania <[email protected]>

Pre-boot guest payload needs to be encrypted and VMM has copied it

"has to have copied it over" I presume?

True, payload is being copied in patch 10/64 now.

Quoted 12 lines
Quoted 2 lines

over to the private-fd. Add support to get the pfn from the memfile fd for encrypting the payload in-place.

Why is that a good thing?

I guess with UPM you're supposed to get the PFN of that encrypted guest payload from that memslot.

IOW, such commit messages are too laconic for my taste and you could try to explain more why this is happening instead of me having to "reverse-deduce" what you're doing from the code...

I am updating the SEV related patches, will add more details in commit and send.

Regards
Nikunj

#61 lore raw

RFC PATCH v7 12/64 KVM: SEV: Implement .fault_is_private callback

In reply to Michael Roth

KVM MMU will use this to determine whether an #NPF should be serviced with restricted memory or not.

---
 arch/x86/kvm/svm/sev.c | 23 +++++++++++++++++++++++
 arch/x86/kvm/svm/svm.c |  2 ++
 arch/x86/kvm/svm/svm.h |  2 ++
 3 files changed, 27 insertions(+)
arch/x86/kvm/svm/sev.c +23 -0
index ae4920aeb281..6579ed218f6a 100644
@@ -3179,3 +3179,26 @@ void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector)
 
 	ghcb_set_sw_exit_info_2(svm->sev_es.ghcb, 1);
 }
+
+int sev_fault_is_private(struct kvm *kvm, gpa_t gpa, u64 error_code, bool *private_fault)
+{
+	gfn_t gfn = gpa_to_gfn(gpa);
+
+	if (!kvm_is_upm_enabled(kvm) || !sev_guest(kvm))
+		goto out_unhandled;
+
+	/*
+	 * For SEV, the hypervisor is not aware of implicit conversions in the
+	 * guest, so it relies purely on explicit conversions via
+	 * KVM_EXIT_HYPERCALL, so the resulting handling by userspace should
+	 * update the backing memory source accordingly. Therefore, the backing
+	 * source is the only indicator of whether the fault should be treated
+	 * as private or not.
+	 */
+	*private_fault = kvm_mem_is_private(kvm, gfn);
+
+	return 1;
+
+out_unhandled:
+	return 0;
+}
arch/x86/kvm/svm/svm.c +2 -0
index 7f3e4d91c0c6..fc7885869f7e 100644
@@ -4830,6 +4830,8 @@ static struct kvm_x86_ops svm_x86_ops __initdata = {
 
 	.vcpu_deliver_sipi_vector = svm_vcpu_deliver_sipi_vector,
 	.vcpu_get_apicv_inhibit_reasons = avic_vcpu_get_apicv_inhibit_reasons,
+
+	.fault_is_private = sev_fault_is_private,
 };
 
 /*
arch/x86/kvm/svm/svm.h +2 -0
index 4826e6cc611b..c760ec51a910 100644
@@ -683,6 +683,8 @@ void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector);
 void sev_es_prepare_switch_to_guest(struct sev_es_save_area *hostsa);
 void sev_es_unmap_ghcb(struct vcpu_svm *svm);
 
+int sev_fault_is_private(struct kvm *kvm, gpa_t gpa, u64 error_code, bool *private_fault);
+
 /* vmenter.S */
 
 void __svm_sev_es_vcpu_run(struct vcpu_svm *svm, bool spec_ctrl_intercepted);
Signature · 1 line
2.25.1
#62 lore raw

RFC PATCH v7 13/64 x86/cpufeatures: Add SEV-SNP CPU feature

In reply to Michael Roth

From: Brijesh Singh <[email protected]>

Add CPU feature detection for Secure Encrypted Virtualization with Secure Nested Paging. This feature adds a strong memory integrity protection to help prevent malicious hypervisor-based attacks like data replay, memory re-mapping, and more.

---
 arch/x86/include/asm/cpufeatures.h       | 1 +
 arch/x86/kernel/cpu/amd.c                | 5 +++--
 tools/arch/x86/include/asm/cpufeatures.h | 1 +
 3 files changed, 5 insertions(+), 2 deletions(-)
arch/x86/include/asm/cpufeatures.h +1 -0
index 1419c4e04d45..480b4eaef310 100644
@@ -420,6 +420,7 @@
 #define X86_FEATURE_SEV			(19*32+ 1) /* AMD Secure Encrypted Virtualization */
 #define X86_FEATURE_VM_PAGE_FLUSH	(19*32+ 2) /* "" VM Page Flush MSR is supported */
 #define X86_FEATURE_SEV_ES		(19*32+ 3) /* AMD Secure Encrypted Virtualization - Encrypted State */
+#define X86_FEATURE_SEV_SNP		(19*32+ 4) /* AMD Secure Encrypted Virtualization - Secure Nested Paging */
 #define X86_FEATURE_V_TSC_AUX		(19*32+ 9) /* "" Virtual TSC_AUX */
 #define X86_FEATURE_SME_COHERENT	(19*32+10) /* "" AMD hardware-enforced cache coherency */
 
arch/x86/kernel/cpu/amd.c +3 -2
index 860b60273df3..c7884198ad5b 100644
@@ -558,8 +558,8 @@ static void early_detect_mem_encrypt(struct cpuinfo_x86 *c)
 	 *	      SME feature (set in scattered.c).
 	 *	      If the kernel has not enabled SME via any means then
 	 *	      don't advertise the SME feature.
-	 *   For SEV: If BIOS has not enabled SEV then don't advertise the
-	 *            SEV and SEV_ES feature (set in scattered.c).
+	 *   For SEV: If BIOS has not enabled SEV then don't advertise SEV and
+	 *	      any additional functionality based on it.
 	 *
 	 *   In all cases, since support for SME and SEV requires long mode,
 	 *   don't advertise the feature under CONFIG_X86_32.
@@ -594,6 +594,7 @@ static void early_detect_mem_encrypt(struct cpuinfo_x86 *c)
 clear_sev:
 		setup_clear_cpu_cap(X86_FEATURE_SEV);
 		setup_clear_cpu_cap(X86_FEATURE_SEV_ES);
+		setup_clear_cpu_cap(X86_FEATURE_SEV_SNP);
 	}
 }
 
tools/arch/x86/include/asm/cpufeatures.h +1 -0
index b71f4f2ecdd5..e81606fcd2ab 100644
@@ -417,6 +417,7 @@
 #define X86_FEATURE_SEV			(19*32+ 1) /* AMD Secure Encrypted Virtualization */
 #define X86_FEATURE_VM_PAGE_FLUSH	(19*32+ 2) /* "" VM Page Flush MSR is supported */
 #define X86_FEATURE_SEV_ES		(19*32+ 3) /* AMD Secure Encrypted Virtualization - Encrypted State */
+#define X86_FEATURE_SEV_SNP		(19*32+ 4) /* AMD Secure Encrypted Virtualization - Secure Nested Paging */
 #define X86_FEATURE_V_TSC_AUX		(19*32+ 9) /* "" Virtual TSC_AUX */
 #define X86_FEATURE_SME_COHERENT	(19*32+10) /* "" AMD hardware-enforced cache coherency */
 
Signature · 1 line
2.25.1
maintainer
#63 lore raw

Re: [PATCH RFC v7 13/64] x86/cpufeatures: Add SEV-SNP CPU feature

Quoted 8 lines · Michael Roth

From: Brijesh Singh <[email protected]>

Add CPU feature detection for Secure Encrypted Virtualization with Secure Nested Paging. This feature adds a strong memory integrity protection to help prevent malicious hypervisor-based attacks like data replay, memory re-mapping, and more.

That points to some review feedback I've given - dunno if it is relevant.

Quoted 2 lines

I read this as Jarkko has handled this patch too. Is that the case?

Quoted 2 lines

Those last two are ok - you took ovef from Ashish.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#64 lore raw

Re: [PATCH RFC v7 13/64] x86/cpufeatures: Add SEV-SNP CPU feature

Quoted 10 lines · Borislav Petkov
Quoted 8 lines · Michael Roth

From: Brijesh Singh <[email protected]>

Add CPU feature detection for Secure Encrypted Virtualization with Secure Nested Paging. This feature adds a strong memory integrity protection to help prevent malicious hypervisor-based attacks like data replay, memory re-mapping, and more.

Quoted 7 lines

That points to some review feedback I've given - dunno if it is relevant.

Quoted 2 lines

I read this as Jarkko has handled this patch too. Is that the case?

Yes we shared some patches via an internal tree at some stages.

-Mike

Quoted 13 lines
Quoted 3 lines

Those last two are ok - you took ovef from Ashish.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer
#65 lore raw

Re: [PATCH RFC v7 13/64] x86/cpufeatures: Add SEV-SNP CPU feature

Quoted 14 lines · Michael Roth
Quoted 13 lines · Borislav Petkov
Quoted 8 lines · Michael Roth

From: Brijesh Singh <[email protected]>

Add CPU feature detection for Secure Encrypted Virtualization with Secure Nested Paging. This feature adds a strong memory integrity protection to help prevent malicious hypervisor-based attacks like data replay, memory re-mapping, and more.

That points to some review feedback I've given - dunno if it is relevant.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

I read this as Jarkko has handled this patch too. Is that the case?

Yes we shared some patches via an internal tree at some stages.

In the sense that, he took Brijesh's patch, then he did something with it(?) and then Ashish took it from him and then you took it from Ashish?

This is how I'm reading this SOB chain at least...

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#66 lore raw

Re: [PATCH RFC v7 13/64] x86/cpufeatures: Add SEV-SNP CPU feature

Quoted 18 lines · Borislav Petkov
Quoted 17 lines · Michael Roth
Quoted 16 lines · Borislav Petkov
Quoted 8 lines · Michael Roth

From: Brijesh Singh <[email protected]>

Add CPU feature detection for Secure Encrypted Virtualization with Secure Nested Paging. This feature adds a strong memory integrity protection to help prevent malicious hypervisor-based attacks like data replay, memory re-mapping, and more.

That points to some review feedback I've given - dunno if it is relevant.

Quoted 2 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

I read this as Jarkko has handled this patch too. Is that the case?

Yes we shared some patches via an internal tree at some stages.

In the sense that, he took Brijesh's patch, then he did something with it(?) and then Ashish took it from him and then you took it from Ashish?

Yes, I think he rebased Ashish's tree on a newer tree and added his SoB on patches that required any conflict resolutions or changes on his end, so we kept those intact since then.

-Mike

Quoted 8 lines

This is how I'm reading this SOB chain at least...

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer
#67 lore raw

Re: [PATCH RFC v7 13/64] x86/cpufeatures: Add SEV-SNP CPU feature

Quoted 3 lines · Michael Roth

Yes, I think he rebased Ashish's tree on a newer tree and added his SoB on patches that required any conflict resolutions or changes on his end, so we kept those intact since then.

Ok, that makes sense.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#68 lore raw

RFC PATCH v7 14/64 x86/sev: Add the host SEV-SNP initialization support

In reply to Michael Roth

From: Brijesh Singh <[email protected]>

The memory integrity guarantees of SEV-SNP are enforced through a new structure called the Reverse Map Table (RMP). The RMP is a single data structure shared across the system that contains one entry for every 4K page of DRAM that may be used by SEV-SNP VMs. The goal of RMP is to track the owner of each page of memory. Pages of memory can be owned by the hypervisor, owned by a specific VM or owned by the AMD-SP. See APM2 section 15.36.3 for more detail on RMP.

The RMP table is used to enforce access control to memory. The table itself is not directly writable by the software. New CPU instructions (RMPUPDATE, PVALIDATE, RMPADJUST) are used to manipulate the RMP entries.

Based on the platform configuration, the BIOS reserves the memory used for the RMP table. The start and end address of the RMP table must be queried by reading the RMP_BASE and RMP_END MSRs. If the RMP_BASE and RMP_END are not set then disable the SEV-SNP feature.

The SEV-SNP feature is enabled only after the RMP table is successfully initialized.

Also set SYSCFG.MFMD when enabling SNP as SEV-SNP FW >= 1.51 requires that SYSCFG.MFMD must be se

RMP table entry format is non-architectural and it can vary by processor and is defined by the PPR. Restrict SNP support on the known CPU model and family for which the RMP table entry format is currently defined for.

Signed-off-b: Ashish Kalra <[email protected]>

---
 arch/x86/include/asm/disabled-features.h |   8 +-
 arch/x86/include/asm/msr-index.h         |  11 +-
 arch/x86/kernel/sev.c                    | 180 +++++++++++++++++++++++
 3 files changed, 197 insertions(+), 2 deletions(-)
arch/x86/include/asm/disabled-features.h +7 -1
index 33d2cd04d254..9b5a2cc8064a 100644
@@ -87,6 +87,12 @@
 # define DISABLE_TDX_GUEST	(1 << (X86_FEATURE_TDX_GUEST & 31))
 #endif
 
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+# define DISABLE_SEV_SNP	0
+#else
+# define DISABLE_SEV_SNP	(1 << (X86_FEATURE_SEV_SNP & 31))
+#endif
+
 /*
  * Make sure to add features to the correct mask
  */
@@ -110,7 +116,7 @@
 			 DISABLE_ENQCMD)
 #define DISABLED_MASK17	0
 #define DISABLED_MASK18	0
-#define DISABLED_MASK19	0
+#define DISABLED_MASK19	(DISABLE_SEV_SNP)
 #define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 20)
 
 #endif /* _ASM_X86_DISABLED_FEATURES_H */
arch/x86/include/asm/msr-index.h +10 -1
index 10ac52705892..35100c630617 100644
@@ -565,6 +565,8 @@
 #define MSR_AMD64_SEV_ENABLED		BIT_ULL(MSR_AMD64_SEV_ENABLED_BIT)
 #define MSR_AMD64_SEV_ES_ENABLED	BIT_ULL(MSR_AMD64_SEV_ES_ENABLED_BIT)
 #define MSR_AMD64_SEV_SNP_ENABLED	BIT_ULL(MSR_AMD64_SEV_SNP_ENABLED_BIT)
+#define MSR_AMD64_RMP_BASE		0xc0010132
+#define MSR_AMD64_RMP_END		0xc0010133
 
 #define MSR_AMD64_VIRT_SPEC_CTRL	0xc001011f
 
@@ -649,7 +651,14 @@
 #define MSR_K8_TOP_MEM2			0xc001001d
 #define MSR_AMD64_SYSCFG		0xc0010010
 #define MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT	23
-#define MSR_AMD64_SYSCFG_MEM_ENCRYPT	BIT_ULL(MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT)
+#define MSR_AMD64_SYSCFG_MEM_ENCRYPT		BIT_ULL(MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT)
+#define MSR_AMD64_SYSCFG_SNP_EN_BIT		24
+#define MSR_AMD64_SYSCFG_SNP_EN		BIT_ULL(MSR_AMD64_SYSCFG_SNP_EN_BIT)
+#define MSR_AMD64_SYSCFG_SNP_VMPL_EN_BIT	25
+#define MSR_AMD64_SYSCFG_SNP_VMPL_EN		BIT_ULL(MSR_AMD64_SYSCFG_SNP_VMPL_EN_BIT)
+#define MSR_AMD64_SYSCFG_MFDM_BIT		19
+#define MSR_AMD64_SYSCFG_MFDM			BIT_ULL(MSR_AMD64_SYSCFG_MFDM_BIT)
+
 #define MSR_K8_INT_PENDING_MSG		0xc0010055
 /* C1E active bits in int pending message */
 #define K8_INTP_C1E_ACTIVE_MASK		0x18000000
arch/x86/kernel/sev.c +180 -0

207 lines not on this page · raw message

Signature · 1 line
2.25.1
#69 lore raw

Re: [PATCH RFC v7 14/64] x86/sev: Add the host SEV-SNP initialization support

arch/x86/include/asm/disabled-features.h line 95

Quoted 6 lines · Michael Roth
arch/x86/include/asm/disabled-features.h +6 -0
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+# define DISABLE_SEV_SNP       0
+#else
+# define DISABLE_SEV_SNP       (1 << (X86_FEATURE_SEV_SNP & 31))
+#endif
+

Would it make sense to split the SEV-* feature family into their own config flag(s) ?
I'm thinking in the context of SEV-SNP running on systems with Transparent SME enabled in the bios. In this case, enabling CONFIG_AMD_MEM_ENCRYPT will also enable SME in the kernel, which is a bit strange and not necessarily useful.
Commit 4e2c87949f2b ("crypto: ccp - When TSME and SME both detected notify user") highlights it.

--
Sabin.

Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

maintainer
#70 lore raw

Re: [PATCH RFC v7 14/64] x86/sev: Add the host SEV-SNP initialization support

Quoted 19 lines · Sabin Rapan

arch/x86/include/asm/disabled-features.h line 95

Quoted 6 lines · Michael Roth
arch/x86/include/asm/disabled-features.h +6 -0
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+# define DISABLE_SEV_SNP       0
+#else
+# define DISABLE_SEV_SNP       (1 << (X86_FEATURE_SEV_SNP & 31))
+#endif
+

Would it make sense to split the SEV-* feature family into their own config flag(s) ?
I'm thinking in the context of SEV-SNP running on systems with Transparent SME enabled in the bios. In this case, enabling CONFIG_AMD_MEM_ENCRYPT will also enable SME in the kernel, which is a bit strange and not necessarily useful.
Commit 4e2c87949f2b ("crypto: ccp - When TSME and SME both detected notify user") highlights it.

Yes, we plan to move the SNP host initialization stuff into a separate source file and under a different config flag such as CONFIG_KVM_AMD_SEV or something.

Thanks,
Ashish

#71 lore raw

Re: [PATCH RFC v7 14/64] x86/sev: Add the host SEV-SNP initialization support

In reply to Michael Roth
Quoted 199 lines · Michael Roth

From: Brijesh Singh <[email protected]>

The memory integrity guarantees of SEV-SNP are enforced through a new structure called the Reverse Map Table (RMP). The RMP is a single data structure shared across the system that contains one entry for every 4K page of DRAM that may be used by SEV-SNP VMs. The goal of RMP is to track the owner of each page of memory. Pages of memory can be owned by the hypervisor, owned by a specific VM or owned by the AMD-SP. See APM2 section 15.36.3 for more detail on RMP.

The RMP table is used to enforce access control to memory. The table itself is not directly writable by the software. New CPU instructions (RMPUPDATE, PVALIDATE, RMPADJUST) are used to manipulate the RMP entries.

Based on the platform configuration, the BIOS reserves the memory used for the RMP table. The start and end address of the RMP table must be queried by reading the RMP_BASE and RMP_END MSRs. If the RMP_BASE and RMP_END are not set then disable the SEV-SNP feature.

The SEV-SNP feature is enabled only after the RMP table is successfully initialized.

Also set SYSCFG.MFMD when enabling SNP as SEV-SNP FW >= 1.51 requires that SYSCFG.MFMD must be se

RMP table entry format is non-architectural and it can vary by processor and is defined by the PPR. Restrict SNP support on the known CPU model and family for which the RMP table entry format is currently defined for.

Signed-off-b: Ashish Kalra <[email protected]>

---
 arch/x86/include/asm/disabled-features.h |   8 +-
 arch/x86/include/asm/msr-index.h         |  11 +-
 arch/x86/kernel/sev.c                    | 180 +++++++++++++++++++++++
 3 files changed, 197 insertions(+), 2 deletions(-)
arch/x86/include/asm/disabled-features.h +7 -1
index 33d2cd04d254..9b5a2cc8064a 100644
@@ -87,6 +87,12 @@
 # define DISABLE_TDX_GUEST	(1 << (X86_FEATURE_TDX_GUEST & 31))
 #endif
 
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+# define DISABLE_SEV_SNP	0
+#else
+# define DISABLE_SEV_SNP	(1 << (X86_FEATURE_SEV_SNP & 31))
+#endif
+
 /*
  * Make sure to add features to the correct mask
  */
@@ -110,7 +116,7 @@
 			 DISABLE_ENQCMD)
 #define DISABLED_MASK17	0
 #define DISABLED_MASK18	0
-#define DISABLED_MASK19	0
+#define DISABLED_MASK19	(DISABLE_SEV_SNP)
 #define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 20)
 
 #endif /* _ASM_X86_DISABLED_FEATURES_H */
arch/x86/include/asm/msr-index.h +10 -1
index 10ac52705892..35100c630617 100644
@@ -565,6 +565,8 @@
 #define MSR_AMD64_SEV_ENABLED		BIT_ULL(MSR_AMD64_SEV_ENABLED_BIT)
 #define MSR_AMD64_SEV_ES_ENABLED	BIT_ULL(MSR_AMD64_SEV_ES_ENABLED_BIT)
 #define MSR_AMD64_SEV_SNP_ENABLED	BIT_ULL(MSR_AMD64_SEV_SNP_ENABLED_BIT)
+#define MSR_AMD64_RMP_BASE		0xc0010132
+#define MSR_AMD64_RMP_END		0xc0010133
 
 #define MSR_AMD64_VIRT_SPEC_CTRL	0xc001011f
 
@@ -649,7 +651,14 @@
 #define MSR_K8_TOP_MEM2			0xc001001d
 #define MSR_AMD64_SYSCFG		0xc0010010
 #define MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT	23
-#define MSR_AMD64_SYSCFG_MEM_ENCRYPT	BIT_ULL(MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT)
+#define MSR_AMD64_SYSCFG_MEM_ENCRYPT		BIT_ULL(MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT)
+#define MSR_AMD64_SYSCFG_SNP_EN_BIT		24
+#define MSR_AMD64_SYSCFG_SNP_EN		BIT_ULL(MSR_AMD64_SYSCFG_SNP_EN_BIT)
+#define MSR_AMD64_SYSCFG_SNP_VMPL_EN_BIT	25
+#define MSR_AMD64_SYSCFG_SNP_VMPL_EN		BIT_ULL(MSR_AMD64_SYSCFG_SNP_VMPL_EN_BIT)
+#define MSR_AMD64_SYSCFG_MFDM_BIT		19
+#define MSR_AMD64_SYSCFG_MFDM			BIT_ULL(MSR_AMD64_SYSCFG_MFDM_BIT)
+
 #define MSR_K8_INT_PENDING_MSG		0xc0010055
 /* C1E active bits in int pending message */
 #define K8_INTP_C1E_ACTIVE_MASK		0x18000000
arch/x86/kernel/sev.c +70 -0
index a428c62330d3..687a91284506 100644
@@ -22,6 +22,9 @@
 #include <linux/efi.h>
 #include <linux/platform_device.h>
 #include <linux/io.h>
+#include <linux/cpumask.h>
+#include <linux/iommu.h>
+#include <linux/amd-iommu.h>
 
 #include <asm/cpu_entry_area.h>
 #include <asm/stacktrace.h>
@@ -38,6 +41,7 @@
 #include <asm/apic.h>
 #include <asm/cpuid.h>
 #include <asm/cmdline.h>
+#include <asm/iommu.h>
 
 #define DR7_RESET_VALUE        0x400
 
@@ -57,6 +61,12 @@
 #define AP_INIT_CR0_DEFAULT		0x60000010
 #define AP_INIT_MXCSR_DEFAULT		0x1f80
 
+/*
+ * The first 16KB from the RMP_BASE is used by the processor for the
+ * bookkeeping, the range needs to be added during the RMP entry lookup.
+ */
+#define RMPTABLE_CPU_BOOKKEEPING_SZ	0x4000
+
 /* For early boot hypervisor communication in SEV-ES enabled guests */
 static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
 
@@ -69,6 +79,9 @@ static struct ghcb *boot_ghcb __section(".data");
 /* Bitmap of SEV features supported by the hypervisor */
 static u64 sev_hv_features __ro_after_init;
 
+static unsigned long rmptable_start __ro_after_init;
+static unsigned long rmptable_end __ro_after_init;
+
 /* #VC handler runtime per-CPU data */
 struct sev_es_runtime_data {
 	struct ghcb ghcb_page;
@@ -2260,3 +2273,170 @@ static int __init snp_init_platform_device(void)
 	return 0;
 }
 device_initcall(snp_init_platform_device);
+
+#undef pr_fmt
+#define pr_fmt(fmt)	"SEV-SNP: " fmt
+
+static int __mfd_enable(unsigned int cpu)
+{
+	u64 val;
+
+	if (!cpu_feature_enabled(X86_FEATURE_SEV_SNP))
+		return 0;
+
+	rdmsrl(MSR_AMD64_SYSCFG, val);
+
+	val |= MSR_AMD64_SYSCFG_MFDM;
+
+	wrmsrl(MSR_AMD64_SYSCFG, val);
+
+	return 0;
+}
+
+static __init void mfd_enable(void *arg)
+{
+	__mfd_enable(smp_processor_id());
+}
+
+static int __snp_enable(unsigned int cpu)
+{
+	u64 val;
+
+	if (!cpu_feature_enabled(X86_FEATURE_SEV_SNP))
+		return 0;
+
+	rdmsrl(MSR_AMD64_SYSCFG, val);
+
+	val |= MSR_AMD64_SYSCFG_SNP_EN;
+	val |= MSR_AMD64_SYSCFG_SNP_VMPL_EN;
+
+	wrmsrl(MSR_AMD64_SYSCFG, val);
+
+	return 0;
+}
+
+static __init void snp_enable(void *arg)
+{
+	__snp_enable(smp_processor_id());
+}
+
+static bool get_rmptable_info(u64 *start, u64 *len)
+{
+	u64 calc_rmp_sz, rmp_sz, rmp_base, rmp_end;
+
+	rdmsrl(MSR_AMD64_RMP_BASE, rmp_base);
+	rdmsrl(MSR_AMD64_RMP_END, rmp_end);
+
+	if (!rmp_base || !rmp_end) {
+		pr_err("Memory for the RMP table has not been reserved by BIOS\n");
+		return false;

arch/x86/kernel/sev.c static int __init snp_init_platform_device(void) · line 2342

Quoted 10 lines
arch/x86/kernel/sev.c +10 -0
+	}
+
+	rmp_sz = rmp_end - rmp_base + 1;
+
+	/*
+	 * Calculate the amount the memory that must be reserved by the BIOS to
+	 * address the whole RAM. The reserved memory should also cover the
+	 * RMP table itself.
+	 */
+	calc_rmp_sz = (((rmp_sz >> PAGE_SHIFT) + totalram_pages()) << 4) + RMPTABLE_CPU_BOOKKEEPING_SZ;

Since the rmptable is indexed by page number, I believe this check should be using max_pfn:

    calc_rmp_sz = (max_pfn << 4) + RMPTABLE_CPU_BOOKKEEPING_SZ;

This accounts for holes/offsets in the memory map which lead to the top of memory having pfn > totalram_pages().

Quoted 103 lines
arch/x86/kernel/sev.c +100 -0
+
+	if (calc_rmp_sz > rmp_sz) {
+		pr_err("Memory reserved for the RMP table does not cover full system RAM (expected 0x%llx got 0x%llx)\n",
+		       calc_rmp_sz, rmp_sz);
+		return false;
+	}
+
+	*start = rmp_base;
+	*len = rmp_sz;
+
+	pr_info("RMP table physical address [0x%016llx - 0x%016llx]\n", rmp_base, rmp_end);
+
+	return true;
+}
+
+static __init int __snp_rmptable_init(void)
+{
+	u64 rmp_base, sz;
+	void *start;
+	u64 val;
+
+	if (!get_rmptable_info(&rmp_base, &sz))
+		return 1;
+
+	start = memremap(rmp_base, sz, MEMREMAP_WB);
+	if (!start) {
+		pr_err("Failed to map RMP table addr 0x%llx size 0x%llx\n", rmp_base, sz);
+		return 1;
+	}
+
+	/*
+	 * Check if SEV-SNP is already enabled, this can happen in case of
+	 * kexec boot.
+	 */
+	rdmsrl(MSR_AMD64_SYSCFG, val);
+	if (val & MSR_AMD64_SYSCFG_SNP_EN)
+		goto skip_enable;
+
+	/* Initialize the RMP table to zero */
+	memset(start, 0, sz);
+
+	/* Flush the caches to ensure that data is written before SNP is enabled. */
+	wbinvd_on_all_cpus();
+
+	/* MFDM must be enabled on all the CPUs prior to enabling SNP. */
+	on_each_cpu(mfd_enable, NULL, 1);
+
+	/* Enable SNP on all CPUs. */
+	on_each_cpu(snp_enable, NULL, 1);
+
+skip_enable:
+	rmptable_start = (unsigned long)start;
+	rmptable_end = rmptable_start + sz - 1;
+
+	return 0;
+}
+
+static int __init snp_rmptable_init(void)
+{
+	int family, model;
+
+	if (!cpu_feature_enabled(X86_FEATURE_SEV_SNP))
+		return 0;
+
+	family = boot_cpu_data.x86;
+	model  = boot_cpu_data.x86_model;
+
+	/*
+	 * RMP table entry format is not architectural and it can vary by processor and
+	 * is defined by the per-processor PPR. Restrict SNP support on the known CPU
+	 * model and family for which the RMP table entry format is currently defined for.
+	 */
+	if (family != 0x19 || model > 0xaf)
+		goto nosnp;
+
+	if (amd_iommu_snp_enable())
+		goto nosnp;
+
+	if (__snp_rmptable_init())
+		goto nosnp;
+
+	cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "x86/rmptable_init:online", __snp_enable, NULL);
+
+	return 0;
+
+nosnp:
+	setup_clear_cpu_cap(X86_FEATURE_SEV_SNP);
+	return -ENOSYS;
+}
+
+/*
+ * This must be called after the PCI subsystem. This is because amd_iommu_snp_enable()
+ * is called to ensure the IOMMU supports the SEV-SNP feature, which can only be
+ * called after subsys_initcall().
+ *
+ * NOTE: IOMMU is enforced by SNP to ensure that hypervisor cannot program DMA
+ * directly into guest private memory. In case of SNP, the IOMMU ensures that
+ * the page(s) used for DMA are hypervisor owned.
+ */
+fs_initcall(snp_rmptable_init);
Signature · 1 line
2.25.1
maintainer
#72 lore raw

Re: [PATCH RFC v7 14/64] x86/sev: Add the host SEV-SNP initialization support

Hello Jeremi,

Quoted 214 lines · Jeremi Piotrowski
Quoted 199 lines · Michael Roth

From: Brijesh Singh <[email protected]>

The memory integrity guarantees of SEV-SNP are enforced through a new structure called the Reverse Map Table (RMP). The RMP is a single data structure shared across the system that contains one entry for every 4K page of DRAM that may be used by SEV-SNP VMs. The goal of RMP is to track the owner of each page of memory. Pages of memory can be owned by the hypervisor, owned by a specific VM or owned by the AMD-SP. See APM2 section 15.36.3 for more detail on RMP.

The RMP table is used to enforce access control to memory. The table itself is not directly writable by the software. New CPU instructions (RMPUPDATE, PVALIDATE, RMPADJUST) are used to manipulate the RMP entries.

Based on the platform configuration, the BIOS reserves the memory used for the RMP table. The start and end address of the RMP table must be queried by reading the RMP_BASE and RMP_END MSRs. If the RMP_BASE and RMP_END are not set then disable the SEV-SNP feature.

The SEV-SNP feature is enabled only after the RMP table is successfully initialized.

Also set SYSCFG.MFMD when enabling SNP as SEV-SNP FW >= 1.51 requires that SYSCFG.MFMD must be se

RMP table entry format is non-architectural and it can vary by processor and is defined by the PPR. Restrict SNP support on the known CPU model and family for which the RMP table entry format is currently defined for.

Signed-off-b: Ashish Kalra <[email protected]>

---
  arch/x86/include/asm/disabled-features.h |   8 +-
  arch/x86/include/asm/msr-index.h         |  11 +-
  arch/x86/kernel/sev.c                    | 180 +++++++++++++++++++++++
  3 files changed, 197 insertions(+), 2 deletions(-)
arch/x86/include/asm/disabled-features.h +7 -1
index 33d2cd04d254..9b5a2cc8064a 100644
@@ -87,6 +87,12 @@
  # define DISABLE_TDX_GUEST	(1 << (X86_FEATURE_TDX_GUEST & 31))
  #endif
 
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+# define DISABLE_SEV_SNP	0
+#else
+# define DISABLE_SEV_SNP	(1 << (X86_FEATURE_SEV_SNP & 31))
+#endif
+
  /*
   * Make sure to add features to the correct mask
   */
@@ -110,7 +116,7 @@
  			 DISABLE_ENQCMD)
  #define DISABLED_MASK17	0
  #define DISABLED_MASK18	0
-#define DISABLED_MASK19	0
+#define DISABLED_MASK19	(DISABLE_SEV_SNP)
  #define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 20)
 
  #endif /* _ASM_X86_DISABLED_FEATURES_H */
arch/x86/include/asm/msr-index.h +10 -1
index 10ac52705892..35100c630617 100644
@@ -565,6 +565,8 @@
  #define MSR_AMD64_SEV_ENABLED		BIT_ULL(MSR_AMD64_SEV_ENABLED_BIT)
  #define MSR_AMD64_SEV_ES_ENABLED	BIT_ULL(MSR_AMD64_SEV_ES_ENABLED_BIT)
  #define MSR_AMD64_SEV_SNP_ENABLED	BIT_ULL(MSR_AMD64_SEV_SNP_ENABLED_BIT)
+#define MSR_AMD64_RMP_BASE		0xc0010132
+#define MSR_AMD64_RMP_END		0xc0010133
 
  #define MSR_AMD64_VIRT_SPEC_CTRL	0xc001011f
 
@@ -649,7 +651,14 @@
  #define MSR_K8_TOP_MEM2			0xc001001d
  #define MSR_AMD64_SYSCFG		0xc0010010
  #define MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT	23
-#define MSR_AMD64_SYSCFG_MEM_ENCRYPT	BIT_ULL(MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT)
+#define MSR_AMD64_SYSCFG_MEM_ENCRYPT		BIT_ULL(MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT)
+#define MSR_AMD64_SYSCFG_SNP_EN_BIT		24
+#define MSR_AMD64_SYSCFG_SNP_EN		BIT_ULL(MSR_AMD64_SYSCFG_SNP_EN_BIT)
+#define MSR_AMD64_SYSCFG_SNP_VMPL_EN_BIT	25
+#define MSR_AMD64_SYSCFG_SNP_VMPL_EN		BIT_ULL(MSR_AMD64_SYSCFG_SNP_VMPL_EN_BIT)
+#define MSR_AMD64_SYSCFG_MFDM_BIT		19
+#define MSR_AMD64_SYSCFG_MFDM			BIT_ULL(MSR_AMD64_SYSCFG_MFDM_BIT)
+
  #define MSR_K8_INT_PENDING_MSG		0xc0010055
  /* C1E active bits in int pending message */
  #define K8_INTP_C1E_ACTIVE_MASK		0x18000000
arch/x86/kernel/sev.c +70 -0
index a428c62330d3..687a91284506 100644
@@ -22,6 +22,9 @@
  #include <linux/efi.h>
  #include <linux/platform_device.h>
  #include <linux/io.h>
+#include <linux/cpumask.h>
+#include <linux/iommu.h>
+#include <linux/amd-iommu.h>
 
  #include <asm/cpu_entry_area.h>
  #include <asm/stacktrace.h>
@@ -38,6 +41,7 @@
  #include <asm/apic.h>
  #include <asm/cpuid.h>
  #include <asm/cmdline.h>
+#include <asm/iommu.h>
 
  #define DR7_RESET_VALUE        0x400
 
@@ -57,6 +61,12 @@
  #define AP_INIT_CR0_DEFAULT		0x60000010
  #define AP_INIT_MXCSR_DEFAULT		0x1f80
 
+/*
+ * The first 16KB from the RMP_BASE is used by the processor for the
+ * bookkeeping, the range needs to be added during the RMP entry lookup.
+ */
+#define RMPTABLE_CPU_BOOKKEEPING_SZ	0x4000
+
  /* For early boot hypervisor communication in SEV-ES enabled guests */
  static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
 
@@ -69,6 +79,9 @@ static struct ghcb *boot_ghcb __section(".data");
  /* Bitmap of SEV features supported by the hypervisor */
  static u64 sev_hv_features __ro_after_init;
 
+static unsigned long rmptable_start __ro_after_init;
+static unsigned long rmptable_end __ro_after_init;
+
  /* #VC handler runtime per-CPU data */
  struct sev_es_runtime_data {
  	struct ghcb ghcb_page;
@@ -2260,3 +2273,170 @@ static int __init snp_init_platform_device(void)
  	return 0;
  }
  device_initcall(snp_init_platform_device);
+
+#undef pr_fmt
+#define pr_fmt(fmt)	"SEV-SNP: " fmt
+
+static int __mfd_enable(unsigned int cpu)
+{
+	u64 val;
+
+	if (!cpu_feature_enabled(X86_FEATURE_SEV_SNP))
+		return 0;
+
+	rdmsrl(MSR_AMD64_SYSCFG, val);
+
+	val |= MSR_AMD64_SYSCFG_MFDM;
+
+	wrmsrl(MSR_AMD64_SYSCFG, val);
+
+	return 0;
+}
+
+static __init void mfd_enable(void *arg)
+{
+	__mfd_enable(smp_processor_id());
+}
+
+static int __snp_enable(unsigned int cpu)
+{
+	u64 val;
+
+	if (!cpu_feature_enabled(X86_FEATURE_SEV_SNP))
+		return 0;
+
+	rdmsrl(MSR_AMD64_SYSCFG, val);
+
+	val |= MSR_AMD64_SYSCFG_SNP_EN;
+	val |= MSR_AMD64_SYSCFG_SNP_VMPL_EN;
+
+	wrmsrl(MSR_AMD64_SYSCFG, val);
+
+	return 0;
+}
+
+static __init void snp_enable(void *arg)
+{
+	__snp_enable(smp_processor_id());
+}
+
+static bool get_rmptable_info(u64 *start, u64 *len)
+{
+	u64 calc_rmp_sz, rmp_sz, rmp_base, rmp_end;
+
+	rdmsrl(MSR_AMD64_RMP_BASE, rmp_base);
+	rdmsrl(MSR_AMD64_RMP_END, rmp_end);
+
+	if (!rmp_base || !rmp_end) {
+		pr_err("Memory for the RMP table has not been reserved by BIOS\n");
+		return false;

arch/x86/kernel/sev.c static int __init snp_init_platform_device(void) · line 2342

Quoted 10 lines
arch/x86/kernel/sev.c +10 -0
+	}
+
+	rmp_sz = rmp_end - rmp_base + 1;
+
+	/*
+	 * Calculate the amount the memory that must be reserved by the BIOS to
+	 * address the whole RAM. The reserved memory should also cover the
+	 * RMP table itself.
+	 */
+	calc_rmp_sz = (((rmp_sz >> PAGE_SHIFT) + totalram_pages()) << 4) + RMPTABLE_CPU_BOOKKEEPING_SZ;

Since the rmptable is indexed by page number, I believe this check should be using max_pfn:

Quoted 5 lines
     calc_rmp_sz = (max_pfn << 4) + RMPTABLE_CPU_BOOKKEEPING_SZ;

This accounts for holes/offsets in the memory map which lead to the top of memory having pfn > totalram_pages().

I agree that this check should use max. addressable pfn to account for holes in the physical address map. The BIOS will probably also be computing RMP table size to cover the entire physical memory, which should be max. addressable PFN.

But, then we primarly need to check that all available RAM pages are covered by the RMP table, so the above check is sufficient for that, right ?

Also, i assume that max_pfn will take into account any hotplugged memory as i do know that totalram_pages() handles hotplugged memory.

Thanks,
Ashish

Quoted 15 lines
Quoted 15 lines
arch/x86/kernel/sev.c +15 -0
+
+	if (calc_rmp_sz > rmp_sz) {
+		pr_err("Memory reserved for the RMP table does not cover full system RAM (expected 0x%llx got 0x%llx)\n",
+		       calc_rmp_sz, rmp_sz);
+		return false;
+	}
+
+	*start = rmp_base;
+	*len = rmp_sz;
+
+	pr_info("RMP table physical address [0x%016llx - 0x%016llx]\n", rmp_base, rmp_end);
+
+	return true;
+}
+
maintainer
#73 lore raw

Re: [PATCH RFC v7 14/64] x86/sev: Add the host SEV-SNP initialization support

Quoted 222 lines · Ashish

Hello Jeremi,

Quoted 213 lines · Jeremi Piotrowski
Quoted 202 lines · Michael Roth

From: Brijesh Singh <[email protected]>

The memory integrity guarantees of SEV-SNP are enforced through a new structure called the Reverse Map Table (RMP). The RMP is a single data structure shared across the system that contains one entry for every 4K page of DRAM that may be used by SEV-SNP VMs. The goal of RMP is to track the owner of each page of memory. Pages of memory can be owned by the hypervisor, owned by a specific VM or owned by the AMD-SP. See APM2 section 15.36.3 for more detail on RMP.

The RMP table is used to enforce access control to memory. The table itself is not directly writable by the software. New CPU instructions (RMPUPDATE, PVALIDATE, RMPADJUST) are used to manipulate the RMP entries.

Based on the platform configuration, the BIOS reserves the memory used for the RMP table. The start and end address of the RMP table must be queried by reading the RMP_BASE and RMP_END MSRs. If the RMP_BASE and RMP_END are not set then disable the SEV-SNP feature.

The SEV-SNP feature is enabled only after the RMP table is successfully initialized.

Also set SYSCFG.MFMD when enabling SNP as SEV-SNP FW >= 1.51 requires that SYSCFG.MFMD must be se

RMP table entry format is non-architectural and it can vary by processor and is defined by the PPR. Restrict SNP support on the known CPU model and family for which the RMP table entry format is currently defined for.

Signed-off-b: Ashish Kalra <[email protected]>

---
  arch/x86/include/asm/disabled-features.h |   8 +- arch/x86/include/asm/msr-index.h         |  11 +- arch/x86/kernel/sev.c                    | 180 +++++++++++++++++++++++

  3 files changed, 197 insertions(+), 2 deletions(-)
arch/x86/include/asm/disabled-features.h +7 -1
index 33d2cd04d254..9b5a2cc8064a 100644
@@ -87,6 +87,12 @@
   # define DISABLE_TDX_GUEST    (1 << (X86_FEATURE_TDX_GUEST & 31))
   #endif
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+# define DISABLE_SEV_SNP    0
+#else
+# define DISABLE_SEV_SNP    (1 << (X86_FEATURE_SEV_SNP & 31))
+#endif
+
   /*
    * Make sure to add features to the correct mask
    */
@@ -110,7 +116,7 @@
                DISABLE_ENQCMD)
   #define DISABLED_MASK17    0
   #define DISABLED_MASK18    0
-#define DISABLED_MASK19    0
+#define DISABLED_MASK19    (DISABLE_SEV_SNP)
   #define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 20)
   #endif /* _ASM_X86_DISABLED_FEATURES_H */
arch/x86/include/asm/msr-index.h +7 -1
index 10ac52705892..35100c630617 100644
@@ -565,6 +565,8 @@
   #define MSR_AMD64_SEV_ENABLED        BIT_ULL(MSR_AMD64_SEV_ENABLED_BIT)
   #define MSR_AMD64_SEV_ES_ENABLED    BIT_ULL(MSR_AMD64_SEV_ES_ENABLED_BIT)
   #define MSR_AMD64_SEV_SNP_ENABLED    BIT_ULL(MSR_AMD64_SEV_SNP_ENABLED_BIT)
+#define MSR_AMD64_RMP_BASE        0xc0010132
+#define MSR_AMD64_RMP_END        0xc0010133
   #define MSR_AMD64_VIRT_SPEC_CTRL    0xc001011f
@@ -649,7 +651,14 @@
   #define MSR_K8_TOP_MEM2            0xc001001d
   #define MSR_AMD64_SYSCFG        0xc0010010
   #define MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT    23
-#define MSR_AMD64_SYSCFG_MEM_ENCRYPT    
 BIT_ULL(MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT)
+#define MSR_AMD64_SYSCFG_MEM_ENCRYPT        
 BIT_ULL(MSR_AMD64_SYSCFG_MEM_ENCRYPT_BIT)
+#define MSR_AMD64_SYSCFG_SNP_EN_BIT        24
+#define MSR_AMD64_SYSCFG_SNP_EN        
 BIT_ULL(MSR_AMD64_SYSCFG_SNP_EN_BIT)
+#define MSR_AMD64_SYSCFG_SNP_VMPL_EN_BIT    25
+#define MSR_AMD64_SYSCFG_SNP_VMPL_EN        

BIT_ULL(MSR_AMD64_SYSCFG_SNP_VMPL_EN_BIT)

arch/x86/include/asm/msr-index.h +2 -0
+#define MSR_AMD64_SYSCFG_MFDM_BIT        19
+#define MSR_AMD64_SYSCFG_MFDM            

BIT_ULL(MSR_AMD64_SYSCFG_MFDM_BIT)
+
  #define MSR_K8_INT_PENDING_MSG        0xc0010055 /* C1E active bits in int pending message */ #define K8_INTP_C1E_ACTIVE_MASK        0x18000000

arch/x86/kernel/sev.c +7 -0
index a428c62330d3..687a91284506 100644
@@ -22,6 +22,9 @@
   #include <linux/efi.h>
   #include <linux/platform_device.h>
   #include <linux/io.h>
+#include <linux/cpumask.h>
+#include <linux/iommu.h>
+#include <linux/amd-iommu.h>
   #include <asm/cpu_entry_area.h>
   #include <asm/stacktrace.h>
@@ -38,6 +41,7 @@
   #include <asm/apic.h>
   #include <asm/cpuid.h>
   #include <asm/cmdline.h>
+#include <asm/iommu.h>
   #define DR7_RESET_VALUE        0x400
@@ -57,6 +61,12 @@
   #define AP_INIT_CR0_DEFAULT        0x60000010
   #define AP_INIT_MXCSR_DEFAULT        0x1f80
+/*
+ * The first 16KB from the RMP_BASE is used by the processor for the
+ * bookkeeping, the range needs to be added during the RMP entry 

lookup.
+ */

arch/x86/kernel/sev.c +2 -0
+#define RMPTABLE_CPU_BOOKKEEPING_SZ    0x4000
+

  /* For early boot hypervisor communication in SEV-ES enabled guests */ static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);

arch/x86/kernel/sev.c +59 -0
@@ -69,6 +79,9 @@ static struct ghcb *boot_ghcb __section(".data");
   /* Bitmap of SEV features supported by the hypervisor */
   static u64 sev_hv_features __ro_after_init;
+static unsigned long rmptable_start __ro_after_init;
+static unsigned long rmptable_end __ro_after_init;
+
   /* #VC handler runtime per-CPU data */
   struct sev_es_runtime_data {
       struct ghcb ghcb_page;
@@ -2260,3 +2273,170 @@ static int __init snp_init_platform_device(void)
       return 0;
   }
   device_initcall(snp_init_platform_device);
+
+#undef pr_fmt
+#define pr_fmt(fmt)    "SEV-SNP: " fmt
+
+static int __mfd_enable(unsigned int cpu)
+{
+    u64 val;
+
+    if (!cpu_feature_enabled(X86_FEATURE_SEV_SNP))
+        return 0;
+
+    rdmsrl(MSR_AMD64_SYSCFG, val);
+
+    val |= MSR_AMD64_SYSCFG_MFDM;
+
+    wrmsrl(MSR_AMD64_SYSCFG, val);
+
+    return 0;
+}
+
+static __init void mfd_enable(void *arg)
+{
+    __mfd_enable(smp_processor_id());
+}
+
+static int __snp_enable(unsigned int cpu)
+{
+    u64 val;
+
+    if (!cpu_feature_enabled(X86_FEATURE_SEV_SNP))
+        return 0;
+
+    rdmsrl(MSR_AMD64_SYSCFG, val);
+
+    val |= MSR_AMD64_SYSCFG_SNP_EN;
+    val |= MSR_AMD64_SYSCFG_SNP_VMPL_EN;
+
+    wrmsrl(MSR_AMD64_SYSCFG, val);
+
+    return 0;
+}
+
+static __init void snp_enable(void *arg)
+{
+    __snp_enable(smp_processor_id());
+}
+
+static bool get_rmptable_info(u64 *start, u64 *len)
+{
+    u64 calc_rmp_sz, rmp_sz, rmp_base, rmp_end;
+
+    rdmsrl(MSR_AMD64_RMP_BASE, rmp_base);
+    rdmsrl(MSR_AMD64_RMP_END, rmp_end);
+
+    if (!rmp_base || !rmp_end) {
+        pr_err("Memory for the RMP table has not been reserved by 

BIOS\n");
+        return false;
+    }

arch/x86/kernel/sev.c +5 -0
+
+    rmp_sz = rmp_end - rmp_base + 1;
+
+    /*
+     * Calculate the amount the memory that must be reserved by the 
Quoted 6 lines

BIOS to
+     * address the whole RAM. The reserved memory should also cover the +     * RMP table itself.
+     */
+    calc_rmp_sz = (((rmp_sz >> PAGE_SHIFT) + totalram_pages()) << 4)
+ RMPTABLE_CPU_BOOKKEEPING_SZ;

Since the rmptable is indexed by page number, I believe this check should be using max_pfn:

Quoted 5 lines

     calc_rmp_sz = (max_pfn << 4) + RMPTABLE_CPU_BOOKKEEPING_SZ;

This accounts for holes/offsets in the memory map which lead to the top of memory having pfn > totalram_pages().

Quoted 6 lines

I agree that this check should use max. addressable pfn to account for holes in the physical address map. The BIOS will probably also be computing RMP table size to cover the entire physical memory, which should be max. addressable PFN.

But, then we primarly need to check that all available RAM pages are covered by the RMP table, so the above check is sufficient for that, right ?

Also, i assume that max_pfn will take into account any hotplugged memory as i do know that totalram_pages() handles hotplugged memory.

But essentially you are correct, as RMP table is indexed by PFN, we have to take into account these physical memory holes so that we can have entries for the max DRAM SPA, so i will fix this to use the max. addressable PFN, i.e., max_pfn.

Thanks,
Ashish

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines

+
+    if (calc_rmp_sz > rmp_sz) {
+        pr_err("Memory reserved for the RMP table does not cover full system RAM (expected 0x%llx got 0x%llx)\n",
+               calc_rmp_sz, rmp_sz);
+        return false;
+    }

arch/x86/kernel/sev.c +5 -0
+
+    *start = rmp_base;
+    *len = rmp_sz;
+
+    pr_info("RMP table physical address [0x%016llx - 0x%016llx]\n", 

rmp_base, rmp_end);

arch/x86/kernel/sev.c +4 -0
+
+    return true;
+}
+
maintainer
#74 lore raw

Re: [PATCH RFC v7 14/64] x86/sev: Add the host SEV-SNP initialization support

In reply to Michael Roth
Quoted 19 lines · Michael Roth

From: Brijesh Singh <[email protected]>

The memory integrity guarantees of SEV-SNP are enforced through a new structure called the Reverse Map Table (RMP). The RMP is a single data structure shared across the system that contains one entry for every 4K page of DRAM that may be used by SEV-SNP VMs. The goal of RMP is to track the owner of each page of memory. Pages of memory can be owned by the hypervisor, owned by a specific VM or owned by the AMD-SP. See APM2 section 15.36.3 for more detail on RMP.

The RMP table is used to enforce access control to memory. The table itself is not directly writable by the software. New CPU instructions (RMPUPDATE, PVALIDATE, RMPADJUST) are used to manipulate the RMP entries.

Based on the platform configuration, the BIOS reserves the memory used for the RMP table. The start and end address of the RMP table must be queried by reading the RMP_BASE and RMP_END MSRs. If the RMP_BASE and RMP_END are not set then disable the SEV-SNP feature.

Quoted 5 lines

The SEV-SNP feature is enabled only after the RMP table is successfully initialized.

Also set SYSCFG.MFMD when enabling SNP as SEV-SNP FW >= 1.51 requires that SYSCFG.MFMD must be se

set.

Quoted 7 lines

RMP table entry format is non-architectural and it can vary by processor and is defined by the PPR. Restrict SNP support on the known CPU model and family for which the RMP table entry format is currently defined for.

Signed-off-b: Ashish Kalra <[email protected]>

	     ^^

Somebody ate a 'y' here. :)

Quoted 6 lines
---
 arch/x86/include/asm/disabled-features.h |   8 +-
 arch/x86/include/asm/msr-index.h         |  11 +-
 arch/x86/kernel/sev.c                    | 180 +++++++++++++++++++++++
 3 files changed, 197 insertions(+), 2 deletions(-)

...

Quoted 1 line

+static __init int __snp_rmptable_init(void)

Why is this one carved out of snp_rmptable_init() ?

Quoted 13 lines
arch/x86/kernel/sev.c +13 -0
+{
+	u64 rmp_base, sz;
+	void *start;
+	u64 val;
+
+	if (!get_rmptable_info(&rmp_base, &sz))
+		return 1;
+
+	start = memremap(rmp_base, sz, MEMREMAP_WB);
+	if (!start) {
+		pr_err("Failed to map RMP table addr 0x%llx size 0x%llx\n", rmp_base, sz);
+		return 1;
+	}

arch/x86/kernel/sev.c static int __init snp_init_platform_device(void) · line 2381

Quoted 10 lines
arch/x86/kernel/sev.c +10 -0
+
+	/*
+	 * Check if SEV-SNP is already enabled, this can happen in case of
+	 * kexec boot.
+	 */
+	rdmsrl(MSR_AMD64_SYSCFG, val);
+	if (val & MSR_AMD64_SYSCFG_SNP_EN)
+		goto skip_enable;
+
+	/* Initialize the RMP table to zero */

Useless comment.

arch/x86/kernel/sev.c static int __init snp_init_platform_device(void) · line 2391

Quoted 10 lines
arch/x86/kernel/sev.c +10 -0
+	memset(start, 0, sz);
+
+	/* Flush the caches to ensure that data is written before SNP is enabled. */
+	wbinvd_on_all_cpus();
+
+	/* MFDM must be enabled on all the CPUs prior to enabling SNP. */
+	on_each_cpu(mfd_enable, NULL, 1);
+
+	/* Enable SNP on all CPUs. */
+	on_each_cpu(snp_enable, NULL, 1);

What happens if someone boots the machine with maxcpus=N, where N is less than all CPUs on the machine? The hotplug notifier should handle it but have you checked that it works fine?

Quoted 6 lines
arch/x86/kernel/sev.c +6 -0
+skip_enable:
+	rmptable_start = (unsigned long)start;
+	rmptable_end = rmptable_start + sz - 1;
+
+	return 0;
+}

arch/x86/kernel/sev.c static int __init snp_init_platform_device(void) · line 2408

Quoted 10 lines
arch/x86/kernel/sev.c +10 -0
+
+static int __init snp_rmptable_init(void)
+{
+	int family, model;
+
+	if (!cpu_feature_enabled(X86_FEATURE_SEV_SNP))
+		return 0;
+
+	family = boot_cpu_data.x86;
+	model  = boot_cpu_data.x86_model;

Looks useless - just use boot_cpu_data directly below.

Quoted 13 lines
arch/x86/kernel/sev.c +13 -0
+
+	/*
+	 * RMP table entry format is not architectural and it can vary by processor and
+	 * is defined by the per-processor PPR. Restrict SNP support on the known CPU
+	 * model and family for which the RMP table entry format is currently defined for.
+	 */
+	if (family != 0x19 || model > 0xaf)
+		goto nosnp;
+
+	if (amd_iommu_snp_enable())
+		goto nosnp;
+
+	if (__snp_rmptable_init())

arch/x86/kernel/sev.c static int __init snp_init_platform_device(void) · line 2431

Quoted 10 lines
arch/x86/kernel/sev.c +10 -0
+		goto nosnp;
+
+	cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "x86/rmptable_init:online", __snp_enable, NULL);
+
+	return 0;
+
+nosnp:
+	setup_clear_cpu_cap(X86_FEATURE_SEV_SNP);
+	return -ENOSYS;
+}

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#75 lore raw

RFC PATCH v7 15/64 x86/sev: Add RMP entry lookup helpers

In reply to Michael Roth

From: Brijesh Singh <[email protected]>

The snp_lookup_rmpentry() can be used by the host to read the RMP entry for a given page. The RMP entry format is documented in AMD PPR, see
https://bugzilla.kernel.org/attachment.cgi?id=296015.

---
 arch/x86/include/asm/sev.h |  4 +-
 arch/x86/kernel/sev.c      | 83 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 86 insertions(+), 1 deletion(-)
arch/x86/include/asm/sev.h +3 -1
index ebc271bb6d8e..8d3ce2ad27da 100644
@@ -83,7 +83,7 @@ extern bool handle_vc_boot_ghcb(struct pt_regs *regs);
 
 /* RMP page size */
 #define RMP_PG_SIZE_4K			0
-
+#define RMP_TO_X86_PG_LEVEL(level)	(((level) == RMP_PG_SIZE_4K) ? PG_LEVEL_4K : PG_LEVEL_2M)
 #define RMPADJUST_VMSA_PAGE_BIT		BIT(16)
 
 /* SNP Guest message request */
@@ -197,6 +197,7 @@ void snp_set_wakeup_secondary_cpu(void);
 bool snp_init(struct boot_params *bp);
 void __init __noreturn snp_abort(void);
 int snp_issue_guest_request(u64 exit_code, struct snp_req_data *input, unsigned long *fw_err);
+int snp_lookup_rmpentry(u64 pfn, int *level);
 #else
 static inline void sev_es_ist_enter(struct pt_regs *regs) { }
 static inline void sev_es_ist_exit(void) { }
@@ -221,6 +222,7 @@ static inline int snp_issue_guest_request(u64 exit_code, struct snp_req_data *in
 {
 	return -ENOTTY;
 }
+static inline int snp_lookup_rmpentry(u64 pfn, int *level) { return 0; }
 #endif
 
 #endif
arch/x86/kernel/sev.c +83 -0

97 lines not on this page · raw message

Signature · 1 line
2.25.1
#76 lore raw

RFC PATCH v7 16/64 x86/sev: Add helper functions for RMPUPDATE and PSMASH instruction

In reply to Michael Roth

From: Brijesh Singh <[email protected]>

The RMPUPDATE instruction writes a new RMP entry in the RMP Table. The hypervisor will use the instruction to add pages to the RMP table. See APM3 for details on the instruction operations.

The PSMASH instruction expands a 2MB RMP entry into a corresponding set of contiguous 4KB-Page RMP entries. The hypervisor will use this instruction to adjust the RMP entry without invalidating the previous RMP entry.

Add the following external interface API functions:

int psmash(u64 pfn);
psmash is used to smash a 2MB aligned page into 4K pages while preserving the Validated bit in the RMP.

int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int asid, bool immutable); Used to assign a page to guest using the RMPUPDATE instruction.

int rmp_make_shared(u64 pfn, enum pg_level level);
Used to transition a page to hypervisor/shared state using the RMPUPDATE instruction.

[mdr: add RMPUPDATE retry logic for transient FAIL_OVERLAP errors]
---
 arch/x86/include/asm/sev.h | 24 ++++++++++
 arch/x86/kernel/sev.c      | 95 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 119 insertions(+)
arch/x86/include/asm/sev.h +24 -0
index 8d3ce2ad27da..4eeedcaca593 100644
@@ -80,10 +80,15 @@ extern bool handle_vc_boot_ghcb(struct pt_regs *regs);
 
 /* Software defined (when rFlags.CF = 1) */
 #define PVALIDATE_FAIL_NOUPDATE		255
+/* RMUPDATE detected 4K page and 2MB page overlap. */
+#define RMPUPDATE_FAIL_OVERLAP		7
 
 /* RMP page size */
 #define RMP_PG_SIZE_4K			0
+#define RMP_PG_SIZE_2M			1
 #define RMP_TO_X86_PG_LEVEL(level)	(((level) == RMP_PG_SIZE_4K) ? PG_LEVEL_4K : PG_LEVEL_2M)
+#define X86_TO_RMP_PG_LEVEL(level)	(((level) == PG_LEVEL_4K) ? RMP_PG_SIZE_4K : RMP_PG_SIZE_2M)
+
 #define RMPADJUST_VMSA_PAGE_BIT		BIT(16)
 
 /* SNP Guest message request */
@@ -133,6 +138,15 @@ struct snp_secrets_page_layout {
 	u8 rsvd3[3840];
 } __packed;
 
+struct rmp_state {
+	u64 gpa;
+	u8 assigned;
+	u8 pagesize;
+	u8 immutable;
+	u8 rsvd;
+	u32 asid;
+} __packed;
+
 #ifdef CONFIG_AMD_MEM_ENCRYPT
 extern struct static_key_false sev_es_enable_key;
 extern void __sev_es_ist_enter(struct pt_regs *regs);
@@ -198,6 +212,9 @@ bool snp_init(struct boot_params *bp);
 void __init __noreturn snp_abort(void);
 int snp_issue_guest_request(u64 exit_code, struct snp_req_data *input, unsigned long *fw_err);
 int snp_lookup_rmpentry(u64 pfn, int *level);
+int psmash(u64 pfn);
+int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int asid, bool immutable);
+int rmp_make_shared(u64 pfn, enum pg_level level);
 #else
 static inline void sev_es_ist_enter(struct pt_regs *regs) { }
 static inline void sev_es_ist_exit(void) { }
@@ -223,6 +240,13 @@ static inline int snp_issue_guest_request(u64 exit_code, struct snp_req_data *in
 	return -ENOTTY;
 }
 static inline int snp_lookup_rmpentry(u64 pfn, int *level) { return 0; }
+static inline int psmash(u64 pfn) { return -ENXIO; }
+static inline int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int asid,
+				   bool immutable)
+{
+	return -ENODEV;
+}
+static inline int rmp_make_shared(u64 pfn, enum pg_level level) { return -ENODEV; }
 #endif
 
 #endif
arch/x86/kernel/sev.c +95 -0

98 lines not on this page · raw message

Signature · 1 line
2.25.1
#77 lore raw

Re: [PATCH RFC v7 16/64] x86/sev: Add helper functions for RMPUPDATE and PSMASH instruction

Quoted 102 lines · Michael Roth

From: Brijesh Singh <[email protected]>

The RMPUPDATE instruction writes a new RMP entry in the RMP Table. The hypervisor will use the instruction to add pages to the RMP table. See APM3 for details on the instruction operations.

The PSMASH instruction expands a 2MB RMP entry into a corresponding set of contiguous 4KB-Page RMP entries. The hypervisor will use this instruction to adjust the RMP entry without invalidating the previous RMP entry.

Add the following external interface API functions:

int psmash(u64 pfn);
psmash is used to smash a 2MB aligned page into 4K pages while preserving the Validated bit in the RMP.

int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int asid, bool immutable); Used to assign a page to guest using the RMPUPDATE instruction.

int rmp_make_shared(u64 pfn, enum pg_level level);
Used to transition a page to hypervisor/shared state using the RMPUPDATE instruction.

[mdr: add RMPUPDATE retry logic for transient FAIL_OVERLAP errors]
---
  arch/x86/include/asm/sev.h | 24 ++++++++++
  arch/x86/kernel/sev.c      | 95 ++++++++++++++++++++++++++++++++++++++
  2 files changed, 119 insertions(+)
arch/x86/include/asm/sev.h +24 -0
index 8d3ce2ad27da..4eeedcaca593 100644
@@ -80,10 +80,15 @@ extern bool handle_vc_boot_ghcb(struct pt_regs *regs);
 
  /* Software defined (when rFlags.CF = 1) */
  #define PVALIDATE_FAIL_NOUPDATE                255
+/* RMUPDATE detected 4K page and 2MB page overlap. */
+#define RMPUPDATE_FAIL_OVERLAP         7
 
  /* RMP page size */
  #define RMP_PG_SIZE_4K                 0
+#define RMP_PG_SIZE_2M                 1
  #define RMP_TO_X86_PG_LEVEL(level)     (((level) == RMP_PG_SIZE_4K) ? PG_LEVEL_4K : PG_LEVEL_2M)
+#define X86_TO_RMP_PG_LEVEL(level)     (((level) == PG_LEVEL_4K) ? RMP_PG_SIZE_4K : RMP_PG_SIZE_2M)
+
  #define RMPADJUST_VMSA_PAGE_BIT                BIT(16)
 
  /* SNP Guest message request */
@@ -133,6 +138,15 @@ struct snp_secrets_page_layout {
         u8 rsvd3[3840];
  } __packed;
 
+struct rmp_state {
+       u64 gpa;
+       u8 assigned;
+       u8 pagesize;
+       u8 immutable;
+       u8 rsvd;
+       u32 asid;
+} __packed;
+
  #ifdef CONFIG_AMD_MEM_ENCRYPT
  extern struct static_key_false sev_es_enable_key;
  extern void __sev_es_ist_enter(struct pt_regs *regs);
@@ -198,6 +212,9 @@ bool snp_init(struct boot_params *bp);
  void __init __noreturn snp_abort(void);
  int snp_issue_guest_request(u64 exit_code, struct snp_req_data *input, unsigned long *fw_err);
  int snp_lookup_rmpentry(u64 pfn, int *level);
+int psmash(u64 pfn);
+int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int asid, bool immutable);
+int rmp_make_shared(u64 pfn, enum pg_level level);
  #else
  static inline void sev_es_ist_enter(struct pt_regs *regs) { }
  static inline void sev_es_ist_exit(void) { }
@@ -223,6 +240,13 @@ static inline int snp_issue_guest_request(u64 exit_code, struct snp_req_data *in
         return -ENOTTY;
  }
  static inline int snp_lookup_rmpentry(u64 pfn, int *level) { return 0; }
+static inline int psmash(u64 pfn) { return -ENXIO; }
+static inline int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int asid,
+                                  bool immutable)
+{
+       return -ENODEV;
+}
+static inline int rmp_make_shared(u64 pfn, enum pg_level level) { return -ENODEV; }
  #endif
 
  #endif
arch/x86/kernel/sev.c +2 -0
index 706675561f49..67035d34adad 100644
@@ -2523,3 +2523,98 @@ int snp_lookup_rmpentry(u64 pfn, int *level)
         return !!rmpentry_assigned(e);
  }
  EXPORT_SYMBOL_GPL(snp_lookup_rmpentry);
+
+/*

arch/x86/kernel/sev.c int snp_lookup_rmpentry(u64 pfn, int *level) · line 2537

Quoted 10 lines
arch/x86/kernel/sev.c +10 -0
+ * psmash is used to smash a 2MB aligned page into 4K
+ * pages while preserving the Validated bit in the RMP.
+ */
+int psmash(u64 pfn)
+{
+       unsigned long paddr = pfn << PAGE_SHIFT;
+       int ret;
+
+       if (!pfn_valid(pfn))
+               return -EINVAL;

We (and many other clouds) use a neat trick to reduce the number of struct pages Linux allocates for guest memory: In its simplest form, add mem= to the kernel cmdline and mmap() /dev/mem to access the reserved memory instead.

This means that the system covers more RAM than Linux contains, which means pfn_valid() is no longer a good indication whether a page is indeed valid. KVM handles this case fine, but this code does not.

Is there any particular reason why we need this check (and similar ones below and in other RMP related patches) in the first place? I would expect that PSMASH and friends return failure codes for invalid pfns.

Alex

Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

maintainer
#78 lore raw

Re: [PATCH RFC v7 16/64] x86/sev: Add helper functions for RMPUPDATE and PSMASH instruction

Quoted 124 lines · Alexander Graf
Quoted 108 lines · Michael Roth

From: Brijesh Singh <[email protected]>

The RMPUPDATE instruction writes a new RMP entry in the RMP Table. The hypervisor will use the instruction to add pages to the RMP table. See APM3 for details on the instruction operations.

The PSMASH instruction expands a 2MB RMP entry into a corresponding set of contiguous 4KB-Page RMP entries. The hypervisor will use this instruction to adjust the RMP entry without invalidating the previous RMP entry.

Add the following external interface API functions:

int psmash(u64 pfn);
psmash is used to smash a 2MB aligned page into 4K pages while preserving the Validated bit in the RMP.

int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int asid, bool immutable); Used to assign a page to guest using the RMPUPDATE instruction.

int rmp_make_shared(u64 pfn, enum pg_level level);
Used to transition a page to hypervisor/shared state using the RMPUPDATE instruction.

[mdr: add RMPUPDATE retry logic for transient FAIL_OVERLAP errors]

---
  arch/x86/include/asm/sev.h | 24 ++++++++++ arch/x86/kernel/sev.c      | 95 ++++++++++++++++++++++++++++++++++++++

  2 files changed, 119 insertions(+)
arch/x86/include/asm/sev.h +5 -0
index 8d3ce2ad27da..4eeedcaca593 100644
@@ -80,10 +80,15 @@ extern bool handle_vc_boot_ghcb(struct pt_regs 
 *regs);
 
   /* Software defined (when rFlags.CF = 1) */
   #define PVALIDATE_FAIL_NOUPDATE                255
+/* RMUPDATE detected 4K page and 2MB page overlap. */
+#define RMPUPDATE_FAIL_OVERLAP         7
 
   /* RMP page size */
   #define RMP_PG_SIZE_4K                 0
+#define RMP_PG_SIZE_2M                 1
   #define RMP_TO_X86_PG_LEVEL(level)     (((level) == RMP_PG_SIZE_4K) ? PG_LEVEL_4K : PG_LEVEL_2M)
+#define X86_TO_RMP_PG_LEVEL(level)     (((level) == PG_LEVEL_4K) ? 
 RMP_PG_SIZE_4K : RMP_PG_SIZE_2M)
+
   #define RMPADJUST_VMSA_PAGE_BIT                BIT(16)

  /* SNP Guest message request */

arch/x86/include/asm/sev.h +12 -0
@@ -133,6 +138,15 @@ struct snp_secrets_page_layout {
          u8 rsvd3[3840];
   } __packed;
 
+struct rmp_state {
+       u64 gpa;
+       u8 assigned;
+       u8 pagesize;
+       u8 immutable;
+       u8 rsvd;
+       u32 asid;
+} __packed;
+
   #ifdef CONFIG_AMD_MEM_ENCRYPT
   extern struct static_key_false sev_es_enable_key;
   extern void __sev_es_ist_enter(struct pt_regs *regs);
@@ -198,6 +212,9 @@ bool snp_init(struct boot_params *bp);
   void __init __noreturn snp_abort(void);
   int snp_issue_guest_request(u64 exit_code, struct snp_req_data *input, unsigned long *fw_err);
   int snp_lookup_rmpentry(u64 pfn, int *level);
+int psmash(u64 pfn);
+int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int asid, 
 bool immutable);
+int rmp_make_shared(u64 pfn, enum pg_level level);
   #else
   static inline void sev_es_ist_enter(struct pt_regs *regs) { }

  static inline void sev_es_ist_exit(void) { }

arch/x86/include/asm/sev.h +0 -0
@@ -223,6 +240,13 @@ static inline int snp_issue_guest_request(u64 

exit_code, struct snp_req_data *in
         return -ENOTTY;
  }
  static inline int snp_lookup_rmpentry(u64 pfn, int *level) { return 0; }

arch/x86/include/asm/sev.h +2 -0
+static inline int psmash(u64 pfn) { return -ENXIO; }
+static inline int rmp_make_private(u64 pfn, u64 gpa, enum pg_level 

level, int asid,
+                                  bool immutable)

arch/x86/include/asm/sev.h +4 -0
+{
+       return -ENODEV;
+}
+static inline int rmp_make_shared(u64 pfn, enum pg_level level) { 

return -ENODEV; }
  #endif

  #endif

arch/x86/kernel/sev.c +2 -0
index 706675561f49..67035d34adad 100644
@@ -2523,3 +2523,98 @@ int snp_lookup_rmpentry(u64 pfn, int *level)
          return !!rmpentry_assigned(e);
   }
   EXPORT_SYMBOL_GPL(snp_lookup_rmpentry);
+
+/*

arch/x86/kernel/sev.c int snp_lookup_rmpentry(u64 pfn, int *level) · line 2537

Quoted 10 lines
arch/x86/kernel/sev.c +10 -0
+ * psmash is used to smash a 2MB aligned page into 4K
+ * pages while preserving the Validated bit in the RMP.
+ */
+int psmash(u64 pfn)
+{
+       unsigned long paddr = pfn << PAGE_SHIFT;
+       int ret;
+
+       if (!pfn_valid(pfn))
+               return -EINVAL;

We (and many other clouds) use a neat trick to reduce the number of struct pages Linux allocates for guest memory: In its simplest form, add mem= to the kernel cmdline and mmap() /dev/mem to access the reserved memory instead.

Quoted 1 line

This means that the system covers more RAM than Linux contains, which means pfn_valid() is no longer a good indication whether a page is indeed valid. KVM handles this case fine, but this code does not.

Hmm...but then is also using max_pfn reliable ?

Quoted 3 lines

Is there any particular reason why we need this check (and similar ones below and in other RMP related patches) in the first place. I would > expect that PSMASH and friends return failure codes for invalid pfns.

Yes, PSMASH does out of bounds check on the input SPA and additionally checks if SPA is 2M aligned, so guess we can rely on using PSMASH failing on invalid pfns.

Thanks,
Ashish

#79 lore raw

Re: [PATCH RFC v7 16/64] x86/sev: Add helper functions for RMPUPDATE and PSMASH instruction

Quoted 135 lines · Ashish
Quoted 133 lines · Alexander Graf
Quoted 114 lines · Michael Roth

From: Brijesh Singh <[email protected]>

The RMPUPDATE instruction writes a new RMP entry in the RMP Table. The hypervisor will use the instruction to add pages to the RMP table. See APM3 for details on the instruction operations.

The PSMASH instruction expands a 2MB RMP entry into a corresponding set of contiguous 4KB-Page RMP entries. The hypervisor will use this instruction to adjust the RMP entry without invalidating the previous RMP entry.

Add the following external interface API functions:

int psmash(u64 pfn);
psmash is used to smash a 2MB aligned page into 4K pages while preserving the Validated bit in the RMP.

int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int asid, bool immutable);
Used to assign a page to guest using the RMPUPDATE instruction.

int rmp_make_shared(u64 pfn, enum pg_level level); Used to transition a page to hypervisor/shared state using the RMPUPDATE instruction.

[mdr: add RMPUPDATE retry logic for transient FAIL_OVERLAP errors]

---
  arch/x86/include/asm/sev.h | 24 ++++++++++
  arch/x86/kernel/sev.c      | 95
++++++++++++++++++++++++++++++++++++++

  2 files changed, 119 insertions(+)
arch/x86/include/asm/sev.h +5 -0
index 8d3ce2ad27da..4eeedcaca593 100644
@@ -80,10 +80,15 @@ extern bool handle_vc_boot_ghcb(struct pt_regs
 *regs);
 
   /* Software defined (when rFlags.CF = 1) */
   #define PVALIDATE_FAIL_NOUPDATE                255
+/* RMUPDATE detected 4K page and 2MB page overlap. */
+#define RMPUPDATE_FAIL_OVERLAP         7
 
   /* RMP page size */
   #define RMP_PG_SIZE_4K                 0
+#define RMP_PG_SIZE_2M                 1
   #define RMP_TO_X86_PG_LEVEL(level)     (((level) == RMP_PG_SIZE_4K)
 ? PG_LEVEL_4K : PG_LEVEL_2M)
+#define X86_TO_RMP_PG_LEVEL(level)     (((level) == PG_LEVEL_4K) ?
 RMP_PG_SIZE_4K : RMP_PG_SIZE_2M)
+

  #define RMPADJUST_VMSA_PAGE_BIT                BIT(16)

  /* SNP Guest message request */

arch/x86/include/asm/sev.h +9 -0
@@ -133,6 +138,15 @@ struct snp_secrets_page_layout {
          u8 rsvd3[3840];
   } __packed;
 
+struct rmp_state {
+       u64 gpa;
+       u8 assigned;
+       u8 pagesize;
+       u8 immutable;
+       u8 rsvd;
+       u32 asid;
+} __packed;
+
   #ifdef CONFIG_AMD_MEM_ENCRYPT
   extern struct static_key_false sev_es_enable_key;
   extern void __sev_es_ist_enter(struct pt_regs *regs);
@@ -198,6 +212,9 @@ bool snp_init(struct boot_params *bp);
   void __init __noreturn snp_abort(void);

  int snp_issue_guest_request(u64 exit_code, struct snp_req_data *input, unsigned long *fw_err);
  int snp_lookup_rmpentry(u64 pfn, int *level);

arch/x86/include/asm/sev.h +2 -0
+int psmash(u64 pfn);
+int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int asid,

bool immutable);
+int rmp_make_shared(u64 pfn, enum pg_level level); #else
  static inline void sev_es_ist_enter(struct pt_regs *regs) { } static inline void sev_es_ist_exit(void) { }

arch/x86/include/asm/sev.h +0 -0
@@ -223,6 +240,13 @@ static inline int snp_issue_guest_request(u64

exit_code, struct snp_req_data *in
         return -ENOTTY;
  }
  static inline int snp_lookup_rmpentry(u64 pfn, int *level) { return 0; }

arch/x86/include/asm/sev.h +2 -0
+static inline int psmash(u64 pfn) { return -ENXIO; }
+static inline int rmp_make_private(u64 pfn, u64 gpa, enum pg_level

level, int asid,
+                                  bool immutable)

arch/x86/include/asm/sev.h +4 -0
+{
+       return -ENODEV;
+}
+static inline int rmp_make_shared(u64 pfn, enum pg_level level) {

return -ENODEV; }
  #endif

  #endif

arch/x86/kernel/sev.c +2 -0
index 706675561f49..67035d34adad 100644
@@ -2523,3 +2523,98 @@ int snp_lookup_rmpentry(u64 pfn, int *level)
          return !!rmpentry_assigned(e);
   }
   EXPORT_SYMBOL_GPL(snp_lookup_rmpentry);
+
+/*

arch/x86/kernel/sev.c int snp_lookup_rmpentry(u64 pfn, int *level) · line 2537

Quoted 10 lines
arch/x86/kernel/sev.c +10 -0
+ * psmash is used to smash a 2MB aligned page into 4K
+ * pages while preserving the Validated bit in the RMP.
+ */
+int psmash(u64 pfn)
+{
+       unsigned long paddr = pfn << PAGE_SHIFT;
+       int ret;
+
+       if (!pfn_valid(pfn))
+               return -EINVAL;

We (and many other clouds) use a neat trick to reduce the number of struct pages Linux allocates for guest memory: In its simplest form, add mem= to the kernel cmdline and mmap() /dev/mem to access the reserved memory instead.

Quoted 5 lines
Quoted 3 lines

This means that the system covers more RAM than Linux contains, which means pfn_valid() is no longer a good indication whether a page is indeed valid. KVM handles this case fine, but this code does not.

Hmm...but then is also using max_pfn reliable ?

I would expect it to not be reliable as it only looks at E820_TYPE_RAM, yes. Do you rely on max_pfn anywhere?

Quoted 9 lines
Quoted 5 lines

Is there any particular reason why we need this check (and similar ones
below and in other RMP related patches) in the first place. I would > expect that PSMASH and friends return failure codes for invalid pfns.

Yes, PSMASH does out of bounds check on the input SPA and additionally checks if SPA is 2M aligned, so guess we can rely on using PSMASH failing on invalid pfns.

Perfect, please remove all the superfluous checks then. If you want to make our life easy, I'd recommend you try to try the patch set with mem= passed on the host and tell QEMU to mmap() /dev/mem for guest RAM. That way you should be able to find any other pitfalls :)

Alex

Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

maintainer
#80 lore raw

Re: [PATCH RFC v7 16/64] x86/sev: Add helper functions for RMPUPDATE and PSMASH instruction

Quoted 137 lines · Alexander Graf
Quoted 135 lines · Ashish
Quoted 133 lines · Alexander Graf
Quoted 114 lines · Michael Roth

From: Brijesh Singh <[email protected]>

The RMPUPDATE instruction writes a new RMP entry in the RMP Table. The hypervisor will use the instruction to add pages to the RMP table. See APM3 for details on the instruction operations.

The PSMASH instruction expands a 2MB RMP entry into a corresponding set of contiguous 4KB-Page RMP entries. The hypervisor will use this instruction to adjust the RMP entry without invalidating the previous RMP entry.

Add the following external interface API functions:

int psmash(u64 pfn);
psmash is used to smash a 2MB aligned page into 4K pages while preserving the Validated bit in the RMP.

int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int asid, bool immutable);
Used to assign a page to guest using the RMPUPDATE instruction.

int rmp_make_shared(u64 pfn, enum pg_level level); Used to transition a page to hypervisor/shared state using the RMPUPDATE instruction.

[mdr: add RMPUPDATE retry logic for transient FAIL_OVERLAP errors]

---
  arch/x86/include/asm/sev.h | 24 ++++++++++
  arch/x86/kernel/sev.c      | 95
++++++++++++++++++++++++++++++++++++++

  2 files changed, 119 insertions(+)
arch/x86/include/asm/sev.h +5 -0
index 8d3ce2ad27da..4eeedcaca593 100644
@@ -80,10 +80,15 @@ extern bool handle_vc_boot_ghcb(struct pt_regs
 *regs);
 
   /* Software defined (when rFlags.CF = 1) */
   #define PVALIDATE_FAIL_NOUPDATE                255
+/* RMUPDATE detected 4K page and 2MB page overlap. */
+#define RMPUPDATE_FAIL_OVERLAP         7
 
   /* RMP page size */
   #define RMP_PG_SIZE_4K                 0
+#define RMP_PG_SIZE_2M                 1
   #define RMP_TO_X86_PG_LEVEL(level)     (((level) == RMP_PG_SIZE_4K)
 ? PG_LEVEL_4K : PG_LEVEL_2M)
+#define X86_TO_RMP_PG_LEVEL(level)     (((level) == PG_LEVEL_4K) ?
 RMP_PG_SIZE_4K : RMP_PG_SIZE_2M)
+

  #define RMPADJUST_VMSA_PAGE_BIT                BIT(16)

  /* SNP Guest message request */

arch/x86/include/asm/sev.h +9 -0
@@ -133,6 +138,15 @@ struct snp_secrets_page_layout {
          u8 rsvd3[3840];
   } __packed;
 
+struct rmp_state {
+       u64 gpa;
+       u8 assigned;
+       u8 pagesize;
+       u8 immutable;
+       u8 rsvd;
+       u32 asid;
+} __packed;
+
   #ifdef CONFIG_AMD_MEM_ENCRYPT
   extern struct static_key_false sev_es_enable_key;
   extern void __sev_es_ist_enter(struct pt_regs *regs);
@@ -198,6 +212,9 @@ bool snp_init(struct boot_params *bp);
   void __init __noreturn snp_abort(void);

  int snp_issue_guest_request(u64 exit_code, struct snp_req_data *input, unsigned long *fw_err);
  int snp_lookup_rmpentry(u64 pfn, int *level);

arch/x86/include/asm/sev.h +2 -0
+int psmash(u64 pfn);
+int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int asid,

bool immutable);
+int rmp_make_shared(u64 pfn, enum pg_level level); #else
  static inline void sev_es_ist_enter(struct pt_regs *regs) { } static inline void sev_es_ist_exit(void) { }

arch/x86/include/asm/sev.h +0 -0
@@ -223,6 +240,13 @@ static inline int snp_issue_guest_request(u64

exit_code, struct snp_req_data *in
         return -ENOTTY;
  }
  static inline int snp_lookup_rmpentry(u64 pfn, int *level) { return 0; }

arch/x86/include/asm/sev.h +2 -0
+static inline int psmash(u64 pfn) { return -ENXIO; }
+static inline int rmp_make_private(u64 pfn, u64 gpa, enum pg_level

level, int asid,
+                                  bool immutable)

arch/x86/include/asm/sev.h +4 -0
+{
+       return -ENODEV;
+}
+static inline int rmp_make_shared(u64 pfn, enum pg_level level) {

return -ENODEV; }
  #endif

  #endif

arch/x86/kernel/sev.c +2 -0
index 706675561f49..67035d34adad 100644
@@ -2523,3 +2523,98 @@ int snp_lookup_rmpentry(u64 pfn, int *level)
          return !!rmpentry_assigned(e);
   }
   EXPORT_SYMBOL_GPL(snp_lookup_rmpentry);
+
+/*

arch/x86/kernel/sev.c int snp_lookup_rmpentry(u64 pfn, int *level) · line 2537

Quoted 10 lines
arch/x86/kernel/sev.c +10 -0
+ * psmash is used to smash a 2MB aligned page into 4K
+ * pages while preserving the Validated bit in the RMP.
+ */
+int psmash(u64 pfn)
+{
+       unsigned long paddr = pfn << PAGE_SHIFT;
+       int ret;
+
+       if (!pfn_valid(pfn))
+               return -EINVAL;

We (and many other clouds) use a neat trick to reduce the number of struct pages Linux allocates for guest memory: In its simplest form, add mem= to the kernel cmdline and mmap() /dev/mem to access the reserved memory instead.

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

This means that the system covers more RAM than Linux contains, which means pfn_valid() is no longer a good indication whether a page is indeed valid. KVM handles this case fine, but this code does not.

Hmm...but then is also using max_pfn reliable ?

I would expect it to not be reliable as it only looks at E820_TYPE_RAM, yes. Do you rely on max_pfn anywhere?

We use it to check if the RMP table is covering the whole system RAM, to get the max. addressable PFN, which should be fine.

Thanks,
Ashish

#81 lore raw

Re: [PATCH RFC v7 16/64] x86/sev: Add helper functions for RMPUPDATE and PSMASH instruction

In reply to Michael Roth
Quoted 32 lines · Michael Roth

From: Brijesh Singh <[email protected]>

The RMPUPDATE instruction writes a new RMP entry in the RMP Table. The hypervisor will use the instruction to add pages to the RMP table. See APM3 for details on the instruction operations.

The PSMASH instruction expands a 2MB RMP entry into a corresponding set of contiguous 4KB-Page RMP entries. The hypervisor will use this instruction to adjust the RMP entry without invalidating the previous RMP entry.

Add the following external interface API functions:

int psmash(u64 pfn);
psmash is used to smash a 2MB aligned page into 4K pages while preserving the Validated bit in the RMP.

int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int asid, bool immutable); Used to assign a page to guest using the RMPUPDATE instruction.

int rmp_make_shared(u64 pfn, enum pg_level level);
Used to transition a page to hypervisor/shared state using the RMPUPDATE instruction.

[mdr: add RMPUPDATE retry logic for transient FAIL_OVERLAP errors]
---
  arch/x86/include/asm/sev.h | 24 ++++++++++
  arch/x86/kernel/sev.c      | 95 ++++++++++++++++++++++++++++++++++++++
  2 files changed, 119 insertions(+)

...

Quoted 11 lines
arch/x86/kernel/sev.c +11 -0
+
+static int rmpupdate(u64 pfn, struct rmp_state *val)
+{
+	unsigned long paddr = pfn << PAGE_SHIFT;
+	int retries = 0;
+	int ret;
+
+	if (!cpu_feature_enabled(X86_FEATURE_SEV_SNP))
+		return -ENXIO;
+
+retry:

arch/x86/kernel/sev.c int snp_lookup_rmpentry(u64 pfn, int *level) · line 2571

Quoted 10 lines
arch/x86/kernel/sev.c +10 -0
+	/* Binutils version 2.36 supports the RMPUPDATE mnemonic. */
+	asm volatile(".byte 0xF2, 0x0F, 0x01, 0xFE"
+		     : "=a"(ret)
+		     : "a"(paddr), "c"((unsigned long)val)
+		     : "memory", "cc");
+
+	if (ret) {
+		if (!retries) {
+			pr_err("RMPUPDATE failed, ret: %d, pfn: %llx, npages: %d, level: %d, retrying (max: %d)...\n",
+			       ret, pfn, npages, level, 2 * num_present_cpus());

This patch isn't bisectable - 'npages' isn't defined in this patch - it's defined later in Patch18

otherwise LGTM

Regards,
Liam

Quoted 11 lines
arch/x86/kernel/sev.c +11 -0
+			dump_stack();
+		}
+		retries++;
+		if (retries < 2 * num_present_cpus())
+			goto retry;
+	} else if (retries > 0) {
+		pr_err("RMPUPDATE for pfn %llx succeeded after %d retries\n", pfn, retries);
+	}
+
+	return ret;
+}
#82 lore raw

RFC PATCH v7 17/64 x86/mm/pat: Introduce set_memory_p

In reply to Michael Roth

From: Ashish Kalra <[email protected]>

set_memory_p() provides a way to change atributes of a memory range to be marked as present.

---
 arch/x86/include/asm/set_memory.h |  3 ++-
 arch/x86/mm/pat/set_memory.c      | 12 ++++++------
 2 files changed, 8 insertions(+), 7 deletions(-)
arch/x86/include/asm/set_memory.h +2 -1
index b45c4d27fd46..56be492eb2d1 100644
@@ -12,7 +12,7 @@
  * Cacheability  : UnCached, WriteCombining, WriteThrough, WriteBack
  * Executability : eXecutable, NoteXecutable
  * Read/Write    : ReadOnly, ReadWrite
- * Presence      : NotPresent
+ * Presence      : NotPresent, Present
  * Encryption    : Encrypted, Decrypted
  *
  * Within a category, the attributes are mutually exclusive.
@@ -44,6 +44,7 @@ int set_memory_uc(unsigned long addr, int numpages);
 int set_memory_wc(unsigned long addr, int numpages);
 int set_memory_wb(unsigned long addr, int numpages);
 int set_memory_np(unsigned long addr, int numpages);
+int set_memory_p(unsigned long addr, int numpages);
 int set_memory_4k(unsigned long addr, int numpages);
 int set_memory_encrypted(unsigned long addr, int numpages);
 int set_memory_decrypted(unsigned long addr, int numpages);
arch/x86/mm/pat/set_memory.c +6 -6
index 2e5a045731de..b1f79062c4a5 100644
@@ -1993,17 +1993,12 @@ int set_mce_nospec(unsigned long pfn)
 	return rc;
 }
 
-static int set_memory_p(unsigned long *addr, int numpages)
-{
-	return change_page_attr_set(addr, numpages, __pgprot(_PAGE_PRESENT), 0);
-}
-
 /* Restore full speculative operation to the pfn. */
 int clear_mce_nospec(unsigned long pfn)
 {
 	unsigned long addr = (unsigned long) pfn_to_kaddr(pfn);
 
-	return set_memory_p(&addr, 1);
+	return set_memory_p(addr, 1);
 }
 EXPORT_SYMBOL_GPL(clear_mce_nospec);
 #endif /* CONFIG_X86_64 */
@@ -2039,6 +2034,11 @@ int set_memory_np(unsigned long addr, int numpages)
 	return change_page_attr_clear(&addr, numpages, __pgprot(_PAGE_PRESENT), 0);
 }
 
+int set_memory_p(unsigned long addr, int numpages)
+{
+	return change_page_attr_set(&addr, numpages, __pgprot(_PAGE_PRESENT), 0);
+}
+
 int set_memory_np_noalias(unsigned long addr, int numpages)
 {
 	int cpa_flags = CPA_NO_CHECK_ALIAS;
Signature · 1 line
2.25.1
#83 lore raw

RFC PATCH v7 18/64 x86/sev: Invalidate pages from the direct map when adding them to the RMP table

In reply to Michael Roth

From: Brijesh Singh <[email protected]>

The integrity guarantee of SEV-SNP is enforced through the RMP table. The RMP is used with standard x86 and IOMMU page tables to enforce memory restrictions and page access rights. The RMP check is enforced as soon as SEV-SNP is enabled globally in the system. When hardware encounters an RMP-check failure, it raises a page-fault exception.

The rmp_make_private() and rmp_make_shared() helpers are used to add or remove the pages from the RMP table. Improve the rmp_make_private() to invalidate state so that pages cannot be used in the direct-map after they are added the RMP table, and restored to their default valid permission after the pages are removed from the RMP table.

---
 arch/x86/kernel/sev.c | 38 +++++++++++++++++++++++++++++++++++++-
 1 file changed, 37 insertions(+), 1 deletion(-)
arch/x86/kernel/sev.c +37 -1
index 67035d34adad..e2b38c3551be 100644
@@ -2549,15 +2549,40 @@ int psmash(u64 pfn)
 }
 EXPORT_SYMBOL_GPL(psmash);
 
+static int restore_direct_map(u64 pfn, int npages)
+{
+	return set_memory_p((unsigned long)pfn_to_kaddr(pfn), npages);
+}
+
+static int invalidate_direct_map(unsigned long pfn, int npages)
+{
+	return set_memory_np((unsigned long)pfn_to_kaddr(pfn), npages);
+}
+
 static int rmpupdate(u64 pfn, struct rmp_state *val)
 {
 	unsigned long paddr = pfn << PAGE_SHIFT;
+	int ret, level, npages;
 	int retries = 0;
-	int ret;
 
 	if (!cpu_feature_enabled(X86_FEATURE_SEV_SNP))
 		return -ENXIO;
 
+	level = RMP_TO_X86_PG_LEVEL(val->pagesize);
+	npages = page_level_size(level) / PAGE_SIZE;
+
+	/*
+	 * If page is getting assigned in the RMP table then unmap it from the
+	 * direct map.
+	 */
+	if (val->assigned) {
+		if (invalidate_direct_map(pfn, npages)) {
+			pr_err("Failed to unmap %d pages at pfn 0x%llx from the direct_map\n",
+			       npages, pfn);
+			return -EFAULT;
+		}
+	}
+
 retry:
 	/* Binutils version 2.36 supports the RMPUPDATE mnemonic. */
 	asm volatile(".byte 0xF2, 0x0F, 0x01, 0xFE"
@@ -2578,6 +2603,17 @@ static int rmpupdate(u64 pfn, struct rmp_state *val)
 		pr_err("RMPUPDATE for pfn %llx succeeded after %d retries\n", pfn, retries);
 	}
 
+	/*
+	 * Restore the direct map after the page is removed from the RMP table.
+	 */
+	if (!ret && !val->assigned) {
+		if (restore_direct_map(pfn, npages)) {
+			pr_err("Failed to map %d pages at pfn 0x%llx into the direct_map\n",
+			       npages, pfn);
+			return -EFAULT;
+		}
+	}
+
 	return ret;
 }
 
Signature · 1 line
2.25.1
#84 lore raw

RFC PATCH v7 19/64 x86/traps: Define RMP violation #PF error code

In reply to Michael Roth

From: Brijesh Singh <[email protected]>

Bit 31 in the page fault-error bit will be set when processor encounters an RMP violation.

While at it, use the BIT_ULL() macro.

Signed-off by: Ashish Kalra <[email protected]>

---
 arch/x86/include/asm/trap_pf.h | 18 +++++++++++-------
 arch/x86/mm/fault.c            |  1 +
 2 files changed, 12 insertions(+), 7 deletions(-)
arch/x86/include/asm/trap_pf.h +11 -7
index 10b1de500ab1..295be06f8db7 100644
@@ -2,6 +2,8 @@
 #ifndef _ASM_X86_TRAP_PF_H
 #define _ASM_X86_TRAP_PF_H
 
+#include <linux/bits.h>  /* BIT() macro */
+
 /*
  * Page fault error code bits:
  *
@@ -12,15 +14,17 @@
  *   bit 4 ==				1: fault was an instruction fetch
  *   bit 5 ==				1: protection keys block access
  *   bit 15 ==				1: SGX MMU page-fault
+ *   bit 31 ==				1: fault was due to RMP violation
  */
 enum x86_pf_error_code {
-	X86_PF_PROT	=		1 << 0,
-	X86_PF_WRITE	=		1 << 1,
-	X86_PF_USER	=		1 << 2,
-	X86_PF_RSVD	=		1 << 3,
-	X86_PF_INSTR	=		1 << 4,
-	X86_PF_PK	=		1 << 5,
-	X86_PF_SGX	=		1 << 15,
+	X86_PF_PROT	=		BIT(0),
+	X86_PF_WRITE	=		BIT(1),
+	X86_PF_USER	=		BIT(2),
+	X86_PF_RSVD	=		BIT(3),
+	X86_PF_INSTR	=		BIT(4),
+	X86_PF_PK	=		BIT(5),
+	X86_PF_SGX	=		BIT(15),
+	X86_PF_RMP	=		BIT(31),
 };
 
 #endif /* _ASM_X86_TRAP_PF_H */
arch/x86/mm/fault.c +1 -0
index 7b0d4ab894c8..f8193b99e9c8 100644
@@ -567,6 +567,7 @@ show_fault_oops(struct pt_regs *regs, unsigned long error_code, unsigned long ad
 		 !(error_code & X86_PF_PROT) ? "not-present page" :
 		 (error_code & X86_PF_RSVD)  ? "reserved bit violation" :
 		 (error_code & X86_PF_PK)    ? "protection keys violation" :
+		 (error_code & X86_PF_RMP)   ? "RMP violation" :
 					       "permissions violation");
 
 	if (!(error_code & X86_PF_USER) && user_mode(regs)) {
Signature · 1 line
2.25.1
#85 lore raw

RFC PATCH v7 20/64 x86/fault: Add support to handle the RMP fault for user address

In reply to Michael Roth

From: Brijesh Singh <[email protected]>

When SEV-SNP is enabled globally, a write from the host goes through the RMP check. When the host writes to pages, hardware checks the following conditions at the end of page walk:

1. Assigned bit in the RMP table is zero (i.e page is shared).
2. If the page table entry that gives the sPA indicates that the target page size is a large page, then all RMP entries for the 4KB constituting pages of the target must have the assigned bit 0.
3. Immutable bit in the RMP table is not zero.

The hardware will raise page fault if one of the above conditions is not met. Try resolving the fault instead of taking fault again and again. If the host attempts to write to the guest private memory then send the SIGBUS signal to kill the process. If the page level between the host and RMP entry does not match, then split the address to keep the RMP and host page levels in sync.

---
 arch/x86/mm/fault.c      | 97 ++++++++++++++++++++++++++++++++++++++++
 include/linux/mm.h       |  3 +-
 include/linux/mm_types.h |  3 ++
 mm/memory.c              | 10 +++++
 4 files changed, 112 insertions(+), 1 deletion(-)
arch/x86/mm/fault.c +97 -0
index f8193b99e9c8..d611051dcf1e 100644
@@ -33,6 +33,7 @@
 #include <asm/kvm_para.h>		/* kvm_handle_async_pf		*/
 #include <asm/vdso.h>			/* fixup_vdso_exception()	*/
 #include <asm/irq_stack.h>
+#include <asm/sev.h>			/* snp_lookup_rmpentry()	*/
 
 #define CREATE_TRACE_POINTS
 #include <asm/trace/exceptions.h>
@@ -414,6 +415,7 @@ static void dump_pagetable(unsigned long address)
 	pr_cont("PTE %lx", pte_val(*pte));
 out:
 	pr_cont("\n");
+
 	return;
 bad:
 	pr_info("BAD\n");
@@ -1240,6 +1242,90 @@ do_kern_addr_fault(struct pt_regs *regs, unsigned long hw_error_code,
 }
 NOKPROBE_SYMBOL(do_kern_addr_fault);
 
+enum rmp_pf_ret {
+	RMP_PF_SPLIT	= 0,
+	RMP_PF_RETRY	= 1,
+	RMP_PF_UNMAP	= 2,
+};
+
+/*
+ * The goal of RMP faulting routine is really to check whether the
+ * page that faulted should be accessible.  That can be determined
+ * simply by looking at the RMP entry for the 4k address being accessed.
+ * If that entry has Assigned=1 then it's a bad address. It could be
+ * because the 2MB region was assigned as a large page, or it could be
+ * because the region is all 4k pages and that 4k was assigned.
+ * In either case, it's a bad access.
+ * There are basically two main possibilities:
+ * 1. The 2M entry has Assigned=1 and Page_Size=1. Then all 511 middle
+ * entries also have Assigned=1. This entire 2M region is a guest page.
+ * 2. The 2M entry has Assigned=0 and Page_Size=0. Then the 511 middle
+ * entries can be anything, this region consists of individual 4k assignments.
+ */
+static int handle_user_rmp_page_fault(struct pt_regs *regs, unsigned long error_code,
+				      unsigned long address)
+{
+	int rmp_level, level;
+	pgd_t *pgd;
+	pte_t *pte;
+	u64 pfn;
+
+	pgd = __va(read_cr3_pa());
+	pgd += pgd_index(address);
+
+	pte = lookup_address_in_pgd(pgd, address, &level);
+
+	/*
+	 * It can happen if there was a race between an unmap event and
+	 * the RMP fault delivery.
+	 */
+	if (!pte || !pte_present(*pte))
+		return RMP_PF_UNMAP;
+
+	/*
+	 * RMP page fault handler follows this algorithm:
+	 * 1. Compute the pfn for the 4kb page being accessed
+	 * 2. Read that RMP entry -- If it is assigned then kill the process
+	 * 3. Otherwise, check the level from the host page table
+	 *    If level=PG_LEVEL_4K then the page is already smashed
+	 *    so just retry the instruction
+	 * 4. If level=PG_LEVEL_2M/1G, then the host page needs to be split
+	 */
+
+	pfn = pte_pfn(*pte);
+
+	/* If its large page then calculte the fault pfn */
+	if (level > PG_LEVEL_4K)
+		pfn = pfn | PFN_DOWN(address & (page_level_size(level) - 1));
+
+	/*
+	 * If its a guest private page, then the fault cannot be resolved.
+	 * Send a SIGBUS to terminate the process.
+	 *
+	 * As documented in APM vol3 pseudo-code for RMPUPDATE, when the 2M range
+	 * is covered by a valid (Assigned=1) 2M entry, the middle 511 4k entries
+	 * also have Assigned=1. This means that if there is an access to a page
+	 * which happens to lie within an Assigned 2M entry, the 4k RMP entry
+	 * will also have Assigned=1. Therefore, the kernel should see that
+	 * the page is not a valid page and the fault cannot be resolved.
+	 */
+	if (snp_lookup_rmpentry(pfn, &rmp_level)) {
+		pr_info("Fatal RMP page fault, terminating process, entry assigned for pfn 0x%llx\n",
+			pfn);
+		do_sigbus(regs, error_code, address, VM_FAULT_SIGBUS);
+		return RMP_PF_RETRY;
+	}
+
+	/*
+	 * The backing page level is higher than the RMP page level, request
+	 * to split the page.
+	 */
+	if (level > rmp_level)
+		return RMP_PF_SPLIT;
+
+	return RMP_PF_RETRY;
+}
+
 /*
  * Handle faults in the user portion of the address space.  Nothing in here
  * should check X86_PF_USER without a specific justification: for almost
@@ -1337,6 +1423,17 @@ void do_user_addr_fault(struct pt_regs *regs,
 	if (error_code & X86_PF_INSTR)
 		flags |= FAULT_FLAG_INSTRUCTION;
 
+	/*
+	 * If its an RMP violation, try resolving it.
+	 */
+	if (error_code & X86_PF_RMP) {
+		if (handle_user_rmp_page_fault(regs, error_code, address))
+			return;
+
+		/* Ask to split the page */
+		flags |= FAULT_FLAG_PAGE_SPLIT;
+	}
+
 #ifdef CONFIG_X86_64
 	/*
 	 * Faults in the vsyscall page might need emulation.  The
include/linux/mm.h +2 -1
index 3c84f4e48cd7..2fd8e16d149c 100644
@@ -466,7 +466,8 @@ static inline bool fault_flag_allow_retry_first(enum fault_flag flags)
 	{ FAULT_FLAG_USER,		"USER" }, \
 	{ FAULT_FLAG_REMOTE,		"REMOTE" }, \
 	{ FAULT_FLAG_INSTRUCTION,	"INSTRUCTION" }, \
-	{ FAULT_FLAG_INTERRUPTIBLE,	"INTERRUPTIBLE" }
+	{ FAULT_FLAG_INTERRUPTIBLE,	"INTERRUPTIBLE" }, \
+	{ FAULT_FLAG_PAGE_SPLIT,	"PAGESPLIT" }
 
 /*
  * vm_fault is filled by the pagefault handler and passed to the vma's
include/linux/mm_types.h +3 -0
index 500e536796ca..06ba34d51638 100644
@@ -962,6 +962,8 @@ typedef struct {
  *                      mapped R/O.
  * @FAULT_FLAG_ORIG_PTE_VALID: whether the fault has vmf->orig_pte cached.
  *                        We should only access orig_pte if this flag set.
+ * @FAULT_FLAG_PAGE_SPLIT: The fault was due page size mismatch, split the
+ *                         region to smaller page size and retry.
  *
  * About @FAULT_FLAG_ALLOW_RETRY and @FAULT_FLAG_TRIED: we can specify
  * whether we would allow page faults to retry by specifying these two
@@ -999,6 +1001,7 @@ enum fault_flag {
 	FAULT_FLAG_INTERRUPTIBLE =	1 << 9,
 	FAULT_FLAG_UNSHARE =		1 << 10,
 	FAULT_FLAG_ORIG_PTE_VALID =	1 << 11,
+	FAULT_FLAG_PAGE_SPLIT =		1 << 12,
 };
 
 typedef unsigned int __bitwise zap_flags_t;
mm/memory.c +10 -0
index f88c351aecd4..e68da7e403c6 100644
@@ -4996,6 +4996,12 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)
 	return 0;
 }
 
+static int handle_split_page_fault(struct vm_fault *vmf)
+{
+	__split_huge_pmd(vmf->vma, vmf->pmd, vmf->address, false, NULL);
+	return 0;
+}
+
 /*
  * By the time we get here, we already hold the mm semaphore
  *
@@ -5078,6 +5084,10 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
 				pmd_migration_entry_wait(mm, vmf.pmd);
 			return 0;
 		}
+
+		if (flags & FAULT_FLAG_PAGE_SPLIT)
+			return handle_split_page_fault(&vmf);
+
 		if (pmd_trans_huge(vmf.orig_pmd) || pmd_devmap(vmf.orig_pmd)) {
 			if (pmd_protnone(vmf.orig_pmd) && vma_is_accessible(vma))
 				return do_huge_pmd_numa_page(&vmf);
Signature · 1 line
2.25.1
#86 lore raw

Re: [PATCH RFC v7 20/64] x86/fault: Add support to handle the RMP fault for user address

Quoted 6 lines · On Wed, 14 Dec 2022 13:40:12 -0600 Michael Roth <[email protected]> wrote:

From: Brijesh Singh <[email protected]>

When SEV-SNP is enabled globally, a write from the host goes through the RMP check. When the host writes to pages, hardware checks the following conditions at the end of page walk:

Quoted 6 lines

1. Assigned bit in the RMP table is zero (i.e page is shared).
2. If the page table entry that gives the sPA indicates that the target page size is a large page, then all RMP entries for the 4KB constituting pages of the target must have the assigned bit 0.
3. Immutable bit in the RMP table is not zero.

Just being curious. AMD APM table 15-37 "RMP Page Assignment Settings" shows Immuable bit is "don't care" when a page is owned by the hypervisor. The table 15-39 "RMP Memory Access Checks" shows the hardware will do "Hypervisor-owned" check for host data write and page table access. I suppose "Hypervisor-owned" check means HW will check if the RMP entry is configured according to the table 15-37 (Assign bit = 0, ASID = 0, Immutable = X)

None of them mentions that Immutable bit in the related RMP-entry should be 1 for hypervisor-owned page.

I can understand 1) 2). Can you explain more about 3)?

Quoted 212 lines

The hardware will raise page fault if one of the above conditions is not met. Try resolving the fault instead of taking fault again and again. If the host attempts to write to the guest private memory then send the SIGBUS signal to kill the process. If the page level between the host and RMP entry does not match, then split the address to keep the RMP and host page levels in sync.

---
 arch/x86/mm/fault.c      | 97 ++++++++++++++++++++++++++++++++++++++++
 include/linux/mm.h       |  3 +-
 include/linux/mm_types.h |  3 ++
 mm/memory.c              | 10 +++++
 4 files changed, 112 insertions(+), 1 deletion(-)
arch/x86/mm/fault.c +97 -0
index f8193b99e9c8..d611051dcf1e 100644
@@ -33,6 +33,7 @@
 #include <asm/kvm_para.h>		/* kvm_handle_async_pf		*/
 #include <asm/vdso.h>			/* fixup_vdso_exception()	*/
 #include <asm/irq_stack.h>
+#include <asm/sev.h>			/* snp_lookup_rmpentry()	*/
 
 #define CREATE_TRACE_POINTS
 #include <asm/trace/exceptions.h>
@@ -414,6 +415,7 @@ static void dump_pagetable(unsigned long address)
 	pr_cont("PTE %lx", pte_val(*pte));
 out:
 	pr_cont("\n");
+
 	return;
 bad:
 	pr_info("BAD\n");
@@ -1240,6 +1242,90 @@ do_kern_addr_fault(struct pt_regs *regs, unsigned long hw_error_code,
 }
 NOKPROBE_SYMBOL(do_kern_addr_fault);
 
+enum rmp_pf_ret {
+	RMP_PF_SPLIT	= 0,
+	RMP_PF_RETRY	= 1,
+	RMP_PF_UNMAP	= 2,
+};
+
+/*
+ * The goal of RMP faulting routine is really to check whether the
+ * page that faulted should be accessible.  That can be determined
+ * simply by looking at the RMP entry for the 4k address being accessed.
+ * If that entry has Assigned=1 then it's a bad address. It could be
+ * because the 2MB region was assigned as a large page, or it could be
+ * because the region is all 4k pages and that 4k was assigned.
+ * In either case, it's a bad access.
+ * There are basically two main possibilities:
+ * 1. The 2M entry has Assigned=1 and Page_Size=1. Then all 511 middle
+ * entries also have Assigned=1. This entire 2M region is a guest page.
+ * 2. The 2M entry has Assigned=0 and Page_Size=0. Then the 511 middle
+ * entries can be anything, this region consists of individual 4k assignments.
+ */
+static int handle_user_rmp_page_fault(struct pt_regs *regs, unsigned long error_code,
+				      unsigned long address)
+{
+	int rmp_level, level;
+	pgd_t *pgd;
+	pte_t *pte;
+	u64 pfn;
+
+	pgd = __va(read_cr3_pa());
+	pgd += pgd_index(address);
+
+	pte = lookup_address_in_pgd(pgd, address, &level);
+
+	/*
+	 * It can happen if there was a race between an unmap event and
+	 * the RMP fault delivery.
+	 */
+	if (!pte || !pte_present(*pte))
+		return RMP_PF_UNMAP;
+
+	/*
+	 * RMP page fault handler follows this algorithm:
+	 * 1. Compute the pfn for the 4kb page being accessed
+	 * 2. Read that RMP entry -- If it is assigned then kill the process
+	 * 3. Otherwise, check the level from the host page table
+	 *    If level=PG_LEVEL_4K then the page is already smashed
+	 *    so just retry the instruction
+	 * 4. If level=PG_LEVEL_2M/1G, then the host page needs to be split
+	 */
+
+	pfn = pte_pfn(*pte);
+
+	/* If its large page then calculte the fault pfn */
+	if (level > PG_LEVEL_4K)
+		pfn = pfn | PFN_DOWN(address & (page_level_size(level) - 1));
+
+	/*
+	 * If its a guest private page, then the fault cannot be resolved.
+	 * Send a SIGBUS to terminate the process.
+	 *
+	 * As documented in APM vol3 pseudo-code for RMPUPDATE, when the 2M range
+	 * is covered by a valid (Assigned=1) 2M entry, the middle 511 4k entries
+	 * also have Assigned=1. This means that if there is an access to a page
+	 * which happens to lie within an Assigned 2M entry, the 4k RMP entry
+	 * will also have Assigned=1. Therefore, the kernel should see that
+	 * the page is not a valid page and the fault cannot be resolved.
+	 */
+	if (snp_lookup_rmpentry(pfn, &rmp_level)) {
+		pr_info("Fatal RMP page fault, terminating process, entry assigned for pfn 0x%llx\n",
+			pfn);
+		do_sigbus(regs, error_code, address, VM_FAULT_SIGBUS);
+		return RMP_PF_RETRY;
+	}
+
+	/*
+	 * The backing page level is higher than the RMP page level, request
+	 * to split the page.
+	 */
+	if (level > rmp_level)
+		return RMP_PF_SPLIT;
+
+	return RMP_PF_RETRY;
+}
+
 /*
  * Handle faults in the user portion of the address space.  Nothing in here
  * should check X86_PF_USER without a specific justification: for almost
@@ -1337,6 +1423,17 @@ void do_user_addr_fault(struct pt_regs *regs,
 	if (error_code & X86_PF_INSTR)
 		flags |= FAULT_FLAG_INSTRUCTION;
 
+	/*
+	 * If its an RMP violation, try resolving it.
+	 */
+	if (error_code & X86_PF_RMP) {
+		if (handle_user_rmp_page_fault(regs, error_code, address))
+			return;
+
+		/* Ask to split the page */
+		flags |= FAULT_FLAG_PAGE_SPLIT;
+	}
+
 #ifdef CONFIG_X86_64
 	/*
 	 * Faults in the vsyscall page might need emulation.  The
include/linux/mm.h +2 -1
index 3c84f4e48cd7..2fd8e16d149c 100644
@@ -466,7 +466,8 @@ static inline bool fault_flag_allow_retry_first(enum fault_flag flags)
 	{ FAULT_FLAG_USER,		"USER" }, \
 	{ FAULT_FLAG_REMOTE,		"REMOTE" }, \
 	{ FAULT_FLAG_INSTRUCTION,	"INSTRUCTION" }, \
-	{ FAULT_FLAG_INTERRUPTIBLE,	"INTERRUPTIBLE" }
+	{ FAULT_FLAG_INTERRUPTIBLE,	"INTERRUPTIBLE" }, \
+	{ FAULT_FLAG_PAGE_SPLIT,	"PAGESPLIT" }
 
 /*
  * vm_fault is filled by the pagefault handler and passed to the vma's
include/linux/mm_types.h +3 -0
index 500e536796ca..06ba34d51638 100644
@@ -962,6 +962,8 @@ typedef struct {
  *                      mapped R/O.
  * @FAULT_FLAG_ORIG_PTE_VALID: whether the fault has vmf->orig_pte cached.
  *                        We should only access orig_pte if this flag set.
+ * @FAULT_FLAG_PAGE_SPLIT: The fault was due page size mismatch, split the
+ *                         region to smaller page size and retry.
  *
  * About @FAULT_FLAG_ALLOW_RETRY and @FAULT_FLAG_TRIED: we can specify
  * whether we would allow page faults to retry by specifying these two
@@ -999,6 +1001,7 @@ enum fault_flag {
 	FAULT_FLAG_INTERRUPTIBLE =	1 << 9,
 	FAULT_FLAG_UNSHARE =		1 << 10,
 	FAULT_FLAG_ORIG_PTE_VALID =	1 << 11,
+	FAULT_FLAG_PAGE_SPLIT =		1 << 12,
 };
 
 typedef unsigned int __bitwise zap_flags_t;
mm/memory.c +10 -0
index f88c351aecd4..e68da7e403c6 100644
@@ -4996,6 +4996,12 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)
 	return 0;
 }
 
+static int handle_split_page_fault(struct vm_fault *vmf)
+{
+	__split_huge_pmd(vmf->vma, vmf->pmd, vmf->address, false, NULL);
+	return 0;
+}
+
 /*
  * By the time we get here, we already hold the mm semaphore
  *
@@ -5078,6 +5084,10 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
 				pmd_migration_entry_wait(mm, vmf.pmd);
 			return 0;
 		}
+
+		if (flags & FAULT_FLAG_PAGE_SPLIT)
+			return handle_split_page_fault(&vmf);
+
 		if (pmd_trans_huge(vmf.orig_pmd) || pmd_devmap(vmf.orig_pmd)) {
 			if (pmd_protnone(vmf.orig_pmd) && vma_is_accessible(vma))
 				return do_huge_pmd_numa_page(&vmf);
#87 lore raw

RFC PATCH v7 21/64 x86/fault: fix handle_split_page_fault() to work with memfd backed pages

In reply to Michael Roth

From: Hugh Dickins <[email protected]>

When the address is backed by a memfd, the code to split the page does nothing more than remove the PMD from the page tables. So immediately install a PTE to ensure that any other pages in that 2MB region are brought back as in 4K pages.

---
 mm/memory.c | 5 +++++
 1 file changed, 5 insertions(+)
mm/memory.c +5 -0
index e68da7e403c6..33c9020ba1f8 100644
@@ -4999,6 +4999,11 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)
 static int handle_split_page_fault(struct vm_fault *vmf)
 {
 	__split_huge_pmd(vmf->vma, vmf->pmd, vmf->address, false, NULL);
+	/*
+	 * Install a PTE immediately to ensure that any other pages in
+	 * this 2MB region are brought back in as 4K pages.
+	 */
+	__pte_alloc(vmf->vma->vm_mm, vmf->pmd);
 	return 0;
 }
 
Signature · 1 line
2.25.1
#88 lore raw

Re: [PATCH RFC v7 21/64] x86/fault: fix handle_split_page_fault() to work with memfd backed pages

Quoted 11 lines · Michael Roth

From: Hugh Dickins <[email protected]>

When the address is backed by a memfd, the code to split the page does nothing more than remove the PMD from the page tables. So immediately install a PTE to ensure that any other pages in that 2MB region are brought back as in 4K pages.

Hah, it's good to see this again, but it was "Suggested-by" me, not "Signed-off-by" me. And was a neat pragmatic one-liner workaround for the immediate problem we had, but came with caveats.

The problem is that we have one wind blowing in the split direction, and another wind (khugepaged) blowing in the collapse direction, and who wins for how long depends on factors I've not fully got to grips with (and is liable to differ between kernel releases).

Good and bad timing to see it. I was just yesterday reviewing a patch to the collapsing wind, which reminded me of an improvement yet to be made there, thinking I'd like to try it sometime; but recallng that someone somewhere relies on the splitting wind, and doesn't want the collapsing wind to blow any harder - now you remind me who!

Bad timing in that I don't have any quick answer on the right thing to do instead, and can't give it the thought it needs at the moment - perhaps others can chime in more usefully.

Hugh

p.s. I don't know where "handle_split_page_fault" comes in, but "x86/fault" in the subject looks wrong, since this appears to be in generic code; and "memfd" seems inappropriate too, but perhaps you have a situation where only memfds can reach handle_split_page_fault().

Quoted 22 lines
---
 mm/memory.c | 5 +++++
 1 file changed, 5 insertions(+)
mm/memory.c +5 -0
index e68da7e403c6..33c9020ba1f8 100644
@@ -4999,6 +4999,11 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)
 static int handle_split_page_fault(struct vm_fault *vmf)
 {
 	__split_huge_pmd(vmf->vma, vmf->pmd, vmf->address, false, NULL);
+	/*
+	 * Install a PTE immediately to ensure that any other pages in
+	 * this 2MB region are brought back in as 4K pages.
+	 */
+	__pte_alloc(vmf->vma->vm_mm, vmf->pmd);
 	return 0;
 }
 
Signature · 1 line
2.25.1
#89 lore raw

RFC PATCH v7 22/64 x86/fault: Return pfn from dump_pagetable() for SEV-specific fault handling.

In reply to Michael Roth

From: Ashish Kalra <[email protected]>

Return pfn from dump_pagetable() to do SEV-specific fault handling. Used for handling SNP RMP page fault.

---
 arch/x86/mm/fault.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
arch/x86/mm/fault.c +11 -4
index d611051dcf1e..ded53879f98d 100644
@@ -311,7 +311,7 @@ static bool low_pfn(unsigned long pfn)
 	return pfn < max_low_pfn;
 }
 
-static void dump_pagetable(unsigned long address)
+static unsigned long dump_pagetable(unsigned long address)
 {
 	pgd_t *base = __va(read_cr3_pa());
 	pgd_t *pgd = &base[pgd_index(address)];
@@ -345,8 +345,10 @@ static void dump_pagetable(unsigned long address)
 
 	pte = pte_offset_kernel(pmd, address);
 	pr_cont("*pte = %0*Lx ", sizeof(*pte) * 2, (u64)pte_val(*pte));
+	return 0;
 out:
 	pr_cont("\n");
+	return 0;
 }
 
 #else /* CONFIG_X86_64: */
@@ -367,10 +369,11 @@ static int bad_address(void *p)
 	return get_kernel_nofault(dummy, (unsigned long *)p);
 }
 
-static void dump_pagetable(unsigned long address)
+static unsigned long dump_pagetable(unsigned long address)
 {
 	pgd_t *base = __va(read_cr3_pa());
 	pgd_t *pgd = base + pgd_index(address);
+	unsigned long pfn;
 	p4d_t *p4d;
 	pud_t *pud;
 	pmd_t *pmd;
@@ -388,6 +391,7 @@ static void dump_pagetable(unsigned long address)
 	if (bad_address(p4d))
 		goto bad;
 
+	pfn = p4d_pfn(*p4d);
 	pr_cont("P4D %lx ", p4d_val(*p4d));
 	if (!p4d_present(*p4d) || p4d_large(*p4d))
 		goto out;
@@ -396,6 +400,7 @@ static void dump_pagetable(unsigned long address)
 	if (bad_address(pud))
 		goto bad;
 
+	pfn = pud_pfn(*pud);
 	pr_cont("PUD %lx ", pud_val(*pud));
 	if (!pud_present(*pud) || pud_large(*pud))
 		goto out;
@@ -404,6 +409,7 @@ static void dump_pagetable(unsigned long address)
 	if (bad_address(pmd))
 		goto bad;
 
+	pfn = pmd_pfn(*pmd);
 	pr_cont("PMD %lx ", pmd_val(*pmd));
 	if (!pmd_present(*pmd) || pmd_large(*pmd))
 		goto out;
@@ -412,13 +418,14 @@ static void dump_pagetable(unsigned long address)
 	if (bad_address(pte))
 		goto bad;
 
+	pfn = pte_pfn(*pte);
 	pr_cont("PTE %lx", pte_val(*pte));
 out:
 	pr_cont("\n");
-
-	return;
+	return pfn;
 bad:
 	pr_info("BAD\n");
+	return -1;
 }
 
 #endif /* CONFIG_X86_64 */
Signature · 1 line
2.25.1
#90 lore raw

RFC PATCH v7 23/64 x86/fault: Add support to dump RMP entry on fault

In reply to Michael Roth

From: Brijesh Singh <[email protected]>

When SEV-SNP is enabled globally, a write from the host goes through the RMP check. If the hardware encounters the check failure, then it raises the #PF (with RMP set). Dump the RMP entry at the faulting pfn to help the debug.

---
 arch/x86/include/asm/sev.h |  2 ++
 arch/x86/kernel/sev.c      | 43 ++++++++++++++++++++++++++++++++++++++
 arch/x86/mm/fault.c        |  7 ++++++-
 3 files changed, 51 insertions(+), 1 deletion(-)
arch/x86/include/asm/sev.h +2 -0
index 4eeedcaca593..2916f4150ac7 100644
@@ -215,6 +215,7 @@ int snp_lookup_rmpentry(u64 pfn, int *level);
 int psmash(u64 pfn);
 int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int asid, bool immutable);
 int rmp_make_shared(u64 pfn, enum pg_level level);
+void sev_dump_rmpentry(u64 pfn);
 #else
 static inline void sev_es_ist_enter(struct pt_regs *regs) { }
 static inline void sev_es_ist_exit(void) { }
@@ -247,6 +248,7 @@ static inline int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int as
 	return -ENODEV;
 }
 static inline int rmp_make_shared(u64 pfn, enum pg_level level) { return -ENODEV; }
+static inline void sev_dump_rmpentry(u64 pfn) {}
 #endif
 
 #endif
arch/x86/kernel/sev.c +43 -0

49 lines not on this page · raw message

arch/x86/mm/fault.c +6 -1
index ded53879f98d..f2b16dcfbd9a 100644
@@ -536,6 +536,8 @@ static void show_ldttss(const struct desc_ptr *gdt, const char *name, u16 index)
 static void
 show_fault_oops(struct pt_regs *regs, unsigned long error_code, unsigned long address)
 {
+	unsigned long pfn;
+
 	if (!oops_may_print())
 		return;
 
@@ -608,7 +610,10 @@ show_fault_oops(struct pt_regs *regs, unsigned long error_code, unsigned long ad
 		show_ldttss(&gdt, "TR", tr);
 	}
 
-	dump_pagetable(address);
+	pfn = dump_pagetable(address);
+
+	if (error_code & X86_PF_RMP)
+		sev_dump_rmpentry(pfn);
 }
 
 static noinline void
Signature · 1 line
2.25.1
#91 lore raw

Re: [PATCH RFC v7 23/64] x86/fault: Add support to dump RMP entry on fault

Quoted 43 lines · Michael Roth

From: Brijesh Singh <[email protected]>

When SEV-SNP is enabled globally, a write from the host goes through the RMP check. If the hardware encounters the check failure, then it raises the #PF (with RMP set). Dump the RMP entry at the faulting pfn to help the debug.

---
 arch/x86/include/asm/sev.h |  2 ++
 arch/x86/kernel/sev.c      | 43 ++++++++++++++++++++++++++++++++++++++
 arch/x86/mm/fault.c        |  7 ++++++-
 3 files changed, 51 insertions(+), 1 deletion(-)
arch/x86/include/asm/sev.h +2 -0
index 4eeedcaca593..2916f4150ac7 100644
@@ -215,6 +215,7 @@ int snp_lookup_rmpentry(u64 pfn, int *level);
 int psmash(u64 pfn);
 int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int asid, bool immutable);
 int rmp_make_shared(u64 pfn, enum pg_level level);
+void sev_dump_rmpentry(u64 pfn);
 #else
 static inline void sev_es_ist_enter(struct pt_regs *regs) { }
 static inline void sev_es_ist_exit(void) { }
@@ -247,6 +248,7 @@ static inline int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int as
        return -ENODEV;
 }
 static inline int rmp_make_shared(u64 pfn, enum pg_level level) { return -ENODEV; }
+static inline void sev_dump_rmpentry(u64 pfn) {}
 #endif
 
 #endif
arch/x86/kernel/sev.c +0 -0
index e2b38c3551be..1dd1b36bdfea 100644
@@ -2508,6 +2508,49 @@ static struct rmpentry *__snp_lookup_rmpentry(u64 pfn, int *level)
        return entry;

arch/x86/kernel/sev.c static struct rmpentry *__snp_lookup_rmpentry(u64 pfn, int *level) · line 2518

Quoted 10 lines
arch/x86/kernel/sev.c +8 -0
 }
 
+void sev_dump_rmpentry(u64 pfn)
+{
+       unsigned long pfn_end;
+       struct rmpentry *e;
+       int level;
+
+       e = __snp_lookup_rmpentry(pfn, &level);
+       if (!e) {

if (IS_ERR(e)) {

Quoted 16 lines
arch/x86/kernel/sev.c +16 -0
+               pr_info("failed to read RMP entry pfn 0x%llx\n", pfn);
+               return;
+       }
+
+       if (rmpentry_assigned(e)) {
+               pr_info("RMPEntry paddr 0x%llx [assigned=%d immutable=%d pagesize=%d gpa=0x%lx"
+                       " asid=%d vmsa=%d validated=%d]\n", pfn << PAGE_SHIFT,
+                       rmpentry_assigned(e), e->info.immutable, rmpentry_pagesize(e),
+                       (unsigned long)e->info.gpa, e->info.asid, e->info.vmsa,
+                       e->info.validated);
+               return;
+       }
+
+       /*
+        * If the RMP entry at the faulting pfn was not assigned, then not sure
+        * what caused the RMP violation. To get some useful debug information,

arch/x86/kernel/sev.c static struct rmpentry *__snp_lookup_rmpentry(u64 pfn, int *level) · line 2544

Quoted 10 lines
arch/x86/kernel/sev.c +10 -0
+        * iterate through the entire 2MB region, and dump the RMP entries if
+        * one of the bit in the RMP entry is set.
+        */
+       pfn = pfn & ~(PTRS_PER_PMD - 1);
+       pfn_end = pfn + PTRS_PER_PMD;
+
+       while (pfn < pfn_end) {
+               e = __snp_lookup_rmpentry(pfn, &level);
+               if (!e)
+                       return;

if (IS_ERR(e))

      continue;
Quoted 40 lines
arch/x86/kernel/sev.c +9 -0
+
+               if (e->low || e->high)
+                       pr_info("RMPEntry paddr 0x%llx: [high=0x%016llx low=0x%016llx]\n",
+                               pfn << PAGE_SHIFT, e->high, e->low);
+               pfn++;
+       }
+}
+EXPORT_SYMBOL_GPL(sev_dump_rmpentry);
+
 /*
  * Return 1 if the RMP entry is assigned, 0 if it exists but is not assigned,
  * and -errno if there is no corresponding RMP entry.
arch/x86/mm/fault.c +6 -2
index ded53879f98d..f2b16dcfbd9a 100644
@@ -536,6 +536,8 @@ static void show_ldttss(const struct desc_ptr *gdt, const char *name, u16 index)
 static void
 show_fault_oops(struct pt_regs *regs, unsigned long error_code, unsigned long address)
 {
+       unsigned long pfn;
+
        if (!oops_may_print())
                return;
 
@@ -608,7 +610,10 @@ show_fault_oops(struct pt_regs *regs, unsigned long error_code, unsigned long ad
                show_ldttss(&gdt, "TR", tr);
        }
 
-       dump_pagetable(address);
+       pfn = dump_pagetable(address);
+
+       if (error_code & X86_PF_RMP)
+               sev_dump_rmpentry(pfn);
 }
 
 static noinline void
--

2.25.1

maintainer
#92 lore raw

Re: [PATCH RFC v7 23/64] x86/fault: Add support to dump RMP entry on fault

Quoted 46 lines · Alper Gun
Quoted 45 lines · Michael Roth

From: Brijesh Singh <[email protected]>

When SEV-SNP is enabled globally, a write from the host goes through the RMP check. If the hardware encounters the check failure, then it raises the #PF (with RMP set). Dump the RMP entry at the faulting pfn to help the debug.

---
  arch/x86/include/asm/sev.h |  2 ++
  arch/x86/kernel/sev.c      | 43 ++++++++++++++++++++++++++++++++++++++
  arch/x86/mm/fault.c        |  7 ++++++-
  3 files changed, 51 insertions(+), 1 deletion(-)
arch/x86/include/asm/sev.h +2 -0
index 4eeedcaca593..2916f4150ac7 100644
@@ -215,6 +215,7 @@ int snp_lookup_rmpentry(u64 pfn, int *level);
  int psmash(u64 pfn);
  int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int asid, bool immutable);
  int rmp_make_shared(u64 pfn, enum pg_level level);
+void sev_dump_rmpentry(u64 pfn);
  #else
  static inline void sev_es_ist_enter(struct pt_regs *regs) { }
  static inline void sev_es_ist_exit(void) { }
@@ -247,6 +248,7 @@ static inline int rmp_make_private(u64 pfn, u64 gpa, enum pg_level level, int as
         return -ENODEV;
  }
  static inline int rmp_make_shared(u64 pfn, enum pg_level level) { return -ENODEV; }
+static inline void sev_dump_rmpentry(u64 pfn) {}
  #endif
 
  #endif
arch/x86/kernel/sev.c +0 -0
index e2b38c3551be..1dd1b36bdfea 100644
@@ -2508,6 +2508,49 @@ static struct rmpentry *__snp_lookup_rmpentry(u64 pfn, int *level)
         return entry;
  }
 
Quoted 10 lines

arch/x86/kernel/sev.c static struct rmpentry *__snp_lookup_rmpentry(u64 pfn, int *level) · line 2518

Quoted 8 lines
arch/x86/kernel/sev.c +8 -0
+void sev_dump_rmpentry(u64 pfn)
+{
+       unsigned long pfn_end;
+       struct rmpentry *e;
+       int level;
+
+       e = __snp_lookup_rmpentry(pfn, &level);
+       if (!e) {

if (IS_ERR(e)) {

Yes, this needs to be fixed to IS_ERR(e)

Quoted 27 lines
Quoted 16 lines
arch/x86/kernel/sev.c +16 -0
+               pr_info("failed to read RMP entry pfn 0x%llx\n", pfn);
+               return;
+       }
+
+       if (rmpentry_assigned(e)) {
+               pr_info("RMPEntry paddr 0x%llx [assigned=%d immutable=%d pagesize=%d gpa=0x%lx"
+                       " asid=%d vmsa=%d validated=%d]\n", pfn << PAGE_SHIFT,
+                       rmpentry_assigned(e), e->info.immutable, rmpentry_pagesize(e),
+                       (unsigned long)e->info.gpa, e->info.asid, e->info.vmsa,
+                       e->info.validated);
+               return;
+       }
+
+       /*
+        * If the RMP entry at the faulting pfn was not assigned, then not sure
+        * what caused the RMP violation. To get some useful debug information,

arch/x86/kernel/sev.c static struct rmpentry *__snp_lookup_rmpentry(u64 pfn, int *level) · line 2544

Quoted 10 lines
arch/x86/kernel/sev.c +10 -0
+        * iterate through the entire 2MB region, and dump the RMP entries if
+        * one of the bit in the RMP entry is set.
+        */
+       pfn = pfn & ~(PTRS_PER_PMD - 1);
+       pfn_end = pfn + PTRS_PER_PMD;
+
+       while (pfn < pfn_end) {
+               e = __snp_lookup_rmpentry(pfn, &level);
+               if (!e)
+                       return;

if (IS_ERR(e))

Quoted 1 line
       continue;

Again, this is correct, but then it should be :

if (IS_ERR(e)) {
	pfn++;
	continue;
}

Thanks,
Ashish

Quoted 40 lines
Quoted 40 lines
arch/x86/kernel/sev.c +9 -0
+
+               if (e->low || e->high)
+                       pr_info("RMPEntry paddr 0x%llx: [high=0x%016llx low=0x%016llx]\n",
+                               pfn << PAGE_SHIFT, e->high, e->low);
+               pfn++;
+       }
+}
+EXPORT_SYMBOL_GPL(sev_dump_rmpentry);
+
  /*
   * Return 1 if the RMP entry is assigned, 0 if it exists but is not assigned,
   * and -errno if there is no corresponding RMP entry.
arch/x86/mm/fault.c +6 -2
index ded53879f98d..f2b16dcfbd9a 100644
@@ -536,6 +536,8 @@ static void show_ldttss(const struct desc_ptr *gdt, const char *name, u16 index)
  static void
  show_fault_oops(struct pt_regs *regs, unsigned long error_code, unsigned long address)
  {
+       unsigned long pfn;
+
         if (!oops_may_print())
                 return;
 
@@ -608,7 +610,10 @@ show_fault_oops(struct pt_regs *regs, unsigned long error_code, unsigned long ad
                 show_ldttss(&gdt, "TR", tr);
         }
 
-       dump_pagetable(address);
+       pfn = dump_pagetable(address);
+
+       if (error_code & X86_PF_RMP)
+               sev_dump_rmpentry(pfn);
  }
 
  static noinline void
--

2.25.1

#93 lore raw

RFC PATCH v7 24/64 crypto:ccp: Define the SEV-SNP commands

In reply to Michael Roth

From: Brijesh Singh <[email protected]>

AMD introduced the next generation of SEV called SEV-SNP (Secure Nested Paging). SEV-SNP builds upon existing SEV and SEV-ES functionality while adding new hardware security protection.

Define the commands and structures used to communicate with the AMD-SP when creating and managing the SEV-SNP guests. The SEV-SNP firmware spec is available at developer.amd.com/sev.

---
 drivers/crypto/ccp/sev-dev.c |  16 +++
 include/linux/psp-sev.h      | 247 +++++++++++++++++++++++++++++++++++
 include/uapi/linux/psp-sev.h |  42 ++++++
 3 files changed, 305 insertions(+)
drivers/crypto/ccp/sev-dev.c +16 -0
index 06fc7156c04f..9d84720a41d7 100644
@@ -126,6 +126,8 @@ static int sev_cmd_buffer_len(int cmd)
 	switch (cmd) {
 	case SEV_CMD_INIT:			return sizeof(struct sev_data_init);
 	case SEV_CMD_INIT_EX:                   return sizeof(struct sev_data_init_ex);
+	case SEV_CMD_SNP_SHUTDOWN_EX:		return sizeof(struct sev_data_snp_shutdown_ex);
+	case SEV_CMD_SNP_INIT_EX:		return sizeof(struct sev_data_snp_init_ex);
 	case SEV_CMD_PLATFORM_STATUS:		return sizeof(struct sev_user_data_status);
 	case SEV_CMD_PEK_CSR:			return sizeof(struct sev_data_pek_csr);
 	case SEV_CMD_PEK_CERT_IMPORT:		return sizeof(struct sev_data_pek_cert_import);
@@ -154,6 +156,20 @@ static int sev_cmd_buffer_len(int cmd)
 	case SEV_CMD_GET_ID:			return sizeof(struct sev_data_get_id);
 	case SEV_CMD_ATTESTATION_REPORT:	return sizeof(struct sev_data_attestation_report);
 	case SEV_CMD_SEND_CANCEL:		return sizeof(struct sev_data_send_cancel);
+	case SEV_CMD_SNP_GCTX_CREATE:		return sizeof(struct sev_data_snp_addr);
+	case SEV_CMD_SNP_LAUNCH_START:		return sizeof(struct sev_data_snp_launch_start);
+	case SEV_CMD_SNP_LAUNCH_UPDATE:		return sizeof(struct sev_data_snp_launch_update);
+	case SEV_CMD_SNP_ACTIVATE:		return sizeof(struct sev_data_snp_activate);
+	case SEV_CMD_SNP_DECOMMISSION:		return sizeof(struct sev_data_snp_addr);
+	case SEV_CMD_SNP_PAGE_RECLAIM:		return sizeof(struct sev_data_snp_page_reclaim);
+	case SEV_CMD_SNP_GUEST_STATUS:		return sizeof(struct sev_data_snp_guest_status);
+	case SEV_CMD_SNP_LAUNCH_FINISH:		return sizeof(struct sev_data_snp_launch_finish);
+	case SEV_CMD_SNP_DBG_DECRYPT:		return sizeof(struct sev_data_snp_dbg);
+	case SEV_CMD_SNP_DBG_ENCRYPT:		return sizeof(struct sev_data_snp_dbg);
+	case SEV_CMD_SNP_PAGE_UNSMASH:		return sizeof(struct sev_data_snp_page_unsmash);
+	case SEV_CMD_SNP_PLATFORM_STATUS:	return sizeof(struct sev_data_snp_addr);
+	case SEV_CMD_SNP_GUEST_REQUEST:		return sizeof(struct sev_data_snp_guest_request);
+	case SEV_CMD_SNP_CONFIG:		return sizeof(struct sev_user_data_snp_config);
 	default:				return 0;
 	}
 
include/linux/psp-sev.h +247 -0

259 lines not on this page · raw message

include/uapi/linux/psp-sev.h +42 -0

54 lines not on this page · raw message

Signature · 1 line
2.25.1
#94 lore raw

Re: [PATCH RFC v7 24/64] crypto:ccp: Define the SEV-SNP commands

Quoted 5 lines · Michael Roth

From: Brijesh Singh <[email protected]>

include/uapi/linux/psp-sev.h +3 -0
+/*
+ * struct sev_user_data_snp_config - system wide configuration value for SNP.
+ *

include/uapi/linux/psp-sev.h struct sev_user_data_get_id2 { · line 191

Quoted 10 lines
include/uapi/linux/psp-sev.h +10 -0
+ * @reported_tcb: The TCB version to report in the guest attestation report.
+ * @mask_chip_id: Indicates that the CHID_ID field in the attestation report
+ * will always be zero.
+ */
+struct sev_user_data_snp_config {
+       __u64 reported_tcb;     /* In */
+       __u32 mask_chip_id;     /* In */
+       __u8 rsvd[52];
+} __packed;
+

Based on table 45 section 8.6.1 in
https://www.amd.com/system/files/TechDocs/56860.pdf I think this should be

struct sev_user_data_snp_config {
       __u64 reported_tcb;     /* In */
       __u32 mask_chip_id:1;     /* In */
       __u32 mask_chip_key:1;     /* In */
       __u32 rsvd:30;
       __u8 rsvd1[52];
} __packed;

--
Sabin.

Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

#95 lore raw

RFC PATCH v7 25/64 crypto: ccp: Add support to initialize the AMD-SP for SEV-SNP

In reply to Michael Roth

From: Brijesh Singh <[email protected]>

Before SNP VMs can be launched, the platform must be appropriately configured and initialized. Platform initialization is accomplished via the SNP_INIT command. Make sure to do a WBINVD and issue DF_FLUSH command to prepare for the first SNP guest launch after INIT.

During the execution of SNP_INIT command, the firmware configures and enables SNP security policy enforcement in many system components. Some system components write to regions of memory reserved by early x86 firmware (e.g. UEFI). Other system components write to regions provided by the operation system, hypervisor, or x86 firmware. Such system components can only write to HV-fixed pages or Default pages. They will error when attempting to write to other page states after SNP_INIT enables their SNP enforcement.

Starting in SNP firmware v1.52, the SNP_INIT_EX command takes a list of system physical address ranges to convert into the HV-fixed page states during the RMP initialization. If INIT_RMP is 1, hypervisors should provide all system physical address ranges that the hypervisor will never assign to a guest until the next RMP re-initialization. For instance, the memory that UEFI reserves should be included in the range list. This allows system components that occasionally write to memory (e.g. logging to UEFI reserved regions) to not fail due to RMP initialization and SNP enablement.

---
 drivers/crypto/ccp/sev-dev.c | 225 +++++++++++++++++++++++++++++++++++
 drivers/crypto/ccp/sev-dev.h |   2 +
 include/linux/psp-sev.h      |  17 +++
 3 files changed, 244 insertions(+)
drivers/crypto/ccp/sev-dev.c +225 -0
index 9d84720a41d7..af20420bd6c2 100644
@@ -26,6 +26,7 @@
 #include <linux/fs_struct.h>
 
 #include <asm/smp.h>
+#include <asm/e820/types.h>
 
 #include "psp-dev.h"
 #include "sev-dev.h"
@@ -34,6 +35,10 @@
 #define SEV_FW_FILE		"amd/sev.fw"
 #define SEV_FW_NAME_SIZE	64
 
+/* Minimum firmware version required for the SEV-SNP support */
+#define SNP_MIN_API_MAJOR	1
+#define SNP_MIN_API_MINOR	51
+
 static DEFINE_MUTEX(sev_cmd_mutex);
 static struct sev_misc_dev *misc_dev;
 
@@ -76,6 +81,13 @@ static void *sev_es_tmr;
 #define NV_LENGTH (32 * 1024)
 static void *sev_init_ex_buffer;
 
+/*
+ * SEV_DATA_RANGE_LIST:
+ *   Array containing range of pages that firmware transitions to HV-fixed
+ *   page state.
+ */
+struct sev_data_range_list *snp_range_list;
+
 static inline bool sev_version_greater_or_equal(u8 maj, u8 min)
 {
 	struct sev_device *sev = psp_master->sev_data;
@@ -830,6 +842,186 @@ static int sev_update_firmware(struct device *dev)
 	return ret;
 }
 
+static void snp_set_hsave_pa(void *arg)
+{
+	wrmsrl(MSR_VM_HSAVE_PA, 0);
+}
+
+static int snp_filter_reserved_mem_regions(struct resource *rs, void *arg)
+{
+	struct sev_data_range_list *range_list = arg;
+	struct sev_data_range *range = &range_list->ranges[range_list->num_elements];
+	size_t size;
+
+	if ((range_list->num_elements * sizeof(struct sev_data_range) +
+	     sizeof(struct sev_data_range_list)) > PAGE_SIZE)
+		return -E2BIG;
+
+	switch (rs->desc) {
+	case E820_TYPE_RESERVED:
+	case E820_TYPE_PMEM:
+	case E820_TYPE_ACPI:
+		range->base = rs->start & PAGE_MASK;
+		size = (rs->end + 1) - rs->start;
+		range->page_count = size >> PAGE_SHIFT;
+		range_list->num_elements++;
+		break;
+	default:
+		break;
+	}
+
+	return 0;
+}
+
+static int __sev_snp_init_locked(int *error)
+{
+	struct psp_device *psp = psp_master;
+	struct sev_data_snp_init_ex data;
+	struct sev_device *sev;
+	int rc = 0;
+
+	if (!psp || !psp->sev_data)
+		return -ENODEV;
+
+	sev = psp->sev_data;
+
+	if (sev->snp_initialized)
+		return 0;
+
+	/*
+	 * The SNP_INIT requires the MSR_VM_HSAVE_PA must be set to 0h
+	 * across all cores.
+	 */
+	on_each_cpu(snp_set_hsave_pa, NULL, 1);
+
+	/*
+	 * Starting in SNP firmware v1.52, the SNP_INIT_EX command takes a list of
+	 * system physical address ranges to convert into the HV-fixed page states
+	 * during the RMP initialization.  For instance, the memory that UEFI
+	 * reserves should be included in the range list. This allows system
+	 * components that occasionally write to memory (e.g. logging to UEFI
+	 * reserved regions) to not fail due to RMP initialization and SNP enablement.
+	 */
+	if (sev_version_greater_or_equal(SNP_MIN_API_MAJOR, 52)) {
+		/*
+		 * Firmware checks that the pages containing the ranges enumerated
+		 * in the RANGES structure are either in the Default page state or in the
+		 * firmware page state.
+		 */
+		snp_range_list = sev_fw_alloc(PAGE_SIZE);
+		if (!snp_range_list) {
+			dev_err(sev->dev,
+				"SEV: SNP_INIT_EX range list memory allocation failed\n");
+			return -ENOMEM;
+		}
+
+		memset(snp_range_list, 0, PAGE_SIZE);
+
+		/*
+		 * Retrieve all reserved memory regions setup by UEFI from the e820 memory map
+		 * to be setup as HV-fixed pages.
+		 */
+
+		rc = walk_iomem_res_desc(IORES_DESC_NONE, IORESOURCE_MEM, 0, ~0,
+					 snp_range_list, snp_filter_reserved_mem_regions);
+		if (rc) {
+			dev_err(sev->dev,
+				"SEV: SNP_INIT_EX walk_iomem_res_desc failed rc = %d\n", rc);
+			return rc;
+		}
+
+		memset(&data, 0, sizeof(data));
+		data.init_rmp = 1;
+		data.list_paddr_en = 1;
+		data.list_paddr = __pa(snp_range_list);
+
+		rc = __sev_do_cmd_locked(SEV_CMD_SNP_INIT_EX, &data, error);
+		if (rc)
+			return rc;
+	} else {
+		rc = __sev_do_cmd_locked(SEV_CMD_SNP_INIT, NULL, error);
+		if (rc)
+			return rc;
+	}
+
+	/* Prepare for first SNP guest launch after INIT */
+	wbinvd_on_all_cpus();
+	rc = __sev_do_cmd_locked(SEV_CMD_SNP_DF_FLUSH, NULL, error);
+	if (rc)
+		return rc;
+
+	sev->snp_initialized = true;
+	dev_dbg(sev->dev, "SEV-SNP firmware initialized\n");
+
+	return rc;
+}
+
+int sev_snp_init(int *error, bool init_on_probe)
+{
+	int rc;
+
+	if (!cpu_feature_enabled(X86_FEATURE_SEV_SNP))
+		return -ENODEV;
+
+	if (init_on_probe && !psp_init_on_probe)
+		return 0;
+
+	mutex_lock(&sev_cmd_mutex);
+	rc = __sev_snp_init_locked(error);
+	mutex_unlock(&sev_cmd_mutex);
+
+	return rc;
+}
+EXPORT_SYMBOL_GPL(sev_snp_init);
+
+static int __sev_snp_shutdown_locked(int *error)
+{
+	struct sev_device *sev = psp_master->sev_data;
+	struct sev_data_snp_shutdown_ex data;
+	int ret;
+
+	if (!sev->snp_initialized)
+		return 0;
+
+	memset(&data, 0, sizeof(data));
+	data.length = sizeof(data);
+	data.iommu_snp_shutdown = 1;
+
+	wbinvd_on_all_cpus();
+
+retry:
+	ret = __sev_do_cmd_locked(SEV_CMD_SNP_SHUTDOWN_EX, &data, error);
+	/* SHUTDOWN may require DF_FLUSH */
+	if (*error == SEV_RET_DFFLUSH_REQUIRED) {
+		ret = __sev_do_cmd_locked(SEV_CMD_SNP_DF_FLUSH, NULL, NULL);
+		if (ret) {
+			dev_err(sev->dev, "SEV-SNP DF_FLUSH failed\n");
+			return ret;
+		}
+		goto retry;
+	}
+	if (ret) {
+		dev_err(sev->dev, "SEV-SNP firmware shutdown failed\n");
+		return ret;
+	}
+
+	sev->snp_initialized = false;
+	dev_dbg(sev->dev, "SEV-SNP firmware shutdown\n");
+
+	return ret;
+}
+
+static int sev_snp_shutdown(int *error)
+{
+	int rc;
+
+	mutex_lock(&sev_cmd_mutex);
+	rc = __sev_snp_shutdown_locked(error);
+	mutex_unlock(&sev_cmd_mutex);
+
+	return rc;
+}
+
 static int sev_ioctl_do_pek_import(struct sev_issue_cmd *argp, bool writable)
 {
 	struct sev_device *sev = psp_master->sev_data;
@@ -1270,6 +1462,8 @@ int sev_dev_init(struct psp_device *psp)
 
 static void sev_firmware_shutdown(struct sev_device *sev)
 {
+	int error;
+
 	sev_platform_shutdown(NULL);
 
 	if (sev_es_tmr) {
@@ -1286,6 +1480,14 @@ static void sev_firmware_shutdown(struct sev_device *sev)
 			   get_order(NV_LENGTH));
 		sev_init_ex_buffer = NULL;
 	}
+
+	if (snp_range_list) {
+		free_pages((unsigned long)snp_range_list,
+			   get_order(PAGE_SIZE));
+		snp_range_list = NULL;
+	}
+
+	sev_snp_shutdown(&error);
 }
 
 void sev_dev_destroy(struct psp_device *psp)
@@ -1341,6 +1543,26 @@ void sev_pci_init(void)
 		}
 	}
 
+	/*
+	 * If boot CPU supports SNP, then first attempt to initialize
+	 * the SNP firmware.
+	 */
+	if (cpu_feature_enabled(X86_FEATURE_SEV_SNP)) {
+		if (!sev_version_greater_or_equal(SNP_MIN_API_MAJOR, SNP_MIN_API_MINOR)) {
+			dev_err(sev->dev, "SEV-SNP support requires firmware version >= %d:%d\n",
+				SNP_MIN_API_MAJOR, SNP_MIN_API_MINOR);
+		} else {
+			rc = sev_snp_init(&error, true);
+			if (rc) {
+				/*
+				 * Don't abort the probe if SNP INIT failed,
+				 * continue to initialize the legacy SEV firmware.
+				 */
+				dev_err(sev->dev, "SEV-SNP: failed to INIT error %#x\n", error);
+			}
+		}
+	}
+
 	/* Obtain the TMR memory area for SEV-ES use */
 	sev_es_tmr = sev_fw_alloc(SEV_ES_TMR_SIZE);
 	if (!sev_es_tmr)
@@ -1356,6 +1578,9 @@ void sev_pci_init(void)
 		dev_err(sev->dev, "SEV: failed to INIT error %#x, rc %d\n",
 			error, rc);
 
+	dev_info(sev->dev, "SEV%s API:%d.%d build:%d\n", sev->snp_initialized ?
+		"-SNP" : "", sev->api_major, sev->api_minor, sev->build);
+
 	return;
 
 err:
drivers/crypto/ccp/sev-dev.h +2 -0
index 666c21eb81ab..34767657beb5 100644
@@ -52,6 +52,8 @@ struct sev_device {
 	u8 build;
 
 	void *cmd_buf;
+
+	bool snp_initialized;
 };
 
 int sev_dev_init(struct psp_device *psp);
include/linux/psp-sev.h +17 -0
index 31b045e1926f..8cfe92e82743 100644
@@ -794,6 +794,21 @@ struct sev_data_snp_shutdown_ex {
  */
 int sev_platform_init(int *error);
 
+/**
+ * sev_snp_init - perform SEV SNP_INIT command
+ *
+ * @error: SEV command return code
+ * @init_on_probe: indicates if called during module probe/init
+ *
+ * Returns:
+ * 0 if the SEV successfully processed the command
+ * -%ENODEV    if the SEV device is not available
+ * -%ENOTSUPP  if the SEV does not support SEV
+ * -%ETIMEDOUT if the SEV command timed out
+ * -%EIO       if the SEV returned a non-zero return code
+ */
+int sev_snp_init(int *error, bool init_on_probe);
+
 /**
  * sev_platform_status - perform SEV PLATFORM_STATUS command
  *
@@ -901,6 +916,8 @@ sev_platform_status(struct sev_user_data_status *status, int *error) { return -E
 
 static inline int sev_platform_init(int *error) { return -ENODEV; }
 
+static inline int sev_snp_init(int *error, bool init_on_probe) { return -ENODEV; }
+
 static inline int
 sev_guest_deactivate(struct sev_data_deactivate *data, int *error) { return -ENODEV; }
 
Signature · 1 line
2.25.1
#96 lore raw

Re: [PATCH RFC v7 25/64] crypto: ccp: Add support to initialize the AMD-SP for SEV-SNP

Quoted 112 lines · Michael Roth

From: Brijesh Singh <[email protected]>

Before SNP VMs can be launched, the platform must be appropriately configured and initialized. Platform initialization is accomplished via the SNP_INIT command. Make sure to do a WBINVD and issue DF_FLUSH command to prepare for the first SNP guest launch after INIT.

During the execution of SNP_INIT command, the firmware configures and enables SNP security policy enforcement in many system components. Some system components write to regions of memory reserved by early x86 firmware (e.g. UEFI). Other system components write to regions provided by the operation system, hypervisor, or x86 firmware. Such system components can only write to HV-fixed pages or Default pages. They will error when attempting to write to other page states after SNP_INIT enables their SNP enforcement.

Starting in SNP firmware v1.52, the SNP_INIT_EX command takes a list of system physical address ranges to convert into the HV-fixed page states during the RMP initialization. If INIT_RMP is 1, hypervisors should provide all system physical address ranges that the hypervisor will never assign to a guest until the next RMP re-initialization. For instance, the memory that UEFI reserves should be included in the range list. This allows system components that occasionally write to memory (e.g. logging to UEFI reserved regions) to not fail due to RMP initialization and SNP enablement.

---
 drivers/crypto/ccp/sev-dev.c | 225 +++++++++++++++++++++++++++++++++++
 drivers/crypto/ccp/sev-dev.h |   2 +
 include/linux/psp-sev.h      |  17 +++
 3 files changed, 244 insertions(+)
drivers/crypto/ccp/sev-dev.c +47 -0
index 9d84720a41d7..af20420bd6c2 100644
@@ -26,6 +26,7 @@
 #include <linux/fs_struct.h>
 
 #include <asm/smp.h>
+#include <asm/e820/types.h>
 
 #include "psp-dev.h"
 #include "sev-dev.h"
@@ -34,6 +35,10 @@
 #define SEV_FW_FILE		"amd/sev.fw"
 #define SEV_FW_NAME_SIZE	64
 
+/* Minimum firmware version required for the SEV-SNP support */
+#define SNP_MIN_API_MAJOR	1
+#define SNP_MIN_API_MINOR	51
+
 static DEFINE_MUTEX(sev_cmd_mutex);
 static struct sev_misc_dev *misc_dev;
 
@@ -76,6 +81,13 @@ static void *sev_es_tmr;
 #define NV_LENGTH (32 * 1024)
 static void *sev_init_ex_buffer;
 
+/*
+ * SEV_DATA_RANGE_LIST:
+ *   Array containing range of pages that firmware transitions to HV-fixed
+ *   page state.
+ */
+struct sev_data_range_list *snp_range_list;
+
 static inline bool sev_version_greater_or_equal(u8 maj, u8 min)
 {
 	struct sev_device *sev = psp_master->sev_data;
@@ -830,6 +842,186 @@ static int sev_update_firmware(struct device *dev)
 	return ret;
 }
 
+static void snp_set_hsave_pa(void *arg)
+{
+	wrmsrl(MSR_VM_HSAVE_PA, 0);
+}
+
+static int snp_filter_reserved_mem_regions(struct resource *rs, void *arg)
+{
+	struct sev_data_range_list *range_list = arg;
+	struct sev_data_range *range = &range_list->ranges[range_list->num_elements];
+	size_t size;
+
+	if ((range_list->num_elements * sizeof(struct sev_data_range) +
+	     sizeof(struct sev_data_range_list)) > PAGE_SIZE)
+		return -E2BIG;
+
+	switch (rs->desc) {
+	case E820_TYPE_RESERVED:
+	case E820_TYPE_PMEM:
+	case E820_TYPE_ACPI:
+		range->base = rs->start & PAGE_MASK;
+		size = (rs->end + 1) - rs->start;
+		range->page_count = size >> PAGE_SHIFT;
+		range_list->num_elements++;
+		break;
+	default:
+		break;
+	}
+
+	return 0;
+}
+
+static int __sev_snp_init_locked(int *error)
+{
+	struct psp_device *psp = psp_master;
+	struct sev_data_snp_init_ex data;

drivers/crypto/ccp/sev-dev.c static int sev_update_firmware(struct device *dev) · line 889

Quoted 10 lines
drivers/crypto/ccp/sev-dev.c +10 -0
+	struct sev_device *sev;
+	int rc = 0;
+
+	if (!psp || !psp->sev_data)
+		return -ENODEV;
+
+	sev = psp->sev_data;
+
+	if (sev->snp_initialized)
+		return 0;

Shouldn't this follow this check:

        if (sev->state == SEV_STATE_INIT) {
                /* debug printk about possible incorrect call order */
                return -ENODEV;
        }

It is game over for SNP, if SEV_CMD_INIT{_EX} got first, which means that this should not proceed.

BR, Jarkko

maintainer
#97 lore raw

Re: [PATCH RFC v7 25/64] crypto: ccp: Add support to initialize the AMD-SP for SEV-SNP

Hello Jarkko,

Quoted 126 lines · Jarkko Sakkinen
Quoted 112 lines · Michael Roth

From: Brijesh Singh <[email protected]>

Before SNP VMs can be launched, the platform must be appropriately configured and initialized. Platform initialization is accomplished via the SNP_INIT command. Make sure to do a WBINVD and issue DF_FLUSH command to prepare for the first SNP guest launch after INIT.

During the execution of SNP_INIT command, the firmware configures and enables SNP security policy enforcement in many system components. Some system components write to regions of memory reserved by early x86 firmware (e.g. UEFI). Other system components write to regions provided by the operation system, hypervisor, or x86 firmware. Such system components can only write to HV-fixed pages or Default pages. They will error when attempting to write to other page states after SNP_INIT enables their SNP enforcement.

Starting in SNP firmware v1.52, the SNP_INIT_EX command takes a list of system physical address ranges to convert into the HV-fixed page states during the RMP initialization. If INIT_RMP is 1, hypervisors should provide all system physical address ranges that the hypervisor will never assign to a guest until the next RMP re-initialization. For instance, the memory that UEFI reserves should be included in the range list. This allows system components that occasionally write to memory (e.g. logging to UEFI reserved regions) to not fail due to RMP initialization and SNP enablement.

---
  drivers/crypto/ccp/sev-dev.c | 225 +++++++++++++++++++++++++++++++++++
  drivers/crypto/ccp/sev-dev.h |   2 +
  include/linux/psp-sev.h      |  17 +++
  3 files changed, 244 insertions(+)
drivers/crypto/ccp/sev-dev.c +47 -0
index 9d84720a41d7..af20420bd6c2 100644
@@ -26,6 +26,7 @@
  #include <linux/fs_struct.h>
 
  #include <asm/smp.h>
+#include <asm/e820/types.h>
 
  #include "psp-dev.h"
  #include "sev-dev.h"
@@ -34,6 +35,10 @@
  #define SEV_FW_FILE		"amd/sev.fw"
  #define SEV_FW_NAME_SIZE	64
 
+/* Minimum firmware version required for the SEV-SNP support */
+#define SNP_MIN_API_MAJOR	1
+#define SNP_MIN_API_MINOR	51
+
  static DEFINE_MUTEX(sev_cmd_mutex);
  static struct sev_misc_dev *misc_dev;
 
@@ -76,6 +81,13 @@ static void *sev_es_tmr;
  #define NV_LENGTH (32 * 1024)
  static void *sev_init_ex_buffer;
 
+/*
+ * SEV_DATA_RANGE_LIST:
+ *   Array containing range of pages that firmware transitions to HV-fixed
+ *   page state.
+ */
+struct sev_data_range_list *snp_range_list;
+
  static inline bool sev_version_greater_or_equal(u8 maj, u8 min)
  {
  	struct sev_device *sev = psp_master->sev_data;
@@ -830,6 +842,186 @@ static int sev_update_firmware(struct device *dev)
  	return ret;
  }
 
+static void snp_set_hsave_pa(void *arg)
+{
+	wrmsrl(MSR_VM_HSAVE_PA, 0);
+}
+
+static int snp_filter_reserved_mem_regions(struct resource *rs, void *arg)
+{
+	struct sev_data_range_list *range_list = arg;
+	struct sev_data_range *range = &range_list->ranges[range_list->num_elements];
+	size_t size;
+
+	if ((range_list->num_elements * sizeof(struct sev_data_range) +
+	     sizeof(struct sev_data_range_list)) > PAGE_SIZE)
+		return -E2BIG;
+
+	switch (rs->desc) {
+	case E820_TYPE_RESERVED:
+	case E820_TYPE_PMEM:
+	case E820_TYPE_ACPI:
+		range->base = rs->start & PAGE_MASK;
+		size = (rs->end + 1) - rs->start;
+		range->page_count = size >> PAGE_SHIFT;
+		range_list->num_elements++;
+		break;
+	default:
+		break;
+	}
+
+	return 0;
+}
+
+static int __sev_snp_init_locked(int *error)
+{
+	struct psp_device *psp = psp_master;
+	struct sev_data_snp_init_ex data;

drivers/crypto/ccp/sev-dev.c static int sev_update_firmware(struct device *dev) · line 889

Quoted 10 lines
drivers/crypto/ccp/sev-dev.c +10 -0
+	struct sev_device *sev;
+	int rc = 0;
+
+	if (!psp || !psp->sev_data)
+		return -ENODEV;
+
+	sev = psp->sev_data;
+
+	if (sev->snp_initialized)
+		return 0;

Shouldn't this follow this check:

Quoted 7 lines
         if (sev->state == SEV_STATE_INIT) {
                 /* debug printk about possible incorrect call order */
                 return -ENODEV;
         }

It is game over for SNP, if SEV_CMD_INIT{_EX} got first, which means that this should not proceed.

But, how will SEV_CMD_INIT_EX happen before as sev_pci_init() which is invoked during CCP module load/initialization, will first try to do sev_snp_init() if SNP is supported, before it invokes sev_platform_init() to do SEV firmware initialization ?

Thanks,
Ashish

#98 lore raw

Re: [PATCH RFC v7 25/64] crypto: ccp: Add support to initialize the AMD-SP for SEV-SNP

Quoted 128 lines · Ashish

Hello Jarkko,

Quoted 125 lines · Jarkko Sakkinen
Quoted 106 lines · Michael Roth

From: Brijesh Singh <[email protected]>

Before SNP VMs can be launched, the platform must be appropriately configured and initialized. Platform initialization is accomplished via the SNP_INIT command. Make sure to do a WBINVD and issue DF_FLUSH command to prepare for the first SNP guest launch after INIT.

During the execution of SNP_INIT command, the firmware configures and enables SNP security policy enforcement in many system components. Some system components write to regions of memory reserved by early x86 firmware (e.g. UEFI). Other system components write to regions provided by the operation system, hypervisor, or x86 firmware. Such system components can only write to HV-fixed pages or Default pages. They will error when attempting to write to other page states after SNP_INIT enables their SNP enforcement.

Starting in SNP firmware v1.52, the SNP_INIT_EX command takes a list of system physical address ranges to convert into the HV-fixed page states during the RMP initialization. If INIT_RMP is 1, hypervisors should provide all system physical address ranges that the hypervisor will never assign to a guest until the next RMP re-initialization. For instance, the memory that UEFI reserves should be included in the range list. This allows system components that occasionally write to memory (e.g. logging to UEFI reserved regions) to not fail due to RMP initialization and SNP enablement.

---
  drivers/crypto/ccp/sev-dev.c | 225 +++++++++++++++++++++++++++++++++++
  drivers/crypto/ccp/sev-dev.h |   2 +
  include/linux/psp-sev.h      |  17 +++
  3 files changed, 244 insertions(+)
drivers/crypto/ccp/sev-dev.c +47 -0
index 9d84720a41d7..af20420bd6c2 100644
@@ -26,6 +26,7 @@
  #include <linux/fs_struct.h>
  #include <asm/smp.h>
+#include <asm/e820/types.h>
  #include "psp-dev.h"
  #include "sev-dev.h"
@@ -34,6 +35,10 @@
  #define SEV_FW_FILE		"amd/sev.fw"
  #define SEV_FW_NAME_SIZE	64
+/* Minimum firmware version required for the SEV-SNP support */
+#define SNP_MIN_API_MAJOR	1
+#define SNP_MIN_API_MINOR	51
+
  static DEFINE_MUTEX(sev_cmd_mutex);
  static struct sev_misc_dev *misc_dev;
@@ -76,6 +81,13 @@ static void *sev_es_tmr;
  #define NV_LENGTH (32 * 1024)
  static void *sev_init_ex_buffer;
+/*
+ * SEV_DATA_RANGE_LIST:
+ *   Array containing range of pages that firmware transitions to HV-fixed
+ *   page state.
+ */
+struct sev_data_range_list *snp_range_list;
+
  static inline bool sev_version_greater_or_equal(u8 maj, u8 min)
  {
  	struct sev_device *sev = psp_master->sev_data;
@@ -830,6 +842,186 @@ static int sev_update_firmware(struct device *dev)
  	return ret;
  }
+static void snp_set_hsave_pa(void *arg)
+{
+	wrmsrl(MSR_VM_HSAVE_PA, 0);
+}
+
+static int snp_filter_reserved_mem_regions(struct resource *rs, void *arg)
+{
+	struct sev_data_range_list *range_list = arg;
+	struct sev_data_range *range = &range_list->ranges[range_list->num_elements];
+	size_t size;
+
+	if ((range_list->num_elements * sizeof(struct sev_data_range) +
+	     sizeof(struct sev_data_range_list)) > PAGE_SIZE)
+		return -E2BIG;
+
+	switch (rs->desc) {
+	case E820_TYPE_RESERVED:
+	case E820_TYPE_PMEM:
+	case E820_TYPE_ACPI:
+		range->base = rs->start & PAGE_MASK;
+		size = (rs->end + 1) - rs->start;
+		range->page_count = size >> PAGE_SHIFT;
+		range_list->num_elements++;
+		break;
+	default:
+		break;
+	}
+
+	return 0;
+}
+
+static int __sev_snp_init_locked(int *error)
+{
+	struct psp_device *psp = psp_master;
+	struct sev_data_snp_init_ex data;

drivers/crypto/ccp/sev-dev.c static int sev_update_firmware(struct device *dev) · line 888

Quoted 10 lines
drivers/crypto/ccp/sev-dev.c +10 -0
+	struct sev_device *sev;
+	int rc = 0;
+
+	if (!psp || !psp->sev_data)
+		return -ENODEV;
+
+	sev = psp->sev_data;
+
+	if (sev->snp_initialized)
+		return 0;

Shouldn't this follow this check:

         if (sev->state == SEV_STATE_INIT) {
                 /* debug printk about possible incorrect call order */
                 return -ENODEV;
         }
Quoted 8 lines
Quoted 2 lines

It is game over for SNP, if SEV_CMD_INIT{_EX} got first, which means that this should not proceed.

But, how will SEV_CMD_INIT_EX happen before as sev_pci_init() which is invoked during CCP module load/initialization, will first try to do sev_snp_init() if SNP is supported, before it invokes sev_platform_init() to do SEV firmware initialization ?

Because the symbol is exported outside the driver to be called by other subsystems, you need to have a santiy check for the call order, as it is a hardware constraint. Otherwise, any unconsidered change in either side could unknowingily break the kernel.

Alternatively, you could choose not to export sev_snp_init(). It is supported by the fact that the call in sev_guest_init() is does nothing useful (for the reasons you already wrote).

BR, Jarkko

#99 lore raw

Re: [PATCH RFC v7 25/64] crypto: ccp: Add support to initialize the AMD-SP for SEV-SNP

In reply to Michael Roth
Quoted 9 lines · Michael Roth
drivers/crypto/ccp/sev-dev.c +9 -0
+	/*
+	 * If boot CPU supports SNP, then first attempt to initialize
+	 * the SNP firmware.
+	 */
+	if (cpu_feature_enabled(X86_FEATURE_SEV_SNP)) {
+		if (!sev_version_greater_or_equal(SNP_MIN_API_MAJOR, SNP_MIN_API_MINOR)) {
+			dev_err(sev->dev, "SEV-SNP support requires firmware version >= %d:%d\n",
+				SNP_MIN_API_MAJOR, SNP_MIN_API_MINOR);
+		} else {

drivers/crypto/ccp/sev-dev.c void sev_pci_init(void) · line 1564

Quoted 10 lines
drivers/crypto/ccp/sev-dev.c +10 -0
+			rc = sev_snp_init(&error, true);
+			if (rc) {
+				/*
+				 * Don't abort the probe if SNP INIT failed,
+				 * continue to initialize the legacy SEV firmware.
+				 */
+				dev_err(sev->dev, "SEV-SNP: failed to INIT error %#x\n", error);
+			}
+		}
+	}

I think this is not right as there is a dep between sev init and this, and there is about a dozen of call sites already __sev_platform_init_locked().

Instead there should be __sev_snp_init_locked() that would be called as part of __sev_platform_init_locked() flow.

Also TMR allocation should be moved inside __sev_platform_init_locked, given that it needs to be marked into RMP after SNP init.

BR, Jarkko

maintainer
#100 lore raw

Re: [PATCH RFC v7 25/64] crypto: ccp: Add support to initialize the AMD-SP for SEV-SNP

Hello Jarkko,

Quoted 15 lines · Jarkko Sakkinen
Quoted 14 lines · Michael Roth
drivers/crypto/ccp/sev-dev.c +14 -0
+	/*
+	 * If boot CPU supports SNP, then first attempt to initialize
+	 * the SNP firmware.
+	 */
+	if (cpu_feature_enabled(X86_FEATURE_SEV_SNP)) {
+		if (!sev_version_greater_or_equal(SNP_MIN_API_MAJOR, SNP_MIN_API_MINOR)) {
+			dev_err(sev->dev, "SEV-SNP support requires firmware version >= %d:%d\n",
+				SNP_MIN_API_MAJOR, SNP_MIN_API_MINOR);
+		} else {
+			rc = sev_snp_init(&error, true);
+			if (rc) {
+				/*
+				 * Don't abort the probe if SNP INIT failed,
+				 * continue to initialize the legacy SEV firmware.
Quoted 9 lines

drivers/crypto/ccp/sev-dev.c void sev_pci_init(void) · line 1564

Quoted 5 lines
drivers/crypto/ccp/sev-dev.c +5 -0
+				 */
+				dev_err(sev->dev, "SEV-SNP: failed to INIT error %#x\n", error);
+			}
+		}
+	}

I think this is not right as there is a dep between sev init and this, and there is about a dozen of call sites already __sev_platform_init_locked().

sev_init ?

As this is invoked during CCP module load/initialization, shouldn't this get invoked before any other call sites invoking __sev_platform_init_locked() ?

Thanks,
Ashish

Quoted 8 lines

Instead there should be __sev_snp_init_locked() that would be called as part of __sev_platform_init_locked() flow.

Also TMR allocation should be moved inside __sev_platform_init_locked, given that it needs to be marked into RMP after SNP init.

BR, Jarkko

#101 lore raw

Re: [PATCH RFC v7 25/64] crypto: ccp: Add support to initialize the AMD-SP for SEV-SNP

Quoted 28 lines · Ashish

Hello Jarkko,

Quoted 15 lines · Jarkko Sakkinen
Quoted 14 lines · Michael Roth
drivers/crypto/ccp/sev-dev.c +14 -0
+	/*
+	 * If boot CPU supports SNP, then first attempt to initialize
+	 * the SNP firmware.
+	 */
+	if (cpu_feature_enabled(X86_FEATURE_SEV_SNP)) {
+		if (!sev_version_greater_or_equal(SNP_MIN_API_MAJOR, SNP_MIN_API_MINOR)) {
+			dev_err(sev->dev, "SEV-SNP support requires firmware version >= %d:%d\n",
+				SNP_MIN_API_MAJOR, SNP_MIN_API_MINOR);
+		} else {
+			rc = sev_snp_init(&error, true);
+			if (rc) {
+				/*
+				 * Don't abort the probe if SNP INIT failed,
+				 * continue to initialize the legacy SEV firmware.
Quoted 9 lines

drivers/crypto/ccp/sev-dev.c void sev_pci_init(void) · line 1564

Quoted 5 lines
drivers/crypto/ccp/sev-dev.c +5 -0
+				 */
+				dev_err(sev->dev, "SEV-SNP: failed to INIT error %#x\n", error);
+			}
+		}
+	}

I think this is not right as there is a dep between sev init and this, and there is about a dozen of call sites already __sev_platform_init_locked().

Quoted 4 lines

sev_init ?

As this is invoked during CCP module load/initialization, shouldn't this get invoked before any other call sites invoking __sev_platform_init_locked() ?

Then it should not be exported because this the only working call site.

However, the benefit of __sev_platform_init_locked() addressing SNP init is that psp_init_on_probe can also postpone SNP init without possibility to any side effects (other call sites than sev_guest_init()).

BR, Jarkko

#102 lore raw

RFC PATCH v7 26/64 crypto:ccp: Provide API to issue SEV and SNP commands

In reply to Michael Roth

From: Brijesh Singh <[email protected]>

Make sev_do_cmd() a generic API interface for the hypervisor to issue commands to manage an SEV and SNP guest. The commands for SEV and SNP are defined in the SEV and SEV-SNP firmware specifications.

---
 drivers/crypto/ccp/sev-dev.c |  3 ++-
 include/linux/psp-sev.h      | 17 +++++++++++++++++
 2 files changed, 19 insertions(+), 1 deletion(-)
drivers/crypto/ccp/sev-dev.c +2 -1
index af20420bd6c2..35f605936f1b 100644
@@ -415,7 +415,7 @@ static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
 	return ret;
 }
 
-static int sev_do_cmd(int cmd, void *data, int *psp_ret)
+int sev_do_cmd(int cmd, void *data, int *psp_ret)
 {
 	int rc;
 
@@ -425,6 +425,7 @@ static int sev_do_cmd(int cmd, void *data, int *psp_ret)
 
 	return rc;
 }
+EXPORT_SYMBOL_GPL(sev_do_cmd);
 
 static int __sev_init_locked(int *error)
 {
include/linux/psp-sev.h +17 -0
index 8cfe92e82743..46f61e3ae33b 100644
@@ -907,6 +907,20 @@ int sev_guest_df_flush(int *error);
  */
 int sev_guest_decommission(struct sev_data_decommission *data, int *error);
 
+/**
+ * sev_do_cmd - perform SEV command
+ *
+ * @error: SEV command return code
+ *
+ * Returns:
+ * 0 if the SEV successfully processed the command
+ * -%ENODEV    if the SEV device is not available
+ * -%ENOTSUPP  if the SEV does not support SEV
+ * -%ETIMEDOUT if the SEV command timed out
+ * -%EIO       if the SEV returned a non-zero return code
+ */
+int sev_do_cmd(int cmd, void *data, int *psp_ret);
+
 void *psp_copy_user_blob(u64 uaddr, u32 len);
 
 #else	/* !CONFIG_CRYPTO_DEV_SP_PSP */
@@ -924,6 +938,9 @@ sev_guest_deactivate(struct sev_data_deactivate *data, int *error) { return -ENO
 static inline int
 sev_guest_decommission(struct sev_data_decommission *data, int *error) { return -ENODEV; }
 
+static inline int
+sev_do_cmd(int cmd, void *data, int *psp_ret) { return -ENODEV; }
+
 static inline int
 sev_guest_activate(struct sev_data_activate *data, int *error) { return -ENODEV; }
 
Signature · 1 line
2.25.1
#103 lore raw

RFC PATCH v7 27/64 crypto: ccp: Introduce snp leaked pages list

In reply to Michael Roth

From: Ashish Kalra <[email protected]>

Pages are unsafe to be released back to the page-allocator, if they have been transitioned to firmware/guest state and can't be reclaimed or transitioned back to hypervisor/shared state. In this case add them to an internal leaked pages list to ensure that they are not freed or touched/accessed to cause fatal page faults.

---
 drivers/crypto/ccp/sev-dev.c | 28 ++++++++++++++++++++++++++++
 include/linux/psp-sev.h      |  8 ++++++++
 2 files changed, 36 insertions(+)
drivers/crypto/ccp/sev-dev.c +28 -0
index 35f605936f1b..eca4e59b0f44 100644
@@ -42,6 +42,12 @@
 static DEFINE_MUTEX(sev_cmd_mutex);
 static struct sev_misc_dev *misc_dev;
 
+/* list of pages which are leaked and cannot be reclaimed */
+static LIST_HEAD(snp_leaked_pages_list);
+static DEFINE_SPINLOCK(snp_leaked_pages_list_lock);
+
+static atomic_long_t snp_nr_leaked_pages = ATOMIC_LONG_INIT(0);
+
 static int psp_cmd_timeout = 100;
 module_param(psp_cmd_timeout, int, 0644);
 MODULE_PARM_DESC(psp_cmd_timeout, " default timeout value, in seconds, for PSP commands");
@@ -188,6 +194,28 @@ static int sev_cmd_buffer_len(int cmd)
 	return 0;
 }
 
+void snp_mark_pages_offline(unsigned long pfn, unsigned int npages)
+{
+	struct page *page = pfn_to_page(pfn);
+
+	WARN(1, "psc failed, pfn 0x%lx pages %d (marked offline)\n", pfn, npages);
+
+	spin_lock(&snp_leaked_pages_list_lock);
+	while (npages--) {
+		/*
+		 * Reuse the page's buddy list for chaining into the leaked
+		 * pages list. This page should not be on a free list currently
+		 * and is also unsafe to be added to a free list.
+		 */
+		list_add_tail(&page->buddy_list, &snp_leaked_pages_list);
+		sev_dump_rmpentry(pfn);
+		pfn++;
+	}
+	spin_unlock(&snp_leaked_pages_list_lock);
+	atomic_long_inc(&snp_nr_leaked_pages);
+}
+EXPORT_SYMBOL_GPL(snp_mark_pages_offline);
+
 static void *sev_fw_alloc(unsigned long len)
 {
 	struct page *page;
include/linux/psp-sev.h +8 -0
index 46f61e3ae33b..8edf5c548fbf 100644
@@ -923,6 +923,12 @@ int sev_do_cmd(int cmd, void *data, int *psp_ret);
 
 void *psp_copy_user_blob(u64 uaddr, u32 len);
 
+/**
+ * sev_mark_pages_offline - insert non-reclaimed firmware/guest pages
+ * into a leaked pages list.
+ */
+void snp_mark_pages_offline(unsigned long pfn, unsigned int npages);
+
 #else	/* !CONFIG_CRYPTO_DEV_SP_PSP */
 
 static inline int
@@ -951,6 +957,8 @@ sev_issue_cmd_external_user(struct file *filep, unsigned int id, void *data, int
 
 static inline void *psp_copy_user_blob(u64 __user uaddr, u32 len) { return ERR_PTR(-EINVAL); }
 
+void snp_mark_pages_offline(unsigned long pfn, unsigned int npages) {}
+
 #endif	/* CONFIG_CRYPTO_DEV_SP_PSP */
 
 #endif	/* __PSP_SEV_H__ */
Signature · 1 line
2.25.1
#104 lore raw

RFC PATCH v7 28/64 crypto: ccp: Handle the legacy TMR allocation when SNP is enabled

In reply to Michael Roth

From: Brijesh Singh <[email protected]>

The behavior and requirement for the SEV-legacy command is altered when the SNP firmware is in the INIT state. See SEV-SNP firmware specification for more details.

Allocate the Trusted Memory Region (TMR) as a 2mb sized/aligned region when SNP is enabled to satisfy new requirements for the SNP. Continue allocating a 1mb region for !SNP configuration.

While at it, provide API that can be used by others to allocate a page that can be used by the firmware. The immediate user for this API will be the KVM driver. The KVM driver to need to allocate a firmware context page during the guest creation. The context page need to be updated by the firmware. See the SEV-SNP specification for further details.

---
 drivers/crypto/ccp/sev-dev.c | 148 +++++++++++++++++++++++++++++++++--
 include/linux/psp-sev.h      |   9 +++
 2 files changed, 149 insertions(+), 8 deletions(-)
drivers/crypto/ccp/sev-dev.c +140 -8
index eca4e59b0f44..4c12e98a1219 100644
@@ -94,6 +94,13 @@ static void *sev_init_ex_buffer;
  */
 struct sev_data_range_list *snp_range_list;
 
+/* When SEV-SNP is enabled the TMR needs to be 2MB aligned and 2MB size. */
+#define SEV_SNP_ES_TMR_SIZE	(2 * 1024 * 1024)
+
+static size_t sev_es_tmr_size = SEV_ES_TMR_SIZE;
+
+static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret);
+
 static inline bool sev_version_greater_or_equal(u8 maj, u8 min)
 {
 	struct sev_device *sev = psp_master->sev_data;
@@ -216,11 +223,134 @@ void snp_mark_pages_offline(unsigned long pfn, unsigned int npages)
 }
 EXPORT_SYMBOL_GPL(snp_mark_pages_offline);
 
+static int snp_reclaim_pages(unsigned long paddr, unsigned int npages, bool locked)
+{
+	/* Cbit maybe set in the paddr */
+	unsigned long pfn = __sme_clr(paddr) >> PAGE_SHIFT;
+	int ret, err, i, n = 0;
+
+	if (!pfn_valid(pfn)) {
+		pr_err("%s: Invalid PFN %lx\n", __func__, pfn);
+		return 0;
+	}
+
+	for (i = 0; i < npages; i++, pfn++, n++) {
+		paddr = pfn << PAGE_SHIFT;
+
+		if (locked)
+			ret = __sev_do_cmd_locked(SEV_CMD_SNP_PAGE_RECLAIM, &paddr, &err);
+		else
+			ret = sev_do_cmd(SEV_CMD_SNP_PAGE_RECLAIM, &paddr, &err);
+
+		if (ret)
+			goto cleanup;
+
+		ret = rmp_make_shared(pfn, PG_LEVEL_4K);
+		if (ret)
+			goto cleanup;
+	}
+
+	return 0;
+
+cleanup:
+	/*
+	 * If failed to reclaim the page then page is no longer safe to
+	 * be release back to the system, leak it.
+	 */
+	snp_mark_pages_offline(pfn, npages - n);
+	return ret;
+}
+
+static int rmp_mark_pages_firmware(unsigned long paddr, unsigned int npages, bool locked)
+{
+	/* Cbit maybe set in the paddr */
+	unsigned long pfn = __sme_clr(paddr) >> PAGE_SHIFT;
+	int rc, n = 0, i;
+
+	for (i = 0; i < npages; i++, n++, pfn++) {
+		rc = rmp_make_private(pfn, 0, PG_LEVEL_4K, 0, true);
+		if (rc)
+			goto cleanup;
+	}
+
+	return 0;
+
+cleanup:
+	/*
+	 * Try unrolling the firmware state changes by
+	 * reclaiming the pages which were already changed to the
+	 * firmware state.
+	 */
+	snp_reclaim_pages(paddr, n, locked);
+
+	return rc;
+}
+
+static struct page *__snp_alloc_firmware_pages(gfp_t gfp_mask, int order, bool locked)
+{
+	unsigned long npages = 1ul << order, paddr;
+	struct sev_device *sev;
+	struct page *page;
+
+	if (!psp_master || !psp_master->sev_data)
+		return NULL;
+
+	page = alloc_pages(gfp_mask, order);
+	if (!page)
+		return NULL;
+
+	/* If SEV-SNP is initialized then add the page in RMP table. */
+	sev = psp_master->sev_data;
+	if (!sev->snp_initialized)
+		return page;
+
+	paddr = __pa((unsigned long)page_address(page));
+	if (rmp_mark_pages_firmware(paddr, npages, locked))
+		return NULL;
+
+	return page;
+}
+
+void *snp_alloc_firmware_page(gfp_t gfp_mask)
+{
+	struct page *page;
+
+	page = __snp_alloc_firmware_pages(gfp_mask, 0, false);
+
+	return page ? page_address(page) : NULL;
+}
+EXPORT_SYMBOL_GPL(snp_alloc_firmware_page);
+
+static void __snp_free_firmware_pages(struct page *page, int order, bool locked)
+{
+	struct sev_device *sev = psp_master->sev_data;
+	unsigned long paddr, npages = 1ul << order;
+
+	if (!page)
+		return;
+
+	paddr = __pa((unsigned long)page_address(page));
+	if (sev->snp_initialized &&
+	    snp_reclaim_pages(paddr, npages, locked))
+		return;
+
+	__free_pages(page, order);
+}
+
+void snp_free_firmware_page(void *addr)
+{
+	if (!addr)
+		return;
+
+	__snp_free_firmware_pages(virt_to_page(addr), 0, false);
+}
+EXPORT_SYMBOL_GPL(snp_free_firmware_page);
+
 static void *sev_fw_alloc(unsigned long len)
 {
 	struct page *page;
 
-	page = alloc_pages(GFP_KERNEL, get_order(len));
+	page = __snp_alloc_firmware_pages(GFP_KERNEL, get_order(len), false);
 	if (!page)
 		return NULL;
 
@@ -468,7 +598,7 @@ static int __sev_init_locked(int *error)
 		data.tmr_address = __pa(sev_es_tmr);
 
 		data.flags |= SEV_INIT_FLAGS_SEV_ES;
-		data.tmr_len = SEV_ES_TMR_SIZE;
+		data.tmr_len = sev_es_tmr_size;
 	}
 
 	return __sev_do_cmd_locked(SEV_CMD_INIT, &data, error);
@@ -491,7 +621,7 @@ static int __sev_init_ex_locked(int *error)
 		data.tmr_address = __pa(sev_es_tmr);
 
 		data.flags |= SEV_INIT_FLAGS_SEV_ES;
-		data.tmr_len = SEV_ES_TMR_SIZE;
+		data.tmr_len = sev_es_tmr_size;
 	}
 
 	return __sev_do_cmd_locked(SEV_CMD_INIT_EX, &data, error);
@@ -982,6 +1112,8 @@ static int __sev_snp_init_locked(int *error)
 	sev->snp_initialized = true;
 	dev_dbg(sev->dev, "SEV-SNP firmware initialized\n");
 
+	sev_es_tmr_size = SEV_SNP_ES_TMR_SIZE;
+
 	return rc;
 }
 
@@ -1499,8 +1631,9 @@ static void sev_firmware_shutdown(struct sev_device *sev)
 		/* The TMR area was encrypted, flush it from the cache */
 		wbinvd_on_all_cpus();
 
-		free_pages((unsigned long)sev_es_tmr,
-			   get_order(SEV_ES_TMR_SIZE));
+		__snp_free_firmware_pages(virt_to_page(sev_es_tmr),
+					  get_order(sev_es_tmr_size),
+					  false);
 		sev_es_tmr = NULL;
 	}
 
@@ -1511,8 +1644,7 @@ static void sev_firmware_shutdown(struct sev_device *sev)
 	}
 
 	if (snp_range_list) {
-		free_pages((unsigned long)snp_range_list,
-			   get_order(PAGE_SIZE));
+		snp_free_firmware_page(snp_range_list);
 		snp_range_list = NULL;
 	}
 
@@ -1593,7 +1725,7 @@ void sev_pci_init(void)
 	}
 
 	/* Obtain the TMR memory area for SEV-ES use */
-	sev_es_tmr = sev_fw_alloc(SEV_ES_TMR_SIZE);
+	sev_es_tmr = sev_fw_alloc(sev_es_tmr_size);
 	if (!sev_es_tmr)
 		dev_warn(sev->dev,
 			 "SEV: TMR allocation failed, SEV-ES support unavailable\n");
include/linux/psp-sev.h +9 -0
index 8edf5c548fbf..d19744807471 100644
@@ -922,6 +922,8 @@ int sev_guest_decommission(struct sev_data_decommission *data, int *error);
 int sev_do_cmd(int cmd, void *data, int *psp_ret);
 
 void *psp_copy_user_blob(u64 uaddr, u32 len);
+void *snp_alloc_firmware_page(gfp_t mask);
+void snp_free_firmware_page(void *addr);
 
 /**
  * sev_mark_pages_offline - insert non-reclaimed firmware/guest pages
@@ -959,6 +961,13 @@ static inline void *psp_copy_user_blob(u64 __user uaddr, u32 len) { return ERR_P
 
 void snp_mark_pages_offline(unsigned long pfn, unsigned int npages) {}
 
+static inline void *snp_alloc_firmware_page(gfp_t mask)
+{
+	return NULL;
+}
+
+static inline void snp_free_firmware_page(void *addr) { }
+
 #endif	/* CONFIG_CRYPTO_DEV_SP_PSP */
 
 #endif	/* __PSP_SEV_H__ */
Signature · 1 line
2.25.1
#105 lore raw

RFC PATCH v7 29/64 crypto: ccp: Handle the legacy SEV command when SNP is enabled

In reply to Michael Roth

From: Brijesh Singh <[email protected]>

The behavior of the SEV-legacy commands is altered when the SNP firmware is in the INIT state. When SNP is in INIT state, all the SEV-legacy commands that cause the firmware to write to memory must be in the firmware state before issuing the command..

A command buffer may contains a system physical address that the firmware may write to. There are two cases that need to be handled:

1) system physical address points to a guest memory
2) system physical address points to a host memory

To handle the case #1, change the page state to the firmware in the RMP table before issuing the command and restore the state to shared after the command completes.

For the case #2, use a bounce buffer to complete the request.

---
 drivers/crypto/ccp/sev-dev.c | 370 ++++++++++++++++++++++++++++++++++-
 drivers/crypto/ccp/sev-dev.h |  12 ++
 2 files changed, 372 insertions(+), 10 deletions(-)
drivers/crypto/ccp/sev-dev.c +360 -10
index 4c12e98a1219..5eb2e8f364d4 100644
@@ -286,6 +286,30 @@ static int rmp_mark_pages_firmware(unsigned long paddr, unsigned int npages, boo
 	return rc;
 }
 
+static int rmp_mark_pages_shared(unsigned long paddr, unsigned int npages)
+{
+	/* Cbit maybe set in the paddr */
+	unsigned long pfn = __sme_clr(paddr) >> PAGE_SHIFT;
+	int rc, n = 0, i;
+
+	for (i = 0; i < npages; i++, pfn++, n++) {
+		rc = rmp_make_shared(pfn, PG_LEVEL_4K);
+		if (rc)
+			goto cleanup;
+	}
+
+	return 0;
+
+cleanup:
+	/*
+	 * If failed to change the page state to shared, then its not safe
+	 * to release the page back to the system, leak it.
+	 */
+	snp_mark_pages_offline(pfn, npages - n);
+
+	return rc;
+}
+
 static struct page *__snp_alloc_firmware_pages(gfp_t gfp_mask, int order, bool locked)
 {
 	unsigned long npages = 1ul << order, paddr;
@@ -487,12 +511,295 @@ static int sev_write_init_ex_file_if_required(int cmd_id)
 	return sev_write_init_ex_file();
 }
 
+static int alloc_snp_host_map(struct sev_device *sev)
+{
+	struct page *page;
+	int i;
+
+	for (i = 0; i < MAX_SNP_HOST_MAP_BUFS; i++) {
+		struct snp_host_map *map = &sev->snp_host_map[i];
+
+		memset(map, 0, sizeof(*map));
+
+		page = alloc_pages(GFP_KERNEL_ACCOUNT, get_order(SEV_FW_BLOB_MAX_SIZE));
+		if (!page)
+			return -ENOMEM;
+
+		map->host = page_address(page);
+	}
+
+	return 0;
+}
+
+static void free_snp_host_map(struct sev_device *sev)
+{
+	int i;
+
+	for (i = 0; i < MAX_SNP_HOST_MAP_BUFS; i++) {
+		struct snp_host_map *map = &sev->snp_host_map[i];
+
+		if (map->host) {
+			__free_pages(virt_to_page(map->host), get_order(SEV_FW_BLOB_MAX_SIZE));
+			memset(map, 0, sizeof(*map));
+		}
+	}
+}
+
+static int map_firmware_writeable(u64 *paddr, u32 len, bool guest, struct snp_host_map *map)
+{
+	unsigned int npages = PAGE_ALIGN(len) >> PAGE_SHIFT;
+
+	map->active = false;
+
+	if (!paddr || !len)
+		return 0;
+
+	map->paddr = *paddr;
+	map->len = len;
+
+	/* If paddr points to a guest memory then change the page state to firmwware. */
+	if (guest) {
+		if (rmp_mark_pages_firmware(*paddr, npages, true))
+			return -EFAULT;
+
+		goto done;
+	}
+
+	if (!map->host)
+		return -ENOMEM;
+
+	/* Check if the pre-allocated buffer can be used to fullfil the request. */
+	if (len > SEV_FW_BLOB_MAX_SIZE)
+		return -EINVAL;
+
+	/* Transition the pre-allocated buffer to the firmware state. */
+	if (rmp_mark_pages_firmware(__pa(map->host), npages, true))
+		return -EFAULT;
+
+	/* Set the paddr to use pre-allocated firmware buffer */
+	*paddr = __psp_pa(map->host);
+
+done:
+	map->active = true;
+	return 0;
+}
+
+static int unmap_firmware_writeable(u64 *paddr, u32 len, bool guest, struct snp_host_map *map)
+{
+	unsigned int npages = PAGE_ALIGN(len) >> PAGE_SHIFT;
+
+	if (!map->active)
+		return 0;
+
+	/* If paddr points to a guest memory then restore the page state to hypervisor. */
+	if (guest) {
+		if (snp_reclaim_pages(*paddr, npages, true))
+			return -EFAULT;
+
+		goto done;
+	}
+
+	/*
+	 * Transition the pre-allocated buffer to hypervisor state before the access.
+	 *
+	 * This is because while changing the page state to firmware, the kernel unmaps
+	 * the pages from the direct map, and to restore the direct map the pages must
+	 * be transitioned back to the shared state.
+	 */
+	if (snp_reclaim_pages(__pa(map->host), npages, true))
+		return -EFAULT;
+
+	/* Copy the response data firmware buffer to the callers buffer. */
+	memcpy(__va(__sme_clr(map->paddr)), map->host, min_t(size_t, len, map->len));
+	*paddr = map->paddr;
+
+done:
+	map->active = false;
+	return 0;
+}
+
+static bool sev_legacy_cmd_buf_writable(int cmd)
+{
+	switch (cmd) {
+	case SEV_CMD_PLATFORM_STATUS:
+	case SEV_CMD_GUEST_STATUS:
+	case SEV_CMD_LAUNCH_START:
+	case SEV_CMD_RECEIVE_START:
+	case SEV_CMD_LAUNCH_MEASURE:
+	case SEV_CMD_SEND_START:
+	case SEV_CMD_SEND_UPDATE_DATA:
+	case SEV_CMD_SEND_UPDATE_VMSA:
+	case SEV_CMD_PEK_CSR:
+	case SEV_CMD_PDH_CERT_EXPORT:
+	case SEV_CMD_GET_ID:
+	case SEV_CMD_ATTESTATION_REPORT:
+		return true;
+	default:
+		return false;
+	}
+}
+
+#define prep_buffer(name, addr, len, guest, map) \
+	func(&((typeof(name *))cmd_buf)->addr, ((typeof(name *))cmd_buf)->len, guest, map)
+
+static int __snp_cmd_buf_copy(int cmd, void *cmd_buf, bool to_fw, int fw_err)
+{
+	int (*func)(u64 *paddr, u32 len, bool guest, struct snp_host_map *map);
+	struct sev_device *sev = psp_master->sev_data;
+	bool from_fw = !to_fw;
+
+	/*
+	 * After the command is completed, change the command buffer memory to
+	 * hypervisor state.
+	 *
+	 * The immutable bit is automatically cleared by the firmware, so
+	 * no not need to reclaim the page.
+	 */
+	if (from_fw && sev_legacy_cmd_buf_writable(cmd)) {
+		if (rmp_mark_pages_shared(__pa(cmd_buf), 1))
+			return -EFAULT;
+
+		/* No need to go further if firmware failed to execute command. */
+		if (fw_err)
+			return 0;
+	}
+
+	if (to_fw)
+		func = map_firmware_writeable;
+	else
+		func = unmap_firmware_writeable;
+
+	/*
+	 * A command buffer may contains a system physical address. If the address
+	 * points to a host memory then use an intermediate firmware page otherwise
+	 * change the page state in the RMP table.
+	 */
+	switch (cmd) {
+	case SEV_CMD_PDH_CERT_EXPORT:
+		if (prep_buffer(struct sev_data_pdh_cert_export, pdh_cert_address,
+				pdh_cert_len, false, &sev->snp_host_map[0]))
+			goto err;
+		if (prep_buffer(struct sev_data_pdh_cert_export, cert_chain_address,
+				cert_chain_len, false, &sev->snp_host_map[1]))
+			goto err;
+		break;
+	case SEV_CMD_GET_ID:
+		if (prep_buffer(struct sev_data_get_id, address, len,
+				false, &sev->snp_host_map[0]))
+			goto err;
+		break;
+	case SEV_CMD_PEK_CSR:
+		if (prep_buffer(struct sev_data_pek_csr, address, len,
+				false, &sev->snp_host_map[0]))
+			goto err;
+		break;
+	case SEV_CMD_LAUNCH_UPDATE_DATA:
+		if (prep_buffer(struct sev_data_launch_update_data, address, len,
+				true, &sev->snp_host_map[0]))
+			goto err;
+		break;
+	case SEV_CMD_LAUNCH_UPDATE_VMSA:
+		if (prep_buffer(struct sev_data_launch_update_vmsa, address, len,
+				true, &sev->snp_host_map[0]))
+			goto err;
+		break;
+	case SEV_CMD_LAUNCH_MEASURE:
+		if (prep_buffer(struct sev_data_launch_measure, address, len,
+				false, &sev->snp_host_map[0]))
+			goto err;
+		break;
+	case SEV_CMD_LAUNCH_UPDATE_SECRET:
+		if (prep_buffer(struct sev_data_launch_secret, guest_address, guest_len,
+				true, &sev->snp_host_map[0]))
+			goto err;
+		break;
+	case SEV_CMD_DBG_DECRYPT:
+		if (prep_buffer(struct sev_data_dbg, dst_addr, len, false,
+				&sev->snp_host_map[0]))
+			goto err;
+		break;
+	case SEV_CMD_DBG_ENCRYPT:
+		if (prep_buffer(struct sev_data_dbg, dst_addr, len, true,
+				&sev->snp_host_map[0]))
+			goto err;
+		break;
+	case SEV_CMD_ATTESTATION_REPORT:
+		if (prep_buffer(struct sev_data_attestation_report, address, len,
+				false, &sev->snp_host_map[0]))
+			goto err;
+		break;
+	case SEV_CMD_SEND_START:
+		if (prep_buffer(struct sev_data_send_start, session_address,
+				session_len, false, &sev->snp_host_map[0]))
+			goto err;
+		break;
+	case SEV_CMD_SEND_UPDATE_DATA:
+		if (prep_buffer(struct sev_data_send_update_data, hdr_address, hdr_len,
+				false, &sev->snp_host_map[0]))
+			goto err;
+		if (prep_buffer(struct sev_data_send_update_data, trans_address,
+				trans_len, false, &sev->snp_host_map[1]))
+			goto err;
+		break;
+	case SEV_CMD_SEND_UPDATE_VMSA:
+		if (prep_buffer(struct sev_data_send_update_vmsa, hdr_address, hdr_len,
+				false, &sev->snp_host_map[0]))
+			goto err;
+		if (prep_buffer(struct sev_data_send_update_vmsa, trans_address,
+				trans_len, false, &sev->snp_host_map[1]))
+			goto err;
+		break;
+	case SEV_CMD_RECEIVE_UPDATE_DATA:
+		if (prep_buffer(struct sev_data_receive_update_data, guest_address,
+				guest_len, true, &sev->snp_host_map[0]))
+			goto err;
+		break;
+	case SEV_CMD_RECEIVE_UPDATE_VMSA:
+		if (prep_buffer(struct sev_data_receive_update_vmsa, guest_address,
+				guest_len, true, &sev->snp_host_map[0]))
+			goto err;
+		break;
+	default:
+		break;
+	}
+
+	/* The command buffer need to be in the firmware state. */
+	if (to_fw && sev_legacy_cmd_buf_writable(cmd)) {
+		if (rmp_mark_pages_firmware(__pa(cmd_buf), 1, true))
+			return -EFAULT;
+	}
+
+	return 0;
+
+err:
+	return -EINVAL;
+}
+
+static inline bool need_firmware_copy(int cmd)
+{
+	struct sev_device *sev = psp_master->sev_data;
+
+	/* After SNP is INIT'ed, the behavior of legacy SEV command is changed. */
+	return ((cmd < SEV_CMD_SNP_INIT) && sev->snp_initialized) ? true : false;
+}
+
+static int snp_aware_copy_to_firmware(int cmd, void *data)
+{
+	return __snp_cmd_buf_copy(cmd, data, true, 0);
+}
+
+static int snp_aware_copy_from_firmware(int cmd, void *data, int fw_err)
+{
+	return __snp_cmd_buf_copy(cmd, data, false, fw_err);
+}
+
 static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
 {
 	struct psp_device *psp = psp_master;
 	struct sev_device *sev;
 	unsigned int phys_lsb, phys_msb;
 	unsigned int reg, ret = 0;
+	void *cmd_buf;
 	int buf_len;
 
 	if (!psp || !psp->sev_data)
@@ -512,12 +819,28 @@ static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
 	 * work for some memory, e.g. vmalloc'd addresses, and @data may not be
 	 * physically contiguous.
 	 */
-	if (data)
-		memcpy(sev->cmd_buf, data, buf_len);
+	if (data) {
+		if (sev->cmd_buf_active > 2)
+			return -EBUSY;
+
+		cmd_buf = sev->cmd_buf_active ? sev->cmd_buf_backup : sev->cmd_buf;
+
+		memcpy(cmd_buf, data, buf_len);
+		sev->cmd_buf_active++;
+
+		/*
+		 * The behavior of the SEV-legacy commands is altered when the
+		 * SNP firmware is in the INIT state.
+		 */
+		if (need_firmware_copy(cmd) && snp_aware_copy_to_firmware(cmd, sev->cmd_buf))
+			return -EFAULT;
+	} else {
+		cmd_buf = sev->cmd_buf;
+	}
 
 	/* Get the physical address of the command buffer */
-	phys_lsb = data ? lower_32_bits(__psp_pa(sev->cmd_buf)) : 0;
-	phys_msb = data ? upper_32_bits(__psp_pa(sev->cmd_buf)) : 0;
+	phys_lsb = data ? lower_32_bits(__psp_pa(cmd_buf)) : 0;
+	phys_msb = data ? upper_32_bits(__psp_pa(cmd_buf)) : 0;
 
 	dev_dbg(sev->dev, "sev command id %#x buffer 0x%08x%08x timeout %us\n",
 		cmd, phys_msb, phys_lsb, psp_timeout);
@@ -560,15 +883,24 @@ static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
 		ret = sev_write_init_ex_file_if_required(cmd);
 	}
 
-	print_hex_dump_debug("(out): ", DUMP_PREFIX_OFFSET, 16, 2, data,
-			     buf_len, false);
-
 	/*
 	 * Copy potential output from the PSP back to data.  Do this even on
 	 * failure in case the caller wants to glean something from the error.
 	 */
-	if (data)
-		memcpy(data, sev->cmd_buf, buf_len);
+	if (data) {
+		/*
+		 * Restore the page state after the command completes.
+		 */
+		if (need_firmware_copy(cmd) &&
+		    snp_aware_copy_from_firmware(cmd, cmd_buf, ret))
+			return -EFAULT;
+
+		memcpy(data, cmd_buf, buf_len);
+		sev->cmd_buf_active--;
+	}
+
+	print_hex_dump_debug("(out): ", DUMP_PREFIX_OFFSET, 16, 2, data,
+			     buf_len, false);
 
 	return ret;
 }
@@ -1579,10 +1911,12 @@ int sev_dev_init(struct psp_device *psp)
 	if (!sev)
 		goto e_err;
 
-	sev->cmd_buf = (void *)devm_get_free_pages(dev, GFP_KERNEL, 0);
+	sev->cmd_buf = (void *)devm_get_free_pages(dev, GFP_KERNEL, 1);
 	if (!sev->cmd_buf)
 		goto e_sev;
 
+	sev->cmd_buf_backup = (uint8_t *)sev->cmd_buf + PAGE_SIZE;
+
 	psp->sev_data = sev;
 
 	sev->dev = dev;
@@ -1648,6 +1982,12 @@ static void sev_firmware_shutdown(struct sev_device *sev)
 		snp_range_list = NULL;
 	}
 
+	/*
+	 * The host map need to clear the immutable bit so it must be free'd before the
+	 * SNP firmware shutdown.
+	 */
+	free_snp_host_map(sev);
+
 	sev_snp_shutdown(&error);
 }
 
@@ -1722,6 +2062,14 @@ void sev_pci_init(void)
 				dev_err(sev->dev, "SEV-SNP: failed to INIT error %#x\n", error);
 			}
 		}
+
+		/*
+		 * Allocate the intermediate buffers used for the legacy command handling.
+		 */
+		if (alloc_snp_host_map(sev)) {
+			dev_notice(sev->dev, "Failed to alloc host map (disabling legacy SEV)\n");
+			goto skip_legacy;
+		}
 	}
 
 	/* Obtain the TMR memory area for SEV-ES use */
@@ -1739,12 +2087,14 @@ void sev_pci_init(void)
 		dev_err(sev->dev, "SEV: failed to INIT error %#x, rc %d\n",
 			error, rc);
 
+skip_legacy:
 	dev_info(sev->dev, "SEV%s API:%d.%d build:%d\n", sev->snp_initialized ?
 		"-SNP" : "", sev->api_major, sev->api_minor, sev->build);
 
 	return;
 
 err:
+	free_snp_host_map(sev);
 	psp_master->sev_data = NULL;
 }
 
drivers/crypto/ccp/sev-dev.h +12 -0
index 34767657beb5..19d79f9d4212 100644
@@ -29,11 +29,20 @@
 #define SEV_CMDRESP_CMD_SHIFT		16
 #define SEV_CMDRESP_IOC			BIT(0)
 
+#define MAX_SNP_HOST_MAP_BUFS		2
+
 struct sev_misc_dev {
 	struct kref refcount;
 	struct miscdevice misc;
 };
 
+struct snp_host_map {
+	u64 paddr;
+	u32 len;
+	void *host;
+	bool active;
+};
+
 struct sev_device {
 	struct device *dev;
 	struct psp_device *psp;
@@ -52,8 +61,11 @@ struct sev_device {
 	u8 build;
 
 	void *cmd_buf;
+	void *cmd_buf_backup;
+	int cmd_buf_active;
 
 	bool snp_initialized;
+	struct snp_host_map snp_host_map[MAX_SNP_HOST_MAP_BUFS];
 };
 
 int sev_dev_init(struct psp_device *psp);
Signature · 1 line
2.25.1
#106 lore raw

Re: [PATCH RFC v7 29/64] crypto: ccp: Handle the legacy SEV command when SNP is enabled

Quoted 204 lines · Michael Roth

From: Brijesh Singh <[email protected]>

The behavior of the SEV-legacy commands is altered when the SNP firmware is in the INIT state. When SNP is in INIT state, all the SEV-legacy commands that cause the firmware to write to memory must be in the firmware state before issuing the command..

A command buffer may contains a system physical address that the firmware may write to. There are two cases that need to be handled:

1) system physical address points to a guest memory
2) system physical address points to a host memory

To handle the case #1, change the page state to the firmware in the RMP table before issuing the command and restore the state to shared after the command completes.

For the case #2, use a bounce buffer to complete the request.

---
 drivers/crypto/ccp/sev-dev.c | 370 ++++++++++++++++++++++++++++++++++-
 drivers/crypto/ccp/sev-dev.h |  12 ++
 2 files changed, 372 insertions(+), 10 deletions(-)
drivers/crypto/ccp/sev-dev.c +161 -0
index 4c12e98a1219..5eb2e8f364d4 100644
@@ -286,6 +286,30 @@ static int rmp_mark_pages_firmware(unsigned long paddr, unsigned int npages, boo
        return rc;
 }
 
+static int rmp_mark_pages_shared(unsigned long paddr, unsigned int npages)
+{
+       /* Cbit maybe set in the paddr */
+       unsigned long pfn = __sme_clr(paddr) >> PAGE_SHIFT;
+       int rc, n = 0, i;
+
+       for (i = 0; i < npages; i++, pfn++, n++) {
+               rc = rmp_make_shared(pfn, PG_LEVEL_4K);
+               if (rc)
+                       goto cleanup;
+       }
+
+       return 0;
+
+cleanup:
+       /*
+        * If failed to change the page state to shared, then its not safe
+        * to release the page back to the system, leak it.
+        */
+       snp_mark_pages_offline(pfn, npages - n);
+
+       return rc;
+}
+
 static struct page *__snp_alloc_firmware_pages(gfp_t gfp_mask, int order, bool locked)
 {
        unsigned long npages = 1ul << order, paddr;
@@ -487,12 +511,295 @@ static int sev_write_init_ex_file_if_required(int cmd_id)
        return sev_write_init_ex_file();
 }
 
+static int alloc_snp_host_map(struct sev_device *sev)
+{
+       struct page *page;
+       int i;
+
+       for (i = 0; i < MAX_SNP_HOST_MAP_BUFS; i++) {
+               struct snp_host_map *map = &sev->snp_host_map[i];
+
+               memset(map, 0, sizeof(*map));
+
+               page = alloc_pages(GFP_KERNEL_ACCOUNT, get_order(SEV_FW_BLOB_MAX_SIZE));
+               if (!page)
+                       return -ENOMEM;
+
+               map->host = page_address(page);
+       }
+
+       return 0;
+}
+
+static void free_snp_host_map(struct sev_device *sev)
+{
+       int i;
+
+       for (i = 0; i < MAX_SNP_HOST_MAP_BUFS; i++) {
+               struct snp_host_map *map = &sev->snp_host_map[i];
+
+               if (map->host) {
+                       __free_pages(virt_to_page(map->host), get_order(SEV_FW_BLOB_MAX_SIZE));
+                       memset(map, 0, sizeof(*map));
+               }
+       }
+}
+
+static int map_firmware_writeable(u64 *paddr, u32 len, bool guest, struct snp_host_map *map)
+{
+       unsigned int npages = PAGE_ALIGN(len) >> PAGE_SHIFT;
+
+       map->active = false;
+
+       if (!paddr || !len)
+               return 0;
+
+       map->paddr = *paddr;
+       map->len = len;
+
+       /* If paddr points to a guest memory then change the page state to firmwware. */
+       if (guest) {
+               if (rmp_mark_pages_firmware(*paddr, npages, true))
+                       return -EFAULT;
+
+               goto done;
+       }
+
+       if (!map->host)
+               return -ENOMEM;
+
+       /* Check if the pre-allocated buffer can be used to fullfil the request. */
+       if (len > SEV_FW_BLOB_MAX_SIZE)
+               return -EINVAL;
+
+       /* Transition the pre-allocated buffer to the firmware state. */
+       if (rmp_mark_pages_firmware(__pa(map->host), npages, true))
+               return -EFAULT;
+
+       /* Set the paddr to use pre-allocated firmware buffer */
+       *paddr = __psp_pa(map->host);
+
+done:
+       map->active = true;
+       return 0;
+}
+
+static int unmap_firmware_writeable(u64 *paddr, u32 len, bool guest, struct snp_host_map *map)
+{
+       unsigned int npages = PAGE_ALIGN(len) >> PAGE_SHIFT;
+
+       if (!map->active)
+               return 0;
+
+       /* If paddr points to a guest memory then restore the page state to hypervisor. */
+       if (guest) {
+               if (snp_reclaim_pages(*paddr, npages, true))
+                       return -EFAULT;
+
+               goto done;
+       }
+
+       /*
+        * Transition the pre-allocated buffer to hypervisor state before the access.
+        *
+        * This is because while changing the page state to firmware, the kernel unmaps
+        * the pages from the direct map, and to restore the direct map the pages must
+        * be transitioned back to the shared state.
+        */
+       if (snp_reclaim_pages(__pa(map->host), npages, true))
+               return -EFAULT;
+
+       /* Copy the response data firmware buffer to the callers buffer. */
+       memcpy(__va(__sme_clr(map->paddr)), map->host, min_t(size_t, len, map->len));
+       *paddr = map->paddr;
+
+done:
+       map->active = false;
+       return 0;
+}
+
+static bool sev_legacy_cmd_buf_writable(int cmd)
+{
+       switch (cmd) {
+       case SEV_CMD_PLATFORM_STATUS:
+       case SEV_CMD_GUEST_STATUS:
+       case SEV_CMD_LAUNCH_START:
+       case SEV_CMD_RECEIVE_START:
+       case SEV_CMD_LAUNCH_MEASURE:
+       case SEV_CMD_SEND_START:
+       case SEV_CMD_SEND_UPDATE_DATA:
+       case SEV_CMD_SEND_UPDATE_VMSA:
+       case SEV_CMD_PEK_CSR:
+       case SEV_CMD_PDH_CERT_EXPORT:
+       case SEV_CMD_GET_ID:
+       case SEV_CMD_ATTESTATION_REPORT:
+               return true;
+       default:
+               return false;
+       }
+}
+
+#define prep_buffer(name, addr, len, guest, map) \
+       func(&((typeof(name *))cmd_buf)->addr, ((typeof(name *))cmd_buf)->len, guest, map)
+
+static int __snp_cmd_buf_copy(int cmd, void *cmd_buf, bool to_fw, int fw_err)
+{
+       int (*func)(u64 *paddr, u32 len, bool guest, struct snp_host_map *map);
+       struct sev_device *sev = psp_master->sev_data;
+       bool from_fw = !to_fw;
+

drivers/crypto/ccp/sev-dev.c static int sev_write_init_ex_file_if_required(int cmd_id) · line 660

Quoted 10 lines
drivers/crypto/ccp/sev-dev.c +10 -0
+       /*
+        * After the command is completed, change the command buffer memory to
+        * hypervisor state.
+        *
+        * The immutable bit is automatically cleared by the firmware, so
+        * no not need to reclaim the page.
+        */
+       if (from_fw && sev_legacy_cmd_buf_writable(cmd)) {
+               if (rmp_mark_pages_shared(__pa(cmd_buf), 1))
+                       return -EFAULT;

If we return here, we will skip calling unmap_firmware_writeable and we will leak some pages in firmware state.

Quoted 99 lines
drivers/crypto/ccp/sev-dev.c +99 -0
+
+               /* No need to go further if firmware failed to execute command. */
+               if (fw_err)
+                       return 0;
+       }
+
+       if (to_fw)
+               func = map_firmware_writeable;
+       else
+               func = unmap_firmware_writeable;
+
+       /*
+        * A command buffer may contains a system physical address. If the address
+        * points to a host memory then use an intermediate firmware page otherwise
+        * change the page state in the RMP table.
+        */
+       switch (cmd) {
+       case SEV_CMD_PDH_CERT_EXPORT:
+               if (prep_buffer(struct sev_data_pdh_cert_export, pdh_cert_address,
+                               pdh_cert_len, false, &sev->snp_host_map[0]))
+                       goto err;
+               if (prep_buffer(struct sev_data_pdh_cert_export, cert_chain_address,
+                               cert_chain_len, false, &sev->snp_host_map[1]))
+                       goto err;
+               break;
+       case SEV_CMD_GET_ID:
+               if (prep_buffer(struct sev_data_get_id, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_PEK_CSR:
+               if (prep_buffer(struct sev_data_pek_csr, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_UPDATE_DATA:
+               if (prep_buffer(struct sev_data_launch_update_data, address, len,
+                               true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_UPDATE_VMSA:
+               if (prep_buffer(struct sev_data_launch_update_vmsa, address, len,
+                               true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_MEASURE:
+               if (prep_buffer(struct sev_data_launch_measure, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_UPDATE_SECRET:
+               if (prep_buffer(struct sev_data_launch_secret, guest_address, guest_len,
+                               true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_DBG_DECRYPT:
+               if (prep_buffer(struct sev_data_dbg, dst_addr, len, false,
+                               &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_DBG_ENCRYPT:
+               if (prep_buffer(struct sev_data_dbg, dst_addr, len, true,
+                               &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_ATTESTATION_REPORT:
+               if (prep_buffer(struct sev_data_attestation_report, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_SEND_START:
+               if (prep_buffer(struct sev_data_send_start, session_address,
+                               session_len, false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_SEND_UPDATE_DATA:
+               if (prep_buffer(struct sev_data_send_update_data, hdr_address, hdr_len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               if (prep_buffer(struct sev_data_send_update_data, trans_address,
+                               trans_len, false, &sev->snp_host_map[1]))
+                       goto err;
+               break;
+       case SEV_CMD_SEND_UPDATE_VMSA:
+               if (prep_buffer(struct sev_data_send_update_vmsa, hdr_address, hdr_len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               if (prep_buffer(struct sev_data_send_update_vmsa, trans_address,
+                               trans_len, false, &sev->snp_host_map[1]))
+                       goto err;
+               break;
+       case SEV_CMD_RECEIVE_UPDATE_DATA:
+               if (prep_buffer(struct sev_data_receive_update_data, guest_address,
+                               guest_len, true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_RECEIVE_UPDATE_VMSA:
+               if (prep_buffer(struct sev_data_receive_update_vmsa, guest_address,
+                               guest_len, true, &sev->snp_host_map[0]))

drivers/crypto/ccp/sev-dev.c static int sev_write_init_ex_file_if_required(int cmd_id) · line 769

Quoted 10 lines
drivers/crypto/ccp/sev-dev.c +10 -0
+                       goto err;
+               break;
+       default:
+               break;
+       }
+
+       /* The command buffer need to be in the firmware state. */
+       if (to_fw && sev_legacy_cmd_buf_writable(cmd)) {
+               if (rmp_mark_pages_firmware(__pa(cmd_buf), 1, true))
+                       return -EFAULT;

This function moves two separate pages to firmware state. First calling map_firmware_writeable and second calling rmp_mark_pages_firmware for cmd_buf.
In case rmp_mark_pages_firmware fails for cmd_buf, the page which has already moved to firmware state in map_firmware_writeable should be reclaimed.
This is a problem especially if we leak a guest owned page in firmware state. Since this is used only by legacy SEV VMs, these leaked pages will never be reclaimed back when destroying these VMs.

Quoted 197 lines
drivers/crypto/ccp/sev-dev.c +80 -10
+       }
+
+       return 0;
+
+err:
+       return -EINVAL;
+}
+
+static inline bool need_firmware_copy(int cmd)
+{
+       struct sev_device *sev = psp_master->sev_data;
+
+       /* After SNP is INIT'ed, the behavior of legacy SEV command is changed. */
+       return ((cmd < SEV_CMD_SNP_INIT) && sev->snp_initialized) ? true : false;
+}
+
+static int snp_aware_copy_to_firmware(int cmd, void *data)
+{
+       return __snp_cmd_buf_copy(cmd, data, true, 0);
+}
+
+static int snp_aware_copy_from_firmware(int cmd, void *data, int fw_err)
+{
+       return __snp_cmd_buf_copy(cmd, data, false, fw_err);
+}
+
 static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
 {
        struct psp_device *psp = psp_master;
        struct sev_device *sev;
        unsigned int phys_lsb, phys_msb;
        unsigned int reg, ret = 0;
+       void *cmd_buf;
        int buf_len;
 
        if (!psp || !psp->sev_data)
@@ -512,12 +819,28 @@ static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
         * work for some memory, e.g. vmalloc'd addresses, and @data may not be
         * physically contiguous.
         */
-       if (data)
-               memcpy(sev->cmd_buf, data, buf_len);
+       if (data) {
+               if (sev->cmd_buf_active > 2)
+                       return -EBUSY;
+
+               cmd_buf = sev->cmd_buf_active ? sev->cmd_buf_backup : sev->cmd_buf;
+
+               memcpy(cmd_buf, data, buf_len);
+               sev->cmd_buf_active++;
+
+               /*
+                * The behavior of the SEV-legacy commands is altered when the
+                * SNP firmware is in the INIT state.
+                */
+               if (need_firmware_copy(cmd) && snp_aware_copy_to_firmware(cmd, sev->cmd_buf))
+                       return -EFAULT;
+       } else {
+               cmd_buf = sev->cmd_buf;
+       }
 
        /* Get the physical address of the command buffer */
-       phys_lsb = data ? lower_32_bits(__psp_pa(sev->cmd_buf)) : 0;
-       phys_msb = data ? upper_32_bits(__psp_pa(sev->cmd_buf)) : 0;
+       phys_lsb = data ? lower_32_bits(__psp_pa(cmd_buf)) : 0;
+       phys_msb = data ? upper_32_bits(__psp_pa(cmd_buf)) : 0;
 
        dev_dbg(sev->dev, "sev command id %#x buffer 0x%08x%08x timeout %us\n",
                cmd, phys_msb, phys_lsb, psp_timeout);
@@ -560,15 +883,24 @@ static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
                ret = sev_write_init_ex_file_if_required(cmd);
        }
 
-       print_hex_dump_debug("(out): ", DUMP_PREFIX_OFFSET, 16, 2, data,
-                            buf_len, false);
-
        /*
         * Copy potential output from the PSP back to data.  Do this even on
         * failure in case the caller wants to glean something from the error.
         */
-       if (data)
-               memcpy(data, sev->cmd_buf, buf_len);
+       if (data) {
+               /*
+                * Restore the page state after the command completes.
+                */
+               if (need_firmware_copy(cmd) &&
+                   snp_aware_copy_from_firmware(cmd, cmd_buf, ret))
+                       return -EFAULT;
+
+               memcpy(data, cmd_buf, buf_len);
+               sev->cmd_buf_active--;
+       }
+
+       print_hex_dump_debug("(out): ", DUMP_PREFIX_OFFSET, 16, 2, data,
+                            buf_len, false);
 
        return ret;
 }
@@ -1579,10 +1911,12 @@ int sev_dev_init(struct psp_device *psp)
        if (!sev)
                goto e_err;
 
-       sev->cmd_buf = (void *)devm_get_free_pages(dev, GFP_KERNEL, 0);
+       sev->cmd_buf = (void *)devm_get_free_pages(dev, GFP_KERNEL, 1);
        if (!sev->cmd_buf)
                goto e_sev;
 
+       sev->cmd_buf_backup = (uint8_t *)sev->cmd_buf + PAGE_SIZE;
+
        psp->sev_data = sev;
 
        sev->dev = dev;
@@ -1648,6 +1982,12 @@ static void sev_firmware_shutdown(struct sev_device *sev)
                snp_range_list = NULL;
        }
 
+       /*
+        * The host map need to clear the immutable bit so it must be free'd before the
+        * SNP firmware shutdown.
+        */
+       free_snp_host_map(sev);
+
        sev_snp_shutdown(&error);
 }
 
@@ -1722,6 +2062,14 @@ void sev_pci_init(void)
                                dev_err(sev->dev, "SEV-SNP: failed to INIT error %#x\n", error);
                        }
                }
+
+               /*
+                * Allocate the intermediate buffers used for the legacy command handling.
+                */
+               if (alloc_snp_host_map(sev)) {
+                       dev_notice(sev->dev, "Failed to alloc host map (disabling legacy SEV)\n");
+                       goto skip_legacy;
+               }
        }
 
        /* Obtain the TMR memory area for SEV-ES use */
@@ -1739,12 +2087,14 @@ void sev_pci_init(void)
                dev_err(sev->dev, "SEV: failed to INIT error %#x, rc %d\n",
                        error, rc);
 
+skip_legacy:
        dev_info(sev->dev, "SEV%s API:%d.%d build:%d\n", sev->snp_initialized ?
                "-SNP" : "", sev->api_major, sev->api_minor, sev->build);
 
        return;
 
 err:
+       free_snp_host_map(sev);
        psp_master->sev_data = NULL;
 }
 
drivers/crypto/ccp/sev-dev.h +12 -1
index 34767657beb5..19d79f9d4212 100644
@@ -29,11 +29,20 @@
 #define SEV_CMDRESP_CMD_SHIFT          16
 #define SEV_CMDRESP_IOC                        BIT(0)
 
+#define MAX_SNP_HOST_MAP_BUFS          2
+
 struct sev_misc_dev {
        struct kref refcount;
        struct miscdevice misc;
 };
 
+struct snp_host_map {
+       u64 paddr;
+       u32 len;
+       void *host;
+       bool active;
+};
+
 struct sev_device {
        struct device *dev;
        struct psp_device *psp;
@@ -52,8 +61,11 @@ struct sev_device {
        u8 build;
 
        void *cmd_buf;
+       void *cmd_buf_backup;
+       int cmd_buf_active;
 
        bool snp_initialized;
+       struct snp_host_map snp_host_map[MAX_SNP_HOST_MAP_BUFS];
 };
 
 int sev_dev_init(struct psp_device *psp);
--

2.25.1

maintainer
#107 lore raw

Re: [PATCH RFC v7 29/64] crypto: ccp: Handle the legacy SEV command when SNP is enabled

Hello Alper,

Quoted 210 lines · Alper Gun
Quoted 209 lines · Michael Roth

From: Brijesh Singh <[email protected]>

The behavior of the SEV-legacy commands is altered when the SNP firmware is in the INIT state. When SNP is in INIT state, all the SEV-legacy commands that cause the firmware to write to memory must be in the firmware state before issuing the command..

A command buffer may contains a system physical address that the firmware may write to. There are two cases that need to be handled:

1) system physical address points to a guest memory
2) system physical address points to a host memory

To handle the case #1, change the page state to the firmware in the RMP table before issuing the command and restore the state to shared after the command completes.

For the case #2, use a bounce buffer to complete the request.

---
  drivers/crypto/ccp/sev-dev.c | 370 ++++++++++++++++++++++++++++++++++-
  drivers/crypto/ccp/sev-dev.h |  12 ++
  2 files changed, 372 insertions(+), 10 deletions(-)
drivers/crypto/ccp/sev-dev.c +166 -0
index 4c12e98a1219..5eb2e8f364d4 100644
@@ -286,6 +286,30 @@ static int rmp_mark_pages_firmware(unsigned long paddr, unsigned int npages, boo
         return rc;
  }
 
+static int rmp_mark_pages_shared(unsigned long paddr, unsigned int npages)
+{
+       /* Cbit maybe set in the paddr */
+       unsigned long pfn = __sme_clr(paddr) >> PAGE_SHIFT;
+       int rc, n = 0, i;
+
+       for (i = 0; i < npages; i++, pfn++, n++) {
+               rc = rmp_make_shared(pfn, PG_LEVEL_4K);
+               if (rc)
+                       goto cleanup;
+       }
+
+       return 0;
+
+cleanup:
+       /*
+        * If failed to change the page state to shared, then its not safe
+        * to release the page back to the system, leak it.
+        */
+       snp_mark_pages_offline(pfn, npages - n);
+
+       return rc;
+}
+
  static struct page *__snp_alloc_firmware_pages(gfp_t gfp_mask, int order, bool locked)
  {
         unsigned long npages = 1ul << order, paddr;
@@ -487,12 +511,295 @@ static int sev_write_init_ex_file_if_required(int cmd_id)
         return sev_write_init_ex_file();
  }
 
+static int alloc_snp_host_map(struct sev_device *sev)
+{
+       struct page *page;
+       int i;
+
+       for (i = 0; i < MAX_SNP_HOST_MAP_BUFS; i++) {
+               struct snp_host_map *map = &sev->snp_host_map[i];
+
+               memset(map, 0, sizeof(*map));
+
+               page = alloc_pages(GFP_KERNEL_ACCOUNT, get_order(SEV_FW_BLOB_MAX_SIZE));
+               if (!page)
+                       return -ENOMEM;
+
+               map->host = page_address(page);
+       }
+
+       return 0;
+}
+
+static void free_snp_host_map(struct sev_device *sev)
+{
+       int i;
+
+       for (i = 0; i < MAX_SNP_HOST_MAP_BUFS; i++) {
+               struct snp_host_map *map = &sev->snp_host_map[i];
+
+               if (map->host) {
+                       __free_pages(virt_to_page(map->host), get_order(SEV_FW_BLOB_MAX_SIZE));
+                       memset(map, 0, sizeof(*map));
+               }
+       }
+}
+
+static int map_firmware_writeable(u64 *paddr, u32 len, bool guest, struct snp_host_map *map)
+{
+       unsigned int npages = PAGE_ALIGN(len) >> PAGE_SHIFT;
+
+       map->active = false;
+
+       if (!paddr || !len)
+               return 0;
+
+       map->paddr = *paddr;
+       map->len = len;
+
+       /* If paddr points to a guest memory then change the page state to firmwware. */
+       if (guest) {
+               if (rmp_mark_pages_firmware(*paddr, npages, true))
+                       return -EFAULT;
+
+               goto done;
+       }
+
+       if (!map->host)
+               return -ENOMEM;
+
+       /* Check if the pre-allocated buffer can be used to fullfil the request. */
+       if (len > SEV_FW_BLOB_MAX_SIZE)
+               return -EINVAL;
+
+       /* Transition the pre-allocated buffer to the firmware state. */
+       if (rmp_mark_pages_firmware(__pa(map->host), npages, true))
+               return -EFAULT;
+
+       /* Set the paddr to use pre-allocated firmware buffer */
+       *paddr = __psp_pa(map->host);
+
+done:
+       map->active = true;
+       return 0;
+}
+
+static int unmap_firmware_writeable(u64 *paddr, u32 len, bool guest, struct snp_host_map *map)
+{
+       unsigned int npages = PAGE_ALIGN(len) >> PAGE_SHIFT;
+
+       if (!map->active)
+               return 0;
+
+       /* If paddr points to a guest memory then restore the page state to hypervisor. */
+       if (guest) {
+               if (snp_reclaim_pages(*paddr, npages, true))
+                       return -EFAULT;
+
+               goto done;
+       }
+
+       /*
+        * Transition the pre-allocated buffer to hypervisor state before the access.
+        *
+        * This is because while changing the page state to firmware, the kernel unmaps
+        * the pages from the direct map, and to restore the direct map the pages must
+        * be transitioned back to the shared state.
+        */
+       if (snp_reclaim_pages(__pa(map->host), npages, true))
+               return -EFAULT;
+
+       /* Copy the response data firmware buffer to the callers buffer. */
+       memcpy(__va(__sme_clr(map->paddr)), map->host, min_t(size_t, len, map->len));
+       *paddr = map->paddr;
+
+done:
+       map->active = false;
+       return 0;
+}
+
+static bool sev_legacy_cmd_buf_writable(int cmd)
+{
+       switch (cmd) {
+       case SEV_CMD_PLATFORM_STATUS:
+       case SEV_CMD_GUEST_STATUS:
+       case SEV_CMD_LAUNCH_START:
+       case SEV_CMD_RECEIVE_START:
+       case SEV_CMD_LAUNCH_MEASURE:
+       case SEV_CMD_SEND_START:
+       case SEV_CMD_SEND_UPDATE_DATA:
+       case SEV_CMD_SEND_UPDATE_VMSA:
+       case SEV_CMD_PEK_CSR:
+       case SEV_CMD_PDH_CERT_EXPORT:
+       case SEV_CMD_GET_ID:
+       case SEV_CMD_ATTESTATION_REPORT:
+               return true;
+       default:
+               return false;
+       }
+}
+
+#define prep_buffer(name, addr, len, guest, map) \
+       func(&((typeof(name *))cmd_buf)->addr, ((typeof(name *))cmd_buf)->len, guest, map)
+
+static int __snp_cmd_buf_copy(int cmd, void *cmd_buf, bool to_fw, int fw_err)
+{
+       int (*func)(u64 *paddr, u32 len, bool guest, struct snp_host_map *map);
+       struct sev_device *sev = psp_master->sev_data;
+       bool from_fw = !to_fw;
+
+       /*
+        * After the command is completed, change the command buffer memory to
+        * hypervisor state.
+        *
+        * The immutable bit is automatically cleared by the firmware, so
Quoted 8 lines

drivers/crypto/ccp/sev-dev.c static int sev_write_init_ex_file_if_required(int cmd_id) · line 660

Quoted 5 lines
drivers/crypto/ccp/sev-dev.c +5 -0
+        * no not need to reclaim the page.
+        */
+       if (from_fw && sev_legacy_cmd_buf_writable(cmd)) {
+               if (rmp_mark_pages_shared(__pa(cmd_buf), 1))
+                       return -EFAULT;

If we return here, we will skip calling unmap_firmware_writeable and we will leak some pages in firmware state.

Do you mean those (guest) pages which were transitioned to firmware state as part of snp_aware_copy_to_firmware()->_snp_cmd_buf_copy()->map_firmware_writeable()?

Quoted 112 lines
Quoted 100 lines
drivers/crypto/ccp/sev-dev.c +99 -0
+
+               /* No need to go further if firmware failed to execute command. */
+               if (fw_err)
+                       return 0;
+       }
+
+       if (to_fw)
+               func = map_firmware_writeable;
+       else
+               func = unmap_firmware_writeable;
+
+       /*
+        * A command buffer may contains a system physical address. If the address
+        * points to a host memory then use an intermediate firmware page otherwise
+        * change the page state in the RMP table.
+        */
+       switch (cmd) {
+       case SEV_CMD_PDH_CERT_EXPORT:
+               if (prep_buffer(struct sev_data_pdh_cert_export, pdh_cert_address,
+                               pdh_cert_len, false, &sev->snp_host_map[0]))
+                       goto err;
+               if (prep_buffer(struct sev_data_pdh_cert_export, cert_chain_address,
+                               cert_chain_len, false, &sev->snp_host_map[1]))
+                       goto err;
+               break;
+       case SEV_CMD_GET_ID:
+               if (prep_buffer(struct sev_data_get_id, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_PEK_CSR:
+               if (prep_buffer(struct sev_data_pek_csr, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_UPDATE_DATA:
+               if (prep_buffer(struct sev_data_launch_update_data, address, len,
+                               true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_UPDATE_VMSA:
+               if (prep_buffer(struct sev_data_launch_update_vmsa, address, len,
+                               true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_MEASURE:
+               if (prep_buffer(struct sev_data_launch_measure, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_UPDATE_SECRET:
+               if (prep_buffer(struct sev_data_launch_secret, guest_address, guest_len,
+                               true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_DBG_DECRYPT:
+               if (prep_buffer(struct sev_data_dbg, dst_addr, len, false,
+                               &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_DBG_ENCRYPT:
+               if (prep_buffer(struct sev_data_dbg, dst_addr, len, true,
+                               &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_ATTESTATION_REPORT:
+               if (prep_buffer(struct sev_data_attestation_report, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_SEND_START:
+               if (prep_buffer(struct sev_data_send_start, session_address,
+                               session_len, false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_SEND_UPDATE_DATA:
+               if (prep_buffer(struct sev_data_send_update_data, hdr_address, hdr_len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               if (prep_buffer(struct sev_data_send_update_data, trans_address,
+                               trans_len, false, &sev->snp_host_map[1]))
+                       goto err;
+               break;
+       case SEV_CMD_SEND_UPDATE_VMSA:
+               if (prep_buffer(struct sev_data_send_update_vmsa, hdr_address, hdr_len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               if (prep_buffer(struct sev_data_send_update_vmsa, trans_address,
+                               trans_len, false, &sev->snp_host_map[1]))
+                       goto err;
+               break;
+       case SEV_CMD_RECEIVE_UPDATE_DATA:
+               if (prep_buffer(struct sev_data_receive_update_data, guest_address,
+                               guest_len, true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_RECEIVE_UPDATE_VMSA:
+               if (prep_buffer(struct sev_data_receive_update_vmsa, guest_address,
+                               guest_len, true, &sev->snp_host_map[0]))

drivers/crypto/ccp/sev-dev.c static int sev_write_init_ex_file_if_required(int cmd_id) · line 769

Quoted 10 lines
drivers/crypto/ccp/sev-dev.c +10 -0
+                       goto err;
+               break;
+       default:
+               break;
+       }
+
+       /* The command buffer need to be in the firmware state. */
+       if (to_fw && sev_legacy_cmd_buf_writable(cmd)) {
+               if (rmp_mark_pages_firmware(__pa(cmd_buf), 1, true))
+                       return -EFAULT;

This function moves two separate pages to firmware state. First

Quoted 9 lines

calling map_firmware_writeable and second calling rmp_mark_pages_firmware for cmd_buf.
In case rmp_mark_pages_firmware fails for cmd_buf, the page which has already moved to firmware state in map_firmware_writeable should be reclaimed.
This is a problem especially if we leak a guest owned page in firmware state. Since this is used only by legacy SEV VMs, these leaked pages will never be reclaimed back when destroying these VMs.

Yes, this looks to be an inherent issue with the original patch, as you mentioned there are two pages - guest owned page and the HV cmd_buf, and failure to transition the cmd_buf back to HV/shared state has no corresponding recovery/reclaim for the transitioned guest page.

Thanks,
Ashish

Quoted 197 lines
Quoted 197 lines
drivers/crypto/ccp/sev-dev.c +80 -10
+       }
+
+       return 0;
+
+err:
+       return -EINVAL;
+}
+
+static inline bool need_firmware_copy(int cmd)
+{
+       struct sev_device *sev = psp_master->sev_data;
+
+       /* After SNP is INIT'ed, the behavior of legacy SEV command is changed. */
+       return ((cmd < SEV_CMD_SNP_INIT) && sev->snp_initialized) ? true : false;
+}
+
+static int snp_aware_copy_to_firmware(int cmd, void *data)
+{
+       return __snp_cmd_buf_copy(cmd, data, true, 0);
+}
+
+static int snp_aware_copy_from_firmware(int cmd, void *data, int fw_err)
+{
+       return __snp_cmd_buf_copy(cmd, data, false, fw_err);
+}
+
  static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
  {
         struct psp_device *psp = psp_master;
         struct sev_device *sev;
         unsigned int phys_lsb, phys_msb;
         unsigned int reg, ret = 0;
+       void *cmd_buf;
         int buf_len;
 
         if (!psp || !psp->sev_data)
@@ -512,12 +819,28 @@ static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
          * work for some memory, e.g. vmalloc'd addresses, and @data may not be
          * physically contiguous.
          */
-       if (data)
-               memcpy(sev->cmd_buf, data, buf_len);
+       if (data) {
+               if (sev->cmd_buf_active > 2)
+                       return -EBUSY;
+
+               cmd_buf = sev->cmd_buf_active ? sev->cmd_buf_backup : sev->cmd_buf;
+
+               memcpy(cmd_buf, data, buf_len);
+               sev->cmd_buf_active++;
+
+               /*
+                * The behavior of the SEV-legacy commands is altered when the
+                * SNP firmware is in the INIT state.
+                */
+               if (need_firmware_copy(cmd) && snp_aware_copy_to_firmware(cmd, sev->cmd_buf))
+                       return -EFAULT;
+       } else {
+               cmd_buf = sev->cmd_buf;
+       }
 
         /* Get the physical address of the command buffer */
-       phys_lsb = data ? lower_32_bits(__psp_pa(sev->cmd_buf)) : 0;
-       phys_msb = data ? upper_32_bits(__psp_pa(sev->cmd_buf)) : 0;
+       phys_lsb = data ? lower_32_bits(__psp_pa(cmd_buf)) : 0;
+       phys_msb = data ? upper_32_bits(__psp_pa(cmd_buf)) : 0;
 
         dev_dbg(sev->dev, "sev command id %#x buffer 0x%08x%08x timeout %us\n",
                 cmd, phys_msb, phys_lsb, psp_timeout);
@@ -560,15 +883,24 @@ static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
                 ret = sev_write_init_ex_file_if_required(cmd);
         }
 
-       print_hex_dump_debug("(out): ", DUMP_PREFIX_OFFSET, 16, 2, data,
-                            buf_len, false);
-
         /*
          * Copy potential output from the PSP back to data.  Do this even on
          * failure in case the caller wants to glean something from the error.
          */
-       if (data)
-               memcpy(data, sev->cmd_buf, buf_len);
+       if (data) {
+               /*
+                * Restore the page state after the command completes.
+                */
+               if (need_firmware_copy(cmd) &&
+                   snp_aware_copy_from_firmware(cmd, cmd_buf, ret))
+                       return -EFAULT;
+
+               memcpy(data, cmd_buf, buf_len);
+               sev->cmd_buf_active--;
+       }
+
+       print_hex_dump_debug("(out): ", DUMP_PREFIX_OFFSET, 16, 2, data,
+                            buf_len, false);
 
         return ret;
  }
@@ -1579,10 +1911,12 @@ int sev_dev_init(struct psp_device *psp)
         if (!sev)
                 goto e_err;
 
-       sev->cmd_buf = (void *)devm_get_free_pages(dev, GFP_KERNEL, 0);
+       sev->cmd_buf = (void *)devm_get_free_pages(dev, GFP_KERNEL, 1);
         if (!sev->cmd_buf)
                 goto e_sev;
 
+       sev->cmd_buf_backup = (uint8_t *)sev->cmd_buf + PAGE_SIZE;
+
         psp->sev_data = sev;
 
         sev->dev = dev;
@@ -1648,6 +1982,12 @@ static void sev_firmware_shutdown(struct sev_device *sev)
                 snp_range_list = NULL;
         }
 
+       /*
+        * The host map need to clear the immutable bit so it must be free'd before the
+        * SNP firmware shutdown.
+        */
+       free_snp_host_map(sev);
+
         sev_snp_shutdown(&error);
  }
 
@@ -1722,6 +2062,14 @@ void sev_pci_init(void)
                                 dev_err(sev->dev, "SEV-SNP: failed to INIT error %#x\n", error);
                         }
                 }
+
+               /*
+                * Allocate the intermediate buffers used for the legacy command handling.
+                */
+               if (alloc_snp_host_map(sev)) {
+                       dev_notice(sev->dev, "Failed to alloc host map (disabling legacy SEV)\n");
+                       goto skip_legacy;
+               }
         }
 
         /* Obtain the TMR memory area for SEV-ES use */
@@ -1739,12 +2087,14 @@ void sev_pci_init(void)
                 dev_err(sev->dev, "SEV: failed to INIT error %#x, rc %d\n",
                         error, rc);
 
+skip_legacy:
         dev_info(sev->dev, "SEV%s API:%d.%d build:%d\n", sev->snp_initialized ?
                 "-SNP" : "", sev->api_major, sev->api_minor, sev->build);
 
         return;
 
  err:
+       free_snp_host_map(sev);
         psp_master->sev_data = NULL;
  }
 
drivers/crypto/ccp/sev-dev.h +12 -1
index 34767657beb5..19d79f9d4212 100644
@@ -29,11 +29,20 @@
  #define SEV_CMDRESP_CMD_SHIFT          16
  #define SEV_CMDRESP_IOC                        BIT(0)
 
+#define MAX_SNP_HOST_MAP_BUFS          2
+
  struct sev_misc_dev {
         struct kref refcount;
         struct miscdevice misc;
  };
 
+struct snp_host_map {
+       u64 paddr;
+       u32 len;
+       void *host;
+       bool active;
+};
+
  struct sev_device {
         struct device *dev;
         struct psp_device *psp;
@@ -52,8 +61,11 @@ struct sev_device {
         u8 build;
 
         void *cmd_buf;
+       void *cmd_buf_backup;
+       int cmd_buf_active;
 
         bool snp_initialized;
+       struct snp_host_map snp_host_map[MAX_SNP_HOST_MAP_BUFS];
  };
 
  int sev_dev_init(struct psp_device *psp);
--

2.25.1

#108 lore raw

Re: [PATCH RFC v7 29/64] crypto: ccp: Handle the legacy SEV command when SNP is enabled

Quoted 223 lines · Ashish

Hello Alper,

Quoted 210 lines · Alper Gun
Quoted 209 lines · Michael Roth

From: Brijesh Singh <[email protected]>

The behavior of the SEV-legacy commands is altered when the SNP firmware is in the INIT state. When SNP is in INIT state, all the SEV-legacy commands that cause the firmware to write to memory must be in the firmware state before issuing the command..

A command buffer may contains a system physical address that the firmware may write to. There are two cases that need to be handled:

1) system physical address points to a guest memory
2) system physical address points to a host memory

To handle the case #1, change the page state to the firmware in the RMP table before issuing the command and restore the state to shared after the command completes.

For the case #2, use a bounce buffer to complete the request.

---
  drivers/crypto/ccp/sev-dev.c | 370 ++++++++++++++++++++++++++++++++++-
  drivers/crypto/ccp/sev-dev.h |  12 ++
  2 files changed, 372 insertions(+), 10 deletions(-)
drivers/crypto/ccp/sev-dev.c +166 -0
index 4c12e98a1219..5eb2e8f364d4 100644
@@ -286,6 +286,30 @@ static int rmp_mark_pages_firmware(unsigned long paddr, unsigned int npages, boo
         return rc;
  }
 
+static int rmp_mark_pages_shared(unsigned long paddr, unsigned int npages)
+{
+       /* Cbit maybe set in the paddr */
+       unsigned long pfn = __sme_clr(paddr) >> PAGE_SHIFT;
+       int rc, n = 0, i;
+
+       for (i = 0; i < npages; i++, pfn++, n++) {
+               rc = rmp_make_shared(pfn, PG_LEVEL_4K);
+               if (rc)
+                       goto cleanup;
+       }
+
+       return 0;
+
+cleanup:
+       /*
+        * If failed to change the page state to shared, then its not safe
+        * to release the page back to the system, leak it.
+        */
+       snp_mark_pages_offline(pfn, npages - n);
+
+       return rc;
+}
+
  static struct page *__snp_alloc_firmware_pages(gfp_t gfp_mask, int order, bool locked)
  {
         unsigned long npages = 1ul << order, paddr;
@@ -487,12 +511,295 @@ static int sev_write_init_ex_file_if_required(int cmd_id)
         return sev_write_init_ex_file();
  }
 
+static int alloc_snp_host_map(struct sev_device *sev)
+{
+       struct page *page;
+       int i;
+
+       for (i = 0; i < MAX_SNP_HOST_MAP_BUFS; i++) {
+               struct snp_host_map *map = &sev->snp_host_map[i];
+
+               memset(map, 0, sizeof(*map));
+
+               page = alloc_pages(GFP_KERNEL_ACCOUNT, get_order(SEV_FW_BLOB_MAX_SIZE));
+               if (!page)
+                       return -ENOMEM;
+
+               map->host = page_address(page);
+       }
+
+       return 0;
+}
+
+static void free_snp_host_map(struct sev_device *sev)
+{
+       int i;
+
+       for (i = 0; i < MAX_SNP_HOST_MAP_BUFS; i++) {
+               struct snp_host_map *map = &sev->snp_host_map[i];
+
+               if (map->host) {
+                       __free_pages(virt_to_page(map->host), get_order(SEV_FW_BLOB_MAX_SIZE));
+                       memset(map, 0, sizeof(*map));
+               }
+       }
+}
+
+static int map_firmware_writeable(u64 *paddr, u32 len, bool guest, struct snp_host_map *map)
+{
+       unsigned int npages = PAGE_ALIGN(len) >> PAGE_SHIFT;
+
+       map->active = false;
+
+       if (!paddr || !len)
+               return 0;
+
+       map->paddr = *paddr;
+       map->len = len;
+
+       /* If paddr points to a guest memory then change the page state to firmwware. */
+       if (guest) {
+               if (rmp_mark_pages_firmware(*paddr, npages, true))
+                       return -EFAULT;
+
+               goto done;
+       }
+
+       if (!map->host)
+               return -ENOMEM;
+
+       /* Check if the pre-allocated buffer can be used to fullfil the request. */
+       if (len > SEV_FW_BLOB_MAX_SIZE)
+               return -EINVAL;
+
+       /* Transition the pre-allocated buffer to the firmware state. */
+       if (rmp_mark_pages_firmware(__pa(map->host), npages, true))
+               return -EFAULT;
+
+       /* Set the paddr to use pre-allocated firmware buffer */
+       *paddr = __psp_pa(map->host);
+
+done:
+       map->active = true;
+       return 0;
+}
+
+static int unmap_firmware_writeable(u64 *paddr, u32 len, bool guest, struct snp_host_map *map)
+{
+       unsigned int npages = PAGE_ALIGN(len) >> PAGE_SHIFT;
+
+       if (!map->active)
+               return 0;
+
+       /* If paddr points to a guest memory then restore the page state to hypervisor. */
+       if (guest) {
+               if (snp_reclaim_pages(*paddr, npages, true))
+                       return -EFAULT;
+
+               goto done;
+       }
+
+       /*
+        * Transition the pre-allocated buffer to hypervisor state before the access.
+        *
+        * This is because while changing the page state to firmware, the kernel unmaps
+        * the pages from the direct map, and to restore the direct map the pages must
+        * be transitioned back to the shared state.
+        */
+       if (snp_reclaim_pages(__pa(map->host), npages, true))
+               return -EFAULT;
+
+       /* Copy the response data firmware buffer to the callers buffer. */
+       memcpy(__va(__sme_clr(map->paddr)), map->host, min_t(size_t, len, map->len));
+       *paddr = map->paddr;
+
+done:
+       map->active = false;
+       return 0;
+}
+
+static bool sev_legacy_cmd_buf_writable(int cmd)
+{
+       switch (cmd) {
+       case SEV_CMD_PLATFORM_STATUS:
+       case SEV_CMD_GUEST_STATUS:
+       case SEV_CMD_LAUNCH_START:
+       case SEV_CMD_RECEIVE_START:
+       case SEV_CMD_LAUNCH_MEASURE:
+       case SEV_CMD_SEND_START:
+       case SEV_CMD_SEND_UPDATE_DATA:
+       case SEV_CMD_SEND_UPDATE_VMSA:
+       case SEV_CMD_PEK_CSR:
+       case SEV_CMD_PDH_CERT_EXPORT:
+       case SEV_CMD_GET_ID:
+       case SEV_CMD_ATTESTATION_REPORT:
+               return true;
+       default:
+               return false;
+       }
+}
+
+#define prep_buffer(name, addr, len, guest, map) \
+       func(&((typeof(name *))cmd_buf)->addr, ((typeof(name *))cmd_buf)->len, guest, map)
+
+static int __snp_cmd_buf_copy(int cmd, void *cmd_buf, bool to_fw, int fw_err)
+{
+       int (*func)(u64 *paddr, u32 len, bool guest, struct snp_host_map *map);
+       struct sev_device *sev = psp_master->sev_data;
+       bool from_fw = !to_fw;
+
+       /*
+        * After the command is completed, change the command buffer memory to
+        * hypervisor state.
+        *
+        * The immutable bit is automatically cleared by the firmware, so
Quoted 8 lines

drivers/crypto/ccp/sev-dev.c static int sev_write_init_ex_file_if_required(int cmd_id) · line 660

Quoted 5 lines
drivers/crypto/ccp/sev-dev.c +5 -0
+        * no not need to reclaim the page.
+        */
+       if (from_fw && sev_legacy_cmd_buf_writable(cmd)) {
+               if (rmp_mark_pages_shared(__pa(cmd_buf), 1))
+                       return -EFAULT;

If we return here, we will skip calling unmap_firmware_writeable and we will leak some pages in firmware state.

Quoted 3 lines

Do you mean those (guest) pages which were transitioned to firmware state as part of snp_aware_copy_to_firmware()->_snp_cmd_buf_copy()->map_firmware_writeable()?

yes, if we return here, these guest pages will be left in firmware state.

Quoted 6 lines
Quoted 6 lines

drivers/crypto/ccp/sev-dev.c static int sev_write_init_ex_file_if_required(int cmd_id) · line 664

Quoted 6 lines
drivers/crypto/ccp/sev-dev.c +4 -0
+
+               /* No need to go further if firmware failed to execute command. */
+               if (fw_err)
+                       return 0;

Same thing also here, we are possibly leaving guest pages in the firmware state.

Quoted 322 lines
Quoted 107 lines
Quoted 95 lines
drivers/crypto/ccp/sev-dev.c +95 -0
+       }
+
+       if (to_fw)
+               func = map_firmware_writeable;
+       else
+               func = unmap_firmware_writeable;
+
+       /*
+        * A command buffer may contains a system physical address. If the address
+        * points to a host memory then use an intermediate firmware page otherwise
+        * change the page state in the RMP table.
+        */
+       switch (cmd) {
+       case SEV_CMD_PDH_CERT_EXPORT:
+               if (prep_buffer(struct sev_data_pdh_cert_export, pdh_cert_address,
+                               pdh_cert_len, false, &sev->snp_host_map[0]))
+                       goto err;
+               if (prep_buffer(struct sev_data_pdh_cert_export, cert_chain_address,
+                               cert_chain_len, false, &sev->snp_host_map[1]))
+                       goto err;
+               break;
+       case SEV_CMD_GET_ID:
+               if (prep_buffer(struct sev_data_get_id, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_PEK_CSR:
+               if (prep_buffer(struct sev_data_pek_csr, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_UPDATE_DATA:
+               if (prep_buffer(struct sev_data_launch_update_data, address, len,
+                               true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_UPDATE_VMSA:
+               if (prep_buffer(struct sev_data_launch_update_vmsa, address, len,
+                               true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_MEASURE:
+               if (prep_buffer(struct sev_data_launch_measure, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_UPDATE_SECRET:
+               if (prep_buffer(struct sev_data_launch_secret, guest_address, guest_len,
+                               true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_DBG_DECRYPT:
+               if (prep_buffer(struct sev_data_dbg, dst_addr, len, false,
+                               &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_DBG_ENCRYPT:
+               if (prep_buffer(struct sev_data_dbg, dst_addr, len, true,
+                               &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_ATTESTATION_REPORT:
+               if (prep_buffer(struct sev_data_attestation_report, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_SEND_START:
+               if (prep_buffer(struct sev_data_send_start, session_address,
+                               session_len, false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_SEND_UPDATE_DATA:
+               if (prep_buffer(struct sev_data_send_update_data, hdr_address, hdr_len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               if (prep_buffer(struct sev_data_send_update_data, trans_address,
+                               trans_len, false, &sev->snp_host_map[1]))
+                       goto err;
+               break;
+       case SEV_CMD_SEND_UPDATE_VMSA:
+               if (prep_buffer(struct sev_data_send_update_vmsa, hdr_address, hdr_len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               if (prep_buffer(struct sev_data_send_update_vmsa, trans_address,
+                               trans_len, false, &sev->snp_host_map[1]))
+                       goto err;
+               break;
+       case SEV_CMD_RECEIVE_UPDATE_DATA:
+               if (prep_buffer(struct sev_data_receive_update_data, guest_address,
+                               guest_len, true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_RECEIVE_UPDATE_VMSA:
+               if (prep_buffer(struct sev_data_receive_update_vmsa, guest_address,
+                               guest_len, true, &sev->snp_host_map[0]))

drivers/crypto/ccp/sev-dev.c static int sev_write_init_ex_file_if_required(int cmd_id) · line 769

Quoted 10 lines
drivers/crypto/ccp/sev-dev.c +10 -0
+                       goto err;
+               break;
+       default:
+               break;
+       }
+
+       /* The command buffer need to be in the firmware state. */
+       if (to_fw && sev_legacy_cmd_buf_writable(cmd)) {
+               if (rmp_mark_pages_firmware(__pa(cmd_buf), 1, true))
+                       return -EFAULT;

This function moves two separate pages to firmware state. First

Quoted 9 lines

calling map_firmware_writeable and second calling rmp_mark_pages_firmware for cmd_buf.
In case rmp_mark_pages_firmware fails for cmd_buf, the page which has already moved to firmware state in map_firmware_writeable should be reclaimed.
This is a problem especially if we leak a guest owned page in firmware state. Since this is used only by legacy SEV VMs, these leaked pages will never be reclaimed back when destroying these VMs.

Yes, this looks to be an inherent issue with the original patch, as you mentioned there are two pages - guest owned page and the HV cmd_buf, and failure to transition the cmd_buf back to HV/shared state has no corresponding recovery/reclaim for the transitioned guest page.

Thanks,
Ashish

Quoted 197 lines
Quoted 197 lines
drivers/crypto/ccp/sev-dev.c +80 -10
+       }
+
+       return 0;
+
+err:
+       return -EINVAL;
+}
+
+static inline bool need_firmware_copy(int cmd)
+{
+       struct sev_device *sev = psp_master->sev_data;
+
+       /* After SNP is INIT'ed, the behavior of legacy SEV command is changed. */
+       return ((cmd < SEV_CMD_SNP_INIT) && sev->snp_initialized) ? true : false;
+}
+
+static int snp_aware_copy_to_firmware(int cmd, void *data)
+{
+       return __snp_cmd_buf_copy(cmd, data, true, 0);
+}
+
+static int snp_aware_copy_from_firmware(int cmd, void *data, int fw_err)
+{
+       return __snp_cmd_buf_copy(cmd, data, false, fw_err);
+}
+
  static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
  {
         struct psp_device *psp = psp_master;
         struct sev_device *sev;
         unsigned int phys_lsb, phys_msb;
         unsigned int reg, ret = 0;
+       void *cmd_buf;
         int buf_len;
 
         if (!psp || !psp->sev_data)
@@ -512,12 +819,28 @@ static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
          * work for some memory, e.g. vmalloc'd addresses, and @data may not be
          * physically contiguous.
          */
-       if (data)
-               memcpy(sev->cmd_buf, data, buf_len);
+       if (data) {
+               if (sev->cmd_buf_active > 2)
+                       return -EBUSY;
+
+               cmd_buf = sev->cmd_buf_active ? sev->cmd_buf_backup : sev->cmd_buf;
+
+               memcpy(cmd_buf, data, buf_len);
+               sev->cmd_buf_active++;
+
+               /*
+                * The behavior of the SEV-legacy commands is altered when the
+                * SNP firmware is in the INIT state.
+                */
+               if (need_firmware_copy(cmd) && snp_aware_copy_to_firmware(cmd, sev->cmd_buf))
+                       return -EFAULT;
+       } else {
+               cmd_buf = sev->cmd_buf;
+       }
 
         /* Get the physical address of the command buffer */
-       phys_lsb = data ? lower_32_bits(__psp_pa(sev->cmd_buf)) : 0;
-       phys_msb = data ? upper_32_bits(__psp_pa(sev->cmd_buf)) : 0;
+       phys_lsb = data ? lower_32_bits(__psp_pa(cmd_buf)) : 0;
+       phys_msb = data ? upper_32_bits(__psp_pa(cmd_buf)) : 0;
 
         dev_dbg(sev->dev, "sev command id %#x buffer 0x%08x%08x timeout %us\n",
                 cmd, phys_msb, phys_lsb, psp_timeout);
@@ -560,15 +883,24 @@ static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
                 ret = sev_write_init_ex_file_if_required(cmd);
         }
 
-       print_hex_dump_debug("(out): ", DUMP_PREFIX_OFFSET, 16, 2, data,
-                            buf_len, false);
-
         /*
          * Copy potential output from the PSP back to data.  Do this even on
          * failure in case the caller wants to glean something from the error.
          */
-       if (data)
-               memcpy(data, sev->cmd_buf, buf_len);
+       if (data) {
+               /*
+                * Restore the page state after the command completes.
+                */
+               if (need_firmware_copy(cmd) &&
+                   snp_aware_copy_from_firmware(cmd, cmd_buf, ret))
+                       return -EFAULT;
+
+               memcpy(data, cmd_buf, buf_len);
+               sev->cmd_buf_active--;
+       }
+
+       print_hex_dump_debug("(out): ", DUMP_PREFIX_OFFSET, 16, 2, data,
+                            buf_len, false);
 
         return ret;
  }
@@ -1579,10 +1911,12 @@ int sev_dev_init(struct psp_device *psp)
         if (!sev)
                 goto e_err;
 
-       sev->cmd_buf = (void *)devm_get_free_pages(dev, GFP_KERNEL, 0);
+       sev->cmd_buf = (void *)devm_get_free_pages(dev, GFP_KERNEL, 1);
         if (!sev->cmd_buf)
                 goto e_sev;
 
+       sev->cmd_buf_backup = (uint8_t *)sev->cmd_buf + PAGE_SIZE;
+
         psp->sev_data = sev;
 
         sev->dev = dev;
@@ -1648,6 +1982,12 @@ static void sev_firmware_shutdown(struct sev_device *sev)
                 snp_range_list = NULL;
         }
 
+       /*
+        * The host map need to clear the immutable bit so it must be free'd before the
+        * SNP firmware shutdown.
+        */
+       free_snp_host_map(sev);
+
         sev_snp_shutdown(&error);
  }
 
@@ -1722,6 +2062,14 @@ void sev_pci_init(void)
                                 dev_err(sev->dev, "SEV-SNP: failed to INIT error %#x\n", error);
                         }
                 }
+
+               /*
+                * Allocate the intermediate buffers used for the legacy command handling.
+                */
+               if (alloc_snp_host_map(sev)) {
+                       dev_notice(sev->dev, "Failed to alloc host map (disabling legacy SEV)\n");
+                       goto skip_legacy;
+               }
         }
 
         /* Obtain the TMR memory area for SEV-ES use */
@@ -1739,12 +2087,14 @@ void sev_pci_init(void)
                 dev_err(sev->dev, "SEV: failed to INIT error %#x, rc %d\n",
                         error, rc);
 
+skip_legacy:
         dev_info(sev->dev, "SEV%s API:%d.%d build:%d\n", sev->snp_initialized ?
                 "-SNP" : "", sev->api_major, sev->api_minor, sev->build);
 
         return;
 
  err:
+       free_snp_host_map(sev);
         psp_master->sev_data = NULL;
  }
 
drivers/crypto/ccp/sev-dev.h +12 -1
index 34767657beb5..19d79f9d4212 100644
@@ -29,11 +29,20 @@
  #define SEV_CMDRESP_CMD_SHIFT          16
  #define SEV_CMDRESP_IOC                        BIT(0)
 
+#define MAX_SNP_HOST_MAP_BUFS          2
+
  struct sev_misc_dev {
         struct kref refcount;
         struct miscdevice misc;
  };
 
+struct snp_host_map {
+       u64 paddr;
+       u32 len;
+       void *host;
+       bool active;
+};
+
  struct sev_device {
         struct device *dev;
         struct psp_device *psp;
@@ -52,8 +61,11 @@ struct sev_device {
         u8 build;
 
         void *cmd_buf;
+       void *cmd_buf_backup;
+       int cmd_buf_active;
 
         bool snp_initialized;
+       struct snp_host_map snp_host_map[MAX_SNP_HOST_MAP_BUFS];
  };
 
  int sev_dev_init(struct psp_device *psp);
--

2.25.1

maintainer
#109 lore raw

Re: [PATCH RFC v7 29/64] crypto: ccp: Handle the legacy SEV command when SNP is enabled

Quoted 228 lines · Alper Gun
Quoted 223 lines · Ashish

Hello Alper,

Quoted 210 lines · Alper Gun
Quoted 209 lines · Michael Roth

From: Brijesh Singh <[email protected]>

The behavior of the SEV-legacy commands is altered when the SNP firmware is in the INIT state. When SNP is in INIT state, all the SEV-legacy commands that cause the firmware to write to memory must be in the firmware state before issuing the command..

A command buffer may contains a system physical address that the firmware may write to. There are two cases that need to be handled:

1) system physical address points to a guest memory
2) system physical address points to a host memory

To handle the case #1, change the page state to the firmware in the RMP table before issuing the command and restore the state to shared after the command completes.

For the case #2, use a bounce buffer to complete the request.

---
   drivers/crypto/ccp/sev-dev.c | 370 ++++++++++++++++++++++++++++++++++-
   drivers/crypto/ccp/sev-dev.h |  12 ++
   2 files changed, 372 insertions(+), 10 deletions(-)
drivers/crypto/ccp/sev-dev.c +166 -0
index 4c12e98a1219..5eb2e8f364d4 100644
@@ -286,6 +286,30 @@ static int rmp_mark_pages_firmware(unsigned long paddr, unsigned int npages, boo
          return rc;
   }
 
+static int rmp_mark_pages_shared(unsigned long paddr, unsigned int npages)
+{
+       /* Cbit maybe set in the paddr */
+       unsigned long pfn = __sme_clr(paddr) >> PAGE_SHIFT;
+       int rc, n = 0, i;
+
+       for (i = 0; i < npages; i++, pfn++, n++) {
+               rc = rmp_make_shared(pfn, PG_LEVEL_4K);
+               if (rc)
+                       goto cleanup;
+       }
+
+       return 0;
+
+cleanup:
+       /*
+        * If failed to change the page state to shared, then its not safe
+        * to release the page back to the system, leak it.
+        */
+       snp_mark_pages_offline(pfn, npages - n);
+
+       return rc;
+}
+
   static struct page *__snp_alloc_firmware_pages(gfp_t gfp_mask, int order, bool locked)
   {
          unsigned long npages = 1ul << order, paddr;
@@ -487,12 +511,295 @@ static int sev_write_init_ex_file_if_required(int cmd_id)
          return sev_write_init_ex_file();
   }
 
+static int alloc_snp_host_map(struct sev_device *sev)
+{
+       struct page *page;
+       int i;
+
+       for (i = 0; i < MAX_SNP_HOST_MAP_BUFS; i++) {
+               struct snp_host_map *map = &sev->snp_host_map[i];
+
+               memset(map, 0, sizeof(*map));
+
+               page = alloc_pages(GFP_KERNEL_ACCOUNT, get_order(SEV_FW_BLOB_MAX_SIZE));
+               if (!page)
+                       return -ENOMEM;
+
+               map->host = page_address(page);
+       }
+
+       return 0;
+}
+
+static void free_snp_host_map(struct sev_device *sev)
+{
+       int i;
+
+       for (i = 0; i < MAX_SNP_HOST_MAP_BUFS; i++) {
+               struct snp_host_map *map = &sev->snp_host_map[i];
+
+               if (map->host) {
+                       __free_pages(virt_to_page(map->host), get_order(SEV_FW_BLOB_MAX_SIZE));
+                       memset(map, 0, sizeof(*map));
+               }
+       }
+}
+
+static int map_firmware_writeable(u64 *paddr, u32 len, bool guest, struct snp_host_map *map)
+{
+       unsigned int npages = PAGE_ALIGN(len) >> PAGE_SHIFT;
+
+       map->active = false;
+
+       if (!paddr || !len)
+               return 0;
+
+       map->paddr = *paddr;
+       map->len = len;
+
+       /* If paddr points to a guest memory then change the page state to firmwware. */
+       if (guest) {
+               if (rmp_mark_pages_firmware(*paddr, npages, true))
+                       return -EFAULT;
+
+               goto done;
+       }
+
+       if (!map->host)
+               return -ENOMEM;
+
+       /* Check if the pre-allocated buffer can be used to fullfil the request. */
+       if (len > SEV_FW_BLOB_MAX_SIZE)
+               return -EINVAL;
+
+       /* Transition the pre-allocated buffer to the firmware state. */
+       if (rmp_mark_pages_firmware(__pa(map->host), npages, true))
+               return -EFAULT;
+
+       /* Set the paddr to use pre-allocated firmware buffer */
+       *paddr = __psp_pa(map->host);
+
+done:
+       map->active = true;
+       return 0;
+}
+
+static int unmap_firmware_writeable(u64 *paddr, u32 len, bool guest, struct snp_host_map *map)
+{
+       unsigned int npages = PAGE_ALIGN(len) >> PAGE_SHIFT;
+
+       if (!map->active)
+               return 0;
+
+       /* If paddr points to a guest memory then restore the page state to hypervisor. */
+       if (guest) {
+               if (snp_reclaim_pages(*paddr, npages, true))
+                       return -EFAULT;
+
+               goto done;
+       }
+
+       /*
+        * Transition the pre-allocated buffer to hypervisor state before the access.
+        *
+        * This is because while changing the page state to firmware, the kernel unmaps
+        * the pages from the direct map, and to restore the direct map the pages must
+        * be transitioned back to the shared state.
+        */
+       if (snp_reclaim_pages(__pa(map->host), npages, true))
+               return -EFAULT;
+
+       /* Copy the response data firmware buffer to the callers buffer. */
+       memcpy(__va(__sme_clr(map->paddr)), map->host, min_t(size_t, len, map->len));
+       *paddr = map->paddr;
+
+done:
+       map->active = false;
+       return 0;
+}
+
+static bool sev_legacy_cmd_buf_writable(int cmd)
+{
+       switch (cmd) {
+       case SEV_CMD_PLATFORM_STATUS:
+       case SEV_CMD_GUEST_STATUS:
+       case SEV_CMD_LAUNCH_START:
+       case SEV_CMD_RECEIVE_START:
+       case SEV_CMD_LAUNCH_MEASURE:
+       case SEV_CMD_SEND_START:
+       case SEV_CMD_SEND_UPDATE_DATA:
+       case SEV_CMD_SEND_UPDATE_VMSA:
+       case SEV_CMD_PEK_CSR:
+       case SEV_CMD_PDH_CERT_EXPORT:
+       case SEV_CMD_GET_ID:
+       case SEV_CMD_ATTESTATION_REPORT:
+               return true;
+       default:
+               return false;
+       }
+}
+
+#define prep_buffer(name, addr, len, guest, map) \
+       func(&((typeof(name *))cmd_buf)->addr, ((typeof(name *))cmd_buf)->len, guest, map)
+
+static int __snp_cmd_buf_copy(int cmd, void *cmd_buf, bool to_fw, int fw_err)
+{
+       int (*func)(u64 *paddr, u32 len, bool guest, struct snp_host_map *map);
+       struct sev_device *sev = psp_master->sev_data;
+       bool from_fw = !to_fw;
+
+       /*
+        * After the command is completed, change the command buffer memory to
+        * hypervisor state.
+        *
+        * The immutable bit is automatically cleared by the firmware, so
Quoted 8 lines

drivers/crypto/ccp/sev-dev.c static int sev_write_init_ex_file_if_required(int cmd_id) · line 660

Quoted 5 lines
drivers/crypto/ccp/sev-dev.c +5 -0
+        * no not need to reclaim the page.
+        */
+       if (from_fw && sev_legacy_cmd_buf_writable(cmd)) {
+               if (rmp_mark_pages_shared(__pa(cmd_buf), 1))
+                       return -EFAULT;

If we return here, we will skip calling unmap_firmware_writeable and we will leak some pages in firmware state.

Quoted 3 lines

Do you mean those (guest) pages which were transitioned to firmware state as part of snp_aware_copy_to_firmware()->_snp_cmd_buf_copy()->map_firmware_writeable()?

Quoted 9 lines

yes, if we return here, these guest pages will be left in firmware state.

Quoted 6 lines
Quoted 6 lines

drivers/crypto/ccp/sev-dev.c static int sev_write_init_ex_file_if_required(int cmd_id) · line 664

Quoted 6 lines
drivers/crypto/ccp/sev-dev.c +4 -0
+
+               /* No need to go further if firmware failed to execute command. */
+               if (fw_err)
+                       return 0;

Same thing also here, we are possibly leaving guest pages in the firmware state.

Yes, i agree, both are inhererent bugs in the original patch and need to be fixed.

Thanks,
Ashish

Quoted 322 lines
Quoted 322 lines
Quoted 107 lines
Quoted 95 lines
drivers/crypto/ccp/sev-dev.c +95 -0
+       }
+
+       if (to_fw)
+               func = map_firmware_writeable;
+       else
+               func = unmap_firmware_writeable;
+
+       /*
+        * A command buffer may contains a system physical address. If the address
+        * points to a host memory then use an intermediate firmware page otherwise
+        * change the page state in the RMP table.
+        */
+       switch (cmd) {
+       case SEV_CMD_PDH_CERT_EXPORT:
+               if (prep_buffer(struct sev_data_pdh_cert_export, pdh_cert_address,
+                               pdh_cert_len, false, &sev->snp_host_map[0]))
+                       goto err;
+               if (prep_buffer(struct sev_data_pdh_cert_export, cert_chain_address,
+                               cert_chain_len, false, &sev->snp_host_map[1]))
+                       goto err;
+               break;
+       case SEV_CMD_GET_ID:
+               if (prep_buffer(struct sev_data_get_id, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_PEK_CSR:
+               if (prep_buffer(struct sev_data_pek_csr, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_UPDATE_DATA:
+               if (prep_buffer(struct sev_data_launch_update_data, address, len,
+                               true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_UPDATE_VMSA:
+               if (prep_buffer(struct sev_data_launch_update_vmsa, address, len,
+                               true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_MEASURE:
+               if (prep_buffer(struct sev_data_launch_measure, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_UPDATE_SECRET:
+               if (prep_buffer(struct sev_data_launch_secret, guest_address, guest_len,
+                               true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_DBG_DECRYPT:
+               if (prep_buffer(struct sev_data_dbg, dst_addr, len, false,
+                               &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_DBG_ENCRYPT:
+               if (prep_buffer(struct sev_data_dbg, dst_addr, len, true,
+                               &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_ATTESTATION_REPORT:
+               if (prep_buffer(struct sev_data_attestation_report, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_SEND_START:
+               if (prep_buffer(struct sev_data_send_start, session_address,
+                               session_len, false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_SEND_UPDATE_DATA:
+               if (prep_buffer(struct sev_data_send_update_data, hdr_address, hdr_len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               if (prep_buffer(struct sev_data_send_update_data, trans_address,
+                               trans_len, false, &sev->snp_host_map[1]))
+                       goto err;
+               break;
+       case SEV_CMD_SEND_UPDATE_VMSA:
+               if (prep_buffer(struct sev_data_send_update_vmsa, hdr_address, hdr_len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               if (prep_buffer(struct sev_data_send_update_vmsa, trans_address,
+                               trans_len, false, &sev->snp_host_map[1]))
+                       goto err;
+               break;
+       case SEV_CMD_RECEIVE_UPDATE_DATA:
+               if (prep_buffer(struct sev_data_receive_update_data, guest_address,
+                               guest_len, true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_RECEIVE_UPDATE_VMSA:
+               if (prep_buffer(struct sev_data_receive_update_vmsa, guest_address,
+                               guest_len, true, &sev->snp_host_map[0]))

drivers/crypto/ccp/sev-dev.c static int sev_write_init_ex_file_if_required(int cmd_id) · line 769

Quoted 10 lines
drivers/crypto/ccp/sev-dev.c +10 -0
+                       goto err;
+               break;
+       default:
+               break;
+       }
+
+       /* The command buffer need to be in the firmware state. */
+       if (to_fw && sev_legacy_cmd_buf_writable(cmd)) {
+               if (rmp_mark_pages_firmware(__pa(cmd_buf), 1, true))
+                       return -EFAULT;

This function moves two separate pages to firmware state. First

Quoted 9 lines

calling map_firmware_writeable and second calling rmp_mark_pages_firmware for cmd_buf.
In case rmp_mark_pages_firmware fails for cmd_buf, the page which has already moved to firmware state in map_firmware_writeable should be reclaimed.
This is a problem especially if we leak a guest owned page in firmware state. Since this is used only by legacy SEV VMs, these leaked pages will never be reclaimed back when destroying these VMs.

Yes, this looks to be an inherent issue with the original patch, as you mentioned there are two pages - guest owned page and the HV cmd_buf, and failure to transition the cmd_buf back to HV/shared state has no corresponding recovery/reclaim for the transitioned guest page.

Thanks,
Ashish

Quoted 197 lines
Quoted 197 lines
drivers/crypto/ccp/sev-dev.c +80 -10
+       }
+
+       return 0;
+
+err:
+       return -EINVAL;
+}
+
+static inline bool need_firmware_copy(int cmd)
+{
+       struct sev_device *sev = psp_master->sev_data;
+
+       /* After SNP is INIT'ed, the behavior of legacy SEV command is changed. */
+       return ((cmd < SEV_CMD_SNP_INIT) && sev->snp_initialized) ? true : false;
+}
+
+static int snp_aware_copy_to_firmware(int cmd, void *data)
+{
+       return __snp_cmd_buf_copy(cmd, data, true, 0);
+}
+
+static int snp_aware_copy_from_firmware(int cmd, void *data, int fw_err)
+{
+       return __snp_cmd_buf_copy(cmd, data, false, fw_err);
+}
+
   static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
   {
          struct psp_device *psp = psp_master;
          struct sev_device *sev;
          unsigned int phys_lsb, phys_msb;
          unsigned int reg, ret = 0;
+       void *cmd_buf;
          int buf_len;
 
          if (!psp || !psp->sev_data)
@@ -512,12 +819,28 @@ static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
           * work for some memory, e.g. vmalloc'd addresses, and @data may not be
           * physically contiguous.
           */
-       if (data)
-               memcpy(sev->cmd_buf, data, buf_len);
+       if (data) {
+               if (sev->cmd_buf_active > 2)
+                       return -EBUSY;
+
+               cmd_buf = sev->cmd_buf_active ? sev->cmd_buf_backup : sev->cmd_buf;
+
+               memcpy(cmd_buf, data, buf_len);
+               sev->cmd_buf_active++;
+
+               /*
+                * The behavior of the SEV-legacy commands is altered when the
+                * SNP firmware is in the INIT state.
+                */
+               if (need_firmware_copy(cmd) && snp_aware_copy_to_firmware(cmd, sev->cmd_buf))
+                       return -EFAULT;
+       } else {
+               cmd_buf = sev->cmd_buf;
+       }
 
          /* Get the physical address of the command buffer */
-       phys_lsb = data ? lower_32_bits(__psp_pa(sev->cmd_buf)) : 0;
-       phys_msb = data ? upper_32_bits(__psp_pa(sev->cmd_buf)) : 0;
+       phys_lsb = data ? lower_32_bits(__psp_pa(cmd_buf)) : 0;
+       phys_msb = data ? upper_32_bits(__psp_pa(cmd_buf)) : 0;
 
          dev_dbg(sev->dev, "sev command id %#x buffer 0x%08x%08x timeout %us\n",
                  cmd, phys_msb, phys_lsb, psp_timeout);
@@ -560,15 +883,24 @@ static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
                  ret = sev_write_init_ex_file_if_required(cmd);
          }
 
-       print_hex_dump_debug("(out): ", DUMP_PREFIX_OFFSET, 16, 2, data,
-                            buf_len, false);
-
          /*
           * Copy potential output from the PSP back to data.  Do this even on
           * failure in case the caller wants to glean something from the error.
           */
-       if (data)
-               memcpy(data, sev->cmd_buf, buf_len);
+       if (data) {
+               /*
+                * Restore the page state after the command completes.
+                */
+               if (need_firmware_copy(cmd) &&
+                   snp_aware_copy_from_firmware(cmd, cmd_buf, ret))
+                       return -EFAULT;
+
+               memcpy(data, cmd_buf, buf_len);
+               sev->cmd_buf_active--;
+       }
+
+       print_hex_dump_debug("(out): ", DUMP_PREFIX_OFFSET, 16, 2, data,
+                            buf_len, false);
 
          return ret;
   }
@@ -1579,10 +1911,12 @@ int sev_dev_init(struct psp_device *psp)
          if (!sev)
                  goto e_err;
 
-       sev->cmd_buf = (void *)devm_get_free_pages(dev, GFP_KERNEL, 0);
+       sev->cmd_buf = (void *)devm_get_free_pages(dev, GFP_KERNEL, 1);
          if (!sev->cmd_buf)
                  goto e_sev;
 
+       sev->cmd_buf_backup = (uint8_t *)sev->cmd_buf + PAGE_SIZE;
+
          psp->sev_data = sev;
 
          sev->dev = dev;
@@ -1648,6 +1982,12 @@ static void sev_firmware_shutdown(struct sev_device *sev)
                  snp_range_list = NULL;
          }
 
+       /*
+        * The host map need to clear the immutable bit so it must be free'd before the
+        * SNP firmware shutdown.
+        */
+       free_snp_host_map(sev);
+
          sev_snp_shutdown(&error);
   }
 
@@ -1722,6 +2062,14 @@ void sev_pci_init(void)
                                  dev_err(sev->dev, "SEV-SNP: failed to INIT error %#x\n", error);
                          }
                  }
+
+               /*
+                * Allocate the intermediate buffers used for the legacy command handling.
+                */
+               if (alloc_snp_host_map(sev)) {
+                       dev_notice(sev->dev, "Failed to alloc host map (disabling legacy SEV)\n");
+                       goto skip_legacy;
+               }
          }
 
          /* Obtain the TMR memory area for SEV-ES use */
@@ -1739,12 +2087,14 @@ void sev_pci_init(void)
                  dev_err(sev->dev, "SEV: failed to INIT error %#x, rc %d\n",
                          error, rc);
 
+skip_legacy:
          dev_info(sev->dev, "SEV%s API:%d.%d build:%d\n", sev->snp_initialized ?
                  "-SNP" : "", sev->api_major, sev->api_minor, sev->build);
 
          return;
 
   err:
+       free_snp_host_map(sev);
          psp_master->sev_data = NULL;
   }
 
drivers/crypto/ccp/sev-dev.h +12 -1
index 34767657beb5..19d79f9d4212 100644
@@ -29,11 +29,20 @@
   #define SEV_CMDRESP_CMD_SHIFT          16
   #define SEV_CMDRESP_IOC                        BIT(0)
 
+#define MAX_SNP_HOST_MAP_BUFS          2
+
   struct sev_misc_dev {
          struct kref refcount;
          struct miscdevice misc;
   };
 
+struct snp_host_map {
+       u64 paddr;
+       u32 len;
+       void *host;
+       bool active;
+};
+
   struct sev_device {
          struct device *dev;
          struct psp_device *psp;
@@ -52,8 +61,11 @@ struct sev_device {
          u8 build;
 
          void *cmd_buf;
+       void *cmd_buf_backup;
+       int cmd_buf_active;
 
          bool snp_initialized;
+       struct snp_host_map snp_host_map[MAX_SNP_HOST_MAP_BUFS];
   };
 
   int sev_dev_init(struct psp_device *psp);
--

2.25.1

#110 lore raw

Re: [PATCH RFC v7 29/64] crypto: ccp: Handle the legacy SEV command when SNP is enabled

In reply to Michael Roth
Quoted 369 lines · Michael Roth

From: Brijesh Singh <[email protected]>

The behavior of the SEV-legacy commands is altered when the SNP firmware is in the INIT state. When SNP is in INIT state, all the SEV-legacy commands that cause the firmware to write to memory must be in the firmware state before issuing the command..

A command buffer may contains a system physical address that the firmware may write to. There are two cases that need to be handled:

1) system physical address points to a guest memory
2) system physical address points to a host memory

To handle the case #1, change the page state to the firmware in the RMP table before issuing the command and restore the state to shared after the command completes.

For the case #2, use a bounce buffer to complete the request.

---
 drivers/crypto/ccp/sev-dev.c | 370 ++++++++++++++++++++++++++++++++++-
 drivers/crypto/ccp/sev-dev.h |  12 ++
 2 files changed, 372 insertions(+), 10 deletions(-)
drivers/crypto/ccp/sev-dev.c +311 -2
index 4c12e98a1219..5eb2e8f364d4 100644
@@ -286,6 +286,30 @@ static int rmp_mark_pages_firmware(unsigned long paddr, unsigned int npages, boo
        return rc;
 }
 
+static int rmp_mark_pages_shared(unsigned long paddr, unsigned int npages)
+{
+       /* Cbit maybe set in the paddr */
+       unsigned long pfn = __sme_clr(paddr) >> PAGE_SHIFT;
+       int rc, n = 0, i;
+
+       for (i = 0; i < npages; i++, pfn++, n++) {
+               rc = rmp_make_shared(pfn, PG_LEVEL_4K);
+               if (rc)
+                       goto cleanup;
+       }
+
+       return 0;
+
+cleanup:
+       /*
+        * If failed to change the page state to shared, then its not safe
+        * to release the page back to the system, leak it.
+        */
+       snp_mark_pages_offline(pfn, npages - n);
+
+       return rc;
+}
+
 static struct page *__snp_alloc_firmware_pages(gfp_t gfp_mask, int order, bool locked)
 {
        unsigned long npages = 1ul << order, paddr;
@@ -487,12 +511,295 @@ static int sev_write_init_ex_file_if_required(int cmd_id)
        return sev_write_init_ex_file();
 }
 
+static int alloc_snp_host_map(struct sev_device *sev)
+{
+       struct page *page;
+       int i;
+
+       for (i = 0; i < MAX_SNP_HOST_MAP_BUFS; i++) {
+               struct snp_host_map *map = &sev->snp_host_map[i];
+
+               memset(map, 0, sizeof(*map));
+
+               page = alloc_pages(GFP_KERNEL_ACCOUNT, get_order(SEV_FW_BLOB_MAX_SIZE));
+               if (!page)
+                       return -ENOMEM;
+
+               map->host = page_address(page);
+       }
+
+       return 0;
+}
+
+static void free_snp_host_map(struct sev_device *sev)
+{
+       int i;
+
+       for (i = 0; i < MAX_SNP_HOST_MAP_BUFS; i++) {
+               struct snp_host_map *map = &sev->snp_host_map[i];
+
+               if (map->host) {
+                       __free_pages(virt_to_page(map->host), get_order(SEV_FW_BLOB_MAX_SIZE));
+                       memset(map, 0, sizeof(*map));
+               }
+       }
+}
+
+static int map_firmware_writeable(u64 *paddr, u32 len, bool guest, struct snp_host_map *map)
+{
+       unsigned int npages = PAGE_ALIGN(len) >> PAGE_SHIFT;
+
+       map->active = false;
+
+       if (!paddr || !len)
+               return 0;
+
+       map->paddr = *paddr;
+       map->len = len;
+
+       /* If paddr points to a guest memory then change the page state to firmwware. */
+       if (guest) {
+               if (rmp_mark_pages_firmware(*paddr, npages, true))
+                       return -EFAULT;
+
+               goto done;
+       }
+
+       if (!map->host)
+               return -ENOMEM;
+
+       /* Check if the pre-allocated buffer can be used to fullfil the request. */
+       if (len > SEV_FW_BLOB_MAX_SIZE)
+               return -EINVAL;
+
+       /* Transition the pre-allocated buffer to the firmware state. */
+       if (rmp_mark_pages_firmware(__pa(map->host), npages, true))
+               return -EFAULT;
+
+       /* Set the paddr to use pre-allocated firmware buffer */
+       *paddr = __psp_pa(map->host);
+
+done:
+       map->active = true;
+       return 0;
+}
+
+static int unmap_firmware_writeable(u64 *paddr, u32 len, bool guest, struct snp_host_map *map)
+{
+       unsigned int npages = PAGE_ALIGN(len) >> PAGE_SHIFT;
+
+       if (!map->active)
+               return 0;
+
+       /* If paddr points to a guest memory then restore the page state to hypervisor. */
+       if (guest) {
+               if (snp_reclaim_pages(*paddr, npages, true))
+                       return -EFAULT;
+
+               goto done;
+       }
+
+       /*
+        * Transition the pre-allocated buffer to hypervisor state before the access.
+        *
+        * This is because while changing the page state to firmware, the kernel unmaps
+        * the pages from the direct map, and to restore the direct map the pages must
+        * be transitioned back to the shared state.
+        */
+       if (snp_reclaim_pages(__pa(map->host), npages, true))
+               return -EFAULT;
+
+       /* Copy the response data firmware buffer to the callers buffer. */
+       memcpy(__va(__sme_clr(map->paddr)), map->host, min_t(size_t, len, map->len));
+       *paddr = map->paddr;
+
+done:
+       map->active = false;
+       return 0;
+}
+
+static bool sev_legacy_cmd_buf_writable(int cmd)
+{
+       switch (cmd) {
+       case SEV_CMD_PLATFORM_STATUS:
+       case SEV_CMD_GUEST_STATUS:
+       case SEV_CMD_LAUNCH_START:
+       case SEV_CMD_RECEIVE_START:
+       case SEV_CMD_LAUNCH_MEASURE:
+       case SEV_CMD_SEND_START:
+       case SEV_CMD_SEND_UPDATE_DATA:
+       case SEV_CMD_SEND_UPDATE_VMSA:
+       case SEV_CMD_PEK_CSR:
+       case SEV_CMD_PDH_CERT_EXPORT:
+       case SEV_CMD_GET_ID:
+       case SEV_CMD_ATTESTATION_REPORT:
+               return true;
+       default:
+               return false;
+       }
+}
+
+#define prep_buffer(name, addr, len, guest, map) \
+       func(&((typeof(name *))cmd_buf)->addr, ((typeof(name *))cmd_buf)->len, guest, map)
+
+static int __snp_cmd_buf_copy(int cmd, void *cmd_buf, bool to_fw, int fw_err)
+{
+       int (*func)(u64 *paddr, u32 len, bool guest, struct snp_host_map *map);
+       struct sev_device *sev = psp_master->sev_data;
+       bool from_fw = !to_fw;
+
+       /*
+        * After the command is completed, change the command buffer memory to
+        * hypervisor state.
+        *
+        * The immutable bit is automatically cleared by the firmware, so
+        * no not need to reclaim the page.
+        */
+       if (from_fw && sev_legacy_cmd_buf_writable(cmd)) {
+               if (rmp_mark_pages_shared(__pa(cmd_buf), 1))
+                       return -EFAULT;
+
+               /* No need to go further if firmware failed to execute command. */
+               if (fw_err)
+                       return 0;
+       }
+
+       if (to_fw)
+               func = map_firmware_writeable;
+       else
+               func = unmap_firmware_writeable;
+
+       /*
+        * A command buffer may contains a system physical address. If the address
+        * points to a host memory then use an intermediate firmware page otherwise
+        * change the page state in the RMP table.
+        */
+       switch (cmd) {
+       case SEV_CMD_PDH_CERT_EXPORT:
+               if (prep_buffer(struct sev_data_pdh_cert_export, pdh_cert_address,
+                               pdh_cert_len, false, &sev->snp_host_map[0]))
+                       goto err;
+               if (prep_buffer(struct sev_data_pdh_cert_export, cert_chain_address,
+                               cert_chain_len, false, &sev->snp_host_map[1]))
+                       goto err;
+               break;
+       case SEV_CMD_GET_ID:
+               if (prep_buffer(struct sev_data_get_id, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_PEK_CSR:
+               if (prep_buffer(struct sev_data_pek_csr, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_UPDATE_DATA:
+               if (prep_buffer(struct sev_data_launch_update_data, address, len,
+                               true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_UPDATE_VMSA:
+               if (prep_buffer(struct sev_data_launch_update_vmsa, address, len,
+                               true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_MEASURE:
+               if (prep_buffer(struct sev_data_launch_measure, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_LAUNCH_UPDATE_SECRET:
+               if (prep_buffer(struct sev_data_launch_secret, guest_address, guest_len,
+                               true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_DBG_DECRYPT:
+               if (prep_buffer(struct sev_data_dbg, dst_addr, len, false,
+                               &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_DBG_ENCRYPT:
+               if (prep_buffer(struct sev_data_dbg, dst_addr, len, true,
+                               &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_ATTESTATION_REPORT:
+               if (prep_buffer(struct sev_data_attestation_report, address, len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_SEND_START:
+               if (prep_buffer(struct sev_data_send_start, session_address,
+                               session_len, false, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_SEND_UPDATE_DATA:
+               if (prep_buffer(struct sev_data_send_update_data, hdr_address, hdr_len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               if (prep_buffer(struct sev_data_send_update_data, trans_address,
+                               trans_len, false, &sev->snp_host_map[1]))
+                       goto err;
+               break;
+       case SEV_CMD_SEND_UPDATE_VMSA:
+               if (prep_buffer(struct sev_data_send_update_vmsa, hdr_address, hdr_len,
+                               false, &sev->snp_host_map[0]))
+                       goto err;
+               if (prep_buffer(struct sev_data_send_update_vmsa, trans_address,
+                               trans_len, false, &sev->snp_host_map[1]))
+                       goto err;
+               break;
+       case SEV_CMD_RECEIVE_UPDATE_DATA:
+               if (prep_buffer(struct sev_data_receive_update_data, guest_address,
+                               guest_len, true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       case SEV_CMD_RECEIVE_UPDATE_VMSA:
+               if (prep_buffer(struct sev_data_receive_update_vmsa, guest_address,
+                               guest_len, true, &sev->snp_host_map[0]))
+                       goto err;
+               break;
+       default:
+               break;
+       }
+
+       /* The command buffer need to be in the firmware state. */
+       if (to_fw && sev_legacy_cmd_buf_writable(cmd)) {
+               if (rmp_mark_pages_firmware(__pa(cmd_buf), 1, true))
+                       return -EFAULT;
+       }
+
+       return 0;
+
+err:
+       return -EINVAL;
+}
+
+static inline bool need_firmware_copy(int cmd)
+{
+       struct sev_device *sev = psp_master->sev_data;
+
+       /* After SNP is INIT'ed, the behavior of legacy SEV command is changed. */
+       return ((cmd < SEV_CMD_SNP_INIT) && sev->snp_initialized) ? true : false;
+}
+
+static int snp_aware_copy_to_firmware(int cmd, void *data)
+{
+       return __snp_cmd_buf_copy(cmd, data, true, 0);
+}
+
+static int snp_aware_copy_from_firmware(int cmd, void *data, int fw_err)
+{
+       return __snp_cmd_buf_copy(cmd, data, false, fw_err);
+}
+
 static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
 {
        struct psp_device *psp = psp_master;
        struct sev_device *sev;
        unsigned int phys_lsb, phys_msb;
        unsigned int reg, ret = 0;
+       void *cmd_buf;
        int buf_len;
 
        if (!psp || !psp->sev_data)
@@ -512,12 +819,28 @@ static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
         * work for some memory, e.g. vmalloc'd addresses, and @data may not be
         * physically contiguous.
         */
-       if (data)
-               memcpy(sev->cmd_buf, data, buf_len);
+       if (data) {
+               if (sev->cmd_buf_active > 2)
+                       return -EBUSY;
+

drivers/crypto/ccp/sev-dev.c static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret) · line 835

Quoted 10 lines
drivers/crypto/ccp/sev-dev.c +10 -0
+               cmd_buf = sev->cmd_buf_active ? sev->cmd_buf_backup : sev->cmd_buf;
+
+               memcpy(cmd_buf, data, buf_len);
+               sev->cmd_buf_active++;
+
+               /*
+                * The behavior of the SEV-legacy commands is altered when the
+                * SNP firmware is in the INIT state.
+                */
+               if (need_firmware_copy(cmd) && snp_aware_copy_to_firmware(cmd, sev->cmd_buf))

I believe this should be cmd_buf instead of sev->cmd_buf. snp_aware_copy_to_firmware(cmd, cmd_buf)

Quoted 140 lines
drivers/crypto/ccp/sev-dev.c +39 -8
+                       return -EFAULT;
+       } else {
+               cmd_buf = sev->cmd_buf;
+       }
 
        /* Get the physical address of the command buffer */
-       phys_lsb = data ? lower_32_bits(__psp_pa(sev->cmd_buf)) : 0;
-       phys_msb = data ? upper_32_bits(__psp_pa(sev->cmd_buf)) : 0;
+       phys_lsb = data ? lower_32_bits(__psp_pa(cmd_buf)) : 0;
+       phys_msb = data ? upper_32_bits(__psp_pa(cmd_buf)) : 0;
 
        dev_dbg(sev->dev, "sev command id %#x buffer 0x%08x%08x timeout %us\n",
                cmd, phys_msb, phys_lsb, psp_timeout);
@@ -560,15 +883,24 @@ static int __sev_do_cmd_locked(int cmd, void *data, int *psp_ret)
                ret = sev_write_init_ex_file_if_required(cmd);
        }
 
-       print_hex_dump_debug("(out): ", DUMP_PREFIX_OFFSET, 16, 2, data,
-                            buf_len, false);
-
        /*
         * Copy potential output from the PSP back to data.  Do this even on
         * failure in case the caller wants to glean something from the error.
         */
-       if (data)
-               memcpy(data, sev->cmd_buf, buf_len);
+       if (data) {
+               /*
+                * Restore the page state after the command completes.
+                */
+               if (need_firmware_copy(cmd) &&
+                   snp_aware_copy_from_firmware(cmd, cmd_buf, ret))
+                       return -EFAULT;
+
+               memcpy(data, cmd_buf, buf_len);
+               sev->cmd_buf_active--;
+       }
+
+       print_hex_dump_debug("(out): ", DUMP_PREFIX_OFFSET, 16, 2, data,
+                            buf_len, false);
 
        return ret;
 }
@@ -1579,10 +1911,12 @@ int sev_dev_init(struct psp_device *psp)
        if (!sev)
                goto e_err;
 
-       sev->cmd_buf = (void *)devm_get_free_pages(dev, GFP_KERNEL, 0);
+       sev->cmd_buf = (void *)devm_get_free_pages(dev, GFP_KERNEL, 1);
        if (!sev->cmd_buf)
                goto e_sev;
 
+       sev->cmd_buf_backup = (uint8_t *)sev->cmd_buf + PAGE_SIZE;
+
        psp->sev_data = sev;
 
        sev->dev = dev;
@@ -1648,6 +1982,12 @@ static void sev_firmware_shutdown(struct sev_device *sev)
                snp_range_list = NULL;
        }
 
+       /*
+        * The host map need to clear the immutable bit so it must be free'd before the
+        * SNP firmware shutdown.
+        */
+       free_snp_host_map(sev);
+
        sev_snp_shutdown(&error);
 }
 
@@ -1722,6 +2062,14 @@ void sev_pci_init(void)
                                dev_err(sev->dev, "SEV-SNP: failed to INIT error %#x\n", error);
                        }
                }
+
+               /*
+                * Allocate the intermediate buffers used for the legacy command handling.
+                */
+               if (alloc_snp_host_map(sev)) {
+                       dev_notice(sev->dev, "Failed to alloc host map (disabling legacy SEV)\n");
+                       goto skip_legacy;
+               }
        }
 
        /* Obtain the TMR memory area for SEV-ES use */
@@ -1739,12 +2087,14 @@ void sev_pci_init(void)
                dev_err(sev->dev, "SEV: failed to INIT error %#x, rc %d\n",
                        error, rc);
 
+skip_legacy:
        dev_info(sev->dev, "SEV%s API:%d.%d build:%d\n", sev->snp_initialized ?
                "-SNP" : "", sev->api_major, sev->api_minor, sev->build);
 
        return;
 
 err:
+       free_snp_host_map(sev);
        psp_master->sev_data = NULL;
 }
 
drivers/crypto/ccp/sev-dev.h +12 -1
index 34767657beb5..19d79f9d4212 100644
@@ -29,11 +29,20 @@
 #define SEV_CMDRESP_CMD_SHIFT          16
 #define SEV_CMDRESP_IOC                        BIT(0)
 
+#define MAX_SNP_HOST_MAP_BUFS          2
+
 struct sev_misc_dev {
        struct kref refcount;
        struct miscdevice misc;
 };
 
+struct snp_host_map {
+       u64 paddr;
+       u32 len;
+       void *host;
+       bool active;
+};
+
 struct sev_device {
        struct device *dev;
        struct psp_device *psp;
@@ -52,8 +61,11 @@ struct sev_device {
        u8 build;
 
        void *cmd_buf;
+       void *cmd_buf_backup;
+       int cmd_buf_active;
 
        bool snp_initialized;
+       struct snp_host_map snp_host_map[MAX_SNP_HOST_MAP_BUFS];
 };
 
 int sev_dev_init(struct psp_device *psp);
--

2.25.1