Skip to contentSkip to conversation

Add PCI pass-thru support to Hyper-V Confidential VMs

PATCH Superseded Michael Kelley 6 participants 39 messages Dec 2, 2022 – Jan 20, 2023

Subsystems (10)
Networking Network drivers x86 PCI +6 more subsystems

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v4 cover Add PCI pass-thru support to Hyper-V Confidential VMs

This patch series adds support for PCI pass-thru devices to Hyper-V Confidential VMs (also called "Isolation VMs"). But in preparation, it first changes how private (encrypted) vs. shared (decrypted) memory is handled in Hyper-V SEV-SNP guest VMs. The new approach builds on the confidential computing (coco) mechanisms introduced in the 5.19 kernel for TDX support and significantly reduces the amount of Hyper-V specific code. Furthermore, with this new approach a proposed RFC patch set for generic DMA layer functionality[1] is no longer necessary.

Background
==========
Hyper-V guests on AMD SEV-SNP hardware have the option of using the "virtual Top Of Memory" (vTOM) feature specified by the SEV-SNP architecture. With vTOM, shared vs. private memory accesses are controlled by splitting the guest physical address space into two halves. vTOM is the dividing line where the uppermost bit of the physical address space is set; e.g., with 47 bits of guest physical address space, vTOM is 0x400000000000 (bit 46 is set). Guest physical memory is accessible at two parallel physical addresses -- one below vTOM and one above vTOM. Accesses below vTOM are private (encrypted) while accesses above vTOM are shared (decrypted). In this sense, vTOM is like the GPA.SHARED bit in Intel TDX.

In Hyper-V's use of vTOM, the normal guest OS runs at VMPL2, while a Hyper-V provided "paravisor" runs at VMPL0 in the guest VM. (VMPL is Virtual Machine Privilege Level. See AMD's SEV-SNP spec for more details.) The paravisor provides emulation for various system devices like the IO-APIC as part of the guest VM. Accesses to such devices made by the normal guest OS trap to the paravisor and are emulated in the guest VM context instead of in the Hyper-V host. This emulation is invisible to the normal guest OS, but with the quirk that memory mapped I/O accesses to these devices must be treated as private, not shared as would be the case for other device accesses.

Support for Hyper-V guests using vTOM was added to the Linux kernel in two patch sets[2][3]. This support treats the vTOM bit as part of the physical address. For accessing shared (decrypted) memory, the core approach is to create a second kernel virtual mapping that maps to parallel physical addresses above vTOM, while leaving the original mapping unchanged. Most of the code for creating that second virtual mapping is confined to Hyper-V specific areas, but there are are also changes to generic swiotlb code.

Changes in this patch set
=========================
In preparation for supporting PCI pass-thru devices, this patch set changes the core approach for handling vTOM. In the new approach, the vTOM bit is treated as a protection flag, and not as part of the physical address. This new approach is like the approach for the GPA.SHARED bit in Intel TDX. Furthermore, there's no need to create a second kernel virtual mapping. When memory is changed between private and shared using set_memory_decrypted() and set_memory_encrypted(), the PTEs for the existing kernel mapping are changed to add or remove the vTOM bit just as with TDX. The hypercalls to change the memory status on the host side are made using the existing callback mechanism. Everything just works, with a minor tweak to map the IO-APIC to use private accesses as mentioned above.

With the new handling of vTOM in place, existing Hyper-V code that creates the second kernel virtual mapping still works, but it is now redundant as the original kernel virtual mapping (as updated) maps to the same physical address. To simplify things going forward, this patch set removes the code that creates the second kernel virtual mapping. And since a second kernel virtual mapping is no longer needed, changes to the DMA layer proposed as an RFC[1] are no longer needed.

Finally, to support PCI pass-thru in a Confidential VM, Hyper-V requires that all accesses to PCI config space be emulated using a hypercall. This patch set adds functions to invoke those hypercalls and uses them in the config space access functions in the Hyper-V PCI driver. Lastly, the Hyper-V PCI driver is marked as allowed to be used in a Confidential VM. The Hyper-V PCI driver has been hardened against a malicious Hyper-V in a previous patch set.[4]

Patch Organization
==================
Patches 1 thru 5 are prepatory patches that account for slightly different assumptions when running in a Hyper-V VM with vTOM, fix some minor bugs, and make temporary tweaks to avoid needing a single large patch to make the transition from the old approach to the new approach.

Patch 6 enables the new approach to handling vTOM for Hyper-V guest VMs. This is the core patch after which the new approach is in effect.

Patches 7 thru 10 remove existing code for creating the second kernel virtual mapping that is no longer necessary with the new approach.

Patch 11 updates existing code so that it no longer assumes that the vTOM bit is part of the physical address.

Patches 12 and 13 add new hypercalls for accessing MMIO space and use those hypercalls for PCI config space. They also enable the Hyper-V vPCI driver to be used in a Confidential VM.

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://lore.kernel.org/all/[email protected]/

---

Changes in v4:
* Remove previous Patch 1 from this series and submit separately [Dave Hansen & Boris Petkov]

* Patch 1: Change the name of the new CC_ATTR that controls whether the IO-APIC is mapped decrypted [Boris Petkov]

* Patch 4: Use sme_me_mask directly instead of calling the getter function. Add Fixes: tag. [Tom Lendacky]

* Patch 6: Remove CC_VENDOR_HYPERV and merge associated vTOM functionality under CC_VENDOR_AMD. [Boris Petkov]

* Patch 8: Use bitwise OR to pick up the vTOM bit in shared_gpa_boundary rather than adding it

Changes in v3:
* Patch 1: Tweak the code fix to cleanly separate the page alignment and physical address masking [Dave Hansen]

* Patch 2: Change the name of the new CC_ATTR that controls whether the IO-APIC is mapped decrypted [Dave Hansen]

* Patch 5 (now patch 7): Add CC_ATTR_MEM_ENCRYPT to what Hyper-V vTOM reports as 'true'. With the addition, Patches 5 and 6 are new to accomodate working correctly with Hyper-V VMs using vTOM. [Tom Lendacky]

Changes in v2:
* Patch 11: Include more detail in the error message if an MMIO hypercall fails. [Bjorn Helgaas]

* Patch 12: Restore removed memory barriers. It seems like these barriers should not be needed because of the spin_unlock() calls, but commit bdd74440d9e8 indicates that they are. This patch series will leave the barriers unchanged; whether they are really needed can be sorted out separately. [Boqun Feng]

Michael Kelley (13):
x86/ioapic: Gate decrypted mapping on cc_platform_has() attribute x86/hyperv: Reorder code in prep for subsequent patch
Drivers: hv: Explicitly request decrypted in vmap_pfn() calls x86/mm: Handle decryption/re-encryption of bss_decrypted consistently
init: Call mem_encrypt_init() after Hyper-V hypercall init is done x86/hyperv: Change vTOM handling to use standard coco mechanisms
swiotlb: Remove bounce buffer remapping for Hyper-V
Drivers: hv: vmbus: Remove second mapping of VMBus monitor pages
Drivers: hv: vmbus: Remove second way of mapping ring buffers
hv_netvsc: Remove second mapping of send and recv buffers
Drivers: hv: Don't remap addresses that are above shared_gpa_boundary
PCI: hv: Add hypercalls to read/write MMIO space
PCI: hv: Enable PCI pass-thru devices in Confidential VMs

 arch/x86/coco/core.c                |  37 ++++--
 arch/x86/hyperv/hv_init.c           |  18 +--
 arch/x86/hyperv/ivm.c               | 128 ++++++++++----------
 arch/x86/include/asm/coco.h         |   1 -
 arch/x86/include/asm/hyperv-tlfs.h  |   3 +
 arch/x86/include/asm/mshyperv.h     |   8 +-
 arch/x86/include/asm/msr-index.h    |   1 +
 arch/x86/kernel/apic/io_apic.c      |   3 +-
 arch/x86/kernel/cpu/mshyperv.c      |  22 ++--
 arch/x86/mm/mem_encrypt_amd.c       |  10 +-
 arch/x86/mm/pat/set_memory.c        |   3 -
 drivers/hv/Kconfig                  |   1 -
 drivers/hv/channel_mgmt.c           |   2 +-
 drivers/hv/connection.c             | 113 +++++-------------
 drivers/hv/hv.c                     |  23 ++--
 drivers/hv/hv_common.c              |  11 --
 drivers/hv/hyperv_vmbus.h           |   2 -
 drivers/hv/ring_buffer.c            |  62 ++++------
 drivers/net/hyperv/hyperv_net.h     |   2 -
 drivers/net/hyperv/netvsc.c         |  48 +-------
 drivers/pci/controller/pci-hyperv.c | 232 ++++++++++++++++++++++++++----------
 include/asm-generic/hyperv-tlfs.h   |  22 ++++
 include/asm-generic/mshyperv.h      |   2 -
 include/linux/cc_platform.h         |  12 ++
 include/linux/swiotlb.h             |   2 -
 init/main.c                         |  19 +--
 kernel/dma/swiotlb.c                |  45 +------
 27 files changed, 398 insertions(+), 434 deletions(-)
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH v4 1/13 x86/ioapic: Gate decrypted mapping on cc_platform_has() attribute

Current code always maps the IO-APIC as shared (decrypted) in a confidential VM. But Hyper-V guest VMs on AMD SEV-SNP with vTOM enabled use a paravisor running in VMPL0 to emulate the IO-APIC. In such a case, the IO-APIC must be accessed as private (encrypted).

Fix this by gating the IO-APIC decrypted mapping on a new cc_platform_has() attribute that a subsequent patch in the series will set only for guests using vTOM.

---
 arch/x86/kernel/apic/io_apic.c |  3 ++-
 include/linux/cc_platform.h    | 12 ++++++++++++
 2 files changed, 14 insertions(+), 1 deletion(-)
arch/x86/kernel/apic/io_apic.c +2 -1
index a868b76..2b70e2e 100644
@@ -2686,7 +2686,8 @@ static void io_apic_set_fixmap(enum fixed_addresses idx, phys_addr_t phys)
 	 * Ensure fixmaps for IOAPIC MMIO respect memory encryption pgprot
 	 * bits, just like normal ioremap():
 	 */
-	flags = pgprot_decrypted(flags);
+	if (!cc_platform_has(CC_ATTR_ACCESS_IOAPIC_ENCRYPTED))
+		flags = pgprot_decrypted(flags);
 
 	__set_fixmap(idx, phys, flags);
 }
include/linux/cc_platform.h +12 -0
index cb0d6cd..7b63a7d 100644
@@ -90,6 +90,18 @@ enum cc_attr {
 	 * Examples include TDX Guest.
 	 */
 	CC_ATTR_HOTPLUG_DISABLED,
+
+	/**
+	 * @CC_ATTR_ACCESS_IOAPIC_ENCRYPTED: Guest VM IO-APIC is encrypted
+	 *
+	 * The platform/OS is running as a guest/virtual machine with
+	 * an IO-APIC that is emulated by a paravisor running in the
+	 * guest VM context. As such, the IO-APIC is accessed in the
+	 * encrypted portion of the guest physical address space.
+	 *
+	 * Examples include Hyper-V SEV-SNP guests using vTOM.
+	 */
+	CC_ATTR_ACCESS_IOAPIC_ENCRYPTED,
 };
 
 #ifdef CONFIG_ARCH_HAS_CC_PLATFORM
Signature · 1 line
1.8.3.1
maintainer
#2 lore raw

Re: [Patch v4 01/13] x86/ioapic: Gate decrypted mapping on cc_platform_has() attribute

Quoted 4 lines · Michael Kelley

Current code always maps the IO-APIC as shared (decrypted) in a confidential VM. But Hyper-V guest VMs on AMD SEV-SNP with vTOM enabled use a paravisor running in VMPL0 to emulate the IO-APIC. In such a case, the IO-APIC must be accessed as private (encrypted).

Lemme see I understand this correctly:

the paravisor is emulating the IO-APIC in the lower range of the address space, under the vTOM which is accessed encrypted.

That's why you need to access it encrypted in the guest.

Close?

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#3 lore raw

RE: [Patch v4 01/13] x86/ioapic: Gate decrypted mapping on cc_platform_has() attribute

From: Borislav Petkov <[email protected]> Sent: Tuesday, December 6, 2022 11:23 AM

Quoted 7 lines · Michael Kelley
Quoted 4 lines

Current code always maps the IO-APIC as shared (decrypted) in a confidential VM. But Hyper-V guest VMs on AMD SEV-SNP with vTOM enabled use a paravisor running in VMPL0 to emulate the IO-APIC. In such a case, the IO-APIC must be accessed as private (encrypted).

Quoted 9 lines

Lemme see I understand this correctly:

the paravisor is emulating the IO-APIC in the lower range of the address space, under the vTOM which is accessed encrypted.

That's why you need to access it encrypted in the guest.

Close?

Exactly correct.

Michael

maintainer
#4 lore raw

Re: [Patch v4 01/13] x86/ioapic: Gate decrypted mapping on cc_platform_has() attribute

Quoted 1 line · Michael Kelley (LINUX)

Exactly correct.

Ok, thanks.

Let's put that in the commit message and get rid of the "subsequent patch" wording as patch order in git is ambiguous.

IOW, something like this:

Current code always maps the IO-APIC as shared (decrypted) in a confidential VM. But Hyper-V guest VMs on AMD SEV-SNP with vTOM enabled use a paravisor running in VMPL0 to emulate the IO-APIC.

In such a case, the IO-APIC must be accessed as private (encrypted) because the paravisor emulates the IO-APIC in the lower half of the vTOM where all accesses must be encrypted.

Add a new CC attribute which determines how the IO-APIC MMIO mapping should be established depending on the platform the kernel is running on as a guest.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#5 lore raw

RE: [Patch v4 01/13] x86/ioapic: Gate decrypted mapping on cc_platform_has() attribute

From: Borislav Petkov <[email protected]> Sent: Thursday, December 29, 2022 3:39 AM

Quoted 15 lines · Michael Kelley (LINUX)
Quoted 1 line

Exactly correct.

Ok, thanks.

Let's put that in the commit message and get rid of the "subsequent patch" wording as patch order in git is ambiguous.

IOW, something like this:

Current code always maps the IO-APIC as shared (decrypted) in a confidential VM. But Hyper-V guest VMs on AMD SEV-SNP with vTOM enabled use a paravisor running in VMPL0 to emulate the IO-APIC.

Quoted 8 lines

In such a case, the IO-APIC must be accessed as private (encrypted) because the paravisor emulates the IO-APIC in the lower half of the vTOM where all accesses must be encrypted.

Add a new CC attribute which determines how the IO-APIC MMIO mapping should be established depending on the platform the kernel is running on as a guest.

Works for me. I'll adopt this wording in v5.

Michael

Reviewed-by on v4 1/13
#6 lore raw

Re: [Patch v4 01/13] x86/ioapic: Gate decrypted mapping on cc_platform_has() attribute

In reply to Michael Kelley
Quoted 5 lines · Michael Kelley

Current code always maps the IO-APIC as shared (decrypted) in a confidential VM. But Hyper-V guest VMs on AMD SEV-SNP with vTOM enabled use a paravisor running in VMPL0 to emulate the IO-APIC. In such a case, the IO-APIC must be accessed as private (encrypted).

Quoted 7 lines

Fix this by gating the IO-APIC decrypted mapping on a new cc_platform_has() attribute that a subsequent patch in the series will set only for guests using vTOM.

---

Looks good to me.

Quoted 41 lines
 arch/x86/kernel/apic/io_apic.c |  3 ++-
 include/linux/cc_platform.h    | 12 ++++++++++++
 2 files changed, 14 insertions(+), 1 deletion(-)
arch/x86/kernel/apic/io_apic.c +2 -1
index a868b76..2b70e2e 100644
@@ -2686,7 +2686,8 @@ static void io_apic_set_fixmap(enum fixed_addresses idx, phys_addr_t phys)
 	 * Ensure fixmaps for IOAPIC MMIO respect memory encryption pgprot
 	 * bits, just like normal ioremap():
 	 */
-	flags = pgprot_decrypted(flags);
+	if (!cc_platform_has(CC_ATTR_ACCESS_IOAPIC_ENCRYPTED))
+		flags = pgprot_decrypted(flags);
 
 	__set_fixmap(idx, phys, flags);
 }
include/linux/cc_platform.h +12 -0
index cb0d6cd..7b63a7d 100644
@@ -90,6 +90,18 @@ enum cc_attr {
 	 * Examples include TDX Guest.
 	 */
 	CC_ATTR_HOTPLUG_DISABLED,
+
+	/**
+	 * @CC_ATTR_ACCESS_IOAPIC_ENCRYPTED: Guest VM IO-APIC is encrypted
+	 *
+	 * The platform/OS is running as a guest/virtual machine with
+	 * an IO-APIC that is emulated by a paravisor running in the
+	 * guest VM context. As such, the IO-APIC is accessed in the
+	 * encrypted portion of the guest physical address space.
+	 *
+	 * Examples include Hyper-V SEV-SNP guests using vTOM.
+	 */
+	CC_ATTR_ACCESS_IOAPIC_ENCRYPTED,
 };
 
 #ifdef CONFIG_ARCH_HAS_CC_PLATFORM
Signature · 2 lines
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
#7 lore raw

PATCH v4 2/13 x86/hyperv: Reorder code in prep for subsequent patch

In reply to Michael Kelley

Reorder some code as preparation for a subsequent patch. No functional change.

---
 arch/x86/hyperv/ivm.c | 68 +++++++++++++++++++++++++--------------------------
 1 file changed, 34 insertions(+), 34 deletions(-)
arch/x86/hyperv/ivm.c +34 -34
index 1dbcbd9..f33c67e 100644
@@ -235,40 +235,6 @@ void hv_ghcb_msr_read(u64 msr, u64 *value)
 EXPORT_SYMBOL_GPL(hv_ghcb_msr_read);
 #endif
 
-enum hv_isolation_type hv_get_isolation_type(void)
-{
-	if (!(ms_hyperv.priv_high & HV_ISOLATION))
-		return HV_ISOLATION_TYPE_NONE;
-	return FIELD_GET(HV_ISOLATION_TYPE, ms_hyperv.isolation_config_b);
-}
-EXPORT_SYMBOL_GPL(hv_get_isolation_type);
-
-/*
- * hv_is_isolation_supported - Check system runs in the Hyper-V
- * isolation VM.
- */
-bool hv_is_isolation_supported(void)
-{
-	if (!cpu_feature_enabled(X86_FEATURE_HYPERVISOR))
-		return false;
-
-	if (!hypervisor_is_type(X86_HYPER_MS_HYPERV))
-		return false;
-
-	return hv_get_isolation_type() != HV_ISOLATION_TYPE_NONE;
-}
-
-DEFINE_STATIC_KEY_FALSE(isolation_type_snp);
-
-/*
- * hv_isolation_type_snp - Check system runs in the AMD SEV-SNP based
- * isolation VM.
- */
-bool hv_isolation_type_snp(void)
-{
-	return static_branch_unlikely(&isolation_type_snp);
-}
-
 /*
  * hv_mark_gpa_visibility - Set pages visible to host via hvcall.
  *
@@ -387,3 +353,37 @@ void hv_unmap_memory(void *addr)
 {
 	vunmap(addr);
 }
+
+enum hv_isolation_type hv_get_isolation_type(void)
+{
+	if (!(ms_hyperv.priv_high & HV_ISOLATION))
+		return HV_ISOLATION_TYPE_NONE;
+	return FIELD_GET(HV_ISOLATION_TYPE, ms_hyperv.isolation_config_b);
+}
+EXPORT_SYMBOL_GPL(hv_get_isolation_type);
+
+/*
+ * hv_is_isolation_supported - Check system runs in the Hyper-V
+ * isolation VM.
+ */
+bool hv_is_isolation_supported(void)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_HYPERVISOR))
+		return false;
+
+	if (!hypervisor_is_type(X86_HYPER_MS_HYPERV))
+		return false;
+
+	return hv_get_isolation_type() != HV_ISOLATION_TYPE_NONE;
+}
+
+DEFINE_STATIC_KEY_FALSE(isolation_type_snp);
+
+/*
+ * hv_isolation_type_snp - Check system runs in the AMD SEV-SNP based
+ * isolation VM.
+ */
+bool hv_isolation_type_snp(void)
+{
+	return static_branch_unlikely(&isolation_type_snp);
+}
Signature · 1 line
1.8.3.1
#8 lore raw

PATCH v4 3/13 Drivers: hv: Explicitly request decrypted in vmap_pfn() calls

In reply to Michael Kelley

In preparation for a subsequent patch, update vmap_pfn() calls to explicitly request that the mapping be for decrypted access to the memory. There's no change in functionality since the PFNs passed to vmap_pfn() are above the shared_gpa_boundary, implicitly producing a decrypted mapping. But explicitly requesting decrypted allows the code to work before and after a subsequent patch that will cause vmap_pfn() to mask the PFNs to being below the shared_gpa_boundary. While another subsesquent patch removes the vmap_pfn() calls entirely, this temporary tweak avoids the need for a large patch that makes all the changes at once.

---
 arch/x86/hyperv/ivm.c    | 2 +-
 drivers/hv/ring_buffer.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/hyperv/ivm.c +1 -1
index f33c67e..e8be4c2 100644
@@ -343,7 +343,7 @@ void *hv_map_memory(void *addr, unsigned long size)
 		pfns[i] = vmalloc_to_pfn(addr + i * PAGE_SIZE) +
 			(ms_hyperv.shared_gpa_boundary >> PAGE_SHIFT);
 
-	vaddr = vmap_pfn(pfns, size / PAGE_SIZE, PAGE_KERNEL_IO);
+	vaddr = vmap_pfn(pfns, size / PAGE_SIZE, pgprot_decrypted(PAGE_KERNEL_NOENC));
 	kfree(pfns);
 
 	return vaddr;
drivers/hv/ring_buffer.c +1 -1
index c6692fd..57667b29 100644
@@ -211,7 +211,7 @@ int hv_ringbuffer_init(struct hv_ring_buffer_info *ring_info,
 
 		ring_info->ring_buffer = (struct hv_ring_buffer *)
 			vmap_pfn(pfns_wraparound, page_cnt * 2 - 1,
-				 PAGE_KERNEL);
+				 pgprot_decrypted(PAGE_KERNEL_NOENC));
 		kfree(pfns_wraparound);
 
 		if (!ring_info->ring_buffer)
Signature · 1 line
1.8.3.1
maintainer
#9 lore raw

Re: [Patch v4 03/13] Drivers: hv: Explicitly request decrypted in vmap_pfn() calls

Quoted 1 line · Michael Kelley

In preparation for a subsequent patch,

Yeah, once those are all in git, the concept of "subsequent patch" becomes ambiguous depending on how you're sorting them. So try to read that commit message out of the context of all those "subsequent patches" and see if it still makes sense.

IOW, you should strive for your commit messages to make sense on their own, without referencing other patches. If possible, that is.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#10 lore raw

PATCH v4 4/13 x86/mm: Handle decryption/re-encryption of bss_decrypted consistently

In reply to Michael Kelley

Current code in sme_postprocess_startup() decrypts the bss_decrypted section when sme_me_mask is non-zero. But code in mem_encrypt_free_decrypted_mem() re-encrypts the unused portion based on CC_ATTR_MEM_ENCRYPT. In a Hyper-V guest VM using vTOM, these conditions are not equivalent as sme_me_mask is always zero when using vTOM. Consequently, mem_encrypt_free_decrypted_mem() attempts to re-encrypt memory that was never decrypted.

Fix this in mem_encrypt_free_decrypted_mem() by conditioning the re-encryption on the same test for non-zero sme_me_mask. Hyper-V guests using vTOM don't need the bss_decrypted section to be decrypted, so skipping the decryption/re-encryption doesn't cause a problem.

Fixes e9d1d2bb75b2 ("treewide: Replace the use of mem_encrypt_active() with cc_platform_has()")
---
 arch/x86/mm/mem_encrypt_amd.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
arch/x86/mm/mem_encrypt_amd.c +7 -3
index 9c4d8db..e0b51c0 100644
@@ -513,10 +513,14 @@ void __init mem_encrypt_free_decrypted_mem(void)
 	npages = (vaddr_end - vaddr) >> PAGE_SHIFT;
 
 	/*
-	 * The unused memory range was mapped decrypted, change the encryption
-	 * attribute from decrypted to encrypted before freeing it.
+	 * If the unused memory range was mapped decrypted, change the encryption
+	 * attribute from decrypted to encrypted before freeing it. Base the
+	 * re-encryption on the same condition used for the decryption in
+	 * sme_postprocess_startup(). Higher level abstractions, such as
+	 * CC_ATTR_MEM_ENCRYPT, aren't necessarily equivalent in a Hyper-V VM
+	 * using vTOM, where sme_me_mask is always zero.
 	 */
-	if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) {
+	if (sme_me_mask) {
 		r = set_memory_encrypted(vaddr, npages);
 		if (r) {
 			pr_warn("failed to free unused decrypted pages\n");
Signature · 1 line
1.8.3.1
maintainer
#11 lore raw

Re: [Patch v4 04/13] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently

Quoted 8 lines · Michael Kelley

Current code in sme_postprocess_startup() decrypts the bss_decrypted section when sme_me_mask is non-zero. But code in mem_encrypt_free_decrypted_mem() re-encrypts the unused portion based on CC_ATTR_MEM_ENCRYPT. In a Hyper-V guest VM using vTOM, these conditions are not equivalent as sme_me_mask is always zero when using vTOM. Consequently, mem_encrypt_free_decrypted_mem() attempts to re-encrypt memory that was never decrypted.

Quoted 5 lines

Fix this in mem_encrypt_free_decrypted_mem() by conditioning the re-encryption on the same test for non-zero sme_me_mask. Hyper-V guests using vTOM don't need the bss_decrypted section to be decrypted, so skipping the decryption/re-encryption doesn't cause a problem.

Lemme simplify the formulations a bit:

"sme_postprocess_startup() decrypts the bss_decrypted ection when me_mask sme_is non-zero.

mem_encrypt_free_decrypted_mem() re-encrypts the unused portion based on CC_ATTR_MEM_ENCRYPT.

In a Hyper-V guest VM using vTOM, these conditions are not equivalent as sme_me_mask is always zero when using vTOM. Consequently, mem_encrypt_free_decrypted_mem() attempts to re-encrypt memory that was never decrypted.

So check sme_me_mask in mem_encrypt_free_decrypted_mem() too.

Hyper-V guests using vTOM don't need the bss_decrypted section to be decrypted, so skipping the decryption/re-encryption doesn't cause a problem."

Quoted 1 line
Fixes e9d1d2bb75b2 ("treewide: Replace the use of mem_encrypt_active() with cc_platform_has()")

So when you say Fixes - this is an issue only for vTOM-using VMs and before yours, there were none. And yours needs more enablement than just this patch.

So does this one really need to be backported to stable@?

I'm asking because there's AI which will pick it up based on this Fixes tag up but that AI is still not that smart to replace us all. :-)

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#12 lore raw

RE: [Patch v4 04/13] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently

From: Borislav Petkov <[email protected]> Sent: Thursday, December 29, 2022 4:18 AM

Quoted 29 lines · Michael Kelley
Quoted 8 lines

Current code in sme_postprocess_startup() decrypts the bss_decrypted section when sme_me_mask is non-zero. But code in mem_encrypt_free_decrypted_mem() re-encrypts the unused portion based on CC_ATTR_MEM_ENCRYPT. In a Hyper-V guest VM using vTOM, these conditions are not equivalent as sme_me_mask is always zero when using vTOM. Consequently, mem_encrypt_free_decrypted_mem() attempts to re-encrypt memory that was never decrypted.

Quoted 5 lines

Fix this in mem_encrypt_free_decrypted_mem() by conditioning the re-encryption on the same test for non-zero sme_me_mask. Hyper-V guests using vTOM don't need the bss_decrypted section to be decrypted, so skipping the decryption/re-encryption doesn't cause a problem.

Lemme simplify the formulations a bit:

"sme_postprocess_startup() decrypts the bss_decrypted ection when me_mask sme_is non-zero.

mem_encrypt_free_decrypted_mem() re-encrypts the unused portion based on CC_ATTR_MEM_ENCRYPT.

In a Hyper-V guest VM using vTOM, these conditions are not equivalent as sme_me_mask is always zero when using vTOM. Consequently, mem_encrypt_free_decrypted_mem() attempts to re-encrypt memory that was never decrypted.

Quoted 5 lines

So check sme_me_mask in mem_encrypt_free_decrypted_mem() too.

Hyper-V guests using vTOM don't need the bss_decrypted section to be decrypted, so skipping the decryption/re-encryption doesn't cause a problem."

Work for me. I'll pick up the new wording in v5.

Quoted 4 lines
Quoted 2 lines
Fixes e9d1d2bb75b2 ("treewide: Replace the use of mem_encrypt_active() with

cc_platform_has()")

Quoted 9 lines

So when you say Fixes - this is an issue only for vTOM-using VMs and before yours, there were none. And yours needs more enablement than just this patch.

So does this one really need to be backported to stable@?

I'm asking because there's AI which will pick it up based on this Fixes tag up but that AI is still not that smart to replace us all. :-)

I'm ambivalent on the backport to stable. One might argue that older kernel versions are conceptually wrong in using different conditions for the decryption and re-encryption. But as you said, they aren't broken from a practical standpoint because sme_me_mask and CC_ATTR_MEM_ENCRYPT are equivalent prior to my patch set. However, the email thread with Sathyanarayanan Kuppuswamy, Tom Lendacky,
and Dexuan Cui concluded that a Fixes: tag is appropriate. See
https://lore.kernel.org/lkml/[email protected]/
and
https://lore.kernel.org/lkml/BYAPR21MB1688A31ED795ED1B5ACB6D26D7099@BYAPR21MB1688.namprd21.prod.outlook.com/

Michael

maintainer
#13 lore raw

Re: [Patch v4 04/13] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently

Quoted 7 lines · Michael Kelley (LINUX)

I'm ambivalent on the backport to stable. One might argue that older kernel versions are conceptually wrong in using different conditions for the decryption and re-encryption. But as you said, they aren't broken from a practical standpoint because sme_me_mask and CC_ATTR_MEM_ENCRYPT are equivalent prior to my patch set. However, the email thread with Sathyanarayanan Kuppuswamy, Tom Lendacky, and Dexuan Cui concluded that a Fixes: tag is appropriate.

Right, just talked to Tom offlist.

A Fixes tag triggers a lot of backporting activity and if it is not really needed, then let's leave it out.

If distros decide to pick up vTOM support, then they'll pick up the whole set anyway.

And if we decide we really need it backported for whatever reason, we will simply send it into stable and the same backporting activity will be triggered then. But then we'd at least have a concrete reason for it.

Makes sense?

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#14 lore raw

RE: [Patch v4 04/13] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently

From: Borislav Petkov <[email protected]> Sent: Monday, January 9, 2023 11:11 AM

Quoted 15 lines · Michael Kelley (LINUX)
Quoted 7 lines

I'm ambivalent on the backport to stable. One might argue that older kernel versions are conceptually wrong in using different conditions for the decryption and re-encryption. But as you said, they aren't broken from a practical standpoint because sme_me_mask and CC_ATTR_MEM_ENCRYPT are equivalent prior to my patch set. However, the email thread with Sathyanarayanan Kuppuswamy, Tom Lendacky, and Dexuan Cui concluded that a Fixes: tag is appropriate.

Right, just talked to Tom offlist.

A Fixes tag triggers a lot of backporting activity and if it is not really needed, then let's leave it out.

Quoted 9 lines

If distros decide to pick up vTOM support, then they'll pick up the whole set anyway.

And if we decide we really need it backported for whatever reason, we will simply send it into stable and the same backporting activity will be triggered then. But then we'd at least have a concrete reason for it.

Makes sense?

Yep, that matches my thinking. I've avoided marking something for stable unless it fixes something that is actually broken.

Michael

#15 lore raw

Re: [Patch v4 04/13] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently

In reply to Borislav Petkov
Quoted 15 lines · Borislav Petkov
Quoted 8 lines · Michael Kelley

Current code in sme_postprocess_startup() decrypts the bss_decrypted section when sme_me_mask is non-zero. But code in mem_encrypt_free_decrypted_mem() re-encrypts the unused portion based on CC_ATTR_MEM_ENCRYPT. In a Hyper-V guest VM using vTOM, these conditions are not equivalent as sme_me_mask is always zero when using vTOM. Consequently, mem_encrypt_free_decrypted_mem() attempts to re-encrypt memory that was never decrypted.

Quoted 5 lines

Fix this in mem_encrypt_free_decrypted_mem() by conditioning the re-encryption on the same test for non-zero sme_me_mask. Hyper-V guests using vTOM don't need the bss_decrypted section to be decrypted, so skipping the decryption/re-encryption doesn't cause a problem.

Quoted 4 lines

Lemme simplify the formulations a bit:

"sme_postprocess_startup() decrypts the bss_decrypted ection when me_mask sme_is non-zero.

s/ection/section/

(In case you copy/paste this text without noticing the typo)

maintainer
#16 lore raw

Re: [Patch v4 04/13] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently

Quoted 6 lines · Bjorn Helgaas
Quoted 2 lines

"sme_postprocess_startup() decrypts the bss_decrypted ection when me_mask sme_is non-zero.

s/ection/section/

(In case you copy/paste this text without noticing the typo)

Thanks.

My par-agraph reformating helper does mangle words like that sometimes. The correct sentence should be:

"sme_postprocess_startup() decrypts the bss_decrypted section when sme_me_mask is non-zero."

/me makes a mental note to switch to the vim builtin instead.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#17 lore raw

PATCH v4 5/13 init: Call mem_encrypt_init() after Hyper-V hypercall init is done

In reply to Michael Kelley

Full Hyper-V initialization, including support for hypercalls, is done as an apic_post_init callback via late_time_init(). mem_encrypt_init() needs to make hypercalls when it marks swiotlb memory as decrypted. But mem_encrypt_init() is currently called a few lines before late_time_init(), so the hypercalls don't work.

Fix this by moving mem_encrypt_init() after late_time_init() and related clock initializations. The intervening initializations don't do any I/O that requires the swiotlb, so moving mem_encrypt_init() slightly later has no impact.

---
 init/main.c | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)
init/main.c +11 -8
index e1c3911..5a7c466 100644
@@ -1088,14 +1088,6 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)
 	 */
 	locking_selftest();
 
-	/*
-	 * This needs to be called before any devices perform DMA
-	 * operations that might use the SWIOTLB bounce buffers. It will
-	 * mark the bounce buffers as decrypted so that their usage will
-	 * not cause "plain-text" data to be decrypted when accessed.
-	 */
-	mem_encrypt_init();
-
 #ifdef CONFIG_BLK_DEV_INITRD
 	if (initrd_start && !initrd_below_start_ok &&
 	    page_to_pfn(virt_to_page((void *)initrd_start)) < min_low_pfn) {
@@ -1112,6 +1104,17 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)
 		late_time_init();
 	sched_clock_init();
 	calibrate_delay();
+
+	/*
+	 * This needs to be called before any devices perform DMA
+	 * operations that might use the SWIOTLB bounce buffers. It will
+	 * mark the bounce buffers as decrypted so that their usage will
+	 * not cause "plain-text" data to be decrypted when accessed. It
+	 * must be called after late_time_init() so that Hyper-V x86/x64
+	 * hypercalls work when the SWIOTLB bounce buffers are decrypted.
+	 */
+	mem_encrypt_init();
+
 	pid_idr_init();
 	anon_vma_init();
 #ifdef CONFIG_X86
Signature · 1 line
1.8.3.1
#18 lore raw

Re: [Patch v4 05/13] init: Call mem_encrypt_init() after Hyper-V hypercall init is done

Quoted 5 lines · Michael Kelley

Full Hyper-V initialization, including support for hypercalls, is done as an apic_post_init callback via late_time_init(). mem_encrypt_init() needs to make hypercalls when it marks swiotlb memory as decrypted. But mem_encrypt_init() is currently called a few lines before late_time_init(), so the hypercalls don't work.

Did you consider moving hyper-v hypercall initialization before
mem_encrypt_init(). Is there any dependency issue?

Quoted 49 lines

Fix this by moving mem_encrypt_init() after late_time_init() and related clock initializations. The intervening initializations don't do any I/O that requires the swiotlb, so moving mem_encrypt_init() slightly later has no impact.

---
 init/main.c | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)
init/main.c +11 -8
index e1c3911..5a7c466 100644
@@ -1088,14 +1088,6 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)
 	 */
 	locking_selftest();
 
-	/*
-	 * This needs to be called before any devices perform DMA
-	 * operations that might use the SWIOTLB bounce buffers. It will
-	 * mark the bounce buffers as decrypted so that their usage will
-	 * not cause "plain-text" data to be decrypted when accessed.
-	 */
-	mem_encrypt_init();
-
 #ifdef CONFIG_BLK_DEV_INITRD
 	if (initrd_start && !initrd_below_start_ok &&
 	    page_to_pfn(virt_to_page((void *)initrd_start)) < min_low_pfn) {
@@ -1112,6 +1104,17 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)
 		late_time_init();
 	sched_clock_init();
 	calibrate_delay();
+
+	/*
+	 * This needs to be called before any devices perform DMA
+	 * operations that might use the SWIOTLB bounce buffers. It will
+	 * mark the bounce buffers as decrypted so that their usage will
+	 * not cause "plain-text" data to be decrypted when accessed. It
+	 * must be called after late_time_init() so that Hyper-V x86/x64
+	 * hypercalls work when the SWIOTLB bounce buffers are decrypted.
+	 */
+	mem_encrypt_init();
+
 	pid_idr_init();
 	anon_vma_init();
 #ifdef CONFIG_X86
Signature · 2 lines
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
#19 lore raw

RE: [Patch v4 05/13] init: Call mem_encrypt_init() after Hyper-V hypercall init is done

From: Sathyanarayanan Kuppuswamy <[email protected]>

Quoted 10 lines · Michael Kelley
Quoted 5 lines

Full Hyper-V initialization, including support for hypercalls, is done as an apic_post_init callback via late_time_init(). mem_encrypt_init() needs to make hypercalls when it marks swiotlb memory as decrypted. But mem_encrypt_init() is currently called a few lines before late_time_init(), so the hypercalls don't work.

Quoted 2 lines

Did you consider moving hyper-v hypercall initialization before
mem_encrypt_init(). Is there any dependency issue?

Hyper-V initialization has historically been done using the callbacks that exist in the x86 initialization paths, rather than adding explicit Hyper-V init calls. As noted above, the full Hyper-V init is done on the apic_post_init callback via late_time_init(). Conceivably we could add an explicit call to do the Hyper-V init, but I think there's still a problem in putting that Hyper-V init prior to the current location of mem_encrypt_init(). I'd have to go check the history, but I think the Hyper-V init needs to happen after the APIC is initialized.

It seems like moving mem_encrypt_init() slightly later is the cleaner long-term solution. Are you aware of a likely problem arising in the future with moving mem_encrypt_init()?

Michael

Quoted 7 lines
Quoted 7 lines

Fix this by moving mem_encrypt_init() after late_time_init() and related clock initializations. The intervening initializations don't do any I/O that requires the swiotlb, so moving mem_encrypt_init() slightly later has no impact.

#20 lore raw

Re: [Patch v4 05/13] init: Call mem_encrypt_init() after Hyper-V hypercall init is done

Quoted 14 lines · Michael Kelley (LINUX)

From: Sathyanarayanan Kuppuswamy <[email protected]>

Quoted 10 lines · Michael Kelley
Quoted 5 lines

Full Hyper-V initialization, including support for hypercalls, is done as an apic_post_init callback via late_time_init(). mem_encrypt_init() needs to make hypercalls when it marks swiotlb memory as decrypted. But mem_encrypt_init() is currently called a few lines before late_time_init(), so the hypercalls don't work.

Quoted 2 lines

Did you consider moving hyper-v hypercall initialization before
mem_encrypt_init(). Is there any dependency issue?

Quoted 8 lines

Hyper-V initialization has historically been done using the callbacks that exist in the x86 initialization paths, rather than adding explicit Hyper-V init calls. As noted above, the full Hyper-V init is done on the apic_post_init callback via late_time_init(). Conceivably we could add an explicit call to do the Hyper-V init, but I think there's still a problem in putting that Hyper-V init prior to the current location of mem_encrypt_init(). I'd have to go check the history, but I think the Hyper-V init needs to happen after the APIC is initialized.

Ok. If there is a dependency or complexity issue, I recommend adding that detail in the commit log.

Quoted 4 lines

It seems like moving mem_encrypt_init() slightly later is the cleaner long-term solution. Are you aware of a likely problem arising in the future with moving mem_encrypt_init()?

I did not investigate in depth, but there appears to be no problem with moving mem_encrypt_init(). But my point is, if it is possible to fix this easily by changing Hyper-v specific initialization, we should consider it first before considering moving the common mem_encrypt_init() function.

Quoted 10 lines

Michael

Quoted 7 lines
Quoted 7 lines

Fix this by moving mem_encrypt_init() after late_time_init() and related clock initializations. The intervening initializations don't do any I/O that requires the swiotlb, so moving mem_encrypt_init() slightly later has no impact.

Signature · 2 lines
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
#21 lore raw

PATCH v4 6/13 x86/hyperv: Change vTOM handling to use standard coco mechanisms

In reply to Michael Kelley

Hyper-V guests on AMD SEV-SNP hardware have the option of using the "virtual Top Of Memory" (vTOM) feature specified by the SEV-SNP architecture. With vTOM, shared vs. private memory accesses are controlled by splitting the guest physical address space into two halves. vTOM is the dividing line where the uppermost bit of the physical address space is set; e.g., with 47 bits of guest physical address space, vTOM is 0x400000000000 (bit 46 is set). Guest physical memory is accessible at two parallel physical addresses -- one below vTOM and one above vTOM. Accesses below vTOM are private (encrypted) while accesses above vTOM are shared (decrypted). In this sense, vTOM is like the GPA.SHARED bit in Intel TDX.

Support for Hyper-V guests using vTOM was added to the Linux kernel in two patch sets[1][2]. This support treats the vTOM bit as part of the physical address. For accessing shared (decrypted) memory, these patch sets create a second kernel virtual mapping that maps to physical addresses above vTOM.

A better approach is to treat the vTOM bit as a protection flag, not as part of the physical address. This new approach is like the approach for the GPA.SHARED bit in Intel TDX. Rather than creating a second kernel virtual mapping, the existing mapping is updated using recently added coco mechanisms. When memory is changed between private and shared using set_memory_decrypted() and set_memory_encrypted(), the PTEs for the existing kernel mapping are changed to add or remove the vTOM bit in the guest physical address, just as with TDX. The hypercalls to change the memory status on the host side are made using the existing callback mechanism. Everything just works, with a minor tweak to map the IO-APIC to use private accesses.

To accomplish the switch in approach, the following must be done in this single patch:

* Update Hyper-V initialization to set the cc_mask based on vTOM and do other coco initialization.

* Update physical_mask so the vTOM bit is no longer treated as part of the physical address

* Remove CC_VENDOR_HYPERV and merge the associated vTOM functionality under CC_VENDOR_AMD. Update cc_mkenc() and cc_mkdec() to set/clear the vTOM bit as a protection flag.

* Code already exists to make hypercalls to inform Hyper-V about pages changing between shared and private. Update this code to run as a callback from __set_memory_enc_pgtable().

* Remove the Hyper-V special case from __set_memory_enc_dec()

* Remove the Hyper-V specific call to swiotlb_update_mem_attributes() since mem_encrypt_init() will now do it.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

---
 arch/x86/coco/core.c             | 37 ++++++++++++++++++++--------
 arch/x86/hyperv/hv_init.c        | 11 ---------
 arch/x86/hyperv/ivm.c            | 52 +++++++++++++++++++++++++++++++---------
 arch/x86/include/asm/coco.h      |  1 -
 arch/x86/include/asm/mshyperv.h  |  8 ++-----
 arch/x86/include/asm/msr-index.h |  1 +
 arch/x86/kernel/cpu/mshyperv.c   | 15 ++++++------
 arch/x86/mm/pat/set_memory.c     |  3 ---
 8 files changed, 78 insertions(+), 50 deletions(-)
arch/x86/coco/core.c +27 -10
index 49b44f8..c361c52 100644
@@ -44,6 +44,24 @@ static bool intel_cc_platform_has(enum cc_attr attr)
 static bool amd_cc_platform_has(enum cc_attr attr)
 {
 #ifdef CONFIG_AMD_MEM_ENCRYPT
+
+	/*
+	 * Handle the SEV-SNP vTOM case where sme_me_mask must be zero,
+	 * and the other levels of SME/SEV functionality, including C-bit
+	 * based SEV-SNP, must not be enabled.
+	 */
+	if (sev_status & MSR_AMD64_SNP_VTOM_ENABLED) {
+		switch (attr) {
+		case CC_ATTR_GUEST_MEM_ENCRYPT:
+		case CC_ATTR_MEM_ENCRYPT:
+		case CC_ATTR_ACCESS_IOAPIC_ENCRYPTED:
+			return true;
+		default:
+			return false;
+		}
+	}
+
+	/* Handle the C-bit case */
 	switch (attr) {
 	case CC_ATTR_MEM_ENCRYPT:
 		return sme_me_mask;
@@ -76,11 +94,6 @@ static bool amd_cc_platform_has(enum cc_attr attr)
 #endif
 }
 
-static bool hyperv_cc_platform_has(enum cc_attr attr)
-{
-	return attr == CC_ATTR_GUEST_MEM_ENCRYPT;
-}
-
 bool cc_platform_has(enum cc_attr attr)
 {
 	switch (vendor) {
@@ -88,8 +101,6 @@ bool cc_platform_has(enum cc_attr attr)
 		return amd_cc_platform_has(attr);
 	case CC_VENDOR_INTEL:
 		return intel_cc_platform_has(attr);
-	case CC_VENDOR_HYPERV:
-		return hyperv_cc_platform_has(attr);
 	default:
 		return false;
 	}
@@ -103,11 +114,14 @@ u64 cc_mkenc(u64 val)
 	 * encryption status of the page.
 	 *
 	 * - for AMD, bit *set* means the page is encrypted
-	 * - for Intel *clear* means encrypted.
+	 * - for AMD with vTOM and for Intel, *clear* means encrypted
 	 */
 	switch (vendor) {
 	case CC_VENDOR_AMD:
-		return val | cc_mask;
+		if (sev_status & MSR_AMD64_SNP_VTOM_ENABLED)
+			return val & ~cc_mask;
+		else
+			return val | cc_mask;
 	case CC_VENDOR_INTEL:
 		return val & ~cc_mask;
 	default:
@@ -120,7 +134,10 @@ u64 cc_mkdec(u64 val)
 	/* See comment in cc_mkenc() */
 	switch (vendor) {
 	case CC_VENDOR_AMD:
-		return val & ~cc_mask;
+		if (sev_status & MSR_AMD64_SNP_VTOM_ENABLED)
+			return val | cc_mask;
+		else
+			return val & ~cc_mask;
 	case CC_VENDOR_INTEL:
 		return val | cc_mask;
 	default:
arch/x86/hyperv/hv_init.c +0 -11
index a823fde..6dbfb26 100644
@@ -29,7 +29,6 @@
 #include <linux/syscore_ops.h>
 #include <clocksource/hyperv_timer.h>
 #include <linux/highmem.h>
-#include <linux/swiotlb.h>
 
 int hyperv_init_cpuhp;
 u64 hv_current_partition_id = ~0ull;
@@ -504,16 +503,6 @@ void __init hyperv_init(void)
 	/* Query the VMs extended capability once, so that it can be cached. */
 	hv_query_ext_cap(0);
 
-#ifdef CONFIG_SWIOTLB
-	/*
-	 * Swiotlb bounce buffer needs to be mapped in extra address
-	 * space. Map function doesn't work in the early place and so
-	 * call swiotlb_update_mem_attributes() here.
-	 */
-	if (hv_is_isolation_supported())
-		swiotlb_update_mem_attributes();
-#endif
-
 	return;
 
 clean_guest_os_id:
arch/x86/hyperv/ivm.c +41 -11

98 lines not on this page · raw message

arch/x86/include/asm/coco.h +0 -1
index 3d98c3a..d2c6a2e 100644
@@ -7,7 +7,6 @@
 enum cc_vendor {
 	CC_VENDOR_NONE,
 	CC_VENDOR_AMD,
-	CC_VENDOR_HYPERV,
 	CC_VENDOR_INTEL,
 };
 
arch/x86/include/asm/mshyperv.h +2 -6
index 6d502f3..010768d 100644
@@ -172,18 +172,19 @@ static inline void hv_apic_init(void) {}
 int hv_map_ioapic_interrupt(int ioapic_id, bool level, int vcpu, int vector,
 		struct hv_interrupt_entry *entry);
 int hv_unmap_ioapic_interrupt(int ioapic_id, struct hv_interrupt_entry *entry);
-int hv_set_mem_host_visibility(unsigned long addr, int numpages, bool visible);
 
 #ifdef CONFIG_AMD_MEM_ENCRYPT
 void hv_ghcb_msr_write(u64 msr, u64 value);
 void hv_ghcb_msr_read(u64 msr, u64 *value);
 bool hv_ghcb_negotiate_protocol(void);
 void hv_ghcb_terminate(unsigned int set, unsigned int reason);
+void hv_vtom_init(void);
 #else
 static inline void hv_ghcb_msr_write(u64 msr, u64 value) {}
 static inline void hv_ghcb_msr_read(u64 msr, u64 *value) {}
 static inline bool hv_ghcb_negotiate_protocol(void) { return false; }
 static inline void hv_ghcb_terminate(unsigned int set, unsigned int reason) {}
+static inline void hv_vtom_init(void) {}
 #endif
 
 extern bool hv_isolation_type_snp(void);
@@ -239,11 +240,6 @@ static inline int hyperv_flush_guest_mapping_range(u64 as,
 }
 static inline void hv_set_register(unsigned int reg, u64 value) { }
 static inline u64 hv_get_register(unsigned int reg) { return 0; }
-static inline int hv_set_mem_host_visibility(unsigned long addr, int numpages,
-					     bool visible)
-{
-	return -1;
-}
 #endif /* CONFIG_HYPERV */
 
 
arch/x86/include/asm/msr-index.h +1 -0
index 37ff475..6a6e70e 100644
@@ -565,6 +565,7 @@
 #define MSR_AMD64_SEV_ENABLED		BIT_ULL(MSR_AMD64_SEV_ENABLED_BIT)
 #define MSR_AMD64_SEV_ES_ENABLED	BIT_ULL(MSR_AMD64_SEV_ES_ENABLED_BIT)
 #define MSR_AMD64_SEV_SNP_ENABLED	BIT_ULL(MSR_AMD64_SEV_SNP_ENABLED_BIT)
+#define MSR_AMD64_SNP_VTOM_ENABLED	BIT_ULL(3)
 
 #define MSR_AMD64_VIRT_SPEC_CTRL	0xc001011f
 
arch/x86/kernel/cpu/mshyperv.c +7 -8
index 8316139..b080795 100644
@@ -33,7 +33,6 @@
 #include <asm/nmi.h>
 #include <clocksource/hyperv_timer.h>
 #include <asm/numa.h>
-#include <asm/coco.h>
 
 /* Is Linux running as the root partition? */
 bool hv_root_partition;
@@ -325,8 +324,10 @@ static void __init ms_hyperv_init_platform(void)
 	if (ms_hyperv.priv_high & HV_ISOLATION) {
 		ms_hyperv.isolation_config_a = cpuid_eax(HYPERV_CPUID_ISOLATION_CONFIG);
 		ms_hyperv.isolation_config_b = cpuid_ebx(HYPERV_CPUID_ISOLATION_CONFIG);
-		ms_hyperv.shared_gpa_boundary =
-			BIT_ULL(ms_hyperv.shared_gpa_boundary_bits);
+
+		if (ms_hyperv.shared_gpa_boundary_active)
+			ms_hyperv.shared_gpa_boundary =
+				BIT_ULL(ms_hyperv.shared_gpa_boundary_bits);
 
 		pr_info("Hyper-V: Isolation Config: Group A 0x%x, Group B 0x%x\n",
 			ms_hyperv.isolation_config_a, ms_hyperv.isolation_config_b);
@@ -337,11 +338,6 @@ static void __init ms_hyperv_init_platform(void)
 			swiotlb_unencrypted_base = ms_hyperv.shared_gpa_boundary;
 #endif
 		}
-		/* Isolation VMs are unenlightened SEV-based VMs, thus this check: */
-		if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) {
-			if (hv_get_isolation_type() != HV_ISOLATION_TYPE_NONE)
-				cc_set_vendor(CC_VENDOR_HYPERV);
-		}
 	}
 
 	if (hv_max_functions_eax >= HYPERV_CPUID_NESTED_FEATURES) {
@@ -410,6 +406,9 @@ static void __init ms_hyperv_init_platform(void)
 	i8253_clear_counter_on_shutdown = false;
 
 #if IS_ENABLED(CONFIG_HYPERV)
+	if ((hv_get_isolation_type() == HV_ISOLATION_TYPE_VBS) ||
+	    (hv_get_isolation_type() == HV_ISOLATION_TYPE_SNP))
+		hv_vtom_init();
 	/*
 	 * Setup the hook to get control post apic initialization.
 	 */
arch/x86/mm/pat/set_memory.c +0 -3
index 356758b..b037954 100644
@@ -2175,9 +2175,6 @@ static int __set_memory_enc_pgtable(unsigned long addr, int numpages, bool enc)
 
 static int __set_memory_enc_dec(unsigned long addr, int numpages, bool enc)
 {
-	if (hv_is_isolation_supported())
-		return hv_set_mem_host_visibility(addr, numpages, !enc);
-
 	if (cc_platform_has(CC_ATTR_MEM_ENCRYPT))
 		return __set_memory_enc_pgtable(addr, numpages, enc);
 
Signature · 1 line
1.8.3.1
maintainer
#22 lore raw

Re: [Patch v4 06/13] x86/hyperv: Change vTOM handling to use standard coco mechanisms

Quoted 30 lines · Michael Kelley

Hyper-V guests on AMD SEV-SNP hardware have the option of using the "virtual Top Of Memory" (vTOM) feature specified by the SEV-SNP architecture. With vTOM, shared vs. private memory accesses are controlled by splitting the guest physical address space into two halves. vTOM is the dividing line where the uppermost bit of the physical address space is set; e.g., with 47 bits of guest physical address space, vTOM is 0x400000000000 (bit 46 is set). Guest physical memory is accessible at two parallel physical addresses -- one below vTOM and one above vTOM. Accesses below vTOM are private (encrypted) while accesses above vTOM are shared (decrypted). In this sense, vTOM is like the GPA.SHARED bit in Intel TDX.

Support for Hyper-V guests using vTOM was added to the Linux kernel in two patch sets[1][2]. This support treats the vTOM bit as part of the physical address. For accessing shared (decrypted) memory, these patch sets create a second kernel virtual mapping that maps to physical addresses above vTOM.

A better approach is to treat the vTOM bit as a protection flag, not as part of the physical address. This new approach is like the approach for the GPA.SHARED bit in Intel TDX. Rather than creating a second kernel virtual mapping, the existing mapping is updated using recently added coco mechanisms. When memory is changed between private and shared using set_memory_decrypted() and set_memory_encrypted(), the PTEs for the existing kernel mapping are changed to add or remove the vTOM bit in the guest physical address, just as with TDX. The hypercalls to change the memory status on the host side are made using the existing callback mechanism. Everything just works, with a minor tweak to map the IO-APIC to use private accesses.

Quoted 2 lines

To accomplish the switch in approach, the following must be done in this single patch:

s/in this single patch//

Quoted 39 lines

* Update Hyper-V initialization to set the cc_mask based on vTOM and do other coco initialization.

* Update physical_mask so the vTOM bit is no longer treated as part of the physical address

* Remove CC_VENDOR_HYPERV and merge the associated vTOM functionality under CC_VENDOR_AMD. Update cc_mkenc() and cc_mkdec() to set/clear the vTOM bit as a protection flag.

* Code already exists to make hypercalls to inform Hyper-V about pages changing between shared and private. Update this code to run as a callback from __set_memory_enc_pgtable().

* Remove the Hyper-V special case from __set_memory_enc_dec()

* Remove the Hyper-V specific call to swiotlb_update_mem_attributes() since mem_encrypt_init() will now do it.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

---
 arch/x86/coco/core.c             | 37 ++++++++++++++++++++--------
 arch/x86/hyperv/hv_init.c        | 11 ---------
 arch/x86/hyperv/ivm.c            | 52 +++++++++++++++++++++++++++++++---------
 arch/x86/include/asm/coco.h      |  1 -
 arch/x86/include/asm/mshyperv.h  |  8 ++-----
 arch/x86/include/asm/msr-index.h |  1 +
 arch/x86/kernel/cpu/mshyperv.c   | 15 ++++++------
 arch/x86/mm/pat/set_memory.c     |  3 ---
 8 files changed, 78 insertions(+), 50 deletions(-)
arch/x86/coco/core.c +0 -0
index 49b44f8..c361c52 100644

arch/x86/coco/core.c static bool intel_cc_platform_has(enum cc_attr attr) · line 53

Quoted 10 lines
arch/x86/coco/core.c +7 -0
@@ -44,6 +44,24 @@ static bool intel_cc_platform_has(enum cc_attr attr)
 static bool amd_cc_platform_has(enum cc_attr attr)
 {
 #ifdef CONFIG_AMD_MEM_ENCRYPT
+
+	/*
+	 * Handle the SEV-SNP vTOM case where sme_me_mask must be zero,
+	 * and the other levels of SME/SEV functionality, including C-bit
+	 * based SEV-SNP, must not be enabled.
+	 */
+	if (sev_status & MSR_AMD64_SNP_VTOM_ENABLED) {
		return amd_cc_platform_vtom();

or so and then stick that switch in there.

This way it looks kinda grafted in front and with a function call with a telling name it says it is a special case...

arch/x86/coco/core.c static bool intel_cc_platform_has(enum cc_attr attr) · line 62

Quoted 9 lines
arch/x86/coco/core.c +9 -0
+		switch (attr) {
+		case CC_ATTR_GUEST_MEM_ENCRYPT:
+		case CC_ATTR_MEM_ENCRYPT:
+		case CC_ATTR_ACCESS_IOAPIC_ENCRYPTED:
+			return true;
+		default:
+			return false;
+		}
+	}

The rest looks kinda nice, I gotta say. I can't complain. :)

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#23 lore raw

RE: [Patch v4 06/13] x86/hyperv: Change vTOM handling to use standard coco mechanisms

From: Borislav Petkov <[email protected]> Sent: Monday, January 9, 2023 8:39 AM

Quoted 1 line

[snip]

Quoted 23 lines
Quoted 5 lines
arch/x86/coco/core.c +0 -0
index 49b44f8..c361c52 100644

arch/x86/coco/core.c static bool intel_cc_platform_has(enum cc_attr attr) · line 53

Quoted 10 lines
arch/x86/coco/core.c +7 -0
@@ -44,6 +44,24 @@ static bool intel_cc_platform_has(enum cc_attr attr)
 static bool amd_cc_platform_has(enum cc_attr attr)
 {
 #ifdef CONFIG_AMD_MEM_ENCRYPT
+
+	/*
+	 * Handle the SEV-SNP vTOM case where sme_me_mask must be zero,
+	 * and the other levels of SME/SEV functionality, including C-bit
+	 * based SEV-SNP, must not be enabled.
+	 */
+	if (sev_status & MSR_AMD64_SNP_VTOM_ENABLED) {
		return amd_cc_platform_vtom();

or so and then stick that switch in there.

This way it looks kinda grafted in front and with a function call with a telling name it says it is a special case...

Quoted 12 lines

arch/x86/coco/core.c static bool intel_cc_platform_has(enum cc_attr attr) · line 62

Quoted 9 lines
arch/x86/coco/core.c +9 -0
+		switch (attr) {
+		case CC_ATTR_GUEST_MEM_ENCRYPT:
+		case CC_ATTR_MEM_ENCRYPT:
+		case CC_ATTR_ACCESS_IOAPIC_ENCRYPTED:
+			return true;
+		default:
+			return false;
+		}
+	}

The rest looks kinda nice, I gotta say. I can't complain. :)

OK. I have no objection to putting that code in a separate "helper" function. The only slight messiness is that the helper function must be separately wrapped in #ifdef CONFIG_AMD_MEM_ENCRYPT, or marked __maybe_unused.

Michael

maintainer
#24 lore raw

Re: [Patch v4 06/13] x86/hyperv: Change vTOM handling to use standard coco mechanisms

Quoted 4 lines · Michael Kelley (LINUX)

OK. I have no objection to putting that code in a separate "helper" function. The only slight messiness is that the helper function must be separately wrapped in #ifdef CONFIG_AMD_MEM_ENCRYPT, or marked __maybe_unused.

I'd vote for __maybe_unused as that doesn't mean more ifdeffery.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#25 lore raw

PATCH v4 7/13 swiotlb: Remove bounce buffer remapping for Hyper-V

In reply to Michael Kelley

With changes to how Hyper-V guest VMs flip memory between private (encrypted) and shared (decrypted), creating a second kernel virtual mapping for shared memory is no longer necessary. Everything needed for the transition to shared is handled by set_memory_decrypted().

As such, remove swiotlb_unencrypted_base and the associated code.

---
 arch/x86/kernel/cpu/mshyperv.c |  7 +------
 include/linux/swiotlb.h        |  2 --
 kernel/dma/swiotlb.c           | 45 +-----------------------------------------
 3 files changed, 2 insertions(+), 52 deletions(-)
arch/x86/kernel/cpu/mshyperv.c +1 -6
index b080795..c87e411 100644
@@ -18,7 +18,6 @@
 #include <linux/kexec.h>
 #include <linux/i8253.h>
 #include <linux/random.h>
-#include <linux/swiotlb.h>
 #include <asm/processor.h>
 #include <asm/hypervisor.h>
 #include <asm/hyperv-tlfs.h>
@@ -332,12 +331,8 @@ static void __init ms_hyperv_init_platform(void)
 		pr_info("Hyper-V: Isolation Config: Group A 0x%x, Group B 0x%x\n",
 			ms_hyperv.isolation_config_a, ms_hyperv.isolation_config_b);
 
-		if (hv_get_isolation_type() == HV_ISOLATION_TYPE_SNP) {
+		if (hv_get_isolation_type() == HV_ISOLATION_TYPE_SNP)
 			static_branch_enable(&isolation_type_snp);
-#ifdef CONFIG_SWIOTLB
-			swiotlb_unencrypted_base = ms_hyperv.shared_gpa_boundary;
-#endif
-		}
 	}
 
 	if (hv_max_functions_eax >= HYPERV_CPUID_NESTED_FEATURES) {
include/linux/swiotlb.h +0 -2
index 35bc4e2..13d7075 100644
@@ -185,6 +185,4 @@ static inline bool is_swiotlb_for_alloc(struct device *dev)
 }
 #endif /* CONFIG_DMA_RESTRICTED_POOL */
 
-extern phys_addr_t swiotlb_unencrypted_base;
-
 #endif /* __LINUX_SWIOTLB_H */
kernel/dma/swiotlb.c +1 -44

74 lines not on this page · raw message

Signature · 1 line
1.8.3.1
maintainer Acked-by on v4 7/13
#26 lore raw

Re: [Patch v4 07/13] swiotlb: Remove bounce buffer remapping for Hyper-V

Quoted 8 lines · Michael Kelley

With changes to how Hyper-V guest VMs flip memory between private (encrypted) and shared (decrypted), creating a second kernel virtual mapping for shared memory is no longer necessary. Everything needed for the transition to shared is handled by set_memory_decrypted().

As such, remove swiotlb_unencrypted_base and the associated code.

Quoted 7 lines
---
 arch/x86/kernel/cpu/mshyperv.c |  7 +------
 include/linux/swiotlb.h        |  2 --
 kernel/dma/swiotlb.c           | 45 +-----------------------------------------
 3 files changed, 2 insertions(+), 52 deletions(-)

Patches removing crap are always nice:

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#27 lore raw

PATCH v4 8/13 Drivers: hv: vmbus: Remove second mapping of VMBus monitor pages

In reply to Michael Kelley

With changes to how Hyper-V guest VMs flip memory between private (encrypted) and shared (decrypted), creating a second kernel virtual mapping for shared memory is no longer necessary. Everything needed for the transition to shared is handled by set_memory_decrypted().

As such, remove the code to create and manage the second mapping for VMBus monitor pages. Because set_memory_decrypted() and set_memory_encrypted() are no-ops in normal VMs, it's not even necessary to test for being in a Confidential VM (a.k.a., "Isolation VM").

---
 drivers/hv/connection.c   | 113 ++++++++++++----------------------------------
 drivers/hv/hyperv_vmbus.h |   2 -
 2 files changed, 28 insertions(+), 87 deletions(-)
drivers/hv/connection.c +28 -85
index 9dc27e5..f670cfd 100644
@@ -104,8 +104,14 @@ int vmbus_negotiate_version(struct vmbus_channel_msginfo *msginfo, u32 version)
 		vmbus_connection.msg_conn_id = VMBUS_MESSAGE_CONNECTION_ID;
 	}
 
-	msg->monitor_page1 = vmbus_connection.monitor_pages_pa[0];
-	msg->monitor_page2 = vmbus_connection.monitor_pages_pa[1];
+	/*
+	 * shared_gpa_boundary is zero in non-SNP VMs, so it's safe to always
+	 * bitwise OR it
+	 */
+	msg->monitor_page1 = virt_to_phys(vmbus_connection.monitor_pages[0]) |
+				ms_hyperv.shared_gpa_boundary;
+	msg->monitor_page2 = virt_to_phys(vmbus_connection.monitor_pages[1]) |
+				ms_hyperv.shared_gpa_boundary;
 
 	msg->target_vcpu = hv_cpu_number_to_vp_number(VMBUS_CONNECT_CPU);
 
@@ -219,72 +225,27 @@ int vmbus_connect(void)
 	 * Setup the monitor notification facility. The 1st page for
 	 * parent->child and the 2nd page for child->parent
 	 */
-	vmbus_connection.monitor_pages[0] = (void *)hv_alloc_hyperv_zeroed_page();
-	vmbus_connection.monitor_pages[1] = (void *)hv_alloc_hyperv_zeroed_page();
+	vmbus_connection.monitor_pages[0] = (void *)hv_alloc_hyperv_page();
+	vmbus_connection.monitor_pages[1] = (void *)hv_alloc_hyperv_page();
 	if ((vmbus_connection.monitor_pages[0] == NULL) ||
 	    (vmbus_connection.monitor_pages[1] == NULL)) {
 		ret = -ENOMEM;
 		goto cleanup;
 	}
 
-	vmbus_connection.monitor_pages_original[0]
-		= vmbus_connection.monitor_pages[0];
-	vmbus_connection.monitor_pages_original[1]
-		= vmbus_connection.monitor_pages[1];
-	vmbus_connection.monitor_pages_pa[0]
-		= virt_to_phys(vmbus_connection.monitor_pages[0]);
-	vmbus_connection.monitor_pages_pa[1]
-		= virt_to_phys(vmbus_connection.monitor_pages[1]);
-
-	if (hv_is_isolation_supported()) {
-		ret = set_memory_decrypted((unsigned long)
-					   vmbus_connection.monitor_pages[0],
-					   1);
-		ret |= set_memory_decrypted((unsigned long)
-					    vmbus_connection.monitor_pages[1],
-					    1);
-		if (ret)
-			goto cleanup;
-
-		/*
-		 * Isolation VM with AMD SNP needs to access monitor page via
-		 * address space above shared gpa boundary.
-		 */
-		if (hv_isolation_type_snp()) {
-			vmbus_connection.monitor_pages_pa[0] +=
-				ms_hyperv.shared_gpa_boundary;
-			vmbus_connection.monitor_pages_pa[1] +=
-				ms_hyperv.shared_gpa_boundary;
-
-			vmbus_connection.monitor_pages[0]
-				= memremap(vmbus_connection.monitor_pages_pa[0],
-					   HV_HYP_PAGE_SIZE,
-					   MEMREMAP_WB);
-			if (!vmbus_connection.monitor_pages[0]) {
-				ret = -ENOMEM;
-				goto cleanup;
-			}
-
-			vmbus_connection.monitor_pages[1]
-				= memremap(vmbus_connection.monitor_pages_pa[1],
-					   HV_HYP_PAGE_SIZE,
-					   MEMREMAP_WB);
-			if (!vmbus_connection.monitor_pages[1]) {
-				ret = -ENOMEM;
-				goto cleanup;
-			}
-		}
-
-		/*
-		 * Set memory host visibility hvcall smears memory
-		 * and so zero monitor pages here.
-		 */
-		memset(vmbus_connection.monitor_pages[0], 0x00,
-		       HV_HYP_PAGE_SIZE);
-		memset(vmbus_connection.monitor_pages[1], 0x00,
-		       HV_HYP_PAGE_SIZE);
+	ret = set_memory_decrypted((unsigned long)
+				vmbus_connection.monitor_pages[0], 1);
+	ret |= set_memory_decrypted((unsigned long)
+				vmbus_connection.monitor_pages[1], 1);
+	if (ret)
+		goto cleanup;
 
-	}
+	/*
+	 * Set_memory_decrypted() will change the memory contents if
+	 * decryption occurs, so zero monitor pages here.
+	 */
+	memset(vmbus_connection.monitor_pages[0], 0x00, HV_HYP_PAGE_SIZE);
+	memset(vmbus_connection.monitor_pages[1], 0x00, HV_HYP_PAGE_SIZE);
 
 	msginfo = kzalloc(sizeof(*msginfo) +
 			  sizeof(struct vmbus_channel_initiate_contact),
@@ -376,31 +337,13 @@ void vmbus_disconnect(void)
 		vmbus_connection.int_page = NULL;
 	}
 
-	if (hv_is_isolation_supported()) {
-		/*
-		 * memunmap() checks input address is ioremap address or not
-		 * inside. It doesn't unmap any thing in the non-SNP CVM and
-		 * so not check CVM type here.
-		 */
-		memunmap(vmbus_connection.monitor_pages[0]);
-		memunmap(vmbus_connection.monitor_pages[1]);
-
-		set_memory_encrypted((unsigned long)
-			vmbus_connection.monitor_pages_original[0],
-			1);
-		set_memory_encrypted((unsigned long)
-			vmbus_connection.monitor_pages_original[1],
-			1);
-	}
+	set_memory_encrypted((unsigned long)vmbus_connection.monitor_pages[0], 1);
+	set_memory_encrypted((unsigned long)vmbus_connection.monitor_pages[1], 1);
 
-	hv_free_hyperv_page((unsigned long)
-		vmbus_connection.monitor_pages_original[0]);
-	hv_free_hyperv_page((unsigned long)
-		vmbus_connection.monitor_pages_original[1]);
-	vmbus_connection.monitor_pages_original[0] =
-		vmbus_connection.monitor_pages[0] = NULL;
-	vmbus_connection.monitor_pages_original[1] =
-		vmbus_connection.monitor_pages[1] = NULL;
+	hv_free_hyperv_page((unsigned long)vmbus_connection.monitor_pages[0]);
+	hv_free_hyperv_page((unsigned long)vmbus_connection.monitor_pages[1]);
+	vmbus_connection.monitor_pages[0] = NULL;
+	vmbus_connection.monitor_pages[1] = NULL;
 }
 
 /*
drivers/hv/hyperv_vmbus.h +0 -2
index dc673ed..167ac51 100644
@@ -241,8 +241,6 @@ struct vmbus_connection {
 	 * is child->parent notification
 	 */
 	struct hv_monitor_page *monitor_pages[2];
-	void *monitor_pages_original[2];
-	phys_addr_t monitor_pages_pa[2];
 	struct list_head chn_msg_list;
 	spinlock_t channelmsg_lock;
 
Signature · 1 line
1.8.3.1
#28 lore raw

PATCH v4 9/13 Drivers: hv: vmbus: Remove second way of mapping ring buffers

In reply to Michael Kelley

With changes to how Hyper-V guest VMs flip memory between private (encrypted) and shared (decrypted), it's no longer necessary to have separate code paths for mapping VMBus ring buffers for for normal VMs and for Confidential VMs.

As such, remove the code path that uses vmap_pfn(), and set the protection flags argument to vmap() to account for the difference between normal and Confidential VMs.

---
 drivers/hv/ring_buffer.c | 62 ++++++++++++++++--------------------------------
 1 file changed, 20 insertions(+), 42 deletions(-)
drivers/hv/ring_buffer.c +20 -42
index 57667b29..3783d9d 100644
@@ -186,8 +186,6 @@ int hv_ringbuffer_init(struct hv_ring_buffer_info *ring_info,
 		       struct page *pages, u32 page_cnt, u32 max_pkt_size)
 {
 	struct page **pages_wraparound;
-	unsigned long *pfns_wraparound;
-	u64 pfn;
 	int i;
 
 	BUILD_BUG_ON((sizeof(struct hv_ring_buffer) != PAGE_SIZE));
@@ -196,50 +194,30 @@ int hv_ringbuffer_init(struct hv_ring_buffer_info *ring_info,
 	 * First page holds struct hv_ring_buffer, do wraparound mapping for
 	 * the rest.
 	 */
-	if (hv_isolation_type_snp()) {
-		pfn = page_to_pfn(pages) +
-			PFN_DOWN(ms_hyperv.shared_gpa_boundary);
+	pages_wraparound = kcalloc(page_cnt * 2 - 1,
+				   sizeof(struct page *),
+				   GFP_KERNEL);
+	if (!pages_wraparound)
+		return -ENOMEM;
 
-		pfns_wraparound = kcalloc(page_cnt * 2 - 1,
-			sizeof(unsigned long), GFP_KERNEL);
-		if (!pfns_wraparound)
-			return -ENOMEM;
-
-		pfns_wraparound[0] = pfn;
-		for (i = 0; i < 2 * (page_cnt - 1); i++)
-			pfns_wraparound[i + 1] = pfn + i % (page_cnt - 1) + 1;
-
-		ring_info->ring_buffer = (struct hv_ring_buffer *)
-			vmap_pfn(pfns_wraparound, page_cnt * 2 - 1,
-				 pgprot_decrypted(PAGE_KERNEL_NOENC));
-		kfree(pfns_wraparound);
-
-		if (!ring_info->ring_buffer)
-			return -ENOMEM;
-
-		/* Zero ring buffer after setting memory host visibility. */
-		memset(ring_info->ring_buffer, 0x00, PAGE_SIZE * page_cnt);
-	} else {
-		pages_wraparound = kcalloc(page_cnt * 2 - 1,
-					   sizeof(struct page *),
-					   GFP_KERNEL);
-		if (!pages_wraparound)
-			return -ENOMEM;
-
-		pages_wraparound[0] = pages;
-		for (i = 0; i < 2 * (page_cnt - 1); i++)
-			pages_wraparound[i + 1] =
-				&pages[i % (page_cnt - 1) + 1];
+	pages_wraparound[0] = pages;
+	for (i = 0; i < 2 * (page_cnt - 1); i++)
+		pages_wraparound[i + 1] =
+			&pages[i % (page_cnt - 1) + 1];
 
-		ring_info->ring_buffer = (struct hv_ring_buffer *)
-			vmap(pages_wraparound, page_cnt * 2 - 1, VM_MAP,
-				PAGE_KERNEL);
+	ring_info->ring_buffer = (struct hv_ring_buffer *)
+		vmap(pages_wraparound, page_cnt * 2 - 1, VM_MAP,
+			pgprot_decrypted(PAGE_KERNEL_NOENC));
 
-		kfree(pages_wraparound);
-		if (!ring_info->ring_buffer)
-			return -ENOMEM;
-	}
+	kfree(pages_wraparound);
+	if (!ring_info->ring_buffer)
+		return -ENOMEM;
 
+	/*
+	 * Ensure the header page is zero'ed since
+	 * encryption status may have changed.
+	 */
+	memset(ring_info->ring_buffer, 0, HV_HYP_PAGE_SIZE);
 
 	ring_info->ring_buffer->read_index =
 		ring_info->ring_buffer->write_index = 0;
Signature · 1 line
1.8.3.1
#29 lore raw

PATCH v4 10/13 hv_netvsc: Remove second mapping of send and recv buffers

In reply to Michael Kelley

With changes to how Hyper-V guest VMs flip memory between private (encrypted) and shared (decrypted), creating a second kernel virtual mapping for shared memory is no longer necessary. Everything needed for the transition to shared is handled by set_memory_decrypted().

As such, remove the code to create and manage the second mapping for the pre-allocated send and recv buffers. This mapping is the last user of hv_map_memory()/hv_unmap_memory(), so delete these functions as well. Finally, hv_map_memory() is the last user of vmap_pfn() in Hyper-V guest code, so remove the Kconfig selection of VMAP_PFN.

---
 arch/x86/hyperv/ivm.c           | 28 ------------------------
 drivers/hv/Kconfig              |  1 -
 drivers/hv/hv_common.c          | 11 ----------
 drivers/net/hyperv/hyperv_net.h |  2 --
 drivers/net/hyperv/netvsc.c     | 48 ++---------------------------------------
 include/asm-generic/mshyperv.h  |  2 --
 6 files changed, 2 insertions(+), 90 deletions(-)
arch/x86/hyperv/ivm.c +0 -28
index 8e2717d..83b71bd 100644
@@ -356,34 +356,6 @@ void __init hv_vtom_init(void)
 
 #endif /* CONFIG_AMD_MEM_ENCRYPT */
 
-/*
- * hv_map_memory - map memory to extra space in the AMD SEV-SNP Isolation VM.
- */
-void *hv_map_memory(void *addr, unsigned long size)
-{
-	unsigned long *pfns = kcalloc(size / PAGE_SIZE,
-				      sizeof(unsigned long), GFP_KERNEL);
-	void *vaddr;
-	int i;
-
-	if (!pfns)
-		return NULL;
-
-	for (i = 0; i < size / PAGE_SIZE; i++)
-		pfns[i] = vmalloc_to_pfn(addr + i * PAGE_SIZE) +
-			(ms_hyperv.shared_gpa_boundary >> PAGE_SHIFT);
-
-	vaddr = vmap_pfn(pfns, size / PAGE_SIZE, pgprot_decrypted(PAGE_KERNEL_NOENC));
-	kfree(pfns);
-
-	return vaddr;
-}
-
-void hv_unmap_memory(void *addr)
-{
-	vunmap(addr);
-}
-
 enum hv_isolation_type hv_get_isolation_type(void)
 {
 	if (!(ms_hyperv.priv_high & HV_ISOLATION))
drivers/hv/Kconfig +0 -1
index 0747a8f..9a074cb 100644
@@ -8,7 +8,6 @@ config HYPERV
 		|| (ARM64 && !CPU_BIG_ENDIAN))
 	select PARAVIRT
 	select X86_HV_CALLBACK_VECTOR if X86
-	select VMAP_PFN
 	help
 	  Select this option to run Linux as a Hyper-V client operating
 	  system.
drivers/hv/hv_common.c +0 -11
index ae68298..566735f 100644
@@ -308,14 +308,3 @@ u64 __weak hv_ghcb_hypercall(u64 control, void *input, void *output, u32 input_s
 	return HV_STATUS_INVALID_PARAMETER;
 }
 EXPORT_SYMBOL_GPL(hv_ghcb_hypercall);
-
-void __weak *hv_map_memory(void *addr, unsigned long size)
-{
-	return NULL;
-}
-EXPORT_SYMBOL_GPL(hv_map_memory);
-
-void __weak hv_unmap_memory(void *addr)
-{
-}
-EXPORT_SYMBOL_GPL(hv_unmap_memory);
drivers/net/hyperv/hyperv_net.h +0 -2
index dd5919e..33d51e3 100644
@@ -1139,7 +1139,6 @@ struct netvsc_device {
 
 	/* Receive buffer allocated by us but manages by NetVSP */
 	void *recv_buf;
-	void *recv_original_buf;
 	u32 recv_buf_size; /* allocated bytes */
 	struct vmbus_gpadl recv_buf_gpadl_handle;
 	u32 recv_section_cnt;
@@ -1148,7 +1147,6 @@ struct netvsc_device {
 
 	/* Send buffer allocated by us */
 	void *send_buf;
-	void *send_original_buf;
 	u32 send_buf_size;
 	struct vmbus_gpadl send_buf_gpadl_handle;
 	u32 send_section_cnt;
drivers/net/hyperv/netvsc.c +2 -46

84 lines not on this page · raw message

include/asm-generic/mshyperv.h +0 -2
index bfb9eb9..6fabc4a 100644
@@ -267,8 +267,6 @@ static inline int cpumask_to_vpset_noself(struct hv_vpset *vpset,
 void hyperv_cleanup(void);
 bool hv_query_ext_cap(u64 cap_query);
 void hv_setup_dma_ops(struct device *dev, bool coherent);
-void *hv_map_memory(void *addr, unsigned long size);
-void hv_unmap_memory(void *addr);
 #else /* CONFIG_HYPERV */
 static inline bool hv_is_hyperv_initialized(void) { return false; }
 static inline bool hv_is_hibernation_supported(void) { return false; }
Signature · 1 line
1.8.3.1
#30 lore raw

PATCH v4 11/13 Drivers: hv: Don't remap addresses that are above shared_gpa_boundary

In reply to Michael Kelley

With the vTOM bit now treated as a protection flag and not part of the physical address, avoid remapping physical addresses with vTOM set since technically such addresses aren't valid. Use ioremap_cache() instead of memremap() to ensure that the mapping provides decrypted access, which will correctly set the vTOM bit as a protection flag.

While this change is not required for correctness with the current implementation of memremap(), for general code hygiene it's better to not depend on the mapping functions doing something reasonable with a physical address that is out-of-range.

While here, fix typos in two error messages.

---
 arch/x86/hyperv/hv_init.c |  7 +++++--
 drivers/hv/hv.c           | 23 +++++++++++++----------
 2 files changed, 18 insertions(+), 12 deletions(-)
arch/x86/hyperv/hv_init.c +5 -2
index 6dbfb26..9070a78 100644
@@ -63,7 +63,10 @@ static int hyperv_init_ghcb(void)
 	 * memory boundary and map it here.
 	 */
 	rdmsrl(MSR_AMD64_SEV_ES_GHCB, ghcb_gpa);
-	ghcb_va = memremap(ghcb_gpa, HV_HYP_PAGE_SIZE, MEMREMAP_WB);
+
+	/* Mask out vTOM bit. ioremap_cache() maps decrypted */
+	ghcb_gpa &= ~ms_hyperv.shared_gpa_boundary;
+	ghcb_va = (void *)ioremap_cache(ghcb_gpa, HV_HYP_PAGE_SIZE);
 	if (!ghcb_va)
 		return -ENOMEM;
 
@@ -217,7 +220,7 @@ static int hv_cpu_die(unsigned int cpu)
 	if (hv_ghcb_pg) {
 		ghcb_va = (void **)this_cpu_ptr(hv_ghcb_pg);
 		if (*ghcb_va)
-			memunmap(*ghcb_va);
+			iounmap(*ghcb_va);
 		*ghcb_va = NULL;
 	}
 
drivers/hv/hv.c +13 -10
index 4d6480d..410e6c4 100644
@@ -217,11 +217,13 @@ void hv_synic_enable_regs(unsigned int cpu)
 	simp.simp_enabled = 1;
 
 	if (hv_isolation_type_snp()) {
+		/* Mask out vTOM bit. ioremap_cache() maps decrypted */
+		u64 base = (simp.base_simp_gpa << HV_HYP_PAGE_SHIFT) &
+				~ms_hyperv.shared_gpa_boundary;
 		hv_cpu->synic_message_page
-			= memremap(simp.base_simp_gpa << HV_HYP_PAGE_SHIFT,
-				   HV_HYP_PAGE_SIZE, MEMREMAP_WB);
+			= (void *)ioremap_cache(base, HV_HYP_PAGE_SIZE);
 		if (!hv_cpu->synic_message_page)
-			pr_err("Fail to map syinc message page.\n");
+			pr_err("Fail to map synic message page.\n");
 	} else {
 		simp.base_simp_gpa = virt_to_phys(hv_cpu->synic_message_page)
 			>> HV_HYP_PAGE_SHIFT;
@@ -234,12 +236,13 @@ void hv_synic_enable_regs(unsigned int cpu)
 	siefp.siefp_enabled = 1;
 
 	if (hv_isolation_type_snp()) {
-		hv_cpu->synic_event_page =
-			memremap(siefp.base_siefp_gpa << HV_HYP_PAGE_SHIFT,
-				 HV_HYP_PAGE_SIZE, MEMREMAP_WB);
-
+		/* Mask out vTOM bit. ioremap_cache() maps decrypted */
+		u64 base = (siefp.base_siefp_gpa << HV_HYP_PAGE_SHIFT) &
+				~ms_hyperv.shared_gpa_boundary;
+		hv_cpu->synic_event_page
+			= (void *)ioremap_cache(base, HV_HYP_PAGE_SIZE);
 		if (!hv_cpu->synic_event_page)
-			pr_err("Fail to map syinc event page.\n");
+			pr_err("Fail to map synic event page.\n");
 	} else {
 		siefp.base_siefp_gpa = virt_to_phys(hv_cpu->synic_event_page)
 			>> HV_HYP_PAGE_SHIFT;
@@ -316,7 +319,7 @@ void hv_synic_disable_regs(unsigned int cpu)
 	 */
 	simp.simp_enabled = 0;
 	if (hv_isolation_type_snp())
-		memunmap(hv_cpu->synic_message_page);
+		iounmap(hv_cpu->synic_message_page);
 	else
 		simp.base_simp_gpa = 0;
 
@@ -326,7 +329,7 @@ void hv_synic_disable_regs(unsigned int cpu)
 	siefp.siefp_enabled = 0;
 
 	if (hv_isolation_type_snp())
-		memunmap(hv_cpu->synic_event_page);
+		iounmap(hv_cpu->synic_event_page);
 	else
 		siefp.base_siefp_gpa = 0;
 
Signature · 1 line
1.8.3.1
#31 lore raw

PATCH v4 12/13 PCI: hv: Add hypercalls to read/write MMIO space

In reply to Michael Kelley

To support PCI pass-thru devices in Confidential VMs, Hyper-V has added hypercalls to read and write MMIO space. Add the appropriate definitions to hyperv-tlfs.h and implement functions to make the hypercalls. These functions are used in a subsequent patch.

---
 arch/x86/include/asm/hyperv-tlfs.h  |  3 ++
 drivers/pci/controller/pci-hyperv.c | 64 +++++++++++++++++++++++++++++++++++++
 include/asm-generic/hyperv-tlfs.h   | 22 +++++++++++++
 3 files changed, 89 insertions(+)
arch/x86/include/asm/hyperv-tlfs.h +3 -0
index 6d9368e..4d67c38 100644
@@ -117,6 +117,9 @@
 /* Recommend using enlightened VMCS */
 #define HV_X64_ENLIGHTENED_VMCS_RECOMMENDED		BIT(14)
 
+/* Use hypercalls for MMIO config space access */
+#define HV_X64_USE_MMIO_HYPERCALLS			BIT(21)
+
 /*
  * CPU management features identification.
  * These are HYPERV_CPUID_CPU_MANAGEMENT_FEATURES.EAX bits.
drivers/pci/controller/pci-hyperv.c +64 -0

70 lines not on this page · raw message

include/asm-generic/hyperv-tlfs.h +22 -0
index b17c6ee..fcab07b 100644
@@ -168,6 +168,8 @@ struct ms_hyperv_tsc_page {
 #define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_SPACE 0x00af
 #define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_LIST 0x00b0
 #define HVCALL_MODIFY_SPARSE_GPA_PAGE_HOST_VISIBILITY 0x00db
+#define HVCALL_MMIO_READ			0x0106
+#define HVCALL_MMIO_WRITE			0x0107
 
 /* Extended hypercalls */
 #define HV_EXT_CALL_QUERY_CAPABILITIES		0x8001
@@ -790,4 +792,24 @@ struct hv_memory_hint {
 	union hv_gpa_page_range ranges[];
 } __packed;
 
+/* Data structures for HVCALL_MMIO_READ and HVCALL_MMIO_WRITE */
+#define HV_HYPERCALL_MMIO_MAX_DATA_LENGTH 64
+
+struct hv_mmio_read_input {
+	u64 gpa;
+	u32 size;
+	u32 reserved;
+} __packed;
+
+struct hv_mmio_read_output {
+	u8 data[HV_HYPERCALL_MMIO_MAX_DATA_LENGTH];
+} __packed;
+
+struct hv_mmio_write_input {
+	u64 gpa;
+	u32 size;
+	u32 reserved;
+	u8 data[HV_HYPERCALL_MMIO_MAX_DATA_LENGTH];
+} __packed;
+
 #endif
Signature · 1 line
1.8.3.1
#32 lore raw

PATCH v4 13/13 PCI: hv: Enable PCI pass-thru devices in Confidential VMs

In reply to Michael Kelley

For PCI pass-thru devices in a Confidential VM, Hyper-V requires that PCI config space be accessed via hypercalls. In normal VMs, config space accesses are trapped to the Hyper-V host and emulated. But in a confidential VM, the host can't access guest memory to decode the instruction for emulation, so an explicit hypercall must be used.

Update the PCI config space access functions to use the hypercalls when such use is indicated by Hyper-V flags. Also, set the flag to allow the Hyper-V PCI driver to be loaded and used in a Confidential VM (a.k.a., "Isolation VM"). The driver has previously been hardened against a malicious Hyper-V host[1].

[1] https://lore.kernel.org/all/[email protected]/

---
 drivers/hv/channel_mgmt.c           |   2 +-
 drivers/pci/controller/pci-hyperv.c | 168 ++++++++++++++++++++++--------------
 2 files changed, 105 insertions(+), 65 deletions(-)
drivers/hv/channel_mgmt.c +1 -1
index cc23b90..007f26d 100644
@@ -67,7 +67,7 @@
 	{ .dev_type = HV_PCIE,
 	  HV_PCIE_GUID,
 	  .perf_device = false,
-	  .allowed_in_isolated = false,
+	  .allowed_in_isolated = true,
 	},
 
 	/* Synthetic Frame Buffer */
drivers/pci/controller/pci-hyperv.c +104 -64

225 lines not on this page · raw message

Signature · 1 line
1.8.3.1
maintainer
#33 lore raw
In reply to Michael Kelley
Quoted 8 lines · Michael Kelley

This patch series adds support for PCI pass-thru devices to Hyper-V Confidential VMs (also called "Isolation VMs"). But in preparation, it first changes how private (encrypted) vs. shared (decrypted) memory is handled in Hyper-V SEV-SNP guest VMs. The new approach builds on the confidential computing (coco) mechanisms introduced in the 5.19 kernel for TDX support and significantly reduces the amount of Hyper-V specific code. Furthermore, with this new approach a proposed RFC patch set for generic DMA layer functionality[1] is no longer necessary.

In any case, this is starting to get ready - how do we merge this?

I apply the x86 bits and give Wei an immutable branch to add the rest of the HyperV stuff ontop?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#34 lore raw

From: Borislav Petkov <[email protected]> Sent: Monday, January 9, 2023 10:47 AM

Quoted 11 lines · Michael Kelley
Quoted 8 lines

This patch series adds support for PCI pass-thru devices to Hyper-V Confidential VMs (also called "Isolation VMs"). But in preparation, it first changes how private (encrypted) vs. shared (decrypted) memory is handled in Hyper-V SEV-SNP guest VMs. The new approach builds on the confidential computing (coco) mechanisms introduced in the 5.19 kernel for TDX support and significantly reduces the amount of Hyper-V specific code. Furthermore, with this new approach a proposed RFC patch set for generic DMA layer functionality[1] is no longer necessary.

Quoted 9 lines

In any case, this is starting to get ready - how do we merge this?

I apply the x86 bits and give Wei an immutable branch to add the rest of the HyperV stuff ontop?

--
Regards/Gruss,
Boris.

I'll let Wei respond on handling the merging.

I'll spin a v5 in a few days. Changes will be:
* Address your comments

* Use PAGE_KERNEL in the arch independent Hyper-V code instead of PAGE_KERNEL_NOENC. PAGE_KERNEL_NOENC doesn't exist for ARM64, so it causes compile errors when building for ARM64. Using PAGE_KERNEL means getting sme_me_mask when on x86, but that value will be zero for vTOM VMs.

* Fix a problem with the virtual TPM device getting mapped decrypted. Like the IOAPIC, the vTPM is provided by the paravisor, and needs to be mapped
encrypted. My thinking is to allow hypervisor initialization code to specify
a guest physical address range to be treated as encrypted, and add a check against that range in __ioremap_check_other(), similar to what is done for EFI memory. Thoughts? I don't want to change the vTPM driver, and the devm_* interfaces it uses don't provide an option to map encrypted anyway. But I'm open to other ideas.

Thanks for the review!

Michael

maintainer
#35 lore raw
In reply to Borislav Petkov
Quoted 10 lines · Borislav Petkov
Quoted 8 lines · Michael Kelley

This patch series adds support for PCI pass-thru devices to Hyper-V Confidential VMs (also called "Isolation VMs"). But in preparation, it first changes how private (encrypted) vs. shared (decrypted) memory is handled in Hyper-V SEV-SNP guest VMs. The new approach builds on the confidential computing (coco) mechanisms introduced in the 5.19 kernel for TDX support and significantly reduces the amount of Hyper-V specific code. Furthermore, with this new approach a proposed RFC patch set for generic DMA layer functionality[1] is no longer necessary.

Quoted 4 lines

In any case, this is starting to get ready - how do we merge this?

I apply the x86 bits and give Wei an immutable branch to add the rest of the HyperV stuff ontop?

I can take all the patches if that's easier for you. I don't think anyone else is depending on the x86 patches in this series.

Giving me an immutable branch works too.

Thanks,
Wei.

Quoted 6 lines
Signature · 5 lines
-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer
#36 lore raw
Quoted 20 lines

From: Wei Liu <[email protected]>
Sent: Thursday, January 12, 2023 6:04 AM
To: Borislav Petkov <[email protected]>
[...]

Quoted 10 lines · Borislav Petkov
Quoted 8 lines · Michael Kelley

This patch series adds support for PCI pass-thru devices to Hyper-V Confidential VMs (also called "Isolation VMs"). But in preparation, it first changes how private (encrypted) vs. shared (decrypted) memory is handled in Hyper-V SEV-SNP guest VMs. The new approach builds on the confidential computing (coco) mechanisms introduced in the 5.19 kernel for TDX support and significantly reduces the amount of Hyper-V specific code. Furthermore, with this new approach a proposed RFC patch set for generic DMA layer functionality[1] is no longer necessary.

Quoted 4 lines

In any case, this is starting to get ready - how do we merge this?

I apply the x86 bits and give Wei an immutable branch to add the rest of the HyperV stuff ontop?

Quoted 10 lines

I can take all the patches if that's easier for you. I don't think anyone else is depending on the x86 patches in this series.

Giving me an immutable branch works too.

Thanks,
Wei.

Quoted 3 lines

--
Regards/Gruss,
Boris.

Hi Boris, Wei, any news on this?

maintainer
#37 lore raw
In reply to Wei Liu
Quoted 2 lines · Wei Liu

I can take all the patches if that's easier for you. I don't think anyone else is depending on the x86 patches in this series.

But we have a bunch of changes in tip so I'd prefer if all were in one place.

Quoted 1 line

Giving me an immutable branch works too.

Yap, lemme do that after applying.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer
#38 lore raw
Quoted 10 lines · Borislav Petkov
Quoted 2 lines · Wei Liu

I can take all the patches if that's easier for you. I don't think anyone else is depending on the x86 patches in this series.

But we have a bunch of changes in tip so I'd prefer if all were in one place.

Quoted 1 line

Giving me an immutable branch works too.

Yap, lemme do that after applying.

Ack. Thanks!

Wei.