Add PCI pass-thru support to Hyper-V Confidential VMs
Subsystems (10)
v3 cover Add PCI pass-thru support to Hyper-V Confidential VMs
This patch series adds support for PCI pass-thru devices to Hyper-V Confidential VMs (also called "Isolation VMs"). But in preparation, it first changes how private (encrypted) vs. shared (decrypted) memory is handled in Hyper-V SEV-SNP guest VMs. The new approach builds on the confidential computing (coco) mechanisms introduced in the 5.19 kernel for TDX support and significantly reduces the amount of Hyper-V specific code. Furthermore, with this new approach a proposed RFC patch set for generic DMA layer functionality[1] is no longer necessary.
Background
==========
Hyper-V guests on AMD SEV-SNP hardware have the option of using the "virtual Top Of Memory" (vTOM) feature specified by the SEV-SNP architecture. With vTOM, shared vs. private memory accesses are controlled by splitting the guest physical address space into two halves. vTOM is the dividing line where the uppermost bit of the physical address space is set; e.g., with 47 bits of guest physical address space, vTOM is 0x40000000000 (bit 46 is set). Guest phyiscal memory is accessible at two parallel physical addresses -- one below vTOM and one above vTOM. Accesses below vTOM are private (encrypted) while accesses above vTOM are shared (decrypted). In this sense, vTOM is like the GPA.SHARED bit in Intel TDX.
In Hyper-V's use of vTOM, the normal guest OS runs at VMPL2, while a Hyper-V provided "paravisor" runs at VMPL0 in the guest VM. (VMPL is Virtual Machine Privilege Level. See AMD's SEV-SNP spec for more details.) The paravisor provides emulation for various system devices like the I/O APIC as part of the guest VM. Accesses to such devices made by the normal guest OS trap to the paravisor and are emulated in the guest VM context instead of in the Hyper-V host. This emulation is invisible to the normal guest OS, but with the quirk that memory mapped I/O accesses to these devices must be treated as private, not shared as would be the case for other device accesses.
Support for Hyper-V guests using vTOM was added to the Linux kernel in two patch sets[2][3]. This support treats the vTOM bit as part of the physical address. For accessing shared (decrypted) memory, the core approach is to create a second kernel virtual mapping that maps to parallel physical addresses above vTOM, while leaving the original mapping unchanged. Most of the code for creating that second virtual mapping is confined to Hyper-V specific areas, but there are are also changes to generic swiotlb code.
Changes in this patch set
=========================
In preparation for supporting PCI pass-thru devices, this patch set changes the core approach for handling vTOM. In the new approach, the vTOM bit is treated as a protection flag, and not as part of the physical address. This new approach is like the approach for the GPA.SHARED bit in Intel TDX. Furthermore, there's no need to create a second kernel virtual mapping. When memory is changed between private and shared using set_memory_decrypted() and set_memory_encrypted(), the PTEs for the existing kernel mapping are changed to add or remove the vTOM bit just as with TDX. The hypercalls to change the memory status on the host side are made using the existing callback mechanism. Everything just works, with a minor tweak to map the I/O APIC to use private accesses as mentioned above.
With the new handling of vTOM in place, existing Hyper-V code that creates the second kernel virtual mapping still works, but it is now redundant as the original kernel virtual mapping (as updated) maps to the same physical address. To simplify things going forward, this patch set removes the code that creates the second kernel virtual mapping. And since a second kernel virtual mapping is no longer needed, changes to the DMA layer proposed as an RFC[1] are no longer needed.
Finally, to support PCI pass-thru in a Confidential VM, Hyper-V requires that all accesses to PCI config space be emulated using a hypercall. This patch set adds functions to invoke those hypercalls and uses them in the config space access functions in the Hyper-V PCI driver. Lastly, the Hyper-V PCI driver is marked as allowed to be used in a Confidential VM. The Hyper-V PCI driver has been hardened against a malicious Hyper-V in a previous patch set.[4]
Patch Organization
==================
Patches 1 thru 6 are prepatory patches that account for slightly different assumptions when running in a Hyper-V VM with vTOM, fix some minor bugs, and make temporary tweaks to avoid needing a single large patch to make the transition from the old approach to the new approach.
Patch 7 enables the new approach to handling vTOM for Hyper-V guest VMs. This is the core patch after which the new approach is in effect.
Patches 8 thru 11 remove existing code for creating the second kernel virtual mapping that is no longer necessary with the new approach.
Patch 12 updates existing code so that it no longer assumes that the vTOM bit is part of the physical address.
Patches 13 and 14 add new hypercalls for accessing MMIO space and use those hypercalls for PCI config space. They also enable the Hyper-V vPCI driver to be used in a Confidential VM.
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://lore.kernel.org/all/[email protected]/
---
Changes in v3:
* Patch 1: Tweak the code fix to cleanly separate the page alignment and physical address masking [Dave Hansen]
* Patch 2: Change the name of the new CC_ATTR that controls whether the IO-APIC is mapped decrypted [Dave Hansen]
* Patch 5 (now patch 7): Add CC_ATTR_MEM_ENCRYPT to what Hyper-V vTOM reports as 'true'. With the addition, Patches 5 and 6 are new to accomodate working correctly with Hyper-V VMs using vTOM. [Tom Lendacky]
Changes in v2:
* Patch 11: Include more detail in the error message if an MMIO hypercall fails. [Bjorn Helgaas]
* Patch 12: Restore removed memory barriers. It seems like these barriers should not be needed because of the spin_unlock() calls, but commit bdd74440d9e8 indicates that they are. This patch series will leave the barriers unchanged; whether they are really needed can be sorted out separately. [Boqun Feng]
Michael Kelley (14):
x86/ioremap: Fix page aligned size calculation in __ioremap_caller() x86/ioapic: Gate decrypted mapping on cc_platform_has() attribute x86/hyperv: Reorder code in prep for subsequent patch
Drivers: hv: Explicitly request decrypted in vmap_pfn() calls x86/mm: Handle decryption/re-encryption of bss_decrypted consistently
init: Call mem_encrypt_init() after Hyper-V hypercall init is done x86/hyperv: Change vTOM handling to use standard coco mechanisms
swiotlb: Remove bounce buffer remapping for Hyper-V
Drivers: hv: vmbus: Remove second mapping of VMBus monitor pages
Drivers: hv: vmbus: Remove second way of mapping ring buffers
hv_netvsc: Remove second mapping of send and recv buffers
Drivers: hv: Don't remap addresses that are above shared_gpa_boundary
PCI: hv: Add hypercalls to read/write MMIO space
PCI: hv: Enable PCI pass-thru devices in Confidential VMs
arch/x86/coco/core.c | 11 +- arch/x86/hyperv/hv_init.c | 18 +-- arch/x86/hyperv/ivm.c | 121 +++++++++---------- arch/x86/include/asm/hyperv-tlfs.h | 3 + arch/x86/include/asm/mshyperv.h | 8 +- arch/x86/kernel/apic/io_apic.c | 3 +- arch/x86/kernel/cpu/mshyperv.c | 22 ++-- arch/x86/mm/ioremap.c | 8 +- arch/x86/mm/mem_encrypt_amd.c | 10 +- arch/x86/mm/pat/set_memory.c | 3 - drivers/hv/Kconfig | 1 - drivers/hv/channel_mgmt.c | 2 +- drivers/hv/connection.c | 113 +++++------------- drivers/hv/hv.c | 23 ++-- drivers/hv/hv_common.c | 11 -- drivers/hv/hyperv_vmbus.h | 2 - drivers/hv/ring_buffer.c | 62 ++++------ drivers/net/hyperv/hyperv_net.h | 2 - drivers/net/hyperv/netvsc.c | 48 +------- drivers/pci/controller/pci-hyperv.c | 232 ++++++++++++++++++++++++++---------- include/asm-generic/hyperv-tlfs.h | 22 ++++ include/asm-generic/mshyperv.h | 2 - include/linux/cc_platform.h | 12 ++ include/linux/swiotlb.h | 2 - init/main.c | 19 +-- kernel/dma/swiotlb.c | 45 +------ 26 files changed, 380 insertions(+), 425 deletions(-)
Signature · 1 line
1.8.3.1
PATCH v3 1/14 x86/ioremap: Fix page aligned size calculation in __ioremap_caller()
Current code re-calculates the size after aligning the starting and ending physical addresses on a page boundary. But the re-calculation also embeds the masking of high order bits that exceed the size of the physical address space (via PHYSICAL_PAGE_MASK). If the masking removes any high order bits, the size calculation results in a huge value that is likely to immediately fail.
Fix this by re-calculating the page-aligned size first. Then mask any high order bits using PHYSICAL_PAGE_MASK.
--- arch/x86/mm/ioremap.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
arch/x86/mm/ioremap.c +7 -1
@@ -217,9 +217,15 @@ static void __ioremap_check_mem(resource_size_t addr, unsigned long size,* Mappings have to be page-aligned */ offset = phys_addr & ~PAGE_MASK ; - phys_addr &=PHYSICAL_PAGE_MASK ; + phys_addr &=PAGE_MASK ; size =PAGE_ALIGN (last_addr+1 ) - phys_addr; +/* +* Mask out any bits not part of the actual physical +* address, like memory encryption bits. +*/ + phys_addr &=PHYSICAL_PAGE_MASK ; + retval =memtype_reserve (phys_addr, (u64 )phys_addr + size, pcm, &new_pcm);if (retval) {
Signature · 1 line
1.8.3.1
Borislav Petkov
maintainerRe: [Patch v3 01/14] x86/ioremap: Fix page aligned size calculation in __ioremap_caller()
Quoted 27 lines · Michael Kelley
Current code re-calculates the size after aligning the starting and ending physical addresses on a page boundary. But the re-calculation also embeds the masking of high order bits that exceed the size of the physical address space (via
PHYSICAL_PAGE_MASK). If the masking removes any high order bits, the size calculation results in a huge value that is likely to immediately fail.Fix this by re-calculating the page-aligned size first. Then mask any high order bits using
PHYSICAL_PAGE_MASK.Signed-off-by Michael Kelley <[email protected]>--- arch/x86/mm/ioremap.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)arch/x86/mm/ioremap.c +1
-1@@ -217,9 +217,15 @@ static void __ioremap_check_mem(resource_size_t addr, unsigned long size,* Mappings have to be page-aligned */ offset = phys_addr & ~PAGE_MASK ; - phys_addr &=PHYSICAL_PAGE_MASK ; + phys_addr &=PAGE_MASK ; size =PAGE_ALIGN (last_addr+1 ) - phys_addr;
arch/x86/mm/ioremap.c static void __ioremap_check_mem(resource_size_t addr, unsigned long size, · line 231
Quoted 10 lines
arch/x86/mm/ioremap.c +6
-0+/* +* Mask out any bits not part of the actual physical +* address, like memory encryption bits. +*/ + phys_addr &=PHYSICAL_PAGE_MASK ; + retval =memtype_reserve (phys_addr, (u64 )phys_addr + size, pcm, &new_pcm);if (retval) {
This looks like a fix to me that needs to go to independently to stable. And it would need a Fixes tag.
/me does some git archeology...
I guess this one:
ffa71f33a820 ("x86, ioremap: Fix incorrect physical address handling in PAE mode")
should be old enough so that it goes to all relevant stable kernels...
Hmm?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRE: [Patch v3 01/14] x86/ioremap: Fix page aligned size calculation in __ioremap_caller()
From: Borislav Petkov <[email protected]> Sent: Monday, November 21, 2022 5:33 AM
Quoted 46 lines · Michael Kelley
arch/x86/mm/ioremap.c static void __ioremap_check_mem(resource_size_t addr,
Quoted 20 lines
Current code re-calculates the size after aligning the starting and ending physical addresses on a page boundary. But the re-calculation also embeds the masking of high order bits that exceed the size of the physical address space (via
PHYSICAL_PAGE_MASK). If the masking removes any high order bits, the size calculation results in a huge value that is likely to immediately fail.Fix this by re-calculating the page-aligned size first. Then mask any high order bits using
PHYSICAL_PAGE_MASK.Signed-off-by Michael Kelley <[email protected]>--- arch/x86/mm/ioremap.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)arch/x86/mm/ioremap.c +0
-0@@ -217,9 +217,15 @@ static void __ioremap_check_mem(resource_size_t addr,unsigned long size,
Quoted 7 lines
arch/x86/mm/ioremap.c +1
-1* Mappings have to be page-aligned */ offset = phys_addr & ~PAGE_MASK ; - phys_addr &=PHYSICAL_PAGE_MASK ; + phys_addr &=PAGE_MASK ; size =PAGE_ALIGN (last_addr+1 ) - phys_addr;arch/x86/mm/ioremap.c static void __ioremap_check_mem(resource_size_t addr, unsigned long size,
Quoted 10 lines
arch/x86/mm/ioremap.c +6
-1+/* +* Mask out any bits not part of the actual physical +* address, like memory encryption bits. +*/ + phys_addr &=PHYSICAL_PAGE_MASK ; + retval =memtype_reserve (phys_addr, (u64 )phys_addr + size, pcm, &new_pcm);if (retval) { --This looks like a fix to me that needs to go to independently to stable. And it would need a Fixes tag.
/me does some git archeology...
Quoted 8 lines
I guess this one:
ffa71f33a820("x86, ioremap: Fix incorrect physical address handling in PAE mode")should be old enough so that it goes to all relevant stable kernels...
Hmm?
As discussed in a parallel thread [1], the incorrect code here doesn't have any real impact in already released Linux kernels. It only affects the transition that my patch series implements to change the way vTOM is handled.
I don't know what the tradeoffs are for backporting a fix that doesn't solve a real problem vs. just letting it be. Every backport carries some overhead in the process and there's always a non-zero risk of breaking something. I've leaned away from adding the "Fixes:" tag in such cases. But if it's better to go ahead and add the "Fixes:" tag for what's only a theoretical problem, I'm OK with doing so.
Michael
Borislav Petkov
maintainerRe: [Patch v3 01/14] x86/ioremap: Fix page aligned size calculation in __ioremap_caller()
Quoted 4 lines · Michael Kelley (LINUX)
As discussed in a parallel thread [1], the incorrect code here doesn't have any real impact in already released Linux kernels. It only affects the transition that my patch series implements to change the way vTOM is handled.
Are you sure?
PHYSICAL_PAGE_MASK is controlled by __PHYSICAL_MASK which is determined by CONFIG_DYNAMIC_PHYSICAL_MASK and __PHYSICAL_MASK_SHIFT which all differ depending on configurations and also dynamic.
It is probably still ok, in probably all possible cases even though I wouldn't bet on it.
And this fix is simple and all clear so lemme ask it differently: what would be any downsides in backporting it to stable, just in case?
Quoted 3 lines
I don't know what the tradeoffs are for backporting a fix that doesn't solve a real problem vs. just letting it be. Every backport carries some overhead in the process
Have you seen the deluge of stable fixes? :-)
Quoted 1 line
and there's always a non-zero risk of breaking something.
I don't see how this one would cause any breakage...
Quoted 3 lines
I've leaned away from adding the "Fixes:" tag in such cases. But if it's better to go ahead and add the "Fixes:" tag for what's only a theoretical problem, I'm OK with doing so.
I think this is a good to have fix anyway as it is Obviously Correct(tm).
Unless you have any reservations you haven't shared yet...
Quoted 1 line
Btw, the proper way to reference to a mail message now is simply to do:
https://lore.kernel.org/r/<Message-ID>
as long as it has been posted on some ML which lore archives. And I think it archives all.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRE: [Patch v3 01/14] x86/ioremap: Fix page aligned size calculation in __ioremap_caller()
From: Borislav Petkov <[email protected]> Sent: Monday, November 21, 2022 11:45 AM
Quoted 13 lines · Michael Kelley (LINUX)
Quoted 4 lines
As discussed in a parallel thread [1], the incorrect code here doesn't have any real impact in already released Linux kernels. It only affects the transition that my patch series implements to change the way vTOM is handled.
Are you sure?
PHYSICAL_PAGE_MASKis controlled by__PHYSICAL_MASKwhich is determined byCONFIG_DYNAMIC_PHYSICAL_MASKand__PHYSICAL_MASK_SHIFTwhich all differ depending on configurations and also dynamic.
Quoted 5 lines
It is probably still ok, in probably all possible cases even though I wouldn't bet on it.
And this fix is simple and all clear so lemme ask it differently: what would be any downsides in backporting it to stable, just in case?
None
Quoted 11 lines
Quoted 4 lines
I don't know what the tradeoffs are for backporting a fix that doesn't solve a real problem vs. just letting it be. Every backport carries some overhead in the process
Have you seen the deluge of stable fixes? :-)
Quoted 1 line
and there's always a non-zero risk of breaking something.
I don't see how this one would cause any breakage...
Quoted 9 lines
Quoted 3 lines
I've leaned away from adding the "Fixes:" tag in such cases. But if it's better to go ahead and add the "Fixes:" tag for what's only a theoretical problem, I'm OK with doing so.
I think this is a good to have fix anyway as it is Obviously Correct(tm).
Unless you have any reservations you haven't shared yet...
No reservations. I'll add the "Fixes:" tag.
Michael
Re: [Patch v3 01/14] x86/ioremap: Fix page aligned size calculation in __ioremap_caller()
In reply to Michael KelleyQuoted 11 lines · Michael Kelley
Current code re-calculates the size after aligning the starting and ending physical addresses on a page boundary. But the re-calculation also embeds the masking of high order bits that exceed the size of the physical address space (via
PHYSICAL_PAGE_MASK). If the masking removes any high order bits, the size calculation results in a huge value that is likely to immediately fail.Fix this by re-calculating the page-aligned size first. Then mask any high order bits using
PHYSICAL_PAGE_MASK.Signed-off-by Michael Kelley <[email protected]>
Looks good:
Although I do agree with Boris that this superficially looks like something that's important to backport. It would be best to either beef up the changelog to explain why that's not the case, or to treat this as an actual fix and submit separately.
RE: [Patch v3 01/14] x86/ioremap: Fix page aligned size calculation in __ioremap_caller()
From: Dave Hansen <[email protected]> Sent: Monday, November 21, 2022 10:14 AM
Quoted 14 lines · Michael Kelley
Quoted 11 lines
Current code re-calculates the size after aligning the starting and ending physical addresses on a page boundary. But the re-calculation also embeds the masking of high order bits that exceed the size of the physical address space (via
PHYSICAL_PAGE_MASK). If the masking removes any high order bits, the size calculation results in a huge value that is likely to immediately fail.Fix this by re-calculating the page-aligned size first. Then mask any high order bits using
PHYSICAL_PAGE_MASK.Signed-off-by Michael Kelley <[email protected]>
Quoted 8 lines
Looks good:
Acked-by Dave Hansen <[email protected]>Although I do agree with Boris that this superficially looks like something that's important to backport. It would be best to either beef up the changelog to explain why that's not the case, or to treat this as an actual fix and submit separately.
You and Boris agree and I have no objection, so I'll add the "Fixes:" tag. I'd like to keep the patch as part of this series because it *is* needed to make the series work.
Michael
Borislav Petkov
maintainerRe: [Patch v3 01/14] x86/ioremap: Fix page aligned size calculation in __ioremap_caller()
Quoted 3 lines · Michael Kelley (LINUX)
You and Boris agree and I have no objection, so I'll add the "Fixes:" tag. I'd like to keep the patch as part of this series because it *is* needed to make the series work.
Yeah, no worries. I can take it tomorrow through urgent and send it to Linus this week so whatever you rebase your tree on, it should already have it.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette[tip: x86/urgent] x86/ioremap: Fix page aligned size calculation in __ioremap_caller()
In reply to Michael KelleyThe following commit has been merged into the x86/urgent branch of tip:
The following commit has been merged into the x86/urgent branch of tip:
Commit-ID: 4dbd6a3e90e03130973688fd79e19425f720d999 Gitweb: https://git.kernel.org/tip/4dbd6a3e90e03130973688fd79e19425f720d999 Author: Michael Kelley <[email protected]> AuthorDate: Wed, 16 Nov 2022 10:41:24 -08:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Tue, 22 Nov 2022 12:21:16 +01:00
x86/ioremap: Fix page aligned size calculation in __ioremap_caller()
Current code re-calculates the size after aligning the starting and ending physical addresses on a page boundary. But the re-calculation also embeds the masking of high order bits that exceed the size of the physical address space (via PHYSICAL_PAGE_MASK). If the masking removes any high order bits, the size calculation results in a huge value that is likely to immediately fail.
Fix this by re-calculating the page-aligned size first. Then mask any high order bits using PHYSICAL_PAGE_MASK.
--- arch/x86/mm/ioremap.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
arch/x86/mm/ioremap.c +7 -1
@@ -217,9 +217,15 @@ __ioremap_caller(resource_size_t phys_addr, unsigned long size,* Mappings have to be page-aligned */ offset = phys_addr & ~PAGE_MASK ; - phys_addr &=PHYSICAL_PAGE_MASK ; + phys_addr &=PAGE_MASK ; size =PAGE_ALIGN (last_addr+1 ) - phys_addr; +/* +* Mask out any bits not part of the actual physical +* address, like memory encryption bits. +*/ + phys_addr &=PHYSICAL_PAGE_MASK ; + retval =memtype_reserve (phys_addr, (u64 )phys_addr + size, pcm, &new_pcm);if (retval) {
PATCH v3 2/14 x86/ioapic: Gate decrypted mapping on cc_platform_has() attribute
In reply to Michael KelleyCurrent code always maps the IOAPIC as shared (decrypted) in a confidential VM. But Hyper-V guest VMs on AMD SEV-SNP with vTOM enabled use a paravisor running in VMPL0 to emulate the IOAPIC. In such a case, the IOAPIC must be accessed as private (encrypted).
Fix this by gating the IOAPIC decrypted mapping on a new cc_platform_has() attribute that a subsequent patch in the series will set only for Hyper-V guests.
--- arch/x86/kernel/apic/io_apic.c | 3 ++- include/linux/cc_platform.h | 12 ++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-)
arch/x86/kernel/apic/io_apic.c +2 -1
@@ -2686,7 +2686,8 @@ static void io_apic_set_fixmap(enum fixed_addresses idx, phys_addr_t phys)* Ensure fixmaps for IOAPIC MMIO respect memory encryption pgprot * bits, just like normal ioremap(): */ - flags =pgprot_decrypted (flags); +if (!cc_platform_has (CC_ATTR_EMULATED_IOAPIC )) + flags =pgprot_decrypted (flags);__set_fixmap (idx, phys, flags); }
include/linux/cc_platform.h +12 -0
@@ -90,6 +90,18 @@ enum cc_attr {* Examples include TDX Guest. */ CC_ATTR_HOTPLUG_DISABLED , + +/** +* @CC_ATTR_EMULATED_IOAPIC: Guest VM has an emulated I/O APIC +* +* The platform/OS is running as a guest/virtual machine with +* an I/O APIC that is emulated by a paravisor running in the +* guest VM context. As such, the I/O APIC is accessed in the +* encrypted portion of the guest physical address space. +* +* Examples include Hyper-V SEV-SNP guests using vTOM. +*/ + CC_ATTR_EMULATED_IOAPIC, };#ifdef CONFIG_ARCH_HAS_CC_PLATFORM
Signature · 1 line
1.8.3.1
Re: [Patch v3 02/14] x86/ioapic: Gate decrypted mapping on cc_platform_has() attribute
Quoted 5 lines · Michael Kelley
Current code always maps the IOAPIC as shared (decrypted) in a confidential VM. But Hyper-V guest VMs on AMD SEV-SNP with vTOM enabled use a paravisor running in VMPL0 to emulate the IOAPIC. In such a case, the IOAPIC must be accessed as private (encrypted).
Quoted 7 lines
Fix this by gating the IOAPIC decrypted mapping on a new
cc_platform_has()attribute that a subsequent patch in the series will set only for Hyper-V guests.Signed-off-by Michael Kelley <[email protected]>Reviewed-by Wei Liu <[email protected]>---
Looks fine to me.
Quoted 41 lines
arch/x86/kernel/apic/io_apic.c | 3 ++- include/linux/cc_platform.h | 12 ++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-)arch/x86/kernel/apic/io_apic.c +2
-1@@ -2686,7 +2686,8 @@ static void io_apic_set_fixmap(enum fixed_addresses idx, phys_addr_t phys)* Ensure fixmaps for IOAPIC MMIO respect memory encryption pgprot * bits, just like normal ioremap(): */ - flags =pgprot_decrypted (flags); +if (!cc_platform_has (CC_ATTR_EMULATED_IOAPIC )) + flags =pgprot_decrypted (flags);__set_fixmap (idx, phys, flags); }include/linux/cc_platform.h +12
-0@@ -90,6 +90,18 @@ enum cc_attr {* Examples include TDX Guest. */ CC_ATTR_HOTPLUG_DISABLED , + +/** +* @CC_ATTR_EMULATED_IOAPIC: Guest VM has an emulated I/O APIC +* +* The platform/OS is running as a guest/virtual machine with +* an I/O APIC that is emulated by a paravisor running in the +* guest VM context. As such, the I/O APIC is accessed in the +* encrypted portion of the guest physical address space. +* +* Examples include Hyper-V SEV-SNP guests using vTOM. +*/ + CC_ATTR_EMULATED_IOAPIC, };#ifdef CONFIG_ARCH_HAS_CC_PLATFORM
Signature · 2 lines
Sathyanarayanan Kuppuswamy Linux Kernel Developer
Borislav Petkov
maintainerRe: [Patch v3 02/14] x86/ioapic: Gate decrypted mapping on cc_platform_has() attribute
In reply to Michael KelleyQuoted 3 lines · Michael Kelley
Current code always maps the IOAPIC as shared (decrypted) in a confidential VM. But Hyper-V guest VMs on AMD SEV-SNP with vTOM enabled use a paravisor running in VMPL0 to emulate the IOAPIC.
"IO-APIC" I guess, in all your text.
Quoted 1 line
In such a case, the IOAPIC must be accessed as private (encrypted).
So the condition for the IO-APIC is pretty specific but the naming CC_ATTR_EMULATED_IOAPIC too generic. Other HVs emulate IO-APICs too, right?
If you have to be precise, the proper check should be (pseudo code):
if (cc_vendor(HYPERV) &&
SNP enabled &&
SNP features has vTOM &&
paravisor in use)so I guess you're probably better off calling it
CC_ATTR_ACCESS_IOAPIC_ENCRYPTED
which then gets set on exactly those guests and nothing else.
I'd say.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRE: [Patch v3 02/14] x86/ioapic: Gate decrypted mapping on cc_platform_has() attribute
From: Borislav Petkov <[email protected]> Sent: Monday, November 21, 2022 5:51 AM
Quoted 21 lines · Michael Kelley
Quoted 3 lines
Current code always maps the IOAPIC as shared (decrypted) in a confidential VM. But Hyper-V guest VMs on AMD SEV-SNP with vTOM enabled use a paravisor running in VMPL0 to emulate the IOAPIC.
"IO-APIC" I guess, in all your text.
Quoted 1 line
In such a case, the IOAPIC must be accessed as private (encrypted).
So the condition for the IO-APIC is pretty specific but the naming
CC_ATTR_EMULATED_IOAPICtoo generic. Other HVs emulate IO-APICs too, right?If you have to be precise, the proper check should be (pseudo code):
if (
cc_vendor(HYPERV)&&SNP enabled && SNP features has vTOM && paravisor in use)
Quoted 8 lines
so I guess you're probably better off calling it
CC_ATTR_ACCESS_IOAPIC_ENCRYPTEDwhich then gets set on exactly those guests and nothing else.
I'd say.
I'm OK with naming it very narrowly. When/if there's a more general case later, we can generalize to whatever degree is appropriate.
Michael
Borislav Petkov
maintainerRe: [Patch v3 02/14] x86/ioapic: Gate decrypted mapping on cc_platform_has() attribute
Quoted 2 lines · Michael Kelley (LINUX)
I'm OK with naming it very narrowly. When/if there's a more general case later, we can generalize to whatever degree is appropriate.
Exactly.
Those defines are free to change when we see fit.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v3 3/14 x86/hyperv: Reorder code in prep for subsequent patch
In reply to Michael KelleyReorder some code as preparation for a subsequent patch. No functional change.
--- arch/x86/hyperv/ivm.c | 68 +++++++++++++++++++++++++-------------------------- 1 file changed, 34 insertions(+), 34 deletions(-)
arch/x86/hyperv/ivm.c +34 -34
@@ -235,40 +235,6 @@ void hv_ghcb_msr_read(u64 msr, u64 *value)EXPORT_SYMBOL_GPL (hv_ghcb_msr_read);#endif -enum hv_isolation_type hv_get_isolation_type (void ) -{ -if (!(ms_hyperv.priv_high &HV_ISOLATION )) -return HV_ISOLATION_TYPE_NONE ; -return FIELD_GET (HV_ISOLATION_TYPE , ms_hyperv.isolation_config_b); -} -EXPORT_SYMBOL_GPL (hv_get_isolation_type); - -/* -* hv_is_isolation_supported - Check system runs in the Hyper-V -* isolation VM. -*/ -bool hv_is_isolation_supported (void ) -{ -if (!cpu_feature_enabled (X86_FEATURE_HYPERVISOR )) -return false; - -if (!hypervisor_is_type (X86_HYPER_MS_HYPERV )) -return false; - -return hv_get_isolation_type () !=HV_ISOLATION_TYPE_NONE ; -} - -DEFINE_STATIC_KEY_FALSE (isolation_type_snp); - -/* -* hv_isolation_type_snp - Check system runs in the AMD SEV-SNP based -* isolation VM. -*/ -bool hv_isolation_type_snp (void ) -{ -return static_branch_unlikely (&isolation_type_snp); -} -/* * hv_mark_gpa_visibility - Set pages visible to host via hvcall. *
@@ -387,3 +353,37 @@ void hv_unmap_memory(void *addr) {vunmap (addr); } + +enum hv_isolation_type hv_get_isolation_type (void ) +{ +if (!(ms_hyperv.priv_high &HV_ISOLATION )) +return HV_ISOLATION_TYPE_NONE ; +return FIELD_GET (HV_ISOLATION_TYPE , ms_hyperv.isolation_config_b); +} +EXPORT_SYMBOL_GPL (hv_get_isolation_type); + +/* +* hv_is_isolation_supported - Check system runs in the Hyper-V +* isolation VM. +*/ +bool hv_is_isolation_supported (void ) +{ +if (!cpu_feature_enabled (X86_FEATURE_HYPERVISOR )) +return false; + +if (!hypervisor_is_type (X86_HYPER_MS_HYPERV )) +return false; + +return hv_get_isolation_type () !=HV_ISOLATION_TYPE_NONE ; +} + +DEFINE_STATIC_KEY_FALSE (isolation_type_snp); + +/* +* hv_isolation_type_snp - Check system runs in the AMD SEV-SNP based +* isolation VM. +*/ +bool hv_isolation_type_snp (void ) +{ +return static_branch_unlikely (&isolation_type_snp); +}
Signature · 1 line
1.8.3.1
PATCH v3 4/14 Drivers: hv: Explicitly request decrypted in vmap_pfn() calls
In reply to Michael KelleyIn preparation for a subsequent patch, update vmap_pfn() calls to explicitly request that the mapping be for decrypted access to the memory. There's no change in functionality since the PFNs passed to vmap_pfn() are above the shared_gpa_boundary, implicitly producing a decrypted mapping. But explicitly requesting decrypted allows the code to work before and after a subsequent patch that will cause vmap_pfn() to mask the PFNs to being below the shared_gpa_boundary. While another subsesquent patch removes the vmap_pfn() calls entirely, this temporary tweak avoids the need for a large patch that makes all the changes at once.
--- arch/x86/hyperv/ivm.c | 2 +- drivers/hv/ring_buffer.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/hyperv/ivm.c +1 -1
@@ -343,7 +343,7 @@ void *hv_map_memory(void *addr, unsigned long size) pfns[i] =vmalloc_to_pfn (addr + i *PAGE_SIZE ) + (ms_hyperv.shared_gpa_boundary >>PAGE_SHIFT ); - vaddr =vmap_pfn (pfns, size /PAGE_SIZE ,PAGE_KERNEL_IO ); + vaddr =vmap_pfn (pfns, size /PAGE_SIZE ,pgprot_decrypted (PAGE_KERNEL_NOENC ));kfree (pfns);return vaddr;
drivers/hv/ring_buffer.c +1 -1
@@ -211,7 +211,7 @@ int hv_ringbuffer_init(struct hv_ring_buffer_info *ring_info, ring_info->ring_buffer = (struct hv_ring_buffer *)vmap_pfn (pfns_wraparound, page_cnt *2 -1 , -PAGE_KERNEL ); +pgprot_decrypted (PAGE_KERNEL_NOENC ));kfree (pfns_wraparound);if (!ring_info->ring_buffer)
Signature · 1 line
1.8.3.1
PATCH v3 5/14 x86/mm: Handle decryption/re-encryption of bss_decrypted consistently
In reply to Michael KelleyCurrent code in sme_postprocess_startup() decrypts the bss_decrypted section when sme_me_mask is non-zero. But code in mem_encrypt_free_decrytped_mem() re-encrypts the unused portion based on CC_ATTR_MEM_ENCRYPT. In a Hyper-V guest VM using vTOM, these conditions are not equivalent as sme_me_mask is always zero when using vTOM. Consequently, mem_encrypt_free_decrypted_mem() attempts to re-encrypt memory that was never decrypted.
Fix this in mem_encrypt_free_decrypted_mem() by conditioning the re-encryption on the same test for non-zero sme_me_mask. Hyper-V guests using vTOM don't need the bss_decrypted section to be decrypted, so skipping the decryption/re-encryption doesn't cause a problem.
--- arch/x86/mm/mem_encrypt_amd.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
arch/x86/mm/mem_encrypt_amd.c +7 -3
@@ -513,10 +513,14 @@ void __init mem_encrypt_free_decrypted_mem(void) npages = (vaddr_end - vaddr) >>PAGE_SHIFT ;/* -* The unused memory range was mapped decrypted, change the encryption -* attribute from decrypted to encrypted before freeing it. +* If the unused memory range was mapped decrypted, change the encryption +* attribute from decrypted to encrypted before freeing it. Base the +* re-encryption on the same condition used for the decryption in +* sme_postprocess_startup(). Higher level abstractions, such as +* CC_ATTR_MEM_ENCRYPT, aren't necessarily equivalent in a Hyper-V VM +* using vTOM, where sme_me_mask is always zero. */ -if (cc_platform_has (CC_ATTR_MEM_ENCRYPT )) { +if (sme_get_me_mask ()) { r =set_memory_encrypted (vaddr, npages);if (r) {pr_warn ("failed to free unused decrypted pages\n" );
Signature · 1 line
1.8.3.1
Re: [Patch v3 05/14] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently
Quoted 28 lines · Michael Kelley
Current code in
sme_postprocess_startup()decrypts thebss_decryptedsection whensme_me_maskis non-zero. But code inmem_encrypt_free_decrytped_mem()re-encrypts the unused portion based onCC_ATTR_MEM_ENCRYPT. In a Hyper-V guest VM using vTOM, these conditions are not equivalent assme_me_maskis always zero when using vTOM. Consequently,mem_encrypt_free_decrypted_mem()attempts to re-encrypt memory that was never decrypted.Fix this in
mem_encrypt_free_decrypted_mem()by conditioning the re-encryption on the same test for non-zerosme_me_mask. Hyper-V guests using vTOM don't need thebss_decryptedsection to be decrypted, so skipping the decryption/re-encryption doesn't cause a problem.Signed-off-by Michael Kelley <[email protected]>--- arch/x86/mm/mem_encrypt_amd.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)arch/x86/mm/mem_encrypt_amd.c +0
-1@@ -513,10 +513,14 @@ void __init mem_encrypt_free_decrypted_mem(void) npages = (vaddr_end - vaddr) >>PAGE_SHIFT ;/* -* The unused memory range was mapped decrypted, change the encryption
arch/x86/mm/mem_encrypt_amd.c void __init mem_encrypt_free_decrypted_mem(void) · line 523
Quoted 10 lines
arch/x86/mm/mem_encrypt_amd.c +7
-2-* attribute from decrypted to encrypted before freeing it. +* If the unused memory range was mapped decrypted, change the encryption +* attribute from decrypted to encrypted before freeing it. Base the +* re-encryption on the same condition used for the decryption in +* sme_postprocess_startup(). Higher level abstractions, such as +* CC_ATTR_MEM_ENCRYPT, aren't necessarily equivalent in a Hyper-V VM +* using vTOM, where sme_me_mask is always zero. */ -if (cc_platform_has (CC_ATTR_MEM_ENCRYPT )) { +if (sme_get_me_mask ()) {
To be consistent within this file, you should use sme_me_mask directly.
Thanks,
Tom
Quoted 3 lines
arch/x86/mm/mem_encrypt_amd.c +0
-0r =set_memory_encrypted (vaddr, npages);if (r) {pr_warn ("failed to free unused decrypted pages\n" );
Tom Lendacky
Reviewed-byRe: [Patch v3 05/14] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently
Quoted 9 lines · Tom Lendacky
Quoted 8 lines · Michael Kelley
Current code in
sme_postprocess_startup()decrypts thebss_decryptedsection whensme_me_maskis non-zero. But code inmem_encrypt_free_decrytped_mem()re-encrypts the unused portion based on CC_ATTR_MEM_ENCRYPT. In a Hyper-V guest VM using vTOM, these conditions are not equivalent assme_me_maskis always zero when using vTOM. Consequently,mem_encrypt_free_decrypted_mem()attempts to re-encrypt memory that was never decrypted.
Quoted 7 lines
Quoted 7 lines
Fix this in
mem_encrypt_free_decrypted_mem()by conditioning the re-encryption on the same test for non-zero sme_me_mask. Hyper-V guests using vTOM don't need thebss_decryptedsection to be decrypted, so skipping the decryption/re-encryption doesn't cause a problem.Signed-off-by Michael Kelley <[email protected]>
Meant to add this in the previous reply...
With the change to use sme_me_mask directly
Quoted 32 lines
Quoted 15 lines
---
arch/x86/mm/mem_encrypt_amd.c| 10 +++++++---1 file changed, 7 insertions(+), 3 deletions(-)arch/x86/mm/mem_encrypt_amd.c +1
-2@@ -513,10 +513,14 @@ void __init mem_encrypt_free_decrypted_mem(void) npages = (vaddr_end - vaddr) >> PAGE_SHIFT; /* -* The unused memory range was mapped decrypted, change the encryption -* attribute from decrypted to encrypted before freeing it. +* If the unused memory range was mapped decrypted, change the encryption
Quoted 8 lines
+ * attribute from decrypted to encrypted before freeing it. Base the + * re-encryption on the same condition used for the decryption in + *
sme_postprocess_startup(). Higher level abstractions, such as + *CC_ATTR_MEM_ENCRYPT, aren't necessarily equivalent in a Hyper-V VM + * using vTOM, wheresme_me_maskis always zero.
*/
- if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) {
+ if (sme_get_me_mask()) {To be consistent within this file, you should use
sme_me_maskdirectly.Thanks,
TomQuoted 3 lines
r =
set_memory_encrypted(vaddr, npages);
if (r) {
pr_warn("failed to free unused decrypted pages\n");
RE: [Patch v3 05/14] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently
From: Tom Lendacky <[email protected]> Sent: Wednesday, November 16, 2022 1:16 PM
Quoted 17 lines · Tom Lendacky
Quoted 15 lines · Michael Kelley
Quoted 14 lines
Current code in
sme_postprocess_startup()decrypts thebss_decryptedsection whensme_me_maskis non-zero. But code inmem_encrypt_free_decrytped_mem()re-encrypts the unused portion based on CC_ATTR_MEM_ENCRYPT. In a Hyper-V guest VM using vTOM, these conditions are not equivalent assme_me_maskis always zero when using vTOM. Consequently,mem_encrypt_free_decrypted_mem()attempts to re-encrypt memory that was never decrypted.Fix this in
mem_encrypt_free_decrypted_mem()by conditioning the re-encryption on the same test for non-zero sme_me_mask. Hyper-V guests using vTOM don't need thebss_decryptedsection to be decrypted, so skipping the decryption/re-encryption doesn't cause a problem.
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Signed-off-by Michael Kelley <[email protected]>Meant to add this in the previous reply...
With the change to use
sme_me_maskdirectlyReviewed-by Tom Lendacky <[email protected]>
Thanks for the reviews. And I see your point about sme_me_mask. I had not previously noticed that it is defined in the module, so no need to use a getter function.
Michael
Re: [Patch v3 05/14] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently
In reply to Michael KelleyQuoted 8 lines · Michael Kelley
Current code in
sme_postprocess_startup()decrypts thebss_decryptedsection whensme_me_maskis non-zero. But code inmem_encrypt_free_decrytped_mem()re-encrypts the unused portion based onCC_ATTR_MEM_ENCRYPT. In a Hyper-V guest VM using vTOM, these conditions are not equivalent assme_me_maskis always zero when using vTOM. Consequently,mem_encrypt_free_decrypted_mem()attempts to re-encrypt memory that was never decrypted.
Quoted 6 lines
Fix this in
mem_encrypt_free_decrypted_mem()by conditioning the re-encryption on the same test for non-zerosme_me_mask. Hyper-V guests using vTOM don't need thebss_decryptedsection to be decrypted, so skipping the decryption/re-encryption doesn't cause a problem.
Do you think it needs Fixes tag?
Quoted 27 lines
Signed-off-by Michael Kelley <[email protected]>--- arch/x86/mm/mem_encrypt_amd.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)arch/x86/mm/mem_encrypt_amd.c +7
-3@@ -513,10 +513,14 @@ void __init mem_encrypt_free_decrypted_mem(void) npages = (vaddr_end - vaddr) >>PAGE_SHIFT ;/* -* The unused memory range was mapped decrypted, change the encryption -* attribute from decrypted to encrypted before freeing it. +* If the unused memory range was mapped decrypted, change the encryption +* attribute from decrypted to encrypted before freeing it. Base the +* re-encryption on the same condition used for the decryption in +* sme_postprocess_startup(). Higher level abstractions, such as +* CC_ATTR_MEM_ENCRYPT, aren't necessarily equivalent in a Hyper-V VM +* using vTOM, where sme_me_mask is always zero. */ -if (cc_platform_has (CC_ATTR_MEM_ENCRYPT )) { +if (sme_get_me_mask ()) { r =set_memory_encrypted (vaddr, npages);if (r) {pr_warn ("failed to free unused decrypted pages\n" );
Signature · 2 lines
Sathyanarayanan Kuppuswamy Linux Kernel Developer
RE: [Patch v3 05/14] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently
From: Sathyanarayanan Kuppuswamy <[email protected]>
Quoted 10 lines · Michael Kelley
Quoted 8 lines
Current code in
sme_postprocess_startup()decrypts thebss_decryptedsection whensme_me_maskis non-zero. But code inmem_encrypt_free_decrytped_mem()re-encrypts the unused portion based onCC_ATTR_MEM_ENCRYPT. In a Hyper-V guest VM using vTOM, these conditions are not equivalent assme_me_maskis always zero when using vTOM. Consequently,mem_encrypt_free_decrypted_mem()attempts to re-encrypt memory that was never decrypted.
Quoted 9 lines
Quoted 6 lines
Fix this in
mem_encrypt_free_decrypted_mem()by conditioning the re-encryption on the same test for non-zerosme_me_mask. Hyper-V guests using vTOM don't need thebss_decryptedsection to be decrypted, so skipping the decryption/re-encryption doesn't cause a problem.Do you think it needs Fixes tag?
At least for my purposes, it doesn't. The original assumption that non-zero sme_me_mask and CC_ATTR_MEM_ENCRYPT are equivalent was valid until this patch series where Hyper-V guests are reporting CC_ATTR_MEM_ENCRYPT as "true" but sme_me_mask is zero. This patch series won't be backported, so the old assumption remains valid for older kernels. There's no benefit in backporting the change.
But I had not thought about TDX. In the TDX case, it appears that sme_postprocess_startup() will not decrypt the bss_decrypted section. The corresponding mem_encrypt_free_decrypted_mem() is a no-op unless CONFIG_AMD_MEM_ENCRYPT is set. But maybe if someone builds a kernel image that supports both TDX and AMD encryption, it could break at runtime on a TDX system. I would also note that on a TDX system without CONFIG_AMD_MEM_ENCRYPT, the unused memory in the bss_decrypted section never gets freed.
But check my logic. :-) I'm not averse to adding the Fixes: tag if there's a
scenario for TDX where doing the backport will solve a real problem.
And thanks for reviewing the code!
Michael
Borislav Petkov
maintainerRe: [Patch v3 05/14] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently
Quoted 5 lines · Michael Kelley (LINUX)
But I had not thought about TDX. In the TDX case, it appears that
sme_postprocess_startup()will not decrypt thebss_decryptedsection. The correspondingmem_encrypt_free_decrypted_mem()is a no-op unlessCONFIG_AMD_MEM_ENCRYPTis set. But maybe if someone builds a kernel image that supports both TDX and AMD encryption, it could break
sme_me_mask better be 0 on a kernel with both built in and running as a TDX guest.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [Patch v3 05/14] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently
Quoted 10 lines · Borislav Petkov
Quoted 5 lines · Michael Kelley (LINUX)
But I had not thought about TDX. In the TDX case, it appears that
sme_postprocess_startup()will not decrypt thebss_decryptedsection. The correspondingmem_encrypt_free_decrypted_mem()is a no-op unlessCONFIG_AMD_MEM_ENCRYPTis set. But maybe if someone builds a kernel image that supports both TDX and AMD encryption, it could break
sme_me_maskbetter be 0 on a kernel with both built in and running as a TDX guest.
Yes. It will be 0 in TDX. In sme_enable(), AMD code checks for CPUID support before updating the sme_me_mask.
Signature · 2 lines
Sathyanarayanan Kuppuswamy Linux Kernel Developer
RE: [Patch v3 05/14] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently
From: Sathyanarayanan Kuppuswamy <[email protected]>
Quoted 9 lines · Borislav Petkov
Quoted 7 lines · Michael Kelley (LINUX)
Quoted 5 lines
But I had not thought about TDX. In the TDX case, it appears that
sme_postprocess_startup()will not decrypt thebss_decryptedsection. The correspondingmem_encrypt_free_decrypted_mem()is a no-op unlessCONFIG_AMD_MEM_ENCRYPTis set. But maybe if someone builds a kernel image that supports both TDX and AMD encryption, it could break
Quoted 7 lines
Quoted 3 lines
sme_me_maskbetter be 0 on a kernel with both built in and running as a TDX guest.Yes. It will be 0 in TDX. In
sme_enable(), AMD code checks for CPUID support before updating thesme_me_mask.
Right. But here's my point: With current code and an image built with CONFIG_AMD_MEM_ENCRYPT=y and running as a TDX guest, sme_postprocess_startup() will not decrypt the bss_decrypted section. Then later mem_encrypt_free_decrypted_mem() will run, see that CC_ATTR_MEM_ENCRYPT is true, and try to re-encrypt the memory. In other words, a TDX guest would break in the same way as a Hyper-V vTOM guest would break. This patch fixes the problem for both cases.
The only things I see in the bss_decrypted section are two clock structures In arch/x86/kernel/kvmclock.c, which aren't needed when Hyper-V is the hypervisor. But with a TDX guest on KVM, will *not* decrypting the bss_decrypted section be a problem? I don't know that kvmclock code or why the two clock structures need to be decrypted for AMD mem encryption.
Michael
Borislav Petkov
maintainerRe: [Patch v3 05/14] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently
Quoted 7 lines · Michael Kelley (LINUX)
Right. But here's my point: With current code and an image built with
CONFIG_AMD_MEM_ENCRYPT=yand running as a TDX guest,sme_postprocess_startup()will not decrypt thebss_decryptedsection. Then latermem_encrypt_free_decrypted_mem()will run, see thatCC_ATTR_MEM_ENCRYPTis true, and try to re-encrypt the memory. In other words, a TDX guest would break in the same way as a Hyper-V vTOM guest would break. This patch fixes the problem for both cases.
I guess making the check more concrete by checking sme_me_mask directly along with a comment makes sense.
We need to be very careful here not to fragment the code too much for all the different guest types.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteDexuan Cui
maintainerRE: [Patch v3 05/14] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently
In reply to Michael Kelley (LINUX)Quoted 4 lines
From: Michael Kelley (LINUX) <[email protected]>
Sent: Thursday, November 17, 2022 6:56 PM [...]
Quoted 8 lines
But I had not thought about TDX. In the TDX case, it appears that
sme_postprocess_startup()will not decrypt thebss_decryptedsection. The correspondingmem_encrypt_free_decrypted_mem()is a no-op unlessCONFIG_AMD_MEM_ENCRYPTis set. But maybe if someone builds a kernel image that supports both TDX and AMD encryption, it could break at runtime on a TDX system. I would also note that on a TDX system withoutCONFIG_AMD_MEM_ENCRYPT, the unused memory in thebss_decryptedsection never gets freed.
On a TDX system *with* CONFIG_AMD_MEM_ENCRYPT, the unused memory in the bss_decrypted section also never gets freed due to the below "return;"
I'd suggest a Fixes tag should be added to make sure the distro vendors notice the patch and backport it :-)
BTW, I just posted a similar patch as I didn't notice this patch. I have replied to my patch email, asking people to ignore my patch.
void __init mem_encrypt_free_decrypted_mem(void)
{
unsigned long vaddr, vaddr_end, npages;
int r;
vaddr = (unsigned long)__start_bss_decrypted_unused;
vaddr_end = (unsigned long)__end_bss_decrypted;
npages = (vaddr_end - vaddr) >> PAGE_SHIFT;
/*
* The unused memory range was mapped decrypted, change the encryption
* attribute from decrypted to encrypted before freeing it.
*/
if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) {
r = set_memory_encrypted(vaddr, npages);
if (r) {
pr_warn("failed to free unused decrypted pages\n");
return;
}
}
free_init_pages("unused decrypted", vaddr, vaddr_end);
}Re: [Patch v3 05/14] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently
Quoted 17 lines · Dexuan Cui
Quoted 4 lines
From: Michael Kelley (LINUX) <[email protected]>
Sent: Thursday, November 17, 2022 6:56 PM [...]Quoted 8 lines
But I had not thought about TDX. In the TDX case, it appears that
sme_postprocess_startup()will not decrypt thebss_decryptedsection. The correspondingmem_encrypt_free_decrypted_mem()is a no-op unlessCONFIG_AMD_MEM_ENCRYPTis set. But maybe if someone builds a kernel image that supports both TDX and AMD encryption, it could break at runtime on a TDX system. I would also note that on a TDX system withoutCONFIG_AMD_MEM_ENCRYPT, the unused memory in thebss_decryptedsection never gets freed.On a TDX system *with*
CONFIG_AMD_MEM_ENCRYPT, the unused memory in thebss_decryptedsection also never gets freed due to the below "return;"
Quoted 7 lines
I'd suggest a Fixes tag should be added to make sure the distro vendors notice the patch and backport it :-)
BTW, I just posted a similar patch as I didn't notice this patch. I have replied to my patch email, asking people to ignore my patch.
Fixes b3f0907c71e0 ("x86/mm: Add .bss..decrypted section to hold shared variables")
I think the Fixes: tag should really be:
e9d1d2bb75b2 ("treewide: Replace the use of mem_encrypt_active() with cc_platform_has()")
since mem_encrypt_active() used to return sme_me_mask, so the checks were balanced at that point.
Thanks,
Tom
Quoted 25 lines
void
__initmem_encrypt_free_decrypted_mem(void){ unsigned long vaddr, vaddr_end, npages; int r; vaddr = (unsigned long)__start_bss_decrypted_unused; vaddr_end = (unsigned long)__end_bss_decrypted; npages = (vaddr_end - vaddr) >> PAGE_SHIFT; /* * The unused memory range was mapped decrypted, change the encryption * attribute from decrypted to encrypted before freeing it. */ if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) { r = set_memory_encrypted(vaddr, npages); if (r) { pr_warn("failed to free unused decrypted pages\n"); return; } } free_init_pages("unused decrypted", vaddr, vaddr_end); }
Dexuan Cui
maintainerRE: [Patch v3 05/14] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently
Quoted 11 lines
From: Tom Lendacky <[email protected]>
Sent: Monday, November 28, 2022 6:16 AMQuoted 9 lines
[...]
On a TDX system *with*CONFIG_AMD_MEM_ENCRYPT, the unused memory in thebss_decryptedsection also never gets freed due to the below "return;"I'd suggest a Fixes tag should be added to make sure the distro vendors notice the patch and backport it :-)
[...]Fixes b3f0907c71e0 ("x86/mm: Add .bss..decrypted section to hold shared
Quoted 6 lines
variables")
I think the Fixes: tag should really be:
e9d1d2bb75b2("treewide: Replace the use ofmem_encrypt_active()withcc_platform_has()")
Yes, you're correct.
Borislav Petkov
maintainerRe: [Patch v3 05/14] x86/mm: Handle decryption/re-encryption of bss_decrypted consistently
In reply to Michael KelleyQuoted 3 lines · Michael Kelley
Current code in
sme_postprocess_startup()decrypts thebss_decryptedsection whensme_me_maskis non-zero. But code inmem_encrypt_free_decrytped_mem()re-encrypts the unused portion based
^^
letters flipped.
arch/x86/mm/mem_encrypt_amd.c void __init mem_encrypt_free_decrypted_mem(void) · line 521
Quoted 12 lines
arch/x86/mm/mem_encrypt_amd.c +6
-2@@ -513,10 +513,14 @@ void __init mem_encrypt_free_decrypted_mem(void) npages = (vaddr_end - vaddr) >>PAGE_SHIFT ;/* -* The unused memory range was mapped decrypted, change the encryption -* attribute from decrypted to encrypted before freeing it. +* If the unused memory range was mapped decrypted, change the encryption +* attribute from decrypted to encrypted before freeing it. Base the +* re-encryption on the same condition used for the decryption in +* sme_postprocess_startup(). Higher level abstractions, such as +* CC_ATTR_MEM_ENCRYPT, aren't necessarily equivalent in a Hyper-V VM +* using vTOM, where sme_me_mask is always zero.
Good, an example why one needs to pay attention here.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v3 6/14 init: Call mem_encrypt_init() after Hyper-V hypercall init is done
In reply to Michael KelleyFull Hyper-V initialization, including support for hypercalls, is done as an apic_post_init callback via late_time_init(). mem_encrypt_init() needs to make hypercalls when it marks swiotlb memory as decrypted. But mem_encrypt_init() is currently called a few lines before late_time_init(), so the hypercalls don't work.
Fix this by moving mem_encrypt_init() after late_time_init() and related clock initializations. The intervening initializations don't do any I/O that requires the swiotlb, so moving mem_encrypt_init() slightly later has no impact.
--- init/main.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-)
init/main.c +11 -8
@@ -1088,14 +1088,6 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)*/ locking_selftest (); -/* -* This needs to be called before any devices perform DMA -* operations that might use the SWIOTLB bounce buffers. It will -* mark the bounce buffers as decrypted so that their usage will -* not cause "plain-text" data to be decrypted when accessed. -*/ -mem_encrypt_init (); -#ifdef CONFIG_BLK_DEV_INITRD if (initrd_start && !initrd_below_start_ok &&page_to_pfn (virt_to_page ((void *)initrd_start)) < min_low_pfn) {
@@ -1112,6 +1104,17 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)late_time_init ();sched_clock_init ();calibrate_delay (); + +/* +* This needs to be called before any devices perform DMA +* operations that might use the SWIOTLB bounce buffers. It will +* mark the bounce buffers as decrypted so that their usage will +* not cause "plain-text" data to be decrypted when accessed. It +* must be called after late_time_init() so that Hyper-V x86/x64 +* hypercalls work when the SWIOTLB bounce buffers are decrypted. +*/ +mem_encrypt_init (); +pid_idr_init ();anon_vma_init ();#ifdef CONFIG_X86
Signature · 1 line
1.8.3.1
Re: [Patch v3 06/14] init: Call mem_encrypt_init() after Hyper-V hypercall init is done
Quoted 6 lines · Michael Kelley
Full Hyper-V initialization, including support for hypercalls, is done as an
apic_post_initcallback vialate_time_init().mem_encrypt_init()needs to make hypercalls when it marks swiotlb memory as decrypted. Butmem_encrypt_init()is currently called a few lines beforelate_time_init(), so the hypercalls don't work.
Quoted 6 lines
Fix this by moving
mem_encrypt_init()afterlate_time_init()and related clock initializations. The intervening initializations don't do any I/O that requires the swiotlb, so movingmem_encrypt_init()slightly later has no impact.Signed-off-by Michael Kelley <[email protected]>
Some quick testing with mem_encrypt_init() in the new location hasn't shown any problems under SME/SEV.
Quoted 41 lines
--- init/main.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-)init/main.c +11
-8@@ -1088,14 +1088,6 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)*/ locking_selftest (); -/* -* This needs to be called before any devices perform DMA -* operations that might use the SWIOTLB bounce buffers. It will -* mark the bounce buffers as decrypted so that their usage will -* not cause "plain-text" data to be decrypted when accessed. -*/ -mem_encrypt_init (); -#ifdef CONFIG_BLK_DEV_INITRD if (initrd_start && !initrd_below_start_ok &&page_to_pfn (virt_to_page ((void *)initrd_start)) < min_low_pfn) {@@ -1112,6 +1104,17 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)late_time_init ();sched_clock_init ();calibrate_delay (); + +/* +* This needs to be called before any devices perform DMA +* operations that might use the SWIOTLB bounce buffers. It will +* mark the bounce buffers as decrypted so that their usage will +* not cause "plain-text" data to be decrypted when accessed. It +* must be called after late_time_init() so that Hyper-V x86/x64 +* hypercalls work when the SWIOTLB bounce buffers are decrypted. +*/ +mem_encrypt_init (); +pid_idr_init ();anon_vma_init ();#ifdef CONFIG_X86
Borislav Petkov
maintainerRe: [Patch v3 06/14] init: Call mem_encrypt_init() after Hyper-V hypercall init is done
In reply to Michael KelleyQuoted 4 lines · Michael Kelley
Fix this by moving
mem_encrypt_init()afterlate_time_init()and related clock initializations. The intervening initializations don't do any I/O that requires the swiotlb, so movingmem_encrypt_init()slightly later has no impact.
I hope you're right. Our boot ordering is fragile as hell. But mem_encrypt_init() doesn't do a whole lot of important setup - that has happened a lot earlier already - so I'm not too worried.
But we'll see what breaks in wider testing.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v3 7/14 x86/hyperv: Change vTOM handling to use standard coco mechanisms
In reply to Michael KelleyHyper-V guests on AMD SEV-SNP hardware have the option of using the "virtual Top Of Memory" (vTOM) feature specified by the SEV-SNP architecture. With vTOM, shared vs. private memory accesses are controlled by splitting the guest physical address space into two halves. vTOM is the dividing line where the uppermost bit of the physical address space is set; e.g., with 47 bits of guest physical address space, vTOM is 0x40000000000 (bit 46 is set). Guest phyiscal memory is accessible at two parallel physical addresses -- one below vTOM and one above vTOM. Accesses below vTOM are private (encrypted) while accesses above vTOM are shared (decrypted). In this sense, vTOM is like the GPA.SHARED bit in Intel TDX.
Support for Hyper-V guests using vTOM was added to the Linux kernel in two patch sets[1][2]. This support treats the vTOM bit as part of the physical address. For accessing shared (decrypted) memory, these patch sets create a second kernel virtual mapping that maps to physical addresses above vTOM.
A better approach is to treat the vTOM bit as a protection flag, not as part of the physical address. This new approach is like the approach for the GPA.SHARED bit in Intel TDX. Rather than creating a second kernel virtual mapping, the existing mapping is updated using recently added coco mechanisms. When memory is changed between private and shared using set_memory_decrypted() and set_memory_encrypted(), the PTEs for the existing kernel mapping are changed to add or remove the vTOM bit in the guest physical address, just as with TDX. The hypercalls to change the memory status on the host side are made using the existing callback mechanism. Everything just works, with a minor tweak to map the I/O APIC to use private accesses.
To accomplish the switch in approach, the following must be done in in this single patch:
* Update Hyper-V initialization to set the cc_mask based on vTOM and do other coco initialization.
* Update physical_mask so the vTOM bit is no longer treated as part of the physical address
* Update cc_mkenc() and cc_mkdec() to be active for Hyper-V guests. This makes the vTOM bit part of the protection flags.
* Code already exists to make hypercalls to inform Hyper-V about pages changing between shared and private. Update this code to run as a callback from __set_memory_enc_pgtable().
* Remove the Hyper-V special case from __set_memory_enc_dec()
* Remove the Hyper-V specific call to swiotlb_update_mem_attributes() since mem_encrypt_init() will now do it.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
--- arch/x86/coco/core.c | 11 +++++++++- arch/x86/hyperv/hv_init.c | 11 ---------- arch/x86/hyperv/ivm.c | 45 +++++++++++++++++++++++++++++++---------- arch/x86/include/asm/mshyperv.h | 8 ++------ arch/x86/kernel/cpu/mshyperv.c | 15 +++++++------- arch/x86/mm/pat/set_memory.c | 3 --- 6 files changed, 53 insertions(+), 40 deletions(-)
arch/x86/coco/core.c +10 -1
@@ -78,7 +78,14 @@ static bool amd_cc_platform_has(enum cc_attr attr)static bool hyperv_cc_platform_has (enum cc_attr attr) { -return attr ==CC_ATTR_GUEST_MEM_ENCRYPT ; +switch (attr) { +case CC_ATTR_GUEST_MEM_ENCRYPT : +case CC_ATTR_MEM_ENCRYPT : +case CC_ATTR_EMULATED_IOAPIC : +return true; +default : +return false; + } }bool cc_platform_has (enum cc_attr attr)
@@ -108,6 +115,7 @@ u64 cc_mkenc(u64 val)switch (vendor) {case CC_VENDOR_AMD :return val | cc_mask; +case CC_VENDOR_HYPERV :case CC_VENDOR_INTEL :return val & ~cc_mask;default :
@@ -121,6 +129,7 @@ u64 cc_mkdec(u64 val)switch (vendor) {case CC_VENDOR_AMD :return val & ~cc_mask; +case CC_VENDOR_HYPERV :case CC_VENDOR_INTEL :return val | cc_mask;default :
arch/x86/hyperv/hv_init.c +0 -11
@@ -29,7 +29,6 @@#include <linux/syscore_ops.h> #include <clocksource/hyperv_timer.h> #include <linux/highmem.h> -#include <linux/swiotlb.h> int hyperv_init_cpuhp;u64 hv_current_partition_id = ~0ull ;
@@ -504,16 +503,6 @@ void __init hyperv_init(void)/* Query the VMs extended capability once, so that it can be cached. */ hv_query_ext_cap (0 ); -#ifdef CONFIG_SWIOTLB -/* -* Swiotlb bounce buffer needs to be mapped in extra address -* space. Map function doesn't work in the early place and so -* call swiotlb_update_mem_attributes() here. -*/ -if (hv_is_isolation_supported ()) -swiotlb_update_mem_attributes (); -#endif -return ; clean_guest_os_id:
arch/x86/hyperv/ivm.c +34 -11
91 lines not on this page · raw message
arch/x86/include/asm/mshyperv.h +2 -6
@@ -174,18 +174,19 @@ static inline void hv_apic_init(void) {}int hv_map_ioapic_interrupt (int ioapic_id,bool level,int vcpu,int vector,struct hv_interrupt_entry *entry);int hv_unmap_ioapic_interrupt (int ioapic_id,struct hv_interrupt_entry *entry); -int hv_set_mem_host_visibility (unsigned long addr,int numpages,bool visible);#ifdef CONFIG_AMD_MEM_ENCRYPT void hv_ghcb_msr_write (u64 msr,u64 value);void hv_ghcb_msr_read (u64 msr,u64 *value);bool hv_ghcb_negotiate_protocol (void );void hv_ghcb_terminate (unsigned int set,unsigned int reason); +void hv_vtom_init (void );#else static inline void hv_ghcb_msr_write (u64 msr,u64 value) {}static inline void hv_ghcb_msr_read (u64 msr,u64 *value) {}static inline bool hv_ghcb_negotiate_protocol (void ) {return false; }static inline void hv_ghcb_terminate (unsigned int set,unsigned int reason) {} +static inline void hv_vtom_init (void ) {}#endif extern bool hv_isolation_type_snp (void );
@@ -241,11 +242,6 @@ static inline int hyperv_flush_guest_mapping_range(u64 as, }static inline void hv_set_register (unsigned int reg,u64 value) { }static inline u64 hv_get_register (unsigned int reg) {return 0 ; } -static inline int hv_set_mem_host_visibility (unsigned long addr,int numpages, -bool visible) -{ -return -1 ; -}#endif /* CONFIG_HYPERV */
arch/x86/kernel/cpu/mshyperv.c +7 -8
@@ -33,7 +33,6 @@#include <asm/nmi.h> #include <clocksource/hyperv_timer.h> #include <asm/numa.h> -#include <asm/coco.h> /* Is Linux running as the root partition? */ bool hv_root_partition;
@@ -325,8 +324,10 @@ static void __init ms_hyperv_init_platform(void)if (ms_hyperv.priv_high &HV_ISOLATION ) { ms_hyperv.isolation_config_a =cpuid_eax (HYPERV_CPUID_ISOLATION_CONFIG ); ms_hyperv.isolation_config_b =cpuid_ebx (HYPERV_CPUID_ISOLATION_CONFIG ); - ms_hyperv.shared_gpa_boundary = -BIT_ULL (ms_hyperv.shared_gpa_boundary_bits); + +if (ms_hyperv.shared_gpa_boundary_active) + ms_hyperv.shared_gpa_boundary = +BIT_ULL (ms_hyperv.shared_gpa_boundary_bits);pr_info ("Hyper-V: Isolation Config: Group A 0x%x, Group B 0x%x\n" , ms_hyperv.isolation_config_a, ms_hyperv.isolation_config_b);
@@ -337,11 +338,6 @@ static void __init ms_hyperv_init_platform(void) swiotlb_unencrypted_base = ms_hyperv.shared_gpa_boundary;#endif } -/* Isolation VMs are unenlightened SEV-based VMs, thus this check: */ -if (IS_ENABLED (CONFIG_AMD_MEM_ENCRYPT )) { -if (hv_get_isolation_type () !=HV_ISOLATION_TYPE_NONE ) -cc_set_vendor (CC_VENDOR_HYPERV ); - } }if (hv_max_functions_eax >=HYPERV_CPUID_NESTED_FEATURES ) {
@@ -410,6 +406,9 @@ static void __init ms_hyperv_init_platform(void) i8253_clear_counter_on_shutdown = false;#if IS_ENABLED (CONFIG_HYPERV ) +if ((hv_get_isolation_type () ==HV_ISOLATION_TYPE_VBS ) || + (hv_get_isolation_type () ==HV_ISOLATION_TYPE_SNP )) +hv_vtom_init ();/* * Setup the hook to get control post apic initialization. */
arch/x86/mm/pat/set_memory.c +0 -3
@@ -2126,9 +2126,6 @@ static int __set_memory_enc_pgtable(unsigned long addr, int numpages, bool enc)static int __set_memory_enc_dec (unsigned long addr,int numpages,bool enc) { -if (hv_is_isolation_supported ()) -return hv_set_mem_host_visibility (addr, numpages, !enc); -if (cc_platform_has (CC_ATTR_MEM_ENCRYPT ))return __set_memory_enc_pgtable (addr, numpages, enc);
Signature · 1 line
1.8.3.1
Re: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
Quoted 55 lines · Michael Kelley
Hyper-V guests on AMD SEV-SNP hardware have the option of using the "virtual Top Of Memory" (vTOM) feature specified by the SEV-SNP architecture. With vTOM, shared vs. private memory accesses are controlled by splitting the guest physical address space into two halves. vTOM is the dividing line where the uppermost bit of the physical address space is set; e.g., with 47 bits of guest physical address space, vTOM is
0x40000000000(bit 46 is set). Guest phyiscal memory is accessible at two parallel physical addresses -- one below vTOM and one above vTOM. Accesses below vTOM are private (encrypted) while accesses above vTOM are shared (decrypted). In this sense, vTOM is like the GPA.SHARED bit in Intel TDX.Support for Hyper-V guests using vTOM was added to the Linux kernel in two patch sets[1][2]. This support treats the vTOM bit as part of the physical address. For accessing shared (decrypted) memory, these patch sets create a second kernel virtual mapping that maps to physical addresses above vTOM.
A better approach is to treat the vTOM bit as a protection flag, not as part of the physical address. This new approach is like the approach for the GPA.SHARED bit in Intel TDX. Rather than creating a second kernel virtual mapping, the existing mapping is updated using recently added coco mechanisms. When memory is changed between private and shared using
set_memory_decrypted()andset_memory_encrypted(), the PTEs for the existing kernel mapping are changed to add or remove the vTOM bit in the guest physical address, just as with TDX. The hypercalls to change the memory status on the host side are made using the existing callback mechanism. Everything just works, with a minor tweak to map the I/O APIC to use private accesses.To accomplish the switch in approach, the following must be done in in this single patch:
* Update Hyper-V initialization to set the
cc_maskbased on vTOM and do other coco initialization.* Update
physical_maskso the vTOM bit is no longer treated as part of the physical address* Update
cc_mkenc()andcc_mkdec()to be active for Hyper-V guests. This makes the vTOM bit part of the protection flags.* Code already exists to make hypercalls to inform Hyper-V about pages changing between shared and private. Update this code to run as a callback from
__set_memory_enc_pgtable().* Remove the Hyper-V special case from
__set_memory_enc_dec()* Remove the Hyper-V specific call to
swiotlb_update_mem_attributes()sincemem_encrypt_init()will now do it.[1]https://lore.kernel.org/all/[email protected]/ [2]https://lore.kernel.org/all/[email protected]/
Signed-off-by Michael Kelley<[email protected]>
Quoted 8 lines
--- arch/x86/coco/core.c | 11 +++++++++- arch/x86/hyperv/hv_init.c | 11 ---------- arch/x86/hyperv/ivm.c | 45 +++++++++++++++++++++++++++++++---------- arch/x86/include/asm/mshyperv.h | 8 ++------ arch/x86/kernel/cpu/mshyperv.c | 15 +++++++------- arch/x86/mm/pat/set_memory.c | 3 --- 6 files changed, 53 insertions(+), 40 deletions(-)
Borislav Petkov
maintainerRe: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
In reply to Michael KelleyQuoted 7 lines · Michael Kelley
Hyper-V guests on AMD SEV-SNP hardware have the option of using the "virtual Top Of Memory" (vTOM) feature specified by the SEV-SNP architecture. With vTOM, shared vs. private memory accesses are controlled by splitting the guest physical address space into two halves. vTOM is the dividing line where the uppermost bit of the physical address space is set; e.g., with 47 bits of guest physical address space, vTOM is
0x40000000000(bit 46 is set). Guest phyiscal
Unknown word [phyiscal] in commit message.
Suggestions: ['physical', 'physically', ... ]
Please introduce a spellchecker into your patch creation workflow.
...
Quoted 9 lines
arch/x86/coco/core.c +1
-0@@ -108,6 +115,7 @@ u64 cc_mkenc(u64 val)switch (vendor) {case CC_VENDOR_AMD :return val | cc_mask; +case CC_VENDOR_HYPERV :case CC_VENDOR_INTEL :return val & ~cc_mask;default :
arch/x86/coco/core.c u64 cc_mkdec(u64 val) · line 135
Quoted 7 lines
arch/x86/coco/core.c +1
-0@@ -121,6 +129,7 @@ u64 cc_mkdec(u64 val)switch (vendor) {case CC_VENDOR_AMD :return val & ~cc_mask; +case CC_VENDOR_HYPERV :case CC_VENDOR_INTEL :return val | cc_mask;default :
Uuuh, this needs a BIG FAT COMMENT.
You're running on SNP and yet the enc/dec meaning is flipped. And that's because of vTOM.
What happens if you have other types of SNP-based VMs on HyperV? The isolation VMs thing? Or is that the same?
What happens when you do TDX guests with HyperV?
This becomes wrong then.
I think you need a more finer-grained check here in the sense of "is it a HyperV guest using a paravisor and vTOM is enabled" or so.
Otherwise, I like the removal of the HyperV-specific checks ofc.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRE: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
From: Borislav Petkov <[email protected]> Sent: Monday, November 21, 2022 7:03 AM
Quoted 29 lines · Michael Kelley
...
Quoted 9 lines
arch/x86/coco/core.c +1
-0@@ -108,6 +115,7 @@ u64 cc_mkenc(u64 val)switch (vendor) {case CC_VENDOR_AMD :return val | cc_mask; +case CC_VENDOR_HYPERV :case CC_VENDOR_INTEL :return val & ~cc_mask;default :arch/x86/coco/core.c u64 cc_mkdec(u64 val) · line 135
Quoted 7 lines
arch/x86/coco/core.c +1
-0@@ -121,6 +129,7 @@ u64 cc_mkdec(u64 val)switch (vendor) {case CC_VENDOR_AMD :return val & ~cc_mask; +case CC_VENDOR_HYPERV :case CC_VENDOR_INTEL :return val | cc_mask;default :Uuuh, this needs a BIG FAT COMMENT.
You're running on SNP and yet the enc/dec meaning is flipped. And that's because of vTOM.
What happens if you have other types of SNP-based VMs on HyperV? The isolation VMs thing? Or is that the same?
Quoted 9 lines
What happens when you do TDX guests with HyperV?
This becomes wrong then.
I think you need a more finer-grained check here in the sense of "is it a HyperV guest using a paravisor and vTOM is enabled" or so.
Otherwise, I like the removal of the HyperV-specific checks ofc.
I think the core problem here is the naming and meaning of CC_VENDOR_HYPERV. The name was created originally when the
Hyper-V vTOM handling code was a lot of special cases. With the
changes in this patch series that make the vTOM functionality more mainstream, the name would be better as CC_VENDOR_AMD_VTOM. vTOM is part of the AMD SEV-SNP spec, and it's a different way of doing the encryption from the "C-bit" based approach. As much as possible, I'm trying to not make it be Hyper-V specific, though currently we have N=1 for hypervisors that offer the vTOM option, so it's a little hard to generalize.
With the thinking oriented that way, a Linux guest on Hyper-V using TDX will run with CC_VENDOR_INTEL. A Linux guest on Hyper-V that is fully enlightened to use the "C-bit" will run with CC_VENDOR_AMD.
Dexuan Cui just posted a patch set for initial TDX support on Hyper-V, and I think that runs with CC_VENDOR_INTEL (Dexuan -- correct me if I'm wrong about that -- I haven't reviewed your patches yet). Tianyu Lan has a patch set out for Hyper-V guests using the "C-bit". That patch set still uses CC_VENDOR_HYPERV. Tianyu and I need to work through whether his patch set can run with CC_VENDOR_AMD like everyone else using the "C-bit" approach.
Yes, the polarity of the AMD vTOM bit matches the polarity of the TDX GPA.SHARED bit, and is the opposite polarity of the AMD "C-bit". I'll add a comment to that effect.
Anyway, that's where I think this should go. Does it make sense? Other thoughts?
Michael
Re: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
Quoted 2 lines · Michael Kelley (LINUX)
Anyway, that's where I think this should go. Does it make sense? Other thoughts?
I think hard-coding the C-bit behavior and/or position to a vendor was probably a bad idea. Even the comment:
u64 cc_mkenc(u64 val)
{
/*
* Both AMD and Intel use a bit in the page table to indicate
* encryption status of the page.
*
* - for AMD, bit *set* means the page is encrypted
* - for Intel *clear* means encrypted.
*/
doesn't make a lot of sense now. Maybe we should just have a:
CC_ATTR_ENC_SET
which gets set for the "AMD" behavior and is clear for the "Intel" behavior. Hyper-V code running on AMD can set the attribute to get teh "Intel" behavior.
That sure beats having a Hyper-V vendor.
RE: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
From: Dave Hansen <[email protected]> Sent: Tuesday, November 22, 2022 10:30 AM
Quoted 20 lines · Michael Kelley (LINUX)
Quoted 2 lines
Anyway, that's where I think this should go. Does it make sense? Other thoughts?
I think hard-coding the C-bit behavior and/or position to a vendor was probably a bad idea. Even the comment:
u64
cc_mkenc(u64 val)
{
/*
* Both AMD and Intel use a bit in the page table to indicate
* encryption status of the page.
*
* - for AMD, bit *set* means the page is encrypted
* - for Intel *clear* means encrypted.
*/doesn't make a lot of sense now. Maybe we should just have a:
Quoted 7 lines
CC_ATTR_ENC_SETwhich gets set for the "AMD" behavior and is clear for the "Intel" behavior. Hyper-V code running on AMD can set the attribute to get teh "Intel" behavior.
That sure beats having a Hyper-V vendor.
To me, figuring out the encryption bit polarity and position isn't the hard part to set up. We've got three technologies: TDX, AMD "C-bit", and arguably, AMD vTOM. Future silicon and architectural enhancements will most likely be variations and improvements on these rather than something completely new (not that I'm necessarily
aware of what the processor vendors may have planned). The AMD
"C-bit" technology already has sub-cases (SME, SEV, SEV-ES, SEV-SNP) because of the architectural history. Any of the technologies may get additional subcases over time. Whether those subcases are represented as new CC_VENDOR_* options, or CC_ATTR_* options on an existing CC_VENDOR_* should be driven by the level of commonality with what already exists. There's no hard-and-fast rule. Just do whatever makes the most sense.
I'm proposing AMD vTOM as a separate CC_VENDOR_AMD_VTOM because it is rather different from CC_VENDOR_AMD, and not just in the polarity and position of the encryption bit. But if we really wanted to just make it a sub-case of CC_VENDOR_AMD, I could probably be convinced. The key is cleanly handling all the other attributes like CC_ATTR_GUEST_STATE_ENCRYPT, CC_ATTR_ACCESS_IOAPIC_ENCRYPTED (that I add in this patch set), CC_ATTR_GUEST_UNROLL_STRING_IO, etc. where we want to avoid too many hacky special cases. The current code structure where the CC_VENDOR_* selection determines the CC_ATTR_* values seems to work OK.
Anyway, that's my thinking. CC_VENDOR_HYPERV goes away, but the behavior is still essentially the same when replaced by CC_VENDOR_AMD_VTOM.
Michael
Borislav Petkov
maintainerRe: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
In reply to Michael Kelley (LINUX)Quoted 5 lines · Michael Kelley (LINUX)
I think the core problem here is the naming and meaning of
CC_VENDOR_HYPERV. The name was created originally when the
Hyper-V vTOM handling code was a lot of special cases. With the
changes in this patch series that make the vTOM functionality more mainstream, the name would be better asCC_VENDOR_AMD_VTOM.
No, if CC_VENDOR_HYPERV means different things depending on what kind of guests you're doing, then you should not use a CC_VENDOR at all.
Quoted 5 lines
vTOM is part of the AMD SEV-SNP spec, and it's a different way of doing the encryption from the "C-bit" based approach. As much as possible, I'm trying to not make it be Hyper-V specific, though currently we have N=1 for hypervisors that offer the vTOM option, so it's a little hard to generalize.
Actually, it is very simple to generalize: vTOM and the paravisor and VMPL are all part of the effort to support unenlightened, unmodified guests with SNP.
So, if KVM wants to run Windows NT 4.0 guests as SNP guests, then it probably would need the same contraptions.
Quoted 3 lines
With the thinking oriented that way, a Linux guest on Hyper-V using TDX will run with
CC_VENDOR_INTEL. A Linux guest on Hyper-V that is fully enlightened to use the "C-bit" will run withCC_VENDOR_AMD.
Right.
Quoted 7 lines
Dexuan Cui just posted a patch set for initial TDX support on Hyper-V, and I think that runs with
CC_VENDOR_INTEL(Dexuan -- correct me if I'm wrong about that -- I haven't reviewed your patches yet). Tianyu Lan has a patch set out for Hyper-V guests using the "C-bit". That patch set still usesCC_VENDOR_HYPERV. Tianyu and I need to work through whether his patch set can run withCC_VENDOR_AMDlike everyone else using the "C-bit" approach.
So I'm not sure the vendor is the right approach here. I guess we need to specify the *type* of guest being supported.
Quoted 6 lines
Yes, the polarity of the AMD vTOM bit matches the polarity of the TDX GPA.SHARED bit, and is the opposite polarity of the AMD "C-bit". I'll add a comment to that effect.
Anyway, that's where I think this should go. Does it make sense? Other thoughts?
I think all that polarity doesn't matter as long as we abstract it away with, "mark encrypted" and "mark decrypted".
But it is late now and I could be wrong - I'll look at the rest tomorrow-ish.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRE: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
From: Borislav Petkov <[email protected]> Sent: Tuesday, November 22, 2022 2:18 PM
Quoted 10 lines · Michael Kelley (LINUX)
Quoted 5 lines
I think the core problem here is the naming and meaning of
CC_VENDOR_HYPERV. The name was created originally when the
Hyper-V vTOM handling code was a lot of special cases. With the
changes in this patch series that make the vTOM functionality more mainstream, the name would be better asCC_VENDOR_AMD_VTOM.No, if
CC_VENDOR_HYPERVmeans different things depending on what kind of guests you're doing, then you should not use aCC_VENDORat all.
Agreed. My proposal is to drop CC_VENDOR_HYPERV entirely.
Replace it with CC_VENDOR_AMD_VTOM (or something like that) that is set *only* by Linux guests that are running on AMD SEV-SNP processors and using the vTOM scheme instead of the AMD C-bit scheme.
Quoted 7 lines
Quoted 6 lines
vTOM is part of the AMD SEV-SNP spec, and it's a different way of doing the encryption from the "C-bit" based approach. As much as possible, I'm trying to not make it be Hyper-V specific, though currently we have N=1 for hypervisors that offer the vTOM option, so it's a little hard to generalize.
Quoted 6 lines
Actually, it is very simple to generalize: vTOM and the paravisor and VMPL are all part of the effort to support unenlightened, unmodified guests with SNP.
So, if KVM wants to run Windows NT 4.0 guests as SNP guests, then it probably would need the same contraptions.
Yes, agreed. My point about generalization is that Hyper-V is the only actual implementation today. Edge cases, like whether the IO-APIC is accessed as encrypted or as decrypted don't have a pattern yet. But that's not a blocker. Such cases can be resolved or special-cased later when/if N > 1.
Quoted 6 lines
Quoted 4 lines
With the thinking oriented that way, a Linux guest on Hyper-V using TDX will run with
CC_VENDOR_INTEL. A Linux guest on Hyper-V that is fully enlightened to use the "C-bit" will run withCC_VENDOR_AMD.Right.
Good. We're in agreement. :-)
Quoted 4 lines
Quoted 4 lines
Dexuan Cui just posted a patch set for initial TDX support on Hyper-V, and I think that runs with
CC_VENDOR_INTEL(Dexuan -- correct me if I'm wrong about that -- I haven't reviewed your patches yet).
I confirmed with Dexuan that his new patch set for TDX guests on Hyper-V has the guest running with CC_VENDOR_INTEL, which is what we want.
Quoted 5 lines
Quoted 5 lines
Tianyu Lan
has a patch set out for Hyper-V guests using the "C-bit". That patch set still usesCC_VENDOR_HYPERV. Tianyu and I need to work through whether his patch set can run withCC_VENDOR_AMDlike everyone else using the "C-bit" approach.
I haven't followed up with Tianyu yet.
Quoted 3 lines
So I'm not sure the vendor is the right approach here. I guess we need to specify the *type* of guest being supported.
Yes, calling it the "vendor" turns out to not quite be right because in the AMD case, the technology/architecture/scheme/"type" (or
whatever you want to call it) is not 1:1 with the vendor. Intel has just
one (TDX) while AMD has two (C-bit and vTOM). "vendor" is just a label,
but we should get the label right to avoid future confusion. The key point is that we'll have three top-level types:
* TDX
* AMD with C-bit (and this has some sub-types)
* AMD with vTOM
The CC_ATTR_* values are then derived from the "type".
Quoted 10 lines
Quoted 7 lines
Yes, the polarity of the AMD vTOM bit matches the polarity of the TDX GPA.SHARED bit, and is the opposite polarity of the AMD "C-bit". I'll add a comment to that effect.
Anyway, that's where I think this should go. Does it make sense? Other thoughts?
I think all that polarity doesn't matter as long as we abstract it away with, "mark encrypted" and "mark decrypted".
Agreed.
Michael
RE: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
From: Michael Kelley (LINUX) <[email protected]> Sent: Tuesday, November 22, 2022 4:59 PM
Quoted 85 lines
From: Borislav Petkov <[email protected]> Sent: Tuesday, November 22, 2022 2:18 PM
Quoted 10 lines · Michael Kelley (LINUX)
Quoted 5 lines
I think the core problem here is the naming and meaning of
CC_VENDOR_HYPERV. The name was created originally when the
Hyper-V vTOM handling code was a lot of special cases. With the
changes in this patch series that make the vTOM functionality more mainstream, the name would be better asCC_VENDOR_AMD_VTOM.No, if
CC_VENDOR_HYPERVmeans different things depending on what kind of guests you're doing, then you should not use aCC_VENDORat all.Agreed. My proposal is to drop
CC_VENDOR_HYPERVentirely.
Replace it withCC_VENDOR_AMD_VTOM(or something like that) that is set *only* by Linux guests that are running on AMD SEV-SNP processors and using the vTOM scheme instead of the AMD C-bit scheme.Quoted 7 lines
Quoted 6 lines
vTOM is part of the AMD SEV-SNP spec, and it's a different way of doing the encryption from the "C-bit" based approach. As much as possible, I'm trying to not make it be Hyper-V specific, though currently we have N=1 for hypervisors that offer the vTOM option, so it's a little hard to generalize.
Quoted 6 lines
Actually, it is very simple to generalize: vTOM and the paravisor and VMPL are all part of the effort to support unenlightened, unmodified guests with SNP.
So, if KVM wants to run Windows NT 4.0 guests as SNP guests, then it probably would need the same contraptions.
Yes, agreed. My point about generalization is that Hyper-V is the only actual implementation today. Edge cases, like whether the IO-APIC is accessed as encrypted or as decrypted don't have a pattern yet. But that's not a blocker. Such cases can be resolved or special-cased later when/if N > 1.
Quoted 6 lines
Quoted 4 lines
With the thinking oriented that way, a Linux guest on Hyper-V using TDX will run with
CC_VENDOR_INTEL. A Linux guest on Hyper-V that is fully enlightened to use the "C-bit" will run withCC_VENDOR_AMD.Right.
Good. We're in agreement. :-)
Quoted 4 lines
Quoted 4 lines
Dexuan Cui just posted a patch set for initial TDX support on Hyper-V, and I think that runs with
CC_VENDOR_INTEL(Dexuan -- correct me if I'm wrong about that -- I haven't reviewed your patches yet).I confirmed with Dexuan that his new patch set for TDX guests on Hyper-V has the guest running with
CC_VENDOR_INTEL, which is what we want.Quoted 5 lines
Quoted 5 lines
Tianyu Lan
has a patch set out for Hyper-V guests using the "C-bit". That patch set still usesCC_VENDOR_HYPERV. Tianyu and I need to work through whether his patch set can run withCC_VENDOR_AMDlike everyone else using the "C-bit" approach.I haven't followed up with Tianyu yet.
Quoted 3 lines
So I'm not sure the vendor is the right approach here. I guess we need to specify the *type* of guest being supported.
Yes, calling it the "vendor" turns out to not quite be right because in the AMD case, the technology/architecture/scheme/"type" (or
whatever you want to call it) is not 1:1 with the vendor. Intel has just
one (TDX) while AMD has two (C-bit and vTOM). "vendor" is just a label,
but we should get the label right to avoid future confusion. The key point is that we'll have three top-level types:* TDX
* AMD with C-bit (and this has some sub-types)
* AMD with vTOMThe CC_ATTR_* values are then derived from the "type".
Quoted 5 lines
Quoted 5 lines
Yes, the polarity of the AMD vTOM bit matches the polarity of the TDX GPA.SHARED bit, and is the opposite polarity of the AMD "C-bit". I'll add a comment to that effect.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Anyway, that's where I think this should go. Does it make sense? Other thoughts?
I think all that polarity doesn't matter as long as we abstract it away with, "mark encrypted" and "mark decrypted".
Agreed.
Boris --
Any further comment on this patch? I think we're agreement. For this patch series I propose to change the symbol "CC_VENDOR_HYPERV" to "CC_VENDOR_AMD_VTOM" and the function name hyperv_cc_platform_has() to amd_vtom_cc_platform_has(). There's no functionality change vs. the current version of this patch. The naming changes just make clear that the scope is only for the AMD vTOM encryption scheme.
As discussed, the concept "vendor" isn't quite right either, but let's not add that change to this patch series. I can follow with a separate patch to change symbols with "vendor" or "VENDOR" to something else, which will probably require some discussion as to what that something else should be.
Michael
Borislav Petkov
maintainerRe: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
Quoted 4 lines · Michael Kelley (LINUX)
Any further comment on this patch? I think we're agreement. For this patch series I propose to change the symbol "
CC_VENDOR_HYPERV" to "CC_VENDOR_AMD_VTOM" and the function namehyperv_cc_platform_has()toamd_vtom_cc_platform_has().
That doesn't sound optimal to me.
So, let's clarify things first: those Isolation VMs - are they going to be the paravisors?
I don't see any other option because the unmodified guest must be some old windoze....
So, if they're going to be that, then I guess this should be called
CC_ATTR_PARAVISOR
to denote that it is a thin layer of virt gunk between an unmodified guest and a hypervisor.
And if TDX wants to do that too later, then they can use that flag too.
Yes, no?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRE: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
From: Borislav Petkov <[email protected]> Sent: Monday, November 28, 2022 8:34 AM
Quoted 11 lines · Michael Kelley (LINUX)
Quoted 4 lines
Any further comment on this patch? I think we're agreement. For this patch series I propose to change the symbol "
CC_VENDOR_HYPERV" to "CC_VENDOR_AMD_VTOM" and the function namehyperv_cc_platform_has()toamd_vtom_cc_platform_has().That doesn't sound optimal to me.
So, let's clarify things first: those Isolation VMs - are they going to be the paravisors?
No. The paravisor exists in another layer (VMPL 0 in the AMD vTOM architecture) that isn't visible to the Linux guest. The paravisor is a separate code base that isn't currently open source. All code in this patch series is code that runs in the Linux guest.
From an encryption standpoint, the Linux guest sees the following:
1) Guest memory is encrypted by default
2) The Linux guest must set the vTOM flag in a PTE to access a page as unencrypted.
3) The Linux guest must explicitly notify the hypervisor to change a page from private (encrypted) to shared (decrypted), and vice versa.
Quoted 3 lines
I don't see any other option because the unmodified guest must be some old windoze....
What Windows guests do isn't really relevant. Again, the code in this patch series all runs directly in the Linux guest, not the paravisor. And the Linux guest isn't unmodified. We've added changes to understand vTOM and the need to communicate with the hypervisor about page changes between private and shared. But there are other changes for a fully enlightened guest that we don't have to make when using AMD vTOM, because the paravisor transparently (to the guest -- Linux or Windows) handles those issues.
Quoted 7 lines
So, if they're going to be that, then I guess this should be called
CC_ATTR_PARAVISORto denote that it is a thin layer of virt gunk between an unmodified guest and a hypervisor.
No, the code is this patch series is not that at all. It's not code that is part of the paravisor. It's Linux guest code that understands it is running in an environment where AMD vTOM is the encryption scheme, which is different from the AMD C-bit encryption scheme.
Quoted 3 lines
And if TDX wants to do that too later, then they can use that flag too.
Again, no. What I have proposed as CC_VENDOR_AMD_VTOM is specific to AMD's virtual-Top-of-Memory architecture. The TDX architecture doesn't really have a way to use a paravisor.
To summarize, the code in this patch series is about a 3rd encryption scheme that is used by the guest. It is completely parallel to the AMD
C-bit encryption scheme and the Intel TDX encryption scheme. With
the AMD vTOM scheme, there is a paravisor that transparently emulates some things for the guest so there are fewer code changes needed in the guest, but this patch series is not about that paravisor code.
Michael
Borislav Petkov
maintainerRe: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
Quoted 2 lines · Michael Kelley (LINUX)
2) The Linux guest must set the vTOM flag in a PTE to access a page as unencrypted.
What exactly do you call the "vTOM flag in the PTE"?
I see this here:
"When bit 1 (vTOM) of SEV_FEATURES is set in the VMSA of an SNP-active VM, the VIRTUAL_TOM field is used to determine the C-bit for data accesses instead of the guest page table contents. All data accesses below VIRTUAL_TOM are accessed with an effective C-bit of 1 and all addresses at or above VIRTUAL_TOM are accessed with an effective C-bit of 0."
Now you say
"vTOM is the dividing line where the uppermost bit of the physical address space is set; e.g., with 47 bits of guest physical address space, vTOM is 0x40000000000 (bit 46 is set)."
So on your guests, is VIRTUAL_TOM == 0x400000000000?
Btw, that 0x4000.. does not have bit 46 set but bit 42. Bit 46 set is
0x400000000000
which means one more '0' at the end.
So before we discuss this further, let's agree on the basics first.
Quoted 8 lines
What Windows guests do isn't really relevant. Again, the code in this patch series all runs directly in the Linux guest, not the paravisor. And the Linux guest isn't unmodified. We've added changes to understand vTOM and the need to communicate with the hypervisor about page changes between private and shared. But there are other changes for a fully enlightened guest that we don't have to make when using AMD vTOM, because the paravisor transparently (to the guest -- Linux or Windows) handles those issues.
So this is some other type of guest you wanna run.
Where is the documentation of that thing?
I'd like to know what exactly it is going to use in the kernel.
Quoted 1 line
Again, no. What I have proposed as
CC_VENDOR_AMD_VTOMis
There's no vendor AMD_VTOM!
We did the vendor thing to denote Intel or AMD wrt to confidential computing.
Now you're coming up with something special. It can't be HYPERV because Hyper-V does other types of confidential solutions too, apparently.
Now before this goes any further I'd like for "something special" to be defined properly and not just be a one-off Hyper-V thing.
Quoted 5 lines
specific to AMD's virtual-Top-of-Memory architecture. The TDX architecture doesn't really have a way to use a paravisor.
To summarize, the code in this patch series is about a 3rd encryption scheme that is used by the guest.
Yes, can that third thing be used by other hypervisors or is this Hyper-V special?
Quoted 5 lines
It is completely parallel to the AMD C-bit encryption scheme and the Intel TDX encryption scheme. With the AMD vTOM scheme, there is a paravisor that transparently emulates some things for the guest so there are fewer code changes needed in the guest, but this patch series is not about that paravisor code.
Would other hypervisors want to support such a scheme?
Is this architecture documented somewhere? If so, where?
What would it mean for the kernel to support it.
And so on and so on.
Thanks.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRE: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
From: Borislav Petkov <[email protected]> Sent: Monday, November 28, 2022 9:25 AM
Quoted 22 lines · Michael Kelley (LINUX)
Quoted 2 lines
2) The Linux guest must set the vTOM flag in a PTE to access a page as unencrypted.
What exactly do you call the "vTOM flag in the PTE"?
I see this here:
"When bit 1 (vTOM) of
SEV_FEATURESis set in the VMSA of an SNP-active VM, theVIRTUAL_TOMfield is used to determine the C-bit for data accesses instead of the guest page table contents. All data accesses belowVIRTUAL_TOMare accessed with an effective C-bit of 1 and all addresses at or aboveVIRTUAL_TOMare accessed with an effective C-bit of 0."Now you say
"vTOM is the dividing line where the uppermost bit of the physical address space is set; e.g., with 47 bits of guest physical address space, vTOM is
0x40000000000(bit 46 is set)."
Quoted 7 lines
So on your guests, is
VIRTUAL_TOM==0x400000000000?Btw, that
0x4000.. does not have bit 46 set but bit 42. Bit 46 set is0x400000000000which means one more '0' at the end.
Yeah, looks like I dropped a '0' in my comment text. Will fix.
Quoted 12 lines
So before we discuss this further, let's agree on the basics first.
Quoted 8 lines
What Windows guests do isn't really relevant. Again, the code in this patch series all runs directly in the Linux guest, not the paravisor. And the Linux guest isn't unmodified. We've added changes to understand vTOM and the need to communicate with the hypervisor about page changes between private and shared. But there are other changes for a fully enlightened guest that we don't have to make when using AMD vTOM, because the paravisor transparently (to the guest -- Linux or Windows) handles those issues.
Quoted 5 lines
So this is some other type of guest you wanna run.
Where is the documentation of that thing?
I'd like to know what exactly it is going to use in the kernel.
Standard Linux is the guest. It's fully functional for running general Purpose workloads that want "confidential computing" where guest memory is encrypted and the data in the guest is not visible to the host hypervisor. It's a standard Linux kernel. I'm not sure what you mean by "some other type of guest".
Quoted 7 lines
Quoted 2 lines
Again, no. What I have proposed as
CC_VENDOR_AMD_VTOMisThere's no vendor
AMD_VTOM!We did the vendor thing to denote Intel or AMD wrt to confidential computing.
But vendor AMD effectively offers two different encryption schemes that could be seen by the guest VM. The hypervisor chooses which scheme a particular guest will see. Hyper-V has chosen to present the vTOM scheme to guest VMs, including normal Linux and Windows guests, that have been modestly updated to understand vTOM.
Quoted 3 lines
Now you're coming up with something special. It can't be HYPERV because Hyper-V does other types of confidential solutions too, apparently.
In the future, Hyper-V may also choose to present original AMD C-bit scheme in some guest VMs, depending on the use case. And it will present the Intel TDX scheme when running on that hardware.
Quoted 4 lines
Now before this goes any further I'd like for "something special" to be defined properly and not just be a one-off Hyper-V thing.
Quoted 8 lines
Quoted 5 lines
specific to AMD's virtual-Top-of-Memory architecture. The TDX architecture doesn't really have a way to use a paravisor.
To summarize, the code in this patch series is about a 3rd encryption scheme that is used by the guest.
Yes, can that third thing be used by other hypervisors or is this Hyper-V special?
This third thing is part of the AMD SEV-SNP architecture and could be used by any hypervisor.
Quoted 10 lines
Quoted 6 lines
It is completely parallel to the AMD C-bit encryption scheme and the Intel TDX encryption scheme. With the AMD vTOM scheme, there is a paravisor that transparently emulates some things for the guest so there are fewer code changes needed in the guest, but this patch series is not about that paravisor code.
Would other hypervisors want to support such a scheme?
Is this architecture documented somewhere? If so, where?
The AMD vTOM scheme is documented in the AMD SEV-SNP architecture specs.
Quoted 2 lines
What would it mean for the kernel to support it.
The Linux kernel running as a guest already supports it, at least when
running on Hyper-V. The code went into the 5.15 kernel [1][2] about
a year ago. But that code is more Hyper-V specific than is desirable. This patch set makes the AMD vTOM support more parallel to the Intel TDX and AMD C-bit support.
To my knowledge, KVM does not support the AMD vTOM scheme. Someone from AMD may have a better sense whether adding that support is likely in the future.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Michael
Borislav Petkov
maintainerRe: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
Quoted 5 lines · Michael Kelley (LINUX)
But vendor AMD effectively offers two different encryption schemes that could be seen by the guest VM. The hypervisor chooses which scheme a particular guest will see. Hyper-V has chosen to present the vTOM scheme to guest VMs, including normal Linux and Windows guests, that have been modestly updated to understand vTOM.
If this is a standard SNP guest then you can detect vTOM support using SEV_FEATURES. See this thread here:
https://lore.kernel.org/r/[email protected]
Which then means, you don't need any special gunk except extending this patch above to check SNP has vTOM support.
Quoted 3 lines
In the future, Hyper-V may also choose to present original AMD C-bit scheme in some guest VMs, depending on the use case. And it will present the Intel TDX scheme when running on that hardware.
And all those should JustWork(tm) because we already support such guests.
Quoted 3 lines
To my knowledge, KVM does not support the AMD vTOM scheme. Someone from AMD may have a better sense whether adding that support is likely in the future.
Yah, see above.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRE: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
From: Borislav Petkov <[email protected]> Sent: Monday, November 28, 2022 11:57 AM
Quoted 8 lines · Michael Kelley (LINUX)
Quoted 5 lines
But vendor AMD effectively offers two different encryption schemes that could be seen by the guest VM. The hypervisor chooses which scheme a particular guest will see. Hyper-V has chosen to present the vTOM scheme to guest VMs, including normal Linux and Windows guests, that have been modestly updated to understand vTOM.
Quoted 7 lines
If this is a standard SNP guest then you can detect vTOM support using
SEV_FEATURES. See this thread here:https://lore.kernel.org/all/[email protected]/
Which then means, you don't need any special gunk except extending this patch above to check SNP has vTOM support.
Yes, we could do that. But this patch set is still needed to get the preliminaries in place. The initial code for supporting AMD vTOM that went upstream a year ago is too Hyper-V specific. This patch set removes the Hyper-V specific stuff, and integrates vTOM support into the overall confidential computing framework in arch/x86/coco/core.c. The Hyper-V code was already somewhat there via CC_VENDOR_HYPERV, but that really should be named something else now. So that's why I'm suggesting CC_VENDOR_AMD_VTOM.
Quoted 6 lines
Quoted 4 lines
In the future, Hyper-V may also choose to present original AMD C-bit scheme in some guest VMs, depending on the use case. And it will present the Intel TDX scheme when running on that hardware.
And all those should JustWork(tm) because we already support such guests.
Agreed. That's where we want to be.
Of course, when you go from N=1 hypervisors (i.e., KVM) to N=2 (KVM and Hyper-V, you find some places where incorrect assumptions were made or some generalizations are needed. Dexuan Cui's patch set for TDX support is fixing those places that he has encountered. But with those fixes, the TDX support will JustWork(tm) for Linux guests on Hyper-V.
I haven't gone deeply into the situation with AMD C-bit support on Hyper-V. Tianyu Lan's set of patches for that support is a bit bigger, and I'm planning to look closely to understand whether it's also just fixing incorrect assumptions and such, or whether they really are some differences with Hyper-V. If there are differences, I want to understand why.
Michael
Borislav Petkov
maintainerRe: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
Quoted 1 line · Michael Kelley (LINUX)
So that's why I'm suggesting
CC_VENDOR_AMD_VTOM.
There's no CC_VENDOR_AMD_VTOM. How many times do I need to explain this?! CC_VENDOR is well the CC vendor - not some special case.
IOW, the goal here is for generic SNP functionality to be the same for *all* SNP guests. We won't do the AMD's version of vTOM enablement, Hyper-V's version of vTOM enablement and so on. It must be a single vTOM feature which works on *all* hypervisors as vTOM is a generic SNP feature - not Hyper-V feature.
Quoted 6 lines
Of course, when you go from N=1 hypervisors (i.e., KVM) to N=2 (KVM and Hyper-V, you find some places where incorrect assumptions were made or some generalizations are needed. Dexuan Cui's patch set for TDX support is fixing those places that he has encountered. But with those fixes, the TDX support will JustWork(tm) for Linux guests on Hyper-V.
That sounds like the right direction to take.
Quoted 6 lines
I haven't gone deeply into the situation with AMD C-bit support on Hyper-V. Tianyu Lan's set of patches for that support is a bit bigger, and I'm planning to look closely to understand whether it's also just fixing incorrect assumptions and such, or whether they really are some differences with Hyper-V. If there are differences, I want to understand why.
You and me too. So I guess we should look at Tianyu's set first.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRE: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
From: Borislav Petkov <[email protected]> Sent: Tuesday, November 29, 2022 12:41 AM
Quoted 6 lines · Michael Kelley (LINUX)
Quoted 1 line
So that's why I'm suggesting
CC_VENDOR_AMD_VTOM.There's no
CC_VENDOR_AMD_VTOM. How many times do I need to explain this?!CC_VENDORis well the CC vendor - not some special case.
Let's back up half a step. We have CC_VENDOR_INTEL and CC_VENDOR_AMD because that's a convenient way to identify two fairly different schemes for doing guest encryption. The schemes have some things in common, but the details are pretty different. Tagging the schemes based on the vendor makes sense because the schemes were independently developed by each processor vendor.
But it turns out that AMD really has two fairly different schemes:
the C-bit scheme and the vTOM scheme. The details of these two AMD schemes are pretty different. vTOM is *not* just a minor option on the C-bit scheme. It's an either/or -- a guest VM is either doing the C-bit scheme or the vTOM scheme, not some combination. Linux code in coco/core.c could choose to treat C-bit and vTOM as two sub-schemes under CC_VENDOR_AMD, but that makes the code a bit messy because we end up with "if" statements to figure out whether to do things the C-bit way or the vTOM way. The difference in the C-bit way and the vTOM way is not Hyper-V specific. Any hypervisor running on an AMD processor can make the same choice to offer C-bit VMs or vTOM VMs.
Or we could model the two AMD schemes as two different vendors, which is what I'm suggesting. Doing so recognizes that the two schemes are fairly disjoint, and it makes the code cleaner.
Tom Lendacky -- any thoughts on this question from AMD's standpoint?
Quoted 6 lines
IOW, the goal here is for generic SNP functionality to be the same for *all* SNP guests. We won't do the AMD's version of vTOM enablement, Hyper-V's version of vTOM enablement and so on. It must be a single vTOM feature which works on *all* hypervisors as vTOM is a generic SNP feature - not Hyper-V feature.
Yes, there's only one version of vTOM enablement -- the AMD version.
But the broader AMD SNP functionality is bifurcated: there's the C-bit scheme and the vTOM scheme. The question is how Linux code should model those two different schemes. I'm suggesting that things are cleaner conceptually and in the code if we model the two different AMD schemes as two different vendors.
Michael
Quoted 19 lines
Quoted 7 lines
Of course, when you go from N=1 hypervisors (i.e., KVM) to N=2 (KVM and Hyper-V, you find some places where incorrect assumptions were made or some generalizations are needed. Dexuan Cui's patch set for TDX support is fixing those places that he has encountered. But with those fixes, the TDX support will JustWork(tm) for Linux guests on Hyper-V.
That sounds like the right direction to take.
Quoted 6 lines
I haven't gone deeply into the situation with AMD C-bit support on Hyper-V. Tianyu Lan's set of patches for that support is a bit bigger, and I'm planning to look closely to understand whether it's also just fixing incorrect assumptions and such, or whether they really are some differences with Hyper-V. If there are differences, I want to understand why.
You and me too. So I guess we should look at Tianyu's set first.
Borislav Petkov
maintainerRe: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
Quoted 2 lines · Michael Kelley (LINUX)
But it turns out that AMD really has two fairly different schemes:
the C-bit scheme and the vTOM scheme.
Except it doesn't:
"In the VMSA of an SNP-active guest, the VIRTUAL_TOM field designates a 2MB aligned guest physical address called the virtual top of memory. When bit 1 (vTOM) of SEV_FEATURES is set in the VMSA of an SNP-active VM, the VIRTUAL_TOM..."
So SEV_FEATURES[1] is vTOM and it is part of SNP.
Why do you keep harping on this being something else is beyond me...
I already pointed you to the patch which adds this along with the other SEV_FEATURES.
Quoted 7 lines
The details of these two AMD schemes are pretty different. vTOM is *not* just a minor option on the C-bit scheme. It's an either/or -- a guest VM is either doing the C-bit scheme or the vTOM scheme, not some combination. Linux code in
coco/core.ccould choose to treat C-bit and vTOM as two sub-schemes underCC_VENDOR_AMD, but that makes the code a bit messy because we end up with "if" statements to figure out whether to do things the C-bit way or the vTOM way.
Are you saying that that:
if (cc_vendor == CC_VENDOR_AMD && sev_features & MSR_AMD64_SNP_VTOM_ENABLED)
is messy? Why?
We will have to support vTOM sooner or later.
Quoted 3 lines
Or we could model the two AMD schemes as two different vendors, which is what I'm suggesting. Doing so recognizes that the two schemes are fairly disjoint, and it makes the code cleaner.
How is that any different from the above check?
You *need* some sort of a check to differentiate between the two anyway.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRE: [Patch v3 07/14] x86/hyperv: Change vTOM handling to use standard coco mechanisms
From: Borislav Petkov <[email protected]> Sent: Tuesday, November 29, 2022 9:47 AM
Quoted 36 lines · Michael Kelley (LINUX)
Quoted 2 lines
But it turns out that AMD really has two fairly different schemes:
the C-bit scheme and the vTOM scheme.Except it doesn't:
"In the VMSA of an SNP-active guest, the
VIRTUAL_TOMfield designates a 2MB aligned guest physical address called the virtual top of memory. When bit 1 (vTOM) ofSEV_FEATURESis set in the VMSA of an SNP-active VM, theVIRTUAL_TOM..."So SEV_FEATURES[1] is vTOM and it is part of SNP.
Why do you keep harping on this being something else is beyond me...
I already pointed you to the patch which adds this along with the other
SEV_FEATURES.Quoted 7 lines
The details of these two AMD schemes are pretty different. vTOM is *not* just a minor option on the C-bit scheme. It's an either/or -- a guest VM is either doing the C-bit scheme or the vTOM scheme, not some combination. Linux code in
coco/core.ccould choose to treat C-bit and vTOM as two sub-schemes underCC_VENDOR_AMD, but that makes the code a bit messy because we end up with "if" statements to figure out whether to do things the C-bit way or the vTOM way.Are you saying that that:
if (cc_vendor == CC_VENDOR_AMD && sev_features & MSR_AMD64_SNP_VTOM_ENABLED)is messy? Why?
We will have to support vTOM sooner or later.
Quoted 8 lines
Quoted 3 lines
Or we could model the two AMD schemes as two different vendors, which is what I'm suggesting. Doing so recognizes that the two schemes are fairly disjoint, and it makes the code cleaner.
How is that any different from the above check?
You *need* some sort of a check to differentiate between the two anyway.
Alright. Enough conceptual debate. I'll do a v4 of the patch series with the AMD C-bit and vTOM schemes folder under CC_VENDOR_AMD and
we can see if there's any further feedback. I should have that v4 out later today or tomorrow.
Michael
PATCH v3 8/14 swiotlb: Remove bounce buffer remapping for Hyper-V
In reply to Michael KelleyWith changes to how Hyper-V guest VMs flip memory between private (encrypted) and shared (decrypted), creating a second kernel virtual mapping for shared memory is no longer necessary. Everything needed for the transition to shared is handled by set_memory_decrypted().
As such, remove swiotlb_unencrypted_base and the associated code.
--- arch/x86/kernel/cpu/mshyperv.c | 7 +------ include/linux/swiotlb.h | 2 -- kernel/dma/swiotlb.c | 45 +----------------------------------------- 3 files changed, 2 insertions(+), 52 deletions(-)
arch/x86/kernel/cpu/mshyperv.c +1 -6
@@ -18,7 +18,6 @@#include <linux/kexec.h> #include <linux/i8253.h> #include <linux/random.h> -#include <linux/swiotlb.h> #include <asm/processor.h> #include <asm/hypervisor.h> #include <asm/hyperv-tlfs.h>
@@ -332,12 +331,8 @@ static void __init ms_hyperv_init_platform(void)pr_info ("Hyper-V: Isolation Config: Group A 0x%x, Group B 0x%x\n" , ms_hyperv.isolation_config_a, ms_hyperv.isolation_config_b); -if (hv_get_isolation_type () ==HV_ISOLATION_TYPE_SNP ) { +if (hv_get_isolation_type () ==HV_ISOLATION_TYPE_SNP )static_branch_enable (&isolation_type_snp); -#ifdef CONFIG_SWIOTLB - swiotlb_unencrypted_base = ms_hyperv.shared_gpa_boundary; -#endif - } }if (hv_max_functions_eax >=HYPERV_CPUID_NESTED_FEATURES ) {
include/linux/swiotlb.h +0 -2
@@ -185,6 +185,4 @@ static inline bool is_swiotlb_for_alloc(struct device *dev) }#endif /* CONFIG_DMA_RESTRICTED_POOL */ -extern phys_addr_t swiotlb_unencrypted_base; -#endif /* __LINUX_SWIOTLB_H */
kernel/dma/swiotlb.c +1 -44
74 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH v3 9/14 Drivers: hv: vmbus: Remove second mapping of VMBus monitor pages
In reply to Michael KelleyWith changes to how Hyper-V guest VMs flip memory between private (encrypted) and shared (decrypted), creating a second kernel virtual mapping for shared memory is no longer necessary. Everything needed for the transition to shared is handled by set_memory_decrypted().
As such, remove the code to create and manage the second mapping for VMBus monitor pages. Because set_memory_decrypted() and set_memory_encrypted() are no-ops in normal VMs, it's not even necessary to test for being in a Confidential VM (a.k.a., "Isolation VM").
--- drivers/hv/connection.c | 113 ++++++++++++---------------------------------- drivers/hv/hyperv_vmbus.h | 2 - 2 files changed, 28 insertions(+), 87 deletions(-)
drivers/hv/connection.c +28 -85
@@ -104,8 +104,14 @@ int vmbus_negotiate_version(struct vmbus_channel_msginfo *msginfo, u32 version) vmbus_connection.msg_conn_id =VMBUS_MESSAGE_CONNECTION_ID ; } - msg->monitor_page1 = vmbus_connection.monitor_pages_pa[0 ]; - msg->monitor_page2 = vmbus_connection.monitor_pages_pa[1 ]; +/* +* shared_gpa_boundary is zero in non-SNP VMs, so it's safe to always +* do the add +*/ + msg->monitor_page1 =virt_to_phys (vmbus_connection.monitor_pages[0 ]) + + ms_hyperv.shared_gpa_boundary; + msg->monitor_page2 =virt_to_phys (vmbus_connection.monitor_pages[1 ]) + + ms_hyperv.shared_gpa_boundary; msg->target_vcpu =hv_cpu_number_to_vp_number (VMBUS_CONNECT_CPU );
@@ -219,72 +225,27 @@ int vmbus_connect(void)* Setup the monitor notification facility. The 1st page for * parent->child and the 2nd page for child->parent */ - vmbus_connection.monitor_pages[0 ] = (void *)hv_alloc_hyperv_zeroed_page (); - vmbus_connection.monitor_pages[1 ] = (void *)hv_alloc_hyperv_zeroed_page (); + vmbus_connection.monitor_pages[0 ] = (void *)hv_alloc_hyperv_page (); + vmbus_connection.monitor_pages[1 ] = (void *)hv_alloc_hyperv_page ();if ((vmbus_connection.monitor_pages[0 ] ==NULL ) || (vmbus_connection.monitor_pages[1 ] ==NULL )) { ret = -ENOMEM ; goto cleanup; } - vmbus_connection.monitor_pages_original[0 ] - = vmbus_connection.monitor_pages[0 ]; - vmbus_connection.monitor_pages_original[1 ] - = vmbus_connection.monitor_pages[1 ]; - vmbus_connection.monitor_pages_pa[0 ] - =virt_to_phys (vmbus_connection.monitor_pages[0 ]); - vmbus_connection.monitor_pages_pa[1 ] - =virt_to_phys (vmbus_connection.monitor_pages[1 ]); - -if (hv_is_isolation_supported ()) { - ret =set_memory_decrypted ((unsigned long ) - vmbus_connection.monitor_pages[0 ], -1 ); - ret |=set_memory_decrypted ((unsigned long ) - vmbus_connection.monitor_pages[1 ], -1 ); -if (ret) - goto cleanup; - -/* -* Isolation VM with AMD SNP needs to access monitor page via -* address space above shared gpa boundary. -*/ -if (hv_isolation_type_snp ()) { - vmbus_connection.monitor_pages_pa[0 ] += - ms_hyperv.shared_gpa_boundary; - vmbus_connection.monitor_pages_pa[1 ] += - ms_hyperv.shared_gpa_boundary; - - vmbus_connection.monitor_pages[0 ] - =memremap (vmbus_connection.monitor_pages_pa[0 ], -HV_HYP_PAGE_SIZE , -MEMREMAP_WB ); -if (!vmbus_connection.monitor_pages[0 ]) { - ret = -ENOMEM ; - goto cleanup; - } - - vmbus_connection.monitor_pages[1 ] - =memremap (vmbus_connection.monitor_pages_pa[1 ], -HV_HYP_PAGE_SIZE , -MEMREMAP_WB ); -if (!vmbus_connection.monitor_pages[1 ]) { - ret = -ENOMEM ; - goto cleanup; - } - } - -/* -* Set memory host visibility hvcall smears memory -* and so zero monitor pages here. -*/ -memset (vmbus_connection.monitor_pages[0 ],0x00 , -HV_HYP_PAGE_SIZE ); -memset (vmbus_connection.monitor_pages[1 ],0x00 , -HV_HYP_PAGE_SIZE ); + ret =set_memory_decrypted ((unsigned long ) + vmbus_connection.monitor_pages[0 ],1 ); + ret |=set_memory_decrypted ((unsigned long ) + vmbus_connection.monitor_pages[1 ],1 ); +if (ret) + goto cleanup; - } +/* +* Set_memory_decrypted() will change the memory contents if +* decryption occurs, so zero monitor pages here. +*/ +memset (vmbus_connection.monitor_pages[0 ],0x00 ,HV_HYP_PAGE_SIZE ); +memset (vmbus_connection.monitor_pages[1 ],0x00 ,HV_HYP_PAGE_SIZE ); msginfo = kzalloc(sizeof(*msginfo) + sizeof(struct vmbus_channel_initiate_contact),
@@ -376,31 +337,13 @@ void vmbus_disconnect(void) vmbus_connection.int_page =NULL ; } -if (hv_is_isolation_supported ()) { -/* -* memunmap() checks input address is ioremap address or not -* inside. It doesn't unmap any thing in the non-SNP CVM and -* so not check CVM type here. -*/ -memunmap (vmbus_connection.monitor_pages[0 ]); -memunmap (vmbus_connection.monitor_pages[1 ]); - -set_memory_encrypted ((unsigned long ) - vmbus_connection.monitor_pages_original[0 ], -1 ); -set_memory_encrypted ((unsigned long ) - vmbus_connection.monitor_pages_original[1 ], -1 ); - } +set_memory_encrypted ((unsigned long )vmbus_connection.monitor_pages[0 ],1 ); +set_memory_encrypted ((unsigned long )vmbus_connection.monitor_pages[1 ],1 ); -hv_free_hyperv_page ((unsigned long ) - vmbus_connection.monitor_pages_original[0 ]); -hv_free_hyperv_page ((unsigned long ) - vmbus_connection.monitor_pages_original[1 ]); - vmbus_connection.monitor_pages_original[0 ] = - vmbus_connection.monitor_pages[0 ] =NULL ; - vmbus_connection.monitor_pages_original[1 ] = - vmbus_connection.monitor_pages[1 ] =NULL ; +hv_free_hyperv_page ((unsigned long )vmbus_connection.monitor_pages[0 ]); +hv_free_hyperv_page ((unsigned long )vmbus_connection.monitor_pages[1 ]); + vmbus_connection.monitor_pages[0 ] =NULL ; + vmbus_connection.monitor_pages[1 ] =NULL ; }/*
drivers/hv/hyperv_vmbus.h +0 -2
@@ -241,8 +241,6 @@ struct vmbus_connection {* is child->parent notification */ struct hv_monitor_page *monitor_pages[2 ]; -void *monitor_pages_original[2 ]; -phys_addr_t monitor_pages_pa[2 ];struct list_head chn_msg_list;spinlock_t channelmsg_lock;
Signature · 1 line
1.8.3.1
PATCH v3 10/14 Drivers: hv: vmbus: Remove second way of mapping ring buffers
In reply to Michael KelleyWith changes to how Hyper-V guest VMs flip memory between private (encrypted) and shared (decrypted), it's no longer necessary to have separate code paths for mapping VMBus ring buffers for for normal VMs and for Confidential VMs.
As such, remove the code path that uses vmap_pfn(), and set the protection flags argument to vmap() to account for the difference between normal and Confidential VMs.
--- drivers/hv/ring_buffer.c | 62 ++++++++++++++++-------------------------------- 1 file changed, 20 insertions(+), 42 deletions(-)
drivers/hv/ring_buffer.c +20 -42
@@ -186,8 +186,6 @@ int hv_ringbuffer_init(struct hv_ring_buffer_info *ring_info, struct page *pages, u32 page_cnt, u32 max_pkt_size) { struct page **pages_wraparound; -unsigned long *pfns_wraparound; -u64 pfn;int i;BUILD_BUG_ON ((sizeof (struct hv_ring_buffer ) !=PAGE_SIZE ));
@@ -196,50 +194,30 @@ int hv_ringbuffer_init(struct hv_ring_buffer_info *ring_info,* First page holds struct hv_ring_buffer, do wraparound mapping for * the rest. */ -if (hv_isolation_type_snp ()) { - pfn =page_to_pfn (pages) + -PFN_DOWN (ms_hyperv.shared_gpa_boundary); + pages_wraparound =kcalloc (page_cnt *2 -1 , +sizeof (struct page *), +GFP_KERNEL ); +if (!pages_wraparound) +return -ENOMEM ; - pfns_wraparound =kcalloc (page_cnt *2 -1 , -sizeof (unsigned long ),GFP_KERNEL ); -if (!pfns_wraparound) -return -ENOMEM ; - - pfns_wraparound[0 ] = pfn; -for (i =0 ; i <2 * (page_cnt -1 ); i++) - pfns_wraparound[i +1 ] = pfn + i % (page_cnt -1 ) +1 ; - - ring_info->ring_buffer = (struct hv_ring_buffer *) -vmap_pfn (pfns_wraparound, page_cnt *2 -1 , -pgprot_decrypted (PAGE_KERNEL_NOENC )); -kfree (pfns_wraparound); - -if (!ring_info->ring_buffer) -return -ENOMEM ; - -/* Zero ring buffer after setting memory host visibility. */ -memset (ring_info->ring_buffer,0x00 ,PAGE_SIZE * page_cnt); - }else { - pages_wraparound =kcalloc (page_cnt *2 -1 , -sizeof (struct page *), -GFP_KERNEL ); -if (!pages_wraparound) -return -ENOMEM ; - - pages_wraparound[0 ] = pages; -for (i =0 ; i <2 * (page_cnt -1 ); i++) - pages_wraparound[i +1 ] = - &pages[i % (page_cnt -1 ) +1 ]; + pages_wraparound[0 ] = pages; +for (i =0 ; i <2 * (page_cnt -1 ); i++) + pages_wraparound[i +1 ] = + &pages[i % (page_cnt -1 ) +1 ]; - ring_info->ring_buffer = (struct hv_ring_buffer *) -vmap (pages_wraparound, page_cnt *2 -1 ,VM_MAP , -PAGE_KERNEL ); + ring_info->ring_buffer = (struct hv_ring_buffer *) +vmap (pages_wraparound, page_cnt *2 -1 ,VM_MAP , +pgprot_decrypted (PAGE_KERNEL_NOENC )); -kfree (pages_wraparound); -if (!ring_info->ring_buffer) -return -ENOMEM ; - } +kfree (pages_wraparound); +if (!ring_info->ring_buffer) +return -ENOMEM ; +/* +* Ensure the header page is zero'ed since +* encryption status may have changed. +*/ +memset (ring_info->ring_buffer,0 ,HV_HYP_PAGE_SIZE ); ring_info->ring_buffer->read_index = ring_info->ring_buffer->write_index =0 ;
Signature · 1 line
1.8.3.1
PATCH v3 11/14 hv_netvsc: Remove second mapping of send and recv buffers
In reply to Michael KelleyWith changes to how Hyper-V guest VMs flip memory between private (encrypted) and shared (decrypted), creating a second kernel virtual mapping for shared memory is no longer necessary. Everything needed for the transition to shared is handled by set_memory_decrypted().
As such, remove the code to create and manage the second mapping for the pre-allocated send and recv buffers. This mapping is the last user of hv_map_memory()/hv_unmap_memory(), so delete these functions as well. Finally, hv_map_memory() is the last user of vmap_pfn() in Hyper-V guest code, so remove the Kconfig selection of VMAP_PFN.
--- arch/x86/hyperv/ivm.c | 28 ------------------------ drivers/hv/Kconfig | 1 - drivers/hv/hv_common.c | 11 ---------- drivers/net/hyperv/hyperv_net.h | 2 -- drivers/net/hyperv/netvsc.c | 48 ++--------------------------------------- include/asm-generic/mshyperv.h | 2 -- 6 files changed, 2 insertions(+), 90 deletions(-)
arch/x86/hyperv/ivm.c +0 -28
@@ -349,34 +349,6 @@ void __init hv_vtom_init(void)#endif /* CONFIG_AMD_MEM_ENCRYPT */ -/* -* hv_map_memory - map memory to extra space in the AMD SEV-SNP Isolation VM. -*/ -void *hv_map_memory (void *addr,unsigned long size) -{ -unsigned long *pfns =kcalloc (size /PAGE_SIZE , -sizeof (unsigned long ),GFP_KERNEL ); -void *vaddr; -int i; - -if (!pfns) -return NULL ; - -for (i =0 ; i < size /PAGE_SIZE ; i++) - pfns[i] =vmalloc_to_pfn (addr + i *PAGE_SIZE ) + - (ms_hyperv.shared_gpa_boundary >>PAGE_SHIFT ); - - vaddr =vmap_pfn (pfns, size /PAGE_SIZE ,pgprot_decrypted (PAGE_KERNEL_NOENC )); -kfree (pfns); - -return vaddr; -} - -void hv_unmap_memory (void *addr) -{ -vunmap (addr); -} -enum hv_isolation_type hv_get_isolation_type (void ) {if (!(ms_hyperv.priv_high &HV_ISOLATION ))
drivers/hv/Kconfig +0 -1
@@ -8,7 +8,6 @@ config HYPERV || (ARM64 && !CPU_BIG_ENDIAN)) select PARAVIRT select X86_HV_CALLBACK_VECTOR if X86 - select VMAP_PFN help Select this option to run Linux as a Hyper-V client operating system.
drivers/hv/hv_common.c +0 -11
@@ -308,14 +308,3 @@ u64 __weak hv_ghcb_hypercall(u64 control, void *input, void *output, u32 input_sreturn HV_STATUS_INVALID_PARAMETER ; }EXPORT_SYMBOL_GPL (hv_ghcb_hypercall); - -void __weak *hv_map_memory (void *addr,unsigned long size) -{ -return NULL ; -} -EXPORT_SYMBOL_GPL (hv_map_memory); - -void __weak hv_unmap_memory (void *addr) -{ -} -EXPORT_SYMBOL_GPL (hv_unmap_memory);
drivers/net/hyperv/hyperv_net.h +0 -2
@@ -1139,7 +1139,6 @@ struct netvsc_device {/* Receive buffer allocated by us but manages by NetVSP */ void *recv_buf; -void *recv_original_buf;u32 recv_buf_size;/* allocated bytes */ struct vmbus_gpadl recv_buf_gpadl_handle;u32 recv_section_cnt;
@@ -1148,7 +1147,6 @@ struct netvsc_device {/* Send buffer allocated by us */ void *send_buf; -void *send_original_buf;u32 send_buf_size;struct vmbus_gpadl send_buf_gpadl_handle;u32 send_section_cnt;
drivers/net/hyperv/netvsc.c +2 -46
84 lines not on this page · raw message
include/asm-generic/mshyperv.h +0 -2
@@ -267,8 +267,6 @@ static inline int cpumask_to_vpset_noself(struct hv_vpset *vpset,void hyperv_cleanup (void );bool hv_query_ext_cap (u64 cap_query);void hv_setup_dma_ops (struct device *dev,bool coherent); -void *hv_map_memory (void *addr,unsigned long size); -void hv_unmap_memory (void *addr);#else /* CONFIG_HYPERV */ static inline bool hv_is_hyperv_initialized (void ) {return false; }static inline bool hv_is_hibernation_supported (void ) {return false; }
Signature · 1 line
1.8.3.1
PATCH v3 12/14 Drivers: hv: Don't remap addresses that are above shared_gpa_boundary
In reply to Michael KelleyWith the vTOM bit now treated as a protection flag and not part of the physical address, avoid remapping physical addresses with vTOM set since technically such addresses aren't valid. Use ioremap_cache() instead of memremap() to ensure that the mapping provides decrypted access, which will correctly set the vTOM bit as a protection flag.
While this change is not required for correctness with the current implementation of memremap(), for general code hygiene it's better to not depend on the mapping functions doing something reasonable with a physical address that is out-of-range.
While here, fix typos in two error messages.
--- arch/x86/hyperv/hv_init.c | 7 +++++-- drivers/hv/hv.c | 23 +++++++++++++---------- 2 files changed, 18 insertions(+), 12 deletions(-)
arch/x86/hyperv/hv_init.c +5 -2
@@ -63,7 +63,10 @@ static int hyperv_init_ghcb(void)* memory boundary and map it here. */ rdmsrl (MSR_AMD64_SEV_ES_GHCB , ghcb_gpa); - ghcb_va =memremap (ghcb_gpa,HV_HYP_PAGE_SIZE ,MEMREMAP_WB ); + +/* Mask out vTOM bit. ioremap_cache() maps decrypted */ + ghcb_gpa &= ~ms_hyperv.shared_gpa_boundary; + ghcb_va = (void *)ioremap_cache (ghcb_gpa,HV_HYP_PAGE_SIZE );if (!ghcb_va)return -ENOMEM ;
@@ -219,7 +222,7 @@ static int hv_cpu_die(unsigned int cpu)if (hv_ghcb_pg) { ghcb_va = (void **)this_cpu_ptr (hv_ghcb_pg);if (*ghcb_va) -memunmap (*ghcb_va); +iounmap (*ghcb_va); *ghcb_va =NULL ; }
drivers/hv/hv.c +13 -10
@@ -217,11 +217,13 @@ void hv_synic_enable_regs(unsigned int cpu) simp.simp_enabled =1 ;if (hv_isolation_type_snp ()) { +/* Mask out vTOM bit. ioremap_cache() maps decrypted */ +u64 base = (simp.base_simp_gpa <<HV_HYP_PAGE_SHIFT ) & + ~ms_hyperv.shared_gpa_boundary; hv_cpu->synic_message_page - =memremap (simp.base_simp_gpa <<HV_HYP_PAGE_SHIFT , -HV_HYP_PAGE_SIZE ,MEMREMAP_WB ); + = (void *)ioremap_cache (base,HV_HYP_PAGE_SIZE );if (!hv_cpu->synic_message_page) -pr_err ("Fail to map syinc message page.\n" ); +pr_err ("Fail to map synic message page.\n" ); }else { simp.base_simp_gpa =virt_to_phys (hv_cpu->synic_message_page) >>HV_HYP_PAGE_SHIFT ;
@@ -234,12 +236,13 @@ void hv_synic_enable_regs(unsigned int cpu) siefp.siefp_enabled =1 ;if (hv_isolation_type_snp ()) { - hv_cpu->synic_event_page = -memremap (siefp.base_siefp_gpa <<HV_HYP_PAGE_SHIFT , -HV_HYP_PAGE_SIZE ,MEMREMAP_WB ); - +/* Mask out vTOM bit. ioremap_cache() maps decrypted */ +u64 base = (siefp.base_siefp_gpa <<HV_HYP_PAGE_SHIFT ) & + ~ms_hyperv.shared_gpa_boundary; + hv_cpu->synic_event_page + = (void *)ioremap_cache (base,HV_HYP_PAGE_SIZE );if (!hv_cpu->synic_event_page) -pr_err ("Fail to map syinc event page.\n" ); +pr_err ("Fail to map synic event page.\n" ); }else { siefp.base_siefp_gpa =virt_to_phys (hv_cpu->synic_event_page) >>HV_HYP_PAGE_SHIFT ;
@@ -316,7 +319,7 @@ void hv_synic_disable_regs(unsigned int cpu)*/ simp.simp_enabled =0 ;if (hv_isolation_type_snp ()) -memunmap (hv_cpu->synic_message_page); +iounmap (hv_cpu->synic_message_page);else simp.base_simp_gpa =0 ;
@@ -326,7 +329,7 @@ void hv_synic_disable_regs(unsigned int cpu) siefp.siefp_enabled =0 ;if (hv_isolation_type_snp ()) -memunmap (hv_cpu->synic_event_page); +iounmap (hv_cpu->synic_event_page);else siefp.base_siefp_gpa =0 ;
Signature · 1 line
1.8.3.1
PATCH v3 13/14 PCI: hv: Add hypercalls to read/write MMIO space
In reply to Michael KelleyTo support PCI pass-thru devices in Confidential VMs, Hyper-V has added hypercalls to read and write MMIO space. Add the appropriate definitions to hyperv-tlfs.h and implement functions to make the hypercalls. These functions are used in a subsequent patch.
--- arch/x86/include/asm/hyperv-tlfs.h | 3 ++ drivers/pci/controller/pci-hyperv.c | 64 +++++++++++++++++++++++++++++++++++++ include/asm-generic/hyperv-tlfs.h | 22 +++++++++++++ 3 files changed, 89 insertions(+)
arch/x86/include/asm/hyperv-tlfs.h +3 -0
@@ -117,6 +117,9 @@/* Recommend using enlightened VMCS */ #define HV_X64_ENLIGHTENED_VMCS_RECOMMENDED BIT(14) +/* Use hypercalls for MMIO config space access */ +#define HV_X64_USE_MMIO_HYPERCALLS BIT(21) +/* * CPU management features identification. * These are HYPERV_CPUID_CPU_MANAGEMENT_FEATURES.EAX bits.
drivers/pci/controller/pci-hyperv.c +64 -0
70 lines not on this page · raw message
include/asm-generic/hyperv-tlfs.h +22 -0
@@ -168,6 +168,8 @@ struct ms_hyperv_tsc_page {#define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_SPACE 0x00af#define HVCALL_FLUSH_GUEST_PHYSICAL_ADDRESS_LIST 0x00b0#define HVCALL_MODIFY_SPARSE_GPA_PAGE_HOST_VISIBILITY 0x00db +#define HVCALL_MMIO_READ 0x0106 +#define HVCALL_MMIO_WRITE 0x0107/* Extended hypercalls */ #define HV_EXT_CALL_QUERY_CAPABILITIES 0x8001
@@ -790,4 +792,24 @@ struct hv_memory_hint { union hv_gpa_page_range ranges[]; } __packed; +/* Data structures for HVCALL_MMIO_READ and HVCALL_MMIO_WRITE */ +#define HV_HYPERCALL_MMIO_MAX_DATA_LENGTH 64 + +struct hv_mmio_read_input { +u64 gpa; +u32 size; +u32 reserved; +} __packed; + +struct hv_mmio_read_output { +u8 data[HV_HYPERCALL_MMIO_MAX_DATA_LENGTH ]; +} __packed; + +struct hv_mmio_write_input { +u64 gpa; +u32 size; +u32 reserved; +u8 data[HV_HYPERCALL_MMIO_MAX_DATA_LENGTH ]; +} __packed; +#endif
Signature · 1 line
1.8.3.1
Wei Liu
maintainerRe: [Patch v3 13/14] PCI: hv: Add hypercalls to read/write MMIO space
On Wed, Nov 16, 2022 at 10:41:36AM -0800, Michael Kelley wrote:
[...]
drivers/pci/controller/pci-hyperv.c static int wslot_to_devfn(u32 wslot) · line 1067
Quoted 12 lines
drivers/pci/controller/pci-hyperv.c +11
-0+static void hv_pci_read_mmio (struct device *dev,phys_addr_t gpa,int size,u32 *val) +{ +struct hv_mmio_read_input *in; +struct hv_mmio_read_output *out; +u64 ret; + +/* +* Must be called with interrupts disabled so it is safe +* to use the per-cpu input argument page. Use it for +* both input and output. +*/
Perhaps adding something along this line?
WARN_ON(!irqs_disabled());
I can fold this in if you agree.
Quoted 22 lines
drivers/pci/controller/pci-hyperv.c +22
-0+ in = *this_cpu_ptr (hyperv_pcpu_input_arg); + out = *this_cpu_ptr (hyperv_pcpu_input_arg) +sizeof (*in); + in->gpa = gpa; + in->size = size; + + ret =hv_do_hypercall (HVCALL_MMIO_READ , in, out); +if (hv_result_success (ret)) { +switch (size) { +case 1 : + *val = *(u8 *)(out->data); +break ; +case 2 : + *val = *(u16 *)(out->data); +break ; +default : + *val = *(u32 *)(out->data); +break ; + } + }else +dev_err (dev,"MMIO read hypercall error %llx addr %llx size %d\n" , + ret, gpa, size); +}
drivers/pci/controller/pci-hyperv.c static int wslot_to_devfn(u32 wslot) · line 1099
Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10
-0+ +static void hv_pci_write_mmio (struct device *dev,phys_addr_t gpa,int size,u32 val) +{ +struct hv_mmio_write_input *in; +u64 ret; + +/* +* Must be called with interrupts disabled so it is safe +* to use the per-cpu input argument memory. +*/
Ditto.
Thanks,
Wei.
RE: [Patch v3 13/14] PCI: hv: Add hypercalls to read/write MMIO space
From: Wei Liu <[email protected]> Sent: Thursday, November 17, 2022 7:17 AM
Quoted 19 lines · Michael Kelley
[...]
Quoted 2 lines
+static void hv_pci_read_mmio(struct device *dev,
phys_addr_tgpa, int size, u32*val)
drivers/pci/controller/pci-hyperv.c static int wslot_to_devfn(u32 wslot) · line 1067
Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10
-0+{ +struct hv_mmio_read_input *in; +struct hv_mmio_read_output *out; +u64 ret; + +/* +* Must be called with interrupts disabled so it is safe +* to use the per-cpu input argument page. Use it for +* both input and output. +*/ Perhaps adding something along this line?
Quoted 3 lines
WARN_ON(!irqs_disabled());I can fold this in if you agree.
These two new functions are only called within this module from code that already has interrupts disabled (as added in Patch 14 of the series), so I didn't do the extra check. But I'm OK with adding it. These functions make a hypercall, so the additional check doesn't have enough perf impact to matter.
Michael
Quoted 39 lines
Quoted 15 lines
drivers/pci/controller/pci-hyperv.c +14
-0+ in = *this_cpu_ptr (hyperv_pcpu_input_arg); + out = *this_cpu_ptr (hyperv_pcpu_input_arg) +sizeof (*in); + in->gpa = gpa; + in->size = size; + + ret =hv_do_hypercall (HVCALL_MMIO_READ , in, out); +if (hv_result_success (ret)) { +switch (size) { +case 1 : + *val = *(u8 *)(out->data); +break ; +case 2 : + *val = *(u16 *)(out->data); +break ;drivers/pci/controller/pci-hyperv.c static int wslot_to_devfn(u32 wslot)
Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10
-0+default : + *val = *(u32 *)(out->data); +break ; + } + }else +dev_err (dev,"MMIO read hypercall error %llx addr %llx size %d\n" , + ret, gpa, size); +} + +static void hv_pci_write_mmio(struct device *dev, phys_addr_t gpa, int size, u32val)
drivers/pci/controller/pci-hyperv.c static int wslot_to_devfn(u32 wslot) · line 1099
Quoted 8 lines
drivers/pci/controller/pci-hyperv.c +8
-0+{ +struct hv_mmio_write_input *in; +u64 ret; + +/* +* Must be called with interrupts disabled so it is safe +* to use the per-cpu input argument memory. +*/ Ditto.
Thanks,
Wei.
Wei Liu
maintainerRe: [Patch v3 13/14] PCI: hv: Add hypercalls to read/write MMIO space
Quoted 22 lines · Michael Kelley (LINUX)
From: Wei Liu <[email protected]> Sent: Thursday, November 17, 2022 7:17 AM
Quoted 21 lines · Michael Kelley
[...]
Quoted 2 lines
+static void hv_pci_read_mmio(struct device *dev,
phys_addr_tgpa, int size, u32*val)
drivers/pci/controller/pci-hyperv.c static int wslot_to_devfn(u32 wslot) · line 1067
Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10
-0+{ +struct hv_mmio_read_input *in; +struct hv_mmio_read_output *out; +u64 ret; + +/* +* Must be called with interrupts disabled so it is safe +* to use the per-cpu input argument page. Use it for +* both input and output. +*/ Perhaps adding something along this line?
WARN_ON(!irqs_disabled());
Quoted 7 lines
Quoted 1 line
I can fold this in if you agree.
These two new functions are only called within this module from code that already has interrupts disabled (as added in Patch 14 of the series), so I didn't do the extra check. But I'm OK with adding it. These functions make a hypercall, so the additional check doesn't have enough perf impact to matter.
Okay, not adding them is fine too.
Thanks,
Wei.
Re: [Patch v3 13/14] PCI: hv: Add hypercalls to read/write MMIO space
In reply to Wei LiuQuoted 14 lines · Wei Liu
On Wed, Nov 16, 2022 at 10:41:36AM -0800, Michael Kelley wrote:
[...]drivers/pci/controller/pci-hyperv.c static int wslot_to_devfn(u32 wslot) · line 1067
Quoted 12 lines
drivers/pci/controller/pci-hyperv.c +11
-0+static void hv_pci_read_mmio (struct device *dev,phys_addr_t gpa,int size,u32 *val) +{ +struct hv_mmio_read_input *in; +struct hv_mmio_read_output *out; +u64 ret; + +/* +* Must be called with interrupts disabled so it is safe +* to use the per-cpu input argument page. Use it for +* both input and output. +*/
There's no need to require interrupts to be disabled to safely use a per-cpu variable, simply disabling preemption also provides the necessary protection. And this_cpu_ptr() will complain with CONFIG_DEBUG_PREEMPT=y if preemption isn't disabled.
IIUC, based on the existing code, what is really be guarded against is an IRQ arriving and initiating a different hypercall from IRQ context, and thus corrupting the page from this function's perspective.
Quoted 3 lines
Perhaps adding something along this line?
WARN_ON(!irqs_disabled());
Given that every use of hyperv_pcpu_input_arg except hv_common_cpu_init() disables IRQs, what about adding a helper to retrieve the pointer and assert that IRQs are disabled? I.e. add the sanity for all usage, not just this one-off case.
And since CPUHP_AP_ONLINE_DYN => hv_common_cpu_init() runs after scheduling is activated by CPUHP_AP_SCHED_WAIT_EMPTY, I believe that hv_common_cpu_init() is theoretically broken. Maybe someone can look at that when fixing he KVM vs. Hyper-V issue?
https://lore.kernel.org/linux-hyperv/[email protected]
https://lore.kernel.org/all/[email protected]
Wei Liu
maintainerRe: [Patch v3 13/14] PCI: hv: Add hypercalls to read/write MMIO space
Quoted 21 lines · Sean Christopherson
Quoted 14 lines · Wei Liu
On Wed, Nov 16, 2022 at 10:41:36AM -0800, Michael Kelley wrote:
[...]drivers/pci/controller/pci-hyperv.c static int wslot_to_devfn(u32 wslot) · line 1067
Quoted 12 lines
drivers/pci/controller/pci-hyperv.c +11
-0+static void hv_pci_read_mmio (struct device *dev,phys_addr_t gpa,int size,u32 *val) +{ +struct hv_mmio_read_input *in; +struct hv_mmio_read_output *out; +u64 ret; + +/* +* Must be called with interrupts disabled so it is safe +* to use the per-cpu input argument page. Use it for +* both input and output. +*/ There's no need to require interrupts to be disabled to safely use a per-cpu variable, simply disabling preemption also provides the necessary protection. And
this_cpu_ptr()will complain withCONFIG_DEBUG_PREEMPT=yif preemption isn't disabled.
Quoted 3 lines
IIUC, based on the existing code, what is really be guarded against is an IRQ arriving and initiating a different hypercall from IRQ context, and thus corrupting the page from this function's perspective.
Exactly. Michael's comment did not say this explicitly but that's what's being guarded.
Quoted 9 lines
Quoted 4 lines
Perhaps adding something along this line?
WARN_ON(!irqs_disabled());Given that every use of
hyperv_pcpu_input_argexcepthv_common_cpu_init()disables IRQs, what about adding a helper to retrieve the pointer and assert that IRQs are disabled? I.e. add the sanity for all usage, not just this one-off case.
We can potentially introduce a pair of get/put functions for these pages, but let's not feature-creep here...
Quoted 7 lines
And since
CPUHP_AP_ONLINE_DYN=>hv_common_cpu_init()runs after scheduling is activated byCPUHP_AP_SCHED_WAIT_EMPTY, I believe thathv_common_cpu_init()is theoretically broken. Maybe someone can look at that when fixing he KVM vs. Hyper-V issue?https://lore.kernel.org/linux-hyperv/[email protected]
https://lore.kernel.org/all/[email protected]
I read the mails before have not looked into those since they are only theoretical per those threads. Sorry.
The only scenario I can think of for CPU hotplug right now is the (upcoming) Linux root kernel, I guess we will cross the bridge when we get there.
Thanks,
Wei.
RE: [Patch v3 13/14] PCI: hv: Add hypercalls to read/write MMIO space
In reply to Michael KelleyQuoted 30 lines
-----Original Message-----
From: Michael Kelley (LINUX) <[email protected]>
Sent: Wednesday, November 16, 2022 1:42 PM
To: [email protected]; KY Srinivasan <[email protected]>; Haiyang Zhang <[email protected]>; [email protected]; Dexuan Cui <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Tianyu Lan <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Williams, Dan J <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]
Cc: Michael Kelley (LINUX) <[email protected]>
Subject: [Patch v3 13/14] PCI: hv: Add hypercalls to read/write MMIO spaceTo support PCI pass-thru devices in Confidential VMs, Hyper-V has added hypercalls to read and write MMIO space. Add the appropriate definitions to
hyperv-tlfs.hand implement functions to make the hypercalls. These functions are used in a subsequent patch.
Quoted 4 lines
Co-developed-by Dexuan Cui <[email protected]>Signed-off-by Dexuan Cui <[email protected]>Signed-off-by Michael Kelley <[email protected]>---
One question - Will you put the document in patch 0 directly into some place of the src tree?
RE: [Patch v3 13/14] PCI: hv: Add hypercalls to read/write MMIO space
From: Haiyang Zhang <[email protected]> Sent: Thursday, November 17, 2022 10:33 AM
Quoted 14 lines
Quoted 9 lines
From: Michael Kelley (LINUX) <[email protected]> Sent: Wednesday, November 16, 2022 1:42 PM
To support PCI pass-thru devices in Confidential VMs, Hyper-V has added hypercalls to read and write MMIO space. Add the appropriate definitions to
hyperv-tlfs.hand implement functions to make the hypercalls. These functions are used in a subsequent patch.Quoted 4 lines
Co-developed-by Dexuan Cui <[email protected]>Signed-off-by Dexuan Cui <[email protected]>Signed-off-by Michael Kelley <[email protected]>---
Quoted 5 lines
Reviewed-by Haiyang Zhang <[email protected]>One question - Will you put the document in patch 0 directly into some place of the src tree?
Not directly. Patch 0 is supposed to be a summary of a patch set, and it doesn't have a place in the 'git' repo, though it will live on in the email archives. But the details are captured in the individual patch commit messages, particularly in Patch 7 for this series.
Separately, I want to add Confidential VMs as a topic in the Hyper-V documentation under Documentation/virt/hyperv. I'll do that once our work related to confidential computing is relatively stable.
Michael
PATCH v3 14/14 PCI: hv: Enable PCI pass-thru devices in Confidential VMs
In reply to Michael KelleyFor PCI pass-thru devices in a Confidential VM, Hyper-V requires that PCI config space be accessed via hypercalls. In normal VMs, config space accesses are trapped to the Hyper-V host and emulated. But in a confidential VM, the host can't access guest memory to decode the instruction for emulation, so an explicit hypercall must be used.
Update the PCI config space access functions to use the hypercalls when such use is indicated by Hyper-V flags. Also, set the flag to allow the Hyper-V PCI driver to be loaded and used in a Confidential VM (a.k.a., "Isolation VM"). The driver has previously been hardened against a malicious Hyper-V host[1].
[1] https://lore.kernel.org/all/[email protected]/
--- drivers/hv/channel_mgmt.c | 2 +- drivers/pci/controller/pci-hyperv.c | 168 ++++++++++++++++++++++-------------- 2 files changed, 105 insertions(+), 65 deletions(-)
drivers/hv/channel_mgmt.c +1 -1
@@ -67,7 +67,7 @@ { .dev_type = HV_PCIE,HV_PCIE_GUID , .perf_device = false, - .allowed_in_isolated = false, + .allowed_in_isolated = true, },/* Synthetic Frame Buffer */
drivers/pci/controller/pci-hyperv.c +104 -64
225 lines not on this page · raw message
Signature · 1 line
1.8.3.1