TDX host kernel support

Intel Trusted Domain Extensions (TDX) protects guest VMs from malicious host and certain physical attacks. TDX specs are available in [1].

This series is the initial support to enable TDX with minimal code to allow KVM to create and run TDX guests. KVM support for TDX is being developed separately[2]. A new KVM "guest_memfd()" to support private memory is also being developed[3]. KVM will only support the new "guest_memfd()" infrastructure for TDX.

Also, a few first generations of TDX hardware have an erratum[4], and require additional handing.

This series doesn't aim to support all functionalities, and doesn't aim to resolve all things perfectly. All other optimizations will be posted as follow-up once this initial TDX support is upstreamed.

Hi Dave/Kirill/Peter/Tony/David and all,

Thanks for your review on the previous versions. Appreciate your review on this version and any tag if patches look good to you. Thanks!

This version was based on "Unify TDCALL/SEAMCALL and TDVMCALL assembly" series, which was based on latest tip/x86/tdx, requested by Peter:

https://lore.kernel.org/lkml/[email protected]/

Please also help to review that series. Thanks!

----- Changelog history: ------

- v12 -> v13:
- Major change is reimplementing SEAMCALL related patches, and rebasing due to the "Unify TDCALL/SEAMCALL and TDVMCALL assembly" series.
- Addressed comments from Peter/Kirill/Dave/David/Nikolay/Yuan.
- Other changes please see patch changelog in individual patches.

(Also removed MM list from CC list to stop annoying MM people).

v12: https://lore.kernel.org/lkml/[email protected]/

- v11 -> v12:
- Addressed comments in v11 from Dave/Kirill/David and others.
- Collected review tags from Dave/Kirill/David and others.
- Splitted the SEAMCALL infrastructure patch into 2 patches for better reveiw.
- One more patch to change to keep TDMRs when module initialization is successful for better review.

v11: https://lore.kernel.org/lkml/[email protected]/T/

- v10 -> v11:

- Addressed comments in v10
- Added patches to handle TDX "partial write machine check" erratum.
- Added a new patch to handle running out of entropy in common code.
- Fixed a bug in kexec() support.

v10: https://lore.kernel.org/kvm/[email protected]/

- v9 -> v10:

- Changed the per-cpu initalization handling
- Gave up "ensuring all online cpus are TDX-runnable when TDX module is initialized", but just provide two basic functions, tdx_enable() and tdx_cpu_enable(), to let the user of TDX to make sure the tdx_cpu_enable() has been done successfully when the user wants to use particular cpu for TDX.
- Thus, moved per-cpu initialization out of tdx_enable(). Now tdx_enable() just assumes VMXON and tdx_cpu_enable() has been done on all online cpus before calling it.
- Merged the tdx_enable() skeleton patch and per-cpu initialization patch together to tell better story.
- Moved "SEAMCALL infrastructure" patch before the tdx_enable() patch.

v9: https://lore.kernel.org/lkml/[email protected]/

- v8 -> v9:

- Added patches to handle TDH.SYS.INIT and TDH.SYS.LP.INIT back.
- Other changes please refer to changelog histroy in individual patches.

v8: https://lore.kernel.org/lkml/[email protected]/

- v7 -> v8:

- 200+ LOC removed (from 1800+ -> 1600+).
- Removed patches to do TDH.SYS.INIT and TDH.SYS.LP.INIT (Dave/Peter/Thomas).
- Removed patch to shut down TDX module (Sean).
- For memory hotplug, changed to reject non-TDX memory from arch_add_memory() to memory_notifier (Dan/David).
- Simplified the "skeletion patch" as a result of removing TDH.SYS.LP.INIT patch.
- Refined changelog/comments for most of the patches (to tell better story, remove silly comments, etc) (Dave).
- Added new 'struct tdmr_info_list' struct, and changed all TDMR related patches to use it (Dave).
- Effectively merged patch "Reserve TDX module global KeyID" and "Configure TDX module with TDMRs and global KeyID", and removed the static variable 'tdx_global_keyid', following Dave's suggestion on making tdx_sysinfo local variable.
- For detailed changes please see individual patch changelog history.

v7: https://lore.kernel.org/lkml/[email protected]/T/

- v6 -> v7:
- Added memory hotplug support.
- Changed how to choose the list of "TDX-usable" memory regions from at kernel boot time to TDX module initialization time.
- Addressed comments received in previous versions. (Andi/Dave).
- Improved the commit message and the comments of kexec() support patch, and the patch handles returnning PAMTs back to the kernel when TDX module initialization fails. Please also see "kexec()" section below.
- Changed the documentation patch accordingly.
- For all others please see individual patch changelog history.

v6: https://lore.kernel.org/lkml/[email protected]/T/

- v5 -> v6:

- Removed ACPI CPU/memory hotplug patches. (Intel internal discussion)
- Removed patch to disable driver-managed memory hotplug (Intel internal discussion).
- Added one patch to introduce enum type for TDX supported page size level to replace the hard-coded values in TDX guest code (Dave).
- Added one patch to make TDX depends on X2APIC being enabled (Dave).
- Added one patch to build all boot-time present memory regions as TDX memory during kernel boot.
- Added Reviewed-by from others to some patches.
- For all others please see individual patch changelog history.

v5: https://lore.kernel.org/lkml/[email protected]/T/

- v4 -> v5:

This is essentially a resent of v4. Sorry I forgot to consult get_maintainer.pl when sending out v4, so I forgot to add linux-acpi and linux-mm mailing list and the relevant people for 4 new patches.

There are also very minor code and commit message update from v4:

- Rebased to latest tip/x86/tdx.
- Fixed a checkpatch issue that I missed in v4.
- Removed an obsoleted comment that I missed in patch 6.
- Very minor update to the commit message of patch 12.

For other changes to individual patches since v3, please refer to the changelog histroy of individual patches (I just used v3 -> v5 since there's basically no code change to v4).

v4: https://lore.kernel.org/lkml/98c84c31d8f062a0b50a69ef4d3188bc259f2af2.1654025431.git.kai.huang@intel.com/T/

- v3 -> v4 (addressed Dave's comments, and other comments from others):

- Simplified SEAMRR and TDX keyID detection.
- Added patches to handle ACPI CPU hotplug.
- Added patches to handle ACPI memory hotplug and driver managed memory hotplug.
- Removed tdx_detect() but only use single tdx_init().
- Removed detecting TDX module via P-SEAMLDR.
- Changed from using e820 to using memblock to convert system RAM to TDX memory.
- Excluded legacy PMEM from TDX memory.
- Removed the boot-time command line to disable TDX patch.
- Addressed comments for other individual patches (please see individual patches).
- Improved the documentation patch based on the new implementation.

v3: https://lore.kernel.org/lkml/[email protected]/T/

- V2 -> v3:

- Addressed comments from Isaku.
- Fixed memory leak and unnecessary function argument in the patch to configure the key for the global keyid (patch 17).
- Enhanced a little bit to the patch to get TDX module and CMR information (patch 09).
- Fixed an unintended change in the patch to allocate PAMT (patch 13).
- Addressed comments from Kevin:
- Slightly improvement on commit message to patch 03.
- Removed WARN_ON_ONCE() in the check of cpus_booted_once_mask in seamrr_enabled() (patch 04).
- Changed documentation patch to add TDX host kernel support materials to Documentation/x86/tdx.rst together with TDX guest staff, instead of a standalone file (patch 21)
- Very minor improvement in commit messages.

v2: https://lore.kernel.org/lkml/[email protected]/T/

- RFC (v1) -> v2:
- Rebased to Kirill's latest TDX guest code.
- Fixed two issues that are related to finding all RAM memory regions

    based on e820.

- Minor improvement on comments and commit messages.

v1: https://lore.kernel.org/lkml/[email protected]/T/

== Background ==

TDX introduces a new CPU mode called Secure Arbitration Mode (SEAM) and a new isolated range pointed by the SEAM Ranger Register (SEAMRR). A CPU-attested software module called 'the TDX module' runs in the new isolated region as a trusted hypervisor to create/run protected VMs.

TDX also leverages Intel Multi-Key Total Memory Encryption (MKTME) to provide crypto-protection to the VMs. TDX reserves part of MKTME KeyIDs as TDX private KeyIDs, which are only accessible within the SEAM mode.

TDX is different from AMD SEV/SEV-ES/SEV-SNP, which uses a dedicated secure processor to provide crypto-protection. The firmware runs on the secure processor acts a similar role as the TDX module.

The host kernel communicates with SEAM software via a new SEAMCALL instruction. This is conceptually similar to a guest->host hypercall, except it is made from the host to SEAM software instead.

Before being able to manage TD guests, the TDX module must be loaded and properly initialized. This series assumes the TDX module is loaded by BIOS before the kernel boots.

How to initialize the TDX module is described at TDX module 1.0 specification, chapter "13.Intel TDX Module Lifecycle: Enumeration, Initialization and Shutdown".

== Design Considerations ==

1. Initialize the TDX module at runtime

There are basically two ways the TDX module could be initialized: either in early boot, or at runtime before the first TDX guest is run. This series implements the runtime initialization.

Also, TDX requires a per-cpu initialization SEAMCALL to be done before making any SEAMCALL on that cpu.

This series adds two functions: tdx_cpu_enable() and tdx_enable() to do per-cpu initialization and module initialization respectively.

2. CPU hotplug

TDX doesn't support physical (ACPI) CPU hotplug. A non-buggy BIOS should never support hotpluggable CPU devicee and/or deliver ACPI CPU hotplug event to the kernel. This series doesn't handle physical (ACPI) CPU hotplug at all but depends on the BIOS to behave correctly.

Also, tdx_cpu_enable() will simply return error for any hot-added cpu if something insane happened.

Note TDX works with CPU logical online/offline, thus this series still allows to do logical CPU online/offline.

3. Kernel policy on TDX memory

The TDX module reports a list of "Convertible Memory Region" (CMR) to indicate which memory regions are TDX-capable. The TDX architecture allows the VMM to designate specific convertible memory regions as usable for TDX private memory.

The initial support of TDX guests will only allocate TDX private memory from the global page allocator. This series chooses to designate _all_ system RAM in the core-mm at the time of initializing TDX module as TDX memory to guarantee all pages in the page allocator are TDX pages.

4. Memory Hotplug

After the kernel passes all "TDX-usable" memory regions to the TDX module, the set of "TDX-usable" memory regions are fixed during module's runtime. No more "TDX-usable" memory can be added to the TDX module after that.

To achieve above "to guarantee all pages in the page allocator are TDX pages", this series simply choose to reject any non-TDX-usable memory in memory hotplug.

5. Physical Memory Hotplug

Note TDX assumes convertible memory is always physically present during machine's runtime. A non-buggy BIOS should never support hot-removal of any convertible memory. This implementation doesn't handle ACPI memory removal but depends on the BIOS to behave correctly.

Also, if something insane really happened, 4) makes sure either TDX cannot be enabled or hot-added memory will be rejected after TDX gets enabled.

6. Kexec()

Similar to AMD's SME, in kexec() kernel needs to flush dirty cachelines of TDX private memory otherwise they may silently corrupt the new kernel.

7. TDX erratum

The first few generations of TDX hardware have an erratum. A partial write to a TDX private memory cacheline will silently "poison" the line. Subsequent reads will consume the poison and generate a machine check.

The fast warm reset reboot doesn't reset TDX private memory. With this erratum, all TDX private pages needs to be converted back to normal before a fast warm reset reboot or booting to the new kernel in kexec(). Otherwise, the new kernel may get unexpected machine check.

In normal condition, triggering the erratum in Linux requires some kind of kernel bug involving relatively exotic memory writes to TDX private memory and will manifest via spurious-looking machine checks when reading the affected memory. Machine check handler is improved to deal with such machine check.

[1]: TDX specs
https://software.intel.com/content/www/us/en/develop/articles/intel-trust-domain-extensions.html

[2]: KVM TDX basic feature support
https://lore.kernel.org/lkml/[email protected]/

[3]: KVM: guest_memfd() and per-page attributes
https://lore.kernel.org/all/[email protected]/

[4]: TDX erratum
https://cdrdv2.intel.com/v1/dl/getContent/772415?explicitVersion=true

Kai Huang (22):
x86/virt/tdx: Detect TDX during kernel boot
x86/tdx: Define TDX supported page sizes as macros x86/virt/tdx: Make INTEL_TDX_HOST depend on X86_X2APIC x86/cpu: Detect TDX partial write machine check erratum x86/virt/tdx: Handle SEAMCALL no entropy error in common code x86/virt/tdx: Add SEAMCALL error printing for module initialization x86/virt/tdx: Add skeleton to enable TDX on demand x86/virt/tdx: Get information about TDX module and TDX-capable memory x86/virt/tdx: Use all system memory when initializing TDX module as
TDX memory
x86/virt/tdx: Add placeholder to construct TDMRs to cover all TDX
memory regions
x86/virt/tdx: Fill out TDMRs to cover all TDX memory regions x86/virt/tdx: Allocate and set up PAMTs for TDMRs x86/virt/tdx: Designate reserved areas for all TDMRs x86/virt/tdx: Configure TDX module with the TDMRs and global KeyID x86/virt/tdx: Configure global KeyID on all packages x86/virt/tdx: Initialize all TDMRs
x86/kexec: Flush cache of TDX private memory
x86/virt/tdx: Keep TDMRs when module initialization is successful x86/virt/tdx: Improve readibility of module initialization error
handling
x86/kexec(): Reset TDX private memory on platforms with TDX erratum x86/mce: Improve error log of kernel space TDX #MC due to erratum Documentation/x86: Add documentation for TDX host support

 Documentation/arch/x86/tdx.rst     |  184 +++-
 arch/x86/Kconfig                   |    3 +
 arch/x86/coco/tdx/tdx-shared.c     |    6 +-
 arch/x86/include/asm/cpufeatures.h |    1 +
 arch/x86/include/asm/msr-index.h   |    3 +
 arch/x86/include/asm/shared/tdx.h  |    6 +
 arch/x86/include/asm/tdx.h         |   40 +
 arch/x86/kernel/cpu/intel.c        |   17 +
 arch/x86/kernel/cpu/mce/core.c     |   33 +
 arch/x86/kernel/machine_kexec_64.c |    9 +
 arch/x86/kernel/process.c          |    8 +-
 arch/x86/kernel/reboot.c           |   15 +
 arch/x86/kernel/setup.c            |    2 +
 arch/x86/virt/vmx/tdx/Makefile     |    2 +-
 arch/x86/virt/vmx/tdx/tdx.c        | 1578 ++++++++++++++++++++++++++++
 arch/x86/virt/vmx/tdx/tdx.h        |  145 +++
 16 files changed, 2036 insertions(+), 16 deletions(-)
 create mode 100644 arch/x86/virt/vmx/tdx/tdx.c
 create mode 100644 arch/x86/virt/vmx/tdx/tdx.h

base-commit: 52a93d39b17dc7eb98b6aa3edb93943248e03b2f
prerequisite-patch-id: 0c22a1b8ee1b6034d75dee7ba2acb8f85b07d7e9
prerequisite-patch-id: 806530c9e960c04af1f89772e3f979edab98a408
prerequisite-patch-id: b7ed5b025d879e45768e9aec3dd7155849107fc9
prerequisite-patch-id: dd9b8b1bccc91dbb763e73cedd3519ea425d35e9
prerequisite-patch-id: 780a296b04f5f6e1183b32fd818b68123ea90837
prerequisite-patch-id: 95113eb3c33808cf3c9ae9c25535e03301fbbcd7
prerequisite-patch-id: e3aef1d7ca5cba99993c26ff1f8506ec3262e872
prerequisite-patch-id: f6cf57a358d1c4f671e92db813078f6ed4f772ad
prerequisite-patch-id: 3136f8c16504a77fa46b834f2a4c2de3ab84b477
prerequisite-patch-id: 47d7568b7cc0875aee1789ea2fce0f983184ca6f
prerequisite-patch-id: 94e4aaee87b6cd053a521bc286b53585436f7fad
prerequisite-patch-id: 7490affb624706a64e72f07ac0e215a4c001809e

2.41.0

Intel Trust Domain Extensions (TDX) protects guest VMs from malicious host and certain physical attacks. A CPU-attested software module called 'the TDX module' runs inside a new isolated memory range as a trusted hypervisor to manage and run protected VMs.

Pre-TDX Intel hardware has support for a memory encryption architecture called MKTME. The memory encryption hardware underpinning MKTME is also used for Intel TDX. TDX ends up "stealing" some of the physical address space from the MKTME architecture for crypto-protection to VMs. The BIOS is responsible for partitioning the "KeyID" space between legacy MKTME and TDX. The KeyIDs reserved for TDX are called 'TDX private KeyIDs' or 'TDX KeyIDs' for short.

During machine boot, TDX microcode verifies that the BIOS programmed TDX private KeyIDs consistently and correctly programmed across all CPU packages. The MSRs are locked in this state after verification. This is why MSR_IA32_MKTME_KEYID_PARTITIONING gets used for TDX enumeration:
it indicates not just that the hardware supports TDX, but that all the boot-time security checks passed.

The TDX module is expected to be loaded by the BIOS when it enables TDX, but the kernel needs to properly initialize it before it can be used to create and run any TDX guests. The TDX module will be initialized by the KVM subsystem when KVM wants to use TDX.

Add a new early_initcall(tdx_init) to detect the TDX by detecting TDX private KeyIDs. Also add a function to report whether TDX is enabled by the BIOS. Similar to AMD SME, kexec() will use it to determine whether cache flush is needed.

The TDX module itself requires one TDX KeyID as the 'TDX global KeyID' to protect its metadata. Each TDX guest also needs a TDX KeyID for its own protection. Just use the first TDX KeyID as the global KeyID and leave the rest for TDX guests. If no TDX KeyID is left for TDX guests, disable TDX as initializing the TDX module alone is useless.

---

v12 -> v13:
- Rebase to TDCALL assembly series.

v11 -> v12:
- Improve setting up guest's TDX keyID range (David)
- ++tdx_keyid_start -> tdx_keyid_start + 1
- --nr_tdx_keyids -> nr_tdx_keyids - 1
- 'return -ENODEV' instead of 'goto no_tdx' (Sathy)
- pr_info() -> pr_err() (Isaku)
- Added tags from Isaku/David

v10 -> v11 (David):
- "host kernel" -> "the host kernel"
- "protected VM" -> "confidential VM".
- Moved setting tdx_global_keyid to the end of tdx_init().

v9 -> v10:
- No change.

v8 -> v9:
- Moved MSR macro from local tdx.h to <asm/msr-index.h> (Dave).
- Moved reserving the TDX global KeyID from later patch to here.
- Changed 'tdx_keyid_start' and 'nr_tdx_keyids' to 'tdx_guest_keyid_start' and 'tdx_nr_guest_keyids' to represent KeyIDs can be used by guest. (Dave)
- Slight changelog update according to above changes.

v7 -> v8: (address Dave's comments)
- Improved changelog:
- "KVM user" -> "The TDX module will be initialized by KVM when ..."
- Changed "tdx_int" part to "Just say what this patch is doing"
- Fixed the last sentence of "kexec()" paragraph
- detect_tdx() -> record_keyid_partitioning()
- Improved how to calculate tdx_keyid_start.
- tdx_keyid_num -> nr_tdx_keyids.
- Improved dmesg printing.
- Add comment to clear_tdx().

v6 -> v7:
- No change.

v5 -> v6:
- Removed SEAMRR detection to make code simpler.
- Removed the 'default N' in the KVM_TDX_HOST Kconfig (Kirill).
- Changed to use 'obj-y' in arch/x86/virt/vmx/tdx/Makefile (Kirill).

---
 arch/x86/include/asm/msr-index.h |  3 ++
 arch/x86/include/asm/tdx.h       |  4 ++
 arch/x86/virt/vmx/tdx/Makefile   |  2 +-
 arch/x86/virt/vmx/tdx/tdx.c      | 90 ++++++++++++++++++++++++++++++++
 4 files changed, 98 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/virt/vmx/tdx/tdx.c

index a00a53e15ab7..342fc6011b4d 100644
@@ -523,6 +523,9 @@
 #define MSR_RELOAD_PMC0			0x000014c1
 #define MSR_RELOAD_FIXED_CTR0		0x00001309
 
+/* KeyID partitioning between MKTME and TDX */
+#define MSR_IA32_MKTME_KEYID_PARTITIONING	0x00000087
+
 /*
  * AMD64 MSRs. Not complete. See the architecture manual for a more
  * complete list.

index adcbe3f1de30..a252328734c7 100644
@@ -81,6 +81,10 @@ static inline long tdx_kvm_hypercall(unsigned int nr, unsigned long p1,
 u64 __seamcall(u64 fn, struct tdx_module_args *args);
 u64 __seamcall_ret(u64 fn, struct tdx_module_args *args);
 u64 __seamcall_saved_ret(u64 fn, struct tdx_module_args *args);
+
+bool platform_tdx_enabled(void);
+#else
+static inline bool platform_tdx_enabled(void) { return false; }
 #endif	/* CONFIG_INTEL_TDX_HOST */
 
 #endif /* !__ASSEMBLY__ */

index 46ef8f73aebb..90da47eb85ee 100644
@@ -1,2 +1,2 @@
 # SPDX-License-Identifier: GPL-2.0-only
-obj-y += seamcall.o
+obj-y += seamcall.o tdx.o

2.41.0

...

TDX supports 4K, 2M and 1G page sizes. The corresponding values are defined by the TDX module spec and used as TDX module ABI. Currently, they are used in try_accept_one() when the TDX guest tries to accept a page. However currently try_accept_one() uses hard-coded magic values.

Define TDX supported page sizes as macros and get rid of the hard-coded values in try_accept_one(). TDX host support will need to use them too.

---
 arch/x86/coco/tdx/tdx-shared.c    | 6 +++---
 arch/x86/include/asm/shared/tdx.h | 5 +++++
 2 files changed, 8 insertions(+), 3 deletions(-)

index 78e413269791..1655aa56a0a5 100644
@@ -22,13 +22,13 @@ static unsigned long try_accept_one(phys_addr_t start, unsigned long len,
 	 */
 	switch (pg_level) {
 	case PG_LEVEL_4K:
-		page_size = 0;
+		page_size = TDX_PS_4K;
 		break;
 	case PG_LEVEL_2M:
-		page_size = 1;
+		page_size = TDX_PS_2M;
 		break;
 	case PG_LEVEL_1G:
-		page_size = 2;
+		page_size = TDX_PS_1G;
 		break;
 	default:
 		return 0;

index 8d1427562c63..257a41d0a36d 100644
@@ -52,6 +52,11 @@
 	(TDX_RDX | TDX_RBX | TDX_RSI | TDX_RDI | TDX_R8  | TDX_R9  | \
 	 TDX_R10 | TDX_R11 | TDX_R12 | TDX_R13 | TDX_R14 | TDX_R15)
 
+/* TDX supported page sizes from the TDX module ABI. */
+#define TDX_PS_4K	0
+#define TDX_PS_2M	1
+#define TDX_PS_1G	2
+
 #ifndef __ASSEMBLY__
 
 /*

2.41.0

TDX capable platforms are locked to X2APIC mode and cannot fall back to the legacy xAPIC mode when TDX is enabled by the BIOS. TDX host support requires x2APIC. Make INTEL_TDX_HOST depend on X86_X2APIC.

---
 arch/x86/Kconfig | 1 +
 1 file changed, 1 insertion(+)

index 0558dd98abd7..114f8c6e95d7 100644
@@ -1954,6 +1954,7 @@ config INTEL_TDX_HOST
 	depends on CPU_SUP_INTEL
 	depends on X86_64
 	depends on KVM_INTEL
+	depends on X86_X2APIC
 	help
 	  Intel Trust Domain Extensions (TDX) protects guest VMs from malicious
 	  host and certain physical attacks.  This option enables necessary TDX

2.41.0

TDX memory has integrity and confidentiality protections. Violations of this integrity protection are supposed to only affect TDX operations and are never supposed to affect the host kernel itself. In other words, the host kernel should never, itself, see machine checks induced by the TDX integrity hardware.

Alas, the first few generations of TDX hardware have an erratum. A partial write to a TDX private memory cacheline will silently "poison" the line. Subsequent reads will consume the poison and generate a machine check. According to the TDX hardware spec, neither of these things should have happened.

Virtually all kernel memory accesses operations happen in full cachelines. In practice, writing a "byte" of memory usually reads a 64 byte cacheline of memory, modifies it, then writes the whole line back. Those operations do not trigger this problem.

This problem is triggered by "partial" writes where a write transaction of less than cacheline lands at the memory controller. The CPU does these via non-temporal write instructions (like MOVNTI), or through UC/WC memory mappings. The issue can also be triggered away from the CPU by devices doing partial writes via DMA.

With this erratum, there are additional things need to be done. Similar to other CPU bugs, use a CPU bug bit to indicate this erratum, and detect this erratum during early boot. Note this bug reflects the hardware thus it is detected regardless of whether the kernel is built with TDX support or not.

---

v12 -> v13:
- Added David's tag.

v11 -> v12:
- Added Kirill's tag
- Changed to detect the erratum in early_init_intel() (Kirill)

v10 -> v11:
- New patch

---
 arch/x86/include/asm/cpufeatures.h |  1 +
 arch/x86/kernel/cpu/intel.c        | 17 +++++++++++++++++
 2 files changed, 18 insertions(+)

index cb8ca46213be..dc8701f8d88b 100644
@@ -483,5 +483,6 @@
 #define X86_BUG_RETBLEED		X86_BUG(27) /* CPU is affected by RETBleed */
 #define X86_BUG_EIBRS_PBRSB		X86_BUG(28) /* EIBRS is vulnerable to Post Barrier RSB Predictions */
 #define X86_BUG_SMT_RSB			X86_BUG(29) /* CPU is vulnerable to Cross-Thread Return Address Predictions */
+#define X86_BUG_TDX_PW_MCE		X86_BUG(30) /* CPU may incur #MC if non-TD software does partial write to TDX private memory */
 
 #endif /* _ASM_X86_CPUFEATURES_H */

index 1c4639588ff9..e6c3107adc15 100644
@@ -358,6 +358,21 @@ int intel_microcode_sanity_check(void *mc, bool print_err, int hdr_type)
 }
 EXPORT_SYMBOL_GPL(intel_microcode_sanity_check);
 
+static void check_tdx_erratum(struct cpuinfo_x86 *c)
+{
+	/*
+	 * These CPUs have an erratum.  A partial write from non-TD
+	 * software (e.g. via MOVNTI variants or UC/WC mapping) to TDX
+	 * private memory poisons that memory, and a subsequent read of
+	 * that memory triggers #MC.
+	 */
+	switch (c->x86_model) {
+	case INTEL_FAM6_SAPPHIRERAPIDS_X:
+	case INTEL_FAM6_EMERALDRAPIDS_X:
+		setup_force_cpu_bug(X86_BUG_TDX_PW_MCE);
+	}
+}
+
 static void early_init_intel(struct cpuinfo_x86 *c)
 {
 	u64 misc_enable;

@@ -509,6 +524,8 @@ static void early_init_intel(struct cpuinfo_x86 *c)
 	 */
 	if (detect_extended_topology_early(c) < 0)
 		detect_ht_early(c);
+
+	check_tdx_erratum(c);
 }
 
 static void bsp_init_intel(struct cpuinfo_x86 *c)

2.41.0

This is missing one thing: alluding to how this will be used. We might do that by saying: "To prepare for _____, add ______."

But that's a minor nit.

...

Thanks for suggestion.

I thought I somehow mentioned at last in the changelog:

With this erratum, there are additional things need to be done.
Similar
to other CPU bugs, use a CPU bug bit to indicate this erratum ...

Perhaps it's not clear. How about below?

With this erratum, there are additional things need to be done around
kexec() and machine check handler. To prepare for those changes, add a CPU bugbittoindicatethiserratum.Notethisbugreflectsthe hardwarethusitisdetectedregardlessofwhetherthekernelisbuilt
with TDX support or not.

Thanks!

Some SEAMCALLs use the RDRAND hardware and can fail for the same reasons as RDRAND. Use the kernel RDRAND retry logic for them.

There are three __seamcall*() variants. Add a macro to do the SEAMCALL retry in the common code and define a wrapper for each __seamcall*() variant.

---

v12 -> v13:
- New implementation due to TDCALL assembly series.

---
 arch/x86/include/asm/tdx.h | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

index a252328734c7..cfae8b31a2e9 100644
@@ -24,6 +24,11 @@
 #define TDX_SEAMCALL_GP			(TDX_SW_ERROR | X86_TRAP_GP)
 #define TDX_SEAMCALL_UD			(TDX_SW_ERROR | X86_TRAP_UD)
 
+/*
+ * TDX module SEAMCALL leaf function error codes
+ */
+#define TDX_RND_NO_ENTROPY	0x8000020300000000ULL
+
 #ifndef __ASSEMBLY__
 
 /*

@@ -82,6 +87,28 @@ u64 __seamcall(u64 fn, struct tdx_module_args *args);
 u64 __seamcall_ret(u64 fn, struct tdx_module_args *args);
 u64 __seamcall_saved_ret(u64 fn, struct tdx_module_args *args);
 
+#include <asm/archrandom.h>
+
+#define SEAMCALL_NO_ENTROPY_RETRY(__seamcall_func, __fn, __args)	\
+({									\
+	int ___retry = RDRAND_RETRY_LOOPS;				\
+	u64 ___sret;							\
+									\
+	do {								\
+		___sret = __seamcall_func((__fn), (__args));		\
+	} while (___sret == TDX_RND_NO_ENTROPY && --___retry);		\
+	___sret;							\
+})
+
+#define seamcall(__fn, __args)						\
+	SEAMCALL_NO_ENTROPY_RETRY(__seamcall, (__fn), (__args))
+
+#define seamcall_ret(__fn, __args)					\
+	SEAMCALL_NO_ENTROPY_RETRY(__seamcall_ret, (__fn), (__args))
+
+#define seamcall_saved_ret(__fn, __args)				\
+	SEAMCALL_NO_ENTROPY_RETRY(__seamcall_saved_ret, (__fn), (__args))
+
 bool platform_tdx_enabled(void);
 #else
 static inline bool platform_tdx_enabled(void) { return false; }

2.41.0

Indirect call was a concern when I was implementing those. I didn't know for sure that the compiler can avoid it. I'll change to use above. Thanks!

Yes I verified the checkpatch.pl wouldn't complain if the #define exceeded 80 characters in one line. I'll use #define. Thanks!

TDX module initialization is essentially to make a set of SEAMCALL leafs to complete a state machine involving multiple states. These SEAMCALLs are not expected to fail. In fact, they are not expected to return any non-zero code (except the "running out of entropy error", which can be handled internally already).

Add yet another layer of SEAMCALL wrappers, which treats all non-zero return code as error, to support printing SEAMCALL error upon failure for module initialization.

Other SEAMCALLs may treat some specific error codes as legal (e.g., some can return BUSY legally and expect the caller to retry). The caller can use the wrappers w/o error printing for those cases. The new wrappers can also be improved to suit those cases. Leave this as future work.

SEAMCALL can also return kernel defined error codes for three special
cases: 1) TDX isn't enabled by the BIOS; 2) TDX module isn't loaded; 3) CPU isn't in VMX operation. The first case isn't expected (unless BIOS bug, etc) because SEAMCALL is only expected to be made when the kernel detects TDX is enabled. The second case is only expected to be legal for the very first SEAMCALL during module initialization. The third case can be legal for any SEAMCALL leaf because VMX can be disabled due to emergency reboot.

Also add wrappers to convert the SEAMCALL error code to the kernel error code so that each caller doesn't have to repeat. Blindly print error for the above special cases to save the effort to optimize them.

TDX module can only be initialized once during its life cycle, but the module can be runtime updated by the kernel (not yet supported). After module runtime update, the kernel needs to initialize it again. Use pr_err() to print SEAMCALL error for module initialization, because if using pr_err_once() the SEAMCALL error during module initialization won't be printed after module runtime update.

At last, for now implement those wrappers in tdx.c but they can be moved to <asm/tdx.h> when needed. They are implemented with intention to be shared by other kernel components. After all, in most cases, SEAMCALL failure is unexpected and the caller just wants to print.

---

v12 -> v13:
- New implementation due to TDCALL assembly series.

---
 arch/x86/include/asm/tdx.h  |  1 +
 arch/x86/virt/vmx/tdx/tdx.c | 84 +++++++++++++++++++++++++++++++++++++
 2 files changed, 85 insertions(+)

index cfae8b31a2e9..3b248c94a4a4 100644
@@ -27,6 +27,7 @@
 /*
  * TDX module SEAMCALL leaf function error codes
  */
+#define TDX_SUCCESS		0ULL
 #define TDX_RND_NO_ENTROPY	0x8000020300000000ULL
 
 #ifndef __ASSEMBLY__

2.41.0

Please see below.

Thanks for comments!

I was hoping __SEAMCALL_PRERR() can be used by KVM code but I guess I was over- thinking.  I can remove __SEAMCALL_PRERR() unless Isaku thinks it is useful to KVM.

However maybe it's better to keep __prerr_func in seamcall_err*() as KVM TDX patches use pr_err_ratelimited(). I am hoping KVM can use those to avoid duplication at some level. Also, IMHO having __prerr_func in seamcall_err*() would make SEAMCALL_PRERR() more understandable because we can immediately see pr_err() is used by just looking at SEAMCALL_PRERR().

Anyway I am eager to hear comments from others too. :-)

Be that as it may, I think it warrants at least some mentioning in the changelog. Alternatively in the first iteration of those patches this can be omitted and then it can be introduced at the time the first users shows up. In any case, let's wait for the KVM people to comment.

Yeah agreed. I had below in the changelog but perhaps it's too vague:

"At last, for now implement those wrappers in tdx.c but they can be moved to <asm/tdx.h> when needed. They are implemented with intention to be shared by other kernel components."

Those changes are due to SEAMCALL API change from the TDCALL/VMCALL/SEAMCALL assembly change patchset. I'll work internally to make this stable asap (I tried before but was suggested to sent out to community for feedback).

Also I would appreciate if you could take a look at patch 18/19 (which are separated small patches for better review) and patch 20 (reset PAMT in kexec).

Thanks in advance!

To enable TDX the kernel needs to initialize TDX from two perspectives:
1) Do a set of SEAMCALLs to initialize the TDX module to make it ready to create and run TDX guests; 2) Do the per-cpu initialization SEAMCALL on one logical cpu before the kernel wants to make any other SEAMCALLs on that cpu (including those involved during module initialization and running TDX guests).

The TDX module can be initialized only once in its lifetime. Instead of always initializing it at boot time, this implementation chooses an "on demand" approach to initialize TDX until there is a real need (e.g when requested by KVM). This approach has below pros:

1) It avoids consuming the memory that must be allocated by kernel and given to the TDX module as metadata (~1/256th of the TDX-usable memory), and also saves the CPU cycles of initializing the TDX module (and the metadata) when TDX is not used at all.

2) The TDX module design allows it to be updated while the system is running. The update procedure shares quite a few steps with this "on demand" initialization mechanism. The hope is that much of "on demand" mechanism can be shared with a future "update" mechanism. A boot-time TDX module implementation would not be able to share much code with the update mechanism.

3) Making SEAMCALL requires VMX to be enabled. Currently, only the KVM code mucks with VMX enabling. If the TDX module were to be initialized separately from KVM (like at boot), the boot code would need to be taught how to muck with VMX enabling and KVM would need to be taught how to cope with that. Making KVM itself responsible for TDX initialization lets the rest of the kernel stay blissfully unaware of VMX.

Similar to module initialization, also make the per-cpu initialization "on demand" as it also depends on VMX being enabled.

Add two functions, tdx_enable() and tdx_cpu_enable(), to enable the TDX module and enable TDX on local cpu respectively. For now tdx_enable() is a placeholder. The TODO list will be pared down as functionality is added.

Export both tdx_cpu_enable() and tdx_enable() for KVM use.

In tdx_enable() use a state machine protected by mutex to make sure the initialization will only be done once, as tdx_enable() can be called multiple times (i.e. KVM module can be reloaded) and may be called concurrently by other kernel components in the future.

The per-cpu initialization on each cpu can only be done once during the module's life time. Use a per-cpu variable to track its status to make sure it is only done once in tdx_cpu_enable().

Also, a SEAMCALL to do TDX module global initialization must be done once on any logical cpu before any per-cpu initialization SEAMCALL. Do it inside tdx_cpu_enable() too (if hasn't been done).

tdx_enable() can potentially invoke SEAMCALLs on any online cpus. The per-cpu initialization must be done before those SEAMCALLs are invoked on some cpu. To keep things simple, in tdx_cpu_enable(), always do the per-cpu initialization regardless of whether the TDX module has been initialized or not. And in tdx_enable(), don't call tdx_cpu_enable() but assume the caller has disabled CPU hotplug, done VMXON and tdx_cpu_enable() on all online cpus before calling tdx_enable().

---

v12 -> v13:
- Made tdx_cpu_enable() always be called with IRQ disabled via IPI funciton call (Peter, Kirill).

v11 -> v12:
- Simplified TDX module global init and lp init status tracking (David).
- Added comment around try_init_module_global() for using raw_spin_lock() (Dave).
- Added one sentence to changelog to explain why to expose tdx_enable() and tdx_cpu_enable() (Dave).
- Simplifed comments around tdx_enable() and tdx_cpu_enable() to use lockdep_assert_*() instead. (Dave)
- Removed redundent "TDX" in error message (Dave).

v10 -> v11:
- Return -NODEV instead of -EINVAL when CONFIG_INTEL_TDX_HOST is off.
- Return the actual error code for tdx_enable() instead of -EINVAL.
- Added Isaku's Reviewed-by.

v9 -> v10:
- Merged the patch to handle per-cpu initialization to this patch to tell the story better.
- Changed how to handle the per-cpu initialization to only provide a tdx_cpu_enable() function to let the user of TDX to do it when the user wants to run TDX code on a certain cpu.
- Changed tdx_enable() to not call cpus_read_lock() explicitly, but call lockdep_assert_cpus_held() to assume the caller has done that.
- Improved comments around tdx_enable() and tdx_cpu_enable().
- Improved changelog to tell the story better accordingly.

v8 -> v9:
- Removed detailed TODO list in the changelog (Dave).
- Added back steps to do module global initialization and per-cpu initialization in the TODO list comment.
- Moved the 'enum tdx_module_status_t' from tdx.c to local tdx.h

v7 -> v8:
- Refined changelog (Dave).
- Removed "all BIOS-enabled cpus" related code (Peter/Thomas/Dave).
- Add a "TODO list" comment in init_tdx_module() to list all steps of initializing the TDX Module to tell the story (Dave).
- Made tdx_enable() unverisally return -EINVAL, and removed nonsense comments (Dave).
- Simplified __tdx_enable() to only handle success or failure.
- TDX_MODULE_SHUTDOWN -> TDX_MODULE_ERROR
- Removed TDX_MODULE_NONE (not loaded) as it is not necessary.
- Improved comments (Dave).
- Pointed out 'tdx_module_status' is software thing (Dave).

v6 -> v7:
- No change.

v5 -> v6:
- Added code to set status to TDX_MODULE_NONE if TDX module is not loaded (Chao)
- Added Chao's Reviewed-by.
- Improved comments around cpus_read_lock().

- v3->v5 (no feedback on v4):
- Removed the check that SEAMRR and TDX KeyID have been detected on all present cpus.
- Removed tdx_detect().
- Added num_online_cpus() to MADT-enabled CPUs check within the CPU hotplug lock and return early with error message.
- Improved dmesg printing for TDX module detection and initialization.

---
 arch/x86/include/asm/tdx.h  |   4 +
 arch/x86/virt/vmx/tdx/tdx.c | 157 ++++++++++++++++++++++++++++++++++++
 arch/x86/virt/vmx/tdx/tdx.h |  30 +++++++
 3 files changed, 191 insertions(+)
 create mode 100644 arch/x86/virt/vmx/tdx/tdx.h

index 3b248c94a4a4..fce7abc99bf5 100644
@@ -111,8 +111,12 @@ u64 __seamcall_saved_ret(u64 fn, struct tdx_module_args *args);
 	SEAMCALL_NO_ENTROPY_RETRY(__seamcall_saved_ret, (__fn), (__args))
 
 bool platform_tdx_enabled(void);
+int tdx_cpu_enable(void);
+int tdx_enable(void);
 #else
 static inline bool platform_tdx_enabled(void) { return false; }
+static inline int tdx_cpu_enable(void) { return -ENODEV; }
+static inline int tdx_enable(void)  { return -ENODEV; }
 #endif	/* CONFIG_INTEL_TDX_HOST */
 
 #endif /* !__ASSEMBLY__ */

new file mode 100644
index 000000000000..a3c52270df5b
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _X86_VIRT_TDX_H
+#define _X86_VIRT_TDX_H
+
+/*
+ * This file contains both macros and data structures defined by the TDX
+ * architecture and Linux defined software data structures and functions.
+ * The two should not be mixed together for better readability.  The
+ * architectural definitions come first.
+ */
+
+/*
+ * TDX module SEAMCALL leaf functions
+ */
+#define TDH_SYS_INIT		33
+#define TDH_SYS_LP_INIT		35
+
+/*
+ * Do not put any hardware-defined TDX structure representations below
+ * this comment!
+ */
+
+/* Kernel defined TDX module status during module initialization. */
+enum tdx_module_status_t {
+	TDX_MODULE_UNINITIALIZED,
+	TDX_MODULE_INITIALIZED,
+	TDX_MODULE_ERROR
+};
+
+#endif

2.41.0

<snip>

Yeah can do. Thanks. I didn't do because tdx_cpu_enable() already does it.

Start to transit out the "multi-steps" to initialize the TDX module.

TDX provides increased levels of memory confidentiality and integrity. This requires special hardware support for features like memory encryption and storage of memory integrity checksums. Not all memory satisfies these requirements.

As a result, TDX introduced the concept of a "Convertible Memory Region" (CMR). During boot, the firmware builds a list of all of the memory ranges which can provide the TDX security guarantees.

CMRs tell the kernel which memory is TDX compatible. The kernel takes CMRs (plus a little more metadata) and constructs "TD Memory Regions" (TDMRs). TDMRs let the kernel grant TDX protections to some or all of the CMR areas.

The TDX module also reports necessary information to let the kernel build TDMRs and run TDX guests in structure 'tdsysinfo_struct'. The list of CMRs, along with the TDX module information, is available to the kernel by querying the TDX module.

As a preparation to construct TDMRs, get the TDX module information and the list of CMRs. Print out CMRs to help user to decode which memory regions are TDX convertible.

The 'tdsysinfo_struct' is fairly large (1024 bytes) and contains a lot of info about the TDX module. Fully define the entire structure, but only use the fields necessary to build the TDMRs and pr_info() some basics about the module. The rest of the fields will get used by KVM.

---

v12 -> v13:
- Allocate TDSYSINFO and CMR array separately. (Kirill)
- Added comment around TDH.SYS.INFO. (Peter)

v11 -> v12:
- Changed to use dynamic allocation for TDSYSINFO_STRUCT and CMR array (Kirill).
- Keep SEAMCALL leaf macro definitions in order (Kirill)
- Removed is_cmr_empty() but open code directly (David)
- 'atribute' -> 'attribute' (David)

v10 -> v11:
- No change.

v9 -> v10:
- Added back "start to transit out..." as now per-cpu init has been moved out from tdx_enable().

v8 -> v9:
- Removed "start to trransit out ..." part in changelog since this patch is no longer the first step anymore.
- Changed to declare 'tdsysinfo' and 'cmr_array' as local static, and changed changelog accordingly (Dave).
- Improved changelog to explain why to declare 'tdsysinfo_struct' in full but only use a few members of them (Dave).

v7 -> v8: (Dave)
- Improved changelog to tell this is the first patch to transit out the "multi-steps" init_tdx_module().
- Removed all CMR check/trim code but to depend on later SEAMCALL.
- Variable 'vertical alignment' in print TDX module information.
- Added DECLARE_PADDED_STRUCT() for padded structure.
- Made tdx_sysinfo and tdx_cmr_array[] to be function local variable (and rename them accordingly), and added -Wframe-larger-than=4096 flag to silence the build warning.

v6 -> v7:
- Simplified the check of CMRs due to the fact that TDX actually verifies CMRs (that are passed by the BIOS) before enabling TDX.
- Changed the function name from check_cmrs() -> trim_empty_cmrs().
- Added CMR page aligned check so that later patch can just get the PFN using ">> PAGE_SHIFT".

v5 -> v6:
- Added to also print TDX module's attribute (Isaku).
- Removed all arguments in tdx_gete_sysinfo() to use static variables of 'tdx_sysinfo' and 'tdx_cmr_array' directly as they are all used directly in other functions in later patches.
- Added Isaku's Reviewed-by.

- v3 -> v5 (no feedback on v4):
- Renamed sanitize_cmrs() to check_cmrs().
- Removed unnecessary sanity check against tdx_sysinfo and tdx_cmr_array actual size returned by TDH.SYS.INFO.
- Changed -EFAULT to -EINVAL in couple places.
- Added comments around tdx_sysinfo and tdx_cmr_array saying they are used by TDH.SYS.INFO ABI.
- Changed to pass 'tdx_sysinfo' and 'tdx_cmr_array' as function arguments in tdx_get_sysinfo().
- Changed to only print BIOS-CMR when check_cmrs() fails.

---
 arch/x86/virt/vmx/tdx/tdx.c | 94 ++++++++++++++++++++++++++++++++++++-
 arch/x86/virt/vmx/tdx/tdx.h | 64 +++++++++++++++++++++++++
 2 files changed, 156 insertions(+), 2 deletions(-)

index 898523d8b8b0..61e5baea6ad7 100644
@@ -16,8 +16,11 @@
 #include <linux/spinlock.h>
 #include <linux/percpu-defs.h>
 #include <linux/mutex.h>
+#include <linux/slab.h>
+#include <linux/math.h>
 #include <asm/msr-index.h>
 #include <asm/msr.h>
+#include <asm/page.h>
 #include <asm/tdx.h>
 #include "tdx.h"
 

@@ -184,12 +187,91 @@ int tdx_cpu_enable(void)
 }
 EXPORT_SYMBOL_GPL(tdx_cpu_enable);
 
+static void print_cmrs(struct cmr_info *cmr_array, int nr_cmrs)
+{
+	int i;
+
+	for (i = 0; i < nr_cmrs; i++) {
+		struct cmr_info *cmr = &cmr_array[i];
+
+		/*
+		 * The array of CMRs reported via TDH.SYS.INFO can
+		 * contain tail empty CMRs.  Don't print them.
+		 */
+		if (!cmr->size)
+			break;
+
+		pr_info("CMR: [0x%llx, 0x%llx)\n", cmr->base,
+				cmr->base + cmr->size);
+	}
+}
+
+static int get_tdx_sysinfo(struct tdsysinfo_struct *tdsysinfo,
+			   struct cmr_info *cmr_array)
+{
+	struct tdx_module_args args;
+	int ret;
+
+	/*
+	 * TDH.SYS.INFO writes the TDSYSINFO_STRUCT and the CMR array
+	 * to the buffers provided by the kernel (via RCX and R8
+	 * respectively).  The buffer size of the TDSYSINFO_STRUCT
+	 * (via RDX) and the maximum entries of the CMR array (via R9)
+	 * passed to this SEAMCALL must be at least the size of
+	 * TDSYSINFO_STRUCT and MAX_CMRS respectively.
+	 *
+	 * Upon a successful return, R9 contains the actual entries
+	 * written to the CMR array.
+	 */
+	args.rcx = __pa(tdsysinfo);
+	args.rdx = TDSYSINFO_STRUCT_SIZE;
+	args.r8 = __pa(cmr_array);
+	args.r9 = MAX_CMRS;
+	ret = seamcall_prerr_ret(TDH_SYS_INFO, &args);
+	if (ret)
+		return ret;
+
+	pr_info("TDX module: attributes 0x%x, vendor_id 0x%x, major_version %u, minor_version %u, build_date %u, build_num %u",
+		tdsysinfo->attributes,    tdsysinfo->vendor_id,
+		tdsysinfo->major_version, tdsysinfo->minor_version,
+		tdsysinfo->build_date,    tdsysinfo->build_num);
+
+	print_cmrs(cmr_array, args.r9);
+
+	return 0;
+}
+
 static int init_tdx_module(void)
 {
+	struct tdsysinfo_struct *tdsysinfo;
+	struct cmr_info *cmr_array;
+	int tdsysinfo_size;
+	int cmr_array_size;
+	int ret;
+
+	tdsysinfo_size = round_up(TDSYSINFO_STRUCT_SIZE,
+			TDSYSINFO_STRUCT_ALIGNMENT);
+	tdsysinfo = kzalloc(tdsysinfo_size, GFP_KERNEL);
+	if (!tdsysinfo)
+		return -ENOMEM;
+
+	cmr_array_size = sizeof(struct cmr_info) * MAX_CMRS;
+	cmr_array_size = round_up(cmr_array_size, CMR_INFO_ARRAY_ALIGNMENT);
+	cmr_array = kzalloc(cmr_array_size, GFP_KERNEL);
+	if (!cmr_array) {
+		kfree(tdsysinfo);
+		return -ENOMEM;
+	}
+
+
+	/* Get the TDSYSINFO_STRUCT and CMRs from the TDX module. */
+	ret = get_tdx_sysinfo(tdsysinfo, cmr_array);
+	if (ret)
+		goto out;
+
 	/*
 	 * TODO:
 	 *
-	 *  - Get TDX module information and TDX-capable memory regions.
 	 *  - Build the list of TDX-usable memory regions.
 	 *  - Construct a list of "TD Memory Regions" (TDMRs) to cover
 	 *    all TDX-usable memory regions.

@@ -199,7 +281,15 @@ static int init_tdx_module(void)
 	 *
 	 *  Return error before all steps are done.
 	 */
-	return -EINVAL;
+	ret = -EINVAL;
+out:
+	/*
+	 * For now both @sysinfo and @cmr_array are only used during
+	 * module initialization, so always free them.
+	 */
+	kfree(tdsysinfo);
+	kfree(cmr_array);
+	return ret;
 }
 
 static int __tdx_enable(void)

2.41.0

As a step of initializing the TDX module, the kernel needs to tell the TDX module which memory regions can be used by the TDX module as TDX guest memory.

TDX reports a list of "Convertible Memory Region" (CMR) to tell the kernel which memory is TDX compatible. The kernel needs to build a list of memory regions (out of CMRs) as "TDX-usable" memory and pass them to the TDX module. Once this is done, those "TDX-usable" memory regions are fixed during module's lifetime.

To keep things simple, assume that all TDX-protected memory will come from the page allocator. Make sure all pages in the page allocator *are* TDX-usable memory.

As TDX-usable memory is a fixed configuration, take a snapshot of the memory configuration from memblocks at the time of module initialization (memblocks are modified on memory hotplug). This snapshot is used to enable TDX support for *this* memory configuration only. Use a memory hotplug notifier to ensure that no other RAM can be added outside of this configuration.

This approach requires all memblock memory regions at the time of module initialization to be TDX convertible memory to work, otherwise module initialization will fail in a later SEAMCALL when passing those regions to the module. This approach works when all boot-time "system RAM" is TDX convertible memory, and no non-TDX-convertible memory is hot-added to the core-mm before module initialization.

For instance, on the first generation of TDX machines, both CXL memory and NVDIMM are not TDX convertible memory. Using kmem driver to hot-add any CXL memory or NVDIMM to the core-mm before module initialization will result in failure to initialize the module. The SEAMCALL error code will be available in the dmesg to help user to understand the failure.

---

v12 -> v13:
- Avoided using " ? : " in tdx_memory_notifier(). (Peter)

v11 -> v12:
- Added tags from Dave/Kirill.

v10 -> v11:
- Added Isaku's Reviewed-by.

v9 -> v10:
- Moved empty @tdx_memlist check out of is_tdx_memory() to make the logic better.
- Added Ying's Reviewed-by.

v8 -> v9:
- Replace "The initial support ..." with timeless sentence in both changelog and comments(Dave).
- Fix run-on sentence in changelog, and senstence to explain why to stash off memblock (Dave).
- Tried to improve why to choose this approach and how it work in changelog based on Dave's suggestion.
- Many other comments enhancement (Dave).

v7 -> v8:
- Trimed down changelog (Dave).
- Changed to use PHYS_PFN() and PFN_PHYS() throughout this series (Ying).
- Moved memory hotplug handling from add_arch_memory() to memory_notifier (Dan/David).
- Removed 'nid' from 'struct tdx_memblock' to later patch (Dave).
- {build|free}_tdx_memory() -> {build|}free_tdx_memlist() (Dave).
- Removed pfn_covered_by_cmr() check as no code to trim CMRs now.
- Improve the comment around first 1MB (Dave).
- Added a comment around reserve_real_mode() to point out TDX code relies on first 1MB being reserved (Ying).
- Added comment to explain why the new online memory range cannot cross multiple TDX memory blocks (Dave).
- Improved other comments (Dave).

---
 arch/x86/Kconfig            |   1 +
 arch/x86/kernel/setup.c     |   2 +
 arch/x86/virt/vmx/tdx/tdx.c | 162 +++++++++++++++++++++++++++++++++++-
 arch/x86/virt/vmx/tdx/tdx.h |   6 ++
 4 files changed, 170 insertions(+), 1 deletion(-)

index 114f8c6e95d7..92666bdfacda 100644
@@ -1955,6 +1955,7 @@ config INTEL_TDX_HOST
 	depends on X86_64
 	depends on KVM_INTEL
 	depends on X86_X2APIC
+	select ARCH_KEEP_MEMBLOCK
 	help
 	  Intel Trust Domain Extensions (TDX) protects guest VMs from malicious
 	  host and certain physical attacks.  This option enables necessary TDX

index fd975a4a5200..b84bd190272d 100644
@@ -1161,6 +1161,8 @@ void __init setup_arch(char **cmdline_p)
 	 *
 	 * Moreover, on machines with SandyBridge graphics or in setups that use
 	 * crashkernel the entire 1M is reserved anyway.
+	 *
+	 * Note the host kernel TDX also requires the first 1MB being reserved.
 	 */
 	x86_platform.realmode_reserve();
 

index fff36af71448..39c9c6fdc11e 100644
@@ -91,4 +91,10 @@ enum tdx_module_status_t {
 	TDX_MODULE_ERROR
 };
 
+struct tdx_memblock {
+	struct list_head list;
+	unsigned long start_pfn;
+	unsigned long end_pfn;
+};
+
 #endif

2.41.0

After the kernel selects all TDX-usable memory regions, the kernel needs to pass those regions to the TDX module via data structure "TD Memory Region" (TDMR).

Add a placeholder to construct a list of TDMRs (in multiple steps) to cover all TDX-usable memory regions.

=== Long Version ===

TDX provides increased levels of memory confidentiality and integrity. This requires special hardware support for features like memory encryption and storage of memory integrity checksums. Not all memory satisfies these requirements.

As a result, TDX introduced the concept of a "Convertible Memory Region" (CMR). During boot, the firmware builds a list of all of the memory ranges which can provide the TDX security guarantees. The list of these ranges is available to the kernel by querying the TDX module.

The TDX architecture needs additional metadata to record things like which TD guest "owns" a given page of memory. This metadata essentially serves as the 'struct page' for the TDX module. The space for this metadata is not reserved by the hardware up front and must be allocated by the kernel and given to the TDX module.

Since this metadata consumes space, the VMM can choose whether or not to allocate it for a given area of convertible memory. If it chooses not to, the memory cannot receive TDX protections and can not be used by TDX guests as private memory.

For every memory region that the VMM wants to use as TDX memory, it sets up a "TD Memory Region" (TDMR). Each TDMR represents a physically contiguous convertible range and must also have its own physically contiguous metadata table, referred to as a Physical Address Metadata Table (PAMT), to track status for each page in the TDMR range.

Unlike a CMR, each TDMR requires 1G granularity and alignment. To support physical RAM areas that don't meet those strict requirements, each TDMR permits a number of internal "reserved areas" which can be placed over memory holes. If PAMT metadata is placed within a TDMR it must be covered by one of these reserved areas.

Let's summarize the concepts:

CMR - Firmware-enumerated physical ranges that support TDX. CMRs are
4K aligned.
TDMR - Physical address range which is chosen by the kernel to support
TDX. 1G granularity and alignment required. Each TDMR has reserved areas where TDX memory holes and overlapping PAMTs can be represented.
PAMT - Physically contiguous TDX metadata. One table for each page size
per TDMR. Roughly 1/256th of TDMR in size. 256G TDMR = ~1G PAMT.

As one step of initializing the TDX module, the kernel configures TDX-usable memory regions by passing a list of TDMRs to the TDX module.

Constructing the list of TDMRs consists below steps:

1) Fill out TDMRs to cover all memory regions that the TDX module will use for TD memory.
2) Allocate and set up PAMT for each TDMR.
3) Designate reserved areas for each TDMR.

Add a placeholder to construct TDMRs to do the above steps. To keep things simple, just allocate enough space to hold maximum number of TDMRs up front. Always free the buffer of TDMRs since they are only used during module initialization.

---

v12 -> v13:
- No change.

v11 -> v12:
- Added tags from Dave/Kirill.

v10 -> v11:
- Changed to keep TDMRs after module initialization to deal with TDX erratum in future patches.

v9 -> v10:
- Changed the TDMR list from static variable back to local variable as now TDX module isn't disabled when tdx_cpu_enable() fails.

v8 -> v9:
- Changes around 'struct tdmr_info_list' (Dave):
- Moved the declaration from tdx.c to tdx.h.
- Renamed 'first_tdmr' to 'tdmrs'.
- 'nr_tdmrs' -> 'nr_consumed_tdmrs'.
- Changed 'tdmrs' to 'void *'.
- Improved comments for all structure members.
- Added a missing empty line in alloc_tdmr_list() (Dave).

v7 -> v8:
- Improved changelog to tell this is one step of "TODO list" in init_tdx_module().
- Other changelog improvement suggested by Dave (with "Create TDMRs" to "Fill out TDMRs" to align with the code).
- Added a "TODO list" comment to lay out the steps to construct TDMRs, following the same idea of "TODO list" in tdx_module_init().
- Introduced 'struct tdmr_info_list' (Dave)
- Further added additional members (tdmr_sz/max_tdmrs/nr_tdmrs) to simplify getting TDMR by given index, and reduce passing arguments around functions.
- Added alloc_tdmr_list()/free_tdmr_list() accordingly, which internally uses tdmr_size_single() (Dave).
- tdmr_num -> nr_tdmrs (Dave).

v6 -> v7:
- Improved commit message to explain 'int' overflow cannot happen in cal_tdmr_size() and alloc_tdmr_array(). -- Andy/Dave.

v5 -> v6:
- construct_tdmrs_memblock() -> construct_tdmrs() as 'tdx_memblock' is used instead of memblock.
- Added Isaku's Reviewed-by.

- v3 -> v5 (no feedback on v4):
- Moved calculating TDMR size to this patch.
- Changed to use alloc_pages_exact() to allocate buffer for all TDMRs once, instead of allocating each TDMR individually.
- Removed "crypto protection" in the changelog.
- -EFAULT -> -EINVAL in couple of places.

---
 arch/x86/virt/vmx/tdx/tdx.c | 97 ++++++++++++++++++++++++++++++++++++-
 arch/x86/virt/vmx/tdx/tdx.h | 32 ++++++++++++
 2 files changed, 127 insertions(+), 2 deletions(-)

index d0a758e984a4..64a7afefc214 100644
@@ -24,6 +24,7 @@
 #include <linux/minmax.h>
 #include <linux/sizes.h>
 #include <linux/pfn.h>
+#include <linux/align.h>
 #include <asm/msr-index.h>
 #include <asm/msr.h>
 #include <asm/page.h>

@@ -323,9 +324,84 @@ static int build_tdx_memlist(struct list_head *tmb_list)
 	return ret;
 }
 
+/* Calculate the actual TDMR size */
+static int tdmr_size_single(u16 max_reserved_per_tdmr)
+{
+	int tdmr_sz;
+
+	/*
+	 * The actual size of TDMR depends on the maximum
+	 * number of reserved areas.
+	 */
+	tdmr_sz = sizeof(struct tdmr_info);
+	tdmr_sz += sizeof(struct tdmr_reserved_area) * max_reserved_per_tdmr;
+
+	return ALIGN(tdmr_sz, TDMR_INFO_ALIGNMENT);
+}
+
+static int alloc_tdmr_list(struct tdmr_info_list *tdmr_list,
+			   struct tdsysinfo_struct *sysinfo)
+{
+	size_t tdmr_sz, tdmr_array_sz;
+	void *tdmr_array;
+
+	tdmr_sz = tdmr_size_single(sysinfo->max_reserved_per_tdmr);
+	tdmr_array_sz = tdmr_sz * sysinfo->max_tdmrs;
+
+	/*
+	 * To keep things simple, allocate all TDMRs together.
+	 * The buffer needs to be physically contiguous to make
+	 * sure each TDMR is physically contiguous.
+	 */
+	tdmr_array = alloc_pages_exact(tdmr_array_sz,
+			GFP_KERNEL | __GFP_ZERO);
+	if (!tdmr_array)
+		return -ENOMEM;
+
+	tdmr_list->tdmrs = tdmr_array;
+
+	/*
+	 * Keep the size of TDMR to find the target TDMR
+	 * at a given index in the TDMR list.
+	 */
+	tdmr_list->tdmr_sz = tdmr_sz;
+	tdmr_list->max_tdmrs = sysinfo->max_tdmrs;
+	tdmr_list->nr_consumed_tdmrs = 0;
+
+	return 0;
+}
+
+static void free_tdmr_list(struct tdmr_info_list *tdmr_list)
+{
+	free_pages_exact(tdmr_list->tdmrs,
+			tdmr_list->max_tdmrs * tdmr_list->tdmr_sz);
+}
+
+/*
+ * Construct a list of TDMRs on the preallocated space in @tdmr_list
+ * to cover all TDX memory regions in @tmb_list based on the TDX module
+ * information in @sysinfo.
+ */
+static int construct_tdmrs(struct list_head *tmb_list,
+			   struct tdmr_info_list *tdmr_list,
+			   struct tdsysinfo_struct *sysinfo)
+{
+	/*
+	 * TODO:
+	 *
+	 *  - Fill out TDMRs to cover all TDX memory regions.
+	 *  - Allocate and set up PAMTs for each TDMR.
+	 *  - Designate reserved areas for each TDMR.
+	 *
+	 * Return -EINVAL until constructing TDMRs is done
+	 */
+	return -EINVAL;
+}
+
 static int init_tdx_module(void)
 {
 	struct tdsysinfo_struct *tdsysinfo;
+	struct tdmr_info_list tdmr_list;
 	struct cmr_info *cmr_array;
 	int tdsysinfo_size;
 	int cmr_array_size;

@@ -367,11 +443,19 @@ static int init_tdx_module(void)
 	if (ret)
 		goto out_put_tdxmem;
 
+	/* Allocate enough space for constructing TDMRs */
+	ret = alloc_tdmr_list(&tdmr_list, tdsysinfo);
+	if (ret)
+		goto out_free_tdxmem;
+
+	/* Cover all TDX-usable memory regions in TDMRs */
+	ret = construct_tdmrs(&tdx_memlist, &tdmr_list, tdsysinfo);
+	if (ret)
+		goto out_free_tdmrs;
+
 	/*
 	 * TODO:
 	 *
-	 *  - Construct a list of "TD Memory Regions" (TDMRs) to cover
-	 *    all TDX-usable memory regions.
 	 *  - Configure the TDMRs and the global KeyID to the TDX module.
 	 *  - Configure the global KeyID on all packages.
 	 *  - Initialize all TDMRs.

@@ -379,6 +463,15 @@ static int init_tdx_module(void)
 	 *  Return error before all steps are done.
 	 */
 	ret = -EINVAL;
+out_free_tdmrs:
+	/*
+	 * Always free the buffer of TDMRs as they are only used during
+	 * module initialization.
+	 */
+	free_tdmr_list(&tdmr_list);
+out_free_tdxmem:
+	if (ret)
+		free_tdx_memlist(&tdx_memlist);
 out_put_tdxmem:
 	/*
 	 * @tdx_memlist is written here and read at memory hotplug time.

index 39c9c6fdc11e..536d89928cd6 100644
@@ -79,6 +79,29 @@ struct tdsysinfo_struct {
 	DECLARE_FLEX_ARRAY(struct cpuid_config, cpuid_configs);
 } __packed;
 
+struct tdmr_reserved_area {
+	u64 offset;
+	u64 size;
+} __packed;
+
+#define TDMR_INFO_ALIGNMENT	512
+
+struct tdmr_info {
+	u64 base;
+	u64 size;
+	u64 pamt_1g_base;
+	u64 pamt_1g_size;
+	u64 pamt_2m_base;
+	u64 pamt_2m_size;
+	u64 pamt_4k_base;
+	u64 pamt_4k_size;
+	/*
+	 * Actual number of reserved areas depends on
+	 * 'struct tdsysinfo_struct'::max_reserved_per_tdmr.
+	 */
+	DECLARE_FLEX_ARRAY(struct tdmr_reserved_area, reserved_areas);
+} __packed __aligned(TDMR_INFO_ALIGNMENT);
+
 /*
  * Do not put any hardware-defined TDX structure representations below
  * this comment!

@@ -97,4 +120,13 @@ struct tdx_memblock {
 	unsigned long end_pfn;
 };
 
+struct tdmr_info_list {
+	void *tdmrs;	/* Flexible array to hold 'tdmr_info's */
+	int nr_consumed_tdmrs;	/* How many 'tdmr_info's are in use */
+
+	/* Metadata for finding target 'tdmr_info' and freeing @tdmrs */
+	int tdmr_sz;	/* Size of one 'tdmr_info' */
+	int max_tdmrs;	/* How many 'tdmr_info's are allocated */
+};
+
 #endif

2.41.0

Start to transit out the "multi-steps" to construct a list of "TD Memory Regions" (TDMRs) to cover all TDX-usable memory regions.

The kernel configures TDX-usable memory regions by passing a list of TDMRs "TD Memory Regions" (TDMRs) to the TDX module. Each TDMR contains the information of the base/size of a memory region, the base/size of the associated Physical Address Metadata Table (PAMT) and a list of reserved areas in the region.

Do the first step to fill out a number of TDMRs to cover all TDX memory regions. To keep it simple, always try to use one TDMR for each memory region. As the first step only set up the base/size for each TDMR.

Each TDMR must be 1G aligned and the size must be in 1G granularity. This implies that one TDMR could cover multiple memory regions. If a memory region spans the 1GB boundary and the former part is already covered by the previous TDMR, just use a new TDMR for the remaining part.

TDX only supports a limited number of TDMRs. Disable TDX if all TDMRs are consumed but there is more memory region to cover.

There are fancier things that could be done like trying to merge adjacent TDMRs. This would allow more pathological memory layouts to be supported. But, current systems are not even close to exhausting the existing TDMR resources in practice. For now, keep it simple.

---

v12 -> v13:
- Added Yuan's tag.

v11 -> v12:
- Improved comments around looping over TDX memblock to create TDMRs. (Dave).
- Added code to pr_warn() when consumed TDMRs reaching maximum TDMRs (Dave).
- BIT_ULL(30) -> SZ_1G (Kirill)
- Removed unused TDMR_PFN_ALIGNMENT (Sathy)
- Added tags from Kirill/Sathy

v10 -> v11:
- No update

v9 -> v10:
- No change.

v8 -> v9:

- Added the last paragraph in the changelog (Dave).
- Removed unnecessary type cast in tdmr_entry() (Dave).

---
 arch/x86/virt/vmx/tdx/tdx.c | 103 +++++++++++++++++++++++++++++++++++-
 arch/x86/virt/vmx/tdx/tdx.h |   3 ++
 2 files changed, 105 insertions(+), 1 deletion(-)

index 64a7afefc214..8188f834674e 100644
@@ -377,6 +377,102 @@ static void free_tdmr_list(struct tdmr_info_list *tdmr_list)
 			tdmr_list->max_tdmrs * tdmr_list->tdmr_sz);
 }
 
+/* Get the TDMR from the list at the given index. */
+static struct tdmr_info *tdmr_entry(struct tdmr_info_list *tdmr_list,
+				    int idx)
+{
+	int tdmr_info_offset = tdmr_list->tdmr_sz * idx;
+
+	return (void *)tdmr_list->tdmrs + tdmr_info_offset;
+}
+
+#define TDMR_ALIGNMENT		SZ_1G
+#define TDMR_ALIGN_DOWN(_addr)	ALIGN_DOWN((_addr), TDMR_ALIGNMENT)
+#define TDMR_ALIGN_UP(_addr)	ALIGN((_addr), TDMR_ALIGNMENT)
+
+static inline u64 tdmr_end(struct tdmr_info *tdmr)
+{
+	return tdmr->base + tdmr->size;
+}
+
+/*
+ * Take the memory referenced in @tmb_list and populate the
+ * preallocated @tdmr_list, following all the special alignment
+ * and size rules for TDMR.
+ */
+static int fill_out_tdmrs(struct list_head *tmb_list,
+			  struct tdmr_info_list *tdmr_list)
+{
+	struct tdx_memblock *tmb;
+	int tdmr_idx = 0;
+
+	/*
+	 * Loop over TDX memory regions and fill out TDMRs to cover them.
+	 * To keep it simple, always try to use one TDMR to cover one
+	 * memory region.
+	 *
+	 * In practice TDX supports at least 64 TDMRs.  A 2-socket system
+	 * typically only consumes less than 10 of those.  This code is
+	 * dumb and simple and may use more TMDRs than is strictly
+	 * required.
+	 */
+	list_for_each_entry(tmb, tmb_list, list) {
+		struct tdmr_info *tdmr = tdmr_entry(tdmr_list, tdmr_idx);
+		u64 start, end;
+
+		start = TDMR_ALIGN_DOWN(PFN_PHYS(tmb->start_pfn));
+		end   = TDMR_ALIGN_UP(PFN_PHYS(tmb->end_pfn));
+
+		/*
+		 * A valid size indicates the current TDMR has already
+		 * been filled out to cover the previous memory region(s).
+		 */
+		if (tdmr->size) {
+			/*
+			 * Loop to the next if the current memory region
+			 * has already been fully covered.
+			 */
+			if (end <= tdmr_end(tdmr))
+				continue;
+
+			/* Otherwise, skip the already covered part. */
+			if (start < tdmr_end(tdmr))
+				start = tdmr_end(tdmr);
+
+			/*
+			 * Create a new TDMR to cover the current memory
+			 * region, or the remaining part of it.
+			 */
+			tdmr_idx++;
+			if (tdmr_idx >= tdmr_list->max_tdmrs) {
+				pr_warn("initialization failed: TDMRs exhausted.\n");
+				return -ENOSPC;
+			}
+
+			tdmr = tdmr_entry(tdmr_list, tdmr_idx);
+		}
+
+		tdmr->base = start;
+		tdmr->size = end - start;
+	}
+
+	/* @tdmr_idx is always the index of the last valid TDMR. */
+	tdmr_list->nr_consumed_tdmrs = tdmr_idx + 1;
+
+	/*
+	 * Warn early that kernel is about to run out of TDMRs.
+	 *
+	 * This is an indication that TDMR allocation has to be
+	 * reworked to be smarter to not run into an issue.
+	 */
+	if (tdmr_list->max_tdmrs - tdmr_list->nr_consumed_tdmrs < TDMR_NR_WARN)
+		pr_warn("consumed TDMRs reaching limit: %d used out of %d\n",
+				tdmr_list->nr_consumed_tdmrs,
+				tdmr_list->max_tdmrs);
+
+	return 0;
+}
+
 /*
  * Construct a list of TDMRs on the preallocated space in @tdmr_list
  * to cover all TDX memory regions in @tmb_list based on the TDX module

@@ -386,10 +482,15 @@ static int construct_tdmrs(struct list_head *tmb_list,
 			   struct tdmr_info_list *tdmr_list,
 			   struct tdsysinfo_struct *sysinfo)
 {
+	int ret;
+
+	ret = fill_out_tdmrs(tmb_list, tdmr_list);
+	if (ret)
+		return ret;
+
 	/*
 	 * TODO:
 	 *
-	 *  - Fill out TDMRs to cover all TDX memory regions.
 	 *  - Allocate and set up PAMTs for each TDMR.
 	 *  - Designate reserved areas for each TDMR.
 	 *

index 536d89928cd6..15afd6a56fdc 100644
@@ -120,6 +120,9 @@ struct tdx_memblock {
 	unsigned long end_pfn;
 };
 
+/* Warn if kernel has less than TDMR_NR_WARN TDMRs after allocation */
+#define TDMR_NR_WARN 4
+
 struct tdmr_info_list {
 	void *tdmrs;	/* Flexible array to hold 'tdmr_info's */
 	int nr_consumed_tdmrs;	/* How many 'tdmr_info's are in use */

2.41.0

The TDX module uses additional metadata to record things like which guest "owns" a given page of memory. This metadata, referred as Physical Address Metadata Table (PAMT), essentially serves as the 'struct page' for the TDX module. PAMTs are not reserved by hardware up front. They must be allocated by the kernel and then given to the TDX module during module initialization.

TDX supports 3 page sizes: 4K, 2M, and 1G. Each "TD Memory Region" (TDMR) has 3 PAMTs to track the 3 supported page sizes. Each PAMT must be a physically contiguous area from a Convertible Memory Region (CMR). However, the PAMTs which track pages in one TDMR do not need to reside within that TDMR but can be anywhere in CMRs. If one PAMT overlaps with any TDMR, the overlapping part must be reported as a reserved area in that particular TDMR.

Use alloc_contig_pages() since PAMT must be a physically contiguous area and it may be potentially large (~1/256th of the size of the given TDMR). The downside is alloc_contig_pages() may fail at runtime. One (bad) mitigation is to launch a TDX guest early during system boot to get those PAMTs allocated at early time, but the only way to fix is to add a boot option to allocate or reserve PAMTs during kernel boot.

It is imperfect but will be improved on later.

TDX only supports a limited number of reserved areas per TDMR to cover both PAMTs and memory holes within the given TDMR. If many PAMTs are allocated within a single TDMR, the reserved areas may not be sufficient to cover all of them.

Adopt the following policies when allocating PAMTs for a given TDMR:

- Allocate three PAMTs of the TDMR in one contiguous chunk to minimize the total number of reserved areas consumed for PAMTs.
- Try to first allocate PAMT from the local node of the TDMR for better NUMA locality.

Also dump out how many pages are allocated for PAMTs when the TDX module is initialized successfully. This helps answer the eternal "where did all my memory go?" questions.

---

v12 -> v13:
- Added Kirill and Yuan's tag.
- Removed unintended space. (Yuan)

v11 -> v12:
- Moved TDX_PS_NUM from tdx.c to <asm/tdx.h> (Kirill)
- "<= TDX_PS_1G" -> "< TDX_PS_NUM" (Kirill)
- Changed tdmr_get_pamt() to return base and size instead of base_pfn and npages and related code directly (Dave).
- Simplified PAMT kb counting. (Dave)
- tdmrs_count_pamt_pages() -> tdmr_count_pamt_kb() (Kirill/Dave)

v10 -> v11:
- No update

v9 -> v10:
- Removed code change in disable_tdx_module() as it doesn't exist anymore.

v8 -> v9:
- Added TDX_PS_NR macro instead of open-coding (Dave).
- Better alignment of 'pamt_entry_size' in tdmr_set_up_pamt() (Dave).
- Changed to print out PAMTs in "KBs" instead of "pages" (Dave).
- Added Dave's Reviewed-by.

v7 -> v8: (Dave)
- Changelog:
- Added a sentence to state PAMT allocation will be improved.
- Others suggested by Dave.
- Moved 'nid' of 'struct tdx_memblock' to this patch.
- Improved comments around tdmr_get_nid().
- WARN_ON_ONCE() -> pr_warn() in tdmr_get_nid().
- Other changes due to 'struct tdmr_info_list'.

v6 -> v7:
- Changes due to using macros instead of 'enum' for TDX supported page sizes.

v5 -> v6:
- Rebase due to using 'tdx_memblock' instead of memblock.
- 'int pamt_entry_nr' -> 'unsigned long nr_pamt_entries' (Dave/Sagis).
- Improved comment around tdmr_get_nid() (Dave).
- Improved comment in tdmr_set_up_pamt() around breaking the PAMT into PAMTs for 4K/2M/1G (Dave).
- tdmrs_get_pamt_pages() -> tdmrs_count_pamt_pages() (Dave).

- v3 -> v5 (no feedback on v4):
- Used memblock to get the NUMA node for given TDMR.
- Removed tdmr_get_pamt_sz() helper but use open-code instead.
- Changed to use 'switch .. case..' for each TDX supported page size in tdmr_get_pamt_sz() (the original __tdmr_get_pamt_sz()).
- Added printing out memory used for PAMT allocation when TDX module is initialized successfully.
- Explained downside of alloc_contig_pages() in changelog.
- Addressed other minor comments.

---
 arch/x86/Kconfig                  |   1 +
 arch/x86/include/asm/shared/tdx.h |   1 +
 arch/x86/virt/vmx/tdx/tdx.c       | 215 +++++++++++++++++++++++++++++-
 arch/x86/virt/vmx/tdx/tdx.h       |   1 +
 4 files changed, 213 insertions(+), 5 deletions(-)

index 92666bdfacda..62d7e3f0ff43 100644
@@ -1956,6 +1956,7 @@ config INTEL_TDX_HOST
 	depends on KVM_INTEL
 	depends on X86_X2APIC
 	select ARCH_KEEP_MEMBLOCK
+	depends on CONTIG_ALLOC
 	help
 	  Intel Trust Domain Extensions (TDX) protects guest VMs from malicious
 	  host and certain physical attacks.  This option enables necessary TDX

index 257a41d0a36d..464174c63d80 100644
@@ -56,6 +56,7 @@
 #define TDX_PS_4K	0
 #define TDX_PS_2M	1
 #define TDX_PS_1G	2
+#define TDX_PS_NR	(TDX_PS_1G + 1)
 
 #ifndef __ASSEMBLY__
 

index 15afd6a56fdc..6987af46d096 100644
@@ -118,6 +118,7 @@ struct tdx_memblock {
 	struct list_head list;
 	unsigned long start_pfn;
 	unsigned long end_pfn;
+	int nid;
 };
 
 /* Warn if kernel has less than TDMR_NR_WARN TDMRs after allocation */

2.41.0

As the last step of constructing TDMRs, populate reserved areas for all TDMRs. For each TDMR, put all memory holes within this TDMR to the reserved areas. And for all PAMTs which overlap with this TDMR, put all the overlapping parts to reserved areas too.

---

v12 -> v13:
- Added Yuan's tag.

v11 -> v12:
- Code change due to tdmr_get_pamt() change from returning pfn/npages to base/size
- Added Kirill's tag

v10 -> v11:
- No update

v9 -> v10:
- No change.

v8 -> v9:
- Added comment around 'tdmr_add_rsvd_area()' to point out it doesn't do optimization to save reserved areas. (Dave).

v7 -> v8: (Dave)
- "set_up" -> "populate" in function name change (Dave).
- Improved comment suggested by Dave.
- Other changes due to 'struct tdmr_info_list'.

v6 -> v7:
- No change.

v5 -> v6:
- Rebase due to using 'tdx_memblock' instead of memblock.
- Split tdmr_set_up_rsvd_areas() into two functions to handle memory hole and PAMT respectively.
- Added Isaku's Reviewed-by.

---
 arch/x86/virt/vmx/tdx/tdx.c | 217 ++++++++++++++++++++++++++++++++++--
 1 file changed, 209 insertions(+), 8 deletions(-)

index 0ff83883ae68..9173d9798231 100644
@@ -25,6 +25,7 @@
 #include <linux/sizes.h>
 #include <linux/pfn.h>
 #include <linux/align.h>
+#include <linux/sort.h>
 #include <asm/msr-index.h>
 #include <asm/msr.h>
 #include <asm/page.h>

@@ -670,6 +671,207 @@ static unsigned long tdmrs_count_pamt_kb(struct tdmr_info_list *tdmr_list)
 	return pamt_size / 1024;
 }
 
+static int tdmr_add_rsvd_area(struct tdmr_info *tdmr, int *p_idx, u64 addr,
+			      u64 size, u16 max_reserved_per_tdmr)
+{
+	struct tdmr_reserved_area *rsvd_areas = tdmr->reserved_areas;
+	int idx = *p_idx;
+
+	/* Reserved area must be 4K aligned in offset and size */
+	if (WARN_ON(addr & ~PAGE_MASK || size & ~PAGE_MASK))
+		return -EINVAL;
+
+	if (idx >= max_reserved_per_tdmr) {
+		pr_warn("initialization failed: TDMR [0x%llx, 0x%llx): reserved areas exhausted.\n",
+				tdmr->base, tdmr_end(tdmr));
+		return -ENOSPC;
+	}
+
+	/*
+	 * Consume one reserved area per call.  Make no effort to
+	 * optimize or reduce the number of reserved areas which are
+	 * consumed by contiguous reserved areas, for instance.
+	 */
+	rsvd_areas[idx].offset = addr - tdmr->base;
+	rsvd_areas[idx].size = size;
+
+	*p_idx = idx + 1;
+
+	return 0;
+}
+
+/*
+ * Go through @tmb_list to find holes between memory areas.  If any of
+ * those holes fall within @tdmr, set up a TDMR reserved area to cover
+ * the hole.
+ */
+static int tdmr_populate_rsvd_holes(struct list_head *tmb_list,
+				    struct tdmr_info *tdmr,
+				    int *rsvd_idx,
+				    u16 max_reserved_per_tdmr)
+{
+	struct tdx_memblock *tmb;
+	u64 prev_end;
+	int ret;
+
+	/*
+	 * Start looking for reserved blocks at the
+	 * beginning of the TDMR.
+	 */
+	prev_end = tdmr->base;
+	list_for_each_entry(tmb, tmb_list, list) {
+		u64 start, end;
+
+		start = PFN_PHYS(tmb->start_pfn);
+		end   = PFN_PHYS(tmb->end_pfn);
+
+		/* Break if this region is after the TDMR */
+		if (start >= tdmr_end(tdmr))
+			break;
+
+		/* Exclude regions before this TDMR */
+		if (end < tdmr->base)
+			continue;
+
+		/*
+		 * Skip over memory areas that
+		 * have already been dealt with.
+		 */
+		if (start <= prev_end) {
+			prev_end = end;
+			continue;
+		}
+
+		/* Add the hole before this region */
+		ret = tdmr_add_rsvd_area(tdmr, rsvd_idx, prev_end,
+				start - prev_end,
+				max_reserved_per_tdmr);
+		if (ret)
+			return ret;
+
+		prev_end = end;
+	}
+
+	/* Add the hole after the last region if it exists. */
+	if (prev_end < tdmr_end(tdmr)) {
+		ret = tdmr_add_rsvd_area(tdmr, rsvd_idx, prev_end,
+				tdmr_end(tdmr) - prev_end,
+				max_reserved_per_tdmr);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+/*
+ * Go through @tdmr_list to find all PAMTs.  If any of those PAMTs
+ * overlaps with @tdmr, set up a TDMR reserved area to cover the
+ * overlapping part.
+ */
+static int tdmr_populate_rsvd_pamts(struct tdmr_info_list *tdmr_list,
+				    struct tdmr_info *tdmr,
+				    int *rsvd_idx,
+				    u16 max_reserved_per_tdmr)
+{
+	int i, ret;
+
+	for (i = 0; i < tdmr_list->nr_consumed_tdmrs; i++) {
+		struct tdmr_info *tmp = tdmr_entry(tdmr_list, i);
+		unsigned long pamt_base, pamt_size, pamt_end;
+
+		tdmr_get_pamt(tmp, &pamt_base, &pamt_size);
+		/* Each TDMR must already have PAMT allocated */
+		WARN_ON_ONCE(!pamt_size || !pamt_base);
+
+		pamt_end = pamt_base + pamt_size;
+		/* Skip PAMTs outside of the given TDMR */
+		if ((pamt_end <= tdmr->base) ||
+				(pamt_base >= tdmr_end(tdmr)))
+			continue;
+
+		/* Only mark the part within the TDMR as reserved */
+		if (pamt_base < tdmr->base)
+			pamt_base = tdmr->base;
+		if (pamt_end > tdmr_end(tdmr))
+			pamt_end = tdmr_end(tdmr);
+
+		ret = tdmr_add_rsvd_area(tdmr, rsvd_idx, pamt_base,
+				pamt_end - pamt_base,
+				max_reserved_per_tdmr);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
+/* Compare function called by sort() for TDMR reserved areas */
+static int rsvd_area_cmp_func(const void *a, const void *b)
+{
+	struct tdmr_reserved_area *r1 = (struct tdmr_reserved_area *)a;
+	struct tdmr_reserved_area *r2 = (struct tdmr_reserved_area *)b;
+
+	if (r1->offset + r1->size <= r2->offset)
+		return -1;
+	if (r1->offset >= r2->offset + r2->size)
+		return 1;
+
+	/* Reserved areas cannot overlap.  The caller must guarantee. */
+	WARN_ON_ONCE(1);
+	return -1;
+}
+
+/*
+ * Populate reserved areas for the given @tdmr, including memory holes
+ * (via @tmb_list) and PAMTs (via @tdmr_list).
+ */
+static int tdmr_populate_rsvd_areas(struct tdmr_info *tdmr,
+				    struct list_head *tmb_list,
+				    struct tdmr_info_list *tdmr_list,
+				    u16 max_reserved_per_tdmr)
+{
+	int ret, rsvd_idx = 0;
+
+	ret = tdmr_populate_rsvd_holes(tmb_list, tdmr, &rsvd_idx,
+			max_reserved_per_tdmr);
+	if (ret)
+		return ret;
+
+	ret = tdmr_populate_rsvd_pamts(tdmr_list, tdmr, &rsvd_idx,
+			max_reserved_per_tdmr);
+	if (ret)
+		return ret;
+
+	/* TDX requires reserved areas listed in address ascending order */
+	sort(tdmr->reserved_areas, rsvd_idx, sizeof(struct tdmr_reserved_area),
+			rsvd_area_cmp_func, NULL);
+
+	return 0;
+}
+
+/*
+ * Populate reserved areas for all TDMRs in @tdmr_list, including memory
+ * holes (via @tmb_list) and PAMTs.
+ */
+static int tdmrs_populate_rsvd_areas_all(struct tdmr_info_list *tdmr_list,
+					 struct list_head *tmb_list,
+					 u16 max_reserved_per_tdmr)
+{
+	int i;
+
+	for (i = 0; i < tdmr_list->nr_consumed_tdmrs; i++) {
+		int ret;
+
+		ret = tdmr_populate_rsvd_areas(tdmr_entry(tdmr_list, i),
+				tmb_list, tdmr_list, max_reserved_per_tdmr);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
 /*
  * Construct a list of TDMRs on the preallocated space in @tdmr_list
  * to cover all TDX memory regions in @tmb_list based on the TDX module

@@ -689,14 +891,13 @@ static int construct_tdmrs(struct list_head *tmb_list,
 			sysinfo->pamt_entry_size);
 	if (ret)
 		return ret;
-	/*
-	 * TODO:
-	 *
-	 *  - Designate reserved areas for each TDMR.
-	 *
-	 * Return -EINVAL until constructing TDMRs is done
-	 */
-	return -EINVAL;
+
+	ret = tdmrs_populate_rsvd_areas_all(tdmr_list, tmb_list,
+			sysinfo->max_reserved_per_tdmr);
+	if (ret)
+		tdmrs_free_pamt_all(tdmr_list);
+
+	return ret;
 }
 
 static int init_tdx_module(void)

2.41.0

The TDX module uses a private KeyID as the "global KeyID" for mapping things like the PAMT and other TDX metadata. This KeyID has already been reserved when detecting TDX during the kernel early boot.

After the list of "TD Memory Regions" (TDMRs) has been constructed to cover all TDX-usable memory regions, the next step is to pass them to the TDX module together with the global KeyID.

---

v12 -> v13:
- Added Yuan's tag.

v11 -> v12:
- Added Kirill's tag

v10 -> v11:
- No update

v9 -> v10:
- Code change due to change static 'tdx_tdmr_list' to local 'tdmr_list'.

v8 -> v9:
- Improved changlog to explain why initializing TDMRs can take long time (Dave).
- Improved comments around 'next-to-initialize' address (Dave).

v7 -> v8: (Dave)
- Changelog:
- explicitly call out this is the last step of TDX module initialization.
- Trimed down changelog by removing SEAMCALL name and details.
- Removed/trimmed down unnecessary comments.
- Other changes due to 'struct tdmr_info_list'.

v6 -> v7:
- Removed need_resched() check. -- Andi.

---
 arch/x86/virt/vmx/tdx/tdx.c | 43 ++++++++++++++++++++++++++++++++++++-
 arch/x86/virt/vmx/tdx/tdx.h |  2 ++
 2 files changed, 44 insertions(+), 1 deletion(-)

index 9173d9798231..d059226e8829 100644
@@ -26,6 +26,7 @@
 #include <linux/pfn.h>
 #include <linux/align.h>
 #include <linux/sort.h>
+#include <linux/log2.h>
 #include <asm/msr-index.h>
 #include <asm/msr.h>
 #include <asm/page.h>

@@ -900,6 +901,41 @@ static int construct_tdmrs(struct list_head *tmb_list,
 	return ret;
 }
 
+static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid)
+{
+	struct tdx_module_args args = {};
+	u64 *tdmr_pa_array;
+	size_t array_sz;
+	int i, ret;
+
+	/*
+	 * TDMRs are passed to the TDX module via an array of physical
+	 * addresses of each TDMR.  The array itself also has certain
+	 * alignment requirement.
+	 */
+	array_sz = tdmr_list->nr_consumed_tdmrs * sizeof(u64);
+	array_sz = roundup_pow_of_two(array_sz);
+	if (array_sz < TDMR_INFO_PA_ARRAY_ALIGNMENT)
+		array_sz = TDMR_INFO_PA_ARRAY_ALIGNMENT;
+
+	tdmr_pa_array = kzalloc(array_sz, GFP_KERNEL);
+	if (!tdmr_pa_array)
+		return -ENOMEM;
+
+	for (i = 0; i < tdmr_list->nr_consumed_tdmrs; i++)
+		tdmr_pa_array[i] = __pa(tdmr_entry(tdmr_list, i));
+
+	args.rcx = __pa(tdmr_pa_array);
+	args.rdx = tdmr_list->nr_consumed_tdmrs;
+	args.r8 = global_keyid;
+	ret = seamcall_prerr(TDH_SYS_CONFIG, &args);
+
+	/* Free the array as it is not required anymore. */
+	kfree(tdmr_pa_array);
+
+	return ret;
+}
+
 static int init_tdx_module(void)
 {
 	struct tdsysinfo_struct *tdsysinfo;

@@ -955,16 +991,21 @@ static int init_tdx_module(void)
 	if (ret)
 		goto out_free_tdmrs;
 
+	/* Pass the TDMRs and the global KeyID to the TDX module */
+	ret = config_tdx_module(&tdmr_list, tdx_global_keyid);
+	if (ret)
+		goto out_free_pamts;
+
 	/*
 	 * TODO:
 	 *
-	 *  - Configure the TDMRs and the global KeyID to the TDX module.
 	 *  - Configure the global KeyID on all packages.
 	 *  - Initialize all TDMRs.
 	 *
 	 *  Return error before all steps are done.
 	 */
 	ret = -EINVAL;
+out_free_pamts:
 	if (ret)
 		tdmrs_free_pamt_all(&tdmr_list);
 	else

index 6987af46d096..b8c9e3d016f9 100644
@@ -19,6 +19,7 @@
 #define TDH_SYS_INFO		32
 #define TDH_SYS_INIT		33
 #define TDH_SYS_LP_INIT		35
+#define TDH_SYS_CONFIG		45
 
 struct cmr_info {
 	u64	base;

@@ -85,6 +86,7 @@ struct tdmr_reserved_area {
 } __packed;
 
 #define TDMR_INFO_ALIGNMENT	512
+#define TDMR_INFO_PA_ARRAY_ALIGNMENT	512
 
 struct tdmr_info {
 	u64 base;

2.41.0

After the list of TDMRs and the global KeyID are configured to the TDX module, the kernel needs to configure the key of the global KeyID on all packages using TDH.SYS.KEY.CONFIG.

This SEAMCALL cannot run parallel on different cpus. Loop all online cpus and use smp_call_on_cpu() to call this SEAMCALL on the first cpu of each package.

To keep things simple, this implementation takes no affirmative steps to online cpus to make sure there's at least one cpu for each package. The callers (aka. KVM) can ensure success by ensuring sufficient CPUs are online for this to succeed.

Intel hardware doesn't guarantee cache coherency across different KeyIDs. The PAMTs are transitioning from being used by the kernel mapping (KeyId 0) to the TDX module's "global KeyID" mapping.

This means that the kernel must flush any dirty KeyID-0 PAMT cachelines before the TDX module uses the global KeyID to access the PAMTs. Otherwise, if those dirty cachelines were written back, they would corrupt the TDX module's metadata. Aside: This corruption would be detected by the memory integrity hardware on the next read of the memory with the global KeyID. The result would likely be fatal to the system but would not impact TDX security.

Following the TDX module specification, flush cache before configuring the global KeyID on all packages. Given the PAMT size can be large (~1/256th of system RAM), just use WBINVD on all CPUs to flush.

If TDH.SYS.KEY.CONFIG fails, the TDX module may already have used the global KeyID to write the PAMTs. Therefore, use WBINVD to flush cache before returning the PAMTs back to the kernel. Also convert all PAMTs back to normal by using MOVDIR64B as suggested by the TDX module spec, although on the platform without the "partial write machine check" erratum it's OK to leave PAMTs as is.

---

v12 -> v13:
- Added Yuan's tag.

v11 -> v12:
- Added Kirill's tag
- Improved changelog (Nikolay)

v10 -> v11:
- Convert PAMTs back to normal when module initialization fails.
- Fixed an error in changelog

v9 -> v10:
- Changed to use 'smp_call_on_cpu()' directly to do key configuration.

v8 -> v9:
- Improved changelog (Dave).
- Improved comments to explain the function to configure global KeyID "takes no affirmative action to online any cpu". (Dave).
- Improved other comments suggested by Dave.

v7 -> v8: (Dave)
- Changelog changes:
- Point out this is the step of "multi-steps" of init_tdx_module().
- Removed MOVDIR64B part.
- Other changes due to removing TDH.SYS.SHUTDOWN and TDH.SYS.LP.INIT.
- Changed to loop over online cpus and use smp_call_function_single() directly as the patch to shut down TDX module has been removed.
- Removed MOVDIR64B part in comment.

v6 -> v7:
- Improved changelong and comment to explain why MOVDIR64B isn't used when returning PAMTs back to the kernel.

---
 arch/x86/virt/vmx/tdx/tdx.c | 130 +++++++++++++++++++++++++++++++++++-
 arch/x86/virt/vmx/tdx/tdx.h |   1 +
 2 files changed, 129 insertions(+), 2 deletions(-)

index d059226e8829..2e9fffc35f7e 100644
@@ -30,6 +30,7 @@
 #include <asm/msr-index.h>
 #include <asm/msr.h>
 #include <asm/page.h>
+#include <asm/special_insns.h>
 #include <asm/tdx.h>
 #include "tdx.h"
 

@@ -613,7 +614,8 @@ static void tdmr_get_pamt(struct tdmr_info *tdmr, unsigned long *pamt_base,
 	*pamt_size = pamt_sz;
 }
 
-static void tdmr_free_pamt(struct tdmr_info *tdmr)
+static void tdmr_do_pamt_func(struct tdmr_info *tdmr,
+		void (*pamt_func)(unsigned long base, unsigned long size))
 {
 	unsigned long pamt_base, pamt_size;
 

@@ -626,9 +628,19 @@ static void tdmr_free_pamt(struct tdmr_info *tdmr)
 	if (WARN_ON_ONCE(!pamt_base))
 		return;
 
+	(*pamt_func)(pamt_base, pamt_size);
+}
+
+static void free_pamt(unsigned long pamt_base, unsigned long pamt_size)
+{
 	free_contig_range(pamt_base >> PAGE_SHIFT, pamt_size >> PAGE_SHIFT);
 }
 
+static void tdmr_free_pamt(struct tdmr_info *tdmr)
+{
+	tdmr_do_pamt_func(tdmr, free_pamt);
+}
+
 static void tdmrs_free_pamt_all(struct tdmr_info_list *tdmr_list)
 {
 	int i;

@@ -657,6 +669,41 @@ static int tdmrs_set_up_pamt_all(struct tdmr_info_list *tdmr_list,
 	return ret;
 }
 
+/*
+ * Convert TDX private pages back to normal by using MOVDIR64B to
+ * clear these pages.  Note this function doesn't flush cache of
+ * these TDX private pages.  The caller should make sure of that.
+ */
+static void reset_tdx_pages(unsigned long base, unsigned long size)
+{
+	const void *zero_page = (const void *)page_address(ZERO_PAGE(0));
+	unsigned long phys, end;
+
+	end = base + size;
+	for (phys = base; phys < end; phys += 64)
+		movdir64b(__va(phys), zero_page);
+
+	/*
+	 * MOVDIR64B uses WC protocol.  Use memory barrier to
+	 * make sure any later user of these pages sees the
+	 * updated data.
+	 */
+	mb();
+}
+
+static void tdmr_reset_pamt(struct tdmr_info *tdmr)
+{
+	tdmr_do_pamt_func(tdmr, reset_tdx_pages);
+}
+
+static void tdmrs_reset_pamt_all(struct tdmr_info_list *tdmr_list)
+{
+	int i;
+
+	for (i = 0; i < tdmr_list->nr_consumed_tdmrs; i++)
+		tdmr_reset_pamt(tdmr_entry(tdmr_list, i));
+}
+
 static unsigned long tdmrs_count_pamt_kb(struct tdmr_info_list *tdmr_list)
 {
 	unsigned long pamt_size = 0;

@@ -936,6 +983,50 @@ static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid)
 	return ret;
 }
 
+static int do_global_key_config(void *data)
+{
+	struct tdx_module_args args = {};
+
+	return seamcall_prerr(TDH_SYS_KEY_CONFIG, &args);
+}
+
+/*
+ * Attempt to configure the global KeyID on all physical packages.
+ *
+ * This requires running code on at least one CPU in each package.  If a
+ * package has no online CPUs, that code will not run and TDX module
+ * initialization (TDMR initialization) will fail.
+ *
+ * This code takes no affirmative steps to online CPUs.  Callers (aka.
+ * KVM) can ensure success by ensuring sufficient CPUs are online for
+ * this to succeed.
+ */
+static int config_global_keyid(void)
+{
+	cpumask_var_t packages;
+	int cpu, ret = -EINVAL;
+
+	if (!zalloc_cpumask_var(&packages, GFP_KERNEL))
+		return -ENOMEM;
+
+	for_each_online_cpu(cpu) {
+		if (cpumask_test_and_set_cpu(topology_physical_package_id(cpu),
+					packages))
+			continue;
+
+		/*
+		 * TDH.SYS.KEY.CONFIG cannot run concurrently on
+		 * different cpus, so just do it one by one.
+		 */
+		ret = smp_call_on_cpu(cpu, do_global_key_config, NULL, true);
+		if (ret)
+			break;
+	}
+
+	free_cpumask_var(packages);
+	return ret;
+}
+
 static int init_tdx_module(void)
 {
 	struct tdsysinfo_struct *tdsysinfo;

@@ -996,15 +1087,47 @@ static int init_tdx_module(void)
 	if (ret)
 		goto out_free_pamts;
 
+	/*
+	 * Hardware doesn't guarantee cache coherency across different
+	 * KeyIDs.  The kernel needs to flush PAMT's dirty cachelines
+	 * (associated with KeyID 0) before the TDX module can use the
+	 * global KeyID to access the PAMT.  Given PAMTs are potentially
+	 * large (~1/256th of system RAM), just use WBINVD on all cpus
+	 * to flush the cache.
+	 */
+	wbinvd_on_all_cpus();
+
+	/* Config the key of global KeyID on all packages */
+	ret = config_global_keyid();
+	if (ret)
+		goto out_reset_pamts;
+
 	/*
 	 * TODO:
 	 *
-	 *  - Configure the global KeyID on all packages.
 	 *  - Initialize all TDMRs.
 	 *
 	 *  Return error before all steps are done.
 	 */
 	ret = -EINVAL;
+out_reset_pamts:
+	if (ret) {
+		/*
+		 * Part of PAMTs may already have been initialized by the
+		 * TDX module.  Flush cache before returning PAMTs back
+		 * to the kernel.
+		 */
+		wbinvd_on_all_cpus();
+		/*
+		 * According to the TDX hardware spec, if the platform
+		 * doesn't have the "partial write machine check"
+		 * erratum, any kernel read/write will never cause #MC
+		 * in kernel space, thus it's OK to not convert PAMTs
+		 * back to normal.  But do the conversion anyway here
+		 * as suggested by the TDX spec.
+		 */
+		tdmrs_reset_pamt_all(&tdmr_list);
+	}
 out_free_pamts:
 	if (ret)
 		tdmrs_free_pamt_all(&tdmr_list);

@@ -1060,6 +1183,9 @@ static int __tdx_enable(void)
  * lock to prevent any new cpu from becoming online; 2) done both VMXON
  * and tdx_cpu_enable() on all online cpus.
  *
+ * This function requires there's at least one online cpu for each CPU
+ * package to succeed.
+ *
  * This function can be called in parallel by multiple callers.
  *
  * Return 0 if TDX is enabled successfully, otherwise error.

index b8c9e3d016f9..2427ae40fc3c 100644
@@ -16,6 +16,7 @@
 /*
  * TDX module SEAMCALL leaf functions
  */
+#define TDH_SYS_KEY_CONFIG	31
 #define TDH_SYS_INFO		32
 #define TDH_SYS_INIT		33
 #define TDH_SYS_LP_INIT		35

2.41.0

After the global KeyID has been configured on all packages, initialize all TDMRs to make all TDX-usable memory regions that are passed to the TDX module become usable.

This is the last step of initializing the TDX module.

Initializing TDMRs can be time consuming on large memory systems as it involves initializing all metadata entries for all pages that can be used by TDX guests. Initializing different TDMRs can be parallelized. For now to keep it simple, just initialize all TDMRs one by one. It can be enhanced in the future.

---

v12 -> v13:
- Added Yuan's tag.

v11 -> v12:
- Added Kirill's tag

v10 -> v11:
- No update

v9 -> v10:
- Code change due to change static 'tdx_tdmr_list' to local 'tdmr_list'.

v8 -> v9:
- Improved changlog to explain why initializing TDMRs can take long time (Dave).
- Improved comments around 'next-to-initialize' address (Dave).

v7 -> v8: (Dave)
- Changelog:
- explicitly call out this is the last step of TDX module initialization.
- Trimed down changelog by removing SEAMCALL name and details.
- Removed/trimmed down unnecessary comments.
- Other changes due to 'struct tdmr_info_list'.

v6 -> v7:
- Removed need_resched() check. -- Andi.

---
 arch/x86/virt/vmx/tdx/tdx.c | 60 ++++++++++++++++++++++++++++++++-----
 arch/x86/virt/vmx/tdx/tdx.h |  1 +
 2 files changed, 53 insertions(+), 8 deletions(-)

index 2e9fffc35f7e..b410fe66fd74 100644
@@ -1027,6 +1027,56 @@ static int config_global_keyid(void)
 	return ret;
 }
 
+static int init_tdmr(struct tdmr_info *tdmr)
+{
+	u64 next;
+
+	/*
+	 * Initializing a TDMR can be time consuming.  To avoid long
+	 * SEAMCALLs, the TDX module may only initialize a part of the
+	 * TDMR in each call.
+	 */
+	do {
+		struct tdx_module_args args = {
+			.rcx = tdmr->base,
+		};
+		int ret;
+
+		ret = seamcall_prerr_ret(TDH_SYS_TDMR_INIT, &args);
+		if (ret)
+			return ret;
+		/*
+		 * RDX contains 'next-to-initialize' address if
+		 * TDH.SYS.TDMR.INIT did not fully complete and
+		 * should be retried.
+		 */
+		next = args.rdx;
+		cond_resched();
+		/* Keep making SEAMCALLs until the TDMR is done */
+	} while (next < tdmr->base + tdmr->size);
+
+	return 0;
+}
+
+static int init_tdmrs(struct tdmr_info_list *tdmr_list)
+{
+	int i;
+
+	/*
+	 * This operation is costly.  It can be parallelized,
+	 * but keep it simple for now.
+	 */
+	for (i = 0; i < tdmr_list->nr_consumed_tdmrs; i++) {
+		int ret;
+
+		ret = init_tdmr(tdmr_entry(tdmr_list, i));
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
+
 static int init_tdx_module(void)
 {
 	struct tdsysinfo_struct *tdsysinfo;

@@ -1102,14 +1152,8 @@ static int init_tdx_module(void)
 	if (ret)
 		goto out_reset_pamts;
 
-	/*
-	 * TODO:
-	 *
-	 *  - Initialize all TDMRs.
-	 *
-	 *  Return error before all steps are done.
-	 */
-	ret = -EINVAL;
+	/* Initialize TDMRs to complete the TDX module initialization */
+	ret = init_tdmrs(&tdmr_list);
 out_reset_pamts:
 	if (ret) {
 		/*

index 2427ae40fc3c..6e41b0731e48 100644
@@ -20,6 +20,7 @@
 #define TDH_SYS_INFO		32
 #define TDH_SYS_INIT		33
 #define TDH_SYS_LP_INIT		35
+#define TDH_SYS_TDMR_INIT	36
 #define TDH_SYS_CONFIG		45
 
 struct cmr_info {

2.41.0

There are two problems in terms of using kexec() to boot to a new kernel when the old kernel has enabled TDX: 1) Part of the memory pages are still TDX private pages; 2) There might be dirty cachelines associated with TDX private pages.

The first problem doesn't matter on the platforms w/o the "partial write machine check" erratum. KeyID 0 doesn't have integrity check. If the new kernel wants to use any non-zero KeyID, it needs to convert the memory to that KeyID and such conversion would work from any KeyID.

However the old kernel needs to guarantee there's no dirty cacheline left behind before booting to the new kernel to avoid silent corruption from later cacheline writeback (Intel hardware doesn't guarantee cache coherency across different KeyIDs).

There are two things that the old kernel needs to do to achieve that:

1) Stop accessing TDX private memory mappings:
a. Stop making TDX module SEAMCALLs (TDX global KeyID); b. Stop TDX guests from running (per-guest TDX KeyID).
2) Flush any cachelines from previous TDX private KeyID writes.

For 2), use wbinvd() to flush cache in stop_this_cpu(), following SME support. And in this way 1) happens for free as there's no TDX activity between wbinvd() and the native_halt().

Flushing cache in stop_this_cpu() only flushes cache on remote cpus. On the rebooting cpu which does kexec(), unlike SME which does the cache flush in relocate_kernel(), flush the cache right after stopping remote cpus in machine_shutdown().

There are two reasons to do so: 1) For TDX there's no need to defer cache flush to relocate_kernel() because all TDX activities have been stopped. 2) On the platforms with the above erratum the kernel must convert all TDX private pages back to normal before booting to the new kernel in kexec(), and flushing cache early allows the kernel to convert memory early rather than having to muck with the relocate_kernel() assembly.

Theoretically, cache flush is only needed when the TDX module has been initialized. However initializing the TDX module is done on demand at runtime, and it takes a mutex to read the module status. Just check whether TDX is enabled by the BIOS instead to flush cache.

---
 arch/x86/kernel/process.c |  8 +++++++-
 arch/x86/kernel/reboot.c  | 15 +++++++++++++++
 2 files changed, 22 insertions(+), 1 deletion(-)

index 72015dba72ab..7e85bd9e5f15 100644
@@ -811,8 +811,14 @@ void __noreturn stop_this_cpu(void *dummy)
 	 *
 	 * Test the CPUID bit directly because the machine might've cleared
 	 * X86_FEATURE_SME due to cmdline options.
+	 *
+	 * The TDX module or guests might have left dirty cachelines
+	 * behind.  Flush them to avoid corruption from later writeback.
+	 * Note that this flushes on all systems where TDX is possible,
+	 * but does not actually check that TDX was in use.
 	 */
-	if (c->extended_cpuid_level >= 0x8000001f && (cpuid_eax(0x8000001f) & BIT(0)))
+	if ((c->extended_cpuid_level >= 0x8000001f && (cpuid_eax(0x8000001f) & BIT(0)))
+			|| platform_tdx_enabled())
 		native_wbinvd();
 
 	/*

index 3adbe97015c1..ae7480a213a6 100644
@@ -32,6 +32,7 @@
 #include <asm/realmode.h>
 #include <asm/x86_init.h>
 #include <asm/efi.h>
+#include <asm/tdx.h>
 
 /*
  * Power off function, if any

@@ -695,6 +696,20 @@ void native_machine_shutdown(void)
 	local_irq_disable();
 	stop_other_cpus();
 #endif
+	/*
+	 * stop_other_cpus() has flushed all dirty cachelines of TDX
+	 * private memory on remote cpus.  Unlike SME, which does the
+	 * cache flush on _this_ cpu in the relocate_kernel(), flush
+	 * the cache for _this_ cpu here.  This is because on the
+	 * platforms with "partial write machine check" erratum the
+	 * kernel needs to convert all TDX private pages back to normal
+	 * before booting to the new kernel in kexec(), and the cache
+	 * flush must be done before that.  If the kernel took SME's way,
+	 * it would have to muck with the relocate_kernel() assembly to
+	 * do memory conversion.
+	 */
+	if (platform_tdx_enabled())
+		native_wbinvd();
 
 	lapic_shutdown();
 	restore_boot_irq_mode();

2.41.0

Does TDX support hibernate? I'm wondering about two potential problems:
1. Reading/writing private pages from the direct map on save/restore
2. The seam module needing to be re-inited (the tdx_enable() stuff)

If that's the case you could have something like the below to just block it when TDX could be in use:

index 2b4a946a6ff5..3b1b7202452d 100644
@@ -84,7 +84,8 @@ bool hibernation_available(void)
 {
        return nohibernate == 0 &&
                !security_locked_down(LOCKDOWN_HIBERNATION) &&
-               !secretmem_active() && !cxl_mem_active();
+               !secretmem_active() && !cxl_mem_active() &&
+               !platform_tdx_enabled();
 }
 
 /**

Or maybe better, it could check tdx_module_status? But there is no way to read that variable from hibernate.

No.

There's a whole bunch of volatile state that's generated inside the CPU and never leaves the CPU, like the ephemeral key that protects TDX module memory.

SGX, for instance, never even supported suspend, IIRC. Enclaves just die and have to be rebuilt.

Right. AFAICT TDX cannot survive from S3 either. All TDX keys get lost when system enters S3. However I don't think TDX can be rebuilt after resume like SGX. Let me confirm with TDX guys on this.

I think we can register syscore_ops->suspend for TDX, and refuse to suspend when TDX is enabled. This covers hibernate case too.

In terms of how to check "TDX is enabled", ideally it's better to check whether TDX module is actually initialized, but the worst case is we can use platform_tdx_enabled(). (I need to think more on this)

Hi Dave, Kirill, Rick,

Is this solution overall acceptable?

By "rebuilt" I mean all private data is totally destroyed and rebuilt from scratch. The SGX architecture provides zero help other than delivering a fault and saying: "whoops all your data is gone".

*Ideally* the firmware would have a choke point where it could just tell the OS that it can't suspend rather than the OS having to figure it out.

Right. For TDX I am worrying about SEAMCALL could poison memory thus could trigger #MC inside kernel, or even could trigger #MC inside SEAM, instead of delivering a fault that SGX app/kernel can handle. I am confirming with TDX team.

Agreed. Let me ask TDX team about this too.

On the platforms with the "partial write machine check" erratum, the kexec() needs to convert all TDX private pages back to normal before booting to the new kernel. Otherwise, the new kernel may get unexpected machine check.

There's no existing infrastructure to track TDX private pages. Change to keep TDMRs when module initialization is successful so that they can be used to find PAMTs.

---

v12 -> v13:
- Split "improve error handling" part out as a separate patch.

v11 -> v12 (new patch):
- Defer keeping TDMRs logic to this patch for better review
- Improved error handling logic (Nikolay/Kirill in patch 15)

---
 arch/x86/virt/vmx/tdx/tdx.c | 24 +++++++++++-------------
 1 file changed, 11 insertions(+), 13 deletions(-)

index b410fe66fd74..ea1363ceaa28 100644
@@ -132,6 +132,8 @@ static DEFINE_MUTEX(tdx_module_lock);
 /* All TDX-usable memory regions.  Protected by mem_hotplug_lock. */
 static LIST_HEAD(tdx_memlist);
 
+static struct tdmr_info_list tdx_tdmr_list;
+
 /*
  * Do the module global initialization if not done yet.  It can be
  * done on any cpu.  It's always called with interrupts disabled.

@@ -1080,7 +1082,6 @@ static int init_tdmrs(struct tdmr_info_list *tdmr_list)
 static int init_tdx_module(void)
 {
 	struct tdsysinfo_struct *tdsysinfo;
-	struct tdmr_info_list tdmr_list;
 	struct cmr_info *cmr_array;
 	int tdsysinfo_size;
 	int cmr_array_size;

@@ -1123,17 +1124,17 @@ static int init_tdx_module(void)
 		goto out_put_tdxmem;
 
 	/* Allocate enough space for constructing TDMRs */
-	ret = alloc_tdmr_list(&tdmr_list, tdsysinfo);
+	ret = alloc_tdmr_list(&tdx_tdmr_list, tdsysinfo);
 	if (ret)
 		goto out_free_tdxmem;
 
 	/* Cover all TDX-usable memory regions in TDMRs */
-	ret = construct_tdmrs(&tdx_memlist, &tdmr_list, tdsysinfo);
+	ret = construct_tdmrs(&tdx_memlist, &tdx_tdmr_list, tdsysinfo);
 	if (ret)
 		goto out_free_tdmrs;
 
 	/* Pass the TDMRs and the global KeyID to the TDX module */
-	ret = config_tdx_module(&tdmr_list, tdx_global_keyid);
+	ret = config_tdx_module(&tdx_tdmr_list, tdx_global_keyid);
 	if (ret)
 		goto out_free_pamts;
 

@@ -1153,7 +1154,7 @@ static int init_tdx_module(void)
 		goto out_reset_pamts;
 
 	/* Initialize TDMRs to complete the TDX module initialization */
-	ret = init_tdmrs(&tdmr_list);
+	ret = init_tdmrs(&tdx_tdmr_list);
 out_reset_pamts:
 	if (ret) {
 		/*

@@ -1170,20 +1171,17 @@ static int init_tdx_module(void)
 		 * back to normal.  But do the conversion anyway here
 		 * as suggested by the TDX spec.
 		 */
-		tdmrs_reset_pamt_all(&tdmr_list);
+		tdmrs_reset_pamt_all(&tdx_tdmr_list);
 	}
 out_free_pamts:
 	if (ret)
-		tdmrs_free_pamt_all(&tdmr_list);
+		tdmrs_free_pamt_all(&tdx_tdmr_list);
 	else
 		pr_info("%lu KBs allocated for PAMT.\n",
-				tdmrs_count_pamt_kb(&tdmr_list));
+				tdmrs_count_pamt_kb(&tdx_tdmr_list));
 out_free_tdmrs:
-	/*
-	 * Always free the buffer of TDMRs as they are only used during
-	 * module initialization.
-	 */
-	free_tdmr_list(&tdmr_list);
+	if (ret)
+		free_tdmr_list(&tdx_tdmr_list);
 out_free_tdxmem:
 	if (ret)
 		free_tdx_memlist(&tdx_memlist);

2.41.0

With keeping TDMRs upon successful TDX module initialization, now only put_online_mems() and freeing the buffers of the TDSYSINFO_STRUCT and the CMR array still need to be done even when module initialization is successful. On the other hand, all other four "out_*" labels before them explicitly check the return value and only clean up when module initialization fails.

This isn't ideal. Make all other four "out_*" labels only reachable when module initialization fails to improve the readibility of error handling. Rename them from "out_*" to "err_*" to reflect the fact.

---

v12 -> v13:
- New patch to improve error handling. (Kirill, Nikolay)

---
 arch/x86/virt/vmx/tdx/tdx.c | 67 +++++++++++++++++++------------------
 1 file changed, 34 insertions(+), 33 deletions(-)

index ea1363ceaa28..8ee9f94c0fa7 100644
@@ -1126,17 +1126,17 @@ static int init_tdx_module(void)
 	/* Allocate enough space for constructing TDMRs */
 	ret = alloc_tdmr_list(&tdx_tdmr_list, tdsysinfo);
 	if (ret)
-		goto out_free_tdxmem;
+		goto err_free_tdxmem;
 
 	/* Cover all TDX-usable memory regions in TDMRs */
 	ret = construct_tdmrs(&tdx_memlist, &tdx_tdmr_list, tdsysinfo);
 	if (ret)
-		goto out_free_tdmrs;
+		goto err_free_tdmrs;
 
 	/* Pass the TDMRs and the global KeyID to the TDX module */
 	ret = config_tdx_module(&tdx_tdmr_list, tdx_global_keyid);
 	if (ret)
-		goto out_free_pamts;
+		goto err_free_pamts;
 
 	/*
 	 * Hardware doesn't guarantee cache coherency across different

@@ -1151,40 +1151,16 @@ static int init_tdx_module(void)
 	/* Config the key of global KeyID on all packages */
 	ret = config_global_keyid();
 	if (ret)
-		goto out_reset_pamts;
+		goto err_reset_pamts;
 
 	/* Initialize TDMRs to complete the TDX module initialization */
 	ret = init_tdmrs(&tdx_tdmr_list);
-out_reset_pamts:
-	if (ret) {
-		/*
-		 * Part of PAMTs may already have been initialized by the
-		 * TDX module.  Flush cache before returning PAMTs back
-		 * to the kernel.
-		 */
-		wbinvd_on_all_cpus();
-		/*
-		 * According to the TDX hardware spec, if the platform
-		 * doesn't have the "partial write machine check"
-		 * erratum, any kernel read/write will never cause #MC
-		 * in kernel space, thus it's OK to not convert PAMTs
-		 * back to normal.  But do the conversion anyway here
-		 * as suggested by the TDX spec.
-		 */
-		tdmrs_reset_pamt_all(&tdx_tdmr_list);
-	}
-out_free_pamts:
 	if (ret)
-		tdmrs_free_pamt_all(&tdx_tdmr_list);
-	else
-		pr_info("%lu KBs allocated for PAMT.\n",
-				tdmrs_count_pamt_kb(&tdx_tdmr_list));
-out_free_tdmrs:
-	if (ret)
-		free_tdmr_list(&tdx_tdmr_list);
-out_free_tdxmem:
-	if (ret)
-		free_tdx_memlist(&tdx_memlist);
+		goto err_reset_pamts;
+
+	pr_info("%lu KBs allocated for PAMT.\n",
+			tdmrs_count_pamt_kb(&tdx_tdmr_list));
+
 out_put_tdxmem:
 	/*
 	 * @tdx_memlist is written here and read at memory hotplug time.

@@ -1199,6 +1175,31 @@ static int init_tdx_module(void)
 	kfree(tdsysinfo);
 	kfree(cmr_array);
 	return ret;
+
+err_reset_pamts:
+	/*
+	 * Part of PAMTs may already have been initialized by the
+	 * TDX module.  Flush cache before returning PAMTs back
+	 * to the kernel.
+	 */
+	wbinvd_on_all_cpus();
+	/*
+	 * According to the TDX hardware spec, if the platform
+	 * doesn't have the "partial write machine check"
+	 * erratum, any kernel read/write will never cause #MC
+	 * in kernel space, thus it's OK to not convert PAMTs
+	 * back to normal.  But do the conversion anyway here
+	 * as suggested by the TDX spec.
+	 */
+	tdmrs_reset_pamt_all(&tdx_tdmr_list);
+err_free_pamts:
+	tdmrs_free_pamt_all(&tdx_tdmr_list);
+err_free_tdmrs:
+	free_tdmr_list(&tdx_tdmr_list);
+err_free_tdxmem:
+	free_tdx_memlist(&tdx_memlist);
+	/* Do things irrelevant to module initialization result */
+	goto out_put_tdxmem;
 }
 
 static int __tdx_enable(void)

2.41.0

^ Nit: "readability"

The first few generations of TDX hardware have an erratum. A partial write to a TDX private memory cacheline will silently "poison" the line. Subsequent reads will consume the poison and generate a machine check. According to the TDX hardware spec, neither of these things should have happened.

== Background ==

Virtually all kernel memory accesses operations happen in full cachelines. In practice, writing a "byte" of memory usually reads a 64 byte cacheline of memory, modifies it, then writes the whole line back. Those operations do not trigger this problem.

This problem is triggered by "partial" writes where a write transaction of less than cacheline lands at the memory controller. The CPU does these via non-temporal write instructions (like MOVNTI), or through UC/WC memory mappings. The issue can also be triggered away from the CPU by devices doing partial writes via DMA.

== Problem ==

A fast warm reset doesn't reset TDX private memory. Kexec() can also boot into the new kernel directly. Thus if the old kernel has enabled TDX on the platform with this erratum, the new kernel may get unexpected machine check.

Note that w/o this erratum any kernel read/write on TDX private memory should never cause machine check, thus it's OK for the old kernel to leave TDX private pages as is.

== Solution ==

In short, with this erratum, the kernel needs to explicitly convert all TDX private pages back to normal to give the new kernel a clean slate after kexec(). The BIOS is also expected to disable fast warm reset as a workaround to this erratum, thus this implementation doesn't try to reset TDX private memory for the reboot case in the kernel but depend on the BIOS to enable the workaround.

Convert TDX private pages back to normal after all remote cpus has been stopped and cache flush has been done on all cpus, when no more TDX activity can happen further. Do it in machine_kexec() to avoid the additional overhead to the normal reboot/shutdown as the kernel depends on the BIOS to disable fast warm reset for the reboot case.

For now TDX private memory can only be PAMT pages. It would be ideal to cover all types of TDX private memory here, but there are practical problems to do so:

1) There's no existing infrastructure to track TDX private pages;
2) It's not feasible to query the TDX module about page type because VMX has already been stopped when KVM receives the reboot notifier, plus the result from the TDX module may not be accurate (e.g., the remote CPU could be stopped right before MOVDIR64B).

One temporary solution is to blindly convert all memory pages, but it's problematic to do so too, because not all pages are mapped as writable in the direct mapping. It can be done by switching to the identical mapping created for kexec() or a new page table, but the complexity looks overkill.

Therefore, rather than doing something dramatic, only reset PAMT pages here. Other kernel components which use TDX can do the conversion on their own by intercepting the rebooting/shutdown notifier (KVM already does that).

---

v12 -> v13:
- Improve comments to explain why barrier is needed and ignore WBINVD. (Dave)
- Improve comments to document memory ordering. (Nikolay)
- Made comments/changelog slightly more concise.

v11 -> v12:
- Changed comment/changelog to say kernel doesn't try to handle fast warm reset but depends on BIOS to enable workaround (Kirill)
- Added a new tdx_may_has_private_mem to indicate system may have TDX private memory and PAMTs/TDMRs are stable to access. (Dave).
- Use atomic_t for tdx_may_has_private_mem for build-in memory barrier (Dave)
- Changed calling x86_platform.memory_shutdown() to calling tdx_reset_memory() directly from machine_kexec() to avoid overhead to normal reboot case.

v10 -> v11:
- New patch

---
 arch/x86/include/asm/tdx.h         |  2 +
 arch/x86/kernel/machine_kexec_64.c |  9 ++++
 arch/x86/virt/vmx/tdx/tdx.c        | 84 ++++++++++++++++++++++++++++++
 3 files changed, 95 insertions(+)

index fce7abc99bf5..97a68ced69dc 100644
@@ -113,10 +113,12 @@ u64 __seamcall_saved_ret(u64 fn, struct tdx_module_args *args);
 bool platform_tdx_enabled(void);
 int tdx_cpu_enable(void);
 int tdx_enable(void);
+void tdx_reset_memory(void);
 #else
 static inline bool platform_tdx_enabled(void) { return false; }
 static inline int tdx_cpu_enable(void) { return -ENODEV; }
 static inline int tdx_enable(void)  { return -ENODEV; }
+static inline void tdx_reset_memory(void) { }
 #endif	/* CONFIG_INTEL_TDX_HOST */
 
 #endif /* !__ASSEMBLY__ */

index 1a3e2c05a8a5..03d9689ef808 100644
@@ -28,6 +28,7 @@
 #include <asm/setup.h>
 #include <asm/set_memory.h>
 #include <asm/cpu.h>
+#include <asm/tdx.h>
 
 #ifdef CONFIG_ACPI
 /*

@@ -301,6 +302,14 @@ void machine_kexec(struct kimage *image)
 	void *control_page;
 	int save_ftrace_enabled;
 
+	/*
+	 * For platforms with TDX "partial write machine check" erratum,
+	 * all TDX private pages need to be converted back to normal
+	 * before booting to the new kernel, otherwise the new kernel
+	 * may get unexpected machine check.
+	 */
+	tdx_reset_memory();
+
 #ifdef CONFIG_KEXEC_JUMP
 	if (image->preserve_context)
 		save_processor_state();

2.41.0

Without a ton of knowledge on TDX arch stuff, I'm mostly looked at the kexec flow with respect to anything that might be tinkering with the PAMT. Everything there looked good to me.

But I'm wondering if you want to skip the tdx_reset_memory() in the KEXEC_JUMP/preserve_context case. Somehow (I'm not clear on all the details), kexec can be configured to have the new kernel jump back to the old kernel and resume execution as if nothing happened. Then I think you would want to keep the TDX data around. Does that make any sense?

Good point. Thanks!

Based on my understanding, it should be OK to skip tdx_reset_memory() (or better to) when preserve_context is on. The second kernel shouldn't touch first kernel's memory anyway otherwise it may corrupt the first kernel state (if it does this maliciously or accidentally, then the first kernel isn't guaranteed to work anyway).

In fact, if we do tdx_reset_memory() when preserve_memory is on, we will need to do additional things to mark TDX as dead otherwise after jumping back other kernel code will still believe TDX is alive and continue to use TDX.

I'll do this if I don't hear objection from other people.

Something like below?

b/arch/x86/kernel/machine_kexec_64.c
index 03d9689ef808..73ed01360408 100644

@@ -307,12 +307,18 @@ void machine_kexec(struct kimage *image)
         * all TDX private pages need to be converted back to normal
         * before booting to the new kernel, otherwise the new kernel
         * may get unexpected machine check.
+        *
+        * But skip this when preserve_context is on.  The second kernel
+        * shouldn't touch the first kernel's memory anyway.  Skipping
+        * this also avoids killing TDX in the first kernel, which would
+        * require more complicated handling.
         */
-       tdx_reset_memory();
-
 #ifdef CONFIG_KEXEC_JUMP
        if (image->preserve_context)
                save_processor_state();
+       else
+#else
+       tdx_reset_memory();
 #endif
 
 

I think it may read the memory, is it ok?

Not the most beautiful ifdeffery, I'd just duplicate the tdx_reset_memory() call. But not a strong opinion.