Skip to contentSkip to conversation

PCI/TSM: PCIe Link Encryption Establishment via TDX platform services

PATCH In review Xu Yilun 12 participants 153 messages Mar 27 – Jul 27

Subsystems (12)
x86 KVM ACPI PCI +8 more subsystems
lore
Share & tools
lore

Xu Yilun

author
#0 lore raw

v2 cover PCI/TSM: PCIe Link Encryption Establishment via TDX platform services

This series is based on mainline v7.0-rc2 and targets v7.1 (quite aggressive though). The merge path will be through tsm.git with tip.git acks where needed. I know there are several parallel series on the fly, so Dave you can wait for Dan to review, or ack/nak as you have time, thanks. No KVM change, no acks from kvm.git is needed.

== Overview ==

This series adds a PCI/TSM low-level driver implementation for TDX Connect (the TEE I/O architecture for Intel platforms). PCI/TSM is the Linux PCI core subsystem [1][2] that supports Link Encryption & trust establishment between CoCo-VM and assigned devices, allowing CoCo-VM to accept devices for private memory access (private DMA). This series only implements Link Encryption. It is a pre-requisite for trusted device assignment in TDX system.

Two protocols, SPDM (Security Protocol and Data Model) and PCI IDE (Integrity and Data Encryption) work together to establish the Link Encryption. SPDM creates trust on untrusted transit for key exchanging. IDE performs the actual real-time encryption for data traffic. In TSM world, they are managed by secure firmwares, e.g. TDX Module.

To manage these protocols, TDX Module introduces Extensions to support long running / hard-irq preemptible flows inside. Host invokes these flows via Extension-SEAMCALLs.

This series has 2 distinct parts:

Patches 1-13: TDX core cleanups and TDX Module Extensions enabling Patches 14-31: tdx_host TSM driver for PCIe Link Encryption

[1]: https://lore.kernel.org/linux-coco/[email protected]/
[2]: https://lore.kernel.org/linux-coco/[email protected]/

== Merge notes ==

- Merge conflicts with parallel series:
Sean's VMXON: https://lore.kernel.org/all/[email protected]/
Chao's runtime update: https://lore.kernel.org/all/[email protected]/

- Picked several patches from parallel series:
Patch 1: https://lore.kernel.org/all/[email protected]/
Patch 14: https://lore.kernel.org/all/[email protected]/
Patch 15: https://lore.kernel.org/all/[email protected]/

== Changelog ==
v2:
- Subject change. previously it was:
"PCI/TSM: TDX Connect: SPDM Session and IDE Establishment"
- Remove __free() for core TDX and refactor all tdx_ext functions
- Use kzalloc(PAGE_SIZE, ...) instead of alloc_page() in TDX core
- Check feature0 support before reading optional global metadata
- Split the TDX Module Extensions enabling into small patches
- Enable TDX Module Extensions along with Basic TDX enabling
- Refactor SEAMCALL version handling
- For tdx_page_array, make page allocation method configurable
- For TDX Module Extensions, use contiguous page allocation
- For IOMMU_MT, use a custom page allocation
- Print TDX Extensions memory usage
- Various Changelog & comments refine

v1: https://lore.kernel.org/all/[email protected]/
- No tdx_enable() needed in tdx-host
- Simplify tdx_page_array kAPI, no singleton mode input
- Refactor the handling of TDX_INTERRUPTED_RESUMABLE
- Refine the usage of scope-based cleanup in tdx-host
- Set nr_stream_id in tdx-host, not in PCI ACPI initialization
- Use KEYP table + ECAP bit50 to decide Domain ID reservation
- Refactor IDE Address Association Register setup
- Remove prototype patches
- Refactor tdx_enable_ext() locking because of Sean's change
- Pick ACPICA KEYP patch from ACPICA repo
- Select TDX Connect feature for TDH.SYS.CONFIG, remove temporary solution for TDH.SYS.INIT
- Use Rick's tdx_errno.h movement patch [6]
- Factor out scope-based cleanup patches in mm
- Remove redunant header files, add header files only when first used
- Use dev_err_probe() when possible
- keyp_info_match() refactor
- Use bitfield.h macros for PAGE_LIST_INFO & HPA_ARRAY_T raw value
- Remove reserved fields for spdm_config_info_t
- Simplify return for tdh_ide_stream_block()
- Other small fixes for Jonathan's comments

RFC: https://lore.kernel.org/linux-coco/[email protected]/

Chao Gao (1):
coco/tdx-host: Introduce a "tdx_host" device

Dan Williams (1):
PCI/TSM: Report active IDE streams per host bridge

Dave Jiang (1):
acpi: Add KEYP support to fw_table parsing

Kiryl Shutsemau (1):
x86/tdx: Move all TDX error defines into <asm/shared/tdx_errno.h>

Lu Baolu (2):
iommu/vt-d: Cache max domain ID to avoid redundant calculation iommu/vt-d: Reserve the MSB domain ID bit for the TDX module

Xu Yilun (21):
x86/virt/tdx: Move bit definitions of TDX_FEATURES0 to public header x86/virt/tdx: Add tdx_page_array helpers for new TDX Module objects x86/virt/tdx: Support allocating contiguous pages for tdx_page_array x86/virt/tdx: Extend tdx_page_array to support IOMMU_MT x86/virt/tdx: Read global metadata for TDX Module Extensions/Connect x86/virt/tdx: Embed version info in SEAMCALL leaf function definitions x86/virt/tdx: Configure TDX Module with optional TDX Connect feature x86/virt/tdx: Move tdx_clflush_page() up in the file x86/virt/tdx: Add extra memory to TDX Module for Extensions x86/virt/tdx: Make TDX Module initialize Extensions x86/virt/tdx: Enable the Extensions after basic TDX Module init x86/virt/tdx: Extend tdx_clflush_page() to handle compound pages coco/tdx-host: Support Link TSM for TDX host
x86/virt/tdx: Add a helper to loop on TDX_INTERRUPTED_RESUMABLE iommu/vt-d: Export a helper to do function for each dmar_drhd_unit coco/tdx-host: Setup all trusted IOMMUs on TDX Connect init
mm: Add __free() support for __free_page()
coco/tdx-host: Parse ACPI KEYP table to init IDE for PCI host bridges x86/virt/tdx: Add SEAMCALL wrappers for IDE stream management coco/tdx-host: Implement IDE stream setup/teardown coco/tdx-host: Finally enable SPDM session and IDE Establishment

Zhenzhong Duan (4):
x86/virt/tdx: Add SEAMCALL wrappers for trusted IOMMU setup and clear coco/tdx-host: Add a helper to exchange SPDM messages through DOE x86/virt/tdx: Add SEAMCALL wrappers for SPDM management coco/tdx-host: Implement SPDM session setup

 drivers/virt/coco/Kconfig                     |   2 +
 drivers/virt/coco/tdx-host/Kconfig            |  16 +
 drivers/virt/coco/Makefile                    |   1 +
 drivers/virt/coco/tdx-host/Makefile           |   1 +
 Documentation/ABI/testing/sysfs-class-tsm     |  13 +
 arch/x86/include/asm/shared/tdx.h             |   1 +
 .../vmx => include/asm/shared}/tdx_errno.h    |  30 +-
 arch/x86/include/asm/tdx.h                    |  95 +-
 arch/x86/include/asm/tdx_global_metadata.h    |  14 +
 arch/x86/kvm/vmx/tdx.h                        |   1 -
 arch/x86/virt/vmx/tdx/tdx.h                   |  42 +-
 drivers/iommu/intel/iommu.h                   |   2 +
 include/linux/acpi.h                          |   3 +
 include/linux/dmar.h                          |   2 +
 include/linux/fw_table.h                      |   1 +
 include/linux/gfp.h                           |   1 +
 include/linux/pci-ide.h                       |   4 +
 include/linux/tsm.h                           |   3 +
 arch/x86/virt/vmx/tdx/tdx.c                   | 839 ++++++++++++++-
 arch/x86/virt/vmx/tdx/tdx_global_metadata.c   |  36 +
 drivers/acpi/tables.c                         |  12 +-
 drivers/iommu/intel/dmar.c                    |  67 ++
 drivers/iommu/intel/iommu.c                   |  10 +-
 drivers/pci/ide.c                             |   9 +-
 drivers/virt/coco/tdx-host/tdx-host.c         | 952 ++++++++++++++++++
 drivers/virt/coco/tsm-core.c                  |  97 ++
 lib/fw_table.c                                |   9 +
 27 files changed, 2202 insertions(+), 61 deletions(-)
 create mode 100644 drivers/virt/coco/tdx-host/Kconfig
 create mode 100644 drivers/virt/coco/tdx-host/Makefile
 rename arch/x86/{kvm/vmx => include/asm/shared}/tdx_errno.h (61%)
 create mode 100644 drivers/virt/coco/tdx-host/tdx-host.c

base-commit: 11439c4635edd669ae435eec308f4ab8a0804808

Signature · 1 line
2.25.1
author
#1 lore raw

PATCH v2 1/31 x86/tdx: Move all TDX error defines into <asm/shared/tdx_errno.h>

From: "Kirill A. Shutemov" <[email protected]>

Today there are two separate locations where TDX error codes are defined:

         arch/x86/include/asm/tdx.h
         arch/x86/kvm/vmx/tdx_errno.h

They have some overlap that is already defined similarly. Reduce the duplication and prepare to introduce some helpers for these error codes in the central place by unifying them. Join them at:
asm/shared/tdx_errno.h
...and update the headers that contained the duplicated definitions to include the new unified header.

"asm/shared" is used for sharing TDX code between the early compressed code and the normal kernel code. While the compressed code for the guest doesn't use these error code header definitions today, it does make the types of calls that return the values they define. So place the defines in "shared" location so that it can, but leave such cleanups for future changes.

Also, adjust BITUL() -> _BITULL() to address 32 bit build errors after the move.

---
 arch/x86/include/asm/shared/tdx.h             |  1 +
 .../vmx => include/asm/shared}/tdx_errno.h    | 28 +++++++++++++++----
 arch/x86/include/asm/tdx.h                    | 21 --------------
 arch/x86/kvm/vmx/tdx.h                        |  1 -
 4 files changed, 23 insertions(+), 28 deletions(-)
 rename arch/x86/{kvm/vmx => include/asm/shared}/tdx_errno.h (64%)
arch/x86/include/asm/shared/tdx.h +1 -0
index 8bc074c8d7c6..6a1646fc2b2f 100644
@@ -4,6 +4,7 @@
 
 #include <linux/bits.h>
 #include <linux/types.h>
+#include <asm/shared/tdx_errno.h>
 
 #define TDX_HYPERCALL_STANDARD  0
 
arch/x86/include/asm/shared/tdx_errno.h renamed from arch/x86/kvm/vmx/tdx_errno.h +22 -6
similarity index 64%
rename from arch/x86/kvm/vmx/tdx_errno.h
rename to arch/x86/include/asm/shared/tdx_errno.h
index 6ff4672c4181..8bf6765cf082 100644
@@ -1,14 +1,15 @@
 /* SPDX-License-Identifier: GPL-2.0 */
-/* architectural status code for SEAMCALL */
-
-#ifndef __KVM_X86_TDX_ERRNO_H
-#define __KVM_X86_TDX_ERRNO_H
+#ifndef _ASM_X86_SHARED_TDX_ERRNO_H
+#define _ASM_X86_SHARED_TDX_ERRNO_H
+#include <asm/trapnr.h>
 
+/* Upper 32 bit of the TDX error code encodes the status */
 #define TDX_SEAMCALL_STATUS_MASK		0xFFFFFFFF00000000ULL
 
 /*
- * TDX SEAMCALL Status Codes (returned in RAX)
+ * TDX Status Codes (returned in RAX)
  */
+#define TDX_SUCCESS				0ULL
 #define TDX_NON_RECOVERABLE_VCPU		0x4000000100000000ULL
 #define TDX_NON_RECOVERABLE_TD			0x4000000200000000ULL
 #define TDX_NON_RECOVERABLE_TD_NON_ACCESSIBLE	0x6000000500000000ULL
@@ -17,6 +18,7 @@
 #define TDX_OPERAND_INVALID			0xC000010000000000ULL
 #define TDX_OPERAND_BUSY			0x8000020000000000ULL
 #define TDX_PREVIOUS_TLB_EPOCH_BUSY		0x8000020100000000ULL
+#define TDX_RND_NO_ENTROPY			0x8000020300000000ULL
 #define TDX_PAGE_METADATA_INCORRECT		0xC000030000000000ULL
 #define TDX_VCPU_NOT_ASSOCIATED			0x8000070200000000ULL
 #define TDX_KEY_GENERATION_FAILED		0x8000080000000000ULL
@@ -28,6 +30,20 @@
 #define TDX_EPT_ENTRY_STATE_INCORRECT		0xC0000B0D00000000ULL
 #define TDX_METADATA_FIELD_NOT_READABLE		0xC0000C0200000000ULL
 
+/*
+ * SW-defined error codes.
+ *
+ * Bits 47:40 == 0xFF indicate Reserved status code class that never used by
+ * TDX module.
+ */
+#define TDX_ERROR			_BITULL(63)
+#define TDX_NON_RECOVERABLE		_BITULL(62)
+#define TDX_SW_ERROR			(TDX_ERROR | GENMASK_ULL(47, 40))
+#define TDX_SEAMCALL_VMFAILINVALID	(TDX_SW_ERROR | _ULL(0xFFFF0000))
+
+#define TDX_SEAMCALL_GP			(TDX_SW_ERROR | X86_TRAP_GP)
+#define TDX_SEAMCALL_UD			(TDX_SW_ERROR | X86_TRAP_UD)
+
 /*
  * TDX module operand ID, appears in 31:0 part of error code as
  * detail information
@@ -37,4 +53,4 @@
 #define TDX_OPERAND_ID_SEPT			0x92
 #define TDX_OPERAND_ID_TD_EPOCH			0xa9
 
-#endif /* __KVM_X86_TDX_ERRNO_H */
+#endif /* _ASM_X86_SHARED_TDX_ERRNO_H */
arch/x86/include/asm/tdx.h +0 -21
index 6b338d7f01b7..e040e0467ae4 100644
@@ -9,29 +9,8 @@
 
 #include <asm/errno.h>
 #include <asm/ptrace.h>
-#include <asm/trapnr.h>
 #include <asm/shared/tdx.h>
 
-/*
- * SW-defined error codes.
- *
- * Bits 47:40 == 0xFF indicate Reserved status code class that never used by
- * TDX module.
- */
-#define TDX_ERROR			_BITUL(63)
-#define TDX_NON_RECOVERABLE		_BITUL(62)
-#define TDX_SW_ERROR			(TDX_ERROR | GENMASK_ULL(47, 40))
-#define TDX_SEAMCALL_VMFAILINVALID	(TDX_SW_ERROR | _UL(0xFFFF0000))
-
-#define TDX_SEAMCALL_GP			(TDX_SW_ERROR | X86_TRAP_GP)
-#define TDX_SEAMCALL_UD			(TDX_SW_ERROR | X86_TRAP_UD)
-
-/*
- * TDX module SEAMCALL leaf function error codes
- */
-#define TDX_SUCCESS		0ULL
-#define TDX_RND_NO_ENTROPY	0x8000020300000000ULL
-
 #ifndef __ASSEMBLER__
 
 #include <uapi/asm/mce.h>
arch/x86/kvm/vmx/tdx.h +0 -1
index 45b5183ccb36..ce2720a028ad 100644
@@ -3,7 +3,6 @@
 #define __KVM_X86_VMX_TDX_H
 
 #include "tdx_arch.h"
-#include "tdx_errno.h"
 
 #ifdef CONFIG_KVM_INTEL_TDX
 #include "common.h"
Signature · 1 line
2.25.1
#2 lore raw

Re: [PATCH v2 01/31] x86/tdx: Move all TDX error defines into <asm/shared/tdx_errno.h>

Quoted 20 lines · Xu Yilun

From: "Kirill A. Shutemov" <[email protected]>

Today there are two separate locations where TDX error codes are defined:
         arch/x86/include/asm/tdx.h arch/x86/kvm/vmx/tdx_errno.h

They have some overlap that is already defined similarly. Reduce the duplication and prepare to introduce some helpers for these error codes in the central place by unifying them. Join them at:
        asm/shared/tdx_errno.h
...and update the headers that contained the duplicated definitions to include the new unified header.

"asm/shared" is used for sharing TDX code between the early compressed code and the normal kernel code. While the compressed code for the guest doesn't use these error code header definitions today, it does make the types of calls that return the values they define. So place the defines in "shared" location so that it can, but leave such cleanups for future changes.

Quoted 8 lines

Also, adjust BITUL() -> _BITULL() to address 32 bit build errors after the move.

I think it is missing Kai's RB from sys disable v1, and your sign off.

This patch is in three series now, hence the long SOB chain. And I think it is also the only KVM touch point in the series. If sys disable gets merged ahead of time it can be dropped. But if there is any lag there we should see if Dave will just take it instead of trying to keep it in sync.

#3 lore raw

Re: [PATCH v2 01/31] x86/tdx: Move all TDX error defines into <asm/shared/tdx_errno.h>

Quoted 30 lines · Rick P
Quoted 20 lines · Xu Yilun

From: "Kirill A. Shutemov" <[email protected]>

Today there are two separate locations where TDX error codes are defined:
         arch/x86/include/asm/tdx.h arch/x86/kvm/vmx/tdx_errno.h

They have some overlap that is already defined similarly. Reduce the duplication and prepare to introduce some helpers for these error codes in the central place by unifying them. Join them at:
        asm/shared/tdx_errno.h
...and update the headers that contained the duplicated definitions to include the new unified header.

"asm/shared" is used for sharing TDX code between the early compressed code and the normal kernel code. While the compressed code for the guest doesn't use these error code header definitions today, it does make the types of calls that return the values they define. So place the defines in "shared" location so that it can, but leave such cleanups for future changes.

Quoted 8 lines

Also, adjust BITUL() -> _BITULL() to address 32 bit build errors after the move.

Quoted 6 lines

I think it is missing Kai's RB from sys disable v1, and your sign off.

This patch is in three series now, hence the long SOB chain. And I think it is also the only KVM touch point in the series. If sys disable gets merged ahead of time it can be dropped. But if there is any lag there we should see if Dave will just take it instead of trying to keep it in sync.

Thanks for the heads up.

I think if we are committed to the idea that some TDX related sets may go through different upstreams and some of those sets have a small handful of common infrastructure patches then there are a few options. Either have someone keep an eye for these and publish stable-commits for folks to share, accept that duplication collisions will happen and rebase when they do, or accept that duplication collisions be ok with that small bit of mess showing up in the history.

In this case, for the tsm.git#staging branch, I will replace this with a fresh application of this:

https://lore.kernel.org/all/[email protected]/

Yilun, going forward, if you borrow a patch from another set, be sure to both add your own signed-off-by, but also a:

...to make it extra clear you are including a patch that is already on the list in another set.

I suspect that by the time this set is ready to move from tsm.git#staging to tsm.git#next a stable commit-id may be available for a rebase.

author
#4 lore raw

Re: [PATCH v2 01/31] x86/tdx: Move all TDX error defines into <asm/shared/tdx_errno.h>

Quoted 4 lines

In this case, for the tsm.git#staging branch, I will replace this with a fresh application of this:

https://lore.kernel.org/all/[email protected]/

This is the exact patch I picked, but Kai's RB is in previous version

https://lore.kernel.org/all/[email protected]/

and b4 didn't catch it. I think should add his RB manually.

Quoted 8 lines

Yilun, going forward, if you borrow a patch from another set, be sure to both add your own signed-off-by, but also a:

...to make it extra clear you are including a patch that is already on the list in another set.

Yes. I listed the link in cover letter, but yes add it here should be clearer and necessary.

Quoted 4 lines

I suspect that by the time this set is ready to move from tsm.git#staging to tsm.git#next a stable commit-id may be available for a rebase.

author
#5 lore raw

Re: [PATCH v2 01/31] x86/tdx: Move all TDX error defines into <asm/shared/tdx_errno.h>

In reply to Edgecombe, Rick P
Quoted 1 line

I think it is missing Kai's RB from sys disable v1, and your sign off.

Ah, yes. I picked from the latest post:

https://lore.kernel.org/all/[email protected]/

But didn't realize the older thread has different RB set and b4 didn't catch, will add it manually.

I'll also add my sign off.

Thanks

#6 lore raw

Re: [PATCH v2 01/31] x86/tdx: Move all TDX error defines into <asm/shared/tdx_errno.h>

In reply to Xu Yilun
Quoted 5 lines · Xu Yilun

This is missing a SoB from the person sending the patch.

Honestly, for something coming all from one company, I'd just prefer that folks simplify the tale of woe about how a patch came to be written. Just axe the SoB's and put one on it. Credit folks with a blurb in the changelog if you have to.

author
#7 lore raw

PATCH v2 2/31 x86/virt/tdx: Move bit definitions of TDX_FEATURES0 to public header

In reply to Xu Yilun

Move bit definitions of TDX_FEATURES0 to TDX core public header.

Kernel users get TDX_FEATURES0 bitmap via tdx_get_sysinfo(). It is reasonable to also public the definitions of each bit. TDX Connect (a new TDX feature to enable Trusted I/O virtualization) will add new bits and check them in separate kernel modules.

Take the opportunity to change its type to BIT_ULL since TDX_FEATURES0 is explicitly defined as 64-bit in both TDX Module Specification and TDX core code.

---
 arch/x86/include/asm/tdx.h  | 4 ++++
 arch/x86/virt/vmx/tdx/tdx.h | 3 ---
 2 files changed, 4 insertions(+), 3 deletions(-)
arch/x86/include/asm/tdx.h +4 -0
index e040e0467ae4..65c4da396450 100644
@@ -127,6 +127,10 @@ static __always_inline u64 sc_retry(sc_func_t func, u64 fn,
 int tdx_cpu_enable(void);
 int tdx_enable(void);
 const char *tdx_dump_mce_info(struct mce *m);
+
+/* Bit definitions of TDX_FEATURES0 metadata field */
+#define TDX_FEATURES0_NO_RBP_MOD	BIT_ULL(18)
+
 const struct tdx_sys_info *tdx_get_sysinfo(void);
 
 int tdx_guest_keyid_alloc(void);
arch/x86/virt/vmx/tdx/tdx.h +0 -3
index 82bb82be8567..c641b4632826 100644
@@ -84,9 +84,6 @@ struct tdmr_info {
 	DECLARE_FLEX_ARRAY(struct tdmr_reserved_area, reserved_areas);
 } __packed __aligned(TDMR_INFO_ALIGNMENT);
 
-/* Bit definitions of TDX_FEATURES0 metadata field */
-#define TDX_FEATURES0_NO_RBP_MOD	BIT(18)
-
 /*
  * Do not put any hardware-defined TDX structure representations below
  * this comment!
Signature · 1 line
2.25.1
#8 lore raw

Re: [PATCH v2 02/31] x86/virt/tdx: Move bit definitions of TDX_FEATURES0 to public header

Quoted 1 line · Xu Yilun

Move bit definitions of TDX_FEATURES0 to TDX core public header.

Patch seems reasonable, but not sure about the "public header" language. Maybe more widely accessibly header? Or a better name?

Quoted 12 lines

Kernel users get TDX_FEATURES0 bitmap via tdx_get_sysinfo(). It is reasonable to also public the definitions of each bit. TDX Connect (a new TDX feature to enable Trusted I/O virtualization) will add new bits and check them in separate kernel modules.

Take the opportunity to change its type to BIT_ULL since TDX_FEATURES0 is explicitly defined as 64-bit in both TDX Module Specification and TDX core code.

---

author
#9 lore raw

Re: [PATCH v2 02/31] x86/virt/tdx: Move bit definitions of TDX_FEATURES0 to public header

Quoted 6 lines · Rick P
Quoted 1 line · Xu Yilun

Move bit definitions of TDX_FEATURES0 to TDX core public header.

Patch seems reasonable, but not sure about the "public header" language. Maybe more widely accessibly header? Or a better name?

Yes, I try to make it more explicit, is it better:

x86/virt/tdx: Move TDX_FEATURES0 bit defines to arch x86 header

Move TDX_FEATURES0 bit definitions to arch x86 header.

...

author
#10 lore raw

PATCH v2 3/31 x86/virt/tdx: Add tdx_page_array helpers for new TDX Module objects

In reply to Xu Yilun

Add struct tdx_page_array definition for new TDX Module object types - HPA_ARRAY_T and HPA_LIST_INFO. They are used as input/output parameters in newly defined SEAMCALLs. Also define some helpers to allocate, setup and free tdx_page_array.

HPA_ARRAY_T and HPA_LIST_INFO are similar in most aspects. They both represent a list of pages for TDX Module accessing. There are several use cases for these 2 structures:

- As SEAMCALL inputs. They are claimed by TDX Module as control pages. Control pages are private pages for TDX Module to hold its internal control structures or private data. TDR, TDCS, TDVPR... are existing control pages, just not added via tdx_page_array.
- As SEAMCALL outputs. They were TDX Module control pages and now are released.
- As SEAMCALL inputs. They are just temporary buffers for exchanging data blobs in one SEAMCALL. TDX Module will not hold them for long time.

The 2 structures both need a 'root page' which contains a list of HPAs. They collapse the HPA of the root page and the number of valid HPAs into a 64 bit raw value for SEAMCALL parameters. The root page is always a medium for passing data pages, TDX Module never keeps the root page.

A main difference is HPA_ARRAY_T requires singleton mode when containing just 1 functional page (page0). In this mode the root page is not needed and the HPA field of the raw value directly points to the page0. But in this patch, root page is always allocated for user friendly kAPIs.

Another small difference is HPA_LIST_INFO contains a 'first entry' field which could be filled by TDX Module. This simplifies host by providing the same structure when re-invoke the interrupted SEAMCALL. No need for host to touch this field.

Typical usages of the tdx_page_array:

1. Add control pages:
- struct tdx_page_array *array = tdx_page_array_create(nr_pages);
- seamcall(TDH_XXX_CREATE, array, ...);

2. Release control pages:
- seamcall(TDX_XXX_DELETE, array, &nr_released, &released_hpa);
- tdx_page_array_ctrl_release(array, nr_released, released_hpa);

3. Exchange data blobs:
- struct tdx_page_array *array = tdx_page_array_create(nr_pages);
- seamcall(TDX_XXX, array, ...);
- Read data from array.
- tdx_page_array_free(array);

4. Note the root page contains 512 HPAs at most, if more pages are required, re-populate the tdx_page_array is needed.

- struct tdx_page_array *array = tdx_page_array_alloc(nr_pages);
- for each 512-page bulk
- tdx_page_array_populate(array, offset);
- seamcall(TDH_XXX_ADD, array, ...);

In case 2, SEAMCALLs output the released page array in the form of HPA_ARRAY_T or PAGE_LIST_INFO. Use tdx_page_array_ctrl_release() to check if the output pages match the original input pages. If failed, TDX Module is buggy. In this case the safer way is to leak the control pages, call tdx_page_array_ctrl_leak().

The usage of tdx_page_array will be in following patches.

---
 arch/x86/include/asm/tdx.h  |  37 +++++
 arch/x86/virt/vmx/tdx/tdx.c | 299 ++++++++++++++++++++++++++++++++++++
 2 files changed, 336 insertions(+)
arch/x86/include/asm/tdx.h +37 -0
index 65c4da396450..9173a432b312 100644
@@ -139,6 +139,43 @@ void tdx_guest_keyid_free(unsigned int keyid);
 
 void tdx_quirk_reset_page(struct page *page);
 
+/**
+ * struct tdx_page_array - Represents a list of pages for TDX Module access
+ * @nr_pages: Total number of data pages in the collection
+ * @pages: Array of data page pointers containing all the data
+ *
+ * @offset: Internal: The starting index in @pages, positions the currently
+ *	    populated page window in @root.
+ * @nents: Internal: Number of valid HPAs for the page window in @root
+ * @root: Internal: A single 4KB page holding the 8-byte HPAs of the page
+ *	  window. The page window max size is constrained by the root page,
+ *	  which is 512 HPAs.
+ *
+ * This structure abstracts several TDX Module defined object types, e.g.,
+ * HPA_ARRAY_T and HPA_LIST_INFO. Typically they all use a "root page" as the
+ * medium to exchange a list of data pages between host and TDX Module. This
+ * structure serves as a unified parameter type for SEAMCALL wrappers, where
+ * these hardware object types are needed.
+ */
+struct tdx_page_array {
+	/* public: */
+	unsigned int nr_pages;
+	struct page **pages;
+
+	/* private: */
+	unsigned int offset;
+	unsigned int nents;
+	u64 *root;
+};
+
+void tdx_page_array_free(struct tdx_page_array *array);
+DEFINE_FREE(tdx_page_array_free, struct tdx_page_array *, if (_T) tdx_page_array_free(_T))
+struct tdx_page_array *tdx_page_array_create(unsigned int nr_pages);
+void tdx_page_array_ctrl_leak(struct tdx_page_array *array);
+int tdx_page_array_ctrl_release(struct tdx_page_array *array,
+				unsigned int nr_released,
+				u64 released_hpa);
+
 struct tdx_td {
 	/* TD root structure: */
 	struct page *tdr_page;
arch/x86/virt/vmx/tdx/tdx.c +299 -0

311 lines not on this page · raw message

Signature · 1 line
2.25.1
#11 lore raw

Re: [PATCH v2 03/31] x86/virt/tdx: Add tdx_page_array helpers for new TDX Module objects

Hi,

In general I'm struggling to understand the design decisions. It seems a very specific design and quite a bit of code to manage an array of pages. Questions below.

Quoted 2 lines · Xu Yilun

Add struct tdx_page_array definition for new TDX Module object types - HPA_ARRAY_T and HPA_LIST_INFO.

This is unfortunate. I see you agree in the comments.

Quoted 9 lines

They are used as input/output
parameters in newly defined SEAMCALLs. Also define some helpers to allocate, setup and free tdx_page_array.

HPA_ARRAY_T and HPA_LIST_INFO are similar in most aspects. They both represent a list of pages for TDX Module accessing. There are several use cases for these 2 structures:

Quoted 9 lines

- As SEAMCALL inputs. They are claimed by TDX Module as control pages. Control pages are private pages for TDX Module to hold its internal control structures or private data. TDR, TDCS, TDVPR... are existing control pages, just not added via tdx_page_array.
- As SEAMCALL outputs. They were TDX Module control pages and now are released.
- As SEAMCALL inputs. They are just temporary buffers for exchanging data blobs in one SEAMCALL. TDX Module will not hold them for long time.

This is kind of verbose for what it seems to be trying to say. It's just that these types can be input or output params. The TDX module could hold on to the pages for a long time, or just transiently. For that latter part I think you are trying to say sometimes they need flushing and sometimes they don't?

Quoted 7 lines

The 2 structures both need a 'root page' which contains a list of HPAs. They collapse the HPA of the root page and the number of valid HPAs into a 64 bit raw value for SEAMCALL parameters. The root page is always a medium for passing data pages, TDX Module never keeps the root page.

Quoted 5 lines

A main difference is HPA_ARRAY_T requires singleton mode when containing just 1 functional page (page0). In this mode the root page is not needed and the HPA field of the raw value directly points to the page0. But in this patch, root page is always allocated for user friendly kAPIs.

"singleton mode"? What is it? If it's the case of not needing populate loop, it probably deserves more explanation. I'm not sure, but the populate loop seems to drive a lot of the struct design?

Quoted 5 lines

Another small difference is HPA_LIST_INFO contains a 'first entry' field which could be filled by TDX Module. This simplifies host by providing the same structure when re-invoke the interrupted SEAMCALL. No need for host to touch this field.

Not clear what this is talking about. But I'm starting to wonder if we should be so bold to claim that the differences between the types really simplify the host.

Quoted 10 lines

Typical usages of the tdx_page_array:

1. Add control pages:
- struct tdx_page_array *array = tdx_page_array_create(nr_pages);
- seamcall(TDH_XXX_CREATE, array, ...);

2. Release control pages:
- seamcall(TDX_XXX_DELETE, array, &nr_released, &released_hpa);
- tdx_page_array_ctrl_release(array, nr_released, released_hpa);

So release is mostly needed because of the need to do the wbvind seamcall? And unlike tdx_page_array_free() it returns an error in case that fails. Or other sanity checking. But all the callers do the same thing on error, call tdx_page_array_ctrl_leak().

Just wondering if we could simplify it somehow. There are two helpers and the caller has to know which one to call based on SEAMCALL specifics. What if the seamcall wrapper set a bit in the page array while passing it out. The bit would specify to the helper if it needs to do wbinvd or not. Then the wrappers could encapsulate the type of free needed and not rely on the caller to know. And we only need to have one function for it instead of two.

BTW, do we expect errors from the tdh_phymem_page_wbinvd_hkid() calls here? How could the BUSY happen? If we don't think it can happen in normal runtime, we could just warn and skip the special leak logic. In KVM side there is a place where we can't really handle it for the wbinvd calls. And one where we can. If we need a ton of code to handle a bug somewhere (on kernel side or TDX module), it seems too defensive to me. At least it's not in sync with the rest of TDX.

Especially the quite large tdx_page_array_validate_release() logic should need a justification that there is something very tricky that needs all this checking.

But maybe you can explain what the special risk is.

Quoted 56 lines

3. Exchange data blobs:
- struct tdx_page_array *array = tdx_page_array_create(nr_pages);
- seamcall(TDX_XXX, array, ...);
- Read data from array.

4. Note the root page contains 512 HPAs at most, if more pages are required, re-populate the tdx_page_array is needed.

- struct tdx_page_array *array = tdx_page_array_alloc(nr_pages);
- for each 512-page bulk
- tdx_page_array_populate(array, offset);
- seamcall(TDH_XXX_ADD, array, ...);

In case 2, SEAMCALLs output the released page array in the form of HPA_ARRAY_T or PAGE_LIST_INFO. Use tdx_page_array_ctrl_release() to check if the output pages match the original input pages. If failed, TDX Module is buggy. In this case the safer way is to leak the control pages, call tdx_page_array_ctrl_leak().

The usage of tdx_page_array will be in following patches.

---
 arch/x86/include/asm/tdx.h  |  37 +++++
 arch/x86/virt/vmx/tdx/tdx.c | 299 ++++++++++++++++++++++++++++++++++++
 2 files changed, 336 insertions(+)
arch/x86/include/asm/tdx.h +17 -0
index 65c4da396450..9173a432b312 100644
@@ -139,6 +139,43 @@ void tdx_guest_keyid_free(unsigned int keyid);
 
 void tdx_quirk_reset_page(struct page *page);
 
+/**
+ * struct tdx_page_array - Represents a list of pages for TDX Module access
+ * @nr_pages: Total number of data pages in the collection
+ * @pages: Array of data page pointers containing all the data
+ *
+ * @offset: Internal: The starting index in @pages, positions the currently
+ *	    populated page window in @root.
+ * @nents: Internal: Number of valid HPAs for the page window in @root
+ * @root: Internal: A single 4KB page holding the 8-byte HPAs of the page
+ *	  window. The page window max size is constrained by the root page,
+ *	  which is 512 HPAs.
+ *
+ * This structure abstracts several TDX Module defined object types, e.g.,
+ * HPA_ARRAY_T and HPA_LIST_INFO. Typically they all use a "root page" as the
+ * medium to exchange a list of data pages between host and TDX Module. This
+ * structure serves as a unified parameter type for SEAMCALL wrappers, where
+ * these hardware object types are needed.

arch/x86/include/asm/tdx.h void tdx_guest_keyid_free(unsigned int keyid); · line 168

Quoted 10 lines
arch/x86/include/asm/tdx.h +10 -0
+ */
+struct tdx_page_array {
+	/* public: */
+	unsigned int nr_pages;
+	struct page **pages;
+
+	/* private: */
+	unsigned int offset;
+	unsigned int nents;
+	u64 *root;

pages is going to be an array of struct pointers, and root is a single page of PA's that gets re-used to copy and pass the PA's to the TDX module. Why do we need both? Like just keep an array of PA's that would be the same size as the struct page array. And not need the populate loop?

Pausing for now. Still looking through the callers and it's the end of the day.

author
#12 lore raw

Re: [PATCH v2 03/31] x86/virt/tdx: Add tdx_page_array helpers for new TDX Module objects

Quoted 5 lines · Xu Yilun
Quoted 2 lines

Add struct tdx_page_array definition for new TDX Module object types - HPA_ARRAY_T and HPA_LIST_INFO.

This is unfortunate. I see you agree in the comments.

Yes, basically they are defining the same concept, behave mostly the same but some differences...

Quoted 20 lines
Quoted 10 lines

They are used as input/output
parameters in newly defined SEAMCALLs. Also define some helpers to allocate, setup and free tdx_page_array.

HPA_ARRAY_T and HPA_LIST_INFO are similar in most aspects. They both represent a list of pages for TDX Module accessing. There are several use cases for these 2 structures:

Quoted 9 lines

- As SEAMCALL inputs. They are claimed by TDX Module as control pages. Control pages are private pages for TDX Module to hold its internal control structures or private data. TDR, TDCS, TDVPR... are existing control pages, just not added via tdx_page_array.
- As SEAMCALL outputs. They were TDX Module control pages and now are released.
- As SEAMCALL inputs. They are just temporary buffers for exchanging data blobs in one SEAMCALL. TDX Module will not hold them for long time.

Quoted 1 line

This is kind of verbose for what it seems to be trying to say. It's just that

I assume if you feel the explanation of "what is control page" is off track. I added it cause the term firstly appears in x86 (only in KVM TDX previously), and people ask the definition:

https://lore.kernel.org/all/[email protected]/

Quoted 2 lines

these types can be input or output params. The TDX module could hold on to the pages for a long time, or just transiently.

Mm.. I'm trying to ramp up on the kernel API level flow:

For control pages, it would be hold by TDX Module long time, so host inputs the page array, later TDX Module outputs the page array back. Host need to verify the outputs.

For shared pages, TDX Module's accessing is transient in one SEAMCALL, so only as input, TDX Module never needs to output the array.

I think the verboseness makes the following pseudo code easier to understand.

Quoted 2 lines

For that latter part I think you are
trying to say sometimes they need flushing and sometimes they don't?

Yeah.
control pages => long term => host verifies and releases => flush on release shared pages => transient => no verify and releases => no flush

Maybe I should mention the flushing is already covered by releasing kAPI.

Quoted 8 lines
Quoted 8 lines

The 2 structures both need a 'root page' which contains a list of HPAs. They collapse the HPA of the root page and the number of valid HPAs into a 64 bit raw value for SEAMCALL parameters. The root page is always a medium for passing data pages, TDX Module never keeps the root page.

Quoted 7 lines
Quoted 5 lines

A main difference is HPA_ARRAY_T requires singleton mode when containing just 1 functional page (page0). In this mode the root page is not needed and the HPA field of the raw value directly points to the page0. But in this patch, root page is always allocated for user friendly kAPIs.

"singleton mode"? What is it? If it's the case of not needing populate loop, it

It is the SEAMCALL level detail for HPA_ARRAY_T. It is literally as explained above - the HPA field should be filled by page0, not root page.

Quoted 2 lines

probably deserves more explanation. I'm not sure, but the populate loop seems to drive a lot of the struct design?

The caller is not aware of singleton mode. Actually, I'm trying to make the tdx_page_array independent of HPA_ARRAY_T or HPA_LIST_INFO details when allocating/populating, root page is still populated even not needed for singleton mode. The differences only happen when collaping the struct into u64 SEAMCALL parameters.

Quoted 10 lines
Quoted 6 lines

Another small difference is HPA_LIST_INFO contains a 'first entry' field which could be filled by TDX Module. This simplifies host by providing the same structure when re-invoke the interrupted SEAMCALL. No need for host to touch this field.

Not clear what this is talking about. But I'm starting to wonder if we should be so bold to claim that the differences between the types really simplify the host.

I'm talking about another SEAMCALL level detail. Sometimes TDX Module got interrupted in the middle of page array processing, it needs an anchor to resuming from where it stops, TDX Module record the anchor in the 'first entry'.

By illustrating these SEAMCALL level differences, I want to explain they don't impact the general SW flow and kAPI cares about them internally.

Yes in POC code we do write dedicated code for each type, but it ends up with plenty of similar logics on caller side about root page manipulation. By now, the differences are not much, but I think we should not write copies for every type, we should stop new types.

Please allow me to stop here, will continue later...

Thanks.

#13 lore raw

Re: [PATCH v2 03/31] x86/virt/tdx: Add tdx_page_array helpers for new TDX Module objects

Quoted 29 lines · Xu Yilun
Quoted 5 lines · Xu Yilun
Quoted 2 lines

Add struct tdx_page_array definition for new TDX Module object types - HPA_ARRAY_T and HPA_LIST_INFO.

This is unfortunate. I see you agree in the comments.

Yes, basically they are defining the same concept, behave mostly the same but some differences...

Quoted 20 lines
Quoted 19 lines

They are used as input/output
parameters in newly defined SEAMCALLs. Also define some helpers to allocate, setup and free tdx_page_array.

HPA_ARRAY_T and HPA_LIST_INFO are similar in most aspects. They both represent a list of pages for TDX Module accessing. There are several use cases for these 2 structures:

 - As SEAMCALL inputs. They are claimed by TDX Module as control pages. Control pages are private pages for TDX Module to hold its internal control structures or private data. TDR, TDCS, TDVPR... are existing control pages, just not added via tdx_page_array.
 - As SEAMCALL outputs. They were TDX Module control pages and now are released.
 - As SEAMCALL inputs. They are just temporary buffers for exchanging data blobs in one SEAMCALL. TDX Module will not hold them for long time.

Quoted 8 lines
Quoted 2 lines

This is kind of verbose for what it seems to be trying to say. It's just that

I assume if you feel the explanation of "what is control page" is off track. I added it cause the term firstly appears in x86 (only in KVM TDX previously), and people ask the definition:

https://lore.kernel.org/all/[email protected]/

I meant it more generally.

Quoted 20 lines
Quoted 4 lines

these types can be input or output params. The TDX module could hold on to the
pages for a long time, or just transiently.

Mm.. I'm trying to ramp up on the kernel API level flow:

For control pages, it would be hold by TDX Module long time, so host inputs the page array, later TDX Module outputs the page array back. Host need to verify the outputs.

For shared pages, TDX Module's accessing is transient in one SEAMCALL, so only as input, TDX Module never needs to output the array.

I think the verboseness makes the following pseudo code easier to understand.

Quoted 2 lines

For that latter part I think you are
trying to say sometimes they need flushing and sometimes they don't?

Quoted 6 lines

Yeah.
control pages => long term => host verifies and releases => flush on release shared pages => transient => no verify and releases => no flush

Maybe I should mention the flushing is already covered by releasing kAPI.

I hear:
1. Long time vs short time
2. Accessed as private vs "shared"

I think (2) is the important point, right? Why does (1) matter?

Quoted 24 lines
Quoted 8 lines
Quoted 8 lines

The 2 structures both need a 'root page' which contains a list of HPAs. They collapse the HPA of the root page and the number of valid HPAs into a 64 bit raw value for SEAMCALL parameters. The root page is always a medium for passing data pages, TDX Module never keeps the root page.

Quoted 8 lines
Quoted 5 lines

A main difference is HPA_ARRAY_T requires singleton mode when containing just 1 functional page (page0). In this mode the root page is not needed and the HPA field of the raw value directly points to the page0. But in this patch, root page is always allocated for user friendly kAPIs.

"singleton mode"? What is it? If it's the case of not needing populate loop, it

It is the SEAMCALL level detail for HPA_ARRAY_T. It is literally as explained above - the HPA field should be filled by page0, not root page.

Quoted 3 lines

probably deserves more explanation. I'm not sure, but the populate loop seems to
drive a lot of the struct design?

Quoted 5 lines

The caller is not aware of singleton mode. Actually, I'm trying to make the tdx_page_array independent of HPA_ARRAY_T or HPA_LIST_INFO details when allocating/populating, root page is still populated even not needed for singleton mode. The differences only happen when collaping the struct into u64 SEAMCALL parameters.

It seems tdx_page_array combines two concepts. An array of pages, and the method that the pages get handed to the TDX module. What if we broke apart these concepts?

Quoted 18 lines
Quoted 8 lines
Quoted 7 lines

Another small difference is HPA_LIST_INFO contains a 'first entry' field which could be filled by TDX Module. This simplifies host by providing the same structure when re-invoke the interrupted SEAMCALL. No need for host to touch this field.

Quoted 4 lines

Not clear what this is talking about. But I'm starting to wonder if we should be
so bold to claim that the differences between the types really simplify the host.

I'm talking about another SEAMCALL level detail. Sometimes TDX Module got interrupted in the middle of page array processing, it needs an anchor to resuming from where it stops, TDX Module record the anchor in the 'first entry'.

Quoted 8 lines

By illustrating these SEAMCALL level differences, I want to explain they don't impact the general SW flow and kAPI cares about them internally.

Yes in POC code we do write dedicated code for each type, but it ends up with plenty of similar logics on caller side about root page manipulation. By now, the differences are not much, but I think we should not write copies for every type, we should stop new types.

Hmm, doesn't it seem like this is quickly becoming complicated though, to combine all the different types together? And it seems there are more coming that people want to add to this.

Quoted 4 lines

Please allow me to stop here, will continue later...

Thanks.

#14 lore raw

Re: [PATCH v2 03/31] x86/virt/tdx: Add tdx_page_array helpers for new TDX Module objects

Quoted 9 lines · Rick P
Quoted 4 lines · Xu Yilun

Yes in POC code we do write dedicated code for each type, but it ends up with plenty of similar logics on caller side about root page manipulation. By now, the differences are not much, but I think we should not write copies for every type, we should stop new types.

Hmm, doesn't it seem like this is quickly becoming complicated though, to combine all the different types together? And it seems there are more coming that people want to add to this.

Most of the list entry types are the same with just some bits unused.

Regards,

Tony

#15 lore raw

Re: [PATCH v2 03/31] x86/virt/tdx: Add tdx_page_array helpers for new TDX Module objects

In reply to Edgecombe, Rick P
Quoted 10 lines · Rick P
Quoted 5 lines · Xu Yilun

The caller is not aware of singleton mode. Actually, I'm trying to make the tdx_page_array independent of HPA_ARRAY_T or HPA_LIST_INFO details when allocating/populating, root page is still populated even not needed for singleton mode. The differences only happen when collaping the struct into u64 SEAMCALL parameters.

It seems tdx_page_array combines two concepts. An array of pages, and the method that the pages get handed to the TDX module. What if we broke apart these concepts?

We could add an enum for the LIST_INFO type to intialized the tdx_page_array?

Then the code using the tdx_page_array could initialize the root page based on the LIST_INFO type for the SEAMCALL.

Regards,

Tony

author
#16 lore raw

Re: [PATCH v2 03/31] x86/virt/tdx: Add tdx_page_array helpers for new TDX Module objects

In reply to Edgecombe, Rick P
Quoted 15 lines
Quoted 9 lines

Typical usages of the tdx_page_array:

1. Add control pages:
- struct tdx_page_array *array = tdx_page_array_create(nr_pages);
- seamcall(TDH_XXX_CREATE, array, ...);

2. Release control pages:
- seamcall(TDX_XXX_DELETE, array, &nr_released, &released_hpa);
- tdx_page_array_ctrl_release(array, nr_released, released_hpa);

So release is mostly needed because of the need to do the wbvind seamcall? And unlike tdx_page_array_free() it returns an error in case that fails. Or other sanity checking. But all the callers do the same thing on error, call tdx_page_array_ctrl_leak().

Quoted 6 lines

Just wondering if we could simplify it somehow. There are two helpers and the caller has to know which one to call based on SEAMCALL specifics. What if the seamcall wrapper set a bit in the page array while passing it out. The bit would specify to the helper if it needs to do wbinvd or not. Then the wrappers could encapsulate the type of free needed and not rely on the caller to know. And we only need to have one function for it instead of two.

I like the idea, I can have a try.

But we may need more than a bit to finish the release. On release some SEAMCALLs return the released HPA list and host checks they are sane, otherwise leak. We need to also record these info in the struct.

Quoted 3 lines

BTW, do we expect errors from the tdh_phymem_page_wbinvd_hkid() calls here? How

No, it can't happen in normal runtime.

Quoted 3 lines

could the BUSY happen? If we don't think it can happen in normal runtime, we could just warn and skip the special leak logic. In KVM side there is a place where we can't really handle it for the wbinvd calls. And one where we can. If

But we do leak control pages when wbinvd fails, or even when tdh_phymem_page_reclaim() fails. So anyway we need leak logic, is it? Is it a little insane if we failed to reclaim and return the private page to kernel?

Quoted 7 lines

we need a ton of code to handle a bug somewhere (on kernel side or TDX module), it seems too defensive to me. At least it's not in sync with the rest of TDX.

Especially the quite large tdx_page_array_validate_release() logic should need a justification that there is something very tricky that needs all this checking.

But maybe you can explain what the special risk is.

I don't see the special risk, actually I don't even see the releasing failed once.

But we do check the return value of tdh_phymem_page_reclaim() for a single ctrl page releasing. It is just we also check a list of ctrl pages releasing. The check becomes naturally complex, e.g., if the released number matches, if every HPA in the list matches ...

...

Quoted 6 lines
Quoted 6 lines
arch/x86/include/asm/tdx.h +6 -0
+struct tdx_page_array {
+	/* public: */
+	unsigned int nr_pages;
+	struct page **pages;
+
+	/* private: */
Quoted 8 lines

arch/x86/include/asm/tdx.h void tdx_guest_keyid_free(unsigned int keyid); · line 168

Quoted 3 lines
arch/x86/include/asm/tdx.h +3 -0
+	unsigned int offset;
+	unsigned int nents;
+	u64 *root;

pages is going to be an array of struct pointers, and root is a single page of PA's that gets re-used to copy and pass the PA's to the TDX module. Why do we need both? Like just keep an array of PA's that would be the same size as the struct page array. And not need the populate loop?

We need Linux language, struct page *, for alloc and free. Also need TDX Module language - PA list - for SEAMCALLs. So IIUC, the page to PA populating won't disappear on allocation, the PA to page populating would appear on free.

Besides, host may need to vmap and access the (shared) pages.

Thanks,
Yilun

#17 lore raw

Re: [PATCH v2 03/31] x86/virt/tdx: Add tdx_page_array helpers for new TDX Module objects

Quoted 8 lines · Xu Yilun
Quoted 7 lines

pages is going to be an array of struct pointers, and root is a single page of
PA's that gets re-used to copy and pass the PA's to the TDX module. Why do we
need both? Like just keep an array of PA's that would be the same size as the
struct page array. And not need the populate loop?

Quoted 4 lines

We need Linux language, struct page *, for alloc and free. Also need TDX Module language - PA list - for SEAMCALLs. So IIUC, the page to PA populating won't disappear on allocation, the PA to page populating would appear on free.

Not sure what you mean by this.

Quoted 2 lines

Besides, host may need to vmap and access the (shared) pages.

Some code someday may need to convert a PA to another format? Is that it? Doesn't seem like big problem.

But I'm not sure about this idea yet.

author
#18 lore raw

Re: [PATCH v2 03/31] x86/virt/tdx: Add tdx_page_array helpers for new TDX Module objects

Quoted 9 lines · Rick P
Quoted 8 lines · Xu Yilun
Quoted 7 lines

pages is going to be an array of struct pointers, and root is a single page of
PA's that gets re-used to copy and pass the PA's to the TDX module. Why do we
need both? Like just keep an array of PA's that would be the same size as the
struct page array. And not need the populate loop?

Quoted 6 lines
Quoted 4 lines

We need Linux language, struct page *, for alloc and free. Also need TDX Module language - PA list - for SEAMCALLs. So IIUC, the page to PA populating won't disappear on allocation, the PA to page populating would appear on free.

Not sure what you mean by this.

I mean host use struct page * for memory allocation and free. If we only keep the PA list, we still need to convert PAs back to struct page * and free.

Quoted 5 lines
Quoted 3 lines

Besides, host may need to vmap and access the (shared) pages.

Some code someday may need to convert a PA to another format? Is that it?

No, I don't convert a PA to something else. PA list is only for SEAMCALLs.

Now we use vm_map_ram(array->pages, ...) to map the shared pages, that's another reason I want to keep the struct page ** in struct tdx_page_array.

Anyway we use struct page ** for kernel memory management in several cases, keeping the struct page ** avoids PA -> page populating.

Quoted 3 lines

Doesn't seem like big problem.

But I'm not sure about this idea yet.

#19 lore raw

Re: [PATCH v2 03/31] x86/virt/tdx: Add tdx_page_array helpers for new TDX Module objects

In reply to Xu Yilun

On 27.03.26 г. 18:01 ч., Xu Yilun wrote:
<snip>

Quoted 9 lines
arch/x86/virt/vmx/tdx/tdx.c +9 -0
+/**
+ * tdx_page_array_ctrl_leak() - Leak data pages and free the container
+ * @array: The tdx_page_array to be leaked.
+ *
+ * Call this function when failed to reclaim the control pages. Free the root
+ * page and the holding structures, but orphan the data pages, to prevent the
+ * host from re-allocating and accessing memory that the hardware may still
+ * consider private.
+ */

arch/x86/virt/vmx/tdx/tdx.c static int build_tdx_memlist(struct list_head *tmb_list) · line 425

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +10 -0
+void tdx_page_array_ctrl_leak(struct tdx_page_array *array)
+{
+	if (!array)
+		return;
+
+	kfree(array->pages);
+	kfree(array->root);
+	kfree(array);
+}
+EXPORT_SYMBOL_GPL(tdx_page_array_ctrl_leak);

This instantly raises a red flag if by design an API has the ability to simply leak memory. Under what conditions this might be required, can't we do something to gracefully handle the case when pages cannot be freed instantly, i.e queued freeing or some such ? Simply leaking them is a big NO.

arch/x86/virt/vmx/tdx/tdx.c static int build_tdx_memlist(struct list_head *tmb_list) · line 428

Quoted 3 lines
arch/x86/virt/vmx/tdx/tdx.c +3 -0
+
+static bool tdx_page_array_validate_release(struct tdx_page_array *array,
+					    unsigned int offset,

This function is only ever called with offset of 0, if it's intended to be used later then I'd rather see this argument added in an explicit patch with rationale why it's needed.

Quoted 52 lines
arch/x86/virt/vmx/tdx/tdx.c +50 -0
+					    unsigned int nr_released,
+					    u64 released_hpa)
+{
+	unsigned int nents;
+
+	if (offset >= array->nr_pages)
+		return false;
+
+	nents = umin(array->nr_pages - offset, TDX_PAGE_ARRAY_MAX_NENTS);
+
+	if (nents != nr_released) {
+		pr_err("%s nr_released [%d] doesn't match page array nents [%d]\n",
+		       __func__, nr_released, nents);
+		return false;
+	}
+
+	/*
+	 * Unfortunately TDX has multiple page allocation protocols, check the
+	 * "singleton" case required for HPA_ARRAY_T.
+	 */
+	if (page_to_phys(array->pages[0]) == released_hpa &&
+	    array->nr_pages == 1)
+		return true;
+
+	/* Then check the "non-singleton" case */
+	if (virt_to_phys(array->root) == released_hpa) {
+		u64 *entries = array->root;
+		int i;
+
+		for (i = 0; i < nents; i++) {
+			struct page *page = array->pages[offset + i];
+			u64 val = page_to_phys(page);
+
+			if (val != entries[i]) {
+				pr_err("%s entry[%d] [0x%llx] doesn't match page hpa [0x%llx]\n",
+				       __func__, i, entries[i], val);
+				return false;
+			}
+		}
+
+		return true;
+	}
+
+	pr_err("%s failed to validate, released_hpa [0x%llx], root page hpa [0x%llx], page0 hpa [%#llx], number pages %u\n",
+	       __func__, released_hpa, virt_to_phys(array->root),
+	       page_to_phys(array->pages[0]), array->nr_pages);
+
+	return false;
+}
 
 
+<snip>
author
#20 lore raw

Re: [PATCH v2 03/31] x86/virt/tdx: Add tdx_page_array helpers for new TDX Module objects

Quoted 18 lines · Nikolay Borisov

On 27.03.26 г. 18:01 ч., Xu Yilun wrote:
<snip>

Quoted 13 lines
arch/x86/virt/vmx/tdx/tdx.c +13 -0
+/**
+ * tdx_page_array_ctrl_leak() - Leak data pages and free the container
+ * @array: The tdx_page_array to be leaked.
+ *
+ * Call this function when failed to reclaim the control pages. Free the root
+ * page and the holding structures, but orphan the data pages, to prevent the
+ * host from re-allocating and accessing memory that the hardware may still
+ * consider private.
+ */
+void tdx_page_array_ctrl_leak(struct tdx_page_array *array)
+{
+	if (!array)
+		return;
Quoted 8 lines

arch/x86/virt/vmx/tdx/tdx.c static int build_tdx_memlist(struct list_head *tmb_list) · line 425

Quoted 6 lines
arch/x86/virt/vmx/tdx/tdx.c +6 -0
+
+	kfree(array->pages);
+	kfree(array->root);
+	kfree(array);
+}
+EXPORT_SYMBOL_GPL(tdx_page_array_ctrl_leak);

This instantly raises a red flag if by design an API has the ability to

OK. With the discussion in this thread, I tend to remove this leak API.

Quoted 4 lines

simply leak memory. Under what conditions this might be required, can't we do something to gracefully handle the case when pages cannot be freed instantly, i.e queued freeing or some such ? Simply leaking them is a big NO.

It was intended to be called when failing to reclaim pages from secure firmware, maybe because of firmware bug. In this case kernel has no idea what to do. Leaking is a last resort here, don't expect things still work.

#21 lore raw

Re: [PATCH v2 03/31] x86/virt/tdx: Add tdx_page_array helpers for new TDX Module objects

In reply to Xu Yilun
Quoted 25 lines · Xu Yilun

Add struct tdx_page_array definition for new TDX Module object types - HPA_ARRAY_T and HPA_LIST_INFO. They are used as input/output parameters in newly defined SEAMCALLs. Also define some helpers to allocate, setup and free tdx_page_array.

HPA_ARRAY_T and HPA_LIST_INFO are similar in most aspects. They both represent a list of pages for TDX Module accessing. There are several use cases for these 2 structures:

- As SEAMCALL inputs. They are claimed by TDX Module as control pages. Control pages are private pages for TDX Module to hold its internal control structures or private data. TDR, TDCS, TDVPR... are existing control pages, just not added via tdx_page_array.
- As SEAMCALL outputs. They were TDX Module control pages and now are released.
- As SEAMCALL inputs. They are just temporary buffers for exchanging data blobs in one SEAMCALL. TDX Module will not hold them for long time.

The 2 structures both need a 'root page' which contains a list of HPAs. They collapse the HPA of the root page and the number of valid HPAs into a 64 bit raw value for SEAMCALL parameters. The root page is always a medium for passing data pages, TDX Module never keeps the root page.

Quoted 5 lines

A main difference is HPA_ARRAY_T requires singleton mode when containing just 1 functional page (page0). In this mode the root page is not needed and the HPA field of the raw value directly points to the page0. But in this patch, root page is always allocated for user friendly kAPIs.

I think this undersells the fact that "singleton mode" is a premature optimization that requires complication to take advantage of the benefit (sometimes save a single page allocation). The Linux implementation forfeits that small benefit for the larger gain of cleaner kAPIs.

Quoted 5 lines

Another small difference is HPA_LIST_INFO contains a 'first entry' field which could be filled by TDX Module. This simplifies host by providing the same structure when re-invoke the interrupted SEAMCALL. No need for host to touch this field.

Quoted 8 lines

Typical usages of the tdx_page_array:

1. Add control pages:
- struct tdx_page_array *array = tdx_page_array_create(nr_pages);
- seamcall(TDH_XXX_CREATE, array, ...);

2. Release control pages:
- seamcall(TDX_XXX_DELETE, array, &nr_released, &released_hpa);

It is simply a bug if TDH_XXX_DELETE does not return every resource passed to TDH_XXX_CREATE. The only "leak" case to worry about is that TDH_XXX_DELETE fails. In that case it should be "fatal" (TDX_BUG_ON, system can keep hobbling along, but panic_on_warn() would not be unreasonable). If TDH_XXX_DELETE fails it indicates some catastrophic misunderstanding between Linux and the TDX Module.

So the seamcall in this case has no need for @nr_released or @released_hpa, those should already be known to the kernel.

What is missing is an architectural guarantee that TDH_XXX_DELETE success == "all resources you arranged at TDH_XXX_CREATE time are free". I would hope that is already the case and AUX_PAGE_PA is only an unfortunate distraction. If it can ever be the case that CREATE and DELETE are asymmetric on success then that needs to be corrected and Linux will wait for a future module that can make that guarantee.

I think that cleans up a bulk of the logic here to abandon caring that the module tries to remind us what we are releasing.

author
#22 lore raw

Re: [PATCH v2 03/31] x86/virt/tdx: Add tdx_page_array helpers for new TDX Module objects

Quoted 32 lines
Quoted 5 lines

A main difference is HPA_ARRAY_T requires singleton mode when containing just 1 functional page (page0). In this mode the root page is not needed and the HPA field of the raw value directly points to the page0. But in this patch, root page is always allocated for user friendly kAPIs.

I think this undersells the fact that "singleton mode" is a premature optimization that requires complication to take advantage of the benefit (sometimes save a single page allocation). The Linux implementation forfeits that small benefit for the larger gain of cleaner kAPIs.

Quoted 5 lines

Another small difference is HPA_LIST_INFO contains a 'first entry' field which could be filled by TDX Module. This simplifies host by providing the same structure when re-invoke the interrupted SEAMCALL. No need for host to touch this field.

Quoted 8 lines

Typical usages of the tdx_page_array:

1. Add control pages:
- struct tdx_page_array *array = tdx_page_array_create(nr_pages);
- seamcall(TDH_XXX_CREATE, array, ...);

2. Release control pages:
- seamcall(TDX_XXX_DELETE, array, &nr_released, &released_hpa);

It is simply a bug if TDH_XXX_DELETE does not return every resource passed to TDH_XXX_CREATE. The only "leak" case to worry about is that TDH_XXX_DELETE fails. In that case it should be "fatal" (TDX_BUG_ON, system can keep hobbling along, but panic_on_warn() would not be unreasonable). If TDH_XXX_DELETE fails it indicates some catastrophic misunderstanding between Linux and the TDX Module.

Quoted 6 lines

So the seamcall in this case has no need for @nr_released or @released_hpa, those should already be known to the kernel.

What is missing is an architectural guarantee that TDH_XXX_DELETE success == "all resources you arranged at TDH_XXX_CREATE time are free". I would hope that is already the case and AUX_PAGE_PA is only an

Yes, it is already the case.

Quoted 3 lines

unfortunate distraction. If it can ever be the case that CREATE and DELETE are asymmetric on success then that needs to be corrected and Linux will wait for a future module that can make that guarantee.

Agree. Thanks for this summary as well as the singleton-mode one.

Quoted 3 lines

I think that cleans up a bulk of the logic here to abandon caring that the module tries to remind us what we are releasing.

Yes, I'll delete all these "released page matching" logic.

#23 lore raw

Re: [PATCH v2 03/31] x86/virt/tdx: Add tdx_page_array helpers for new TDX Module objects

In reply to Xu Yilun

Xu Yilun wrote:
[..]

Quoted 5 lines

The usage of tdx_page_array will be in following patches.

[..]

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +10 -0
+static struct tdx_page_array *
+tdx_page_array_alloc(unsigned int nr_pages)
+{
+	struct tdx_page_array *array = NULL;
+	struct page **pages = NULL;
+	u64 *root = NULL;
+	int ret;
+
+	if (!nr_pages)
+		return NULL;

arch/x86/virt/vmx/tdx/tdx.c static int build_tdx_memlist(struct list_head *tmb_list) · line 347

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +10 -0
+
+	array = kzalloc_obj(*array);
+	if (!array)
+		goto out_free;
+
+	root = kzalloc(PAGE_SIZE, GFP_KERNEL);
+	if (!root)
+		goto out_free;
+
+	pages = kcalloc(nr_pages, sizeof(*pages), GFP_KERNEL);

This should now be:

    kzalloc_objs(struct page *, nr_pages);

...oh nevermind you caught that in your incremental fixup. ...but a couple more comments below:

[..]

arch/x86/virt/vmx/tdx/tdx.c static int build_tdx_memlist(struct list_head *tmb_list) · line 528

Quoted 6 lines
arch/x86/virt/vmx/tdx/tdx.c +6 -0
+
+#define HPA_LIST_INFO_FIRST_ENTRY	GENMASK_U64(11, 3)
+#define HPA_LIST_INFO_PFN		GENMASK_U64(51, 12)
+#define HPA_LIST_INFO_LAST_ENTRY	GENMASK_U64(63, 55)
+
+static u64 __maybe_unused hpa_list_info_assign_raw(struct tdx_page_array *array)

2 quick comments:

* I do not understand shipping a __maybe_unused helper in patch 3 that does not get used until patch10.

* The "assign_raw" verb feels strange. I think this probably just want to be called: to_hpa_list_info(struct tdx_page_array *)

arch/x86/virt/vmx/tdx/tdx.c static int build_tdx_memlist(struct list_head *tmb_list) · line 539

Quoted 11 lines
arch/x86/virt/vmx/tdx/tdx.c +11 -0
+{
+	return FIELD_PREP(HPA_LIST_INFO_FIRST_ENTRY, 0) |
+	       FIELD_PREP(HPA_LIST_INFO_PFN,
+			  PFN_DOWN(virt_to_phys(array->root))) |
+	       FIELD_PREP(HPA_LIST_INFO_LAST_ENTRY, array->nents - 1);
+}
+
+#define HPA_ARRAY_T_PFN		GENMASK_U64(51, 12)
+#define HPA_ARRAY_T_SIZE	GENMASK_U64(63, 55)
+
+static u64 __maybe_unused hpa_array_t_assign_raw(struct tdx_page_array *array)

to_hpa_array_t()

arch/x86/virt/vmx/tdx/tdx.c static int build_tdx_memlist(struct list_head *tmb_list) · line 552

Quoted 13 lines
arch/x86/virt/vmx/tdx/tdx.c +13 -0
+{
+	unsigned long pfn;
+
+	if (array->nents == 1)
+		pfn = page_to_pfn(array->pages[array->offset]);
+	else
+		pfn = PFN_DOWN(virt_to_phys(array->root));
+
+	return FIELD_PREP(HPA_ARRAY_T_PFN, pfn) |
+	       FIELD_PREP(HPA_ARRAY_T_SIZE, array->nents - 1);
+}
+
+static u64 __maybe_unused hpa_array_t_release_raw(struct tdx_page_array *array)

It seems too subtle that this function sometimes returns zero and sometimes returns a page that the TDX module will clobber with data that we do not care about.

It is also not clear that "0" is what the module considers a valid value that meets "checks its validity for forward compatibility". I guess we get lucky because all of the calls that need this presently are multi-page cases?

I would feel better if this always returned the root HPA and was called something like:

to_output_clobber(), or to_aux_clobber()

...to make it clear that whatever was there before gets destroyed.

author
#24 lore raw

Re: [PATCH v2 03/31] x86/virt/tdx: Add tdx_page_array helpers for new TDX Module objects

Quoted 10 lines

arch/x86/virt/vmx/tdx/tdx.c static int build_tdx_memlist(struct list_head *tmb_list) · line 528

Quoted 5 lines
arch/x86/virt/vmx/tdx/tdx.c +5 -0
+#define HPA_LIST_INFO_FIRST_ENTRY	GENMASK_U64(11, 3)
+#define HPA_LIST_INFO_PFN		GENMASK_U64(51, 12)
+#define HPA_LIST_INFO_LAST_ENTRY	GENMASK_U64(63, 55)
+
+static u64 __maybe_unused hpa_list_info_assign_raw(struct tdx_page_array *array)

2 quick comments:

* I do not understand shipping a __maybe_unused helper in patch 3 that does not get used until patch10.

You once had a comment wanting to see how a tdx_page_array collapses to a 64-bit raw value for SEAMCALLs in the same patch. So I move the helpers earlier. Do you want to change them back?

Personally, I'd like to keep them here, to better align with the illustration in commit log about why we need the tdx_page_array.

Quoted 3 lines

* The "assign_raw" verb feels strange. I think this probably just want to be called: to_hpa_list_info(struct tdx_page_array *)

It's a better name, thanks.

[...]

Quoted 18 lines

arch/x86/virt/vmx/tdx/tdx.c static int build_tdx_memlist(struct list_head *tmb_list) · line 552

Quoted 13 lines
arch/x86/virt/vmx/tdx/tdx.c +13 -0
+{
+	unsigned long pfn;
+
+	if (array->nents == 1)
+		pfn = page_to_pfn(array->pages[array->offset]);
+	else
+		pfn = PFN_DOWN(virt_to_phys(array->root));
+
+	return FIELD_PREP(HPA_ARRAY_T_PFN, pfn) |
+	       FIELD_PREP(HPA_ARRAY_T_SIZE, array->nents - 1);
+}
+
+static u64 __maybe_unused hpa_array_t_release_raw(struct tdx_page_array *array)

It seems too subtle that this function sometimes returns zero and sometimes returns a page that the TDX module will clobber with data that we do not care about.

Quoted 2 lines

It is also not clear that "0" is what the module considers a valid value that meets "checks its validity for forward compatibility". I guess we

It is the TDX Module's requirement, which is 'too subtle'. TDX Module tries to keep align with the singleton definition for its output hpa_array_t. If TDX Module wants to output multiple released pages, it requires VMM to provide a root page HPA (in input register) so it can write HPA list on the root page. But if it outputs one released page, it directly writes page0 HPA in output register, and doesn't need a root page HPA in input register and enforce its value 0.

That's why we return 0 on singleton mode, otherwise a root page HPA.

Quoted 2 lines

get lucky because all of the calls that need this presently are multi-page cases?

Let me experiment, see if we have chance to simplify things.

Quoted 2 lines

I would feel better if this always returned the root HPA and was called

No, we can't. We must provide 0 for singleton mode. So I think maybe

to_hpa_array_t_released()

Anyway, Linux doesn't need the output hpa_array_t. I've already raised to Module team that don't enforce the medium page input. If VMM doesn't provide the page, don't bother fill it.

Quoted 6 lines

something like:

to_output_clobber(), or to_aux_clobber()

...to make it clear that whatever was there before gets destroyed.

author
#25 lore raw

PATCH v2 4/31 x86/virt/tdx: Support allocating contiguous pages for tdx_page_array

In reply to Xu Yilun

The current tdx_page_array implementation allocates scattered order-0 pages. However, some TDX Module operations benefit from contiguous physical memory. E.g. Enabling TDX Module Extensions (an optional TDX feature) requires ~50MB memory and never returns. Such allocation would at worst cause ~25GB permanently fragmented memory if each allocated page is from a different 2M region.

Support allocating contiguous pages for tdx_page_array by making the allocation method configurable. Change the tdx_page_array_alloc() to accept a custom allocation function pointer and a context parameter. Wrap the specific allocation into a tdx_page_array_alloc_contig() helper.

The foreseeable caller will allocate ~50MB memory with this helper, exceeding the maximum HPAs (512) a root page can hold, the typical usage will be:

- struct tdx_page_array *array = tdx_page_array_alloc_contig(nr_pages);
- for each 512-page bulk
- tdx_page_array_populate(array, offset);
- seamcall(TDH_XXX_ADD, array, ...);

The configurable allocation method would also benefit more tdx_page_array usages. TDX Module may require more specific memory layouts encoded in the root page. Will introduce them in following patches.

---
 arch/x86/virt/vmx/tdx/tdx.c | 42 +++++++++++++++++++++++++++++++++----
 1 file changed, 38 insertions(+), 4 deletions(-)
arch/x86/virt/vmx/tdx/tdx.c +38 -4
index a3021e7e2490..6c4ed80e8e5a 100644
@@ -289,7 +289,8 @@ static void tdx_free_pages_bulk(unsigned int nr_pages, struct page **pages)
 		__free_page(pages[i]);
 }
 
-static int tdx_alloc_pages_bulk(unsigned int nr_pages, struct page **pages)
+static int tdx_alloc_pages_bulk(unsigned int nr_pages, struct page **pages,
+				void *data)
 {
 	unsigned int filled, done = 0;
 
@@ -326,7 +327,10 @@ void tdx_page_array_free(struct tdx_page_array *array)
 EXPORT_SYMBOL_GPL(tdx_page_array_free);
 
 static struct tdx_page_array *
-tdx_page_array_alloc(unsigned int nr_pages)
+tdx_page_array_alloc(unsigned int nr_pages,
+		     int (*alloc_fn)(unsigned int nr_pages,
+				     struct page **pages, void *data),
+		     void *data)
 {
 	struct tdx_page_array *array = NULL;
 	struct page **pages = NULL;
@@ -348,7 +352,7 @@ tdx_page_array_alloc(unsigned int nr_pages)
 	if (!pages)
 		goto out_free;
 
-	ret = tdx_alloc_pages_bulk(nr_pages, pages);
+	ret = alloc_fn(nr_pages, pages, data);
 	if (ret)
 		goto out_free;
 
@@ -388,7 +392,7 @@ struct tdx_page_array *tdx_page_array_create(unsigned int nr_pages)
 	if (nr_pages > TDX_PAGE_ARRAY_MAX_NENTS)
 		return NULL;
 
-	array = tdx_page_array_alloc(nr_pages);
+	array = tdx_page_array_alloc(nr_pages, tdx_alloc_pages_bulk, NULL);
 	if (!array)
 		return NULL;
 
@@ -521,6 +525,36 @@ int tdx_page_array_ctrl_release(struct tdx_page_array *array,
 }
 EXPORT_SYMBOL_GPL(tdx_page_array_ctrl_release);
 
+static int tdx_alloc_pages_contig(unsigned int nr_pages, struct page **pages,
+				  void *data)
+{
+	struct page *page;
+	int i;
+
+	page = alloc_contig_pages(nr_pages, GFP_KERNEL, numa_mem_id(),
+				  &node_online_map);
+	if (!page)
+		return -ENOMEM;
+
+	for (i = 0; i < nr_pages; i++)
+		pages[i] = page + i;
+
+	return 0;
+}
+
+/*
+ * For holding large number of contiguous pages, usually larger than
+ * TDX_PAGE_ARRAY_MAX_NENTS (512).
+ *
+ * Similar to tdx_page_array_alloc(), after allocating with this
+ * function, call tdx_page_array_populate() to populate the tdx_page_array.
+ */
+static __maybe_unused struct tdx_page_array *
+tdx_page_array_alloc_contig(unsigned int nr_pages)
+{
+	return tdx_page_array_alloc(nr_pages, tdx_alloc_pages_contig, NULL);
+}
+
 #define HPA_LIST_INFO_FIRST_ENTRY	GENMASK_U64(11, 3)
 #define HPA_LIST_INFO_PFN		GENMASK_U64(51, 12)
 #define HPA_LIST_INFO_LAST_ENTRY	GENMASK_U64(63, 55)
Signature · 1 line
2.25.1
#26 lore raw

Re: [PATCH v2 04/31] x86/virt/tdx: Support allocating contiguous pages for tdx_page_array

Quoted 47 lines · Xu Yilun

The current tdx_page_array implementation allocates scattered order-0 pages. However, some TDX Module operations benefit from contiguous physical memory. E.g. Enabling TDX Module Extensions (an optional TDX feature) requires ~50MB memory and never returns. Such allocation would at worst cause ~25GB permanently fragmented memory if each allocated page is from a different 2M region.

Support allocating contiguous pages for tdx_page_array by making the allocation method configurable. Change the tdx_page_array_alloc() to accept a custom allocation function pointer and a context parameter. Wrap the specific allocation into a tdx_page_array_alloc_contig() helper.

The foreseeable caller will allocate ~50MB memory with this helper, exceeding the maximum HPAs (512) a root page can hold, the typical usage will be:

- struct tdx_page_array *array = tdx_page_array_alloc_contig(nr_pages);
- for each 512-page bulk
- tdx_page_array_populate(array, offset);
- seamcall(TDH_XXX_ADD, array, ...);

The configurable allocation method would also benefit more tdx_page_array usages. TDX Module may require more specific memory layouts encoded in the root page. Will introduce them in following patches.

---
  arch/x86/virt/vmx/tdx/tdx.c | 42 +++++++++++++++++++++++++++++++++----
  1 file changed, 38 insertions(+), 4 deletions(-)
arch/x86/virt/vmx/tdx/tdx.c +2 -1
index a3021e7e2490..6c4ed80e8e5a 100644
@@ -289,7 +289,8 @@ static void tdx_free_pages_bulk(unsigned int nr_pages, struct page **pages)
  		__free_page(pages[i]);
  }
 
-static int tdx_alloc_pages_bulk(unsigned int nr_pages, struct page **pages)
+static int tdx_alloc_pages_bulk(unsigned int nr_pages, struct page **pages,
+				void *data)
  {
  	unsigned int filled, done = 0;
 

arch/x86/virt/vmx/tdx/tdx.c void tdx_page_array_free(struct tdx_page_array *array) · line 333

Quoted 8 lines
arch/x86/virt/vmx/tdx/tdx.c +4 -1
@@ -326,7 +327,10 @@ void tdx_page_array_free(struct tdx_page_array *array)
  EXPORT_SYMBOL_GPL(tdx_page_array_free);
 
  static struct tdx_page_array *
-tdx_page_array_alloc(unsigned int nr_pages)
+tdx_page_array_alloc(unsigned int nr_pages,
+		     int (*alloc_fn)(unsigned int nr_pages,
+				     struct page **pages, void *data),
+		     void *data)

This interface seems cumbersome, since you will always have separate allocation paths:

Contig, Bulk and Iommu mt pages let's just keep them separate. I.e the flow should be:

1. Do common allocation by calling tdx_page_array_alloc (you aren't passing the alloc function) you just get a bare-bones tdx_page_array struct

2. Do the specific allocation in either :

tdx_page_array_create - for the bulk case
tdx_page_array_alloc_contig - for the contig case
tdx_page_array_create_iommu_mt - for the iommu case. Here you can open code tdx_alloc_pages_iommu_mt.

And keep the specific clearly separate in each function.

Quoted 58 lines
arch/x86/virt/vmx/tdx/tdx.c +32 -2
  {
  	struct tdx_page_array *array = NULL;
  	struct page **pages = NULL;
@@ -348,7 +352,7 @@ tdx_page_array_alloc(unsigned int nr_pages)
  	if (!pages)
  		goto out_free;
 
-	ret = tdx_alloc_pages_bulk(nr_pages, pages);
+	ret = alloc_fn(nr_pages, pages, data);
  	if (ret)
  		goto out_free;
 
@@ -388,7 +392,7 @@ struct tdx_page_array *tdx_page_array_create(unsigned int nr_pages)
  	if (nr_pages > TDX_PAGE_ARRAY_MAX_NENTS)
  		return NULL;
 
-	array = tdx_page_array_alloc(nr_pages);
+	array = tdx_page_array_alloc(nr_pages, tdx_alloc_pages_bulk, NULL);
  	if (!array)
  		return NULL;
 
@@ -521,6 +525,36 @@ int tdx_page_array_ctrl_release(struct tdx_page_array *array,
  }
  EXPORT_SYMBOL_GPL(tdx_page_array_ctrl_release);
 
+static int tdx_alloc_pages_contig(unsigned int nr_pages, struct page **pages,
+				  void *data)
+{
+	struct page *page;
+	int i;
+
+	page = alloc_contig_pages(nr_pages, GFP_KERNEL, numa_mem_id(),
+				  &node_online_map);
+	if (!page)
+		return -ENOMEM;
+
+	for (i = 0; i < nr_pages; i++)
+		pages[i] = page + i;
+
+	return 0;
+}
+
+/*
+ * For holding large number of contiguous pages, usually larger than
+ * TDX_PAGE_ARRAY_MAX_NENTS (512).
+ *
+ * Similar to tdx_page_array_alloc(), after allocating with this
+ * function, call tdx_page_array_populate() to populate the tdx_page_array.
+ */
+static __maybe_unused struct tdx_page_array *
+tdx_page_array_alloc_contig(unsigned int nr_pages)
+{
+	return tdx_page_array_alloc(nr_pages, tdx_alloc_pages_contig, NULL);
+}
+
  #define HPA_LIST_INFO_FIRST_ENTRY	GENMASK_U64(11, 3)
  #define HPA_LIST_INFO_PFN		GENMASK_U64(51, 12)
  #define HPA_LIST_INFO_LAST_ENTRY	GENMASK_U64(63, 55)
author
#27 lore raw

Re: [PATCH v2 04/31] x86/virt/tdx: Support allocating contiguous pages for tdx_page_array

Quoted 16 lines

arch/x86/virt/vmx/tdx/tdx.c void tdx_page_array_free(struct tdx_page_array *array) · line 333

Quoted 6 lines

static struct tdx_page_array *

arch/x86/virt/vmx/tdx/tdx.c +4 -1
-tdx_page_array_alloc(unsigned int nr_pages)
+tdx_page_array_alloc(unsigned int nr_pages,
+		     int (*alloc_fn)(unsigned int nr_pages,
+				     struct page **pages, void *data),
+		     void *data)

This interface seems cumbersome, since you will always have separate allocation paths:

Contig, Bulk and Iommu mt pages let's just keep them separate. I.e the flow should be:

1. Do common allocation by calling tdx_page_array_alloc (you aren't passing the alloc function) you just get a bare-bones tdx_page_array struct

Quoted 8 lines

2. Do the specific allocation in either :

tdx_page_array_create - for the bulk case tdx_page_array_alloc_contig - for the contig case tdx_page_array_create_iommu_mt - for the iommu case. Here you can open code tdx_alloc_pages_iommu_mt.

And keep the specific clearly separate in each function.

Thank for the suggestion. There are also other concerns in this thread, but I'll take this into consideration.

#28 lore raw

Re: [PATCH v2 04/31] x86/virt/tdx: Support allocating contiguous pages for tdx_page_array

In reply to Xu Yilun
Quoted 22 lines · Xu Yilun

The current tdx_page_array implementation allocates scattered order-0 pages. However, some TDX Module operations benefit from contiguous physical memory. E.g. Enabling TDX Module Extensions (an optional TDX feature) requires ~50MB memory and never returns. Such allocation would at worst cause ~25GB permanently fragmented memory if each allocated page is from a different 2M region.

Support allocating contiguous pages for tdx_page_array by making the allocation method configurable. Change the tdx_page_array_alloc() to accept a custom allocation function pointer and a context parameter. Wrap the specific allocation into a tdx_page_array_alloc_contig() helper.

The foreseeable caller will allocate ~50MB memory with this helper, exceeding the maximum HPAs (512) a root page can hold, the typical usage will be:

- struct tdx_page_array *array = tdx_page_array_alloc_contig(nr_pages);
- for each 512-page bulk
- tdx_page_array_populate(array, offset);
- seamcall(TDH_XXX_ADD, array, ...);

Quoted 4 lines

The configurable allocation method would also benefit more tdx_page_array usages. TDX Module may require more specific memory layouts encoded in the root page. Will introduce them in following patches.

Will skip this one as I see it gets massively reworked in your incremental update. The change to make the caller responsible for creating the page array is great.

author
#29 lore raw

PATCH v2 5/31 x86/virt/tdx: Extend tdx_page_array to support IOMMU_MT

In reply to Xu Yilun

IOMMU_MT is another TDX Module defined structure similar to HPA_ARRAY_T and HPA_LIST_INFO. The difference is it requires multi-order contiguous pages for some entries. It adds an additional NUM_PAGES field for every multi-order page entry.

Add a dedicated allocation helper for IOMMU_MT. Fortunately put_page() works well for both single pages and multi-order folios, simplifying the cleanup logic for all allocation methods.

---
 arch/x86/include/asm/tdx.h  |  2 +
 arch/x86/virt/vmx/tdx/tdx.c | 90 +++++++++++++++++++++++++++++++++++--
 2 files changed, 89 insertions(+), 3 deletions(-)
arch/x86/include/asm/tdx.h +2 -0
index 9173a432b312..d5f1d7b7d1e7 100644
@@ -175,6 +175,8 @@ void tdx_page_array_ctrl_leak(struct tdx_page_array *array);
 int tdx_page_array_ctrl_release(struct tdx_page_array *array,
 				unsigned int nr_released,
 				u64 released_hpa);
+struct tdx_page_array *
+tdx_page_array_create_iommu_mt(unsigned int iq_order, unsigned int nr_mt_pages);
 
 struct tdx_td {
 	/* TD root structure: */
arch/x86/virt/vmx/tdx/tdx.c +87 -3

114 lines not on this page · raw message

Signature · 1 line
2.25.1
#30 lore raw

Re: [PATCH v2 05/31] x86/virt/tdx: Extend tdx_page_array to support IOMMU_MT

Quoted 88 lines · Xu Yilun

IOMMU_MT is another TDX Module defined structure similar to HPA_ARRAY_T and HPA_LIST_INFO. The difference is it requires multi-order contiguous pages for some entries. It adds an additional NUM_PAGES field for every multi-order page entry.

Add a dedicated allocation helper for IOMMU_MT. Fortunately put_page() works well for both single pages and multi-order folios, simplifying the cleanup logic for all allocation methods.

---
 arch/x86/include/asm/tdx.h  |  2 +
 arch/x86/virt/vmx/tdx/tdx.c | 90 +++++++++++++++++++++++++++++++++++--
 2 files changed, 89 insertions(+), 3 deletions(-)
arch/x86/include/asm/tdx.h +2 -0
index 9173a432b312..d5f1d7b7d1e7 100644
@@ -175,6 +175,8 @@ void tdx_page_array_ctrl_leak(struct tdx_page_array *array);
 int tdx_page_array_ctrl_release(struct tdx_page_array *array,
 				unsigned int nr_released,
 				u64 released_hpa);
+struct tdx_page_array *
+tdx_page_array_create_iommu_mt(unsigned int iq_order, unsigned int nr_mt_pages);
 
 struct tdx_td {
 	/* TD root structure: */
arch/x86/virt/vmx/tdx/tdx.c +28 -3
index 6c4ed80e8e5a..2b17e0f73dac 100644
@@ -275,8 +275,15 @@ static int tdx_page_array_populate(struct tdx_page_array *array,
 			    TDX_PAGE_ARRAY_MAX_NENTS);
 
 	entries = array->root;
-	for (i = 0; i < array->nents; i++)
-		entries[i] = page_to_phys(array->pages[offset + i]);
+	for (i = 0; i < array->nents; i++) {
+		struct page *page = array->pages[offset + i];
+
+		entries[i] = page_to_phys(page);
+
+		/* Now only for iommu_mt */
+		if (compound_nr(page) > 1)
+			entries[i] |= compound_nr(page);
+	}
 
 	return array->nents;
 }
@@ -286,7 +293,7 @@ static void tdx_free_pages_bulk(unsigned int nr_pages, struct page **pages)
 	int i;
 
 	for (i = 0; i < nr_pages; i++)
-		__free_page(pages[i]);
+		put_page(pages[i]);
 }
 
 static int tdx_alloc_pages_bulk(unsigned int nr_pages, struct page **pages,
@@ -463,6 +470,10 @@ static bool tdx_page_array_validate_release(struct tdx_page_array *array,
 			struct page *page = array->pages[offset + i];
 			u64 val = page_to_phys(page);
 
+			/* Now only for iommu_mt */
+			if (compound_nr(page) > 1)
+				val |= compound_nr(page);
+
 			if (val != entries[i]) {
 				pr_err("%s entry[%d] [0x%llx] doesn't match page hpa [0x%llx]\n",
 				       __func__, i, entries[i], val);
@@ -555,6 +566,79 @@ tdx_page_array_alloc_contig(unsigned int nr_pages)
 	return tdx_page_array_alloc(nr_pages, tdx_alloc_pages_contig, NULL);
 }
 
+static int tdx_alloc_pages_iommu_mt(unsigned int nr_pages, struct page **pages,
+				    void *data)
+{
+	unsigned int iq_order = (unsigned int)(long)data;
+	struct folio *t_iq, *t_ctxiq;
+	int ret;
+
+	/* TODO: folio_alloc_node() is preferred, but need numa info */
+	t_iq = folio_alloc(GFP_KERNEL | __GFP_ZERO, iq_order);
+	if (!t_iq)
+		return -ENOMEM;
+
+	t_ctxiq = folio_alloc(GFP_KERNEL | __GFP_ZERO, iq_order);
+	if (!t_ctxiq) {

arch/x86/virt/vmx/tdx/tdx.c tdx_page_array_alloc_contig(unsigned int nr_pages) · line 592

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +10 -0
+		ret = -ENOMEM;
+		goto out_t_iq;
+	}
+
+	ret = tdx_alloc_pages_bulk(nr_pages - 2, pages + 2, NULL);
+	if (ret)
+		goto out_t_ctxiq;
+
+	pages[0] = folio_page(t_iq, 0);
+	pages[1] = folio_page(t_ctxiq, 0);

To me it seems like this can't really be called a page array any more. The first two u64's are too special. Instead it's a special one-off ABI format passed via a page.

BTW, I can't find TDH.IOMMU.SETUP in the docs. Any pointers?

Quoted 16 lines
arch/x86/virt/vmx/tdx/tdx.c +16 -0
+
+	return 0;
+
+out_t_ctxiq:
+	folio_put(t_ctxiq);
+out_t_iq:
+	folio_put(t_iq);
+
+	return ret;
+}
+
+/**
+ * tdx_page_array_create_iommu_mt() - Create a page array for IOMMU Memory Tables
+ * @iq_order: The allocation order for the IOMMU Invalidation Queue.
+ * @nr_mt_pages: Number of additional order-0 pages for the MT.
+ *

arch/x86/virt/vmx/tdx/tdx.c tdx_page_array_alloc_contig(unsigned int nr_pages) · line 618

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +10 -0
+ * Allocate and populate a specialized tdx_page_array for IOMMU_MT structures.
+ * The resulting array consists of two multi-order folios (at index 0 and 1)
+ * followed by the requested number of order-0 pages.
+ *
+ * Return: Fully populated tdx_page_array or NULL on failure.
+ */
+struct tdx_page_array *
+tdx_page_array_create_iommu_mt(unsigned int iq_order, unsigned int nr_mt_pages)
+{
+	unsigned int nr_pages = nr_mt_pages + 2;

Consider the amount of tricks that are needed to coax the tdx_page_array to populate the handoff page as needed. It adds 2 pages here, then subtracts them later in the callback. Then tweaks the pa in tdx_page_array_populate() to add the length...

Quoted 26 lines
arch/x86/virt/vmx/tdx/tdx.c +23 -0
+	struct tdx_page_array *array;
+	int populated;
+
+	if (nr_pages > TDX_PAGE_ARRAY_MAX_NENTS)
+		return NULL;
+
+	array = tdx_page_array_alloc(nr_pages, tdx_alloc_pages_iommu_mt,
+				     (void *)(long)iq_order);
+	if (!array)
+		return NULL;
+
+	populated = tdx_page_array_populate(array, 0);
+	if (populated != nr_pages)
+		goto out_free;
+
+	return array;
+
+out_free:
+	tdx_page_array_free(array);
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(tdx_page_array_create_iommu_mt);
+
 #define HPA_LIST_INFO_FIRST_ENTRY	GENMASK_U64(11, 3)
 #define HPA_LIST_INFO_PFN		GENMASK_U64(51, 12)
 #define HPA_LIST_INFO_LAST_ENTRY	GENMASK_U64(63, 55)
author
#31 lore raw

Re: [PATCH v2 05/31] x86/virt/tdx: Extend tdx_page_array to support IOMMU_MT

Quoted 29 lines
Quoted 14 lines
arch/x86/virt/vmx/tdx/tdx.c +14 -0
+static int tdx_alloc_pages_iommu_mt(unsigned int nr_pages, struct page **pages,
+				    void *data)
+{
+	unsigned int iq_order = (unsigned int)(long)data;
+	struct folio *t_iq, *t_ctxiq;
+	int ret;
+
+	/* TODO: folio_alloc_node() is preferred, but need numa info */
+	t_iq = folio_alloc(GFP_KERNEL | __GFP_ZERO, iq_order);
+	if (!t_iq)
+		return -ENOMEM;
+
+	t_ctxiq = folio_alloc(GFP_KERNEL | __GFP_ZERO, iq_order);
+	if (!t_ctxiq) {

arch/x86/virt/vmx/tdx/tdx.c tdx_page_array_alloc_contig(unsigned int nr_pages) · line 592

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +10 -0
+		ret = -ENOMEM;
+		goto out_t_iq;
+	}
+
+	ret = tdx_alloc_pages_bulk(nr_pages - 2, pages + 2, NULL);
+	if (ret)
+		goto out_t_ctxiq;
+
+	pages[0] = folio_page(t_iq, 0);
+	pages[1] = folio_page(t_ctxiq, 0);

To me it seems like this can't really be called a page array any more. The first two u64's are too special. Instead it's a special one-off ABI format passed via a page.

Quoted 1 line

BTW, I can't find TDH.IOMMU.SETUP in the docs. Any pointers?

https://cdrdv2.intel.com/v1/dl/getContent/858625

Quoted 23 lines
Quoted 23 lines
arch/x86/virt/vmx/tdx/tdx.c +22 -0
+
+	return 0;
+
+out_t_ctxiq:
+	folio_put(t_ctxiq);
+out_t_iq:
+	folio_put(t_iq);
+
+	return ret;
+}
+
+/**
+ * tdx_page_array_create_iommu_mt() - Create a page array for IOMMU Memory Tables
+ * @iq_order: The allocation order for the IOMMU Invalidation Queue.
+ * @nr_mt_pages: Number of additional order-0 pages for the MT.
+ *
+ * Allocate and populate a specialized tdx_page_array for IOMMU_MT structures.
+ * The resulting array consists of two multi-order folios (at index 0 and 1)
+ * followed by the requested number of order-0 pages.
+ *
+ * Return: Fully populated tdx_page_array or NULL on failure> > + */
+struct tdx_page_array *
Quoted 8 lines

arch/x86/virt/vmx/tdx/tdx.c tdx_page_array_alloc_contig(unsigned int nr_pages) · line 618

Quoted 3 lines
arch/x86/virt/vmx/tdx/tdx.c +3 -0
+tdx_page_array_create_iommu_mt(unsigned int iq_order, unsigned int nr_mt_pages)
+{
+	unsigned int nr_pages = nr_mt_pages + 2;

Consider the amount of tricks that are needed to coax the tdx_page_array to populate the handoff page as needed. It adds 2 pages here, then subtracts them later in the callback. Then tweaks the pa in tdx_page_array_populate() to add the length...

mm.. The tricky part is the specific memory requirement/allocation, the common part is the pa list contained in a root page. Maybe we only model the later, let the specific user does the memory allocation. Is that closer to your "break concepts apart" idea?

Quoted 23 lines
Quoted 23 lines
arch/x86/virt/vmx/tdx/tdx.c +22 -0
+	struct tdx_page_array *array;
+	int populated;
+
+	if (nr_pages > TDX_PAGE_ARRAY_MAX_NENTS)
+		return NULL;
+
+	array = tdx_page_array_alloc(nr_pages, tdx_alloc_pages_iommu_mt,
+				     (void *)(long)iq_order);
+	if (!array)
+		return NULL;
+
+	populated = tdx_page_array_populate(array, 0);
+	if (populated != nr_pages)
+		goto out_free;
+
+	return array;
+
+out_free:
+	tdx_page_array_free(array);
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(tdx_page_array_create_iommu_mt);
#32 lore raw

Re: [PATCH v2 05/31] x86/virt/tdx: Extend tdx_page_array to support IOMMU_MT

Quoted 11 lines · Xu Yilun
Quoted 6 lines

Consider the amount of tricks that are needed to coax the tdx_page_array to populate the handoff page as needed. It adds 2 pages here, then subtracts them
later in the callback. Then tweaks the pa in tdx_page_array_populate() to add
the length...

mm.. The tricky part is the specific memory requirement/allocation, the common part is the pa list contained in a root page. Maybe we only model the later, let the specific user does the memory allocation. Is that closer to your "break concepts apart" idea?

I haven't wrapped my head around this enough to suggest anything is definitely the right approach.

But yes, the idea would be that the allocation of the list of pages to give to the TDX module would be a separate allocation and set of management functions. And the the allocation of the pages that are used to communicate the list of pages (and in this case other args) with the module would be another set. So each type of TDX module arg page format (IOMMU_MT, etc) would be separable, but share the page list allocation part only. It looks like Nikolay was probing along the same path. Not sure if he had the same solution in mind.

So for this:
1. Allocate a list or array of pages using a generic method.
2. Allocate these two IOMMU special pages.
3. Allocate memory needed for the seamcall (root pages)

Hand all three to the wrapper and have it shove them all through in the special way it prefers.

Maybe... Can you write something about the similarities and differences with the three types of lists in that series? Like in a compact form?

Also, how much of the earlier code duplication you wanted to avoid was the leaking and special error handling stuff?

author
#33 lore raw

Re: [PATCH v2 05/31] x86/virt/tdx: Extend tdx_page_array to support IOMMU_MT

Quoted 29 lines · Rick P
Quoted 11 lines · Xu Yilun
Quoted 6 lines

Consider the amount of tricks that are needed to coax the tdx_page_array to populate the handoff page as needed. It adds 2 pages here, then subtracts them
later in the callback. Then tweaks the pa in tdx_page_array_populate() to add
the length...

mm.. The tricky part is the specific memory requirement/allocation, the common part is the pa list contained in a root page. Maybe we only model the later, let the specific user does the memory allocation. Is that closer to your "break concepts apart" idea?

I haven't wrapped my head around this enough to suggest anything is definitely the right approach.

But yes, the idea would be that the allocation of the list of pages to give to the TDX module would be a separate allocation and set of management functions. And the the allocation of the pages that are used to communicate the list of pages (and in this case other args) with the module would be another set. So each type of TDX module arg page format (IOMMU_MT, etc) would be separable, but share the page list allocation part only. It looks like Nikolay was probing along the same path. Not sure if he had the same solution in mind.

So for this:
1. Allocate a list or array of pages using a generic method.
2. Allocate these two IOMMU special pages.
3. Allocate memory needed for the seamcall (root pages)

Quoted 5 lines

Hand all three to the wrapper and have it shove them all through in the special way it prefers.

Maybe... Can you write something about the similarities and differences with the three types of lists in that series? Like in a compact form?

The common part:

64bit obj type root page

   +----------+----------------+    +---------------------+     
   | ...      |  ...           |    | page0 HPA(bit12-51) |--> page0
   +----------+----------------+    +---------------------+     
   |bit 12-51 | root page HPA  |--->| page1 HPA           |--> page1
   +----------+----------------+    +---------------------+
   | ...      |  ...           |    | pageX HPA           |--> pageX
   +----------+----------------+    +---------------------+

The specific objects:

HPA_LIST_INFO root page

   +----------+----------------+    +---------------------+
   |bit 3-11  | first entry    |    | page0 HPA(bit12-51) |
   +----------+----------------+    +---------------------+
   |bit 12-51 | root page HPA  |--->| page1 HPA           |
   +----------+----------------+    +---------------------+
   |bit 55-63 | last entry     |    | pageX HPA           |
   +----------+----------------+    +---------------------+

HPA_ARRAY_T root page HPA_ARRAY_T(singleton mode)

   +----------+----------------+    +---------------------+      +----------+----------------+
   |bit 3-11  | Reserved 0     |    | page0 HPA(bit12-51) |      |bit 3-11  | Reserved 0     |
   +----------+----------------+    +---------------------+      +----------+----------------+
   |bit 12-51 | root page HPA  |--->| page1 HPA           |      |bit 12-51 | page0 HPA      |--> page0
   +----------+----------------+    +---------------------+      +----------+----------------+
   |bit 55-63 | last entry     |    | pageX HPA           |      |bit 55-63 | last entry     |
   +----------+----------------+    +---------------------+      +----------+----------------+

MMIOMT root page

   +----------+----------------+    +-----------------------------+-------------------+
   |bit 3-11  | Reserved 0     |    | 2^order page0 HPA(bit12-51) |num pages(bit 0-11)|
   +----------+----------------+    +-----------------------------+-------------------+
   |bit 12-51 | root page HPA  |--->| 2^order page1 HPA           |num pages          |
   +----------+----------------+    +-----------------------------+-------------------+
   |bit 55-63 | Reserved 0     |    | page2 HPA                   |0                  |
   +----------+----------------+    +-----------------------------+-------------------+
                                    | page3 HPA                   |0                  |
                                    +-----------------------------+-------------------+
                                    | pageX HPA                   |0                  |
                                    +-----------------------------+-------------------+

The same thing is they all have root_page_hpa->root_page->page_hpa_list structure.

The differences:

HPA_LIST_INFO HPA_ARRAY_T IOMMU_MT Note
first entry Y N N start entry in root page
last entry Y Y N last entry in root page
num pages always 0 always 0 Y for multi-order page
singleton N Y N try to save a root page

Quoted 3 lines

Also, how much of the earlier code duplication you wanted to avoid was the leaking and special error handling stuff?

This is indeed a large part, and now we don't need them anymore.

Others are:
- the root_page allocation/population/free
- Too much parameters (struct page **, num_pages, struct page *root...) for seamcall wrappers. Or 3 newly defined structures which looks pretty much the same and need same implementations like tdx_clflush_page().

Thanks,
Yilun

author
#34 lore raw

Re: [PATCH v2 05/31] x86/virt/tdx: Extend tdx_page_array to support IOMMU_MT

In reply to Edgecombe, Rick P
Quoted 24 lines · Rick P
Quoted 11 lines · Xu Yilun
Quoted 6 lines

Consider the amount of tricks that are needed to coax the tdx_page_array to populate the handoff page as needed. It adds 2 pages here, then subtracts them
later in the callback. Then tweaks the pa in tdx_page_array_populate() to add
the length...

mm.. The tricky part is the specific memory requirement/allocation, the common part is the pa list contained in a root page. Maybe we only model the later, let the specific user does the memory allocation. Is that closer to your "break concepts apart" idea?

I haven't wrapped my head around this enough to suggest anything is definitely the right approach.

But yes, the idea would be that the allocation of the list of pages to give to the TDX module would be a separate allocation and set of management functions. And the the allocation of the pages that are used to communicate the list of pages (and in this case other args) with the module would be another set. So each type of TDX module arg page format (IOMMU_MT, etc) would be separable, but share the page list allocation part only. It looks like Nikolay was probing along the same path. Not sure if he had the same solution in mind.

Quoted 7 lines

So for this:
1. Allocate a list or array of pages using a generic method.
2. Allocate these two IOMMU special pages.
3. Allocate memory needed for the seamcall (root pages)

Hand all three to the wrapper and have it shove them all through in the special way it prefers.

I'm drafting some changes and make the tdx_page_array look like:

  struct tdx_page_array {
	/* public: */
	unsigned int nr_pages;
	struct page **pages;

	/* private: */
	u64 *root;
	bool flush_on_free;

};

- I removed the page allocations for tdx_page_array kAPIs. Now the caller needs to allocate the struct page **pages and the page list, then create the tdx_page_array by providing these pages.

struct tdx_page_array *tdx_page_array_create(struct page **pages,
unsigned int nr_pages)

This also means tdx_page_array doesn't have to hold more than 512 pages anymore, it now an exact descriptor for the TDX Module's definitions rather than a manager. It's a chunk of the required memory when we need more than 512 pages. This eliminates the need for 'offset' field and the slide window operations so make the helpers simpler.

- I still keep the generic struct tdx_page_array to represent all kinds of object types (HPA_ARRAY_T, HPA_LIST_INFO, IOMMU_MT), and provide the tdx_page_array to SEAMCALL helpers as parameters. I think this structure is generally good enough to represent a list of pages, keeps type safety compared to a list of HPAs.

- I still record both the page list (struct page **pages) and the HPA list (in u64 *root). struct page **pages works with kernel memory management (e.g. vmap) well while the populated root works with SEAMCALLs.

- I'm not introducing more structures each for an object type, like struct hpa_array, struct hpa_list_info, struct iommu_metadata. They are conceptually the same thing. The iommu_mt supports multi-order pages, hpa_array_t & hpa_list_info don't support. But their bit definitions don't conflict. I can use the same piece of code to populate their root page content.

- Add a flush_on_free field to mark if a cache write back is needed on tdx_page_array_free(), then we don't need 2 free APIs.

I want to clean up my code, then post an incremental patch for preview.

Thanks.

author
#35 lore raw

Re: [PATCH v2 05/31] x86/virt/tdx: Extend tdx_page_array to support IOMMU_MT

Quoted 77 lines · Xu Yilun
Quoted 24 lines · Rick P
Quoted 11 lines · Xu Yilun
Quoted 6 lines

Consider the amount of tricks that are needed to coax the tdx_page_array to populate the handoff page as needed. It adds 2 pages here, then subtracts them
later in the callback. Then tweaks the pa in tdx_page_array_populate() to add
the length...

mm.. The tricky part is the specific memory requirement/allocation, the common part is the pa list contained in a root page. Maybe we only model the later, let the specific user does the memory allocation. Is that closer to your "break concepts apart" idea?

I haven't wrapped my head around this enough to suggest anything is definitely the right approach.

But yes, the idea would be that the allocation of the list of pages to give to the TDX module would be a separate allocation and set of management functions. And the the allocation of the pages that are used to communicate the list of pages (and in this case other args) with the module would be another set. So each type of TDX module arg page format (IOMMU_MT, etc) would be separable, but share the page list allocation part only. It looks like Nikolay was probing along the same path. Not sure if he had the same solution in mind.

Quoted 7 lines

So for this:
1. Allocate a list or array of pages using a generic method.
2. Allocate these two IOMMU special pages.
3. Allocate memory needed for the seamcall (root pages)

Hand all three to the wrapper and have it shove them all through in the special way it prefers.

I'm drafting some changes and make the tdx_page_array look like:

  struct tdx_page_array {
	/* public: */
	unsigned int nr_pages;
	struct page **pages;

	/* private: */
	u64 *root;
	bool flush_on_free;

};

- I removed the page allocations for tdx_page_array kAPIs. Now the caller needs to allocate the struct page **pages and the page list, then create the tdx_page_array by providing these pages.

struct tdx_page_array *tdx_page_array_create(struct page **pages,
unsigned int nr_pages)

This also means tdx_page_array doesn't have to hold more than 512 pages anymore, it now an exact descriptor for the TDX Module's definitions rather than a manager. It's a chunk of the required memory when we need more than 512 pages. This eliminates the need for 'offset' field and the slide window operations so make the helpers simpler.

- I still keep the generic struct tdx_page_array to represent all kinds of object types (HPA_ARRAY_T, HPA_LIST_INFO, IOMMU_MT), and provide the tdx_page_array to SEAMCALL helpers as parameters. I think this structure is generally good enough to represent a list of pages, keeps type safety compared to a list of HPAs.

- I still record both the page list (struct page **pages) and the HPA list (in u64 *root). struct page **pages works with kernel memory management (e.g. vmap) well while the populated root works with SEAMCALLs.

- I'm not introducing more structures each for an object type, like struct hpa_array, struct hpa_list_info, struct iommu_metadata. They are conceptually the same thing. The iommu_mt supports multi-order pages, hpa_array_t & hpa_list_info don't support. But their bit definitions don't conflict. I can use the same piece of code to populate their root page content.

Quoted 4 lines

- Add a flush_on_free field to mark if a cache write back is needed on tdx_page_array_free(), then we don't need 2 free APIs.

I want to clean up my code, then post an incremental patch for preview.

Hi, I end up made the following changes on top of this series:

-------8<--------

 arch/x86/include/asm/tdx.h            |  32 +-
 arch/x86/virt/vmx/tdx/tdx.c           | 561 ++++++++------------------
 drivers/virt/coco/tdx-host/tdx-host.c | 179 ++++++--
 3 files changed, 316 insertions(+), 456 deletions(-)
arch/x86/include/asm/tdx.h +10 -22
index 7bdd66acda5b..31d1101a4f45 100644
@@ -143,15 +143,12 @@ void tdx_quirk_reset_page(struct page *page);
 
 /**
  * struct tdx_page_array - Represents a list of pages for TDX Module access
- * @nr_pages: Total number of data pages in the collection
- * @pages: Array of data page pointers containing all the data
+ * @nr_pages: Number of data pages in the collection (up to 512)
+ * @pages: Array of data page pointers
  *
- * @offset: Internal: The starting index in @pages, positions the currently
- *	    populated page window in @root.
- * @nents: Internal: Number of valid HPAs for the page window in @root
- * @root: Internal: A single 4KB page holding the 8-byte HPAs of the page
- *	  window. The page window max size is constrained by the root page,
- *	  which is 512 HPAs.
+ * @root: Internal: A single 4KB page holding the 8-byte HPAs of the @pages
+ * @flush_on_free: Internal: whether to flush cache when @pages are to be
+ *		   freed.
  *
  * This structure abstracts several TDX Module defined object types, e.g.,
  * HPA_ARRAY_T and HPA_LIST_INFO. Typically they all use a "root page" as the
@@ -165,20 +162,13 @@ struct tdx_page_array {
 	struct page **pages;
 
 	/* private: */
-	unsigned int offset;
-	unsigned int nents;
 	u64 *root;
+	bool flush_on_free;
 };
 
 void tdx_page_array_free(struct tdx_page_array *array);
-DEFINE_FREE(tdx_page_array_free, struct tdx_page_array *, if (_T) tdx_page_array_free(_T))
-struct tdx_page_array *tdx_page_array_create(unsigned int nr_pages);
-void tdx_page_array_ctrl_leak(struct tdx_page_array *array);
-int tdx_page_array_ctrl_release(struct tdx_page_array *array,
-				unsigned int nr_released,
-				u64 released_hpa);
-struct tdx_page_array *
-tdx_page_array_create_iommu_mt(unsigned int iq_order, unsigned int nr_mt_pages);
+struct tdx_page_array *tdx_page_array_create(struct page **pages,
+					     unsigned int nr_pages);
 
 struct tdx_td {
 	/* TD root structure: */
@@ -248,8 +238,7 @@ u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct page *page);
 u64 tdh_iommu_setup(u64 vtbar, struct tdx_page_array *iommu_mt, u64 *iommu_id);
 u64 tdh_iommu_clear(u64 iommu_id, struct tdx_page_array *iommu_mt);
 u64 tdh_spdm_create(u64 func_id, struct tdx_page_array *spdm_mt, u64 *spdm_id);
-u64 tdh_spdm_delete(u64 spdm_id, struct tdx_page_array *spdm_mt,
-		    unsigned int *nr_released, u64 *released_hpa);
+u64 tdh_spdm_delete(u64 spdm_id, struct tdx_page_array *spdm_mt);
 u64 tdh_exec_spdm_connect(u64 spdm_id, struct page *spdm_conf,
 			  struct page *spdm_rsp, struct page *spdm_req,
 			  struct tdx_page_array *spdm_out,
@@ -269,8 +258,7 @@ u64 tdh_ide_stream_create(u64 stream_info, u64 spdm_id,
 			  u64 *rp_ide_id);
 u64 tdh_ide_stream_block(u64 spdm_id, u64 stream_id);
 u64 tdh_ide_stream_delete(u64 spdm_id, u64 stream_id,
-			  struct tdx_page_array *stream_mt,
-			  unsigned int *nr_released, u64 *released_hpa);
+			  struct tdx_page_array *stream_mt);
 u64 tdh_ide_stream_km(u64 spdm_id, u64 stream_id, u64 operation,
 		      struct page *spdm_rsp, struct page *spdm_req,
 		      u64 *spdm_req_len);
arch/x86/virt/vmx/tdx/tdx.c +171 -390

744 lines not on this page · raw message

drivers/virt/coco/tdx-host/tdx-host.c +135 -44

250 lines not on this page · raw message

#36 lore raw

Re: [PATCH v2 05/31] x86/virt/tdx: Extend tdx_page_array to support IOMMU_MT

Xu Yilun wrote:
[..]

Quoted 11 lines
Quoted 11 lines

I'm drafting some changes and make the tdx_page_array look like:

  struct tdx_page_array {
	/* public: */
	unsigned int nr_pages;
	struct page **pages;

	/* private: */
	u64 *root;
	bool flush_on_free;

How about "need_phymem_page_wbinvd"?

That makes it a bit more greppable and not to be confused with other flushing.

[..]

Quoted 13 lines

Hi, I end up made the following changes on top of this series:

-------8<--------

 arch/x86/include/asm/tdx.h            |  32 +-
 arch/x86/virt/vmx/tdx/tdx.c           | 561 ++++++++------------------
 drivers/virt/coco/tdx-host/tdx-host.c | 179 ++++++--
 3 files changed, 316 insertions(+), 456 deletions(-)
arch/x86/virt/vmx/tdx/tdx.c +2 -0
+		ret = tdx_ext_mem_setup(nr_pages, &ext_mem);
 		if (ret)
+			return ret;
 	}

arch/x86/virt/vmx/tdx/tdx.c static int init_tdx_ext(void) · line 1422

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +7 -0
 
+	ret = tdx_ext_init();
+	if (ret)
+		goto out_remove_ext_mem;
+
 	/*
+	 * Extensions memory is never reclaimed once assigned, stop tracking it
+	 * and free the tracking structures.
 	 */
+	tdx_page_array_free(ext_mem.chunk);

Wait, these pages belong to the module now, they can't be freed, or I am missing something?

arch/x86/virt/vmx/tdx/tdx.c static int init_tdx_ext(void) · line 1423

Quoted 1 line
arch/x86/virt/vmx/tdx/tdx.c +1 -0
+	kfree(ext_mem.pages);

Releasing this makes sense.

Quoted 8 lines
arch/x86/virt/vmx/tdx/tdx.c +0 -2
 	pr_info("%lu KB allocated for TDX Module Extensions\n",
 		nr_pages * PAGE_SIZE / 1024);
 
 	return 0;
 
-out_flush:
-	if (ext_mem)

arch/x86/virt/vmx/tdx/tdx.c static int init_tdx_ext(void) · line 1439

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +9 -0
+out_remove_ext_mem:
+	if (nr_pages) {
+		/*
+		 * TDH.EXT.MEM.ADD only collects required memory. TDX.EXT.INIT
+		 * does the actual initialization so if it fails some pages may
+		 * have been touched by the TDX module, flush cache before
+		 * returning these pages to kernel.
+		 */
 		wbinvd_on_all_cpus();
+		tdx_ext_mem_remove(&ext_mem);

This only releases the last populated chunk, not all previous chunks, right?

author
#37 lore raw

Re: [PATCH v2 05/31] x86/virt/tdx: Extend tdx_page_array to support IOMMU_MT

Quoted 10 lines · Dan Williams

Xu Yilun wrote:
[..]

Quoted 8 lines
Quoted 8 lines

I'm drafting some changes and make the tdx_page_array look like:

  struct tdx_page_array {
	/* public: */
	unsigned int nr_pages;
	struct page **pages;
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
	/* private: */
	u64 *root;
	bool flush_on_free;

How about "need_phymem_page_wbinvd"?

Yes.

Quoted 19 lines

That makes it a bit more greppable and not to be confused with other flushing.

[..]

Quoted 14 lines

Hi, I end up made the following changes on top of this series:

-------8<--------

 arch/x86/include/asm/tdx.h            |  32 +-
 arch/x86/virt/vmx/tdx/tdx.c           | 561 ++++++++------------------
 drivers/virt/coco/tdx-host/tdx-host.c | 179 ++++++--
 3 files changed, 316 insertions(+), 456 deletions(-)
arch/x86/virt/vmx/tdx/tdx.c +2 -0
+		ret = tdx_ext_mem_setup(nr_pages, &ext_mem);
 		if (ret)
+			return ret;
 	}
Quoted 12 lines

arch/x86/virt/vmx/tdx/tdx.c static int init_tdx_ext(void) · line 1422

Quoted 9 lines
arch/x86/virt/vmx/tdx/tdx.c +7 -0
+	ret = tdx_ext_init();
+	if (ret)
+		goto out_remove_ext_mem;
+
 	/*
+	 * Extensions memory is never reclaimed once assigned, stop tracking it
+	 * and free the tracking structures.
 	 */
+	tdx_page_array_free(ext_mem.chunk);

Wait, these pages belong to the module now, they can't be freed, or I am missing something?

With this new solution, tdx_page_array is downgraded to a descriptor, doesn't manage the actual data pages/memory any more. So tdx_page_array_free() will not free data pages, only frees the tdx_page_array descriptor.

Quoted 11 lines

arch/x86/virt/vmx/tdx/tdx.c static int init_tdx_ext(void) · line 1423

Quoted 2 lines
arch/x86/virt/vmx/tdx/tdx.c +1 -0
+	kfree(ext_mem.pages);

Releasing this makes sense.

Quoted 6 lines
arch/x86/virt/vmx/tdx/tdx.c +0 -0
 	pr_info("%lu KB allocated for TDX Module Extensions\n",
 		nr_pages * PAGE_SIZE / 1024);
 
 	return 0;
Quoted 15 lines

arch/x86/virt/vmx/tdx/tdx.c static int init_tdx_ext(void) · line 1439

Quoted 12 lines
arch/x86/virt/vmx/tdx/tdx.c +9 -2
-out_flush:
-	if (ext_mem)
+out_remove_ext_mem:
+	if (nr_pages) {
+		/*
+		 * TDH.EXT.MEM.ADD only collects required memory. TDX.EXT.INIT
+		 * does the actual initialization so if it fails some pages may
+		 * have been touched by the TDX module, flush cache before
+		 * returning these pages to kernel.
+		 */
 		wbinvd_on_all_cpus();
+		tdx_ext_mem_remove(&ext_mem);

This only releases the last populated chunk, not all previous chunks, right?

Not true. ext_mem stores all the data pages and the reusable descriptor 'chunk' for SEAMCALL. tdx_ext_mem_remove() removes all the data pages and the 'chunk'.

#38 lore raw

Re: [PATCH v2 05/31] x86/virt/tdx: Extend tdx_page_array to support IOMMU_MT

Quoted 51 lines · Xu Yilun
Quoted 10 lines · Dan Williams

Xu Yilun wrote:
[..]

Quoted 8 lines
Quoted 8 lines

I'm drafting some changes and make the tdx_page_array look like:

  struct tdx_page_array {
	/* public: */
	unsigned int nr_pages;
	struct page **pages;
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
	/* private: */
	u64 *root;
	bool flush_on_free;

How about "need_phymem_page_wbinvd"?

Yes.

Quoted 19 lines

That makes it a bit more greppable and not to be confused with other flushing.

[..]

Quoted 14 lines

Hi, I end up made the following changes on top of this series:

-------8<--------

 arch/x86/include/asm/tdx.h            |  32 +-
 arch/x86/virt/vmx/tdx/tdx.c           | 561 ++++++++------------------
 drivers/virt/coco/tdx-host/tdx-host.c | 179 ++++++--
 3 files changed, 316 insertions(+), 456 deletions(-)
arch/x86/virt/vmx/tdx/tdx.c +2 -0
+		ret = tdx_ext_mem_setup(nr_pages, &ext_mem);
 		if (ret)
+			return ret;
 	}
Quoted 12 lines

arch/x86/virt/vmx/tdx/tdx.c static int init_tdx_ext(void) · line 1422

Quoted 9 lines
arch/x86/virt/vmx/tdx/tdx.c +7 -0
+	ret = tdx_ext_init();
+	if (ret)
+		goto out_remove_ext_mem;
+
 	/*
+	 * Extensions memory is never reclaimed once assigned, stop tracking it
+	 * and free the tracking structures.
 	 */
+	tdx_page_array_free(ext_mem.chunk);

Wait, these pages belong to the module now, they can't be freed, or I am missing something?

Quoted 4 lines

With this new solution, tdx_page_array is downgraded to a descriptor, doesn't manage the actual data pages/memory any more. So tdx_page_array_free() will not free data pages, only frees the tdx_page_array descriptor.

Oh, I was confused by the fact that tdx_page_array_free() still loops through array->pages in the need_wbinvd case. In the case of "never reclaim" it will also "never wbinvd". ...and this why populate has that "WARN_ON_ONCE(array->pages && array->flush_on_free);".

A couple recommendations come to mind:

* s/tdx_page_array_free/tdx_page_array_destroy/

...since "destroy" mirrors create and matches other cases where only metadata is managed.

* Create a new tdx_page_array_repopulate() helper to make it clear which paths depend on being able to repopulate and move the WARN_ON_ONCE() out of the common path that does not repopulate. "repopulate" can have "realloc" semantics where it allocates on first use, but otherwise "populate" gets to not care about the corner cases. Make the WARN case fail repopulate.

Quoted 21 lines
Quoted 5 lines
Quoted 5 lines
arch/x86/virt/vmx/tdx/tdx.c +0 -0
 	pr_info("%lu KB allocated for TDX Module Extensions\n",
 		nr_pages * PAGE_SIZE / 1024);
 
 	return 0;
Quoted 15 lines

arch/x86/virt/vmx/tdx/tdx.c static int init_tdx_ext(void) · line 1439

Quoted 12 lines
arch/x86/virt/vmx/tdx/tdx.c +9 -2
-out_flush:
-	if (ext_mem)
+out_remove_ext_mem:
+	if (nr_pages) {
+		/*
+		 * TDH.EXT.MEM.ADD only collects required memory. TDX.EXT.INIT
+		 * does the actual initialization so if it fails some pages may
+		 * have been touched by the TDX module, flush cache before
+		 * returning these pages to kernel.
+		 */
 		wbinvd_on_all_cpus();
+		tdx_ext_mem_remove(&ext_mem);

This only releases the last populated chunk, not all previous chunks, right?

Quoted 3 lines

Not true. ext_mem stores all the data pages and the reusable descriptor 'chunk' for SEAMCALL. tdx_ext_mem_remove() removes all the data pages and the 'chunk'.

Yes, see that now.

author
#39 lore raw

Re: [PATCH v2 05/31] x86/virt/tdx: Extend tdx_page_array to support IOMMU_MT

Quoted 7 lines

A couple recommendations come to mind:

* s/tdx_page_array_free/tdx_page_array_destroy/

...since "destroy" mirrors create and matches other cases where only metadata is managed.

Quoted 6 lines

* Create a new tdx_page_array_repopulate() helper to make it clear which paths depend on being able to repopulate and move the WARN_ON_ONCE() out of the common path that does not repopulate. "repopulate" can have "realloc" semantics where it allocates on first use, but otherwise "populate" gets to not care about the corner cases. Make the WARN case fail repopulate.

Agree. I end up add a function like that:

/**
* tdx_page_array_repopulate() - repopulate a tdx_page_array
* @array: The array descriptor to reallocate for.
* @pages: Pointer to struct page array for tdx_page_array populating
* @nr_pages: Size of @pages array.
*
* Re-populate the tdx_page_array. If @array is %NULL, it behaves exactly like
* tdx_page_array_create().
*
* Return: Re-populated tdx_page_array or NULL on failure.
*/
static struct tdx_page_array *
tdx_page_array_repopulate(struct tdx_page_array *array, struct page **pages,
unsigned int nr_pages)

{
	struct tdx_page_array *tmp = array;
	int ret;

	if (tmp) {
		/* Don't pass in something partially initialized */
		if (!tmp->root || !tmp->pages || !tmp->nr_pages)
			return NULL;

		/*
		 * When re-populating, the old pages are no longer tracked.
		 * Theoretically they require cache flushing before reclaiming
		 * for other kernel usage, similar to tdx_page_array_destroy().
		 * Since there is no use case to repopulate and then reclaim
		 * old pages yet, just warn to prompt future improvement.
		 */
		if (WARN_ON_ONCE(tmp->need_phymem_page_wbinvd))
			return NULL;
	} else {
		tmp = tdx_page_array_alloc();
		if (!tmp)
			return NULL;
	}

	ret = tdx_page_array_populate(tmp, pages, nr_pages);
	if (ret) {
		/* Only destroy newly allocated object */
		if (!array)
			tdx_page_array_destroy(tmp);

		return NULL;
	}

	return tmp;
}
#40 lore raw

Re: [PATCH v2 05/31] x86/virt/tdx: Extend tdx_page_array to support IOMMU_MT

In reply to Xu Yilun
Quoted 8 lines · Xu Yilun

IOMMU_MT is another TDX Module defined structure similar to HPA_ARRAY_T and HPA_LIST_INFO. The difference is it requires multi-order contiguous pages for some entries. It adds an additional NUM_PAGES field for every multi-order page entry.

Add a dedicated allocation helper for IOMMU_MT. Fortunately put_page() works well for both single pages and multi-order folios, simplifying the cleanup logic for all allocation methods.

Well I guess you can have a 'free_fn' to free the pages you allocated via 'alloc_fn'? Will this simplify the code and at least keep tdx_page_array implementation cleaner?

It's strange that you only have a 'alloc_fn' but doesn't have a 'free_fn' anyway.

author
#41 lore raw

Re: [PATCH v2 05/31] x86/virt/tdx: Extend tdx_page_array to support IOMMU_MT

Quoted 6 lines · Kai
Quoted 5 lines · Xu Yilun

IOMMU_MT is another TDX Module defined structure similar to HPA_ARRAY_T and HPA_LIST_INFO. The difference is it requires multi-order contiguous pages for some entries. It adds an additional NUM_PAGES field for every multi-order page entry.

Quoted 7 lines
Quoted 3 lines

Add a dedicated allocation helper for IOMMU_MT. Fortunately put_page() works well for both single pages and multi-order folios, simplifying the cleanup logic for all allocation methods.

Well I guess you can have a 'free_fn' to free the pages you allocated via 'alloc_fn'? Will this simplify the code and at least keep tdx_page_array implementation cleaner?

mm.. I think code would be simplified with less callbacks.

But anyway, the need for the alloc_fn becomes a sign for me to think about separating the memory allocation and struct tdx_page_array construction. Especially that the IOMMU_MT needs specialized memory layout so better managed by the kernel driver who really uses IOMMU_MT.

Quoted 3 lines

It's strange that you only have a 'alloc_fn' but doesn't have a 'free_fn' anyway.

author
#42 lore raw

PATCH v2 6/31 x86/virt/tdx: Read global metadata for TDX Module Extensions/Connect

In reply to Xu Yilun

Add reading of the global metadata for TDX Module Extensions & TDX Connect. Add them in a batch as TDX Connect is currently the only user of TDX Module Extensions and no way to initialize TDX Module Extensions without firstly enabling TDX Connect.

TDX Module Extensions & TDX Connect are optional features enumerated by TDX_FEATURES0. Check the TDX_FEATURES0 before reading these metadata to avoid failing the whole TDX initialization.

---
 arch/x86/include/asm/tdx.h                  |  2 ++
 arch/x86/include/asm/tdx_global_metadata.h  | 14 ++++++++
 arch/x86/virt/vmx/tdx/tdx_global_metadata.c | 36 +++++++++++++++++++++
 3 files changed, 52 insertions(+)
arch/x86/include/asm/tdx.h +2 -0
index d5f1d7b7d1e7..d7605235aa9b 100644
@@ -129,7 +129,9 @@ int tdx_enable(void);
 const char *tdx_dump_mce_info(struct mce *m);
 
 /* Bit definitions of TDX_FEATURES0 metadata field */
+#define TDX_FEATURES0_TDXCONNECT	BIT_ULL(6)
 #define TDX_FEATURES0_NO_RBP_MOD	BIT_ULL(18)
+#define TDX_FEATURES0_EXT		BIT_ULL(39)
 
 const struct tdx_sys_info *tdx_get_sysinfo(void);
 
arch/x86/include/asm/tdx_global_metadata.h +14 -0
index 060a2ad744bf..e7948bca671a 100644
@@ -34,11 +34,25 @@ struct tdx_sys_info_td_conf {
 	u64 cpuid_config_values[128][2];
 };
 
+struct tdx_sys_info_ext {
+	u16 memory_pool_required_pages;
+	u8 ext_required;
+};
+
+struct tdx_sys_info_connect {
+	u16 ide_mt_page_count;
+	u16 spdm_mt_page_count;
+	u16 iommu_mt_page_count;
+	u16 spdm_max_dev_info_pages;
+};
+
 struct tdx_sys_info {
 	struct tdx_sys_info_features features;
 	struct tdx_sys_info_tdmr tdmr;
 	struct tdx_sys_info_td_ctrl td_ctrl;
 	struct tdx_sys_info_td_conf td_conf;
+	struct tdx_sys_info_ext ext;
+	struct tdx_sys_info_connect connect;
 };
 
 #endif
arch/x86/virt/vmx/tdx/tdx_global_metadata.c +36 -0
index 13ad2663488b..a07f1e7b18e8 100644
@@ -85,6 +85,36 @@ static int get_tdx_sys_info_td_conf(struct tdx_sys_info_td_conf *sysinfo_td_conf
 	return ret;
 }
 
+static int get_tdx_sys_info_ext(struct tdx_sys_info_ext *sysinfo_ext)
+{
+	int ret = 0;
+	u64 val;
+
+	if (!ret && !(ret = read_sys_metadata_field(0x3100000100000000, &val)))
+		sysinfo_ext->memory_pool_required_pages = val;
+	if (!ret && !(ret = read_sys_metadata_field(0x3100000100000001, &val)))
+		sysinfo_ext->ext_required = val;
+
+	return ret;
+}
+
+static int get_tdx_sys_info_connect(struct tdx_sys_info_connect *sysinfo_connect)
+{
+	int ret = 0;
+	u64 val;
+
+	if (!ret && !(ret = read_sys_metadata_field(0x3000000100000001, &val)))
+		sysinfo_connect->ide_mt_page_count = val;
+	if (!ret && !(ret = read_sys_metadata_field(0x3000000100000002, &val)))
+		sysinfo_connect->spdm_mt_page_count = val;
+	if (!ret && !(ret = read_sys_metadata_field(0x3000000100000003, &val)))
+		sysinfo_connect->iommu_mt_page_count = val;
+	if (!ret && !(ret = read_sys_metadata_field(0x3000000100000007, &val)))
+		sysinfo_connect->spdm_max_dev_info_pages = val;
+
+	return ret;
+}
+
 static int get_tdx_sys_info(struct tdx_sys_info *sysinfo)
 {
 	int ret = 0;
@@ -94,5 +124,11 @@ static int get_tdx_sys_info(struct tdx_sys_info *sysinfo)
 	ret = ret ?: get_tdx_sys_info_td_ctrl(&sysinfo->td_ctrl);
 	ret = ret ?: get_tdx_sys_info_td_conf(&sysinfo->td_conf);
 
+	if (sysinfo->features.tdx_features0 & TDX_FEATURES0_EXT)
+		ret = ret ?: get_tdx_sys_info_ext(&sysinfo->ext);
+
+	if (sysinfo->features.tdx_features0 & TDX_FEATURES0_TDXCONNECT)
+		ret = ret ?: get_tdx_sys_info_connect(&sysinfo->connect);
+
 	return ret;
 }
Signature · 1 line
2.25.1
#43 lore raw

Re: [PATCH v2 06/31] x86/virt/tdx: Read global metadata for TDX Module Extensions/Connect

Quoted 65 lines · Xu Yilun

Add reading of the global metadata for TDX Module Extensions & TDX Connect. Add them in a batch as TDX Connect is currently the only user of TDX Module Extensions and no way to initialize TDX Module Extensions without firstly enabling TDX Connect.

TDX Module Extensions & TDX Connect are optional features enumerated by TDX_FEATURES0. Check the TDX_FEATURES0 before reading these metadata to avoid failing the whole TDX initialization.

---
  arch/x86/include/asm/tdx.h                  |  2 ++
  arch/x86/include/asm/tdx_global_metadata.h  | 14 ++++++++
  arch/x86/virt/vmx/tdx/tdx_global_metadata.c | 36 +++++++++++++++++++++
  3 files changed, 52 insertions(+)
arch/x86/include/asm/tdx.h +2 -0
index d5f1d7b7d1e7..d7605235aa9b 100644
@@ -129,7 +129,9 @@ int tdx_enable(void);
  const char *tdx_dump_mce_info(struct mce *m);
 
  /* Bit definitions of TDX_FEATURES0 metadata field */
+#define TDX_FEATURES0_TDXCONNECT	BIT_ULL(6)
  #define TDX_FEATURES0_NO_RBP_MOD	BIT_ULL(18)
+#define TDX_FEATURES0_EXT		BIT_ULL(39)
 
  const struct tdx_sys_info *tdx_get_sysinfo(void);
 
arch/x86/include/asm/tdx_global_metadata.h +14 -0
index 060a2ad744bf..e7948bca671a 100644
@@ -34,11 +34,25 @@ struct tdx_sys_info_td_conf {
  	u64 cpuid_config_values[128][2];
  };
 
+struct tdx_sys_info_ext {
+	u16 memory_pool_required_pages;
+	u8 ext_required;
+};
+
+struct tdx_sys_info_connect {
+	u16 ide_mt_page_count;
+	u16 spdm_mt_page_count;
+	u16 iommu_mt_page_count;
+	u16 spdm_max_dev_info_pages;
+};
+
  struct tdx_sys_info {
  	struct tdx_sys_info_features features;
  	struct tdx_sys_info_tdmr tdmr;
  	struct tdx_sys_info_td_ctrl td_ctrl;
  	struct tdx_sys_info_td_conf td_conf;
+	struct tdx_sys_info_ext ext;
+	struct tdx_sys_info_connect connect;
  };
 
  #endif
arch/x86/virt/vmx/tdx/tdx_global_metadata.c +0 -0
index 13ad2663488b..a07f1e7b18e8 100644

arch/x86/virt/vmx/tdx/tdx_global_metadata.c static int get_tdx_sys_info_td_conf(struct tdx_sys_info_td_conf *sysinfo_td_conf · line 93

Quoted 9 lines
arch/x86/virt/vmx/tdx/tdx_global_metadata.c +6 -0
@@ -85,6 +85,36 @@ static int get_tdx_sys_info_td_conf(struct tdx_sys_info_td_conf *sysinfo_td_conf
  	return ret;
  }
 
+static int get_tdx_sys_info_ext(struct tdx_sys_info_ext *sysinfo_ext)
+{
+	int ret = 0;
+	u64 val;
+
+	if (!ret && !(ret = read_sys_metadata_field(0x3100000100000000, &val)))

nit: This is likely generated by a script/llm because I see no other explanation why !ret is being checked here...

arch/x86/virt/vmx/tdx/tdx_global_metadata.c static int get_tdx_sys_info_td_conf(struct tdx_sys_info_td_conf *sysinfo_td_conf · line 106

Quoted 13 lines
arch/x86/virt/vmx/tdx/tdx_global_metadata.c +13 -0
+		sysinfo_ext->memory_pool_required_pages = val;
+	if (!ret && !(ret = read_sys_metadata_field(0x3100000100000001, &val)))
+		sysinfo_ext->ext_required = val;
+
+	return ret;
+}
+
+static int get_tdx_sys_info_connect(struct tdx_sys_info_connect *sysinfo_connect)
+{
+	int ret = 0;
+	u64 val;
+
+	if (!ret && !(ret = read_sys_metadata_field(0x3000000100000001, &val)))

Ditto
<snip>

author
#44 lore raw

Re: [PATCH v2 06/31] x86/virt/tdx: Read global metadata for TDX Module Extensions/Connect

Quoted 10 lines

arch/x86/virt/vmx/tdx/tdx_global_metadata.c static int get_tdx_sys_info_td_conf(struct tdx_sys_info_td_conf *sysinfo_td_conf · line 93

Quoted 6 lines
arch/x86/virt/vmx/tdx/tdx_global_metadata.c +6 -0
+static int get_tdx_sys_info_ext(struct tdx_sys_info_ext *sysinfo_ext)
+{
+	int ret = 0;
+	u64 val;
+
+	if (!ret && !(ret = read_sys_metadata_field(0x3100000100000000, &val)))

nit: This is likely generated by a script/llm because I see no other explanation why !ret is being checked here...

Yes, the file was generated by script at the beginning.

#45 lore raw

Re: [PATCH v2 06/31] x86/virt/tdx: Read global metadata for TDX Module Extensions/Connect

In reply to Xu Yilun
Quoted 8 lines · Xu Yilun

Add reading of the global metadata for TDX Module Extensions & TDX Connect. Add them in a batch as TDX Connect is currently the only user of TDX Module Extensions and no way to initialize TDX Module Extensions without firstly enabling TDX Connect.

TDX Module Extensions & TDX Connect are optional features enumerated by TDX_FEATURES0. Check the TDX_FEATURES0 before reading these metadata to avoid failing the whole TDX initialization.

Maybe it's better to split this patch into two, one to read generic "TDX Module Extension" related global metadata, and the other to read TDX Connect specific ones?

They are logically two separate things anyway. And there are other features also need to enable TDX Module Extensions (e.g., NRX for migration), and we can just reuse the generic metadata patch from this series.

author
#46 lore raw

Re: [PATCH v2 06/31] x86/virt/tdx: Read global metadata for TDX Module Extensions/Connect

Quoted 10 lines · Kai
Quoted 8 lines · Xu Yilun

Add reading of the global metadata for TDX Module Extensions & TDX Connect. Add them in a batch as TDX Connect is currently the only user of TDX Module Extensions and no way to initialize TDX Module Extensions without firstly enabling TDX Connect.

TDX Module Extensions & TDX Connect are optional features enumerated by TDX_FEATURES0. Check the TDX_FEATURES0 before reading these metadata to avoid failing the whole TDX initialization.

Quoted 7 lines

Maybe it's better to split this patch into two, one to read generic "TDX Module Extension" related global metadata, and the other to read TDX Connect specific ones?

They are logically two separate things anyway. And there are other features also need to enable TDX Module Extensions (e.g., NRX for migration), and we can just reuse the generic metadata patch from this series.

Will do.

#47 lore raw

Re: [PATCH v2 06/31] x86/virt/tdx: Read global metadata for TDX Module Extensions/Connect

In reply to Xu Yilun
Quoted 8 lines · Xu Yilun

Add reading of the global metadata for TDX Module Extensions & TDX Connect. Add them in a batch as TDX Connect is currently the only user of TDX Module Extensions and no way to initialize TDX Module Extensions without firstly enabling TDX Connect.

TDX Module Extensions & TDX Connect are optional features enumerated by TDX_FEATURES0. Check the TDX_FEATURES0 before reading these metadata to avoid failing the whole TDX initialization.

I think it is important to distinguish "optional" module features vs required Linux features. Linux requires all features that a module advertises to succeed at core TDX init time.

Otherwise, this looks ok / consistent with other metadata reading. It sets the precedent that if TDX Connect is advertised it must succeed all core initialization.

author
#48 lore raw

Re: [PATCH v2 06/31] x86/virt/tdx: Read global metadata for TDX Module Extensions/Connect

Quoted 6 lines · Dan Williams
Quoted 5 lines · Xu Yilun

Add reading of the global metadata for TDX Module Extensions & TDX Connect. Add them in a batch as TDX Connect is currently the only user of TDX Module Extensions and no way to initialize TDX Module Extensions without firstly enabling TDX Connect.

Quoted 7 lines
Quoted 3 lines

TDX Module Extensions & TDX Connect are optional features enumerated by TDX_FEATURES0. Check the TDX_FEATURES0 before reading these metadata to avoid failing the whole TDX initialization.

I think it is important to distinguish "optional" module features vs required Linux features. Linux requires all features that a module advertises to succeed at core TDX init time.

Agree. But I want to reduce the scope to only about metadata reading in this patch. So:

TDX Module Extensions is an optional features enumerated by TDX_FEATURES0. But in the implementation, Linux requires that all features that a Module advertises must have a complete, valid set of metadata, and the check must succeed at core TDX initialization time.

Check TDX_FEATURES0 before reading these metadata. If a feature is advertised, a failure in reading associated metadata causes the whole TDX initialization to fail, otherwise skip.

Quoted 4 lines

Otherwise, this looks ok / consistent with other metadata reading. It sets the precedent that if TDX Connect is advertised it must succeed all core initialization.

author
#49 lore raw

PATCH v2 7/31 x86/virt/tdx: Embed version info in SEAMCALL leaf function definitions

In reply to Xu Yilun

Embed version information in SEAMCALL leaf function definitions rather than let the caller open code them. For now, only TDH.VP.INIT is involved.

Don't bother the caller to choose the SEAMCALL version if unnecessary. New version SEAMCALLs are guaranteed to be backward compatible, so ideally kernel doesn't need to keep version history and only uses the latest version SEAMCALLs.

The concern is some old TDX Modules don't recognize new version SEAMCALLs. Multiple SEAMCALL versions co-exist when kernel should support these old Modules. As time goes by, the old Modules deprecate and old version SEAMCALL definitions should disappear.

The old TDX Modules that only support TDH.VP.INIT v0 are all deprecated, so only provide the latest (v1) definition.

---
 arch/x86/virt/vmx/tdx/tdx.h | 23 ++++++++++++++---------
 arch/x86/virt/vmx/tdx/tdx.c |  4 ++--
 2 files changed, 16 insertions(+), 11 deletions(-)
arch/x86/virt/vmx/tdx/tdx.h +14 -9
index c641b4632826..e5a9331df451 100644
@@ -2,6 +2,7 @@
 #ifndef _X86_VIRT_TDX_H
 #define _X86_VIRT_TDX_H
 
+#include <linux/bitfield.h>
 #include <linux/bits.h>
 
 /*
@@ -11,6 +12,18 @@
  * architectural definitions come first.
  */
 
+/*
+ * SEAMCALL leaf:
+ *
+ * Bit 15:0	Leaf number
+ * Bit 23:16	Version number
+ */
+#define SEAMCALL_LEAF			GENMASK(15, 0)
+#define SEAMCALL_VER			GENMASK(23, 16)
+
+#define SEAMCALL_LEAF_VER(l, v)		(FIELD_PREP(SEAMCALL_LEAF, l) | \
+					 FIELD_PREP(SEAMCALL_VER, v))
+
 /*
  * TDX module SEAMCALL leaf functions
  */
@@ -31,7 +44,7 @@
 #define TDH_VP_CREATE			10
 #define TDH_MNG_KEY_FREEID		20
 #define TDH_MNG_INIT			21
-#define TDH_VP_INIT			22
+#define TDH_VP_INIT			SEAMCALL_LEAF_VER(22, 1)
 #define TDH_PHYMEM_PAGE_RDMD		24
 #define TDH_VP_RD			26
 #define TDH_PHYMEM_PAGE_RECLAIM		28
@@ -47,14 +60,6 @@
 #define TDH_VP_WR			43
 #define TDH_SYS_CONFIG			45
 
-/*
- * SEAMCALL leaf:
- *
- * Bit 15:0	Leaf number
- * Bit 23:16	Version number
- */
-#define TDX_VERSION_SHIFT		16
-
 /* TDX page types */
 #define	PT_NDA		0x0
 #define	PT_RSVD		0x1
arch/x86/virt/vmx/tdx/tdx.c +2 -2
index 2b17e0f73dac..130214933c2f 100644
@@ -2202,8 +2202,8 @@ u64 tdh_vp_init(struct tdx_vp *vp, u64 initial_rcx, u32 x2apicid)
 		.r8 = x2apicid,
 	};
 
-	/* apicid requires version == 1. */
-	return seamcall(TDH_VP_INIT | (1ULL << TDX_VERSION_SHIFT), &args);
+	/* apicid requires version == 1. See TDH_VP_INIT definition.*/
+	return seamcall(TDH_VP_INIT, &args);
 }
 EXPORT_SYMBOL_FOR_KVM(tdh_vp_init);
 
Signature · 1 line
2.25.1
author
#50 lore raw

PATCH v2 8/31 x86/virt/tdx: Configure TDX Module with optional TDX Connect feature

In reply to Xu Yilun

TDX Module supports optional TDX features (e.g. TDX Connect & TDX Module Extensions) that won't be enabled by default. It extends TDH.SYS.CONFIG for host to choose to enable them on bootup.

Call TDH.SYS.CONFIG with a new bitmap input parameter to specify which features to enable. The bitmap uses the same definitions as TDX_FEATURES0. But note not all bits in TDX_FEATURES0 are valid for configuration, e.g. TDX Module Extensions is a service that supports TDX Connect, it is implicitly enabled when TDX Connect is enabled. Setting TDX_FEATURES0_EXT in the bitmap has no effect.

TDX Module advances the version of TDH.SYS.CONFIG for the change, so use the latest version (v1) for optional feature enabling. But supporting existing Modules which only support v0 is still necessary until they are deprecated, enumerate via TDX_FEATURES0 to decide which version to use.

TDX Module updates global metadata when optional features are enabled. Host should update the cached tdx_sysinfo to reflect these changes.

---
 arch/x86/virt/vmx/tdx/tdx.h |  3 ++-
 arch/x86/virt/vmx/tdx/tdx.c | 16 +++++++++++++++-
 2 files changed, 17 insertions(+), 2 deletions(-)
arch/x86/virt/vmx/tdx/tdx.h +2 -1
index e5a9331df451..870bb75da3ba 100644
@@ -58,7 +58,8 @@
 #define TDH_PHYMEM_CACHE_WB		40
 #define TDH_PHYMEM_PAGE_WBINVD		41
 #define TDH_VP_WR			43
-#define TDH_SYS_CONFIG			45
+#define TDH_SYS_CONFIG_V0		45
+#define TDH_SYS_CONFIG			SEAMCALL_LEAF_VER(TDH_SYS_CONFIG_V0, 1)
 
 /* TDX page types */
 #define	PT_NDA		0x0
arch/x86/virt/vmx/tdx/tdx.c +15 -1
index 130214933c2f..0c5d6bdd810f 100644
@@ -1353,6 +1353,7 @@ static int construct_tdmrs(struct list_head *tmb_list,
 static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid)
 {
 	struct tdx_module_args args = {};
+	u64 seamcall_fn = TDH_SYS_CONFIG_V0;
 	u64 *tdmr_pa_array;
 	size_t array_sz;
 	int i, ret;
@@ -1377,7 +1378,15 @@ static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid)
 	args.rcx = __pa(tdmr_pa_array);
 	args.rdx = tdmr_list->nr_consumed_tdmrs;
 	args.r8 = global_keyid;
-	ret = seamcall_prerr(TDH_SYS_CONFIG, &args);
+
+	if (tdx_sysinfo.features.tdx_features0 & TDX_FEATURES0_TDXCONNECT) {
+		args.r9 |= TDX_FEATURES0_TDXCONNECT;
+		args.r11 = ktime_get_real_seconds();
+		/* These parameters requires version >= 1 */
+		seamcall_fn = TDH_SYS_CONFIG;
+	}
+
+	ret = seamcall_prerr(seamcall_fn, &args);
 
 	/* Free the array as it is not required anymore. */
 	kfree(tdmr_pa_array);
@@ -1537,6 +1546,11 @@ static int init_tdx_module(void)
 	if (ret)
 		goto err_free_pamts;
 
+	/* configuration to tdx module may change tdx_sysinfo, update it */
+	ret = get_tdx_sys_info(&tdx_sysinfo);
+	if (ret)
+		goto err_reset_pamts;
+
 	/* Config the key of global KeyID on all packages */
 	ret = config_global_keyid();
 	if (ret)
Signature · 1 line
2.25.1
#51 lore raw

Re: [PATCH v2 08/31] x86/virt/tdx: Configure TDX Module with optional TDX Connect feature

Quoted 33 lines · Xu Yilun

TDX Module supports optional TDX features (e.g. TDX Connect & TDX Module Extensions) that won't be enabled by default. It extends TDH.SYS.CONFIG for host to choose to enable them on bootup.

Call TDH.SYS.CONFIG with a new bitmap input parameter to specify which features to enable. The bitmap uses the same definitions as TDX_FEATURES0. But note not all bits in TDX_FEATURES0 are valid for configuration, e.g. TDX Module Extensions is a service that supports TDX Connect, it is implicitly enabled when TDX Connect is enabled. Setting TDX_FEATURES0_EXT in the bitmap has no effect.

TDX Module advances the version of TDH.SYS.CONFIG for the change, so use the latest version (v1) for optional feature enabling. But supporting existing Modules which only support v0 is still necessary until they are deprecated, enumerate via TDX_FEATURES0 to decide which version to use.

TDX Module updates global metadata when optional features are enabled. Host should update the cached tdx_sysinfo to reflect these changes.

---
  arch/x86/virt/vmx/tdx/tdx.h |  3 ++-
  arch/x86/virt/vmx/tdx/tdx.c | 16 +++++++++++++++-
  2 files changed, 17 insertions(+), 2 deletions(-)
arch/x86/virt/vmx/tdx/tdx.h +0 -0
index e5a9331df451..870bb75da3ba 100644

arch/x86/virt/vmx/tdx/tdx.h line 62

Quoted 6 lines
arch/x86/virt/vmx/tdx/tdx.h +2 -1
@@ -58,7 +58,8 @@
  #define TDH_PHYMEM_CACHE_WB		40
  #define TDH_PHYMEM_PAGE_WBINVD		41
  #define TDH_VP_WR			43
-#define TDH_SYS_CONFIG			45
+#define TDH_SYS_CONFIG_V0		45
+#define TDH_SYS_CONFIG			SEAMCALL_LEAF_VER(TDH_SYS_CONFIG_V0, 1)

Since newer versions of tdx module apis are backwards compatible with older ones, and v0 are actually deprecated why have both definitions?

<snip>

author
#52 lore raw

Re: [PATCH v2 08/31] x86/virt/tdx: Configure TDX Module with optional TDX Connect feature

Quoted 31 lines · Nikolay Borisov
Quoted 28 lines · Xu Yilun

TDX Module supports optional TDX features (e.g. TDX Connect & TDX Module Extensions) that won't be enabled by default. It extends TDH.SYS.CONFIG for host to choose to enable them on bootup.

Call TDH.SYS.CONFIG with a new bitmap input parameter to specify which features to enable. The bitmap uses the same definitions as TDX_FEATURES0. But note not all bits in TDX_FEATURES0 are valid for configuration, e.g. TDX Module Extensions is a service that supports TDX Connect, it is implicitly enabled when TDX Connect is enabled. Setting TDX_FEATURES0_EXT in the bitmap has no effect.

TDX Module advances the version of TDH.SYS.CONFIG for the change, so use the latest version (v1) for optional feature enabling. But supporting existing Modules which only support v0 is still necessary until they are deprecated, enumerate via TDX_FEATURES0 to decide which version to use.

TDX Module updates global metadata when optional features are enabled. Host should update the cached tdx_sysinfo to reflect these changes.

---
  arch/x86/virt/vmx/tdx/tdx.h |  3 ++-
  arch/x86/virt/vmx/tdx/tdx.c | 16 +++++++++++++++-
  2 files changed, 17 insertions(+), 2 deletions(-)
Quoted 14 lines

arch/x86/virt/vmx/tdx/tdx.h line 62

Quoted 11 lines
arch/x86/virt/vmx/tdx/tdx.h +2 -1
index e5a9331df451..870bb75da3ba 100644
@@ -58,7 +58,8 @@
  #define TDH_PHYMEM_CACHE_WB		40
  #define TDH_PHYMEM_PAGE_WBINVD		41
  #define TDH_VP_WR			43
-#define TDH_SYS_CONFIG			45
+#define TDH_SYS_CONFIG_V0		45
+#define TDH_SYS_CONFIG			SEAMCALL_LEAF_VER(TDH_SYS_CONFIG_V0, 1)

Since newer versions of tdx module apis are backwards compatible with older ones, and v0 are actually deprecated why have both definitions?

No, for this TDH_SYS_CONFIG SEAMCALL, the situation is different. There is no public TDX Module release yet to support TDH_SYS_CONFIG_V1. So I can't say v0 is deprecated.

Quoted 3 lines

<snip>

#53 lore raw

Re: [PATCH v2 08/31] x86/virt/tdx: Configure TDX Module with optional TDX Connect feature

In reply to Xu Yilun
Quoted 6 lines

TDX Module updates global metadata when optional features are enabled. Host should update the cached tdx_sysinfo to reflect these changes.

[...]

Quoted 25 lines

static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid) {

arch/x86/virt/vmx/tdx/tdx.c +10 -1
 	struct tdx_module_args args = {};
+	u64 seamcall_fn = TDH_SYS_CONFIG_V0;
 	u64 *tdmr_pa_array;
 	size_t array_sz;
 	int i, ret;
@@ -1377,7 +1378,15 @@ static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid)
 	args.rcx = __pa(tdmr_pa_array);
 	args.rdx = tdmr_list->nr_consumed_tdmrs;
 	args.r8 = global_keyid;
-	ret = seamcall_prerr(TDH_SYS_CONFIG, &args);
+
+	if (tdx_sysinfo.features.tdx_features0 & TDX_FEATURES0_TDXCONNECT) {
+		args.r9 |= TDX_FEATURES0_TDXCONNECT;
+		args.r11 = ktime_get_real_seconds();
+		/* These parameters requires version >= 1 */
+		seamcall_fn = TDH_SYS_CONFIG;
+	}
+
+	ret = seamcall_prerr(seamcall_fn, &args);
 
 	/* Free the array as it is not required anymore. */
 	kfree(tdmr_pa_array);

arch/x86/virt/vmx/tdx/tdx.c static int init_tdx_module(void) · line 1553

Quoted 8 lines
arch/x86/virt/vmx/tdx/tdx.c +5 -0
@@ -1537,6 +1546,11 @@ static int init_tdx_module(void)
 	if (ret)
 		goto err_free_pamts;
 
+	/* configuration to tdx module may change tdx_sysinfo, update it */
+	ret = get_tdx_sys_info(&tdx_sysinfo);
+	if (ret)
+		goto err_reset_pamts;
+

How about put this into config_tdx_module()?

In this way you can only update global metadata when there's new feature being opted in, and at the meantime, avoid making init_tdx_module() more complicated.

author
#54 lore raw

Re: [PATCH v2 08/31] x86/virt/tdx: Configure TDX Module with optional TDX Connect feature

Quoted 44 lines · Kai
Quoted 6 lines

TDX Module updates global metadata when optional features are enabled. Host should update the cached tdx_sysinfo to reflect these changes.

[...]

Quoted 25 lines

static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid) {

arch/x86/virt/vmx/tdx/tdx.c +10 -1
 	struct tdx_module_args args = {};
+	u64 seamcall_fn = TDH_SYS_CONFIG_V0;
 	u64 *tdmr_pa_array;
 	size_t array_sz;
 	int i, ret;
@@ -1377,7 +1378,15 @@ static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid)
 	args.rcx = __pa(tdmr_pa_array);
 	args.rdx = tdmr_list->nr_consumed_tdmrs;
 	args.r8 = global_keyid;
-	ret = seamcall_prerr(TDH_SYS_CONFIG, &args);
+
+	if (tdx_sysinfo.features.tdx_features0 & TDX_FEATURES0_TDXCONNECT) {
+		args.r9 |= TDX_FEATURES0_TDXCONNECT;
+		args.r11 = ktime_get_real_seconds();
+		/* These parameters requires version >= 1 */
+		seamcall_fn = TDH_SYS_CONFIG;
+	}
+
+	ret = seamcall_prerr(seamcall_fn, &args);
 
 	/* Free the array as it is not required anymore. */
 	kfree(tdmr_pa_array);

arch/x86/virt/vmx/tdx/tdx.c static int init_tdx_module(void) · line 1553

Quoted 8 lines
arch/x86/virt/vmx/tdx/tdx.c +5 -0
@@ -1537,6 +1546,11 @@ static int init_tdx_module(void)
 	if (ret)
 		goto err_free_pamts;
 
+	/* configuration to tdx module may change tdx_sysinfo, update it */
+	ret = get_tdx_sys_info(&tdx_sysinfo);
+	if (ret)
+		goto err_reset_pamts;
+

How about put this into config_tdx_module()?

Quoted 1 line

In this way you can only update global metadata when there's new feature

mm.. personally I don't like such subtle control, especially when

- We expect one or more features are doomed to be enabled.
- We are pursuing simple TDX enabling process.
- This is still not the exact control. If we really want to be precise, should check feature by feature, that's not worth it.

Quoted 2 lines

being opted in, and at the meantime, avoid making init_tdx_module() more complicated.

#55 lore raw

Re: [PATCH v2 08/31] x86/virt/tdx: Configure TDX Module with optional TDX Connect feature

Quoted 9 lines
Quoted 9 lines

arch/x86/virt/vmx/tdx/tdx.c static int init_tdx_module(void) · line 1553

Quoted 6 lines
arch/x86/virt/vmx/tdx/tdx.c +5 -0
+	/* configuration to tdx module may change tdx_sysinfo, update it */
+	ret = get_tdx_sys_info(&tdx_sysinfo);
+	if (ret)
+		goto err_reset_pamts;
+

How about put this into config_tdx_module()?

Quoted 5 lines
Quoted 1 line

In this way you can only update global metadata when there's new feature

mm.. personally I don't like such subtle control, especially when

- We expect one or more features are doomed to be enabled.

The new kernel may also run on old hardware where no ext features can be supported.

Quoted 4 lines

- We are pursuing simple TDX enabling process.
- This is still not the exact control. If we really want to be precise, should check feature by feature, that's not worth it.

For the record I am not wanting "exact control". It's totally fine to me to get global metadata again if there's any ext feature enabled.

And my main comment actually is that init_tdx_module() is already having many steps, so when the new code could logically fit somewhere else we should stop making init_tdx_module() more complicated.

But no strong opinion, will leave to you.

#56 lore raw

Re: [PATCH v2 08/31] x86/virt/tdx: Configure TDX Module with optional TDX Connect feature

In reply to Xu Yilun
Quoted 27 lines

static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid) {

arch/x86/virt/vmx/tdx/tdx.c +10 -1
 	struct tdx_module_args args = {};
+	u64 seamcall_fn = TDH_SYS_CONFIG_V0;
 	u64 *tdmr_pa_array;
 	size_t array_sz;
 	int i, ret;
@@ -1377,7 +1378,15 @@ static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid)
 	args.rcx = __pa(tdmr_pa_array);
 	args.rdx = tdmr_list->nr_consumed_tdmrs;
 	args.r8 = global_keyid;
-	ret = seamcall_prerr(TDH_SYS_CONFIG, &args);
+
+	if (tdx_sysinfo.features.tdx_features0 & TDX_FEATURES0_TDXCONNECT) {
+		args.r9 |= TDX_FEATURES0_TDXCONNECT;
+		args.r11 = ktime_get_real_seconds();
+		/* These parameters requires version >= 1 */
+		seamcall_fn = TDH_SYS_CONFIG;
+	}
+
+	ret = seamcall_prerr(seamcall_fn, &args);
 
 	/* Free the array as it is not required anymore. */
 	kfree(tdmr_pa_array);
@@ -1537,6 +1546,11 @@ static int init_tdx_module(void)
 	if (ret)

arch/x86/virt/vmx/tdx/tdx.c static int init_tdx_module(void) · line 1556

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +5 -0
 		goto err_free_pamts;
 
+	/* configuration to tdx module may change tdx_sysinfo, update it */
+	ret = get_tdx_sys_info(&tdx_sysinfo);
+	if (ret)
+		goto err_reset_pamts;
+
 	/* Config the key of global KeyID on all packages */
 	ret = config_global_keyid();
 	if (ret)

Maybe a more generic comment:

I don't quite like hard-coding opt-in TDX_FEATURES0_TDXCONNECT inside config_tdx_module(), especially currently we just unconditionally opt it in if the module support this feature.

Initializing TDX Connect (and other features via TDX Module Extensions) consumes more memory. It would be better if we can choose to opt-in when the kernel has enabled TDX Connect (or any other feature via TDX module Extensions) in the Kconfig.

Unfortunately we need to opt-in all these features together during module initialization, so we cannot make tdx_enable() to accept the additional features to enable, and in each in-kernel TDX user, call tdx_enable() with the new feature that that TDX user concerns.

But I think it makes sense to have a dedicated place to calculate all opt-in features. E.g., assuming we eventually are going to support TDX Connect and live migration:

static u64 get_ext_features_tdx_connect(struct tdx_sys_info * sysinfo)

{
	if (!IS_ENABLED(TDX_CONNECT))
		return 0;

	return sysinfo->features.tdx_features0 & TDX_FEATURES0_TDXCONNECT ?
		TDX_FEATURES0_TDXCONNECT : 0;
}

static u64 get_ext_features_live_migration(struct tdx_sys_info *sysinfo)

{
	u64 mig_features = TDX_FEATURES0_NRX | TDX_FEATURES0_NON_BLOCKING;

	if (!IS_ENABLED(TDX_LIVE_MIGRATION))
		return 0;

	return sysinfo->features.tdx_features0 & mig_features;
}

static u64 calculate_ext_features(struct tdx_sys_info *sysinfo)

{
	u64 ext_features = 0;

	ext_features |= get_ext_features_tdx_connect(sysinfo);

	ext_features |= get_ext_features_live_migration(sysinfo);

	return ext_features;
}

int init_tdx_module()

{
	u64 ext_features = calculate_ext_features(&tdx_sysinfo);

	ret = config_tdx_module(&tdx_tdmr_list, &tdx_global_keyid,
				ext_features);

	/* do other initializations like TDH.SYS.KEY.CONFIG */
	...
	/*
	 * TDX Module Extension features must be initialized
	 * after TDH.SYS.KEY.CONFIG.
	 */
	if (ext_features)
		ret = init_tdx_ext();	

	...
}

One nasty thing is per public spec R11 of TDH.SYS.CONFIG needs to be RTC if TDX_CONNECT is on, so we still need some special handing in config_tdx_module():

	if (ext_features & TDX_RFEAURES0_TDXCONNECT)
		args.r11 = ktime_get_real_seconds();

But I think this is acceptable.

#57 lore raw

Re: [PATCH v2 08/31] x86/virt/tdx: Configure TDX Module with optional TDX Connect feature

Quoted 10 lines · Kai

Maybe a more generic comment:

I don't quite like hard-coding opt-in TDX_FEATURES0_TDXCONNECT inside config_tdx_module(), especially currently we just unconditionally opt it in if the module support this feature.

Initializing TDX Connect (and other features via TDX Module Extensions) consumes more memory.  It would be better if we can choose to opt-in when the kernel has enabled TDX Connect (or any other feature via TDX module Extensions) in the Kconfig.

Better how? TDX uses a lot of memory. There are many possible optimizations to reduce this. Why focus on this one? Do we think any TDX users would actually reconfigure there kernel for this reason?

I mean, I don't actually know how much memory this is, but to me the reasoning doesn't seem in balance with the wider TDX situation.

#58 lore raw

Re: [PATCH v2 08/31] x86/virt/tdx: Configure TDX Module with optional TDX Connect feature

Quoted 12 lines · Rick P
Quoted 10 lines · Kai

Maybe a more generic comment:

I don't quite like hard-coding opt-in TDX_FEATURES0_TDXCONNECT inside config_tdx_module(), especially currently we just unconditionally opt it in if the module support this feature.

Initializing TDX Connect (and other features via TDX Module Extensions) consumes more memory.  It would be better if we can choose to opt-in when the kernel has enabled TDX Connect (or any other feature via TDX module Extensions) in the Kconfig.

Quoted 6 lines

Better how? TDX uses a lot of memory. There are many possible optimizations to reduce this. Why focus on this one? Do we think any TDX users would actually reconfigure there kernel for this reason?

I mean, I don't actually know how much memory this is, but to me the reasoning doesn't seem in balance with the wider TDX situation.

In another patch, Yilun said:

For now, TDX Module Extensions consume relatively large amount of memory (~50MB).

Distros tend to enable all I assume. I guess the CSPs tend to enable all as well, but I am not sure whether CSPs can also choose to only enable basic TDX w/o other features like runtime update, TDX Connect etc, depending on how they want to "sell" TDX VMs.

E.g., I assume a TD without GPU passthrough could be cheaper the one which has, right? Can the CSPs host such TDs on dedicated machine pools? Can they choose to disable TDX Connect on these machines?

They might not care about losing ~50M memory, though, but that's a different story, and it could be more in the future.

My thinking is, this series actually introduced a new "TDX_CONNNECT" Kconfig, so why not only consume the memory when it's on?

At last, just my 2cents, I kinda overall don't agree we always assume "everything will be on" but neglect avoiding unnecessary code/cost that is not reachable when some option is off. That would defeat the purpose of having Kconfig option.

#59 lore raw

Re: [PATCH v2 08/31] x86/virt/tdx: Configure TDX Module with optional TDX Connect feature

On 4/1/26 17:40, Huang, Kai wrote:
...

Quoted 2 lines

They might not care about losing ~50M memory, though, but that's a different story, and it could be more in the future.

If anyone _does_ care about 50MB of memory, I expect they'll speak up. Until they do, could we please err on the side of least complexity?

I'm not even sure the Kconfig is worth it to be honest.

#60 lore raw

Re: [PATCH v2 08/31] x86/virt/tdx: Configure TDX Module with optional TDX Connect feature

Quoted 7 lines · Dave Hansen

On 4/1/26 17:40, Huang, Kai wrote:
...

Quoted 2 lines

They might not care about losing ~50M memory, though, but that's a different story, and it could be more in the future.

If anyone _does_ care about 50MB of memory, I expect they'll speak up. Until they do, could we please err on the side of least complexity?

Yeah agreed.

Quoted 2 lines

I'm not even sure the Kconfig is worth it to be honest.

I am not sure either.

My main comment is to make the code more friendly to opt-in more features, actually.

#61 lore raw

Re: [PATCH v2 08/31] x86/virt/tdx: Configure TDX Module with optional TDX Connect feature

In reply to Xu Yilun
Quoted 2 lines · Xu Yilun

TDX Module supports optional TDX features (e.g. TDX Connect & TDX Module Extensions) that won't be enabled by default.

So this is another place where "optional" is misleading. For simplicity there is no mechanism to fallback from TDX Connect operation if present, at least in the core. The only optional aspects would be the mechanism that could be unloaded through the tdx_host driver.

.../me notices that other comments on this patch say the same, but do read on, another important detail about ktime_get_real_seconds() below.

Quoted 9 lines

It extends TDH.SYS.CONFIG for host to choose to enable them on bootup.

Call TDH.SYS.CONFIG with a new bitmap input parameter to specify which features to enable. The bitmap uses the same definitions as TDX_FEATURES0. But note not all bits in TDX_FEATURES0 are valid for configuration, e.g. TDX Module Extensions is a service that supports TDX Connect, it is implicitly enabled when TDX Connect is enabled. Setting TDX_FEATURES0_EXT in the bitmap has no effect.

Quoted 5 lines

TDX Module advances the version of TDH.SYS.CONFIG for the change, so use the latest version (v1) for optional feature enabling. But supporting existing Modules which only support v0 is still necessary until they are deprecated, enumerate via TDX_FEATURES0 to decide which version to use.

I would say this differently, it will always be the case that new kernels are needed to enable new features, but it is unlikely that TDH.SYS.CONFIG ever needs to change again. The v0 -> v1 transition means that feature bits are to be used here on out. So there is little value in worrying about deprecating v0 to save a couple lines of code in 5-7 years when these original TDX platforms sunset.

Quoted 38 lines

TDX Module updates global metadata when optional features are enabled. Host should update the cached tdx_sysinfo to reflect these changes.

---
 arch/x86/virt/vmx/tdx/tdx.h |  3 ++-
 arch/x86/virt/vmx/tdx/tdx.c | 16 +++++++++++++++-
 2 files changed, 17 insertions(+), 2 deletions(-)
arch/x86/virt/vmx/tdx/tdx.h +2 -1
index e5a9331df451..870bb75da3ba 100644
@@ -58,7 +58,8 @@
 #define TDH_PHYMEM_CACHE_WB		40
 #define TDH_PHYMEM_PAGE_WBINVD		41
 #define TDH_VP_WR			43
-#define TDH_SYS_CONFIG			45
+#define TDH_SYS_CONFIG_V0		45
+#define TDH_SYS_CONFIG			SEAMCALL_LEAF_VER(TDH_SYS_CONFIG_V0, 1)
 
 /* TDX page types */
 #define	PT_NDA		0x0
arch/x86/virt/vmx/tdx/tdx.c +1 -0
index 130214933c2f..0c5d6bdd810f 100644
@@ -1353,6 +1353,7 @@ static int construct_tdmrs(struct list_head *tmb_list,
 static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid)
 {
 	struct tdx_module_args args = {};
+	u64 seamcall_fn = TDH_SYS_CONFIG_V0;
 	u64 *tdmr_pa_array;
 	size_t array_sz;
 	int i, ret;

arch/x86/virt/vmx/tdx/tdx.c static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid) · line 1384

Quoted 8 lines
arch/x86/virt/vmx/tdx/tdx.c +4 -1
@@ -1377,7 +1378,15 @@ static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid)
 	args.rcx = __pa(tdmr_pa_array);
 	args.rdx = tdmr_list->nr_consumed_tdmrs;
 	args.r8 = global_keyid;
-	ret = seamcall_prerr(TDH_SYS_CONFIG, &args);
+
+	if (tdx_sysinfo.features.tdx_features0 & TDX_FEATURES0_TDXCONNECT) {
+		args.r9 |= TDX_FEATURES0_TDXCONNECT;
+		args.r11 = ktime_get_real_seconds();

Mainline has reason to not entertain this module requirement. The fact that passing zero is an error is useful to detect unsupported modules. An updated module would accept zero as indicating "VMM requests module disable all policy and mechanisms related to untrusted wall clock time". Specifically, there are several problems with this:

1/ No other TSM implementation requires the VMM to pass in an untrusted time 2/ The wall time may change and may require hooks to keep the module time
up to date, but see point 1/, this would be a TDX special flower hook.
3/ Presumably this allows the module or the guest to do certificate expiration
checks, but that is the responsibility of the relying party. The relying party may have reason to accept an "expired" cert as determined by VMM wall clock, and the guest presumably already has mechanisms to determine untrusted wall clock time from the VMM if it wants. Guests do not need TDX ABI for that.

So I think Linux wants to pass 0 here and wait for modules that accept that as the start of TDX Connect support. As you said, given there are no released modules with TDX Connect there is time to make that first release drop this requirement.

author
#62 lore raw

Re: [PATCH v2 08/31] x86/virt/tdx: Configure TDX Module with optional TDX Connect feature

Quoted 8 lines · Dan Williams
Quoted 2 lines · Xu Yilun

TDX Module supports optional TDX features (e.g. TDX Connect & TDX Module Extensions) that won't be enabled by default.

So this is another place where "optional" is misleading. For simplicity there is no mechanism to fallback from TDX Connect operation if present, at least in the core. The only optional aspects would be the mechanism that could be unloaded through the tdx_host driver.

I see. I think I should use 'extra' instead of 'optional' in all places. I'll rephase the comments.

Quoted 19 lines

.../me notices that other comments on this patch say the same, but do read on, another important detail about ktime_get_real_seconds() below.

Quoted 9 lines

It extends TDH.SYS.CONFIG for host to choose to enable them on bootup.

Call TDH.SYS.CONFIG with a new bitmap input parameter to specify which features to enable. The bitmap uses the same definitions as TDX_FEATURES0. But note not all bits in TDX_FEATURES0 are valid for configuration, e.g. TDX Module Extensions is a service that supports TDX Connect, it is implicitly enabled when TDX Connect is enabled. Setting TDX_FEATURES0_EXT in the bitmap has no effect.

Quoted 5 lines

TDX Module advances the version of TDH.SYS.CONFIG for the change, so use the latest version (v1) for optional feature enabling. But supporting existing Modules which only support v0 is still necessary until they are deprecated, enumerate via TDX_FEATURES0 to decide which version to use.

Quoted 6 lines

I would say this differently, it will always be the case that new kernels are needed to enable new features, but it is unlikely that TDH.SYS.CONFIG ever needs to change again. The v0 -> v1 transition means that feature bits are to be used here on out. So there is little value in worrying about deprecating v0 to save a couple lines of code in 5-7 years when these original TDX platforms sunset.

OK. I'll use your rationale.

Quoted 42 lines
Quoted 42 lines

TDX Module updates global metadata when optional features are enabled. Host should update the cached tdx_sysinfo to reflect these changes.

---
 arch/x86/virt/vmx/tdx/tdx.h |  3 ++-
 arch/x86/virt/vmx/tdx/tdx.c | 16 +++++++++++++++-
 2 files changed, 17 insertions(+), 2 deletions(-)
arch/x86/virt/vmx/tdx/tdx.h +2 -1
index e5a9331df451..870bb75da3ba 100644
@@ -58,7 +58,8 @@
 #define TDH_PHYMEM_CACHE_WB		40
 #define TDH_PHYMEM_PAGE_WBINVD		41
 #define TDH_VP_WR			43
-#define TDH_SYS_CONFIG			45
+#define TDH_SYS_CONFIG_V0		45
+#define TDH_SYS_CONFIG			SEAMCALL_LEAF_VER(TDH_SYS_CONFIG_V0, 1)
 
 /* TDX page types */
 #define	PT_NDA		0x0
arch/x86/virt/vmx/tdx/tdx.c +1 -0
index 130214933c2f..0c5d6bdd810f 100644
@@ -1353,6 +1353,7 @@ static int construct_tdmrs(struct list_head *tmb_list,
 static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid)
 {
 	struct tdx_module_args args = {};
+	u64 seamcall_fn = TDH_SYS_CONFIG_V0;
 	u64 *tdmr_pa_array;
 	size_t array_sz;
 	int i, ret;
@@ -1377,7 +1378,15 @@ static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid)
 	args.rcx = __pa(tdmr_pa_array);
 	args.rdx = tdmr_list->nr_consumed_tdmrs;
 	args.r8 = global_keyid;
Quoted 8 lines

arch/x86/virt/vmx/tdx/tdx.c static int config_tdx_module(struct tdmr_info_list *tdmr_list, u64 global_keyid) · line 1384

Quoted 5 lines
arch/x86/virt/vmx/tdx/tdx.c +4 -1
-	ret = seamcall_prerr(TDH_SYS_CONFIG, &args);
+
+	if (tdx_sysinfo.features.tdx_features0 & TDX_FEATURES0_TDXCONNECT) {
+		args.r9 |= TDX_FEATURES0_TDXCONNECT;
+		args.r11 = ktime_get_real_seconds();

Mainline has reason to not entertain this module requirement. The fact that passing zero is an error is useful to detect unsupported modules.

Ah, it is "useful"... :)

Quoted 14 lines

An updated module would accept zero as indicating "VMM requests module disable all policy and mechanisms related to untrusted wall clock time". Specifically, there are several problems with this:

1/ No other TSM implementation requires the VMM to pass in an untrusted time 2/ The wall time may change and may require hooks to keep the module time
up to date, but see point 1/, this would be a TDX special flower hook.
3/ Presumably this allows the module or the guest to do certificate expiration
checks, but that is the responsibility of the relying party. The relying party may have reason to accept an "expired" cert as determined by VMM wall clock, and the guest presumably already has mechanisms to determine untrusted wall clock time from the VMM if it wants. Guests do not need TDX ABI for that.

Quoted 4 lines

So I think Linux wants to pass 0 here and wait for modules that accept that as the start of TDX Connect support. As you said, given there are no released modules with TDX Connect there is time to make that first release drop this requirement.

I see. Actually the Module is about to remove the r11 RTC.

https://cdrdv2.intel.com/v1/dl/getContent/871617

I'll remove this r11.

author
#63 lore raw

PATCH v2 9/31 x86/virt/tdx: Move tdx_clflush_page() up in the file

In reply to Xu Yilun

Prepare to add more callers earlier in this file, so move this function up in advance.

---
 arch/x86/virt/vmx/tdx/tdx.c | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)
arch/x86/virt/vmx/tdx/tdx.c +11 -11
index 0c5d6bdd810f..4fb56bb442f0 100644
@@ -1502,6 +1502,17 @@ static int init_tdmrs(struct tdmr_info_list *tdmr_list)
 	return 0;
 }
 
+/*
+ * The TDX module exposes a CLFLUSH_BEFORE_ALLOC bit to specify whether
+ * a CLFLUSH of pages is required before handing them to the TDX module.
+ * Be conservative and make the code simpler by doing the CLFLUSH
+ * unconditionally.
+ */
+static void tdx_clflush_page(struct page *page)
+{
+	clflush_cache_range(page_to_virt(page), PAGE_SIZE);
+}
+
 static int init_tdx_module(void)
 {
 	int ret;
@@ -1936,17 +1947,6 @@ static inline u64 tdx_tdr_pa(struct tdx_td *td)
 	return page_to_phys(td->tdr_page);
 }
 
-/*
- * The TDX module exposes a CLFLUSH_BEFORE_ALLOC bit to specify whether
- * a CLFLUSH of pages is required before handing them to the TDX module.
- * Be conservative and make the code simpler by doing the CLFLUSH
- * unconditionally.
- */
-static void tdx_clflush_page(struct page *page)
-{
-	clflush_cache_range(page_to_virt(page), PAGE_SIZE);
-}
-
 noinstr u64 tdh_vp_enter(struct tdx_vp *td, struct tdx_module_args *args)
 {
 	args->rcx = td->tdvpr_pa;
Signature · 1 line
2.25.1
author
#64 lore raw

PATCH v2 10/31 x86/virt/tdx: Add extra memory to TDX Module for Extensions

In reply to Xu Yilun

Adding more memory to TDX Module is the first step to enable Extensions.

Currently, TDX Module memory use is relatively static. But, some new features (called "TDX Module Extensions") need to use memory more dynamically. While 'static' here means the kernel provides necessary amount of memory to TDX Module for its basic functionalities, 'dynamic' means extra memory is needed only if new optional features are to be enabled. So add a new memory feeding process backed by a new SEAMCALL TDH.EXT.MEM.ADD.

The process is mostly the same as adding PAMT. The kernel queries TDX Module how much memory needed, allocates it, hands it over, and never gets it back.

TDH.EXT.MEM.ADD uses tdx_page_array to provide control (private) pages to TDX Module. Introduce a tdx_clflush_page_array() helper to flush shared cache before SEAMCALL, to avoid shared cache write back damages these private pages.

For now, TDX Module Extensions consume relatively large amount of memory (~50MB). Use contiguous page allocation to avoid permanently fragment too much memory. Print this readout value on TDX Module Extensions initialization for visibility.

---
 arch/x86/virt/vmx/tdx/tdx.h |  1 +
 arch/x86/virt/vmx/tdx/tdx.c | 92 ++++++++++++++++++++++++++++++++++++-
 2 files changed, 91 insertions(+), 2 deletions(-)
arch/x86/virt/vmx/tdx/tdx.h +1 -0
index 870bb75da3ba..31ccdfcf518c 100644
@@ -60,6 +60,7 @@
 #define TDH_VP_WR			43
 #define TDH_SYS_CONFIG_V0		45
 #define TDH_SYS_CONFIG			SEAMCALL_LEAF_VER(TDH_SYS_CONFIG_V0, 1)
+#define TDH_EXT_MEM_ADD			61
 
 /* TDX page types */
 #define	PT_NDA		0x0
arch/x86/virt/vmx/tdx/tdx.c +90 -2

110 lines not on this page · raw message

Signature · 1 line
2.25.1
#65 lore raw

Re: [PATCH v2 10/31] x86/virt/tdx: Add extra memory to TDX Module for Extensions

Quoted 67 lines · Xu Yilun

Adding more memory to TDX Module is the first step to enable Extensions.

Currently, TDX Module memory use is relatively static. But, some new features (called "TDX Module Extensions") need to use memory more dynamically. While 'static' here means the kernel provides necessary amount of memory to TDX Module for its basic functionalities, 'dynamic' means extra memory is needed only if new optional features are to be enabled. So add a new memory feeding process backed by a new SEAMCALL TDH.EXT.MEM.ADD.

The process is mostly the same as adding PAMT. The kernel queries TDX Module how much memory needed, allocates it, hands it over, and never gets it back.

TDH.EXT.MEM.ADD uses tdx_page_array to provide control (private) pages to TDX Module. Introduce a tdx_clflush_page_array() helper to flush shared cache before SEAMCALL, to avoid shared cache write back damages these private pages.

For now, TDX Module Extensions consume relatively large amount of memory (~50MB). Use contiguous page allocation to avoid permanently fragment too much memory. Print this readout value on TDX Module Extensions initialization for visibility.

---
 arch/x86/virt/vmx/tdx/tdx.h |  1 +
 arch/x86/virt/vmx/tdx/tdx.c | 92 ++++++++++++++++++++++++++++++++++++-
 2 files changed, 91 insertions(+), 2 deletions(-)
arch/x86/virt/vmx/tdx/tdx.h +1 -0
index 870bb75da3ba..31ccdfcf518c 100644
@@ -60,6 +60,7 @@
 #define TDH_VP_WR			43
 #define TDH_SYS_CONFIG_V0		45
 #define TDH_SYS_CONFIG			SEAMCALL_LEAF_VER(TDH_SYS_CONFIG_V0, 1)
+#define TDH_EXT_MEM_ADD			61
 
 /* TDX page types */
 #define	PT_NDA		0x0
arch/x86/virt/vmx/tdx/tdx.c +2 -2
index 4fb56bb442f0..5fae17c13191 100644
@@ -560,7 +560,7 @@ static int tdx_alloc_pages_contig(unsigned int nr_pages, struct page **pages,
  * Similar to tdx_page_array_alloc(), after allocating with this
  * function, call tdx_page_array_populate() to populate the tdx_page_array.
  */
-static __maybe_unused struct tdx_page_array *
+static struct tdx_page_array *
 tdx_page_array_alloc_contig(unsigned int nr_pages)
 {
 	return tdx_page_array_alloc(nr_pages, tdx_alloc_pages_contig, NULL);
@@ -643,7 +643,7 @@ EXPORT_SYMBOL_GPL(tdx_page_array_create_iommu_mt);
 #define HPA_LIST_INFO_PFN		GENMASK_U64(51, 12)
 #define HPA_LIST_INFO_LAST_ENTRY	GENMASK_U64(63, 55)
 
-static u64 __maybe_unused hpa_list_info_assign_raw(struct tdx_page_array *array)
+static u64 hpa_list_info_assign_raw(struct tdx_page_array *array)
 {
 	return FIELD_PREP(HPA_LIST_INFO_FIRST_ENTRY, 0) |
 	       FIELD_PREP(HPA_LIST_INFO_PFN,

arch/x86/virt/vmx/tdx/tdx.c static void tdx_clflush_page(struct page *page) · line 1516

Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +1 -0
@@ -1513,6 +1513,94 @@ static void tdx_clflush_page(struct page *page)
 	clflush_cache_range(page_to_virt(page), PAGE_SIZE);
 }
 
+static void tdx_clflush_page_array(struct tdx_page_array *array)

It doesn't clflush the page array, it clflushes the current populate chunk. Hmm. Does it suggest that the page array and the format for handing them to the TDX module are two different things?

Quoted 28 lines
arch/x86/virt/vmx/tdx/tdx.c +28 -0
+{
+	for (int i = 0; i < array->nents; i++)
+		tdx_clflush_page(array->pages[array->offset + i]);
+}
+
+static int tdx_ext_mem_add(struct tdx_page_array *ext_mem)
+{
+	struct tdx_module_args args = {
+		.rcx = hpa_list_info_assign_raw(ext_mem),
+	};
+	u64 r;
+
+	tdx_clflush_page_array(ext_mem);
+
+	do {
+		r = seamcall_ret(TDH_EXT_MEM_ADD, &args);
+		cond_resched();
+	} while (r == TDX_INTERRUPTED_RESUMABLE);
+
+	if (r != TDX_SUCCESS)
+		return -EFAULT;
+
+	return 0;
+}
+
+static int tdx_ext_mem_setup(struct tdx_page_array *ext_mem)
+{
+	unsigned int populated, offset = 0;

arch/x86/virt/vmx/tdx/tdx.c static void tdx_clflush_page(struct page *page) · line 1554

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +10 -0
+	int ret;
+
+	/*
+	 * tdx_page_array's root page can hold 512 HPAs at most. We have ~50MB
+	 * memory to add, re-populate the array and add pages bulk by bulk.
+	 */
+	while (1) {
+		populated = tdx_page_array_populate(ext_mem, offset);
+		if (!populated)
+			break;

For this case of populate it seems like it would be ok to keep an array of PA's instead of an array of struct pages. Not sure on it yet.

Quoted 15 lines
arch/x86/virt/vmx/tdx/tdx.c +15 -0
+
+		ret = tdx_ext_mem_add(ext_mem);
+		if (ret)
+			return ret;
+
+		offset += populated;
+	}
+
+	return 0;
+}
+
+static int __maybe_unused init_tdx_ext(void)
+{
+	struct tdx_page_array *ext_mem = NULL;
+	unsigned int nr_pages;

arch/x86/virt/vmx/tdx/tdx.c static void tdx_clflush_page(struct page *page) · line 1579

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +10 -0
+	int ret;
+
+	if (!(tdx_sysinfo.features.tdx_features0 & TDX_FEATURES0_EXT))
+		return 0;
+
+	nr_pages = tdx_sysinfo.ext.memory_pool_required_pages;
+	/*
+	 * memory_pool_required_pages == 0 means no need to add more pages,
+	 * skip the memory setup.
+	 */

Is this ever expected? Extensions are supported, but require no pages?

arch/x86/virt/vmx/tdx/tdx.c static void tdx_clflush_page(struct page *page) · line 1591

Quoted 12 lines
arch/x86/virt/vmx/tdx/tdx.c +12 -0
+	if (nr_pages) {
+		ext_mem = tdx_page_array_alloc_contig(nr_pages);
+		if (!ext_mem)
+			return -ENOMEM;
+
+		ret = tdx_ext_mem_setup(ext_mem);
+		if (ret)
+			goto out_ext_mem;
+	}
+
+	/* Extension memory is never reclaimed once assigned */
+	tdx_page_array_ctrl_leak(ext_mem);

This looks very weird to call "leak" in the success path.

Quoted 15 lines
arch/x86/virt/vmx/tdx/tdx.c +12 -0
+
+	pr_info("%lu KB allocated for TDX Module Extensions\n",
+		nr_pages * PAGE_SIZE / 1024);
+
+	return 0;
+
+out_ext_mem:
+	tdx_page_array_free(ext_mem);
+
+	return ret;
+}
+
 static int init_tdx_module(void)
 {
 	int ret;
#66 lore raw

Re: [PATCH v2 10/31] x86/virt/tdx: Add extra memory to TDX Module for Extensions

In reply to Xu Yilun
Quoted 67 lines · Xu Yilun

Adding more memory to TDX Module is the first step to enable Extensions.

Currently, TDX Module memory use is relatively static. But, some new features (called "TDX Module Extensions") need to use memory more dynamically. While 'static' here means the kernel provides necessary amount of memory to TDX Module for its basic functionalities, 'dynamic' means extra memory is needed only if new optional features are to be enabled. So add a new memory feeding process backed by a new SEAMCALL TDH.EXT.MEM.ADD.

The process is mostly the same as adding PAMT. The kernel queries TDX Module how much memory needed, allocates it, hands it over, and never gets it back.

TDH.EXT.MEM.ADD uses tdx_page_array to provide control (private) pages to TDX Module. Introduce a tdx_clflush_page_array() helper to flush shared cache before SEAMCALL, to avoid shared cache write back damages these private pages.

For now, TDX Module Extensions consume relatively large amount of memory (~50MB). Use contiguous page allocation to avoid permanently fragment too much memory. Print this readout value on TDX Module Extensions initialization for visibility.

---
  arch/x86/virt/vmx/tdx/tdx.h |  1 +
  arch/x86/virt/vmx/tdx/tdx.c | 92 ++++++++++++++++++++++++++++++++++++-
  2 files changed, 91 insertions(+), 2 deletions(-)
arch/x86/virt/vmx/tdx/tdx.h +1 -0
index 870bb75da3ba..31ccdfcf518c 100644
@@ -60,6 +60,7 @@
  #define TDH_VP_WR			43
  #define TDH_SYS_CONFIG_V0		45
  #define TDH_SYS_CONFIG			SEAMCALL_LEAF_VER(TDH_SYS_CONFIG_V0, 1)
+#define TDH_EXT_MEM_ADD			61
 
  /* TDX page types */
  #define	PT_NDA		0x0
arch/x86/virt/vmx/tdx/tdx.c +2 -2
index 4fb56bb442f0..5fae17c13191 100644
@@ -560,7 +560,7 @@ static int tdx_alloc_pages_contig(unsigned int nr_pages, struct page **pages,
   * Similar to tdx_page_array_alloc(), after allocating with this
   * function, call tdx_page_array_populate() to populate the tdx_page_array.
   */
-static __maybe_unused struct tdx_page_array *
+static struct tdx_page_array *
  tdx_page_array_alloc_contig(unsigned int nr_pages)
  {
  	return tdx_page_array_alloc(nr_pages, tdx_alloc_pages_contig, NULL);
@@ -643,7 +643,7 @@ EXPORT_SYMBOL_GPL(tdx_page_array_create_iommu_mt);
  #define HPA_LIST_INFO_PFN		GENMASK_U64(51, 12)
  #define HPA_LIST_INFO_LAST_ENTRY	GENMASK_U64(63, 55)
 
-static u64 __maybe_unused hpa_list_info_assign_raw(struct tdx_page_array *array)
+static u64 hpa_list_info_assign_raw(struct tdx_page_array *array)
  {
  	return FIELD_PREP(HPA_LIST_INFO_FIRST_ENTRY, 0) |
  	       FIELD_PREP(HPA_LIST_INFO_PFN,

arch/x86/virt/vmx/tdx/tdx.c static void tdx_clflush_page(struct page *page) · line 1518

Quoted 6 lines
arch/x86/virt/vmx/tdx/tdx.c +3 -0
@@ -1513,6 +1513,94 @@ static void tdx_clflush_page(struct page *page)
  	clflush_cache_range(page_to_virt(page), PAGE_SIZE);
  }
 
+static void tdx_clflush_page_array(struct tdx_page_array *array)
+{
+	for (int i = 0; i < array->nents; i++)

shouldn't the actual number of entries be adjusted as per offset, similarly to how 'nents' in tdx_page_array_validate_release is calculated?

<snip>

author
#67 lore raw

Re: [PATCH v2 10/31] x86/virt/tdx: Add extra memory to TDX Module for Extensions

Quoted 9 lines
Quoted 9 lines
arch/x86/virt/vmx/tdx/tdx.c +1 -1
@@ -643,7 +643,7 @@ EXPORT_SYMBOL_GPL(tdx_page_array_create_iommu_mt);
  #define HPA_LIST_INFO_PFN		GENMASK_U64(51, 12)
  #define HPA_LIST_INFO_LAST_ENTRY	GENMASK_U64(63, 55)
-static u64 __maybe_unused hpa_list_info_assign_raw(struct tdx_page_array *array)
+static u64 hpa_list_info_assign_raw(struct tdx_page_array *array)
  {
  	return FIELD_PREP(HPA_LIST_INFO_FIRST_ENTRY, 0) |
  	       FIELD_PREP(HPA_LIST_INFO_PFN,
@@ -1513,6 +1513,94 @@ static void tdx_clflush_page(struct page *page)
Quoted 8 lines

arch/x86/virt/vmx/tdx/tdx.c static void tdx_clflush_page(struct page *page) · line 1518

Quoted 5 lines
arch/x86/virt/vmx/tdx/tdx.c +3 -0
  	clflush_cache_range(page_to_virt(page), PAGE_SIZE);
  }
+static void tdx_clflush_page_array(struct tdx_page_array *array)
+{
+	for (int i = 0; i < array->nents; i++)

shouldn't the actual number of entries be adjusted as per offset, similarly to how 'nents' in tdx_page_array_validate_release is calculated?

Actually array->nents is calculated the same way:

static int tdx_page_array_populate(struct tdx_page_array *array,
unsigned int offset)

  {
	...

	array->offset = offset;
	array->nents = umin(array->nr_pages - offset,
			    TDX_PAGE_ARRAY_MAX_NENTS);
	...

}

so IIUC we are good here.

#68 lore raw

Re: [PATCH v2 10/31] x86/virt/tdx: Add extra memory to TDX Module for Extensions

In reply to Xu Yilun

arch/x86/virt/vmx/tdx/tdx.c static void tdx_clflush_page(struct page *page) · line 1534

Quoted 13 lines · Xu Yilun
arch/x86/virt/vmx/tdx/tdx.c +13 -0
+static int tdx_ext_mem_add(struct tdx_page_array *ext_mem)
+{
+	struct tdx_module_args args = {
+		.rcx = hpa_list_info_assign_raw(ext_mem),
+	};
+	u64 r;
+
+	tdx_clflush_page_array(ext_mem);
+
+	do {
+		r = seamcall_ret(TDH_EXT_MEM_ADD, &args);
+		cond_resched();
+	} while (r == TDX_INTERRUPTED_RESUMABLE);

Ditto here. I don't think we should introduce any more cond_resched().

Btw, I think technically we can reuse the seamcall_ir_resched() you introduced later, albeit in which a local '_args' is used as a copy of the original 'args', but that has no harm for the case where we can just use the 'args' to loop.

I am wondering whether we can just use that here, or we just get rid of that helper (then open retry by the callers of these SEAMCALL wrappers), since there will be more cases where we need to manually set 'resume=1' in the SEAMCALL input 'args' when retrying TDX_INTERRUPTED_RESUMABLE.

Unless you have good idea to unify them all?

E.g., we have something like below in our internal KVM code, using macros to do 'resume=1' and retry as the caller wishes. But my understanding is Dave probably won't like macros. :-)

(you may see broken indent/text due to text wrapper and sorry for that.)

/*
* ...
*
* The retry_func and update_args allow the SEAMCALL to be retried in a loop if
* it can still return other error code when there's no race from both KVM and
* vCPUs and can be "retried" until it succeeds.
*/
#define tdh_do_no_vcpus_retry(tdh_func, kvm, retry_func, update_args, args...)\

({                                                                            \
        struct kvm_tdx *__kvm_tdx = to_kvm_tdx(kvm);                          \
        u64 __err;                                                            \
                                                                              \
        lockdep_assert_held_write(&kvm->mmu_lock);                            \
                                                                              \
        __err = retry_func(tdh_func, update_args, args);                      \
        if (unlikely(tdx_operand_busy(__err))) {                              \
                WRITE_ONCE(__kvm_tdx->wait_for_sept_zap, true);               \
                kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE);   \
                                                                              \
                __err = retry_func(tdh_func, update_args, args);              \
                                                                              \
                WRITE_ONCE(__kvm_tdx->wait_for_sept_zap, false);              \
        }                                                                     \
        __err;                                                                \
})                                                                             
#define tdh_intr_retry(tdh_func, update_args, args...)                        \
({                                                                            \
        u64 ____err;                                                          \
                                                                              \
        do {                                                                  \
                ____err = tdh_func(args);                                     \
                                                                              \
                if ((____err & TDX_SEAMCALL_STATUS_MASK) !=		      \
				TDX_INTERRUPTED_RESUMABLE)                   
\	                       
                        break;                                                \
                                                                              \
                update_args;                                                  \
        } while (1);                                                          \
        ____err;                                                              \
})

#define tdh_no_retry(tdh_func, update_args, args...) tdh_func(args)

#define tdh_do_no_vcpus(tdh_func, kvm, args...) \
tdh_do_no_vcpus_retry(tdh_func, kvm, tdh_no_retry, ;, args)

#define tdh_do_no_vcpus_intr_retry(tdh_func, kvm, update_args, args...) \
tdh_do_no_vcpus_retry(tdh_func, kvm, tdh_intr_retry, update_args, args)

author
#69 lore raw

Re: [PATCH v2 10/31] x86/virt/tdx: Add extra memory to TDX Module for Extensions

Quoted 17 lines · Kai

arch/x86/virt/vmx/tdx/tdx.c static void tdx_clflush_page(struct page *page) · line 1534

Quoted 13 lines · Xu Yilun
arch/x86/virt/vmx/tdx/tdx.c +13 -0
+static int tdx_ext_mem_add(struct tdx_page_array *ext_mem)
+{
+	struct tdx_module_args args = {
+		.rcx = hpa_list_info_assign_raw(ext_mem),
+	};
+	u64 r;
+
+	tdx_clflush_page_array(ext_mem);
+
+	do {
+		r = seamcall_ret(TDH_EXT_MEM_ADD, &args);
+		cond_resched();
+	} while (r == TDX_INTERRUPTED_RESUMABLE);

Ditto here. I don't think we should introduce any more cond_resched().

Good to me.

Quoted 4 lines

Btw, I think technically we can reuse the seamcall_ir_resched() you introduced later, albeit in which a local '_args' is used as a copy of the original 'args', but that has no harm for the case where we can just use the 'args' to loop.

No we can't. TDH_EXT_MEM_ADD is designed to use output parameter RCX to override/update input parameter RCX, so the caller doesn't have to do manual parameter update on retry call. Using seamcall_ir_resched() makes each retry use the original RCX, not the updated one.

Quoted 5 lines

I am wondering whether we can just use that here, or we just get rid of that helper (then open retry by the callers of these SEAMCALL wrappers), since there will be more cases where we need to manually set 'resume=1' in the SEAMCALL input 'args' when retrying TDX_INTERRUPTED_RESUMABLE.

I'd like to know why some SEAMCALLs needs resume flag but others don't. If there is chance we don't introduce too much variants for the same thing, that's most friendly to OS. And "no resume flag" is my best preference.

For now, I can see only one SEAMCALL with resume flag in mainline, tdh_phymem_cache_wb(). I'd rather we treat it as an exception and no resume flag any more if possible.

Then we don't have to make all following efforts, they are complex...

Quoted 59 lines

Unless you have good idea to unify them all?

E.g., we have something like below in our internal KVM code, using macros to do 'resume=1' and retry as the caller wishes. But my understanding is Dave probably won't like macros. :-)

(you may see broken indent/text due to text wrapper and sorry for that.)

/*
* ...
*
* The retry_func and update_args allow the SEAMCALL to be retried in a loop if
* it can still return other error code when there's no race from both KVM and
* vCPUs and can be "retried" until it succeeds.
*/
#define tdh_do_no_vcpus_retry(tdh_func, kvm, retry_func, update_args, args...)\

({                                                                            \
        struct kvm_tdx *__kvm_tdx = to_kvm_tdx(kvm);                          \
        u64 __err;                                                            \
                                                                              \
        lockdep_assert_held_write(&kvm->mmu_lock);                            \
                                                                              \
        __err = retry_func(tdh_func, update_args, args);                      \
        if (unlikely(tdx_operand_busy(__err))) {                              \
                WRITE_ONCE(__kvm_tdx->wait_for_sept_zap, true);               \
                kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE);   \
                                                                              \
                __err = retry_func(tdh_func, update_args, args);              \
                                                                              \
                WRITE_ONCE(__kvm_tdx->wait_for_sept_zap, false);              \
        }                                                                     \
        __err;                                                                \
})                                                                             
#define tdh_intr_retry(tdh_func, update_args, args...)                        \
({                                                                            \
        u64 ____err;                                                          \
                                                                              \
        do {                                                                  \
                ____err = tdh_func(args);                                     \
                                                                              \
                if ((____err & TDX_SEAMCALL_STATUS_MASK) !=		      \
				TDX_INTERRUPTED_RESUMABLE)                   
\	                       
                        break;                                                \
                                                                              \
                update_args;                                                  \
        } while (1);                                                          \
        ____err;                                                              \
})

#define tdh_no_retry(tdh_func, update_args, args...) tdh_func(args)

#define tdh_do_no_vcpus(tdh_func, kvm, args...) \
tdh_do_no_vcpus_retry(tdh_func, kvm, tdh_no_retry, ;, args)

#define tdh_do_no_vcpus_intr_retry(tdh_func, kvm, update_args, args...) \
tdh_do_no_vcpus_retry(tdh_func, kvm, tdh_intr_retry, update_args, args)

#70 lore raw

Re: [PATCH v2 10/31] x86/virt/tdx: Add extra memory to TDX Module for Extensions

Quoted 10 lines
Quoted 5 lines

Btw, I think technically we can reuse the seamcall_ir_resched() you introduced later, albeit in which a local '_args' is used as a copy of the original 'args', but that has no harm for the case where we can just use the 'args' to loop.

No we can't. TDH_EXT_MEM_ADD is designed to use output parameter RCX to override/update input parameter RCX, so the caller doesn't have to do manual parameter update on retry call. Using seamcall_ir_resched() makes each retry use the original RCX, not the updated one.

OK I wish there's a comment saying there's additional output besides error code via RAX and we just need to feed the output as input again when retrying the SEAMCALL.

Quoted 10 lines
Quoted 6 lines

I am wondering whether we can just use that here, or we just get rid of that helper (then open retry by the callers of these SEAMCALL wrappers), since there will be more cases where we need to manually set 'resume=1' in the SEAMCALL input 'args' when retrying TDX_INTERRUPTED_RESUMABLE.

I'd like to know why some SEAMCALLs needs resume flag but others don't. If there is chance we don't introduce too much variants for the same thing, that's most friendly to OS. And "no resume flag" is my best preference.

I don't know either.

Quoted 4 lines

For now, I can see only one SEAMCALL with resume flag in mainline, tdh_phymem_cache_wb(). I'd rather we treat it as an exception and no resume flag any more if possible.

Right, but there will be more, and setting 'resumed=1' is even different from how tdh_phymem_cache_wb() does.

#71 lore raw

Re: [PATCH v2 10/31] x86/virt/tdx: Add extra memory to TDX Module for Extensions

In reply to Huang, Kai
Quoted 10 lines · Kai

Ditto here.  I don't think we should introduce any more cond_resched().

Btw, I think technically we can reuse the seamcall_ir_resched() you introduced later, albeit in which a local '_args' is used as a copy of the original 'args', but that has no harm for the case where we can just use the 'args' to loop.

I am wondering whether we can just use that here, or we just get rid of that helper (then open retry by the callers of these SEAMCALL wrappers), since there will be more cases where we need to manually set 'resume=1' in the SEAMCALL input 'args' when retrying TDX_INTERRUPTED_RESUMABLE.

I kind of like the latter option to open code more of this stuff. The stacks of seamcall wrapper macros is already too much.

#72 lore raw

Re: [PATCH v2 10/31] x86/virt/tdx: Add extra memory to TDX Module for Extensions

Quoted 7 lines · Rick P
Quoted 6 lines · Kai

Ditto here.  I don't think we should introduce any more cond_resched().

Btw, I think technically we can reuse the seamcall_ir_resched() you introduced later, albeit in which a local '_args' is used as a copy of the original 'args', but that has no harm for the case where we can just use the 'args' to loop.

Quoted 7 lines
Quoted 4 lines

I am wondering whether we can just use that here, or we just get rid of that helper (then open retry by the callers of these SEAMCALL wrappers), since there will be more cases where we need to manually set 'resume=1' in the SEAMCALL input 'args' when retrying TDX_INTERRUPTED_RESUMABLE.

I kind of like the latter option to open code more of this stuff. The stacks of seamcall wrapper macros is already too much.

Agreed.

And SEAMCALL *users* can actually come up with their own version of wrapper(s) to do the retry. E.g., currently seamcall_ir_resched() is only used for IOMMU SEAMCALLs, and we can put this wrapper in the IOMMU code or coco/tdx-host.

author
#73 lore raw

Re: [PATCH v2 10/31] x86/virt/tdx: Add extra memory to TDX Module for Extensions

Quoted 13 lines · Kai
Quoted 12 lines · Rick P
Quoted 10 lines · Kai

Ditto here.  I don't think we should introduce any more cond_resched().

Btw, I think technically we can reuse the seamcall_ir_resched() you introduced later, albeit in which a local '_args' is used as a copy of the original 'args', but that has no harm for the case where we can just use the 'args' to loop.

I am wondering whether we can just use that here, or we just get rid of that helper (then open retry by the callers of these SEAMCALL wrappers), since there will be more cases where we need to manually set 'resume=1' in the SEAMCALL input 'args' when retrying TDX_INTERRUPTED_RESUMABLE.

Quoted 8 lines
Quoted 2 lines

I kind of like the latter option to open code more of this stuff. The stacks of seamcall wrapper macros is already too much.

Agreed.

And SEAMCALL *users* can actually come up with their own version of wrapper(s) to do the retry. E.g., currently seamcall_ir_resched() is only used for IOMMU SEAMCALLs, and we can put this wrapper in the IOMMU code or coco/tdx-host.

After we have introduced TDX Module Extension, irq preemptable EXT-SEAMCALLs become a common concept. It is irq preemptable so that the secure world remembers and resumes the context, no need for host to remind via resume lag.

Today there are 3 EXT-SEAMCALLs, TDH_SPDM_CONNECT/DISCONNECT/MNG, irq preemption handling is a general requirement for them, and I think it is still true for any further EXT-SEAMCALLs.

So I think a general helper for EXT-SEAMCALLs makes sense.

TDH.IOMMU.SETUP, however, is another case. It is not a EXT-SEAMCALL but happened to follow the same irq-retry handling process. To avoid code duplication we have:

/*
* seamcall_ret_ir_exec() aliases seamcall_ret_ir_resched() for
* documentation purposes. It documents the TDX Module extension
* seamcalls that are long running / hard-irq preemptible flows that
* generate events. The calls using seamcall_ret_ir_resched() are long
* running flows, that periodically yield.
*/
#define seamcall_ret_ir_exec seamcall_ret_ir_resched

TDH.IOMMU.SETUP uses seamcall_ret_ir_resched(), and EXT-SEAMCALLs use seamcall_ret_ir_exec().

How do you think?

#74 lore raw

Re: [PATCH v2 10/31] x86/virt/tdx: Add extra memory to TDX Module for Extensions

Quoted 17 lines · Xu Yilun
Quoted 16 lines · Kai
Quoted 14 lines · Rick P
Quoted 10 lines · Kai

Ditto here.  I don't think we should introduce any more cond_resched().

Btw, I think technically we can reuse the seamcall_ir_resched() you introduced later, albeit in which a local '_args' is used as a copy of the original 'args', but that has no harm for the case where we can just use the 'args' to loop.

I am wondering whether we can just use that here, or we just get rid of that helper (then open retry by the callers of these SEAMCALL wrappers), since there will be more cases where we need to manually set 'resume=1' in the SEAMCALL input 'args' when retrying TDX_INTERRUPTED_RESUMABLE.

I kind of like the latter option to open code more of this stuff. The stacks of seamcall wrapper macros is already too much.

Quoted 9 lines
Quoted 5 lines

Agreed.

And SEAMCALL *users* can actually come up with their own version of wrapper(s) to do the retry. E.g., currently seamcall_ir_resched() is only used for IOMMU SEAMCALLs, and we can put this wrapper in the IOMMU code or coco/tdx-host.

After we have introduced TDX Module Extension, irq preemptable EXT-SEAMCALLs become a common concept.

It has been a concept since before the EXT-SEAMCALLs actually. For instance, TDX live migration using blocking export doesn't need any opt-in via module extension (only the non-blocking way needs), but the SEAMCALLs to export/import TD/vCPU/memory are all interruptible.

In fact, they had the "latency requirement" behind the INTERRUPT_RESUMABLE in mind at the very beginning. It's just at that point all SEAMCALLs were not that heavy.

Quoted 3 lines

It is irq preemptable so that the
secure world remembers and resumes the context, no need for host to remind via resume lag.

The fact is the aforementioned live migration related export/import SEAMCALLs (there are 8 at least, but maybe more) all requires the explicit setting of 'resume=1' (plus using the SEAMCALL output as input for retry). I don't know the story behind this, though. There might be some tricky thing here for the module to remember and manage (e.g., migration has a concept of "migration stream", and the resume is per-stream).

Quoted 6 lines

Today there are 3 EXT-SEAMCALLs, TDH_SPDM_CONNECT/DISCONNECT/MNG, irq preemption handling is a general requirement for them, and I think it is still true for any further EXT-SEAMCALLs.

So I think a general helper for EXT-SEAMCALLs makes sense.

Yes conceptually I agree, but not need to distinguish EXT-SEAMCALLs or not IMHO.

The problem is there isn't a common rule to follow.

E.g., let's say "the module can remember thus no resume flag is needed", how about the SEAMCALL inputs? Can the "output" args be directly used as input for retry, or the original input should always be used?

Not to mention there's existing SEAMCALLs which require explicitly setting 'resume=1'.

I believe we can use some smart hack to implement a common one to cover all cases above, but I am not sure whether it's worth to do (maybe we can have a try to see how does it look like, though, I think).

Given the SEAMCALLs for TDX Connect seem to follow one rule to retry, and live migration SEAMCALLs follow another rule, it seems for now the simplest way is to introduce the needed retry helper in the layer of SEAMCALL *user* (TDX Connect and migration).

Quoted 13 lines

TDH.IOMMU.SETUP, however, is another case. It is not a EXT-SEAMCALL but happened to follow the same irq-retry handling process. To avoid code duplication we have:

/*
* seamcall_ret_ir_exec() aliases seamcall_ret_ir_resched() for
* documentation purposes. It documents the TDX Module extension
* seamcalls that are long running / hard-irq preemptible flows that
* generate events. The calls using seamcall_ret_ir_resched() are long
* running flows, that periodically yield.
*/
#define seamcall_ret_ir_exec seamcall_ret_ir_resched

Quoted 4 lines

TDH.IOMMU.SETUP uses seamcall_ret_ir_resched(), and EXT-SEAMCALLs use seamcall_ret_ir_exec().

How do you think?

Sorry I don't quite get.  What does "exec" postfix mean?

From patch 25, they are all in TDX core, so I don't quite get why we need to distinguish EXT-SEAMCALLs vs normal ones. IMHO it's an additional layer which doesn't actually help address any problem.

Btw, we should really get rid of the "resched()" postfix from the function name since cond_resched() is no longer needed and possibility of rescheduling is implied pretty much all places in the kernel code now (except some special code such as code in IRQ context).

#75 lore raw

Re: [PATCH v2 10/31] x86/virt/tdx: Add extra memory to TDX Module for Extensions

Quoted 3 lines · Kai

I believe we can use some smart hack to implement a common one to cover all cases above, but I am not sure whether it's worth to do (maybe we can have a try to see how does it look like, though, I think).

So the problem is use what as input when retry, and one way is to provide a callback to allow the user to provide a specific function to update the 'args' before retrying.

Something like below? I am not sure I like it though, because as Rick said there's too much SEAMCALL wrapper/macros already.

typedef void (*args_update_func_t)(struct tdx_module_args *args,

                                   struct tdx_module_args *ori);

static __always_inline u64 sc_retry_intr(sc_func_t func, u64 fn,
struct tdx_module_args *args, args_update_func_t update_args)

{
        struct tdx_module_args _args = *args;
        u64 ret;

        do {
                ret = sc_retry(func, fn, &_args);

                if (ret != TDX_INTERRUPT_RESUMABLE)
                        break;
        
                update_args(&_args, args);
        } while (1);
        
        *args = _args;

        return ret;
}                                        
#define seamcall_ret_intr(_fn, _args, _args_update_f)   \
        sc_retry_intr(__seamcall_ret, (_fn), (_args), (_args_update_f))
author
#76 lore raw

Re: [PATCH v2 10/31] x86/virt/tdx: Add extra memory to TDX Module for Extensions

In reply to Huang, Kai
Quoted 24 lines · Kai
Quoted 23 lines · Xu Yilun
Quoted 21 lines · Kai
Quoted 7 lines · Rick P
Quoted 6 lines · Kai

Ditto here.  I don't think we should introduce any more cond_resched().

Btw, I think technically we can reuse the seamcall_ir_resched() you introduced later, albeit in which a local '_args' is used as a copy of the original 'args', but that has no harm for the case where we can just use the 'args' to loop.

Quoted 7 lines
Quoted 4 lines

I am wondering whether we can just use that here, or we just get rid of that helper (then open retry by the callers of these SEAMCALL wrappers), since there will be more cases where we need to manually set 'resume=1' in the SEAMCALL input 'args' when retrying TDX_INTERRUPTED_RESUMABLE.

I kind of like the latter option to open code more of this stuff. The stacks of seamcall wrapper macros is already too much.

Agreed.

And SEAMCALL *users* can actually come up with their own version of wrapper(s) to do the retry. E.g., currently seamcall_ir_resched() is only used for IOMMU SEAMCALLs, and we can put this wrapper in the IOMMU code or coco/tdx-host.

Quoted 5 lines
Quoted 3 lines

After we have introduced TDX Module Extension, irq preemptable EXT-SEAMCALLs become a common concept.

It has been a concept since before the EXT-SEAMCALLs actually. For instance,

No, they look similar but different. EXT-SEAMCALLs are truly irq preemptable and resumable to its context. Other SEAMCALLs just periodically yield and don't have a generic way to save/resume their context. Sometime you need to pass in resume flag on 2nd time, which means the secure world forget where they were and can't really resume all by itself.

What I mean is, EXT-SEAMCALLs should never need to play tricks on input parameters. Just input what is originally inputted, the secure world doesn't need hint to resume itself. So the int-retry process should be common.

Quoted 8 lines

TDX live migration using blocking export doesn't need any opt-in via module extension (only the non-blocking way needs), but the SEAMCALLs to export/import TD/vCPU/memory are all interruptible.

In fact, they had the "latency requirement" behind the INTERRUPT_RESUMABLE in mind at the very beginning. It's just at that point all SEAMCALLs were not that heavy.

Quoted 7 lines
Quoted 3 lines

It is irq preemptable so that the
secure world remembers and resumes the context, no need for host to remind via resume lag.

The fact is the aforementioned live migration related export/import SEAMCALLs (there are 8 at least, but maybe more) all requires the explicit setting of 'resume=1' (plus using the SEAMCALL output as input for retry). I don't know

Yes, so they are not truly interrupt resumable and should be specially treated.

Quoted 11 lines

the story behind this, though. There might be some tricky thing here for the module to remember and manage (e.g., migration has a concept of "migration stream", and the resume is per-stream).

Quoted 6 lines

Today there are 3 EXT-SEAMCALLs, TDH_SPDM_CONNECT/DISCONNECT/MNG, irq preemption handling is a general requirement for them, and I think it is still true for any further EXT-SEAMCALLs.

So I think a general helper for EXT-SEAMCALLs makes sense.

Quoted 7 lines

Yes conceptually I agree, but not need to distinguish EXT-SEAMCALLs or not IMHO.

The problem is there isn't a common rule to follow.

E.g., let's say "the module can remember thus no resume flag is needed", how about the SEAMCALL inputs? Can the "output" args be directly used as input for retry, or the original input should always be used?

Since EXT-SEAMCALLs don't depend on input tricks to resume, there could be a common rule, now it is defined as "the original input should always be used".

Quoted 26 lines

Not to mention there's existing SEAMCALLs which require explicitly setting 'resume=1'.

I believe we can use some smart hack to implement a common one to cover all cases above, but I am not sure whether it's worth to do (maybe we can have a try to see how does it look like, though, I think).

Given the SEAMCALLs for TDX Connect seem to follow one rule to retry, and live migration SEAMCALLs follow another rule, it seems for now the simplest way is to introduce the needed retry helper in the layer of SEAMCALL *user* (TDX Connect and migration).

Quoted 13 lines

TDH.IOMMU.SETUP, however, is another case. It is not a EXT-SEAMCALL but happened to follow the same irq-retry handling process. To avoid code duplication we have:

/*
* seamcall_ret_ir_exec() aliases seamcall_ret_ir_resched() for
* documentation purposes. It documents the TDX Module extension
* seamcalls that are long running / hard-irq preemptible flows that
* generate events. The calls using seamcall_ret_ir_resched() are long
* running flows, that periodically yield.
*/
#define seamcall_ret_ir_exec seamcall_ret_ir_resched

Quoted 6 lines
Quoted 4 lines

TDH.IOMMU.SETUP uses seamcall_ret_ir_resched(), and EXT-SEAMCALLs use seamcall_ret_ir_exec().

How do you think?

Sorry I don't quite get.  What does "exec" postfix mean?

It is 'execution', means EXT-SEAMCALLs can resume their execution. But since you have concern, maybe some better name?

Quoted 3 lines

From patch 25, they are all in TDX core, so I don't quite get why we need to distinguish EXT-SEAMCALLs vs normal ones. IMHO it's an additional layer which

EXT-SEAMCALLs have generic way to resume, while others don't. So we need a helper for EXT-SEAMCALLs. For other SEAMCALLs that happens to process the same way, we are avoiding code duplication, but should clearly distinguish the purpose so make another name as documentation.

But if any concern, we could delete the int-retry support for normal SEAMCALLs, they are not generic as you said.

Quoted 6 lines

doesn't actually help address any problem.

Btw, we should really get rid of the "resched()" postfix from the function name since cond_resched() is no longer needed and possibility of rescheduling is implied pretty much all places in the kernel code now (except some special code such as code in IRQ context).

Yes, thanks to remind me again.

author
#77 lore raw

PATCH v2 11/31 x86/virt/tdx: Make TDX Module initialize Extensions

In reply to Xu Yilun

After providing all required memory to TDX Module, initialize the Extensions via TDH.EXT.INIT, and then Extension-SEAMCALLs can be used.

The initialization of Extensions touches the required memory (previously provided by TDH.EXT.MEM.ADD) in private manner. If failed, flush cache before freeing these memory, to avoid private cache write back damages the shared pages.

TDX should use movdir64b to clear private pages when reclaiming them on older platforms with the X86_BUG_TDX_PW_MCE erratum. For simplicity, don't expect this errata on any TDX Extensions supported platform. So TDX Extensions & all features that require TDX Extensions (e.g. TDX Connect) will not call the clearing helpers.

Note the "ext_required" global metadata specifies if TDH.EXT.INIT call is needed. If 0, the Extensions are already working, so skip the SEAMCALL.

---
 arch/x86/virt/vmx/tdx/tdx.h |  1 +
 arch/x86/virt/vmx/tdx/tdx.c | 45 +++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)
arch/x86/virt/vmx/tdx/tdx.h +1 -0
index 31ccdfcf518c..a26fe94c07ff 100644
@@ -60,6 +60,7 @@
 #define TDH_VP_WR			43
 #define TDH_SYS_CONFIG_V0		45
 #define TDH_SYS_CONFIG			SEAMCALL_LEAF_VER(TDH_SYS_CONFIG_V0, 1)
+#define TDH_EXT_INIT			60
 #define TDH_EXT_MEM_ADD			61
 
 /* TDX page types */
arch/x86/virt/vmx/tdx/tdx.c +45 -0

69 lines not on this page · raw message

Signature · 1 line
2.25.1
#78 lore raw

Re: [PATCH v2 11/31] x86/virt/tdx: Make TDX Module initialize Extensions

Quoted 69 lines · Xu Yilun

After providing all required memory to TDX Module, initialize the Extensions via TDH.EXT.INIT, and then Extension-SEAMCALLs can be used.

The initialization of Extensions touches the required memory (previously provided by TDH.EXT.MEM.ADD) in private manner. If failed, flush cache before freeing these memory, to avoid private cache write back damages the shared pages.

TDX should use movdir64b to clear private pages when reclaiming them on older platforms with the X86_BUG_TDX_PW_MCE erratum. For simplicity, don't expect this errata on any TDX Extensions supported platform. So TDX Extensions & all features that require TDX Extensions (e.g. TDX Connect) will not call the clearing helpers.

Note the "ext_required" global metadata specifies if TDH.EXT.INIT call is needed. If 0, the Extensions are already working, so skip the SEAMCALL.

---
 arch/x86/virt/vmx/tdx/tdx.h |  1 +
 arch/x86/virt/vmx/tdx/tdx.c | 45 +++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)
arch/x86/virt/vmx/tdx/tdx.h +1 -0
index 31ccdfcf518c..a26fe94c07ff 100644
@@ -60,6 +60,7 @@
 #define TDH_VP_WR			43
 #define TDH_SYS_CONFIG_V0		45
 #define TDH_SYS_CONFIG			SEAMCALL_LEAF_VER(TDH_SYS_CONFIG_V0, 1)
+#define TDH_EXT_INIT			60
 #define TDH_EXT_MEM_ADD			61
 
 /* TDX page types */
arch/x86/virt/vmx/tdx/tdx.c +17 -0
index 5fae17c13191..4134f92425da 100644
@@ -1519,6 +1519,23 @@ static void tdx_clflush_page_array(struct tdx_page_array *array)
 		tdx_clflush_page(array->pages[array->offset + i]);
 }
 
+/* Initialize the TDX Module Extensions then Extension-SEAMCALLs can be used */
+static int tdx_ext_init(void)
+{
+	struct tdx_module_args args = {};
+	u64 r;
+
+	do {
+		r = seamcall(TDH_EXT_INIT, &args);
+		cond_resched();
+	} while (r == TDX_INTERRUPTED_RESUMABLE);
+
+	if (r != TDX_SUCCESS)
+		return -EFAULT;
+
+	return 0;
+}
+
 static int tdx_ext_mem_add(struct tdx_page_array *ext_mem)
 {
 	struct tdx_module_args args = {
@@ -1572,6 +1589,17 @@ static int __maybe_unused init_tdx_ext(void)
 	if (!(tdx_sysinfo.features.tdx_features0 & TDX_FEATURES0_EXT))
 		return 0;
 

arch/x86/virt/vmx/tdx/tdx.c static int __maybe_unused init_tdx_ext(void) · line 1601

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +10 -0
+	/*
+	 * With this errata, TDX should use movdir64b to clear private pages
+	 * when reclaiming them. See tdx_quirk_reset_paddr().
+	 *
+	 * Don't expect this errata on any TDX Extensions supported platform.
+	 * All features require TDX Extensions (including TDX Extensions
+	 * itself) will never call tdx_quirk_reset_paddr().
+	 */
+	if (boot_cpu_has_bug(X86_BUG_TDX_PW_MCE))
+		return -ENXIO;

I don't know if we are going to want to sprinkle these over every new feature until the end of time. If this feature will only show up on the platforms with this erratum, then I say we just drop the check.

arch/x86/virt/vmx/tdx/tdx.c static int __maybe_unused init_tdx_ext(void) · line 1620

Quoted 11 lines
arch/x86/virt/vmx/tdx/tdx.c +4 -0
+
 	nr_pages = tdx_sysinfo.ext.memory_pool_required_pages;
 	/*
 	 * memory_pool_required_pages == 0 means no need to add more pages,
@@ -1587,6 +1615,20 @@ static int __maybe_unused init_tdx_ext(void)
 			goto out_ext_mem;
 	}
 
+	/*
+	 * ext_required == 0 means no need to call TDH.EXT.INIT, the Extensions
+	 * are already working.

How does this scenario happen exactly? And why not check it above at the beginning? Before the allocation, so it doesn't need to free.

Is there a scenario where the memory needs to be given, but the extension is already inited?

Quoted 15 lines
arch/x86/virt/vmx/tdx/tdx.c +11 -0
+	 */
+	if (tdx_sysinfo.ext.ext_required) {
+		ret = tdx_ext_init();
+		/*
+		 * Some pages may have been touched by the TDX module.
+		 * Flush cache before returning these pages to kernel.
+		 */
+		if (ret)
+			goto out_flush;
+	}
+
 	/* Extension memory is never reclaimed once assigned */
 	tdx_page_array_ctrl_leak(ext_mem);
 

arch/x86/virt/vmx/tdx/tdx.c static int __maybe_unused init_tdx_ext(void) · line 1641

Quoted 5 lines
arch/x86/virt/vmx/tdx/tdx.c +2 -0
@@ -1595,6 +1637,9 @@ static int __maybe_unused init_tdx_ext(void)
 
 	return 0;
 
+out_flush:
+	if (ext_mem)

For the error path we don't need to be efficient. But also why does it assume tdx_ext_init() can touch the pages, but tdx_ext_mem_add() can't?

Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +1 -0
+		wbinvd_on_all_cpus();
 out_ext_mem:
 	tdx_page_array_free(ext_mem);
 
author
#79 lore raw

Re: [PATCH v2 11/31] x86/virt/tdx: Make TDX Module initialize Extensions

Quoted 7 lines
Quoted 7 lines

static int tdx_ext_mem_add(struct tdx_page_array *ext_mem) {

arch/x86/virt/vmx/tdx/tdx.c +0 -0
 	struct tdx_module_args args = {
@@ -1572,6 +1589,17 @@ static int __maybe_unused init_tdx_ext(void)
 	if (!(tdx_sysinfo.features.tdx_features0 & TDX_FEATURES0_EXT))
 		return 0;
 
Quoted 14 lines

arch/x86/virt/vmx/tdx/tdx.c static int __maybe_unused init_tdx_ext(void) · line 1601

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +10 -0
+	/*
+	 * With this errata, TDX should use movdir64b to clear private pages
+	 * when reclaiming them. See tdx_quirk_reset_paddr().
+	 *
+	 * Don't expect this errata on any TDX Extensions supported platform.
+	 * All features require TDX Extensions (including TDX Extensions
+	 * itself) will never call tdx_quirk_reset_paddr().
+	 */
+	if (boot_cpu_has_bug(X86_BUG_TDX_PW_MCE))
+		return -ENXIO;

I don't know if we are going to want to sprinkle these over every new feature until the end of time. If this feature will only show up on the platforms with this erratum, then I say we just drop the check.

Make sense.

Quoted 16 lines

arch/x86/virt/vmx/tdx/tdx.c static int __maybe_unused init_tdx_ext(void) · line 1620

Quoted 12 lines
arch/x86/virt/vmx/tdx/tdx.c +4 -0
+
 	nr_pages = tdx_sysinfo.ext.memory_pool_required_pages;
 	/*
 	 * memory_pool_required_pages == 0 means no need to add more pages,
@@ -1587,6 +1615,20 @@ static int __maybe_unused init_tdx_ext(void)
 			goto out_ext_mem;
 	}
 
+	/*
+	 * ext_required == 0 means no need to call TDH.EXT.INIT, the Extensions
+	 * are already working.

How does this scenario happen exactly? And why not check it above at the beginning? Before the allocation, so it doesn't need to free.

Quoted 2 lines

Is there a scenario where the memory needs to be given, but the extension is already inited?

mm.. you are right. It leads to something absurd.

I checked with TDX Module team again. The correct understanding is:

- TDX_FEATURES0_EXT bit shows Extensions is supported.
- optional feature bits are selected on TDH_SYS_CONFIG
- If one of the optional feature (e.g. TDX CONNECT) requires Extention, memory_pool_required_pages > 0 && ext_required == 1. Otherwise no need to initialize Extension.

So yes, I should check memory_pool_required_pages && ext_required at the beginning.

Quoted 17 lines
Quoted 17 lines
arch/x86/virt/vmx/tdx/tdx.c +11 -0
+	 */
+	if (tdx_sysinfo.ext.ext_required) {
+		ret = tdx_ext_init();
+		/*
+		 * Some pages may have been touched by the TDX module.
+		 * Flush cache before returning these pages to kernel.
+		 */
+		if (ret)
+			goto out_flush;
+	}
+
 	/* Extension memory is never reclaimed once assigned */
 	tdx_page_array_ctrl_leak(ext_mem);
 
@@ -1595,6 +1637,9 @@ static int __maybe_unused init_tdx_ext(void)
 
Quoted 7 lines

arch/x86/virt/vmx/tdx/tdx.c static int __maybe_unused init_tdx_ext(void) · line 1641

Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +2 -0
 	return 0;
 
+out_flush:
+	if (ext_mem)

For the error path we don't need to be efficient. But also why does it assume tdx_ext_init() can touch the pages, but tdx_ext_mem_add() can't?

The tdx_ext_mem_add() only collects memory, tdx_ext_init() does the actual initialization for Extensions and touches the memory. But the detail of when touching the pages is not specified in SPEC, do you think host doesn't have to tell the difference, just flush when any one of ext-SEAMCALLs is called?

Quoted 7 lines
Quoted 6 lines
arch/x86/virt/vmx/tdx/tdx.c +1 -0
+		wbinvd_on_all_cpus();
 out_ext_mem:
 	tdx_page_array_free(ext_mem);
 
#80 lore raw

Re: [PATCH v2 11/31] x86/virt/tdx: Make TDX Module initialize Extensions

Quoted 14 lines · Xu Yilun
Quoted 9 lines

arch/x86/virt/vmx/tdx/tdx.c static int __maybe_unused init_tdx_ext(void) · line 1620

Quoted 3 lines
arch/x86/virt/vmx/tdx/tdx.c +3 -0
+	/*
+	 * ext_required == 0 means no need to call TDH.EXT.INIT, the Extensions
+	 * are already working.

How does this scenario happen exactly? And why not check it above at the beginning? Before the allocation, so it doesn't need to free.

Is there a scenario where the memory needs to be given, but the extension is already inited?

mm.. you are right. It leads to something absurd.

I checked with TDX Module team again. The correct understanding is:

Quoted 8 lines

 - TDX_FEATURES0_EXT bit shows Extensions is supported.
 - optional feature bits are selected on TDH_SYS_CONFIG
 - If one of the optional feature (e.g. TDX CONNECT) requires Extention, memory_pool_required_pages > 0 && ext_required == 1. Otherwise no need to initialize Extension.

So yes, I should check memory_pool_required_pages && ext_required at the beginning.

My understanding is different:

Per spec, the 'EXT_REQUIRED' global metadata just means "Return true if the TDH.EXT.INIT is required to be called", so I think, architecturally, it's possible that one particular feature only requires additional memory pool but doesn't explicitly need to call TDH.EXT.INIT. Or some feature may not require any additional memory pool but needs TDH.EXT.INIT.  Or require both (such as TDX Connect I presume).

We can safely assume 2) and 3) are not required if no feature is configured in 1) (backward compatibility). But when there is, I think we can just:

1) If 'MEMORY_POOL_REQUIRED_PAGES' is not zero, do TDH.EXT.MEM.ADD
2) If 'EXT_REQUIRED' is true, do TDH.EXT.INIT

author
#81 lore raw

Re: [PATCH v2 11/31] x86/virt/tdx: Make TDX Module initialize Extensions

Quoted 21 lines · Kai
Quoted 20 lines · Xu Yilun
Quoted 9 lines

arch/x86/virt/vmx/tdx/tdx.c static int __maybe_unused init_tdx_ext(void) · line 1620

Quoted 3 lines
arch/x86/virt/vmx/tdx/tdx.c +3 -0
+	/*
+	 * ext_required == 0 means no need to call TDH.EXT.INIT, the Extensions
+	 * are already working.

How does this scenario happen exactly? And why not check it above at the beginning? Before the allocation, so it doesn't need to free.

Is there a scenario where the memory needs to be given, but the extension is already inited?

mm.. you are right. It leads to something absurd.

I checked with TDX Module team again. The correct understanding is:

 - TDX_FEATURES0_EXT bit shows Extensions is supported.
 - optional feature bits are selected on TDH_SYS_CONFIG
 - If one of the optional feature (e.g. TDX CONNECT) requires Extention, memory_pool_required_pages > 0 && ext_required == 1. Otherwise no need to initialize Extension.

Quoted 7 lines
Quoted 2 lines

So yes, I should check memory_pool_required_pages && ext_required at the beginning.

My understanding is different:

Per spec, the 'EXT_REQUIRED' global metadata just means "Return true if the TDH.EXT.INIT is required to be called", so I think, architecturally, it's

Maybe these text should be improved. They just literally tell how, so leads to our disagreement.

Quoted 3 lines

possible that one particular feature only requires additional memory pool but doesn't explicitly need to call TDH.EXT.INIT. Or some feature may not require any additional memory pool but needs TDH.EXT.INIT.

This is different from what I've been told by TDX Module team. Do you have a real setup like that?

My gut feeling also tells me there is little chance that:

1. The Extensions is already working (cause no need to call TDH.EXT.INIT) while we are still adding memory.
2. The Extensions could enable long running / hard-irq preemptible flows with no memory consumption.

#82 lore raw

Re: [PATCH v2 11/31] x86/virt/tdx: Make TDX Module initialize Extensions

Quoted 40 lines · Xu Yilun
Quoted 21 lines · Kai
Quoted 20 lines · Xu Yilun
Quoted 9 lines

arch/x86/virt/vmx/tdx/tdx.c static int __maybe_unused init_tdx_ext(void) · line 1620

Quoted 3 lines
arch/x86/virt/vmx/tdx/tdx.c +3 -0
+	/*
+	 * ext_required == 0 means no need to call TDH.EXT.INIT, the Extensions
+	 * are already working.

How does this scenario happen exactly? And why not check it above at the beginning? Before the allocation, so it doesn't need to free.

Is there a scenario where the memory needs to be given, but the extension is already inited?

mm.. you are right. It leads to something absurd.

I checked with TDX Module team again. The correct understanding is:

 - TDX_FEATURES0_EXT bit shows Extensions is supported.
 - optional feature bits are selected on TDH_SYS_CONFIG
 - If one of the optional feature (e.g. TDX CONNECT) requires Extention, memory_pool_required_pages > 0 && ext_required == 1. Otherwise no need to initialize Extension.

Quoted 7 lines
Quoted 2 lines

So yes, I should check memory_pool_required_pages && ext_required at the beginning.

My understanding is different:

Per spec, the 'EXT_REQUIRED' global metadata just means "Return true if the TDH.EXT.INIT is required to be called", so I think, architecturally, it's

Maybe these text should be improved. They just literally tell how, so leads to our disagreement.

Quoted 3 lines

possible that one particular feature only requires additional memory pool but doesn't explicitly need to call TDH.EXT.INIT. Or some feature may not require any additional memory pool but needs TDH.EXT.INIT.

This is different from what I've been told by TDX Module team. Do you have a real setup like that?

Quoted 6 lines

My gut feeling also tells me there is little chance that:

1. The Extensions is already working (cause no need to call TDH.EXT.INIT) while we are still adding memory.
2. The Extensions could enable long running / hard-irq preemptible flows with no memory consumption.

I don't think we need to guess here. We need to understand what the architecture behaviour is and then write code based on that. If there's anything not clear on architecture, we need to ask the module team to clarify.

Back to the architecture behaviour, the spec "4.4 TDX Module Extension Initialization" says:

1. The host VMM configures the desired TDX features ...

2. Based on the enabled features, the TDX Module checks whether a memory pool is required and if so, calculates its required size.

3. The host VMM reads MEMORY_POOL_REQUIRED_PAGES, the number of missing TDX Module’s memory pool pages, using TDH.SYS.RD.

4. Once the TDX Module has been initialized (TDH.SYS.KEY.CONFIG was called on all packages), the host VMM can call TDH.EXT.MEM.ADD multiple times to add the required number of memory pages to the TDX Module’s memory pool.

5. The host VMM reads EXT_REQUIRED, which indicates whether the TDX Module extension is required to be initialized, using TDH.SYS.RD. If required, the host VMM can then call TDH.EXT.INIT to initialize the TDX Module extension.

So to me it's clear that we need to do things in following:

1. Opt-in ext features in TDH.SYS.CONFIG
2. Read MEMORY_POOL_REQUIRED_PAGES and EXT_REQUIRED
3. After TDH.SYS.KEY.CONFIG, initialize the module extension:
a. If MEMORY_POOL_REQUIRED_PAGES is not zero, do TDH.EXT.MEM.ADD b. If EXT_REQUIRED is not zero, do TDH.EXT.INIT

To me there's no need to make any other assumption here.

#83 lore raw

Re: [PATCH v2 11/31] x86/virt/tdx: Make TDX Module initialize Extensions

Quoted 4 lines · Kai

I don't think we need to guess here.  We need to understand what the architecture behaviour is and then write code based on that.  If there's anything not clear on architecture, we need to ask the module team to clarify.

A general comment...I think this is the wrong attitude. If we have some assumptions that can simplify the kernel code, let's get them in the spec. Or otherwise get an agreement from the TDX module to make them no longer assumptions.

Just silently hoping our assumptions are true is not great either. But uncritically implementing the architecture that is handed down is totally wrong.

#84 lore raw

Re: [PATCH v2 11/31] x86/virt/tdx: Make TDX Module initialize Extensions

Quoted 6 lines · Rick P
Quoted 4 lines · Kai

I don't think we need to guess here.  We need to understand what the architecture behaviour is and then write code based on that.  If there's anything not clear on architecture, we need to ask the module team to clarify.

Quoted 8 lines

A general comment...I think this is the wrong attitude. If we have some assumptions that can simplify the kernel code, let's get them in the spec. Or otherwise get an agreement from the TDX module to make them no longer assumptions.

Just silently hoping our assumptions are true is not great either. But uncritically implementing the architecture that is handed down is totally wrong.

Yes agree in principle.

Maybe I am missing something, but I don't see any big issue for this particular architecture behaviour. If you have any good idea to improve this flow then great.

author
#85 lore raw

PATCH v2 12/31 x86/virt/tdx: Enable the Extensions after basic TDX Module init

In reply to Xu Yilun

The detailed initialization flow for TDX Module Extensions has been fully implemented. Enable the flow after basic TDX Module initialization.

Theoretically, the Extensions can be initialized later when the first usage of the Extension-SEAMCALL comes. That would save or postpone the usage of ~50M memory. But it isn't worth the complexity, the needs for Extensions are vast but the savings are little for a typical TDX capable system (about 0.001% of memory). So just enable it along with the basic TDX.

---
 arch/x86/virt/vmx/tdx/tdx.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/virt/vmx/tdx/tdx.c +5 -1
index 4134f92425da..0e1ad793e648 100644
@@ -1580,7 +1580,7 @@ static int tdx_ext_mem_setup(struct tdx_page_array *ext_mem)
 	return 0;
 }
 
-static int __maybe_unused init_tdx_ext(void)
+static int init_tdx_ext(void)
 {
 	struct tdx_page_array *ext_mem = NULL;
 	unsigned int nr_pages;
@@ -1705,6 +1705,10 @@ static int init_tdx_module(void)
 	if (ret)
 		goto err_reset_pamts;
 
+	ret = init_tdx_ext();
+	if (ret)
+		goto err_reset_pamts;
+
 	pr_info("%lu KB allocated for PAMT\n", tdmrs_count_pamt_kb(&tdx_tdmr_list));
 
 out_put_tdxmem:
Signature · 1 line
2.25.1
author
#86 lore raw

PATCH v2 13/31 x86/virt/tdx: Extend tdx_clflush_page() to handle compound pages

In reply to Xu Yilun

Use page_size() to correctly flush the range that a compound page covers.

Recall that TDX Module requires VMM to provide IOMMU metadata known as IOMMU_MT, which contains some multi-order pages. Like all other metadata, TDX Module will convert these multi-order pages to private so VMM should flush the shared cache beforehand. Extend tdx_clflush_page() to handle this case.

The usage of tdx_clflush_page() for IOMMU_MT will be introduced later, but the change stands as a valid improvement on its own.

---
 arch/x86/virt/vmx/tdx/tdx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/virt/vmx/tdx/tdx.c +1 -1
index 0e1ad793e648..e7d47fbe7057 100644
@@ -1510,7 +1510,7 @@ static int init_tdmrs(struct tdmr_info_list *tdmr_list)
  */
 static void tdx_clflush_page(struct page *page)
 {
-	clflush_cache_range(page_to_virt(page), PAGE_SIZE);
+	clflush_cache_range(page_to_virt(page), page_size(page));
 }
 
 static void tdx_clflush_page_array(struct tdx_page_array *array)
Signature · 1 line
2.25.1
author
#87 lore raw

PATCH v2 14/31 PCI/TSM: Report active IDE streams per host bridge

In reply to Xu Yilun

From: Dan Williams <[email protected]>

The first attempt at an ABI for this failed to account for naming collisions across host bridges:

Commit a4438f06b1db ("PCI/TSM: Report active IDE streams")

Revive this ABI with a per host bridge link that appears at first stream creation for a given host bridge and disappears after the last stream is removed.

For systems with many host bridge objects it allows:

ls /sys/class/tsm/tsmN/pci*/stream*

...to find all the host bridges with active streams without first iterating over all host bridges. Yilun notes that is handy to have this short cut [1] and from an administrator perspective it helps with inventory for constrained stream resources.

---
 Documentation/ABI/testing/sysfs-class-tsm | 13 +++
 include/linux/pci-ide.h                   |  2 +
 include/linux/tsm.h                       |  3 +
 drivers/pci/ide.c                         |  4 +
 drivers/virt/coco/tsm-core.c              | 97 +++++++++++++++++++++++
 5 files changed, 119 insertions(+)
Documentation/ABI/testing/sysfs-class-tsm +13 -0
index 2949468deaf7..1ddb8f357961 100644
@@ -7,3 +7,16 @@ Description:
 		signals when the PCI layer is able to support establishment of
 		link encryption and other device-security features coordinated
 		through a platform tsm.
+
+What:		/sys/class/tsm/tsmN/pciDDDD:BB
+Contact:	[email protected]
+Description:
+		(RO) When a PCIe host bridge has established a secure connection
+		via a TSM to an endpoint, this symlink appears. It facilitates a
+		TSM instance scoped view of PCIe Link Encryption and Secure
+		Session resource consumption across host bridges. The symlink
+		appears when a host bridge has 1 or more IDE streams established
+		with this TSM, and disappears when that number returns to 0. See
+		Documentation/ABI/testing/sysfs-devices-pci-host-bridge for the
+		description of the pciDDDD:BB/streamH.R.E symlink and the
+		pciDDDD:BB/available_secure_streams attribute.
include/linux/pci-ide.h +2 -0
index ae07d9f699c0..381a1bf22a95 100644
@@ -82,6 +82,7 @@ struct pci_ide_regs {
  * @host_bridge_stream: allocated from host bridge @ide_stream_ida pool
  * @stream_id: unique Stream ID (within Partner Port pairing)
  * @name: name of the established Selective IDE Stream in sysfs
+ * @tsm_dev: For TSM established IDE, the TSM device context
  *
  * Negative @stream_id values indicate "uninitialized" on the
  * expectation that with TSM established IDE the TSM owns the stream_id
@@ -93,6 +94,7 @@ struct pci_ide {
 	u8 host_bridge_stream;
 	int stream_id;
 	const char *name;
+	struct tsm_dev *tsm_dev;
 };
 
 /*
include/linux/tsm.h +3 -0
index 381c53244c83..7f72a154b6b2 100644
@@ -123,4 +123,7 @@ int tsm_report_unregister(const struct tsm_report_ops *ops);
 struct tsm_dev *tsm_register(struct device *parent, struct pci_tsm_ops *ops);
 void tsm_unregister(struct tsm_dev *tsm_dev);
 struct tsm_dev *find_tsm_dev(int id);
+struct pci_ide;
+int tsm_ide_stream_register(struct pci_ide *ide);
+void tsm_ide_stream_unregister(struct pci_ide *ide);
 #endif /* __TSM_H */
drivers/pci/ide.c +4 -0
index be74e8f0ae21..b35e8aba7ecb 100644
@@ -11,6 +11,7 @@
 #include <linux/pci_regs.h>
 #include <linux/slab.h>
 #include <linux/sysfs.h>
+#include <linux/tsm.h>
 
 #include "pci.h"
 
@@ -372,6 +373,9 @@ void pci_ide_stream_release(struct pci_ide *ide)
 	if (ide->partner[PCI_IDE_EP].enable)
 		pci_ide_stream_disable(pdev, ide);
 
+	if (ide->tsm_dev)
+		tsm_ide_stream_unregister(ide);
+
 	if (ide->partner[PCI_IDE_RP].setup)
 		pci_ide_stream_teardown(rp, ide);
 
drivers/virt/coco/tsm-core.c +97 -0

119 lines not on this page · raw message

Signature · 1 line
2.25.1
#88 lore raw

Re: [PATCH v2 14/31] PCI/TSM: Report active IDE streams per host bridge

Quoted 15 lines · Xu Yilun

From: Dan Williams <[email protected]>

The first attempt at an ABI for this failed to account for naming collisions across host bridges:

Commit a4438f06b1db ("PCI/TSM: Report active IDE streams")

Revive this ABI with a per host bridge link that appears at first stream creation for a given host bridge and disappears after the last stream is removed.

For systems with many host bridge objects it allows:

ls /sys/class/tsm/tsmN/pci*/stream*

Quoted 7 lines

...to find all the host bridges with active streams without first iterating over all host bridges. Yilun notes that is handy to have this short cut [1] and from an administrator perspective it helps with inventory for constrained stream resources.

Oh, looks like I had yet to get this reviewed on the list. Can I add your review and / or test tags and take this for v7.1?

author
#89 lore raw

Re: [PATCH v2 14/31] PCI/TSM: Report active IDE streams per host bridge

Quoted 21 lines · Dan Williams
Quoted 20 lines · Xu Yilun

From: Dan Williams <[email protected]>

The first attempt at an ABI for this failed to account for naming collisions across host bridges:

Commit a4438f06b1db ("PCI/TSM: Report active IDE streams")

Revive this ABI with a per host bridge link that appears at first stream creation for a given host bridge and disappears after the last stream is removed.

For systems with many host bridge objects it allows:

ls /sys/class/tsm/tsmN/pci*/stream*

...to find all the host bridges with active streams without first iterating over all host bridges. Yilun notes that is handy to have this short cut [1] and from an administrator perspective it helps with inventory for constrained stream resources.

Quoted 5 lines
Quoted 2 lines

Oh, looks like I had yet to get this reviewed on the list. Can I add your review and / or test tags and take this for v7.1?

Yeah, I've added my RB in that series. Hope we could have this one in v7.1.

author
#90 lore raw

PATCH v2 15/31 coco/tdx-host: Introduce a "tdx_host" device

In reply to Xu Yilun

From: Chao Gao <[email protected]>

TDX depends on a platform firmware module that is invoked via instructions similar to vmenter (i.e. enter into a new privileged "root-mode" context to manage private memory and private device mechanisms). It is a software construct that depends on the CPU vmxon state to enable invocation of TDX module ABIs. Unlike other Trusted Execution Environment (TEE) platform implementations that employ a firmware module running on a PCI device with an MMIO mailbox for communication, TDX has no hardware device to point to as the TEE Secure Manager (TSM).

Create a virtual device not only to align with other implementations but also to make it easier to

- expose metadata (e.g., TDX module version, seamldr version etc) to the userspace as device attributes

- implement firmware uploader APIs which are tied to a device. This is needed to support TDX module runtime updates

- enable TDX Connect which will share a common infrastructure with other platform implementations. In the TDX Connect context, every architecture has a TSM, represented by a PCIe or virtual device. The new "tdx_host" device will serve the TSM role.

A faux device is used for TDX because the TDX module is singular within the system and lacks associated platform resources. Using a faux device eliminates the need to create a stub bus.

The call to tdx_get_sysinfo() ensures that the TDX module is ready to provide services.

Note that AMD has a PCI device for the PSP for SEV and ARM CCA will likely have a faux device [1].

---
 drivers/virt/coco/Kconfig             |  2 ++
 drivers/virt/coco/tdx-host/Kconfig    | 10 +++++++
 drivers/virt/coco/Makefile            |  1 +
 drivers/virt/coco/tdx-host/Makefile   |  1 +
 arch/x86/virt/vmx/tdx/tdx.c           |  2 +-
 drivers/virt/coco/tdx-host/tdx-host.c | 43 +++++++++++++++++++++++++++
 6 files changed, 58 insertions(+), 1 deletion(-)
 create mode 100644 drivers/virt/coco/tdx-host/Kconfig
 create mode 100644 drivers/virt/coco/tdx-host/Makefile
 create mode 100644 drivers/virt/coco/tdx-host/tdx-host.c
drivers/virt/coco/Kconfig +2 -0
index df1cfaf26c65..f7691f64fbe3 100644
@@ -17,5 +17,7 @@ source "drivers/virt/coco/arm-cca-guest/Kconfig"
 source "drivers/virt/coco/guest/Kconfig"
 endif
 
+source "drivers/virt/coco/tdx-host/Kconfig"
+
 config TSM
 	bool
drivers/virt/coco/tdx-host/Kconfig new file +10 -0
new file mode 100644
index 000000000000..d35d85ef91c0
@@ -0,0 +1,10 @@
+config TDX_HOST_SERVICES
+	tristate "TDX Host Services Driver"
+	depends on INTEL_TDX_HOST
+	default m
+	help
+	  Enable access to TDX host services like module update and
+	  extensions (e.g. TDX Connect).
+
+	  Say y or m if enabling support for confidential virtual machine
+	  support (CONFIG_INTEL_TDX_HOST). The module is called tdx_host.ko.
drivers/virt/coco/Makefile +1 -0
index cb52021912b3..b323b0ae4f82 100644
@@ -6,6 +6,7 @@ obj-$(CONFIG_EFI_SECRET)	+= efi_secret/
 obj-$(CONFIG_ARM_PKVM_GUEST)	+= pkvm-guest/
 obj-$(CONFIG_SEV_GUEST)		+= sev-guest/
 obj-$(CONFIG_INTEL_TDX_GUEST)	+= tdx-guest/
+obj-$(CONFIG_INTEL_TDX_HOST)	+= tdx-host/
 obj-$(CONFIG_ARM_CCA_GUEST)	+= arm-cca-guest/
 obj-$(CONFIG_TSM) 		+= tsm-core.o
 obj-$(CONFIG_TSM_GUEST)		+= guest/
drivers/virt/coco/tdx-host/Makefile new file +1 -0
new file mode 100644
index 000000000000..e61e749a8dff
@@ -0,0 +1 @@
+obj-$(CONFIG_TDX_HOST_SERVICES) += tdx-host.o
arch/x86/virt/vmx/tdx/tdx.c +1 -1
index e7d47fbe7057..cd0948794b6c 100644
@@ -2057,7 +2057,7 @@ const struct tdx_sys_info *tdx_get_sysinfo(void)
 
 	return p;
 }
-EXPORT_SYMBOL_FOR_KVM(tdx_get_sysinfo);
+EXPORT_SYMBOL_FOR_MODULES(tdx_get_sysinfo, "kvm-intel,tdx-host");
 
 u32 tdx_get_nr_guest_keyids(void)
 {
drivers/virt/coco/tdx-host/tdx-host.c new file +43 -0

43 lines not on this page · raw message

Signature · 1 line
2.25.1
#91 lore raw

Re: [PATCH v2 15/31] coco/tdx-host: Introduce a "tdx_host" device

Quoted 11 lines · Xu Yilun

From: Chao Gao <[email protected]>

TDX depends on a platform firmware module that is invoked via instructions similar to vmenter (i.e. enter into a new privileged "root-mode" context to manage private memory and private device mechanisms). It is a software construct that depends on the CPU vmxon state to enable invocation of TDX module ABIs. Unlike other Trusted Execution Environment (TEE) platform implementations that employ a firmware module running on a PCI device with an MMIO mailbox for communication, TDX has no hardware device to point to as the TEE Secure Manager (TSM).

I'm trying to get up to speed on Confidential IO. Thank you for this summary!

s/TEE Secure Manager (TSM)/TEE Security Manager/

Quoted 37 lines

Create a virtual device not only to align with other implementations but also to make it easier to

- expose metadata (e.g., TDX module version, seamldr version etc) to the userspace as device attributes

- implement firmware uploader APIs which are tied to a device. This is needed to support TDX module runtime updates

- enable TDX Connect which will share a common infrastructure with other platform implementations. In the TDX Connect context, every architecture has a TSM, represented by a PCIe or virtual device. The new "tdx_host" device will serve the TSM role.

A faux device is used for TDX because the TDX module is singular within the system and lacks associated platform resources. Using a faux device eliminates the need to create a stub bus.

The call to tdx_get_sysinfo() ensures that the TDX module is ready to provide services.

Note that AMD has a PCI device for the PSP for SEV and ARM CCA will likely have a faux device [1].

[...snip...]

author
#92 lore raw

PATCH v2 16/31 coco/tdx-host: Support Link TSM for TDX host

In reply to Xu Yilun

Register a Link TSM instance to support host side TSM operations for TDISP, when the TDX Connect support bit is set by TDX Module in tdx_feature0.

This is the main purpose of an independent tdx-host module out of TDX core. Recall that a TEE Security Manager (TSM) is a platform agent that speaks the TEE Device Interface Security Protocol (TDISP) to PCIe devices and manages private memory resources for the platform. An independent tdx-host module allows for device-security enumeration and initialization flows to be deferred from other TDX Module initialization requirements. Crucially, when / if TDX Module init moves earlier in x86 initialization flow this driver is still guaranteed to run after IOMMU and PCI init (i.e. subsys_initcall() vs device_initcall()).

The ability to unload the module, or unbind the driver is also useful for debug and coarse grained transitioning between PCI TSM operation and PCI CMA operation (native kernel PCI device authentication).

For now only verify TDX Connect support in TDX Module and enable TDX Module Extentions. The TSM support are basic boilerplate with operation flows to be added later.

---
 drivers/virt/coco/tdx-host/Kconfig    |   5 +
 drivers/virt/coco/tdx-host/tdx-host.c | 138 +++++++++++++++++++++++++-
 2 files changed, 141 insertions(+), 2 deletions(-)
drivers/virt/coco/tdx-host/Kconfig +5 -0
index d35d85ef91c0..32add81b7d56 100644
@@ -8,3 +8,8 @@ config TDX_HOST_SERVICES
 
 	  Say y or m if enabling support for confidential virtual machine
 	  support (CONFIG_INTEL_TDX_HOST). The module is called tdx_host.ko.
+
+config TDX_CONNECT
+	def_bool y
+	depends on TDX_HOST_SERVICES
+	depends on PCI_TSM
drivers/virt/coco/tdx-host/tdx-host.c +136 -2

159 lines not on this page · raw message

Signature · 1 line
2.25.1
#93 lore raw

Re: [PATCH v2 16/31] coco/tdx-host: Support Link TSM for TDX host

drivers/virt/coco/tdx-host/tdx-host.c static const struct x86_cpu_id tdx_host_ids[] = { · line 113

Quoted 7 lines · Xu Yilun

[...snip...]

drivers/virt/coco/tdx-host/tdx-host.c +4 -0
+static struct pci_tsm_ops tdx_tsm_link_ops = {
+	.probe = tdx_tsm_link_probe,
+	.remove = tdx_tsm_link_remove,
+	.connect = tdx_tsm_link_connect,

Since .connect is called from pci_tsm_connect(), which is in turn called from connect_store, is the plan that the VMM should write to

/sys/bus/pci/devices/.../tsm/connect

to set up SPDM and IDE before starting the guest?

Quoted 6 lines
drivers/virt/coco/tdx-host/tdx-host.c +3 -0
+	.disconnect = tdx_tsm_link_disconnect,
+};
+
 
[...snip...]
author
#94 lore raw

PATCH v2 17/31 acpi: Add KEYP support to fw_table parsing

In reply to Xu Yilun

From: Dave Jiang <[email protected]>

KEYP ACPI table can be parsed using the common fw_table handlers. Add additional support to detect and parse the table.

---
 include/linux/acpi.h     |  3 +++
 include/linux/fw_table.h |  1 +
 drivers/acpi/tables.c    | 12 +++++++++++-
 lib/fw_table.c           |  9 +++++++++
 4 files changed, 24 insertions(+), 1 deletion(-)
include/linux/acpi.h +3 -0
index 4d2f0bed7a06..e5b51bd46600 100644
@@ -247,6 +247,9 @@ int acpi_table_parse_madt(enum acpi_madt_type id,
 int __init_or_acpilib
 acpi_table_parse_cedt(enum acpi_cedt_type id,
 		      acpi_tbl_entry_handler_arg handler_arg, void *arg);
+int __init_or_acpilib
+acpi_table_parse_keyp(enum acpi_keyp_type id,
+		      acpi_tbl_entry_handler_arg handler_arg, void *arg);
 
 int acpi_parse_mcfg (struct acpi_table_header *header);
 void acpi_table_print_madt_entry (struct acpi_subtable_header *madt);
include/linux/fw_table.h +1 -0
index 9bd605b87c4c..293252cb0b7e 100644
@@ -36,6 +36,7 @@ union acpi_subtable_headers {
 	struct acpi_prmt_module_header prmt;
 	struct acpi_cedt_header cedt;
 	struct acpi_cdat_header cdat;
+	struct acpi_keyp_common_header keyp;
 };
 
 int acpi_parse_entries_array(char *id, unsigned long table_size,
drivers/acpi/tables.c +11 -1
index 4286e4af1092..8dc60632faf3 100644
@@ -299,6 +299,16 @@ acpi_table_parse_cedt(enum acpi_cedt_type id,
 }
 EXPORT_SYMBOL_ACPI_LIB(acpi_table_parse_cedt);
 
+int __init_or_acpilib
+acpi_table_parse_keyp(enum acpi_keyp_type id,
+		      acpi_tbl_entry_handler_arg handler_arg, void *arg)
+{
+	return __acpi_table_parse_entries(ACPI_SIG_KEYP,
+					  sizeof(struct acpi_table_keyp), id,
+					  NULL, handler_arg, arg, 0);
+}
+EXPORT_SYMBOL_ACPI_LIB(acpi_table_parse_keyp);
+
 int __init acpi_table_parse_entries(char *id, unsigned long table_size,
 				    int entry_id,
 				    acpi_tbl_entry_handler handler,
@@ -408,7 +418,7 @@ static const char table_sigs[][ACPI_NAMESEG_SIZE] __nonstring_array __initconst
 	ACPI_SIG_PSDT, ACPI_SIG_RSDT, ACPI_SIG_XSDT, ACPI_SIG_SSDT,
 	ACPI_SIG_IORT, ACPI_SIG_NFIT, ACPI_SIG_HMAT, ACPI_SIG_PPTT,
 	ACPI_SIG_NHLT, ACPI_SIG_AEST, ACPI_SIG_CEDT, ACPI_SIG_AGDI,
-	ACPI_SIG_NBFT, ACPI_SIG_SWFT, ACPI_SIG_MPAM};
+	ACPI_SIG_NBFT, ACPI_SIG_SWFT, ACPI_SIG_MPAM, ACPI_SIG_KEYP};
 
 #define ACPI_HEADER_SIZE sizeof(struct acpi_table_header)
 
lib/fw_table.c +9 -0
index 16291814450e..147e3895e94c 100644
@@ -20,6 +20,7 @@ enum acpi_subtable_type {
 	ACPI_SUBTABLE_PRMT,
 	ACPI_SUBTABLE_CEDT,
 	CDAT_SUBTABLE,
+	ACPI_SUBTABLE_KEYP,
 };
 
 struct acpi_subtable_entry {
@@ -41,6 +42,8 @@ acpi_get_entry_type(struct acpi_subtable_entry *entry)
 		return entry->hdr->cedt.type;
 	case CDAT_SUBTABLE:
 		return entry->hdr->cdat.type;
+	case ACPI_SUBTABLE_KEYP:
+		return entry->hdr->keyp.type;
 	}
 	return 0;
 }
@@ -61,6 +64,8 @@ acpi_get_entry_length(struct acpi_subtable_entry *entry)
 		__le16 length = (__force __le16)entry->hdr->cdat.length;
 
 		return le16_to_cpu(length);
+	case ACPI_SUBTABLE_KEYP:
+		return entry->hdr->keyp.length;
 	}
 	}
 	return 0;
@@ -80,6 +85,8 @@ acpi_get_subtable_header_length(struct acpi_subtable_entry *entry)
 		return sizeof(entry->hdr->cedt);
 	case CDAT_SUBTABLE:
 		return sizeof(entry->hdr->cdat);
+	case ACPI_SUBTABLE_KEYP:
+		return sizeof(entry->hdr->keyp);
 	}
 	return 0;
 }
@@ -95,6 +102,8 @@ acpi_get_subtable_type(char *id)
 		return ACPI_SUBTABLE_CEDT;
 	if (strncmp(id, ACPI_SIG_CDAT, 4) == 0)
 		return CDAT_SUBTABLE;
+	if (strncmp(id, ACPI_SIG_KEYP, 4) == 0)
+		return ACPI_SUBTABLE_KEYP;
 	return ACPI_SUBTABLE_COMMON;
 }
 
Signature · 1 line
2.25.1
author
#95 lore raw

PATCH v2 18/31 iommu/vt-d: Cache max domain ID to avoid redundant calculation

In reply to Xu Yilun

From: Lu Baolu <[email protected]>

The cap_ndoms() helper calculates the maximum available domain ID from the value of capability register, which can be inefficient if called repeatedly. Cache the maximum supported domain ID in max_domain_id field during initialization to avoid redundant calls to cap_ndoms() throughout the IOMMU driver.

No functionality change.

---
 drivers/iommu/intel/iommu.h |  1 +
 drivers/iommu/intel/dmar.c  |  1 +
 drivers/iommu/intel/iommu.c | 10 +++++-----
 3 files changed, 7 insertions(+), 5 deletions(-)
drivers/iommu/intel/iommu.h +1 -0
index 599913fb65d5..4a21ab6a311d 100644
@@ -705,6 +705,7 @@ struct intel_iommu {
 	/* mutex to protect domain_ida */
 	struct mutex	did_lock;
 	struct ida	domain_ida; /* domain id allocator */
+	unsigned long	max_domain_id;
 	unsigned long	*copied_tables; /* bitmap of copied tables */
 	spinlock_t	lock; /* protect context, domain ids */
 	struct root_entry *root_entry; /* virtual address */
drivers/iommu/intel/dmar.c +1 -0
index d68c06025cac..93efd1a5dc5b 100644
@@ -1099,6 +1099,7 @@ static int alloc_iommu(struct dmar_drhd_unit *drhd)
 	spin_lock_init(&iommu->lock);
 	ida_init(&iommu->domain_ida);
 	mutex_init(&iommu->did_lock);
+	iommu->max_domain_id = cap_ndoms(iommu->cap);
 
 	ver = readl(iommu->reg + DMAR_VER_REG);
 	pr_info("%s: reg_base_addr %llx ver %d:%d cap %llx ecap %llx\n",
drivers/iommu/intel/iommu.c +5 -5
index ef7613b177b9..9a57f78647ed 100644
@@ -1043,7 +1043,7 @@ int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu)
 	}
 
 	num = ida_alloc_range(&iommu->domain_ida, IDA_START_DID,
-			      cap_ndoms(iommu->cap) - 1, GFP_KERNEL);
+			      iommu->max_domain_id - 1, GFP_KERNEL);
 	if (num < 0) {
 		pr_err("%s: No free domain ids\n", iommu->name);
 		goto err_unlock;
@@ -1107,7 +1107,7 @@ static void copied_context_tear_down(struct intel_iommu *iommu,
 	did_old = context_domain_id(context);
 	context_clear_entry(context);
 
-	if (did_old < cap_ndoms(iommu->cap)) {
+	if (did_old < iommu->max_domain_id) {
 		iommu->flush.flush_context(iommu, did_old,
 					   PCI_DEVID(bus, devfn),
 					   DMA_CCMD_MASK_NOBIT,
@@ -1505,7 +1505,7 @@ static int copy_context_table(struct intel_iommu *iommu,
 			continue;
 
 		did = context_domain_id(&ce);
-		if (did >= 0 && did < cap_ndoms(iommu->cap))
+		if (did >= 0 && did < iommu->max_domain_id)
 			ida_alloc_range(&iommu->domain_ida, did, did, GFP_KERNEL);
 
 		set_context_copied(iommu, bus, devfn);
@@ -2425,7 +2425,7 @@ static ssize_t domains_supported_show(struct device *dev,
 				      struct device_attribute *attr, char *buf)
 {
 	struct intel_iommu *iommu = dev_to_intel_iommu(dev);
-	return sysfs_emit(buf, "%ld\n", cap_ndoms(iommu->cap));
+	return sysfs_emit(buf, "%ld\n", iommu->max_domain_id);
 }
 static DEVICE_ATTR_RO(domains_supported);
 
@@ -2436,7 +2436,7 @@ static ssize_t domains_used_show(struct device *dev,
 	unsigned int count = 0;
 	int id;
 
-	for (id = 0; id < cap_ndoms(iommu->cap); id++)
+	for (id = 0; id < iommu->max_domain_id; id++)
 		if (ida_exists(&iommu->domain_ida, id))
 			count++;
 
Signature · 1 line
2.25.1
Reviewed-by on v2 18/31
#96 lore raw

RE: [PATCH v2 18/31] iommu/vt-d: Cache max domain ID to avoid redundant calculation

Quoted 12 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

From: Lu Baolu <[email protected]>

The cap_ndoms() helper calculates the maximum available domain ID from the value of capability register, which can be inefficient if called repeatedly. Cache the maximum supported domain ID in max_domain_id field
during initialization to avoid redundant calls to cap_ndoms() throughout the IOMMU driver.

Quoted 4 lines

No functionality change.

maintainer
#97 lore raw

Re: [PATCH v2 18/31] iommu/vt-d: Cache max domain ID to avoid redundant calculation

In reply to Xu Yilun
Quoted 10 lines · Xu Yilun

From: Lu Baolu<[email protected]>

The cap_ndoms() helper calculates the maximum available domain ID from the value of capability register, which can be inefficient if called repeatedly. Cache the maximum supported domain ID in max_domain_id field during initialization to avoid redundant calls to cap_ndoms() throughout the IOMMU driver.

No functionality change.

Quoted 7 lines
---
  drivers/iommu/intel/iommu.h |  1 +
  drivers/iommu/intel/dmar.c  |  1 +
  drivers/iommu/intel/iommu.c | 10 +++++-----
  3 files changed, 7 insertions(+), 5 deletions(-)

This patch has been queued for iommu/next. Thanks!

author
#98 lore raw

PATCH v2 19/31 iommu/vt-d: Reserve the MSB domain ID bit for the TDX module

In reply to Xu Yilun

From: Lu Baolu <[email protected]>

The Intel TDX Connect Architecture Specification defines some enhancements for the VT-d architecture to introduce IOMMU support for TEE-IO requests. Section 2.2, 'Trusted DMA' states that:

"I/O TLB and DID Isolation – When IOMMU is enabled to support TDX Connect, the IOMMU restricts the VMM’s DID setting, reserving the MSB bit for the TDX module. The TDX module always sets this reserved bit on the trusted DMA table. IOMMU tags IOTLB, PASID cache, and context entries to indicate whether they were created from TEE-IO transactions, ensuring isolation between TEE and non-TEE requests in translation caches."

Reserve the MSB in the domain ID for the TDX module's use if the enhancement is required, which is detected if the ECAP.TDXCS bit in the VT-d extended capability register is set and the TVM Usable field of the ACPI KEYP table is set.

---
 drivers/iommu/intel/iommu.h |  1 +
 drivers/iommu/intel/dmar.c  | 52 ++++++++++++++++++++++++++++++++++++-
 2 files changed, 52 insertions(+), 1 deletion(-)
drivers/iommu/intel/iommu.h +1 -0
index 4a21ab6a311d..0c2b4e38dee7 100644
@@ -192,6 +192,7 @@
  */
 
 #define ecap_pms(e)		(((e) >> 51) & 0x1)
+#define ecap_tdxc(e)		(((e) >> 50) & 0x1)
 #define ecap_rps(e)		(((e) >> 49) & 0x1)
 #define ecap_smpwc(e)		(((e) >> 48) & 0x1)
 #define ecap_flts(e)		(((e) >> 47) & 0x1)
drivers/iommu/intel/dmar.c +51 -1

64 lines not on this page · raw message

Signature · 1 line
2.25.1
#99 lore raw

Re: [PATCH v2 19/31] iommu/vt-d: Reserve the MSB domain ID bit for the TDX module

kernel test robot noticed the following build warnings:

Hi Xu,

kernel test robot noticed the following build warnings:

[auto build test WARNING on 11439c4635edd669ae435eec308f4ab8a0804808]

url:    https://github.com/intel-lab-lkp/linux/commits/Xu-Yilun/x86-tdx-Move-all-TDX-error-defines-into-asm-shared-tdx_errno-h/20260328-151524
base:   11439c4635edd669ae435eec308f4ab8a0804808
patch link:    https://lore.kernel.org/r/20260327160132.2946114-20-yilun.xu%40linux.intel.com
patch subject: [PATCH v2 19/31] iommu/vt-d: Reserve the MSB domain ID bit for the TDX module
config: i386-randconfig-141-20260328 (https://download.01.org/0day-ci/archive/20260329/[email protected]/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
smatch: v0.5.0-9004-gb810ac53
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260329/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>, old ones prefixed by <<):

>> WARNING: modpost: vmlinux: section mismatch in reference: iommu_max_domain_id+0x55 (section: .text.iommu_max_domain_id) -> acpi_table_parse_keyp (section: .init.text)
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
maintainer
#100 lore raw

Re: [PATCH v2 19/31] iommu/vt-d: Reserve the MSB domain ID bit for the TDX module

Quoted 13 lines · kernel test robot

kernel test robot noticed the following build warnings:

[auto build test WARNING on 11439c4635edd669ae435eec308f4ab8a0804808]

url:https://github.com/intel-lab-lkp/linux/commits/Xu-Yilun/x86-tdx-Move- all-TDX-error-defines-into-asm-shared-tdx_errno-h/20260328-151524
base: 11439c4635edd669ae435eec308f4ab8a0804808
patch link:https://lore.kernel.org/r/20260327160132.2946114-20- yilun.xu%40linux.intel.com
patch subject: [PATCH v2 19/31] iommu/vt-d: Reserve the MSB domain ID bit for the TDX module
config: i386-randconfig-141-20260328 (https://download.01.org/0day-ci/archive/20260329/202603290006.za7iiDgF- [email protected]/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
smatch: v0.5.0-9004-gb810ac53
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260329/202603290006.za7iiDgF- [email protected]/reproduce)

Quoted 8 lines

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot<[email protected]>
| Closes:https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>, old ones prefixed by <<):

Quoted 1 line
Quoted 1 line

WARNING: modpost: vmlinux: section mismatch in reference: iommu_max_domain_id+0x55 (section: .text.iommu_max_domain_id) -> acpi_table_parse_keyp (section: .init.text)

acpi_table_parse_keyp() is marked as __init. But this patch causes the intel iommu driver to call it from a runtime function.

int __init_or_acpilib
acpi_table_parse_keyp(enum acpi_keyp_type id,
acpi_tbl_entry_handler_arg handler_arg, void *arg)
{
return __acpi_table_parse_entries(ACPI_SIG_KEYP,
sizeof(struct
acpi_table_keyp), id,

                                          NULL, handler_arg, arg, 0);
}

One way to solve this might be parsing the table once in the __init context and store the result in variable that could be used after boot.

How about the following additional change (untested)?

drivers/iommu/intel/dmar.c +15 -19
index 097c4a90302f..0b384a58a3a0 100644
@@ -63,6 +63,7 @@ LIST_HEAD(dmar_drhd_units);
 struct acpi_table_header * __initdata dmar_tbl;
 static int dmar_dev_scope_status = 1;
 static DEFINE_IDA(dmar_seq_ids);
+static bool tdx_tvm_usable __ro_after_init;
 
 static int alloc_iommu(struct dmar_drhd_unit *drhd);
 static void free_iommu(struct intel_iommu *iommu);
@@ -915,6 +916,17 @@ dmar_validate_one_drhd(struct acpi_dmar_header *entry, void *arg)
 	return 0;
 }
 
+static void __init intel_iommu_check_tdxc_enhancement(void)
+{
+	int tvm_usable = 0;
+	int ret;
+
+	ret = acpi_table_parse_keyp(ACPI_KEYP_TYPE_CONFIG_UNIT,
+				    keyp_config_unit_tvm_usable, &tvm_usable);
+	if (ret >= 0)
+		tdx_tvm_usable = !!tvm_usable;
+}
+
 void __init detect_intel_iommu(void)
 {
 	int ret;
@@ -923,6 +935,8 @@ void __init detect_intel_iommu(void)
 		.ignore_unhandled = true,
 	};
 
+	intel_iommu_check_tdxc_enhancement();
+
 	down_write(&dmar_global_lock);
 	ret = dmar_table_detect();
 	if (!ret)
@@ -1046,24 +1060,6 @@ static int keyp_config_unit_tvm_usable(union acpi_subtable_headers *header,
 	return 0;
 }
 
-static bool platform_is_tdxc_enhanced(void)
-{
-	static int tvm_usable = -1;
-	int ret;
-
-	/* only need to parse once */
-	if (tvm_usable != -1)
-		return !!tvm_usable;
-
-	tvm_usable = 0;
-	ret = acpi_table_parse_keyp(ACPI_KEYP_TYPE_CONFIG_UNIT,
-				    keyp_config_unit_tvm_usable, &tvm_usable);
-	if (ret < 0)
-		tvm_usable = 0;
-
-	return !!tvm_usable;
-}
-
 static unsigned long iommu_max_domain_id(struct intel_iommu *iommu)
 {
 	unsigned long ndoms = cap_ndoms(iommu->cap);
@@ -1075,7 +1071,7 @@ static unsigned long iommu_max_domain_id(struct intel_iommu *iommu)
 	 * the VMM’s DID setting, reserving the MSB bit for the TDX module. The
 	 * TDX module always sets this reserved bit on the trusted DMA table.
 	 */
-	if (ecap_tdxc(iommu->ecap) && platform_is_tdxc_enhanced()) {
+	if (ecap_tdxc(iommu->ecap) && tdx_tvm_usable) {
 		pr_info_once("Most Significant Bit of domain ID reserved.\n");
 		return ndoms >> 1;
 	}

Thanks,
baolu

author
#101 lore raw

Re: [PATCH v2 19/31] iommu/vt-d: Reserve the MSB domain ID bit for the TDX module

Quoted 40 lines · Baolu Lu
Quoted 19 lines · kernel test robot

kernel test robot noticed the following build warnings:

[auto build test WARNING on 11439c4635edd669ae435eec308f4ab8a0804808]

url:https://github.com/intel-lab-lkp/linux/commits/Xu-Yilun/x86-tdx-Move- all-TDX-error-defines-into-asm-shared-tdx_errno-h/20260328-151524
base: 11439c4635edd669ae435eec308f4ab8a0804808
patch link:https://lore.kernel.org/r/20260327160132.2946114-20-
yilun.xu%40linux.intel.com
patch subject: [PATCH v2 19/31] iommu/vt-d: Reserve the MSB domain ID bit for the TDX module
config: i386-randconfig-141-20260328
(https://download.01.org/0day-ci/archive/20260329/202603290006.za7iiDgF-
[email protected]/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
smatch: v0.5.0-9004-gb810ac53
reproduce (this is a W=1 build):
(https://download.01.org/0day-ci/archive/20260329/202603290006.za7iiDgF-
[email protected]/reproduce)

Quoted 8 lines

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot<[email protected]>
| Closes:https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>, old ones prefixed by <<):

Quoted 1 line
Quoted 1 line

WARNING: modpost: vmlinux: section mismatch in reference: iommu_max_domain_id+0x55 (section: .text.iommu_max_domain_id) -> acpi_table_parse_keyp (section: .init.text)

acpi_table_parse_keyp() is marked as __init. But this patch causes the intel iommu driver to call it from a runtime function.

int __init_or_acpilib
acpi_table_parse_keyp(enum acpi_keyp_type id,
acpi_tbl_entry_handler_arg handler_arg, void *arg)
{
return __acpi_table_parse_entries(ACPI_SIG_KEYP,
sizeof(struct acpi_table_keyp),
id,

Quoted 2 lines
                                          NULL, handler_arg, arg, 0);
}

Is it better we configure ACPI table as library, so that drivers could use it freely at runtime? tdx-host also uses this function.

--------8<--------

drivers/iommu/intel/Kconfig +1 -0
index 5471f814e073..55188d6d38bb 100644
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 # Intel IOMMU support
 config DMAR_TABLE
+       select ACPI_TABLE_LIB
        bool
 
 config DMAR_PERF
maintainer
#102 lore raw

Re: [PATCH v2 19/31] iommu/vt-d: Reserve the MSB domain ID bit for the TDX module

Quoted 55 lines · Xu Yilun
Quoted 40 lines · Baolu Lu
Quoted 19 lines · kernel test robot

kernel test robot noticed the following build warnings:

[auto build test WARNING on 11439c4635edd669ae435eec308f4ab8a0804808]

url:https://github.com/intel-lab-lkp/linux/commits/Xu-Yilun/x86-tdx-Move- all-TDX-error-defines-into-asm-shared-tdx_errno-h/20260328-151524
base: 11439c4635edd669ae435eec308f4ab8a0804808
patch link:https://lore.kernel.org/r/20260327160132.2946114-20-
yilun.xu%40linux.intel.com
patch subject: [PATCH v2 19/31] iommu/vt-d: Reserve the MSB domain ID bit for the TDX module
config: i386-randconfig-141-20260328
(https://download.01.org/0day-ci/archive/20260329/202603290006.za7iiDgF-
[email protected]/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
smatch: v0.5.0-9004-gb810ac53
reproduce (this is a W=1 build):
(https://download.01.org/0day-ci/archive/20260329/202603290006.za7iiDgF-
[email protected]/reproduce)

Quoted 8 lines

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot<[email protected]>
| Closes:https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>, old ones prefixed by <<):

Quoted 1 line
Quoted 1 line

WARNING: modpost: vmlinux: section mismatch in reference: iommu_max_domain_id+0x55 (section: .text.iommu_max_domain_id) -> acpi_table_parse_keyp (section: .init.text)

acpi_table_parse_keyp() is marked as __init. But this patch causes the intel iommu driver to call it from a runtime function.

int __init_or_acpilib
acpi_table_parse_keyp(enum acpi_keyp_type id,
acpi_tbl_entry_handler_arg handler_arg, void *arg)
{
return __acpi_table_parse_entries(ACPI_SIG_KEYP,
sizeof(struct acpi_table_keyp),
id,

Quoted 2 lines
                                           NULL, handler_arg, arg, 0);
}

Is it better we configure ACPI table as library, so that drivers could use it freely at runtime? tdx-host also uses this function.

--------8<--------

drivers/iommu/intel/Kconfig +0 -0
index 5471f814e073..55188d6d38bb 100644

drivers/iommu/intel/Kconfig line 7

Quoted 7 lines
drivers/iommu/intel/Kconfig +1 -0
@@ -1,6 +1,7 @@
  # SPDX-License-Identifier: GPL-2.0-only
  # Intel IOMMU support
  config DMAR_TABLE
+       select ACPI_TABLE_LIB
         bool
 
  config DMAR_PERF

This looks better.

Thanks,
baolu

#103 lore raw

Re: [PATCH v2 19/31] iommu/vt-d: Reserve the MSB domain ID bit for the TDX module

In reply to Xu Yilun
kernel test robot noticed the following build warnings:

Hi Xu,

kernel test robot noticed the following build warnings:

[auto build test WARNING on 11439c4635edd669ae435eec308f4ab8a0804808]

url:    https://github.com/intel-lab-lkp/linux/commits/Xu-Yilun/x86-tdx-Move-all-TDX-error-defines-into-asm-shared-tdx_errno-h/20260328-151524
base:   11439c4635edd669ae435eec308f4ab8a0804808
patch link:    https://lore.kernel.org/r/20260327160132.2946114-20-yilun.xu%40linux.intel.com
patch subject: [PATCH v2 19/31] iommu/vt-d: Reserve the MSB domain ID bit for the TDX module
config: x86_64-defconfig (https://download.01.org/0day-ci/archive/20260329/[email protected]/config)
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260329/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>, old ones prefixed by <<):

>> WARNING: modpost: vmlinux: section mismatch in reference: alloc_iommu.cold+0x49 (section: .text.unlikely) -> acpi_table_parse_keyp (section: .init.text)
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#104 lore raw

RE: [PATCH v2 19/31] iommu/vt-d: Reserve the MSB domain ID bit for the TDX module

In reply to Xu Yilun

Review

Quoted 5 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

(quoted diff) +2 -0
+
+static bool platform_is_tdxc_enhanced(void)

platform_support_tdxc()

Quoted 12 lines
drivers/iommu/intel/dmar.c +11 -0
+{
+	static int tvm_usable = -1;
+	int ret;
+
+	/* only need to parse once */
+	if (tvm_usable != -1)
+		return !!tvm_usable;
+
+	tvm_usable = 0;
+	ret = acpi_table_parse_keyp(ACPI_KEYP_TYPE_CONFIG_UNIT,
+				    keyp_config_unit_tvm_usable,

&tvm_usable);

drivers/iommu/intel/dmar.c static int map_iommu(struct intel_iommu *iommu, struct dmar_drhd_unit *drhd) · line 1062

Quoted 2 lines
drivers/iommu/intel/dmar.c +2 -0
+	if (ret < 0)
+		tvm_usable = 0;

this is useless. tvm_usable is already set to '0' before the function call.

Quoted 20 lines
drivers/iommu/intel/dmar.c +10 -0
+
+	return !!tvm_usable;
+}
+
+static unsigned long iommu_max_domain_id(struct intel_iommu *iommu)
+{
+	unsigned long ndoms = cap_ndoms(iommu->cap);
+
+	/*
+	 * Intel TDX Connect Architecture Specification, Section 2.2 Trusted

DMA

(quoted diff) +2 -0
+	 *
+	 * When IOMMU is enabled to support TDX Connect, the IOMMU

restricts

(quoted diff) +1 -0
+	 * the VMM’s DID setting, reserving the MSB bit for the TDX module.

The

drivers/iommu/intel/dmar.c +4 -0
+	 * TDX module always sets this reserved bit on the trusted DMA table.
+	 */
+	if (ecap_tdxc(iommu->ecap) && platform_is_tdxc_enhanced()) {
+		pr_info_once("Most Significant Bit of domain ID
Quoted 1 line

reserved.\n");

'... reserved for TDX Connect'

drivers/iommu/intel/dmar.c static int map_iommu(struct intel_iommu *iommu, struct dmar_drhd_unit *drhd) · line 1082

Quoted 3 lines
drivers/iommu/intel/dmar.c +3 -0
+		return ndoms >> 1;
+	}
+

Here we need more words to explain the strategy here.

The comment says "When IOMMU is *enabled*...", but the code here just checks the static capability. It's probably a design choice that you don't want to add complexity on recycling DIDs when TDX connect is actually enabled, but it's worth a note here.

btw in patch23 commit msg:

"
There is no dedicated way to enumerate which IOMMU devices support trusted operations. The host has to call TDH.IOMMU.SETUP on all IOMMU devices and tell their trusted capability by the return value. "

which implies that ecap_tdxc() alone doesn't really report the capability?

anyway all of those need a better explanation here...

author
#105 lore raw

Re: [PATCH v2 19/31] iommu/vt-d: Reserve the MSB domain ID bit for the TDX module

Quoted 6 lines

Here we need more words to explain the strategy here.

The comment says "When IOMMU is *enabled*...", but the code here just checks the static capability. It's probably a design choice that you don't want to add complexity on recycling DIDs when TDX connect is actually enabled, but it's worth a note here.

Yes, that's the rationale. I'll add it to comments.

Quoted 10 lines

btw in patch23 commit msg:

"
There is no dedicated way to enumerate which IOMMU devices support trusted operations. The host has to call TDH.IOMMU.SETUP on all IOMMU devices and tell their trusted capability by the return value. "

which implies that ecap_tdxc() alone doesn't really report the capability?

Ah, good catch. Let me explain:

ecap_tdxc does report the capability. This bit is special cause both trusted part & untrusted part access it.

For IOMMU driver (which now handles the untrusted part), it can directly query to this bit and decide what to do.

But for tdx-host driver which handles the trusted part, it shouldn't speculate into the IOMMU for capability enumeration. TDX Module has more concerns about trusted capability, including the related I/O stack capabilities e.g. SPDM/IDE cap... So in patch23 I actually mean we don't have an enumeration SEAMCALL for trusted capability, I will refactor that message:

There is no dedicated *SEAMCALL* to enumerate which IOMMU devices support trusted operations...

Quoted 2 lines

anyway all of those need a better explanation here...

#106 lore raw

RE: [PATCH v2 19/31] iommu/vt-d: Reserve the MSB domain ID bit for the TDX module

Quoted 11 lines

From: Xu Yilun <[email protected]>
Sent: Wednesday, April 22, 2026 2:01 PM

Quoted 6 lines

Here we need more words to explain the strategy here.

The comment says "When IOMMU is *enabled*...", but the code here just checks the static capability. It's probably a design choice that you don't want to add complexity on recycling DIDs when TDX connect is actually enabled, but it's worth a note here.

Yes, that's the rationale. I'll add it to comments.

btw halving the DID space permanently on any platforms supporting TDX connect doesn't sound a good design. It may break usages which already uses more than 50% of the DID space but have no business to do with TDX connect.

It makes more sense to cut it down in-fly when tdx connect is initialized. If the higher half DIDs have been used then fail TDX connect. otherwise adjust the max domain id.

Quoted 17 lines
Quoted 11 lines

btw in patch23 commit msg:

"
There is no dedicated way to enumerate which IOMMU devices support trusted operations. The host has to call TDH.IOMMU.SETUP on all IOMMU devices and tell their trusted capability by the return value. "

which implies that ecap_tdxc() alone doesn't really report the capability?

Ah, good catch. Let me explain:

ecap_tdxc does report the capability. This bit is special cause both trusted part & untrusted part access it.

Quoted 6 lines

For IOMMU driver (which now handles the untrusted part), it can directly query to this bit and decide what to do.

But for tdx-host driver which handles the trusted part, it shouldn't speculate into the IOMMU for capability enumeration. TDX Module has more concerns about trusted capability, including the related I/O stack

I guess "more concerns" means that there are more conditions for TDX module to look at beyond ecap_tdxc(), so it's not appropriate for tdx-host driver to check ecap alone.

Quoted 10 lines

capabilities e.g. SPDM/IDE cap... So in patch23 I actually mean we don't have an enumeration SEAMCALL for trusted capability, I will refactor that message:

There is no dedicated *SEAMCALL* to enumerate which IOMMU devices
support
trusted operations...

Quoted 2 lines

anyway all of those need a better explanation here...

author
#107 lore raw

Re: [PATCH v2 19/31] iommu/vt-d: Reserve the MSB domain ID bit for the TDX module

Quoted 18 lines
Quoted 18 lines
Quoted 9 lines

btw in patch23 commit msg:

"
There is no dedicated way to enumerate which IOMMU devices support trusted operations. The host has to call TDH.IOMMU.SETUP on all IOMMU devices and tell their trusted capability by the return value. "

which implies that ecap_tdxc() alone doesn't really report the capability?

Ah, good catch. Let me explain:

ecap_tdxc does report the capability. This bit is special cause both trusted part & untrusted part access it.

For IOMMU driver (which now handles the untrusted part), it can directly query to this bit and decide what to do.

Quoted 7 lines
Quoted 3 lines

But for tdx-host driver which handles the trusted part, it shouldn't speculate into the IOMMU for capability enumeration. TDX Module has more concerns about trusted capability, including the related I/O stack

I guess "more concerns" means that there are more conditions for TDX module to look at beyond ecap_tdxc(), so it's not appropriate for tdx-host driver to check ecap alone.

Exactly.

Quoted 8 lines
Quoted 8 lines

capabilities e.g. SPDM/IDE cap... So in patch23 I actually mean we don't have an enumeration SEAMCALL for trusted capability, I will refactor that message:

There is no dedicated *SEAMCALL* to enumerate which IOMMU devices
support
trusted operations...

author
#108 lore raw

PATCH v2 20/31 x86/virt/tdx: Add a helper to loop on TDX_INTERRUPTED_RESUMABLE

In reply to Xu Yilun

Add a helper to handle SEAMCALL return code TDX_INTERRUPTED_RESUMABLE.

SEAMCALL returns TDX_INTERRUPTED_RESUMABLE to avoid stalling host for long time. After host has handled the interrupt, it calls the interrupted SEAMCALL again and TDX Module continues to execute. TDX Module made progress in this case and would eventually finish. An infinite loop in host should be safe.

The helper is for SEAMCALL wrappers which output information by using seamcall_ret() or seamcall_saved_ret(). The 2 functions overwrite input arguments by outputs but much SEAMCALLs expect the same inputs to resume.

The helper is not for special cases where the SEAMCALL expects modified inputs to resume. The helper is also not for SEAMCALLs with no output, do {...} while (r == TDX_INTERRUPTED_RESUMABLE) just works.

---
 arch/x86/virt/vmx/tdx/tdx.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
arch/x86/virt/vmx/tdx/tdx.c +23 -0
index cd0948794b6c..294f36048c03 100644
@@ -2084,6 +2084,29 @@ static inline u64 tdx_tdr_pa(struct tdx_td *td)
 	return page_to_phys(td->tdr_page);
 }
 
+static u64 __maybe_unused __seamcall_ir_resched(sc_func_t sc_func, u64 fn,
+						struct tdx_module_args *args)
+{
+	struct tdx_module_args _args;
+	u64 r;
+
+	while (1) {
+		_args = *(args);
+		r = sc_retry(sc_func, fn, &_args);
+		if (r != TDX_INTERRUPTED_RESUMABLE)
+			break;
+
+		cond_resched();
+	}
+
+	*args = _args;
+
+	return r;
+}
+
+#define seamcall_ret_ir_resched(fn, args)	\
+	__seamcall_ir_resched(__seamcall_ret, fn, args)
+
 noinstr u64 tdh_vp_enter(struct tdx_vp *td, struct tdx_module_args *args)
 {
 	args->rcx = td->tdvpr_pa;
Signature · 1 line
2.25.1
#109 lore raw

RE: [PATCH v2 20/31] x86/virt/tdx: Add a helper to loop on TDX_INTERRUPTED_RESUMABLE

arch/x86/virt/vmx/tdx/tdx.c static inline u64 tdx_tdr_pa(struct tdx_td *td) · line 2089

Quoted 7 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

+static u64 __maybe_unused __seamcall_ir_resched(sc_func_t sc_func, u64 fn,

arch/x86/virt/vmx/tdx/tdx.c +2 -0
+						struct tdx_module_args *args)
+{

'ir' sounds redundant with the trailing 'resched'?

not big deal, just a bit confusing when seeing it in IOMMU side where 'ir' also refers to 'interrupt remapping' and is frequently used in irq_remapping.c... :)

author
#110 lore raw

Re: [PATCH v2 20/31] x86/virt/tdx: Add a helper to loop on TDX_INTERRUPTED_RESUMABLE

Quoted 9 lines · Kevin

arch/x86/virt/vmx/tdx/tdx.c static inline u64 tdx_tdr_pa(struct tdx_td *td) · line 2089

Quoted 7 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

+static u64 __maybe_unused __seamcall_ir_resched(sc_func_t sc_func, u64 fn,

arch/x86/virt/vmx/tdx/tdx.c +2 -0
+						struct tdx_module_args *args)
+{

'ir' sounds redundant with the trailing 'resched'?

Mm.. I want to 'ir' to reflect the loop-retry is dedicated for INTERRUPTED_RESUMABLE in TDX context. When you say not big deal, I assume I can keep the naming?

Quoted 4 lines

not big deal, just a bit confusing when seeing it in IOMMU side where 'ir' also refers to 'interrupt remapping' and is frequently used in irq_remapping.c... :)

#111 lore raw

RE: [PATCH v2 20/31] x86/virt/tdx: Add a helper to loop on TDX_INTERRUPTED_RESUMABLE

Quoted 15 lines

From: Xu Yilun <[email protected]>
Sent: Wednesday, April 22, 2026 2:05 PM

Quoted 4 lines · Kevin
Quoted 4 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

+static u64 __maybe_unused __seamcall_ir_resched(sc_func_t sc_func,

u64

Quoted 5 lines

arch/x86/virt/vmx/tdx/tdx.c static inline u64 tdx_tdr_pa(struct tdx_td *td) · line 2089

Quoted 3 lines

fn,

arch/x86/virt/vmx/tdx/tdx.c +2 -0
+						struct tdx_module_args *args)
+{

'ir' sounds redundant with the trailing 'resched'?

Quoted 4 lines

Mm.. I want to 'ir' to reflect the loop-retry is dedicated for INTERRUPTED_RESUMABLE in TDX context. When you say not big deal, I assume I can keep the naming?

that's ok

#112 lore raw

RE: [PATCH v2 20/31] x86/virt/tdx: Add a helper to loop on TDX_INTERRUPTED_RESUMABLE

In reply to Tian, Kevin
Quoted 4 lines · Kevin
Quoted 4 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

+static u64 __maybe_unused __seamcall_ir_resched(sc_func_t sc_func, u64

How about __seamcall_retry_interrupted, to emphasize that this function retries a certain kind of seamcall?

I think resched is part of the implementation, of what's done while retrying, which might not need to be part of the name.

No strong opinion though. Thanks for considering.

Quoted 9 lines

arch/x86/virt/vmx/tdx/tdx.c static inline u64 tdx_tdr_pa(struct tdx_td *td) · line 2089

Quoted 3 lines

fn,

arch/x86/virt/vmx/tdx/tdx.c +2 -0
+						struct tdx_module_args *args)
+{

'ir' sounds redundant with the trailing 'resched'?

not big deal, just a bit confusing when seeing it in IOMMU side where 'ir' also refers to 'interrupt remapping' and is frequently used in irq_remapping.c... :)

#113 lore raw

Re: [PATCH v2 20/31] x86/virt/tdx: Add a helper to loop on TDX_INTERRUPTED_RESUMABLE

In reply to Xu Yilun
Quoted 9 lines · Xu Yilun
arch/x86/virt/vmx/tdx/tdx.c +9 -0
+static u64 __maybe_unused __seamcall_ir_resched(sc_func_t sc_func, u64 fn,
+						struct tdx_module_args *args)
+{
+	struct tdx_module_args _args;
+	u64 r;
+
+	while (1) {
+		_args = *(args);
+		r = sc_retry(sc_func, fn, &_args);

arch/x86/virt/vmx/tdx/tdx.c static inline u64 tdx_tdr_pa(struct tdx_td *td) · line 2105

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +10 -0
+		if (r != TDX_INTERRUPTED_RESUMABLE)
+			break;
+
+		cond_resched();
+	}
+
+	*args = _args;
+
+	return r;
+}

Since commit 7dadeaa6e851e ("sched: Further restrict the preemption modes") for x86 only PREEMPT (full) and PREEMPT_LAZY are possible, even when PREEMPT_DYNAMIC is on.

cond_resched() is useful in PREEMPT_NONE and PREEMPT_VOLUNTARY, but it is basically a RET0 in both PREEMPT_LAZY and PREEMPT. My understanding is we shouldn't add any more cond_resched() for x86 now (see the aforementioned commit changelog for more info).

#114 lore raw

Re: [PATCH v2 20/31] x86/virt/tdx: Add a helper to loop on TDX_INTERRUPTED_RESUMABLE

In reply to Xu Yilun
Quoted 11 lines · Xu Yilun

Add a helper to handle SEAMCALL return code TDX_INTERRUPTED_RESUMABLE.

SEAMCALL returns TDX_INTERRUPTED_RESUMABLE to avoid stalling host for long time. After host has handled the interrupt, it calls the interrupted SEAMCALL again and TDX Module continues to execute. TDX Module made progress in this case and would eventually finish. An infinite loop in host should be safe.

The helper is for SEAMCALL wrappers which output information by using seamcall_ret() or seamcall_saved_ret(). The 2 functions overwrite input arguments by outputs but much SEAMCALLs expect the same inputs to

Did you mean most instead of much?

Quoted 17 lines

resume.

The helper is not for special cases where the SEAMCALL expects modified inputs to resume. The helper is also not for SEAMCALLs with no output, do {...} while (r == TDX_INTERRUPTED_RESUMABLE) just works.

---
 arch/x86/virt/vmx/tdx/tdx.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
arch/x86/virt/vmx/tdx/tdx.c +0 -0
index cd0948794b6c..294f36048c03 100644
@@ -2084,6 +2084,29 @@ static inline u64 tdx_tdr_pa(struct tdx_td *td)
 	return page_to_phys(td->tdr_page);

arch/x86/virt/vmx/tdx/tdx.c static inline u64 tdx_tdr_pa(struct tdx_td *td) · line 2094

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +8 -0
 }
 
+static u64 __maybe_unused __seamcall_ir_resched(sc_func_t sc_func, u64 fn,
+						struct tdx_module_args *args)
+{
+	struct tdx_module_args _args;
+	u64 r;
+
+	while (1) {
+		_args = *(args);

Is this copying meant to ensure that every retry has the same input args? Perhaps a comment should be added so it's obvious that every retry will have the same input, and outputs from each retry are dropped.

Quoted 20 lines
arch/x86/virt/vmx/tdx/tdx.c +15 -1
+		r = sc_retry(sc_func, fn, &_args);
+		if (r != TDX_INTERRUPTED_RESUMABLE)
+			break;
+
+		cond_resched();
+	}
+
+	*args = _args;
+
+	return r;
+}
+
+#define seamcall_ret_ir_resched(fn, args)	\
+	__seamcall_ir_resched(__seamcall_ret, fn, args)
+
 noinstr u64 tdh_vp_enter(struct tdx_vp *td, struct tdx_module_args *args)
 {
 	args->rcx = td->tdvpr_pa;
--

2.25.1

author
#115 lore raw

Re: [PATCH v2 20/31] x86/virt/tdx: Add a helper to loop on TDX_INTERRUPTED_RESUMABLE

Quoted 4 lines
Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +4 -0
+static u64 __maybe_unused __seamcall_ir_resched(sc_func_t sc_func, u64 fn,
+						struct tdx_module_args *args)
+{
+	struct tdx_module_args _args;
Quoted 8 lines

arch/x86/virt/vmx/tdx/tdx.c static inline u64 tdx_tdr_pa(struct tdx_td *td) · line 2094

Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +4 -0
+	u64 r;
+
+	while (1) {
+		_args = *(args);

Is this copying meant to ensure that every retry has the same input args? Perhaps a comment should be added so it's obvious that every retry will have the same input, and outputs from each retry are dropped.

Yes, that's the intent.

But now I've found this is not a good convention for INTERRUPTED_RESUMABLE. If outputs are necessary why we drop them, if outputs are not useful why we output them to overwrite inputs. The fact is that TDX module only outputs all 0's to zero out the inputs, that makes no sense.

So we now switch to another existing convention - if there is valid output, use the output to update the input for next loop, or keep the input unchanged. This way we only need to:

	do {
		r = seamcall(TDH_XX_XX, &args);
	} while (r == TDX_INTERRUPTED_RESUMABLE);

	if (r != TDX_SUCCESS)
		return -EFAULT;

https://lore.kernel.org/kvm/[email protected]/
https://lore.kernel.org/kvm/[email protected]/

Quoted 12 lines
Quoted 12 lines
arch/x86/virt/vmx/tdx/tdx.c +11 -0
+		r = sc_retry(sc_func, fn, &_args);
+		if (r != TDX_INTERRUPTED_RESUMABLE)
+			break;
+
+		cond_resched();
+	}
+
+	*args = _args;
+
+	return r;
+}
author
#116 lore raw

PATCH v2 21/31 x86/virt/tdx: Add SEAMCALL wrappers for trusted IOMMU setup and clear

In reply to Xu Yilun

From: Zhenzhong Duan <[email protected]>

Add SEAMCALLs to setup/clear trusted IOMMU for TDX Connect.

Enable TEE I/O support for a target device requires to setup trusted IOMMU for the related IOMMU device first, even only for enabling physical secure links like SPDM/IDE.

TDH.IOMMU.SETUP takes the register base address (VTBAR) to position an IOMMU device, and outputs an IOMMU_ID as the trusted IOMMU identifier. TDH.IOMMU.CLEAR takes the IOMMU_ID to reverse the setup.

More information see Intel TDX Connect ABI Specification [1] Section 3.2 TDX Connect Host-Side (SEAMCALL) Interface Functions.

[1]: https://cdrdv2.intel.com/v1/dl/getContent/858625

---
 arch/x86/include/asm/tdx.h  |  2 ++
 arch/x86/virt/vmx/tdx/tdx.h |  2 ++
 arch/x86/virt/vmx/tdx/tdx.c | 32 ++++++++++++++++++++++++++++++--
 3 files changed, 34 insertions(+), 2 deletions(-)
arch/x86/include/asm/tdx.h +2 -0
index d7605235aa9b..a59e0e43e465 100644
@@ -245,6 +245,8 @@ u64 tdh_mem_page_remove(struct tdx_td *td, u64 gpa, u64 level, u64 *ext_err1, u6
 u64 tdh_phymem_cache_wb(bool resume);
 u64 tdh_phymem_page_wbinvd_tdr(struct tdx_td *td);
 u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct page *page);
+u64 tdh_iommu_setup(u64 vtbar, struct tdx_page_array *iommu_mt, u64 *iommu_id);
+u64 tdh_iommu_clear(u64 iommu_id, struct tdx_page_array *iommu_mt);
 #else
 static inline void tdx_init(void) { }
 static inline int tdx_cpu_enable(void) { return -ENODEV; }
arch/x86/virt/vmx/tdx/tdx.h +2 -0
index a26fe94c07ff..b25c418f6e61 100644
@@ -62,6 +62,8 @@
 #define TDH_SYS_CONFIG			SEAMCALL_LEAF_VER(TDH_SYS_CONFIG_V0, 1)
 #define TDH_EXT_INIT			60
 #define TDH_EXT_MEM_ADD			61
+#define TDH_IOMMU_SETUP			128
+#define TDH_IOMMU_CLEAR			129
 
 /* TDX page types */
 #define	PT_NDA		0x0
arch/x86/virt/vmx/tdx/tdx.c +30 -2
index 294f36048c03..790713881f1f 100644
@@ -2084,8 +2084,8 @@ static inline u64 tdx_tdr_pa(struct tdx_td *td)
 	return page_to_phys(td->tdr_page);
 }
 
-static u64 __maybe_unused __seamcall_ir_resched(sc_func_t sc_func, u64 fn,
-						struct tdx_module_args *args)
+static u64 __seamcall_ir_resched(sc_func_t sc_func, u64 fn,
+				 struct tdx_module_args *args)
 {
 	struct tdx_module_args _args;
 	u64 r;
@@ -2478,3 +2478,31 @@ void tdx_cpu_flush_cache_for_kexec(void)
 }
 EXPORT_SYMBOL_FOR_KVM(tdx_cpu_flush_cache_for_kexec);
 #endif
+
+u64 tdh_iommu_setup(u64 vtbar, struct tdx_page_array *iommu_mt, u64 *iommu_id)
+{
+	struct tdx_module_args args = {
+		.rcx = vtbar,
+		.rdx = virt_to_phys(iommu_mt->root),
+	};
+	u64 r;
+
+	tdx_clflush_page_array(iommu_mt);
+
+	r = seamcall_ret_ir_resched(TDH_IOMMU_SETUP, &args);
+
+	*iommu_id = args.rcx;
+	return r;
+}
+EXPORT_SYMBOL_FOR_MODULES(tdh_iommu_setup, "tdx-host");
+
+u64 tdh_iommu_clear(u64 iommu_id, struct tdx_page_array *iommu_mt)
+{
+	struct tdx_module_args args = {
+		.rcx = iommu_id,
+		.rdx = virt_to_phys(iommu_mt->root),
+	};
+
+	return seamcall_ret_ir_resched(TDH_IOMMU_CLEAR, &args);
+}
+EXPORT_SYMBOL_FOR_MODULES(tdh_iommu_clear, "tdx-host");
Signature · 1 line
2.25.1
#117 lore raw

RE: [PATCH v2 21/31] x86/virt/tdx: Add SEAMCALL wrappers for trusted IOMMU setup and clear

Quoted 6 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

From: Zhenzhong Duan <[email protected]>

Add SEAMCALLs to setup/clear trusted IOMMU for TDX Connect.

what is 'trusted IOMMU'? a new hardware, or some sensitive resource in the IOMMU which is only visible to TDX module?

If the latter it's clearer to say "trusted configuration in IOMMU".

Quoted 4 lines

Enable TEE I/O support for a target device requires to setup trusted IOMMU for the related IOMMU device first, even only for enabling physical secure links like SPDM/IDE.

this series is just about SPDM/IDE. then the first part about TEE I/O is not really relevant.

Quoted 4 lines

TDH.IOMMU.SETUP takes the register base address (VTBAR) to position an IOMMU device, and outputs an IOMMU_ID as the trusted IOMMU identifier. TDH.IOMMU.CLEAR takes the IOMMU_ID to reverse the setup.

Intel IOMMU is called VT-d. It has a register block but not a PCI device so there is no BAR resource related.

let's just call it 'reg_base'

intel-iommu driver already has its own 'id' definition for each iommu device. It's clearer to add a prefix to this new id, e.g. tdx_iommu_id?

author
#118 lore raw

Re: [PATCH v2 21/31] x86/virt/tdx: Add SEAMCALL wrappers for trusted IOMMU setup and clear

Quoted 9 lines · Kevin
Quoted 6 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

From: Zhenzhong Duan <[email protected]>

Add SEAMCALLs to setup/clear trusted IOMMU for TDX Connect.

what is 'trusted IOMMU'? a new hardware, or some sensitive resource in the IOMMU which is only visible to TDX module?

Some extended HW resources in IOMMU so I think the later.

Quoted 2 lines

If the latter it's clearer to say "trusted configuration in IOMMU".

Yeah. And I realized there are more configuration except IOMMU, so I would say:

Add SEAMCALLs to setup/clear the IOMMU device and related I/O stack to work in trusted (TDX) mode.

Quoted 8 lines
Quoted 5 lines

Enable TEE I/O support for a target device requires to setup trusted IOMMU for the related IOMMU device first, even only for enabling physical secure links like SPDM/IDE.

this series is just about SPDM/IDE. then the first part about TEE I/O is not really relevant.

This is truely obscure. I want to clarify some potential concern about why we need to setup IOMMU when only to enable PCIe link encryption, my re-phase:

With the setup SEAMCALL, TDX Module ensures that related resources in the IOMMU device & I/O stack are in expected state and protected from further untrusted access, so that subsequent SPDM/IDE enabling is secure.

Quoted 10 lines
Quoted 5 lines

TDH.IOMMU.SETUP takes the register base address (VTBAR) to position an IOMMU device, and outputs an IOMMU_ID as the trusted IOMMU identifier. TDH.IOMMU.CLEAR takes the IOMMU_ID to reverse the setup.

Intel IOMMU is called VT-d. It has a register block but not a PCI device so there is no BAR resource related.

let's just call it 'reg_base'

Yes.

Quoted 3 lines

intel-iommu driver already has its own 'id' definition for each iommu device. It's clearer to add a prefix to this new id, e.g. tdx_iommu_id?

Yes.

author
#119 lore raw

PATCH v2 22/31 iommu/vt-d: Export a helper to do function for each dmar_drhd_unit

In reply to Xu Yilun

Enable the tdx-host module to get VTBAR address for every IOMMU device. The VTBAR address is for TDX Module to identify the IOMMU device and setup its trusted configuraion.

---
 include/linux/dmar.h       |  2 ++
 drivers/iommu/intel/dmar.c | 16 ++++++++++++++++
 2 files changed, 18 insertions(+)
include/linux/dmar.h +2 -0
index 692b2b445761..cd8d9f440975 100644
@@ -86,6 +86,8 @@ extern struct list_head dmar_drhd_units;
 				dmar_rcu_check())			\
 		if (i=drhd->iommu, 0) {} else 
 
+int do_for_each_drhd_unit(int (*fn)(struct dmar_drhd_unit *));
+
 static inline bool dmar_rcu_check(void)
 {
 	return rwsem_is_locked(&dmar_global_lock) ||
drivers/iommu/intel/dmar.c +16 -0
index 4f9571eee1d4..eea9ba691f99 100644
@@ -2452,3 +2452,19 @@ bool dmar_platform_optin(void)
 	return ret;
 }
 EXPORT_SYMBOL_GPL(dmar_platform_optin);
+
+int do_for_each_drhd_unit(int (*fn)(struct dmar_drhd_unit *))
+{
+	struct dmar_drhd_unit *drhd;
+	int ret;
+
+	guard(rwsem_read)(&dmar_global_lock);
+
+	for_each_drhd_unit(drhd) {
+		ret = fn(drhd);
+		if (ret)
+			return ret;
+	}
+	return 0;
+}
+EXPORT_SYMBOL_GPL(do_for_each_drhd_unit);
Signature · 1 line
2.25.1
#120 lore raw

RE: [PATCH v2 22/31] iommu/vt-d: Export a helper to do function for each dmar_drhd_unit

include/linux/dmar.h extern struct list_head dmar_drhd_units; · line 91

Quoted 10 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

include/linux/dmar.h +2 -0
@@ -86,6 +86,8 @@ extern struct list_head dmar_drhd_units;
 				dmar_rcu_check())			\
 		if (i=drhd->iommu, 0) {} else
 
+int do_for_each_drhd_unit(int (*fn)(struct dmar_drhd_unit *));
+
 static inline bool dmar_rcu_check(void)

It's a bit weird to insert it here. Move it to follow for_each_iommu().

Quoted 5 lines
drivers/iommu/intel/dmar.c +5 -0
+
+int do_for_each_drhd_unit(int (*fn)(struct dmar_drhd_unit *))
+{
+	struct dmar_drhd_unit *drhd;
+	int ret;

drivers/iommu/intel/dmar.c bool dmar_platform_optin(void) · line 2469

Quoted 10 lines
drivers/iommu/intel/dmar.c +10 -0
+
+	guard(rwsem_read)(&dmar_global_lock);
+
+	for_each_drhd_unit(drhd) {
+		ret = fn(drhd);
+		if (ret)
+			return ret;
+	}
+	return 0;
+}

use for_each_active_drhd_unit(). or is there need to setup the trusted configuration even on ignored iommu?

author
#121 lore raw

Re: [PATCH v2 22/31] iommu/vt-d: Export a helper to do function for each dmar_drhd_unit

Quoted 7 lines · Kevin
Quoted 7 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

include/linux/dmar.h +0 -0
@@ -86,6 +86,8 @@ extern struct list_head dmar_drhd_units;
 				dmar_rcu_check())			\
 		if (i=drhd->iommu, 0) {} else
 
Quoted 5 lines

include/linux/dmar.h extern struct list_head dmar_drhd_units; · line 91

Quoted 3 lines
include/linux/dmar.h +2 -0
+int do_for_each_drhd_unit(int (*fn)(struct dmar_drhd_unit *));
+
 static inline bool dmar_rcu_check(void)

It's a bit weird to insert it here. Move it to follow for_each_iommu().

Sorry, it is following for_each_iommu(), is it?

Quoted 11 lines
Quoted 11 lines
drivers/iommu/intel/dmar.c +10 -0
+
+int do_for_each_drhd_unit(int (*fn)(struct dmar_drhd_unit *))
+{
+	struct dmar_drhd_unit *drhd;
+	int ret;
+
+	guard(rwsem_read)(&dmar_global_lock);
+
+	for_each_drhd_unit(drhd) {
+		ret = fn(drhd);
Quoted 8 lines

drivers/iommu/intel/dmar.c bool dmar_platform_optin(void) · line 2469

Quoted 5 lines
drivers/iommu/intel/dmar.c +5 -0
+		if (ret)
+			return ret;
+	}
+	return 0;
+}

use for_each_active_drhd_unit(). or is there need to setup the trusted configuration even on ignored iommu?

No, for_each_active_drhd_unit() is good.

#122 lore raw

RE: [PATCH v2 22/31] iommu/vt-d: Export a helper to do function for each dmar_drhd_unit

Quoted 11 lines

From: Xu Yilun <[email protected]>
Sent: Wednesday, April 22, 2026 2:34 PM

Quoted 7 lines · Kevin
Quoted 7 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

include/linux/dmar.h +0 -0
@@ -86,6 +86,8 @@ extern struct list_head dmar_drhd_units;
 				dmar_rcu_check())			\
 		if (i=drhd->iommu, 0) {} else
 
Quoted 8 lines
Quoted 5 lines

include/linux/dmar.h extern struct list_head dmar_drhd_units; · line 91

Quoted 3 lines
include/linux/dmar.h +2 -0
+int do_for_each_drhd_unit(int (*fn)(struct dmar_drhd_unit *));
+
 static inline bool dmar_rcu_check(void)

It's a bit weird to insert it here. Move it to follow for_each_iommu().

Sorry, it is following for_each_iommu(), is it?

yeah, I misread it.

#123 lore raw

Re: [PATCH v2 22/31] iommu/vt-d: Export a helper to do function for each dmar_drhd_unit

In reply to Xu Yilun
Quoted 4 lines · Xu Yilun

Enable the tdx-host module to get VTBAR address for every IOMMU device. The VTBAR address is for TDX Module to identify the IOMMU device and setup its trusted configuraion.

I think this commit message doesn't match the commit subject or the change, unless I'm misunderstanding something

Quoted 46 lines
---
 include/linux/dmar.h       |  2 ++
 drivers/iommu/intel/dmar.c | 16 ++++++++++++++++
 2 files changed, 18 insertions(+)
include/linux/dmar.h +2 -0
index 692b2b445761..cd8d9f440975 100644
@@ -86,6 +86,8 @@ extern struct list_head dmar_drhd_units;
 				dmar_rcu_check())			\
 		if (i=drhd->iommu, 0) {} else
 
+int do_for_each_drhd_unit(int (*fn)(struct dmar_drhd_unit *));
+
 static inline bool dmar_rcu_check(void)
 {
 	return rwsem_is_locked(&dmar_global_lock) ||
drivers/iommu/intel/dmar.c +16 -1
index 4f9571eee1d4..eea9ba691f99 100644
@@ -2452,3 +2452,19 @@ bool dmar_platform_optin(void)
 	return ret;
 }
 EXPORT_SYMBOL_GPL(dmar_platform_optin);
+
+int do_for_each_drhd_unit(int (*fn)(struct dmar_drhd_unit *))
+{
+	struct dmar_drhd_unit *drhd;
+	int ret;
+
+	guard(rwsem_read)(&dmar_global_lock);
+
+	for_each_drhd_unit(drhd) {
+		ret = fn(drhd);
+		if (ret)
+			return ret;
+	}
+	return 0;
+}
+EXPORT_SYMBOL_GPL(do_for_each_drhd_unit);
--

2.25.1

author
#124 lore raw

PATCH v2 23/31 coco/tdx-host: Setup all trusted IOMMUs on TDX Connect init

In reply to Xu Yilun

Setup all trusted IOMMUs on TDX Connect initialization and clear all on TDX Connect removal.

Trusted IOMMU setup is the pre-condition for all following TDX Connect operations such as SPDM/IDE setup. It is more of a platform configuration than a standalone IOMMU configuration, so put the implementation in tdx-host driver.

There is no dedicated way to enumerate which IOMMU devices support trusted operations. The host has to call TDH.IOMMU.SETUP on all IOMMU devices and tell their trusted capability by the return value.

---
 drivers/virt/coco/tdx-host/Kconfig    |  1 +
 drivers/virt/coco/tdx-host/tdx-host.c | 85 +++++++++++++++++++++++++++
 2 files changed, 86 insertions(+)
drivers/virt/coco/tdx-host/Kconfig +1 -0
index 32add81b7d56..24e872f8953e 100644
@@ -13,3 +13,4 @@ config TDX_CONNECT
 	def_bool y
 	depends on TDX_HOST_SERVICES
 	depends on PCI_TSM
+	depends on INTEL_IOMMU
drivers/virt/coco/tdx-host/tdx-host.c +85 -0

103 lines not on this page · raw message

Signature · 1 line
2.25.1
#125 lore raw

RE: [PATCH v2 23/31] coco/tdx-host: Setup all trusted IOMMUs on TDX Connect init

Quoted 11 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

Setup all trusted IOMMUs on TDX Connect initialization and clear all on TDX Connect removal.

Trusted IOMMU setup is the pre-condition for all following TDX Connect operations such as SPDM/IDE setup. It is more of a platform configuration than a standalone IOMMU configuration, so put the implementation in tdx-host driver.

not sure what above tries to tell. why is it a platform configuration when you have seamcalls on each IOMMU?

author
#126 lore raw

Re: [PATCH v2 23/31] coco/tdx-host: Setup all trusted IOMMUs on TDX Connect init

Quoted 6 lines · Kevin
Quoted 6 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

Setup all trusted IOMMUs on TDX Connect initialization and clear all on TDX Connect removal.

Quoted 8 lines
Quoted 5 lines

Trusted IOMMU setup is the pre-condition for all following TDX Connect operations such as SPDM/IDE setup. It is more of a platform configuration than a standalone IOMMU configuration, so put the implementation in tdx-host driver.

not sure what above tries to tell. why is it a platform configuration when you have seamcalls on each IOMMU?

This is to say the TDH.IOMMU.SETUP relates to PCIe SPDM/IDE, it is not just about IOMMU. By identifying the

for_each_iommu(iommu)
tdh.iommu.setup(iommu)

as a platform configuration, it justifies why we trigger this configuration at tdx-host driver probe, rather than in some IOMMU/IOMMUFD API.

#127 lore raw

RE: [PATCH v2 23/31] coco/tdx-host: Setup all trusted IOMMUs on TDX Connect init

Quoted 22 lines

From: Xu Yilun <[email protected]>
Sent: Wednesday, April 22, 2026 5:27 PM

Quoted 6 lines · Kevin
Quoted 6 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

Setup all trusted IOMMUs on TDX Connect initialization and clear all on TDX Connect removal.

Quoted 8 lines
Quoted 5 lines

Trusted IOMMU setup is the pre-condition for all following TDX Connect operations such as SPDM/IDE setup. It is more of a platform configuration than a standalone IOMMU configuration, so put the implementation in tdx-host driver.

not sure what above tries to tell. why is it a platform configuration when you have seamcalls on each IOMMU?

This is to say the TDH.IOMMU.SETUP relates to PCIe SPDM/IDE, it is not just about IOMMU. By identifying the

Quoted 6 lines

for_each_iommu(iommu)
tdh.iommu.setup(iommu)

as a platform configuration, it justifies why we trigger this configuration at tdx-host driver probe, rather than in some IOMMU/IOMMUFD API.

iommu drivers also involve PCI, e.g. call pci_enable_ats(), etc.

so having relation to PCIe SPDM/IDE is not an argument of platform vs. IOMMU.

Actually I'm OK to put that logic in tdx-host. Just the explanation here doesn't make much sense...

author
#128 lore raw

Re: [PATCH v2 23/31] coco/tdx-host: Setup all trusted IOMMUs on TDX Connect init

Quoted 25 lines · Kevin
Quoted 25 lines

From: Xu Yilun <[email protected]>
Sent: Wednesday, April 22, 2026 5:27 PM

Quoted 6 lines · Kevin
Quoted 6 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

Setup all trusted IOMMUs on TDX Connect initialization and clear all on TDX Connect removal.

Quoted 8 lines
Quoted 5 lines

Trusted IOMMU setup is the pre-condition for all following TDX Connect operations such as SPDM/IDE setup. It is more of a platform configuration than a standalone IOMMU configuration, so put the implementation in tdx-host driver.

not sure what above tries to tell. why is it a platform configuration when you have seamcalls on each IOMMU?

This is to say the TDH.IOMMU.SETUP relates to PCIe SPDM/IDE, it is not just about IOMMU. By identifying the

for_each_iommu(iommu)
tdh.iommu.setup(iommu)

Quoted 8 lines
Quoted 3 lines

as a platform configuration, it justifies why we trigger this configuration at tdx-host driver probe, rather than in some IOMMU/IOMMUFD API.

iommu drivers also involve PCI, e.g. call pci_enable_ats(), etc.

so having relation to PCIe SPDM/IDE is not an argument of platform vs. IOMMU.

OK, I think I could delete the platform vs. IOMMU thing in commit log.

Quoted 4 lines

Actually I'm OK to put that logic in tdx-host. Just the explanation here doesn't make much sense...

#129 lore raw

Re: [PATCH v2 23/31] coco/tdx-host: Setup all trusted IOMMUs on TDX Connect init

In reply to Xu Yilun
Quoted 8 lines · Xu Yilun

Setup all trusted IOMMUs on TDX Connect initialization and clear all on TDX Connect removal.

Trusted IOMMU setup is the pre-condition for all following TDX Connect operations such as SPDM/IDE setup. It is more of a platform configuration than a standalone IOMMU configuration, so put the implementation in tdx-host driver.

I was wondering about hotplug. There's dmar_device_hotplug(). Does it make sense to do TDH.IOMMU.SETUP from there?

For boot, is the platform-level configuration due to dependencies? Is it because TDX is set up after IOMMUs and hence TDH.IOMMU.SETUP is done from the creation of the faux device?

Quoted 39 lines

There is no dedicated way to enumerate which IOMMU devices support trusted operations. The host has to call TDH.IOMMU.SETUP on all IOMMU devices and tell their trusted capability by the return value.

[...snip...]

drivers/virt/coco/tdx-host/tdx-host.c +28 -0
@@ -119,6 +120,82 @@ static void unregister_link_tsm(void *link)
 	tsm_unregister(link);
 }
 
+static DEFINE_XARRAY(tlink_iommu_xa);
+
+static void tdx_iommu_clear(u64 iommu_id, struct tdx_page_array *iommu_mt)
+{
+	u64 r;
+
+	r = tdh_iommu_clear(iommu_id, iommu_mt);
+	if (r) {
+		pr_err("fail to clear tdx iommu 0x%llx\n", r);
+		goto leak;
+	}
+
+	if (tdx_page_array_ctrl_release(iommu_mt, iommu_mt->nr_pages,
+					virt_to_phys(iommu_mt->root))) {
+		pr_err("fail to release iommu_mt pages\n");
+		goto leak;
+	}
+
+	return;
+
+leak:
+	tdx_page_array_ctrl_leak(iommu_mt);
+}
+
+static int tdx_iommu_enable_one(struct dmar_drhd_unit *drhd)
+{
+	unsigned int nr_pages = tdx_sysinfo->connect.iommu_mt_page_count;
+	u64 r, iommu_id;

drivers/virt/coco/tdx-host/tdx-host.c static void unregister_link_tsm(void *link) · line 160

Quoted 10 lines
drivers/virt/coco/tdx-host/tdx-host.c +10 -0
+	int ret;
+
+	struct tdx_page_array *iommu_mt __free(tdx_page_array_free) =
+		tdx_page_array_create_iommu_mt(1, nr_pages);
+	if (!iommu_mt)
+		return -ENOMEM;
+
+	r = tdh_iommu_setup(drhd->reg_base_addr, iommu_mt, &iommu_id);
+	/* This drhd doesn't support tdx mode, skip. */
+	if ((r & TDX_SEAMCALL_STATUS_MASK)  == TDX_OPERAND_INVALID)

Extra space here before ==.

Is there a chance that TDX_OPERAND_INVALID might be returned for some other reason other than that the drhd doesn't support TDX? I'm concerned that some drhd would be silently skipped for some other error.

Quoted 17 lines
drivers/virt/coco/tdx-host/tdx-host.c +17 -0
+		return 0;
+
+	if (r) {
+		pr_err("fail to enable tdx mode for DRHD[0x%llx]\n",
+		       drhd->reg_base_addr);
+		return -EFAULT;
+	}
+
+	ret = xa_insert(&tlink_iommu_xa, (unsigned long)iommu_id,
+			no_free_ptr(iommu_mt), GFP_KERNEL);
+	if (ret) {
+		tdx_iommu_clear(iommu_id, iommu_mt);
+		return ret;
+	}
+
+	return 0;
+}

drivers/virt/coco/tdx-host/tdx-host.c static void unregister_link_tsm(void *link) · line 187

Quoted 10 lines
drivers/virt/coco/tdx-host/tdx-host.c +10 -0
+
+static void tdx_iommu_disable_all(void *data)
+{
+	struct tdx_page_array *iommu_mt;
+	unsigned long iommu_id;
+
+	xa_for_each(&tlink_iommu_xa, iommu_id, iommu_mt)
+		tdx_iommu_clear(iommu_id, iommu_mt);
+}
+

Is there an equivalent teardown operation? Like on kexec, if TDX is torn down, would tdx_iommu_disable_all() need to be called to undo everything, so that after TDX is started again tdx_iommu_enable_all() would be set up again?

For hot-unplug, would the TDX IOMMU config need to be disabled?

Quoted 31 lines
drivers/virt/coco/tdx-host/tdx-host.c +19 -1
+static int tdx_iommu_enable_all(void)
+{
+	int ret;
+
+	ret = do_for_each_drhd_unit(tdx_iommu_enable_one);
+	if (ret)
+		tdx_iommu_disable_all(NULL);
+
+	return ret;
+}
+
 static int __maybe_unused tdx_connect_init(struct device *dev)
 {
 	struct tsm_dev *link;
@@ -130,6 +207,14 @@ static int __maybe_unused tdx_connect_init(struct device *dev)
 	if (!(tdx_sysinfo->features.tdx_features0 & TDX_FEATURES0_TDXCONNECT))
 		return 0;
 
+	ret = tdx_iommu_enable_all();
+	if (ret)
+		return dev_err_probe(dev, ret, "Enable tdx iommu failed\n");
+
+	ret = devm_add_action_or_reset(dev, tdx_iommu_disable_all, NULL);
+	if (ret)
+		return ret;
+
 	link = tsm_register(dev, &tdx_tsm_link_ops);
 	if (IS_ERR(link))
 		return dev_err_probe(dev, PTR_ERR(link),
--

2.25.1

author
#130 lore raw

PATCH v2 24/31 coco/tdx-host: Add a helper to exchange SPDM messages through DOE

In reply to Xu Yilun

From: Zhenzhong Duan <[email protected]>

TDX host uses this function to exchange TDX Module encrypted data with devices via SPDM. It is unfortunate that TDX passes raw DOE frames with headers included and the PCI DOE core wants payloads separated from headers.

This conversion code is about the same amount of work as teaching the PCI DOE driver to support raw frames. Unless and until another raw frame use case shows up, just do this conversion in the TDX TSM driver.

---
 drivers/virt/coco/tdx-host/tdx-host.c | 61 +++++++++++++++++++++++++++
 1 file changed, 61 insertions(+)
drivers/virt/coco/tdx-host/tdx-host.c +61 -0
index 98ed93ac0153..06f3d194e0a8 100644
@@ -5,11 +5,13 @@
  * Copyright (C) 2025 Intel Corporation
  */
 
+#include <linux/bitfield.h>
 #include <linux/device/faux.h>
 #include <linux/dmar.h>
 #include <linux/module.h>
 #include <linux/mod_devicetable.h>
 #include <linux/pci.h>
+#include <linux/pci-doe.h>
 #include <linux/pci-tsm.h>
 #include <linux/tsm.h>
 
@@ -39,6 +41,65 @@ static struct tdx_tsm_link *to_tdx_tsm_link(struct pci_tsm *tsm)
 	return container_of(tsm, struct tdx_tsm_link, pci.base_tsm);
 }
 
+#define PCI_DOE_DATA_OBJECT_HEADER_1_OFFSET	0
+#define PCI_DOE_DATA_OBJECT_HEADER_2_OFFSET	4
+#define PCI_DOE_DATA_OBJECT_HEADER_SIZE		8
+#define PCI_DOE_DATA_OBJECT_PAYLOAD_OFFSET	PCI_DOE_DATA_OBJECT_HEADER_SIZE
+
+#define PCI_DOE_PROTOCOL_SECURE_SPDM		2
+
+static int __maybe_unused tdx_spdm_msg_exchange(struct tdx_tsm_link *tlink,
+						void *request, size_t request_sz,
+						void *response, size_t response_sz)
+{
+	struct pci_dev *pdev = tlink->pci.base_tsm.pdev;
+	void *req_pl_addr, *resp_pl_addr;
+	size_t req_pl_sz, resp_pl_sz;
+	u32 data, len;
+	u16 vendor;
+	u8 type;
+	int ret;
+
+	/*
+	 * pci_doe() accept DOE PAYLOAD only but request carries DOE HEADER so
+	 * shift the buffers, skip DOE HEADER in request buffer, and fill DOE
+	 * HEADER in response buffer manually.
+	 */
+
+	data = le32_to_cpu(*(__le32 *)(request + PCI_DOE_DATA_OBJECT_HEADER_1_OFFSET));
+	vendor = FIELD_GET(PCI_DOE_DATA_OBJECT_HEADER_1_VID, data);
+	type = FIELD_GET(PCI_DOE_DATA_OBJECT_HEADER_1_TYPE, data);
+
+	data = le32_to_cpu(*(__le32 *)(request + PCI_DOE_DATA_OBJECT_HEADER_2_OFFSET));
+	len = FIELD_GET(PCI_DOE_DATA_OBJECT_HEADER_2_LENGTH, data);
+
+	req_pl_sz = len * sizeof(__le32) - PCI_DOE_DATA_OBJECT_HEADER_SIZE;
+	resp_pl_sz = response_sz - PCI_DOE_DATA_OBJECT_HEADER_SIZE;
+	req_pl_addr = request + PCI_DOE_DATA_OBJECT_HEADER_SIZE;
+	resp_pl_addr = response + PCI_DOE_DATA_OBJECT_HEADER_SIZE;
+
+	ret = pci_tsm_doe_transfer(pdev, type, req_pl_addr, req_pl_sz,
+				   resp_pl_addr, resp_pl_sz);
+	if (ret < 0) {
+		pci_err(pdev, "spdm msg exchange fail %d\n", ret);
+		return ret;
+	}
+
+	data = FIELD_PREP(PCI_DOE_DATA_OBJECT_HEADER_1_VID, vendor) |
+	       FIELD_PREP(PCI_DOE_DATA_OBJECT_HEADER_1_TYPE, type);
+	*(__le32 *)(response + PCI_DOE_DATA_OBJECT_HEADER_1_OFFSET) = cpu_to_le32(data);
+
+	len = (ret + PCI_DOE_DATA_OBJECT_HEADER_SIZE) / sizeof(__le32);
+	data = FIELD_PREP(PCI_DOE_DATA_OBJECT_HEADER_2_LENGTH, len);
+	*(__le32 *)(response + PCI_DOE_DATA_OBJECT_HEADER_2_OFFSET) = cpu_to_le32(data);
+
+	ret += PCI_DOE_DATA_OBJECT_HEADER_SIZE;
+
+	pci_dbg(pdev, "%s complete: vendor 0x%x type 0x%x rsp_sz %d\n",
+		__func__, vendor, type, ret);
+	return ret;
+}
+
 static int tdx_tsm_link_connect(struct pci_dev *pdev)
 {
 	return -ENXIO;
Signature · 1 line
2.25.1
#131 lore raw

RE: [PATCH v2 24/31] coco/tdx-host: Add a helper to exchange SPDM messages through DOE

drivers/virt/coco/tdx-host/tdx-host.c static struct tdx_tsm_link *to_tdx_tsm_link(struct pci_tsm *tsm) · line 55

Quoted 11 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

(quoted diff) +2 -0
+
+static int __maybe_unused tdx_spdm_msg_exchange(struct tdx_tsm_link

*tlink,

(quoted diff) +1 -0
+						void *request, size_t

request_sz,

(quoted diff) +1 -0
+						void *response, size_t

response_sz)

drivers/virt/coco/tdx-host/tdx-host.c +2 -0
+{
+	struct pci_dev *pdev = tlink->pci.base_tsm.pdev;

call it pci_spdm_msg_exchange() and pass in struct pci_dev directly.

there is no other use of tlink in this function. could add a note that this should be moved to pci core when a 2nd user of raw frame comes.

author
#132 lore raw

Re: [PATCH v2 24/31] coco/tdx-host: Add a helper to exchange SPDM messages through DOE

Quoted 5 lines · Kevin
Quoted 5 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

(quoted diff) +2 -0
+
+static int __maybe_unused tdx_spdm_msg_exchange(struct tdx_tsm_link

*tlink,

Quoted 8 lines

drivers/virt/coco/tdx-host/tdx-host.c static struct tdx_tsm_link *to_tdx_tsm_link(struct pci_tsm *tsm) · line 55

Quoted 6 lines
(quoted diff) +1 -0
+						void *request, size_t

request_sz,

(quoted diff) +1 -0
+						void *response, size_t

response_sz)

drivers/virt/coco/tdx-host/tdx-host.c +2 -0
+{
+	struct pci_dev *pdev = tlink->pci.base_tsm.pdev;

call it pci_spdm_msg_exchange() and pass in struct pci_dev directly.

I don't think so. There is kernel managed spdm transfer support WIP, which is another topic. We don't want to mix the namespace with that one.

And we also don't name it tsm_spdm_msg_exchange, cause TSM firmwares output different blobs for vendor TSM drivers to transfer. E.g. TDX Module outputs buffers with DOE header & SPDM header, other vendors (AMD IIRC) outputs buffers with only SPDM header. So this function is TDX specific.

Quoted 3 lines

there is no other use of tlink in this function. could add a note that this should be moved to pci core when a 2nd user of raw frame comes.

#133 lore raw

RE: [PATCH v2 24/31] coco/tdx-host: Add a helper to exchange SPDM messages through DOE

Quoted 19 lines

From: Xu Yilun <[email protected]>
Sent: Wednesday, April 22, 2026 5:41 PM

Quoted 4 lines · Kevin

Review

Quoted 4 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

(quoted diff) +2 -0
+
+static int __maybe_unused tdx_spdm_msg_exchange(struct

tdx_tsm_link

Quoted 9 lines

drivers/virt/coco/tdx-host/tdx-host.c static struct tdx_tsm_link *to_tdx_tsm_link(struct pci_tsm *tsm) · line 55

Quoted 7 lines

*tlink,

(quoted diff) +1 -0
+						void *request, size_t

request_sz,

(quoted diff) +1 -0
+						void *response, size_t

response_sz)

drivers/virt/coco/tdx-host/tdx-host.c +2 -0
+{
+	struct pci_dev *pdev = tlink->pci.base_tsm.pdev;

call it pci_spdm_msg_exchange() and pass in struct pci_dev directly.

Quoted 3 lines

I don't think so. There is kernel managed spdm transfer support WIP, which is another topic. We don't want to mix the namespace with that one.

pci_spdm_raw_msg_exchange() then, since you said currently only one user i.e. tdx?

If the kernel managed spdm doesn't support the raw format, then there won't be conflict.

if it supports (i.e. the 2nd user), then this should be moved to pci core.

Quoted 7 lines

And we also don't name it tsm_spdm_msg_exchange, cause TSM firmwares output different blobs for vendor TSM drivers to transfer. E.g. TDX Module outputs buffers with DOE header & SPDM header, other vendors (AMD IIRC) outputs buffers with only SPDM header. So this function is TDX specific.

TDX is just an user of that. All the logic here is about handling the raw format, nothing specific to tdx.

author
#134 lore raw

Re: [PATCH v2 24/31] coco/tdx-host: Add a helper to exchange SPDM messages through DOE

Quoted 23 lines · Kevin
Quoted 19 lines

From: Xu Yilun <[email protected]>
Sent: Wednesday, April 22, 2026 5:41 PM

Quoted 4 lines · Kevin

Review

Quoted 4 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

(quoted diff) +2 -0
+
+static int __maybe_unused tdx_spdm_msg_exchange(struct

tdx_tsm_link

Quoted 9 lines

drivers/virt/coco/tdx-host/tdx-host.c static struct tdx_tsm_link *to_tdx_tsm_link(struct pci_tsm *tsm) · line 55

Quoted 7 lines

*tlink,

(quoted diff) +1 -0
+						void *request, size_t

request_sz,

(quoted diff) +1 -0
+						void *response, size_t

response_sz)

drivers/virt/coco/tdx-host/tdx-host.c +2 -0
+{
+	struct pci_dev *pdev = tlink->pci.base_tsm.pdev;

call it pci_spdm_msg_exchange() and pass in struct pci_dev directly.

Quoted 3 lines

I don't think so. There is kernel managed spdm transfer support WIP, which is another topic. We don't want to mix the namespace with that one.

Quoted 5 lines

pci_spdm_raw_msg_exchange() then, since you said currently only one user i.e. tdx?

If the kernel managed spdm doesn't support the raw format, then there won't be conflict.

I think kernel managed spdm APIs should not support the raw format (with DOE header). SPDM protocol could be run on various transit so make no sense to have special care for DOE.

So this "raw_msg_exchange" is dedicated for PCI/TSM. Yes there won't be conflict, but it's important "pci_spdm_" prefix only serve kernel managed spdm. PCI_TSM managed SPDM has fundamental logical differences with the kernel managed one.

Quoted 11 lines

if it supports (i.e. the 2nd user), then this should be moved to pci core.

Quoted 7 lines

And we also don't name it tsm_spdm_msg_exchange, cause TSM firmwares output different blobs for vendor TSM drivers to transfer. E.g. TDX Module outputs buffers with DOE header & SPDM header, other vendors (AMD IIRC) outputs buffers with only SPDM header. So this function is TDX specific.

Quoted 2 lines

TDX is just an user of that. All the logic here is about handling the raw format, nothing specific to tdx.

It is special, kernel managed spdm won't expect an all-in-one format, other vendors either. It is TDX's decision to output this all-in-one format for OS to transfer.

author
#135 lore raw

PATCH v2 25/31 x86/virt/tdx: Add SEAMCALL wrappers for SPDM management

In reply to Xu Yilun

From: Zhenzhong Duan <[email protected]>

Add several SEAMCALL wrappers for SPDM management. TDX Module requires HPA_ARRAY_T structure as input/output parameters for these SEAMCALLs. So use tdx_page_array for these wrappers.

- TDH.SPDM.CREATE creates SPDM session metadata buffers for TDX Module.
- TDH.SPDM.DELETE destroys SPDM session metadata and returns these buffers to host, after checking no reference attached to the metadata.
- TDH.SPDM.CONNECT establishes a new SPDM session with the device.
- TDH.SPDM.DISCONNECT tears down the SPDM session with the device.
- TDH.SPDM.MNG supports three SPDM runtime operations: HEARTBEAT, KEY_UPDATE and DEV_INFO_RECOLLECTION.

---
 arch/x86/include/asm/tdx.h  |  13 ++++
 arch/x86/virt/vmx/tdx/tdx.h |   5 ++
 arch/x86/virt/vmx/tdx/tdx.c | 114 +++++++++++++++++++++++++++++++++++-
 3 files changed, 130 insertions(+), 2 deletions(-)
arch/x86/include/asm/tdx.h +13 -0
index a59e0e43e465..8abdad084972 100644
@@ -247,6 +247,19 @@ u64 tdh_phymem_page_wbinvd_tdr(struct tdx_td *td);
 u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct page *page);
 u64 tdh_iommu_setup(u64 vtbar, struct tdx_page_array *iommu_mt, u64 *iommu_id);
 u64 tdh_iommu_clear(u64 iommu_id, struct tdx_page_array *iommu_mt);
+u64 tdh_spdm_create(u64 func_id, struct tdx_page_array *spdm_mt, u64 *spdm_id);
+u64 tdh_spdm_delete(u64 spdm_id, struct tdx_page_array *spdm_mt,
+		    unsigned int *nr_released, u64 *released_hpa);
+u64 tdh_exec_spdm_connect(u64 spdm_id, struct page *spdm_conf,
+			  struct page *spdm_rsp, struct page *spdm_req,
+			  struct tdx_page_array *spdm_out,
+			  u64 *spdm_req_or_out_len);
+u64 tdh_exec_spdm_disconnect(u64 spdm_id, struct page *spdm_rsp,
+			     struct page *spdm_req, u64 *spdm_req_len);
+u64 tdh_exec_spdm_mng(u64 spdm_id, u64 spdm_op, struct page *spdm_param,
+		      struct page *spdm_rsp, struct page *spdm_req,
+		      struct tdx_page_array *spdm_out,
+		      u64 *spdm_req_or_out_len);
 #else
 static inline void tdx_init(void) { }
 static inline int tdx_cpu_enable(void) { return -ENODEV; }
arch/x86/virt/vmx/tdx/tdx.h +5 -0
index b25c418f6e61..4784db2d1d92 100644
@@ -64,6 +64,11 @@
 #define TDH_EXT_MEM_ADD			61
 #define TDH_IOMMU_SETUP			128
 #define TDH_IOMMU_CLEAR			129
+#define TDH_SPDM_CREATE			130
+#define TDH_SPDM_DELETE			131
+#define TDH_SPDM_CONNECT		142
+#define TDH_SPDM_DISCONNECT		143
+#define TDH_SPDM_MNG			144
 
 /* TDX page types */
 #define	PT_NDA		0x0
arch/x86/virt/vmx/tdx/tdx.c +112 -2

135 lines not on this page · raw message

Signature · 1 line
2.25.1
#136 lore raw

RE: [PATCH v2 25/31] x86/virt/tdx: Add SEAMCALL wrappers for SPDM management

Quoted 3 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

here ...

Quoted 2 lines

- TDH.SPDM.MNG supports three SPDM runtime operations: HEARTBEAT, KEY_UPDATE and DEV_INFO_RECOLLECTION.

... but the actual helper just pass whatever ops to TDX module

Quoted 23 lines

+u64 tdh_exec_spdm_mng(u64 spdm_id, u64 spdm_op, struct page *spdm_param,

arch/x86/virt/vmx/tdx/tdx.c +21 -0
+		      struct page *spdm_rsp, struct page *spdm_req,
+		      struct tdx_page_array *spdm_out,
+		      u64 *spdm_req_or_out_len)
+{
+	struct tdx_module_args args = {
+		.rcx = spdm_id,
+		.rdx = spdm_op,
+		.r8 = spdm_param ? page_to_phys(spdm_param) : -1,
+		.r9 = page_to_phys(spdm_rsp),
+		.r10 = page_to_phys(spdm_req),
+		.r11 = spdm_out ? hpa_array_t_assign_raw(spdm_out) : -1,
+	};
+	u64 r;
+
+	r = seamcall_ret_ir_exec(TDH_SPDM_MNG, &args);
+
+	*spdm_req_or_out_len = args.rcx;
+
+	return r;
+}
+EXPORT_SYMBOL_FOR_MODULES(tdh_exec_spdm_mng, "tdx-host");
author
#137 lore raw

Re: [PATCH v2 25/31] x86/virt/tdx: Add SEAMCALL wrappers for SPDM management

Quoted 10 lines · Kevin
Quoted 3 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:01 AM

here ...

Quoted 2 lines

- TDH.SPDM.MNG supports three SPDM runtime operations: HEARTBEAT, KEY_UPDATE and DEV_INFO_RECOLLECTION.

... but the actual helper just pass whatever ops to TDX module

Ah, yes. But only tdx-host driver which implements link encryption cares about these operation code. My preference is only define general the SEAMCALL wrapper format in TDX core as other SEAMCALL do. And leave the specific op code definition in tdx-host driver.

I can revise the commit log as:

- TDH.SPDM.MNG supports various SPDM runtime operations: HEARTBEAT, KEY_UPDATE, DEV_INFO_RECOLLECTION... These operation codes are defined in tdx-host driver.

Thanks.

Quoted 24 lines
Quoted 24 lines

+u64 tdh_exec_spdm_mng(u64 spdm_id, u64 spdm_op, struct page *spdm_param,

arch/x86/virt/vmx/tdx/tdx.c +21 -0
+		      struct page *spdm_rsp, struct page *spdm_req,
+		      struct tdx_page_array *spdm_out,
+		      u64 *spdm_req_or_out_len)
+{
+	struct tdx_module_args args = {
+		.rcx = spdm_id,
+		.rdx = spdm_op,
+		.r8 = spdm_param ? page_to_phys(spdm_param) : -1,
+		.r9 = page_to_phys(spdm_rsp),
+		.r10 = page_to_phys(spdm_req),
+		.r11 = spdm_out ? hpa_array_t_assign_raw(spdm_out) : -1,
+	};
+	u64 r;
+
+	r = seamcall_ret_ir_exec(TDH_SPDM_MNG, &args);
+
+	*spdm_req_or_out_len = args.rcx;
+
+	return r;
+}
+EXPORT_SYMBOL_FOR_MODULES(tdh_exec_spdm_mng, "tdx-host");
author
#138 lore raw

PATCH v2 26/31 mm: Add __free() support for __free_page()

In reply to Xu Yilun

Allow for the declaration of struct page * variables that trigger __free_page() when they go out of scope.

A example usage would be in the following patch:

static struct pci_tsm *tdx_tsm_link_pf0_probe(...)

  {
	...

	struct page *in_msg_page __free(__free_page) =
		alloc_page(GFP_KERNEL | __GFP_ZERO);
	if (!in_msg_page)
		return NULL;

	struct page *out_msg_page __free(__free_page) =
		alloc_page(GFP_KERNEL | __GFP_ZERO);
	if (!out_msg_page)
		return NULL;

	...

	tlink->in_msg = no_free_ptr(in_msg_page);
	tlink->out_msg = no_free_ptr(out_msg_page);

	...

}

---
 include/linux/gfp.h | 1 +
 1 file changed, 1 insertion(+)
include/linux/gfp.h +1 -0
index 51ef13ed756e..d37e5564234e 100644
@@ -391,6 +391,7 @@ extern void free_pages_nolock(struct page *page, unsigned int order);
 extern void free_pages(unsigned long addr, unsigned int order);
 
 #define __free_page(page) __free_pages((page), 0)
+DEFINE_FREE(__free_page, struct page *, if (_T) __free_page(_T))
 #define free_page(addr) free_pages((addr), 0)
 
 void page_alloc_init_cpuhp(void);
Signature · 1 line
2.25.1
Reviewed-by on v2 26/31
#139 lore raw

Re: [PATCH v2 26/31] mm: Add __free() support for __free_page()

Quoted 46 lines · Xu Yilun

Allow for the declaration of struct page * variables that trigger __free_page() when they go out of scope.

A example usage would be in the following patch:

static struct pci_tsm *tdx_tsm_link_pf0_probe(...)

  {
	...

	struct page *in_msg_page __free(__free_page) =
		alloc_page(GFP_KERNEL | __GFP_ZERO);
	if (!in_msg_page)
		return NULL;

	struct page *out_msg_page __free(__free_page) =
		alloc_page(GFP_KERNEL | __GFP_ZERO);
	if (!out_msg_page)
		return NULL;

	...

	tlink->in_msg = no_free_ptr(in_msg_page);
	tlink->out_msg = no_free_ptr(out_msg_page);

	...

}

---
 include/linux/gfp.h | 1 +
 1 file changed, 1 insertion(+)
include/linux/gfp.h +1 -1
index 51ef13ed756e..d37e5564234e 100644
@@ -391,6 +391,7 @@ extern void free_pages_nolock(struct page *page, unsigned int order);
 extern void free_pages(unsigned long addr, unsigned int order);
 
 #define __free_page(page) __free_pages((page), 0)
+DEFINE_FREE(__free_page, struct page *, if (_T) __free_page(_T))
 #define free_page(addr) free_pages((addr), 0)
 
 void page_alloc_init_cpuhp(void);
--
Quoted 1 line

2.25.1

This is nice to have, independent of this series. Perhaps it should be sent separately, and some existing function could be refactored to use this, maybe __tdx_td_init?

author
#140 lore raw

PATCH v2 27/31 coco/tdx-host: Implement SPDM session setup

In reply to Xu Yilun

From: Zhenzhong Duan <[email protected]>

Implementation for a most straightforward SPDM session setup, using all default session options. Retrieve device info data from TDX Module which contains the SPDM negotiation results.

TDH.SPDM.CONNECT/DISCONNECT are TDX Module Extension introduced SEAMCALLs which can run for longer periods and interruptible. But there is resource constraints that limit how many SEAMCALLs of this kind can run simultaneously. The current situation is One SEAMCALL at a time. Otherwise TDX_OPERAND_BUSY is returned. To avoid "broken indefinite" retry, a tdx_ext_lock is used to guard these SEAMCALLs.

---
 arch/x86/include/asm/shared/tdx_errno.h |   2 +
 drivers/virt/coco/tdx-host/tdx-host.c   | 301 +++++++++++++++++++++++-
 2 files changed, 299 insertions(+), 4 deletions(-)
arch/x86/include/asm/shared/tdx_errno.h +2 -0
index 8bf6765cf082..7db04fe30378 100644
@@ -29,6 +29,8 @@
 #define TDX_EPT_WALK_FAILED			0xC0000B0000000000ULL
 #define TDX_EPT_ENTRY_STATE_INCORRECT		0xC0000B0D00000000ULL
 #define TDX_METADATA_FIELD_NOT_READABLE		0xC0000C0200000000ULL
+#define TDX_SPDM_SESSION_KEY_REQUIRE_REFRESH	0xC0000F4500000000ULL
+#define TDX_SPDM_REQUEST			0xC0000F5700000000ULL
 
 /*
  * SW-defined error codes.
drivers/virt/coco/tdx-host/tdx-host.c +297 -4

350 lines not on this page · raw message

Signature · 1 line
2.25.1
#141 lore raw

Re: [PATCH v2 27/31] coco/tdx-host: Implement SPDM session setup

Quoted 51 lines · Xu Yilun

From: Zhenzhong Duan <[email protected]>

Implementation for a most straightforward SPDM session setup, using all default session options. Retrieve device info data from TDX Module which contains the SPDM negotiation results.

TDH.SPDM.CONNECT/DISCONNECT are TDX Module Extension introduced SEAMCALLs which can run for longer periods and interruptible. But there is resource constraints that limit how many SEAMCALLs of this kind can run simultaneously. The current situation is One SEAMCALL at a time. Otherwise TDX_OPERAND_BUSY is returned. To avoid "broken indefinite" retry, a tdx_ext_lock is used to guard these SEAMCALLs.

---
  arch/x86/include/asm/shared/tdx_errno.h |   2 +
  drivers/virt/coco/tdx-host/tdx-host.c   | 301 +++++++++++++++++++++++-
  2 files changed, 299 insertions(+), 4 deletions(-)
arch/x86/include/asm/shared/tdx_errno.h +2 -0
index 8bf6765cf082..7db04fe30378 100644
@@ -29,6 +29,8 @@
  #define TDX_EPT_WALK_FAILED			0xC0000B0000000000ULL
  #define TDX_EPT_ENTRY_STATE_INCORRECT		0xC0000B0D00000000ULL
  #define TDX_METADATA_FIELD_NOT_READABLE		0xC0000C0200000000ULL
+#define TDX_SPDM_SESSION_KEY_REQUIRE_REFRESH	0xC0000F4500000000ULL
+#define TDX_SPDM_REQUEST			0xC0000F5700000000ULL
 
  /*
   * SW-defined error codes.
drivers/virt/coco/tdx-host/tdx-host.c +2 -0
index 06f3d194e0a8..4d127b7c2591 100644
@@ -14,6 +14,7 @@
  #include <linux/pci-doe.h>
  #include <linux/pci-tsm.h>
  #include <linux/tsm.h>
+#include <linux/vmalloc.h>
 
  #include <asm/cpu_device_id.h>
  #include <asm/tdx.h>
@@ -32,8 +33,43 @@ MODULE_DEVICE_TABLE(x86cpu, tdx_host_ids);
   */
  static const struct tdx_sys_info *tdx_sysinfo;
 
+#define TDISP_FUNC_ID		GENMASK(15, 0)

drivers/virt/coco/tdx-host/tdx-host.c MODULE_DEVICE_TABLE(x86cpu, tdx_host_ids); · line 46

Quoted 10 lines
drivers/virt/coco/tdx-host/tdx-host.c +10 -0
+#define TDISP_FUNC_ID_SEGMENT		GENMASK(23, 16)
+#define TDISP_FUNC_ID_SEG_VALID		BIT(24)
+
+static inline u32 tdisp_func_id(struct pci_dev *pdev)
+{
+	u32 func_id;
+
+	func_id = FIELD_PREP(TDISP_FUNC_ID_SEGMENT, pci_domain_nr(pdev->bus));
+	if (func_id)
+		func_id |= TDISP_FUNC_ID_SEG_VALID;

This check implies pci_domain_nr returning 0 is considered invalid. Other callers in the kernel seem to not care, they just use the domain nr, so is this check spurious or intentional ?

drivers/virt/coco/tdx-host/tdx-host.c MODULE_DEVICE_TABLE(x86cpu, tdx_host_ids); · line 56

Quoted 10 lines
drivers/virt/coco/tdx-host/tdx-host.c +10 -0
+	func_id |= FIELD_PREP(TDISP_FUNC_ID,
+			      PCI_DEVID(pdev->bus->number, pdev->devfn));
+
+	return func_id;
+}
+
+struct spdm_config_info_t {
+	u32 vmm_spdm_cap;
+#define SPDM_CAP_HBEAT          BIT(13)
+#define SPDM_CAP_KEY_UPD        BIT(14)

nit: move those defines above the struct definition, they just break the reading flow as it is.

Quoted 19 lines
drivers/virt/coco/tdx-host/tdx-host.c +14 -0
+	u8 spdm_session_policy;
+	u8 certificate_slot_mask;
+	u8 raw_bitstream_requested;
+} __packed;
+
  struct tdx_tsm_link {
  	struct pci_tsm_pf0 pci;
+	u32 func_id;
+	struct page *in_msg;
+	struct page *out_msg;
+
+	u64 spdm_id;
+	struct page *spdm_conf;
+	struct tdx_page_array *spdm_mt;
+	unsigned int dev_info_size;
+	void *dev_info_data;
  };
 
  static struct tdx_tsm_link *to_tdx_tsm_link(struct pci_tsm *tsm)

<snip>

drivers/virt/coco/tdx-host/tdx-host.c static int __maybe_unused tdx_spdm_msg_exchange(struct tdx_tsm_link *tlink, · line 211

Quoted 5 lines
drivers/virt/coco/tdx-host/tdx-host.c +5 -0
+
+static void *tdx_dup_array_data(struct tdx_page_array *array,
+				unsigned int data_size)
+{
+	unsigned int npages = (data_size + PAGE_SIZE - 1) / PAGE_SIZE;

nit: There's DIV_ROUND_UP

Quoted 15 lines
drivers/virt/coco/tdx-host/tdx-host.c +15 -0
+	void *data, *dup_data;
+
+	if (npages > array->nr_pages)
+		return NULL;
+
+	data = vm_map_ram(array->pages, npages, -1);
+	if (!data)
+		return NULL;
+
+	dup_data = kmemdup(data, data_size, GFP_KERNEL);
+	vm_unmap_ram(data, npages);
+
+	return dup_data;
+}
+

<snip>

drivers/virt/coco/tdx-host/tdx-host.c static int __maybe_unused tdx_spdm_msg_exchange(struct tdx_tsm_link *tlink, · line 357

Quoted 11 lines
drivers/virt/coco/tdx-host/tdx-host.c +8 -1
+
+DEFINE_FREE(tdx_spdm_session_teardown, struct tdx_tsm_link *,
+	    if (!IS_ERR_OR_NULL(_T)) tdx_spdm_session_teardown(_T))
+
  static int tdx_tsm_link_connect(struct pci_dev *pdev)
  {
-	return -ENXIO;
+	struct tdx_tsm_link *tlink = to_tdx_tsm_link(pdev->tsm);
+
+	struct tdx_tsm_link *tlink_spdm __free(tdx_spdm_session_teardown) =
+		tdx_spdm_session_setup(tlink);

Is the free() really needed here, either the session is correctly setup and tlink_spdm is returned. But if session_setup() files then what about calling spdm_session_disconnect() on an unestablished session?

Quoted 10 lines
drivers/virt/coco/tdx-host/tdx-host.c +8 -0
+	if (IS_ERR(tlink_spdm)) {
+		pci_err(pdev, "fail to setup spdm session\n");
+		return PTR_ERR(tlink_spdm);
+	}
+
+	retain_and_null_ptr(tlink_spdm);
+
+	return 0;
  }

<snip>

author
#142 lore raw

Re: [PATCH v2 27/31] coco/tdx-host: Implement SPDM session setup

Quoted 7 lines
Quoted 7 lines
drivers/virt/coco/tdx-host/tdx-host.c +7 -0
+#define TDISP_FUNC_ID		GENMASK(15, 0)
+#define TDISP_FUNC_ID_SEGMENT		GENMASK(23, 16)
+#define TDISP_FUNC_ID_SEG_VALID		BIT(24)
+
+static inline u32 tdisp_func_id(struct pci_dev *pdev)
+{
+	u32 func_id;
Quoted 8 lines

drivers/virt/coco/tdx-host/tdx-host.c MODULE_DEVICE_TABLE(x86cpu, tdx_host_ids); · line 46

Quoted 4 lines
drivers/virt/coco/tdx-host/tdx-host.c +4 -0
+
+	func_id = FIELD_PREP(TDISP_FUNC_ID_SEGMENT, pci_domain_nr(pdev->bus));
+	if (func_id)
+		func_id |= TDISP_FUNC_ID_SEG_VALID;

This check implies pci_domain_nr returning 0 is considered invalid. Other callers in the kernel seem to not care, they just use the domain nr, so is this check spurious or intentional ?

This is the func_id format defined in TDISP SPEC, bit 24 is a must. It is not the linux defined SBDF format.

Quoted 14 lines

drivers/virt/coco/tdx-host/tdx-host.c MODULE_DEVICE_TABLE(x86cpu, tdx_host_ids); · line 56

Quoted 11 lines
drivers/virt/coco/tdx-host/tdx-host.c +10 -0
+	func_id |= FIELD_PREP(TDISP_FUNC_ID,
+			      PCI_DEVID(pdev->bus->number, pdev->devfn));
+
+	return func_id;
+}
+
+struct spdm_config_info_t {
+	u32 vmm_spdm_cap;
+#define SPDM_CAP_HBEAT          BIT(13)
+#define SPDM_CAP_KEY_UPD        BIT(14)

nit: move those defines above the struct definition, they just break the reading flow as it is.

Yes.

...

Quoted 6 lines

drivers/virt/coco/tdx-host/tdx-host.c static int __maybe_unused tdx_spdm_msg_exchange(struct tdx_tsm_link *tlink, · line 211

Quoted 4 lines
drivers/virt/coco/tdx-host/tdx-host.c +4 -0
+static void *tdx_dup_array_data(struct tdx_page_array *array,
+				unsigned int data_size)
+{
+	unsigned int npages = (data_size + PAGE_SIZE - 1) / PAGE_SIZE;

nit: There's DIV_ROUND_UP

Yes.

...

Quoted 6 lines
Quoted 6 lines
drivers/virt/coco/tdx-host/tdx-host.c +3 -1
+DEFINE_FREE(tdx_spdm_session_teardown, struct tdx_tsm_link *,
+	    if (!IS_ERR_OR_NULL(_T)) tdx_spdm_session_teardown(_T))
+
  static int tdx_tsm_link_connect(struct pci_dev *pdev)
  {
-	return -ENXIO;
Quoted 8 lines

drivers/virt/coco/tdx-host/tdx-host.c static int __maybe_unused tdx_spdm_msg_exchange(struct tdx_tsm_link *tlink, · line 357

Quoted 4 lines
drivers/virt/coco/tdx-host/tdx-host.c +4 -0
+	struct tdx_tsm_link *tlink = to_tdx_tsm_link(pdev->tsm);
+
+	struct tdx_tsm_link *tlink_spdm __free(tdx_spdm_session_teardown) =
+		tdx_spdm_session_setup(tlink);

Is the free() really needed here, either the session is correctly setup and tlink_spdm is returned. But if session_setup() files then what about calling spdm_session_disconnect() on an unestablished session?

Ah, we have more steps to add, the __free() will take function when the following steps fail.

We may add __free() when we add more steps, but I think that makes the diff harder to read, so I want to keep this style.

Thanks.

Quoted 14 lines
Quoted 11 lines
drivers/virt/coco/tdx-host/tdx-host.c +8 -0
+	if (IS_ERR(tlink_spdm)) {
+		pci_err(pdev, "fail to setup spdm session\n");
+		return PTR_ERR(tlink_spdm);
+	}
+
+	retain_and_null_ptr(tlink_spdm);
+
+	return 0;
  }

<snip>

author
#143 lore raw

PATCH v2 28/31 coco/tdx-host: Parse ACPI KEYP table to init IDE for PCI host bridges

In reply to Xu Yilun

Parse the KEYP Key Configuration Units (KCU), to decide the max IDE streams supported for each host bridge.

The KEYP table points to a number of KCU structures that each associates with a list of root ports (RP) via segment, bus, and devfn. Sanity check the KEYP table, ensure all RPs listed for each KCU are included in one host bridge. Then extact the max IDE streams supported to pci_host_bridge via pci_ide_set_nr_streams().

---
 drivers/virt/coco/tdx-host/tdx-host.c | 111 ++++++++++++++++++++++++++
 1 file changed, 111 insertions(+)
drivers/virt/coco/tdx-host/tdx-host.c +111 -0
index 4d127b7c2591..d5072a68b81a 100644
@@ -5,6 +5,7 @@
  * Copyright (C) 2025 Intel Corporation
  */
 
+#include <linux/acpi.h>
 #include <linux/bitfield.h>
 #include <linux/device/faux.h>
 #include <linux/dmar.h>
@@ -12,6 +13,7 @@
 #include <linux/mod_devicetable.h>
 #include <linux/pci.h>
 #include <linux/pci-doe.h>
+#include <linux/pci-ide.h>
 #include <linux/pci-tsm.h>
 #include <linux/tsm.h>
 #include <linux/vmalloc.h>
@@ -474,6 +476,111 @@ static void unregister_link_tsm(void *link)
 	tsm_unregister(link);
 }
 
+#define KCU_STR_CAP_NUM_STREAMS		GENMASK(8, 0)
+
+/* The bus_end is inclusive */
+struct keyp_hb_info {
+	/* input */
+	u16 segment;
+	u8 bus_start;
+	u8 bus_end;
+	/* output */
+	u8 nr_ide_streams;
+};
+
+static bool keyp_info_match(struct acpi_keyp_rp_info *rp,
+			    struct keyp_hb_info *hb)
+{
+	return rp->segment == hb->segment && rp->bus >= hb->bus_start &&
+	       rp->bus <= hb->bus_end;
+}
+
+static int keyp_config_unit_handler(union acpi_subtable_headers *header,
+				    void *arg, const unsigned long end)
+{
+	struct acpi_keyp_config_unit *acpi_cu =
+		(struct acpi_keyp_config_unit *)&header->keyp;
+	struct keyp_hb_info *hb_info = arg;
+	int rp_size, rp_count, i;
+	void __iomem *addr;
+	bool match = false;
+	u32 cap;
+
+	rp_size = acpi_cu->header.length - sizeof(*acpi_cu);
+	if (rp_size % sizeof(struct acpi_keyp_rp_info))
+		return -EINVAL;
+
+	rp_count = rp_size / sizeof(struct acpi_keyp_rp_info);
+	if (!rp_count || rp_count != acpi_cu->root_port_count)
+		return -EINVAL;
+
+	for (i = 0; i < rp_count; i++) {
+		struct acpi_keyp_rp_info *rp_info = &acpi_cu->rp_info[i];
+
+		if (i == 0) {
+			match = keyp_info_match(rp_info, hb_info);
+			/* The host bridge already matches another KCU */
+			if (match && hb_info->nr_ide_streams)
+				return -EINVAL;
+
+			continue;
+		}
+
+		if (match ^ keyp_info_match(rp_info, hb_info))
+			return -EINVAL;
+	}
+
+	if (!match)
+		return 0;
+
+	addr = ioremap(acpi_cu->register_base_address, sizeof(cap));
+	if (!addr)
+		return -ENOMEM;
+	cap = ioread32(addr);
+	iounmap(addr);
+
+	hb_info->nr_ide_streams = FIELD_GET(KCU_STR_CAP_NUM_STREAMS, cap) + 1;
+
+	return 0;
+}
+
+static u8 keyp_find_nr_ide_stream(u16 segment, u8 bus_start, u8 bus_end)
+{
+	struct keyp_hb_info hb_info = {
+		.segment = segment,
+		.bus_start = bus_start,
+		.bus_end = bus_end,
+	};
+	int rc;
+
+	rc = acpi_table_parse_keyp(ACPI_KEYP_TYPE_CONFIG_UNIT,
+				   keyp_config_unit_handler, &hb_info);
+	if (rc < 0)
+		return 0;
+
+	return hb_info.nr_ide_streams;
+}
+
+static void keyp_setup_nr_ide_stream(struct pci_bus *bus)
+{
+	struct pci_host_bridge *hb = pci_find_host_bridge(bus);
+	u8 nr_ide_streams;
+
+	nr_ide_streams = keyp_find_nr_ide_stream(pci_domain_nr(bus),
+						 bus->busn_res.start,
+						 bus->busn_res.end);
+
+	pci_ide_set_nr_streams(hb, nr_ide_streams);
+}
+
+static void tdx_setup_nr_ide_stream(void)
+{
+	struct pci_bus *bus = NULL;
+
+	while ((bus = pci_find_next_bus(bus)))
+		keyp_setup_nr_ide_stream(bus);
+}
+
 static DEFINE_XARRAY(tlink_iommu_xa);
 
 static void tdx_iommu_clear(u64 iommu_id, struct tdx_page_array *iommu_mt)
@@ -569,6 +676,8 @@ static int __maybe_unused tdx_connect_init(struct device *dev)
 	if (ret)
 		return ret;
 
+	tdx_setup_nr_ide_stream();
+
 	link = tsm_register(dev, &tdx_tsm_link_ops);
 	if (IS_ERR(link))
 		return dev_err_probe(dev, PTR_ERR(link),
@@ -612,5 +721,7 @@ static void __exit tdx_host_exit(void)
 }
 module_exit(tdx_host_exit);
 
+MODULE_IMPORT_NS("ACPI");
+MODULE_IMPORT_NS("PCI_IDE");
 MODULE_DESCRIPTION("TDX Host Services");
 MODULE_LICENSE("GPL");
Signature · 1 line
2.25.1
#144 lore raw

Re: [PATCH v2 28/31] coco/tdx-host: Parse ACPI KEYP table to init IDE for PCI host bridges

Quoted 39 lines · Xu Yilun

Parse the KEYP Key Configuration Units (KCU), to decide the max IDE streams supported for each host bridge.

The KEYP table points to a number of KCU structures that each associates with a list of root ports (RP) via segment, bus, and devfn. Sanity check the KEYP table, ensure all RPs listed for each KCU are included in one host bridge. Then extact the max IDE streams supported to pci_host_bridge via pci_ide_set_nr_streams().

---
 drivers/virt/coco/tdx-host/tdx-host.c | 111 ++++++++++++++++++++++++++
 1 file changed, 111 insertions(+)
drivers/virt/coco/tdx-host/tdx-host.c +2 -0
index 4d127b7c2591..d5072a68b81a 100644
@@ -5,6 +5,7 @@
  * Copyright (C) 2025 Intel Corporation
  */
 
+#include <linux/acpi.h>
 #include <linux/bitfield.h>
 #include <linux/device/faux.h>
 #include <linux/dmar.h>
@@ -12,6 +13,7 @@
 #include <linux/mod_devicetable.h>
 #include <linux/pci.h>
 #include <linux/pci-doe.h>
+#include <linux/pci-ide.h>
 #include <linux/pci-tsm.h>
 #include <linux/tsm.h>
 #include <linux/vmalloc.h>
@@ -474,6 +476,111 @@ static void unregister_link_tsm(void *link)
 	tsm_unregister(link);

drivers/virt/coco/tdx-host/tdx-host.c static void unregister_link_tsm(void *link) · line 486

Quoted 10 lines
drivers/virt/coco/tdx-host/tdx-host.c +8 -0
 }
 
+#define KCU_STR_CAP_NUM_STREAMS		GENMASK(8, 0)
+
+/* The bus_end is inclusive */
+struct keyp_hb_info {
+	/* input */
+	u16 segment;
+	u8 bus_start;
+	u8 bus_end;

Would you consider using bus_last instead of bus_end to indicate that bus_last is inclusive? Or would it be better to say nr_buses to avoid the inclusive/exclusive problem?

Quoted 80 lines
drivers/virt/coco/tdx-host/tdx-host.c +80 -0
+	/* output */
+	u8 nr_ide_streams;
+};
+
+static bool keyp_info_match(struct acpi_keyp_rp_info *rp,
+			    struct keyp_hb_info *hb)
+{
+	return rp->segment == hb->segment && rp->bus >= hb->bus_start &&
+	       rp->bus <= hb->bus_end;
+}
+
+static int keyp_config_unit_handler(union acpi_subtable_headers *header,
+				    void *arg, const unsigned long end)
+{
+	struct acpi_keyp_config_unit *acpi_cu =
+		(struct acpi_keyp_config_unit *)&header->keyp;
+	struct keyp_hb_info *hb_info = arg;
+	int rp_size, rp_count, i;
+	void __iomem *addr;
+	bool match = false;
+	u32 cap;
+
+	rp_size = acpi_cu->header.length - sizeof(*acpi_cu);
+	if (rp_size % sizeof(struct acpi_keyp_rp_info))
+		return -EINVAL;
+
+	rp_count = rp_size / sizeof(struct acpi_keyp_rp_info);
+	if (!rp_count || rp_count != acpi_cu->root_port_count)
+		return -EINVAL;
+
+	for (i = 0; i < rp_count; i++) {
+		struct acpi_keyp_rp_info *rp_info = &acpi_cu->rp_info[i];
+
+		if (i == 0) {
+			match = keyp_info_match(rp_info, hb_info);
+			/* The host bridge already matches another KCU */
+			if (match && hb_info->nr_ide_streams)
+				return -EINVAL;
+
+			continue;
+		}
+
+		if (match ^ keyp_info_match(rp_info, hb_info))
+			return -EINVAL;
+	}
+
+	if (!match)
+		return 0;
+
+	addr = ioremap(acpi_cu->register_base_address, sizeof(cap));
+	if (!addr)
+		return -ENOMEM;
+	cap = ioread32(addr);
+	iounmap(addr);
+
+	hb_info->nr_ide_streams = FIELD_GET(KCU_STR_CAP_NUM_STREAMS, cap) + 1;
+
+	return 0;
+}
+
+static u8 keyp_find_nr_ide_stream(u16 segment, u8 bus_start, u8 bus_end)
+{
+	struct keyp_hb_info hb_info = {
+		.segment = segment,
+		.bus_start = bus_start,
+		.bus_end = bus_end,
+	};
+	int rc;
+
+	rc = acpi_table_parse_keyp(ACPI_KEYP_TYPE_CONFIG_UNIT,
+				   keyp_config_unit_handler, &hb_info);
+	if (rc < 0)
+		return 0;
+
+	return hb_info.nr_ide_streams;
+}
+
+static void keyp_setup_nr_ide_stream(struct pci_bus *bus)
+{
+	struct pci_host_bridge *hb = pci_find_host_bridge(bus);

drivers/virt/coco/tdx-host/tdx-host.c static void unregister_link_tsm(void *link) · line 576

Quoted 10 lines
drivers/virt/coco/tdx-host/tdx-host.c +10 -0
+	u8 nr_ide_streams;
+
+	nr_ide_streams = keyp_find_nr_ide_stream(pci_domain_nr(bus),
+						 bus->busn_res.start,
+						 bus->busn_res.end);
+
+	pci_ide_set_nr_streams(hb, nr_ide_streams);
+}
+
+static void tdx_setup_nr_ide_stream(void)

Was wondering, is nr in the name meant as an abbreviation for number? Why not just tdx_setup_ide_streams(), since this doesn't really set up a specific number of streams.

Same for the naming of keyp_setup_nr_ide_stream().

Quoted 29 lines
drivers/virt/coco/tdx-host/tdx-host.c +11 -1
+{
+	struct pci_bus *bus = NULL;
+
+	while ((bus = pci_find_next_bus(bus)))
+		keyp_setup_nr_ide_stream(bus);
+}
+
 static DEFINE_XARRAY(tlink_iommu_xa);
 
 static void tdx_iommu_clear(u64 iommu_id, struct tdx_page_array *iommu_mt)
@@ -569,6 +676,8 @@ static int __maybe_unused tdx_connect_init(struct device *dev)
 	if (ret)
 		return ret;
 
+	tdx_setup_nr_ide_stream();
+
 	link = tsm_register(dev, &tdx_tsm_link_ops);
 	if (IS_ERR(link))
 		return dev_err_probe(dev, PTR_ERR(link),
@@ -612,5 +721,7 @@ static void __exit tdx_host_exit(void)
 }
 module_exit(tdx_host_exit);
 
+MODULE_IMPORT_NS("ACPI");
+MODULE_IMPORT_NS("PCI_IDE");
 MODULE_DESCRIPTION("TDX Host Services");
 MODULE_LICENSE("GPL");
--

2.25.1

author
#145 lore raw

PATCH v2 29/31 x86/virt/tdx: Add SEAMCALL wrappers for IDE stream management

In reply to Xu Yilun

Add several SEAMCALL wrappers for IDE stream management.

- TDH.IDE.STREAM.CREATE creates IDE stream metadata buffers for TDX Module, and does root port side IDE configuration.
- TDH.IDE.STREAM.BLOCK clears the root port side IDE configuration.
- TDH.IDE.STREAM.DELETE releases the IDE stream metadata buffers.
- TDH.IDE.STREAM.KM deals with the IDE Key Management protocol (IDE-KM)

More information see Intel TDX Connect ABI Specification [1] Section 3.2 TDX Connect Host-Side (SEAMCALL) Interface Functions.

[1]: https://cdrdv2.intel.com/v1/dl/getContent/858625

---
 arch/x86/include/asm/tdx.h  | 14 ++++++
 arch/x86/virt/vmx/tdx/tdx.h |  4 ++
 arch/x86/virt/vmx/tdx/tdx.c | 86 +++++++++++++++++++++++++++++++++++++
 3 files changed, 104 insertions(+)
arch/x86/include/asm/tdx.h +14 -0
index 8abdad084972..7bdd66acda5b 100644
@@ -260,6 +260,20 @@ u64 tdh_exec_spdm_mng(u64 spdm_id, u64 spdm_op, struct page *spdm_param,
 		      struct page *spdm_rsp, struct page *spdm_req,
 		      struct tdx_page_array *spdm_out,
 		      u64 *spdm_req_or_out_len);
+u64 tdh_ide_stream_create(u64 stream_info, u64 spdm_id,
+			  struct tdx_page_array *stream_mt, u64 stream_ctrl,
+			  u64 rid_assoc1, u64 rid_assoc2,
+			  u64 addr_assoc1, u64 addr_assoc2,
+			  u64 addr_assoc3,
+			  u64 *stream_id,
+			  u64 *rp_ide_id);
+u64 tdh_ide_stream_block(u64 spdm_id, u64 stream_id);
+u64 tdh_ide_stream_delete(u64 spdm_id, u64 stream_id,
+			  struct tdx_page_array *stream_mt,
+			  unsigned int *nr_released, u64 *released_hpa);
+u64 tdh_ide_stream_km(u64 spdm_id, u64 stream_id, u64 operation,
+		      struct page *spdm_rsp, struct page *spdm_req,
+		      u64 *spdm_req_len);
 #else
 static inline void tdx_init(void) { }
 static inline int tdx_cpu_enable(void) { return -ENODEV; }
arch/x86/virt/vmx/tdx/tdx.h +4 -0
index 4784db2d1d92..d0a9694432de 100644
@@ -66,6 +66,10 @@
 #define TDH_IOMMU_CLEAR			129
 #define TDH_SPDM_CREATE			130
 #define TDH_SPDM_DELETE			131
+#define TDH_IDE_STREAM_CREATE		132
+#define TDH_IDE_STREAM_BLOCK		133
+#define TDH_IDE_STREAM_DELETE		134
+#define TDH_IDE_STREAM_KM		135
 #define TDH_SPDM_CONNECT		142
 #define TDH_SPDM_DISCONNECT		143
 #define TDH_SPDM_MNG			144
arch/x86/virt/vmx/tdx/tdx.c +86 -0

89 lines not on this page · raw message

Signature · 1 line
2.25.1
author
#146 lore raw

PATCH v2 30/31 coco/tdx-host: Implement IDE stream setup/teardown

In reply to Xu Yilun

Implementation for a most straightforward Selective IDE stream setup. Hard code all parameters for Stream Control Register. And no IDE Key Refresh support.

---
 include/linux/pci-ide.h               |   2 +
 drivers/pci/ide.c                     |   5 +-
 drivers/virt/coco/tdx-host/tdx-host.c | 226 ++++++++++++++++++++++++++
 3 files changed, 231 insertions(+), 2 deletions(-)
include/linux/pci-ide.h +2 -0
index 381a1bf22a95..f0c6975fd429 100644
@@ -106,6 +106,8 @@ struct pci_ide {
 void pci_ide_set_nr_streams(struct pci_host_bridge *hb, u16 nr);
 struct pci_ide_partner *pci_ide_to_settings(struct pci_dev *pdev,
 					    struct pci_ide *ide);
+void pci_ide_stream_to_regs(struct pci_dev *pdev, struct pci_ide *ide,
+			    struct pci_ide_regs *regs);
 struct pci_ide *pci_ide_stream_alloc(struct pci_dev *pdev);
 void pci_ide_stream_free(struct pci_ide *ide);
 int  pci_ide_stream_register(struct pci_ide *ide);
drivers/pci/ide.c +3 -2
index b35e8aba7ecb..1337608448c2 100644
@@ -556,8 +556,8 @@ static void mem_assoc_to_regs(struct pci_bus_region *region,
  * @ide: registered IDE settings descriptor
  * @regs: output register values
  */
-static void pci_ide_stream_to_regs(struct pci_dev *pdev, struct pci_ide *ide,
-				   struct pci_ide_regs *regs)
+void pci_ide_stream_to_regs(struct pci_dev *pdev, struct pci_ide *ide,
+			    struct pci_ide_regs *regs)
 {
 	struct pci_ide_partner *settings = pci_ide_to_settings(pdev, ide);
 	int assoc_idx = 0;
@@ -586,6 +586,7 @@ static void pci_ide_stream_to_regs(struct pci_dev *pdev, struct pci_ide *ide,
 
 	regs->nr_addr = assoc_idx;
 }
+EXPORT_SYMBOL_GPL(pci_ide_stream_to_regs);
 
 /**
  * pci_ide_stream_setup() - program settings to Selective IDE Stream registers
drivers/virt/coco/tdx-host/tdx-host.c +226 -0

251 lines not on this page · raw message

Signature · 1 line
2.25.1
#147 lore raw

RE: [PATCH v2 30/31] coco/tdx-host: Implement IDE stream setup/teardown

Quoted 7 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:02 AM

Implementation for a most straightforward Selective IDE stream setup. Hard code all parameters for Stream Control Register. And no IDE Key Refresh support.

'more straightforward', compared to what?

author
#148 lore raw

Re: [PATCH v2 30/31] coco/tdx-host: Implement IDE stream setup/teardown

Quoted 9 lines · Kevin
Quoted 7 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:02 AM

Implementation for a most straightforward Selective IDE stream setup. Hard code all parameters for Stream Control Register. And no IDE Key Refresh support.

'more straightforward', compared to what?

Actually it is " *most* straightforward", I just mean "very".

#149 lore raw

RE: [PATCH v2 30/31] coco/tdx-host: Implement IDE stream setup/teardown

Quoted 13 lines

From: Xu Yilun <[email protected]>
Sent: Wednesday, April 22, 2026 5:58 PM

Quoted 9 lines · Kevin
Quoted 7 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:02 AM

Implementation for a most straightforward Selective IDE stream setup. Hard code all parameters for Stream Control Register. And no IDE Key Refresh support.

'more straightforward', compared to what?

a typo.

Quoted 2 lines

Actually it is " *most* straightforward", I just mean "very".

When you say "most straightforward", then I want to know what are other options to compare. If you think that the thought practice leading to the 'most' definition is important, then please elaborate.

otherwise I'd just remove that sentence.

author
#150 lore raw

Re: [PATCH v2 30/31] coco/tdx-host: Implement IDE stream setup/teardown

Quoted 14 lines · Kevin
Quoted 13 lines

From: Xu Yilun <[email protected]>
Sent: Wednesday, April 22, 2026 5:58 PM

Quoted 9 lines · Kevin
Quoted 7 lines

From: Xu Yilun <[email protected]>
Sent: Saturday, March 28, 2026 12:02 AM

Implementation for a most straightforward Selective IDE stream setup. Hard code all parameters for Stream Control Register. And no IDE Key Refresh support.

'more straightforward', compared to what?

Quoted 7 lines

a typo.

Quoted 2 lines

Actually it is " *most* straightforward", I just mean "very".

When you say "most straightforward", then I want to know what are other options to compare. If you think that the thought practice

OK, I think the use of "a most" is somewhat misleading. I don't want to emphasize on comparison. I just want to give a summary about the
implementation: hard code all parameters, give no option for configurations, no optional features supported e.g. KEY Refresh.

So is it better I just s/most/very:

Implementation for a very straightforward Selective IDE stream setup...

Quoted 4 lines

leading to the 'most' definition is important, then please elaborate.

otherwise I'd just remove that sentence.

#151 lore raw

Re: [PATCH v2 30/31] coco/tdx-host: Implement IDE stream setup/teardown

In reply to Xu Yilun
Quoted 12 lines · Xu Yilun

[...snip...]

static int tdx_tsm_link_connect(struct pci_dev *pdev) {

drivers/virt/coco/tdx-host/tdx-host.c +2 -0
 	struct tdx_tsm_link *tlink = to_tdx_tsm_link(pdev->tsm);
@@ -362,7 +579,15 @@ static int tdx_tsm_link_connect(struct pci_dev *pdev)
 		return PTR_ERR(tlink_spdm);
 	}
 
+	struct tdx_tsm_link *tlink_ide __free(tdx_ide_stream_teardown) =
+		tdx_ide_stream_setup(tlink);

drivers/virt/coco/tdx-host/tdx-host.c static int tdx_tsm_link_connect(struct pci_dev *pdev) · line 593

Quoted 10 lines
drivers/virt/coco/tdx-host/tdx-host.c +6 -0
+	if (IS_ERR(tlink_ide)) {
+		pci_err(pdev, "fail to setup ide stream\n");
+		return PTR_ERR(tlink_ide);
+	}
+
 	retain_and_null_ptr(tlink_spdm);
+	retain_and_null_ptr(tlink_ide);
 
 	return 0;
 }

Would it be possible to define .connect = tdx_tsm_link_connect in the patch that sets up spdm, and then update tdx_tsm_link_connect in this patch?

I think it would help in showing how the spdm is associated with tdx_tsm_link_ops.

Would it also be possible to bring the definition of .probe in tdx_tsm_link_ops till after this patch too?

Quoted 10 lines
drivers/virt/coco/tdx-host/tdx-host.c +1 -1
@@ -371,6 +596,7 @@ static void tdx_tsm_link_disconnect(struct pci_dev *pdev)
 {
 	struct tdx_tsm_link *tlink = to_tdx_tsm_link(pdev->tsm);
 
+	tdx_ide_stream_teardown(tlink);
 	tdx_spdm_session_teardown(tlink);
 }
 
--

2.25.1

author
#152 lore raw

PATCH v2 31/31 coco/tdx-host: Finally enable SPDM session and IDE Establishment

In reply to Xu Yilun

The basic SPDM session and IDE functionalities are all implemented, enable them.

---
 drivers/virt/coco/tdx-host/tdx-host.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/virt/coco/tdx-host/tdx-host.c +2 -3
index 0f6056945788..7800afb0893d 100644
@@ -883,7 +883,7 @@ static int tdx_iommu_enable_all(void)
 	return ret;
 }
 
-static int __maybe_unused tdx_connect_init(struct device *dev)
+static int tdx_connect_init(struct device *dev)
 {
 	struct tsm_dev *link;
 	int ret;
@@ -914,8 +914,7 @@ static int __maybe_unused tdx_connect_init(struct device *dev)
 
 static int tdx_host_probe(struct faux_device *fdev)
 {
-	/* TODO: do tdx_connect_init() when it is fully implemented. */
-	return 0;
+	return tdx_connect_init(&fdev->dev);
 }
 
 static struct faux_device_ops tdx_host_ops = {
Signature · 1 line
2.25.1