Skip to contentSkip to conversation

Add TDX Guest Support (Debug support)

PATCH New Kuppuswamy Sathyanarayanan 1 participant 5 messages Jun 26, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Add TDX Guest Support (Debug support)

Hi All,

Intel's Trust Domain Extensions (TDX) protect guest VMs from malicious hosts and some physical attacks.

Following patches adds tracepoint support for TDX Guest TDCALL requests and #VE exceptions. It also includes helper function to detect TD-DEBUG mode which will be used by patches in other TDX series to add TD-DEBUG mode specific features support.

This series is the continuation of the following TDX guest related patches.

[set 1] - https://lore.kernel.org/patchwork/project/lkml/list/?series=505232
[set 2] - https://lore.kernel.org/patchwork/project/lkml/list/?series=506230
[set 3] - https://lore.kernel.org/patchwork/project/lkml/list/?series=506231
[set 4] - https://lore.kernel.org/patchwork/project/lkml/list/?series=506232

Also please note that this series alone is not necessarily fully functional. You need to apply all the above 4 patch series to get a fully functional TDX guest.

Kuppuswamy Sathyanarayanan (3):
x86/tdx: Add TDCALL tracepoint
x86/tdx: Expose TDX Guest #VE count in /proc/interrupts x86/tdx: Add tdg_debug_enabled() interface

Sean Christopherson (1):
x86/tdx: Add #VE tracepoint

 arch/x86/include/asm/hardirq.h   |   3 +
 arch/x86/include/asm/tdx.h       |   1 +
 arch/x86/include/asm/trace/tdx.h | 150 +++++++++++++++++++++++++++++++
 arch/x86/kernel/irq.c            |   6 ++
 arch/x86/kernel/tdx.c            |  71 ++++++++++++---
 arch/x86/kernel/traps.c          |   2 +
 6 files changed, 220 insertions(+), 13 deletions(-)
 create mode 100644 arch/x86/include/asm/trace/tdx.h
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/4 x86/tdx: Add #VE tracepoint

From: Sean Christopherson <[email protected]>

Add tracepoint for tracing TDX guest #VE exceptions. It will dump RIP, exit reason, exit qual, GPA, instruction length and instruction info for each #VE exception occurred.

Also, make trace points RCU idle safe to avoid warnings when RCU debugging is enabled.

---
 arch/x86/include/asm/trace/tdx.h | 51 ++++++++++++++++++++++++++++++++
 arch/x86/kernel/tdx.c            |  6 ++++
 2 files changed, 57 insertions(+)
 create mode 100644 arch/x86/include/asm/trace/tdx.h
arch/x86/include/asm/trace/tdx.h new file +51 -0
new file mode 100644
index 000000000000..e045e276009e
@@ -0,0 +1,51 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#if !defined(_TRACE_TDX_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_TDX_H
+
+#include <linux/tracepoint.h>
+
+#include <uapi/asm/vmx.h>
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM tdx
+
+#ifdef CONFIG_INTEL_TDX_GUEST
+
+TRACE_EVENT(tdg_virtualization_exception,
+	    TP_PROTO(u64 rip, u32 exit_reason, u64 exit_qual,
+		     u64 gpa, u32 instr_len, u32 instr_info),
+	    TP_ARGS(rip, exit_reason, exit_qual, gpa, instr_len,
+		    instr_info),
+	    TP_STRUCT__entry(
+			     __field(u64, rip)
+			     __field(u64, exit_qual)
+			     __field(u64, gpa)
+			     __field(u32, exit_reason)
+			     __field(u32, instr_len)
+			     __field(u32, instr_info)
+			     ),
+	    TP_fast_assign(
+			   __entry->rip = rip;
+			   __entry->exit_qual = exit_qual;
+			   __entry->gpa = gpa;
+			   __entry->exit_reason = exit_reason;
+			   __entry->instr_len = instr_len;
+			   __entry->instr_info = instr_info;
+			   ),
+	    TP_printk("reason %s rip 0x%016llx len %u info 0x%08x qual 0x%016llx gpa 0x%016llx",
+		      __print_symbolic(__entry->exit_reason, VMX_EXIT_REASONS),
+		      __entry->rip, __entry->instr_len, __entry->instr_info,
+		      __entry->exit_qual, __entry->gpa
+		      )
+	    );
+
+#endif // CONFIG_INTEL_TDX_GUEST
+
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH asm/trace/
+#undef TRACE_INCLUDE_FILE
+#define TRACE_INCLUDE_FILE tdx
+#endif /* _TRACE_TDX_H */
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
arch/x86/kernel/tdx.c +6 -0
index 4639206d99f3..67687cf3e631 100644
@@ -16,6 +16,9 @@
 #include <linux/sched/signal.h> /* force_sig_fault() */
 #include <linux/swiotlb.h>
 
+#define CREATE_TRACE_POINTS
+#include <asm/trace/tdx.h>
+
 /* TDX Module call Leaf IDs */
 #define TDINFO				1
 #define TDGETVEINFO			3
@@ -407,6 +410,9 @@ int tdg_handle_virtualization_exception(struct pt_regs *regs,
 	unsigned long val;
 	int ret = 0;
 
+	trace_tdg_virtualization_exception_rcuidle(regs->ip, ve->exit_reason,
+		ve->exit_qual, ve->gpa, ve->instr_len, ve->instr_info);
+
 	switch (ve->exit_reason) {
 	case EXIT_REASON_HLT:
 		tdg_halt();
Signature · 1 line
2.25.1
#2 lore raw

PATCH 2/4 x86/tdx: Add TDCALL tracepoint

In reply to Kuppuswamy Sathyanarayanan

Add TDCALL tracepoint in __tdx_module_call() and __tdx_hypercall() helper functions. These two helper functions are core calls for triggering the "TDCALL" instruction. Having a trace point for each TDCALL is useful for debugging and performance analysis, as well as testing.

For __tdx_module_call(), it will dump info about TDCALL leaf ID (RAX) and input parameters (RCX, RDX, R8, R9) in entry path and TDCALL return status (RAX) and output parameters (RCX, RDX, R8, R9, R10, R11) in exit path.

For __tdx_hypercall(), it will dump info about TDG.VP.VMCALL sub ID (R11), and input parameters (R12-R15) in entry path and TDG.VP.VMCALL return status (R10) and output parameters (R11-R15) in exit path.

Also, make trace points RCU idle safe to avoid warnings when RCU debugging is enabled.

---
 arch/x86/include/asm/trace/tdx.h | 99 ++++++++++++++++++++++++++++++++
 arch/x86/kernel/tdx.c            | 60 ++++++++++++++-----
 2 files changed, 146 insertions(+), 13 deletions(-)
arch/x86/include/asm/trace/tdx.h +99 -0
index e045e276009e..a94ac1e73b6c 100644
@@ -39,6 +39,105 @@ TRACE_EVENT(tdg_virtualization_exception,
 		      )
 	    );
 
+TRACE_EVENT(tdx_module_call_enter,
+	    TP_PROTO(u64 id, u64 rcx, u64 rdx, u64 r8, u64 r9),
+	    TP_ARGS(id, rcx, rdx, r8, r9),
+	    TP_STRUCT__entry(
+		__field(u64, id)
+		__field(u64, rcx)
+		__field(u64, rdx)
+		__field(u64, r8)
+		__field(u64, r9)
+		),
+	    TP_fast_assign(
+		__entry->id  = id;
+		__entry->rcx = rcx;
+		__entry->rdx = rdx;
+		__entry->r8  = r8;
+		__entry->r9  = r9;
+		),
+	    TP_printk("id %lld rcx 0x%016llx rdx 0x%016llx r8 0x%016llx r9 0x%016llx",
+		      __entry->id, __entry->rcx, __entry->rdx,
+		      __entry->r8, __entry->r9
+		      )
+	    );
+
+TRACE_EVENT(tdx_module_call_exit,
+	    TP_PROTO(u64 rax, u64 rcx, u64 rdx, u64 r8, u64 r9,
+		     u64 r10, u64 r11),
+	    TP_ARGS(rax, rcx, rdx, r8, r9, r10, r11),
+	    TP_STRUCT__entry(
+		__field(u64, rax)
+		__field(u64, rcx)
+		__field(u64, rdx)
+		__field(u64, r8)
+		__field(u64, r9)
+		__field(u64, r10)
+		__field(u64, r11)
+		),
+	    TP_fast_assign(
+		__entry->rax = rax;
+		__entry->rcx = rcx;
+		__entry->rdx = rdx;
+		__entry->r8  = r8;
+		__entry->r9  = r9;
+		__entry->r10 = r10;
+		__entry->r11 = r11;
+		),
+	    TP_printk("ret %lld rcx 0x%016llx rdx 0x%016llx r8 0x%016llx r9 0x%016llx r10 0x%016llx r11 0x%016llx",
+		      __entry->rax, __entry->rcx, __entry->rdx,
+		      __entry->r8, __entry->r9, __entry->r10, __entry->r11
+		      )
+	    );
+
+TRACE_EVENT(tdx_hypercall_enter,
+	    TP_PROTO(u64 id, u64 r12, u64 r13, u64 r14, u64 r15),
+	    TP_ARGS(id, r12, r13, r14, r15),
+	    TP_STRUCT__entry(
+		__field(u64, id)
+		__field(u64, r12)
+		__field(u64, r13)
+		__field(u64, r14)
+		__field(u64, r15)
+		),
+	    TP_fast_assign(
+		__entry->id  = id;
+		__entry->r12 = r12;
+		__entry->r13 = r13;
+		__entry->r14 = r14;
+		__entry->r15 = r15;
+		),
+	    TP_printk("subfn %lld r12 0x%016llx r13 0x%016llx r14 0x%016llx r15 0x%016llx",
+		      __entry->id, __entry->r12, __entry->r13,
+		      __entry->r14, __entry->r15
+		      )
+	    );
+
+TRACE_EVENT(tdx_hypercall_exit,
+	    TP_PROTO(u64 r10, u64 r11, u64 r12, u64 r13, u64 r14, u64 r15),
+	    TP_ARGS(r10, r11, r12, r13, r14, r15),
+	    TP_STRUCT__entry(
+		__field(u64, r10)
+		__field(u64, r11)
+		__field(u64, r12)
+		__field(u64, r13)
+		__field(u64, r14)
+		__field(u64, r15)
+		),
+	    TP_fast_assign(
+		__entry->r10 = r10;
+		__entry->r11 = r11;
+		__entry->r12 = r12;
+		__entry->r13 = r13;
+		__entry->r14 = r14;
+		__entry->r15 = r15;
+		),
+	    TP_printk("ret %lld r11 0x%016llx r12 0x%016llx r13 0x%016llx r14 0x%016llx r15 0x%016llx",
+		      __entry->r10, __entry->r11, __entry->r12,
+		      __entry->r13, __entry->r14, __entry->r15
+		      )
+	    );
+
 #endif // CONFIG_INTEL_TDX_GUEST
 
 #undef TRACE_INCLUDE_PATH
arch/x86/kernel/tdx.c +47 -13

129 lines not on this page · raw message

Signature · 1 line
2.25.1
#3 lore raw

PATCH 3/4 x86/tdx: Expose TDX Guest #VE count in /proc/interrupts

In reply to Kuppuswamy Sathyanarayanan

Add support to expose TD Guest Virtualization Exception (#VE) count in /proc/interrupts. It is useful in performance analysis of TD Guest.

---
 arch/x86/include/asm/hardirq.h | 3 +++
 arch/x86/kernel/irq.c          | 6 ++++++
 arch/x86/kernel/traps.c        | 2 ++
 3 files changed, 11 insertions(+)
arch/x86/include/asm/hardirq.h +3 -0
index 275e7fd20310..07d79fa9c5c6 100644
@@ -44,6 +44,9 @@ typedef struct {
 	unsigned int irq_hv_reenlightenment_count;
 	unsigned int hyperv_stimer0_count;
 #endif
+#if IS_ENABLED(CONFIG_INTEL_TDX_GUEST)
+	unsigned int tdg_ve_count;
+#endif
 } ____cacheline_aligned irq_cpustat_t;
 
 DECLARE_PER_CPU_SHARED_ALIGNED(irq_cpustat_t, irq_stat);
arch/x86/kernel/irq.c +6 -0
index e28f6a5d14f1..669869bd46ec 100644
@@ -181,6 +181,12 @@ int arch_show_interrupts(struct seq_file *p, int prec)
 		seq_printf(p, "%10u ",
 			   irq_stats(j)->kvm_posted_intr_wakeup_ipis);
 	seq_puts(p, "  Posted-interrupt wakeup event\n");
+#endif
+#if IS_ENABLED(CONFIG_INTEL_TDX_GUEST)
+	seq_printf(p, "%*s: ", prec, "TGV");
+	for_each_online_cpu(j)
+		seq_printf(p, "%10u ", irq_stats(j)->tdg_ve_count);
+	seq_puts(p, "  TDX Guest VE event\n");
 #endif
 	return 0;
 }
arch/x86/kernel/traps.c +2 -0
index d860fdee9cfe..3a6cfde8ca85 100644
@@ -1186,6 +1186,8 @@ DEFINE_IDTENTRY(exc_virtualization_exception)
 
 	RCU_LOCKDEP_WARN(!rcu_is_watching(), "entry code didn't wake RCU");
 
+	inc_irq_stat(tdg_ve_count);
+
 	/*
 	 * NMIs/Machine-checks/Interrupts will be in a disabled state
 	 * till TDGETVEINFO TDCALL is executed. This prevents #VE
Signature · 1 line
2.25.1
#4 lore raw

PATCH 4/4 x86/tdx: Add tdg_debug_enabled() interface

In reply to Kuppuswamy Sathyanarayanan

A guest TD is defined as debuggable if its ATTRIBUTES.DEBUG bit is 1. In this mode, the host VMM can use Intel TDX functions to access secret TD state that is not accessible for non-debuggable TDs. A debuggable TD is, by nature, untrusted.

Since the TD’s ATTRIBUTES are included in the TDG.MR.REPORT, the TD’s debuggability state can be known to any third party to which the TD attests. TD Attributes are initialized during TD INIT call. You can get more details about debug features in Intel Trust Domain Extensions (Intel TDX) Module Architecture specification, sec 13.3.

Add a new interface to detect the TDX debug mode. This will be used by follow-on patches. Examples of its usage are, when adding command line debug options to disable TDX features like driver or port filter, tdg_debug_enabled() is used to make sure it is used only in debug mode.

https://software.intel.com/content/dam/develop/external/us/en/documents/tdx-module-1eas-v0.85.039.pdf

---
 arch/x86/include/asm/tdx.h | 1 +
 arch/x86/kernel/tdx.c      | 5 +++++
 2 files changed, 6 insertions(+)
arch/x86/include/asm/tdx.h +1 -0
index 665c8cf57d5b..1c0d4bf693a3 100644
@@ -67,6 +67,7 @@ enum tdx_map_type {
 
 #ifdef CONFIG_INTEL_TDX_GUEST
 
+bool tdg_debug_enabled(void);
 void __init tdx_early_init(void);
 
 bool tdx_prot_guest_has(unsigned long flag);
arch/x86/kernel/tdx.c +5 -0
index 99e6701e7f48..1226c1b234bd 100644
@@ -123,6 +123,11 @@ phys_addr_t tdg_shared_mask(void)
 	return 1ULL << (td_info.gpa_width - 1);
 }
 
+bool tdg_debug_enabled(void)
+{
+	return td_info.attributes & BIT(0);
+}
+
 static void tdg_get_info(void)
 {
 	u64 ret;
Signature · 1 line
2.25.1