x86: enable FRED for x86-64
Subsystems (16)
v5 cover x86: enable FRED for x86-64
This patch set enables FRED for x86-64.
The Intel flexible return and event delivery (FRED) architecture defines simple new transitions that change privilege level (ring transitions). The FRED architecture was designed with the following goals:
1) Improve overall performance and response time by replacing event delivery through the interrupt descriptor table (IDT event delivery) and event return by the IRET instruction with lower latency transitions.
2) Improve software robustness by ensuring that event delivery establishes the full supervisor context and that event return establishes the full user context.
The new transitions defined by the FRED architecture are FRED event delivery and, for returning from events, two FRED return instructions. FRED event delivery can effect a transition from ring 3 to ring 0, but it is used also to deliver events incident to ring 0. One FRED instruction (ERETU) effects a return from ring 0 to ring 3, while the other (ERETS) returns while remaining in ring 0.
Search for the latest FRED spec in most search engines with this search pattern:
site:intel.com FRED (flexible return and event delivery) specification
As of now there is no publicly avaiable CPU supporting FRED, thus the Intel Simics® Simulator is used as software development and testing vehicles. And it can be downloaded from:
https://www.intel.com/content/www/us/en/developer/articles/tool/simics-simulator.html
To enable FRED, the Simics package 8112 QSP-CPU needs to be installed with CPU model configured as:
$cpu_comp_class = "x86-experimental-fred"
Longer term, we should refactor common code shared by FRED and IDT into common shared files, and contain IDT code using a new config CONFIG_X86_IDT.
Changes since v4:
* Rebased against v6.3-rc1.
* Do NOT use the term "injection", which in the KVM context means to reinject an event into the guest (Sean Christopherson).
* Add the explanation of why to execute "int $2" to invoke the NMI handler in NMI caused VM exits (Sean Christopherson).
* Use cs/ss instead of csx/ssx when initializing the pt_regs structure for calling external_interrupt(), otherwise it breaks i386 build.
Changes since v3:
* Call external_interrupt() to handle IRQ in IRQ caused VM exits.
* Execute "int $2" to handle NMI in NMI caused VM exits.
* Rename csl/ssl of the pt_regs structure to csx/ssx (x for extended) (Andrew Cooper).
Changes since v2:
* Improve comments for changes in arch/x86/include/asm/idtentry.h.
Changes since v1:
* call irqentry_nmi_{enter,exit}() in both IDT and FRED debug fault kernel handler (Peter Zijlstra).
* Initialize a FRED exception handler to fred_bad_event() instead of NULL if no FRED handler defined for an exception vector (Peter Zijlstra).
* Push calling irqentry_{enter,exit}() and instrumentation_{begin,end}() down into individual FRED exception handlers, instead of in the dispatch framework (Peter Zijlstra).
H. Peter Anvin (Intel) (24):
x86/traps: let common_interrupt() handle IRQ_MOVE_CLEANUP_VECTOR x86/traps: add a system interrupt table for system interrupt dispatch x86/traps: add external_interrupt() to dispatch external interrupts x86/cpufeature: add the cpu feature bit for FRED x86/opcode: add ERETU, ERETS instructions to x86-opcode-map x86/objtool: teach objtool about ERETU and ERETS
x86/cpu: add X86_CR4_FRED macro
x86/fred: add Kconfig option for FRED (CONFIG_X86_FRED) x86/fred: if CONFIG_X86_FRED is disabled, disable FRED support x86/cpu: add MSR numbers for FRED configuration
x86/fred: header file with FRED definitions
x86/fred: make unions for the cs and ss fields in struct pt_regs x86/fred: reserve space for the FRED stack frame
x86/fred: add a page fault entry stub for FRED
x86/fred: add a debug fault entry stub for FRED
x86/fred: add a NMI entry stub for FRED
x86/fred: FRED entry/exit and dispatch code
x86/fred: FRED initialization code
x86/fred: update MSR_IA32_FRED_RSP0 during task switch x86/fred: let ret_from_fork() jmp to fred_exit_user when FRED is
enabled
x86/fred: disallow the swapgs instruction when FRED is enabled x86/fred: no ESPFIX needed when FRED is enabled
x86/fred: allow single-step trap and NMI when starting a new thread x86/fred: allow FRED systems to use interrupt vectors 0x10-0x1f
Xin Li (10):
x86/traps: add install_system_interrupt_handler() x86/traps: export external_interrupt() for VMX IRQ reinjection x86/fred: header file for event types
x86/fred: add a machine check entry stub for FRED x86/fred: fixup fault on ERETU by jumping to fred_entrypoint_user x86/ia32: do not modify the DPL bits for a null selector x86/fred: allow dynamic stack frame size
x86/fred: disable FRED by default in its early stage
KVM: x86/vmx: call external_interrupt() to handle IRQ in IRQ caused VM
exits
KVM: x86/vmx: execute "int $2" to handle NMI in NMI caused VM exits
when FRED is enabled
.../admin-guide/kernel-parameters.txt | 4 + arch/x86/Kconfig | 9 + arch/x86/entry/Makefile | 5 +- arch/x86/entry/entry_32.S | 2 +- arch/x86/entry/entry_64.S | 5 + arch/x86/entry/entry_64_fred.S | 59 +++++ arch/x86/entry/entry_fred.c | 234 ++++++++++++++++++ arch/x86/entry/vsyscall/vsyscall_64.c | 2 +- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/entry-common.h | 3 + arch/x86/include/asm/event-type.h | 17 ++ arch/x86/include/asm/extable_fixup_types.h | 4 +- arch/x86/include/asm/fred.h | 131 ++++++++++ arch/x86/include/asm/idtentry.h | 76 +++++- arch/x86/include/asm/irq.h | 5 + arch/x86/include/asm/irq_vectors.h | 15 +- arch/x86/include/asm/msr-index.h | 13 +- arch/x86/include/asm/processor.h | 12 +- arch/x86/include/asm/ptrace.h | 36 ++- arch/x86/include/asm/switch_to.h | 10 +- arch/x86/include/asm/thread_info.h | 35 +-- arch/x86/include/asm/traps.h | 13 + arch/x86/include/asm/vmx.h | 17 +- arch/x86/include/uapi/asm/processor-flags.h | 2 + arch/x86/kernel/Makefile | 1 + arch/x86/kernel/apic/apic.c | 11 +- arch/x86/kernel/apic/vector.c | 8 +- arch/x86/kernel/cpu/acrn.c | 7 +- arch/x86/kernel/cpu/common.c | 88 ++++--- arch/x86/kernel/cpu/mce/core.c | 11 + arch/x86/kernel/cpu/mshyperv.c | 22 +- arch/x86/kernel/espfix_64.c | 8 + arch/x86/kernel/fred.c | 73 ++++++ arch/x86/kernel/head_32.S | 3 +- arch/x86/kernel/idt.c | 6 +- arch/x86/kernel/irq.c | 6 +- arch/x86/kernel/irqinit.c | 7 +- arch/x86/kernel/kvm.c | 4 +- arch/x86/kernel/nmi.c | 28 +++ arch/x86/kernel/process.c | 5 + arch/x86/kernel/process_64.c | 21 +- arch/x86/kernel/signal_32.c | 21 +- arch/x86/kernel/traps.c | 175 +++++++++++-- arch/x86/kvm/vmx/vmx.c | 33 ++- arch/x86/lib/x86-opcode-map.txt | 2 +- arch/x86/mm/extable.c | 28 +++ arch/x86/mm/fault.c | 20 +- drivers/xen/events/events_base.c | 5 +- kernel/fork.c | 6 + tools/arch/x86/include/asm/cpufeatures.h | 1 + .../arch/x86/include/asm/disabled-features.h | 8 +- tools/arch/x86/include/asm/msr-index.h | 13 +- tools/arch/x86/lib/x86-opcode-map.txt | 2 +- tools/objtool/arch/x86/decode.c | 19 +- 55 files changed, 1185 insertions(+), 175 deletions(-) create mode 100644 arch/x86/entry/entry_64_fred.S create mode 100644 arch/x86/entry/entry_fred.c create mode 100644 arch/x86/include/asm/event-type.h create mode 100644 arch/x86/include/asm/fred.h create mode 100644 arch/x86/kernel/fred.c
Signature · 1 line
2.34.1
PATCH v5 1/34 x86/traps: let common_interrupt() handle IRQ_MOVE_CLEANUP_VECTOR
From: "H. Peter Anvin (Intel)" <[email protected]>
IRQ_MOVE_CLEANUP_VECTOR is the only one of the system IRQ vectors that is *below* FIRST_SYSTEM_VECTOR. It is a slow path, so just push it into common_interrupt() just before the spurious interrupt handling.
--- arch/x86/kernel/irq.c | 4 ++++ 1 file changed, 4 insertions(+)
arch/x86/kernel/irq.c +4 -0
@@ -248,6 +248,10 @@ DEFINE_IDTENTRY_IRQ(common_interrupt) desc =__this_cpu_read (vector_irq[vector]);if (likely (!IS_ERR_OR_NULL (desc))) {handle_irq (desc, regs); +#ifdef CONFIG_SMP + }else if (vector ==IRQ_MOVE_CLEANUP_VECTOR ) { +sysvec_irq_move_cleanup (regs); +#endif }else {ack_APIC_irq ();
Signature · 1 line
2.34.1
PATCH v5 2/34 x86/traps: add a system interrupt table for system interrupt dispatch
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Upon receiving an external interrupt, KVM VMX reinjects it through calling the interrupt handler in its IDT descriptor on the current kernel stack, which essentially uses the IDT as an interrupt dispatch table.
However the IDT is one of the lowest level critical data structures between a x86 CPU and the Linux kernel, we should avoid using it *directly* whenever possible, espeically in a software defined manner.
On x86, external interrupts are divided into the following groups
1) system interrupts
2) external device interrupts
With the IDT, system interrupts are dispatched through the IDT directly, while external device interrupts are all routed to the external interrupt dispatch function common_interrupt(), which dispatches external device interrupts through a per-CPU external interrupt dispatch table vector_irq.
To eliminate dispatching external interrupts through the IDT, add a system interrupt handler table for dispatching a system interrupt to its corresponding handler directly. Thus a software based dispatch function will be:
void external_interrupt(struct pt_regs *regs, u8 vector)
{
if (is_system_interrupt(vector))
system_interrupt_handlers[vector_to_sysvec(vector)](regs);
else /* external device interrupt */
common_interrupt(regs, vector);
}What's more, with the Intel FRED (Flexible Return and Event Delivery) architecture, IDT, the hardware based event dispatch table, is gone, and the Linux kernel needs to dispatch events to their handlers with vector to handler mappings, the dispatch function external_interrupt() is also needed.
--- arch/x86/include/asm/idtentry.h | 64 +++++++++++++++++++++++++++------ arch/x86/include/asm/traps.h | 7 ++++ arch/x86/kernel/traps.c | 40 +++++++++++++++++++++ 3 files changed, 100 insertions(+), 11 deletions(-)
arch/x86/include/asm/idtentry.h +53 -11
@@ -167,17 +167,22 @@ __visible noinstr void func(struct pt_regs *regs, unsigned long error_code)/** * DECLARE_IDTENTRY_IRQ - Declare functions for device interrupt IDT entry -* points (common/spurious) +* points (common/spurious) and their corresponding +* software based dispatch handlers in the non-noinstr +* text section * @vector: Vector number (ignored for C) * @func: Function name of the entry point * -* Maps to DECLARE_IDTENTRY_ERRORCODE() +* Maps to DECLARE_IDTENTRY_ERRORCODE(), plus a dispatch function prototype */ #define DECLARE_IDTENTRY_IRQ (vector, func) \ - DECLARE_IDTENTRY_ERRORCODE(vector, func) + DECLARE_IDTENTRY_ERRORCODE(vector, func); \ + void dispatch_##func(struct pt_regs *regs, unsigned long error_code)/** * DEFINE_IDTENTRY_IRQ - Emit code for device interrupt IDT entry points +* and their corresponding software based dispatch +* handlers in the non-noinstr text section * @func: Function name of the entry point * * The vector number is pushed by the low level entry stub and handed
@@ -187,6 +192,9 @@ __visible noinstr void func(struct pt_regs *regs, unsigned long error_code)* irq_enter/exit_rcu() are invoked before the function body and the * KVM L1D flush request is set. Stack switching to the interrupt stack * has to be done in the function body if necessary. +* +* dispatch_func() is a software based dispatch handler in the non-noinstr +* text section. */ #define DEFINE_IDTENTRY_IRQ (func) \ static void __##func(struct pt_regs *regs, u32 vector); \
@@ -204,31 +212,48 @@ __visible noinstr void func(struct pt_regs *regs, \irqentry_exit (regs, state); \ } \ \ +void dispatch_##func(struct pt_regs *regs, unsigned long error_code) \ +{ \ + u32 vector = (u32)(u8)error_code; \ + \ + kvm_set_cpu_l1tf_flush_l1d(); \ + run_irq_on_irqstack_cond(__##func, regs, vector); \ +} \ + \ static noinline void __##func(struct pt_regs *regs, u32 vector)/** * DECLARE_IDTENTRY_SYSVEC - Declare functions for system vector entry points +* and their corresponding software based dispatch +* handlers in the non-noinstr text section * @vector: Vector number (ignored for C) * @func: Function name of the entry point * -* Declares three functions: +* Declares four functions: * - The ASM entry point: asm_##func * - The XEN PV trap entry point: xen_##func (maybe unused) * - The C handler called from the ASM entry point +* - The C handler used in the system interrupt handler table * -* Maps to DECLARE_IDTENTRY(). +* Maps to DECLARE_IDTENTRY(), plus a dispatch table function prototype */ #define DECLARE_IDTENTRY_SYSVEC (vector, func) \ - DECLARE_IDTENTRY(vector, func) + DECLARE_IDTENTRY(vector, func); \ + void dispatch_table_##func(struct pt_regs *regs)/** * DEFINE_IDTENTRY_SYSVEC - Emit code for system vector IDT entry points +* and their corresponding software based dispatch +* handlers in the non-noinstr text section * @func: Function name of the entry point * * irqentry_enter/exit() and irq_enter/exit_rcu() are invoked before the * function body. KVM L1D flush request is set. * -* Runs the function on the interrupt stack if the entry hit kernel mode +* Runs the function on the interrupt stack if the entry hit kernel mode. +* +* dispatch_table_func() is used in the system interrupt handler table for +* system interrupts dispatching. */ #define DEFINE_IDTENTRY_SYSVEC (func) \ static void __##func(struct pt_regs *regs); \
@@ -244,11 +269,19 @@ __visible noinstr void func(struct pt_regs *regs) \irqentry_exit (regs, state); \ } \ \ +void dispatch_table_##func(struct pt_regs *regs) \ +{ \ + kvm_set_cpu_l1tf_flush_l1d(); \ + run_sysvec_on_irqstack_cond(__##func, regs); \ +} \ + \ static noinline void __##func(struct pt_regs *regs)/** * DEFINE_IDTENTRY_SYSVEC_SIMPLE - Emit code for simple system vector IDT -* entry points +* entry points and their corresponding +* software based dispatch handlers in +* the non-noinstr text section * @func: Function name of the entry point * * Runs the function on the interrupted stack. No switch to IRQ stack and
@@ -256,6 +289,9 @@ static noinline void __##func(struct pt_regs *regs)* * Only use for 'empty' vectors like reschedule IPI and KVM posted * interrupt vectors. +* +* dispatch_table_func() is used in the system interrupt handler table for +* system interrupts dispatching. */ #define DEFINE_IDTENTRY_SYSVEC_SIMPLE (func) \ static __always_inline void __##func(struct pt_regs *regs); \
@@ -273,6 +309,14 @@ __visible noinstr void func(struct pt_regs *regs) \irqentry_exit (regs, state); \ } \ \ +void dispatch_table_##func(struct pt_regs *regs) \ +{ \ + __irq_enter_raw(); \ + kvm_set_cpu_l1tf_flush_l1d(); \ + __##func (regs); \ + __irq_exit_raw(); \ +} \ + \ static __always_inline void __##func(struct pt_regs *regs)/**
@@ -634,9 +678,7 @@ DECLARE_IDTENTRY(X86_TRAP_VE, exc_virtualization_exception);/* Device interrupts common/spurious */ DECLARE_IDTENTRY_IRQ (X86_TRAP_OTHER , common_interrupt); -#ifdef CONFIG_X86_LOCAL_APIC DECLARE_IDTENTRY_IRQ (X86_TRAP_OTHER , spurious_interrupt); -#endif /* System vector entry points */ #ifdef CONFIG_X86_LOCAL_APIC
@@ -647,7 +689,7 @@ DECLARE_IDTENTRY_SYSVEC(X86_PLATFORM_IPI_VECTOR, sysvec_x86_platform_ipi);#endif #ifdef CONFIG_SMP -DECLARE_IDTENTRY (RESCHEDULE_VECTOR , sysvec_reschedule_ipi); +DECLARE_IDTENTRY_SYSVEC (RESCHEDULE_VECTOR , sysvec_reschedule_ipi);DECLARE_IDTENTRY_SYSVEC (IRQ_MOVE_CLEANUP_VECTOR , sysvec_irq_move_cleanup);DECLARE_IDTENTRY_SYSVEC (REBOOT_VECTOR , sysvec_reboot);DECLARE_IDTENTRY_SYSVEC (CALL_FUNCTION_SINGLE_VECTOR , sysvec_call_function_single);
arch/x86/include/asm/traps.h +7 -0
@@ -47,4 +47,11 @@ void __noreturn handle_stack_overflow(struct pt_regs *regs, struct stack_info *info);#endif +/* +* How system interrupt handlers are called. +*/ +#define DECLARE_SYSTEM_INTERRUPT_HANDLER (f) \ + void f (struct pt_regs *regs) +typedef DECLARE_SYSTEM_INTERRUPT_HANDLER ((*system_interrupt_handler )); +#endif /* _ASM_X86_TRAPS_H */
arch/x86/kernel/traps.c +40 -0
@@ -1451,6 +1451,46 @@ DEFINE_IDTENTRY_SW(iret_error) }#endif +#define SYSV (x,y) [(x) - FIRST_SYSTEM_VECTOR] = y + +static system_interrupt_handler system_interrupt_handlers[NR_SYSTEM_VECTORS ] = { +#ifdef CONFIG_SMP +SYSV (RESCHEDULE_VECTOR , dispatch_table_sysvec_reschedule_ipi), +SYSV (CALL_FUNCTION_VECTOR , dispatch_table_sysvec_call_function), +SYSV (CALL_FUNCTION_SINGLE_VECTOR , dispatch_table_sysvec_call_function_single), +SYSV (REBOOT_VECTOR , dispatch_table_sysvec_reboot), +#endif + +#ifdef CONFIG_X86_THERMAL_VECTOR +SYSV (THERMAL_APIC_VECTOR , dispatch_table_sysvec_thermal), +#endif + +#ifdef CONFIG_X86_MCE_THRESHOLD +SYSV (THRESHOLD_APIC_VECTOR , dispatch_table_sysvec_threshold), +#endif + +#ifdef CONFIG_X86_MCE_AMD +SYSV (DEFERRED_ERROR_VECTOR , dispatch_table_sysvec_deferred_error), +#endif + +#ifdef CONFIG_X86_LOCAL_APIC +SYSV (LOCAL_TIMER_VECTOR , dispatch_table_sysvec_apic_timer_interrupt), +SYSV (X86_PLATFORM_IPI_VECTOR , dispatch_table_sysvec_x86_platform_ipi), +# ifdef CONFIG_HAVE_KVM +SYSV (POSTED_INTR_VECTOR , dispatch_table_sysvec_kvm_posted_intr_ipi), +SYSV (POSTED_INTR_WAKEUP_VECTOR , dispatch_table_sysvec_kvm_posted_intr_wakeup_ipi), +SYSV (POSTED_INTR_NESTED_VECTOR , dispatch_table_sysvec_kvm_posted_intr_nested_ipi), +# endif +# ifdef CONFIG_IRQ_WORK +SYSV (IRQ_WORK_VECTOR , dispatch_table_sysvec_irq_work), +# endif +SYSV (SPURIOUS_APIC_VECTOR , dispatch_table_sysvec_spurious_apic_interrupt), +SYSV (ERROR_APIC_VECTOR , dispatch_table_sysvec_error_interrupt), +#endif +}; + +#undef SYSV +void __init trap_init (void ) {/* Init cpu_entry_area before IST entries are set up */
Signature · 1 line
2.34.1
PATCH v5 3/34 x86/traps: add install_system_interrupt_handler()
In reply to Xin LiSome kernel components install system interrupt handlers into the IDT, and we need to do the same for system_interrupt_handlers. A new function install_system_interrupt_handler() is added to install a system interrupt handler into both the IDT and system_interrupt_handlers.
--- arch/x86/include/asm/traps.h | 2 ++ arch/x86/kernel/cpu/acrn.c | 7 +++++-- arch/x86/kernel/cpu/mshyperv.c | 22 ++++++++++++++-------- arch/x86/kernel/kvm.c | 4 +++- arch/x86/kernel/traps.c | 8 ++++++++ drivers/xen/events/events_base.c | 5 ++++- 6 files changed, 36 insertions(+), 12 deletions(-)
arch/x86/include/asm/traps.h +2 -0
@@ -41,6 +41,8 @@ void math_emulate(struct math_emu_info *);bool fault_in_kernel_space (unsigned long address); +void install_system_interrupt_handler (unsigned int n,const void *asm_addr,const void *addr); +#ifdef CONFIG_VMAP_STACK void __noreturn handle_stack_overflow(struct pt_regs *regs, unsigned long fault_address,
arch/x86/kernel/cpu/acrn.c +5 -2
@@ -18,6 +18,7 @@#include <asm/hypervisor.h> #include <asm/idtentry.h> #include <asm/irq_regs.h> +#include <asm/traps.h> static u32 __init acrn_detect (void ) {
@@ -26,8 +27,10 @@ static u32 __init acrn_detect(void)static void __init acrn_init_platform (void ) { -/* Setup the IDT for ACRN hypervisor callback */ -alloc_intr_gate (HYPERVISOR_CALLBACK_VECTOR , asm_sysvec_acrn_hv_callback); +/* Install system interrupt handler for ACRN hypervisor callback */ +install_system_interrupt_handler (HYPERVISOR_CALLBACK_VECTOR , + asm_sysvec_acrn_hv_callback, + sysvec_acrn_hv_callback); x86_platform.calibrate_tsc = acrn_get_tsc_khz; x86_platform.calibrate_cpu = acrn_get_tsc_khz;
arch/x86/kernel/cpu/mshyperv.c +14 -8
@@ -29,6 +29,7 @@#include <asm/i8259.h> #include <asm/apic.h> #include <asm/timer.h> +#include <asm/traps.h> #include <asm/reboot.h> #include <asm/nmi.h> #include <clocksource/hyperv_timer.h>
@@ -487,19 +488,24 @@ static void __init ms_hyperv_init_platform(void)*/ x86_platform.apic_post_init = hyperv_init;hyperv_setup_mmu_ops (); -/* Setup the IDT for hypervisor callback */ -alloc_intr_gate (HYPERVISOR_CALLBACK_VECTOR , asm_sysvec_hyperv_callback); -/* Setup the IDT for reenlightenment notifications */ +/* Install system interrupt handler for hypervisor callback */ +install_system_interrupt_handler (HYPERVISOR_CALLBACK_VECTOR , + asm_sysvec_hyperv_callback, + sysvec_hyperv_callback); + +/* Install system interrupt handler for reenlightenment notifications */ if (ms_hyperv.features &HV_ACCESS_REENLIGHTENMENT ) { -alloc_intr_gate (HYPERV_REENLIGHTENMENT_VECTOR , - asm_sysvec_hyperv_reenlightenment); +install_system_interrupt_handler (HYPERV_REENLIGHTENMENT_VECTOR , + asm_sysvec_hyperv_reenlightenment, + sysvec_hyperv_reenlightenment); } -/* Setup the IDT for stimer0 */ +/* Install system interrupt handler for stimer0 */ if (ms_hyperv.misc_features &HV_STIMER_DIRECT_MODE_AVAILABLE ) { -alloc_intr_gate (HYPERV_STIMER0_VECTOR , - asm_sysvec_hyperv_stimer0); +install_system_interrupt_handler (HYPERV_STIMER0_VECTOR , + asm_sysvec_hyperv_stimer0, + sysvec_hyperv_stimer0); }# ifdef CONFIG_SMP
arch/x86/kernel/kvm.c +3 -1
@@ -829,7 +829,9 @@ static void __init kvm_guest_init(void)if (kvm_para_has_feature (KVM_FEATURE_ASYNC_PF_INT ) && kvmapf) {static_branch_enable (&kvm_async_pf_enabled); -alloc_intr_gate (HYPERVISOR_CALLBACK_VECTOR , asm_sysvec_kvm_asyncpf_interrupt); +install_system_interrupt_handler (HYPERVISOR_CALLBACK_VECTOR , + asm_sysvec_kvm_asyncpf_interrupt, + sysvec_kvm_asyncpf_interrupt); }#ifdef CONFIG_SMP
arch/x86/kernel/traps.c +8 -0
@@ -1491,6 +1491,14 @@ static system_interrupt_handler system_interrupt_handlers[NR_SYSTEM_VECTORS] = {#undef SYSV +void __init install_system_interrupt_handler (unsigned int n,const void *asm_addr,const void *addr) +{ +BUG_ON (n <FIRST_SYSTEM_VECTOR ); + + system_interrupt_handlers[n -FIRST_SYSTEM_VECTOR ] = (system_interrupt_handler )addr; +alloc_intr_gate (n, asm_addr); +} +void __init trap_init (void ) {/* Init cpu_entry_area before IST entries are set up */
drivers/xen/events/events_base.c +4 -1
@@ -45,6 +45,7 @@#include <asm/irq.h> #include <asm/io_apic.h> #include <asm/i8259.h> +#include <asm/traps.h> #include <asm/xen/cpuid.h> #include <asm/xen/pci.h> #endif
@@ -2249,7 +2250,9 @@ static __init void xen_alloc_callback_vector(void)return ;pr_info ("Xen HVM callback vector for event delivery is enabled\n" ); - alloc_intr_gate(HYPERVISOR_CALLBACK_VECTOR, asm_sysvec_xen_hvm_callback); +install_system_interrupt_handler (HYPERVISOR_CALLBACK_VECTOR , + asm_sysvec_xen_hvm_callback, + sysvec_xen_hvm_callback); }#else void xen_setup_callback_vector (void ) {}
Signature · 1 line
2.34.1
PATCH v5 4/34 x86/traps: add external_interrupt() to dispatch external interrupts
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Add external_interrupt() to dispatch external interrupts to their handlers. If an external interrupt is a system interrupt, dipatch it through system_interrupt_handler_table, otherwise call into dispatch_common_interrupt().
--- arch/x86/kernel/traps.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)
arch/x86/kernel/traps.c +41 -0
@@ -1499,6 +1499,47 @@ void __init install_system_interrupt_handler(unsigned int n, const void *asm_addalloc_intr_gate (n, asm_addr); } +#ifndef CONFIG_X86_LOCAL_APIC +/* +* Used when local APIC is not compiled into the kernel, but +* external_interrupt() needs dispatch_spurious_interrupt(). +*/ +DEFINE_IDTENTRY_IRQ (spurious_interrupt) +{ +pr_info ("Spurious interrupt (vector 0x%x) on CPU#%d, should never happen.\n" , + vector,smp_processor_id ()); +} +#endif + +/* +* External interrupt dispatch function. +* +* Until/unless dispatch_common_interrupt() can be taught to deal with the +* special system vectors, split the dispatch. +* +* Note: dispatch_common_interrupt() already deals with IRQ_MOVE_CLEANUP_VECTOR. +*/ +int external_interrupt (struct pt_regs *regs,unsigned int vector) +{ +unsigned int sysvec = vector -FIRST_SYSTEM_VECTOR ; + +if (vector <FIRST_EXTERNAL_VECTOR ) { +pr_err ("invalid external interrupt vector %d\n" , vector); +return -EINVAL ; + } + +if (sysvec <NR_SYSTEM_VECTORS ) { +if (system_interrupt_handlers[sysvec]) + system_interrupt_handlers[sysvec](regs); +else +dispatch_spurious_interrupt (regs, vector); + }else { +dispatch_common_interrupt (regs, vector); + } + +return 0 ; +} +void __init trap_init (void ) {/* Init cpu_entry_area before IST entries are set up */
Signature · 1 line
2.34.1
Peter Zijlstra
maintainerRe: [PATCH v5 04/34] x86/traps: add external_interrupt() to dispatch external interrupts
Quoted 24 lines · Xin Li
arch/x86/kernel/traps.c +24
-0+#ifndef CONFIG_X86_LOCAL_APIC +/* +* Used when local APIC is not compiled into the kernel, but +* external_interrupt() needs dispatch_spurious_interrupt(). +*/ +DEFINE_IDTENTRY_IRQ (spurious_interrupt) +{ +pr_info ("Spurious interrupt (vector 0x%x) on CPU#%d, should never happen.\n" , + vector,smp_processor_id ()); +} +#endif + +/* +* External interrupt dispatch function. +* +* Until/unless dispatch_common_interrupt() can be taught to deal with the +* special system vectors, split the dispatch. +* +* Note: dispatch_common_interrupt() already deals with IRQ_MOVE_CLEANUP_VECTOR. +*/ +int external_interrupt (struct pt_regs *regs,unsigned int vector) +{ +unsigned int sysvec = vector -FIRST_SYSTEM_VECTOR ; +
arch/x86/kernel/traps.c void __init install_system_interrupt_handler(unsigned int n, const void *asm_add · line 1535
Quoted 10 lines
arch/x86/kernel/traps.c +10
-0+if (vector <FIRST_EXTERNAL_VECTOR ) { +pr_err ("invalid external interrupt vector %d\n" , vector); +return -EINVAL ; + } + +if (sysvec <NR_SYSTEM_VECTORS ) { +if (system_interrupt_handlers[sysvec]) + system_interrupt_handlers[sysvec](regs); +else +dispatch_spurious_interrupt (regs, vector);
ISTR suggesting you can get rid of this branch if you stuff system_interrupt_handlers[] with dispatch_spurious_interrupt instead of NULL.
Quoted 6 lines
arch/x86/kernel/traps.c +6
-0+ }else { +dispatch_common_interrupt (regs, vector); + } + +return 0 ; +}
Peter Zijlstra
maintainerRe: [PATCH v5 04/34] x86/traps: add external_interrupt() to dispatch external interrupts
Quoted 32 lines · Peter Zijlstra
Quoted 30 lines · Xin Li
arch/x86/kernel/traps.c +30
-0+#ifndef CONFIG_X86_LOCAL_APIC +/* +* Used when local APIC is not compiled into the kernel, but +* external_interrupt() needs dispatch_spurious_interrupt(). +*/ +DEFINE_IDTENTRY_IRQ (spurious_interrupt) +{ +pr_info ("Spurious interrupt (vector 0x%x) on CPU#%d, should never happen.\n" , + vector,smp_processor_id ()); +} +#endif + +/* +* External interrupt dispatch function. +* +* Until/unless dispatch_common_interrupt() can be taught to deal with the +* special system vectors, split the dispatch. +* +* Note: dispatch_common_interrupt() already deals with IRQ_MOVE_CLEANUP_VECTOR. +*/ +int external_interrupt (struct pt_regs *regs,unsigned int vector) +{ +unsigned int sysvec = vector -FIRST_SYSTEM_VECTOR ; + +if (vector <FIRST_EXTERNAL_VECTOR ) { +pr_err ("invalid external interrupt vector %d\n" , vector); +return -EINVAL ; + } + +if (sysvec <NR_SYSTEM_VECTORS ) {
Quoted 8 lines
arch/x86/kernel/traps.c void __init install_system_interrupt_handler(unsigned int n, const void *asm_add · line 1535
Quoted 4 lines
arch/x86/kernel/traps.c +4
-0+if (system_interrupt_handlers[sysvec]) + system_interrupt_handlers[sysvec](regs); +else +dispatch_spurious_interrupt (regs, vector);ISTR suggesting you can get rid of this branch if you stuff system_interrupt_handlers[] with
dispatch_spurious_interruptinstead of NULL.
Ah, I suggested that for another function vector, but it applies here too I suppose :-)
RE: [PATCH v5 04/34] x86/traps: add external_interrupt() to dispatch external interrupts
Quoted 11 lines
Quoted 9 lines
arch/x86/kernel/traps.c void __init install_system_interrupt_handler(unsigned int n, const void *asm_add · line 1535
Quoted 5 lines
arch/x86/kernel/traps.c +5
-0+if (sysvec <NR_SYSTEM_VECTORS ) { +if (system_interrupt_handlers[sysvec]) + system_interrupt_handlers[sysvec](regs); +else +dispatch_spurious_interrupt (regs, vector);ISTR suggesting you can get rid of this branch if you stuff system_interrupt_handlers[] with
dispatch_spurious_interruptinstead of NULL.Ah, I suggested that for another function vector, but it applies here too I suppose :-)
Of course!
We just need to use a wrapper as dispatch_spurious_interrupt() takes an extra parameter "vector".
Thanks!
Xin
RE: [PATCH v5 04/34] x86/traps: add external_interrupt() to dispatch external interrupts
In reply to Peter ZijlstraQuoted 8 lines
arch/x86/kernel/traps.c void __init install_system_interrupt_handler(unsigned int n, const void *asm_add · line 1535
Quoted 5 lines
arch/x86/kernel/traps.c +5
-0+if (sysvec <NR_SYSTEM_VECTORS ) { +if (system_interrupt_handlers[sysvec]) + system_interrupt_handlers[sysvec](regs); +else +dispatch_spurious_interrupt (regs, vector);ISTR suggesting you can get rid of this branch if you stuff system_interrupt_handlers[] with
dispatch_spurious_interruptinstead of NULL.
You're right, however I only fixed one.
PATCH v5 5/34 x86/traps: export external_interrupt() for VMX IRQ reinjection
In reply to Xin LiTo eliminate dispatching IRQ through the IDT, export external_interrupt() for VMX IRQ reinjection.
--- arch/x86/include/asm/traps.h | 2 ++ arch/x86/kernel/traps.c | 14 ++++++++++++++ 2 files changed, 16 insertions(+)
arch/x86/include/asm/traps.h +2 -0
@@ -56,4 +56,6 @@ void __noreturn handle_stack_overflow(struct pt_regs *regs, void f (struct pt_regs *regs) typedef DECLARE_SYSTEM_INTERRUPT_HANDLER((*system_interrupt_handler)); +int external_interrupt (struct pt_regs *regs,unsigned int vector); +#endif /* _ASM_X86_TRAPS_H */
arch/x86/kernel/traps.c +14 -0
@@ -1540,6 +1540,20 @@ int external_interrupt(struct pt_regs *regs, unsigned int vector)return 0 ; } +#if IS_ENABLED (CONFIG_KVM_INTEL ) +/* +* KVM VMX reinjects IRQ on its current stack, it's a sync call +* thus the values in the pt_regs structure are not used in +* executing IRQ handlers, except cs.RPL and flags.IF, which +* are both always 0 in the VMX IRQ reinjection context. +* +* However, the pt_regs structure is sometimes used in stack +* dump, e.g., show_regs(). So let the caller, i.e., KVM VMX +* decide how to initialize the input pt_regs structure. +*/ +EXPORT_SYMBOL_GPL (external_interrupt); +#endif +void __init trap_init (void ) {/* Init cpu_entry_area before IST entries are set up */
Signature · 1 line
2.34.1
Sean Christopherson
maintainerRe: [PATCH v5 05/34] x86/traps: export external_interrupt() for VMX IRQ reinjection
Quoted 32 lines · Xin Li
To eliminate dispatching IRQ through the IDT, export
external_interrupt()for VMX IRQ reinjection.Tested-by Shan Kang <[email protected]>Signed-off-by Xin Li <[email protected]>--- arch/x86/include/asm/traps.h | 2 ++ arch/x86/kernel/traps.c | 14 ++++++++++++++ 2 files changed, 16 insertions(+)arch/x86/include/asm/traps.h +2
-0@@ -56,4 +56,6 @@ void __noreturn handle_stack_overflow(struct pt_regs *regs, void f (struct pt_regs *regs) typedef DECLARE_SYSTEM_INTERRUPT_HANDLER((*system_interrupt_handler)); +int external_interrupt (struct pt_regs *regs,unsigned int vector); +#endif /* _ASM_X86_TRAPS_H */ arch/x86/kernel/traps.c +3
-0@@ -1540,6 +1540,20 @@ int external_interrupt(struct pt_regs *regs, unsigned int vector)return 0 ; } +#if IS_ENABLED (CONFIG_KVM_INTEL ) +/* +* KVM VMX reinjects IRQ on its current stack, it's a sync call
arch/x86/kernel/traps.c int external_interrupt(struct pt_regs *regs, unsigned int vector) · line 1555
Quoted 10 lines
arch/x86/kernel/traps.c +10
-0+* thus the values in the pt_regs structure are not used in +* executing IRQ handlers, except cs.RPL and flags.IF, which +* are both always 0 in the VMX IRQ reinjection context. +* +* However, the pt_regs structure is sometimes used in stack +* dump, e.g., show_regs(). So let the caller, i.e., KVM VMX +* decide how to initialize the input pt_regs structure. +*/ +EXPORT_SYMBOL_GPL (external_interrupt); +#endif
If the x86 maintainers don't object, I would prefer this to be squashed with the actual KVM usage, that way discussions on exactly what the exported API should be can be contained in a single thread.
RE: [PATCH v5 05/34] x86/traps: export external_interrupt() for VMX IRQ reinjection
Quoted 9 lines
Quoted 9 lines
arch/x86/kernel/traps.c +9
-0+#if IS_ENABLED (CONFIG_KVM_INTEL ) +/* +* KVM VMX reinjects IRQ on its current stack, it's a sync call +* thus the values in the pt_regs structure are not used in +* executing IRQ handlers, except cs.RPL and flags.IF, which +* are both always 0 in the VMX IRQ reinjection context. +* +* However, the pt_regs structure is sometimes used in stack +* dump, e.g., show_regs(). So let the caller, i.e., KVM VMX
Quoted 8 lines
arch/x86/kernel/traps.c int external_interrupt(struct pt_regs *regs, unsigned int vector) · line 1555
Quoted 4 lines
+ * decide how to initialize the input
pt_regsstructure.
+ */arch/x86/kernel/traps.c +2
-0+EXPORT_SYMBOL_GPL (external_interrupt); +#endif If the x86 maintainers don't object, I would prefer this to be squashed with the actual KVM usage, that way discussions on exactly what the exported API should be can be contained in a single thread.
The KVM usage is the only one now, thus it does make sense to squash into one.
I'm working on v6 and will merge this patch into the corresponding KVM patch. BTW, I will stop using "reinject" as asked.
PATCH v5 6/34 x86/cpufeature: add the cpu feature bit for FRED
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Add the CPU feature bit for FRED (Flexible Return and Event Delivery).
The Intel flexible return and event delivery (FRED) architecture defines simple new transitions that change privilege level (ring transitions). The FRED architecture was designed with the following goals:
1) Improve overall performance and response time by replacing event delivery through the interrupt descriptor table (IDT event delivery) and event return by the IRET instruction with lower latency transitions.
2) Improve software robustness by ensuring that event delivery establishes the full supervisor context and that event return establishes the full user context.
The new transitions defined by the FRED architecture are FRED event delivery and, for returning from events, two FRED return instructions. FRED event delivery can effect a transition from ring 3 to ring 0, but it is used also to deliver events incident to ring 0. One FRED instruction (ERETU) effects a return from ring 0 to ring 3, while the other (ERETS) returns while remaining in ring 0.
Search for the latest FRED spec in most search engines with this search pattern:
site:intel.com FRED (flexible return and event delivery) specification
--- arch/x86/include/asm/cpufeatures.h | 1 + tools/arch/x86/include/asm/cpufeatures.h | 1 + 2 files changed, 2 insertions(+)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -318,6 +318,7 @@#define X86_FEATURE_FZRM (12*32+10)/* "" Fast zero-length REP MOVSB */ #define X86_FEATURE_FSRS (12*32+11)/* "" Fast short REP STOSB */ #define X86_FEATURE_FSRC (12*32+12)/* "" Fast short REP {CMPSB,SCASB} */ +#define X86_FEATURE_FRED (12*32+17)/* Flexible Return and Event Delivery */ #define X86_FEATURE_LKGS (12*32+18)/* "" Load "kernel" (userspace) GS */ #define X86_FEATURE_AMX_FP16 (12*32+21)/* "" AMX fp16 Support */ #define X86_FEATURE_AVX_IFMA (12*32+23)/* "" Support for VPMADD52[H,L]UQ */
tools/arch/x86/include/asm/cpufeatures.h +1 -0
@@ -312,6 +312,7 @@#define X86_FEATURE_AVX_VNNI (12*32+ 4)/* AVX VNNI instructions */ #define X86_FEATURE_AVX512_BF16 (12*32+ 5)/* AVX512 BFLOAT16 instructions */ #define X86_FEATURE_CMPCCXADD (12*32+ 7)/* "" CMPccXADD instructions */ +#define X86_FEATURE_FRED (12*32+17)/* Flexible Return and Event Delivery */ #define X86_FEATURE_LKGS (12*32+18)/* "" Load "kernel" (userspace) GS */ #define X86_FEATURE_AMX_FP16 (12*32+21)/* "" AMX fp16 Support */ #define X86_FEATURE_AVX_IFMA (12*32+23)/* "" Support for VPMADD52[H,L]UQ */
Signature · 1 line
2.34.1
PATCH v5 7/34 x86/opcode: add ERETU, ERETS instructions to x86-opcode-map
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Add the instruction opcodes used by FRED: ERETU, ERETS.
Opcode number is per public FRED draft spec v3.0.
--- arch/x86/lib/x86-opcode-map.txt | 2 +- tools/arch/x86/lib/x86-opcode-map.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/lib/x86-opcode-map.txt +1 -1
@@ -1052,7 +1052,7 @@ EndTable GrpTable: Grp7 0: SGDT Ms | VMCALL (001),(11B) | VMLAUNCH (010),(11B) | VMRESUME (011),(11B) | VMXOFF (100),(11B) | PCONFIG (101),(11B) | ENCLV (000),(11B) -1: SIDT Ms | MONITOR (000),(11B) | MWAIT (001),(11B) | CLAC (010),(11B) | STAC (011),(11B) | ENCLS (111),(11B) +1: SIDT Ms | MONITOR (000),(11B) | MWAIT (001),(11B) | CLAC (010),(11B) | STAC (011),(11B) | ENCLS (111),(11B) | ERETU (F3),(010),(11B) | ERETS (F2),(010),(11B) 2: LGDT Ms | XGETBV (000),(11B) | XSETBV (001),(11B) | VMFUNC (100),(11B) | XEND (101)(11B) | XTEST (110)(11B) | ENCLU (111),(11B) 3: LIDT Ms 4: SMSW Mw/Rv
tools/arch/x86/lib/x86-opcode-map.txt +1 -1
@@ -1052,7 +1052,7 @@ EndTable GrpTable: Grp7 0: SGDT Ms | VMCALL (001),(11B) | VMLAUNCH (010),(11B) | VMRESUME (011),(11B) | VMXOFF (100),(11B) | PCONFIG (101),(11B) | ENCLV (000),(11B) -1: SIDT Ms | MONITOR (000),(11B) | MWAIT (001),(11B) | CLAC (010),(11B) | STAC (011),(11B) | ENCLS (111),(11B) +1: SIDT Ms | MONITOR (000),(11B) | MWAIT (001),(11B) | CLAC (010),(11B) | STAC (011),(11B) | ENCLS (111),(11B) | ERETU (F3),(010),(11B) | ERETS (F2),(010),(11B) 2: LGDT Ms | XGETBV (000),(11B) | XSETBV (001),(11B) | VMFUNC (100),(11B) | XEND (101)(11B) | XTEST (110)(11B) | ENCLU (111),(11B) 3: LIDT Ms 4: SMSW Mw/Rv
Signature · 1 line
2.34.1
PATCH v5 8/34 x86/objtool: teach objtool about ERETU and ERETS
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Update the objtool decoder to know about the ERETU and ERETS instructions (type INSN_CONTEXT_SWITCH.)
--- tools/objtool/arch/x86/decode.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)
tools/objtool/arch/x86/decode.c +14 -5
@@ -509,11 +509,20 @@ int arch_decode_instruction(struct objtool_file *file, const struct section *secif (op2 ==0x01 ) { -if (modrm ==0xca ) - insn->type =INSN_CLAC ; -else if (modrm ==0xcb ) - insn->type =INSN_STAC ; - +switch (insn_last_prefix_id (&ins)) { +case INAT_PFX_REPE : +case INAT_PFX_REPNE : +if (modrm ==0xca ) +/* eretu/erets */ + insn->type =INSN_CONTEXT_SWITCH ; +break ; +default : +if (modrm ==0xca ) + insn->type =INSN_CLAC ; +else if (modrm ==0xcb ) + insn->type =INSN_STAC ; +break ; + } }else if (op2 >=0x80 && op2 <=0x8f ) { insn->type =INSN_JUMP_CONDITIONAL ;
Signature · 1 line
2.34.1
PATCH v5 9/34 x86/cpu: add X86_CR4_FRED macro
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Add X86_CR4_FRED macro for the FRED bit in %cr4. This bit should be a pinned bit, not to be changed after initialization.
--- arch/x86/include/uapi/asm/processor-flags.h | 2 ++ arch/x86/kernel/cpu/common.c | 11 ++++++++--- 2 files changed, 10 insertions(+), 3 deletions(-)
arch/x86/include/uapi/asm/processor-flags.h +2 -0
@@ -132,6 +132,8 @@#define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT)#define X86_CR4_CET_BIT 23/* enable Control-flow Enforcement Technology */ #define X86_CR4_CET _BITUL(X86_CR4_CET_BIT) +#define X86_CR4_FRED_BIT 32/* enable FRED kernel entry */ +#define X86_CR4_FRED _BITULL(X86_CR4_FRED_BIT)/* * x86-64 Task Priority Register, CR8
arch/x86/kernel/cpu/common.c +8 -3
@@ -412,10 +412,15 @@ static __always_inline void setup_umip(struct cpuinfo_x86 *c)cr4_clear_bits (X86_CR4_UMIP ); } -/* These bits should not change their value after CPU init is finished. */ +/* +* These bits should not change their value after CPU init is finished. +* The explicit cast to unsigned long suppresses a warning on i386 for +* x86-64 only feature bits >= 32. +*/ static const unsigned long cr4_pinned_mask = -X86_CR4_SMEP |X86_CR4_SMAP |X86_CR4_UMIP | -X86_CR4_FSGSBASE |X86_CR4_CET ; + (unsigned long ) + (X86_CR4_SMEP |X86_CR4_SMAP |X86_CR4_UMIP | +X86_CR4_FSGSBASE |X86_CR4_CET |X86_CR4_FRED );static DEFINE_STATIC_KEY_FALSE_RO (cr_pinning);static unsigned long cr4_pinned_bits__ro_after_init ;
Signature · 1 line
2.34.1
PATCH v5 10/34 x86/fred: add Kconfig option for FRED (CONFIG_X86_FRED)
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Add the configuration option CONFIG_X86_FRED to enable FRED.
--- arch/x86/Kconfig | 9 +++++++++ 1 file changed, 9 insertions(+)
arch/x86/Kconfig +9 -0
@@ -500,6 +500,15 @@ config X86_CPU_RESCTRL Say N if unsure. +config X86_FRED + bool "Flexible Return and Event Delivery" + depends on X86_64 + help + When enabled, try to use Flexible Return and Event Delivery + instead of the legacy SYSCALL/SYSENTER/IDT architecture for + ring transitions and exception/interrupt handling if the + system supports. + if X86_32 config X86_BIGSMP bool "Support for big SMP systems with more than 8 CPUs"
Signature · 1 line
2.34.1
PATCH v5 11/34 x86/fred: if CONFIG_X86_FRED is disabled, disable FRED support
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Add CONFIG_X86_FRED to <asm/disabled-features.h> to make cpu_feature_enabled() work correctly with FRED.
--- arch/x86/include/asm/disabled-features.h | 8 +++++++- tools/arch/x86/include/asm/disabled-features.h | 8 +++++++- 2 files changed, 14 insertions(+), 2 deletions(-)
arch/x86/include/asm/disabled-features.h +7 -1
@@ -99,6 +99,12 @@# define DISABLE_TDX_GUEST (1 << (X86_FEATURE_TDX_GUEST & 31))#endif +#ifdef CONFIG_X86_FRED +# define DISABLE_FRED 0 +#else +# define DISABLE_FRED (1 << (X86_FEATURE_FRED & 31)) +#endif +/* * Make sure to add features to the correct mask */
@@ -115,7 +121,7 @@#define DISABLED_MASK10 0#define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ DISABLE_CALL_DEPTH_TRACKING) -#define DISABLED_MASK12 0 +#define DISABLED_MASK12 (DISABLE_FRED)#define DISABLED_MASK13 0#define DISABLED_MASK14 0#define DISABLED_MASK15 0
tools/arch/x86/include/asm/disabled-features.h +7 -1
@@ -99,6 +99,12 @@# define DISABLE_TDX_GUEST (1 << (X86_FEATURE_TDX_GUEST & 31))#endif +#ifdef CONFIG_X86_FRED +# define DISABLE_FRED 0 +#else +# define DISABLE_FRED (1 << (X86_FEATURE_FRED & 31)) +#endif +/* * Make sure to add features to the correct mask */
@@ -115,7 +121,7 @@#define DISABLED_MASK10 0#define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ DISABLE_CALL_DEPTH_TRACKING) -#define DISABLED_MASK12 0 +#define DISABLED_MASK12 (DISABLE_FRED)#define DISABLED_MASK13 0#define DISABLED_MASK14 0#define DISABLED_MASK15 0
Signature · 1 line
2.34.1
PATCH v5 12/34 x86/cpu: add MSR numbers for FRED configuration
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Add MSR numbers for the FRED configuration registers.
--- arch/x86/include/asm/msr-index.h | 13 ++++++++++++- tools/arch/x86/include/asm/msr-index.h | 13 ++++++++++++- 2 files changed, 24 insertions(+), 2 deletions(-)
arch/x86/include/asm/msr-index.h +12 -1
@@ -36,8 +36,19 @@#define EFER_FFXSR (1<<_EFER_FFXSR)#define EFER_AUTOIBRS (1<<_EFER_AUTOIBRS) -/* Intel MSRs. Some also available on other CPUs */ +/* FRED MSRs */ +#define MSR_IA32_FRED_RSP0 0x1cc/* Level 0 stack pointer */ +#define MSR_IA32_FRED_RSP1 0x1cd/* Level 1 stack pointer */ +#define MSR_IA32_FRED_RSP2 0x1ce/* Level 2 stack pointer */ +#define MSR_IA32_FRED_RSP3 0x1cf/* Level 3 stack pointer */ +#define MSR_IA32_FRED_STKLVLS 0x1d0/* Exception stack levels */ +#define MSR_IA32_FRED_SSP0 MSR_IA32_PL0_SSP/* Level 0 shadow stack pointer */ +#define MSR_IA32_FRED_SSP1 0x1d1/* Level 1 shadow stack pointer */ +#define MSR_IA32_FRED_SSP2 0x1d2/* Level 2 shadow stack pointer */ +#define MSR_IA32_FRED_SSP3 0x1d3/* Level 3 shadow stack pointer */ +#define MSR_IA32_FRED_CONFIG 0x1d4/* Entrypoint and interrupt stack level */ +/* Intel MSRs. Some also available on other CPUs */ #define MSR_TEST_CTRL 0x00000033#define MSR_TEST_CTRL_SPLIT_LOCK_DETECT_BIT 29#define MSR_TEST_CTRL_SPLIT_LOCK_DETECT BIT(MSR_TEST_CTRL_SPLIT_LOCK_DETECT_BIT)
tools/arch/x86/include/asm/msr-index.h +12 -1
@@ -34,8 +34,19 @@#define EFER_LMSLE (1<<_EFER_LMSLE)#define EFER_FFXSR (1<<_EFER_FFXSR) -/* Intel MSRs. Some also available on other CPUs */ +/* FRED MSRs */ +#define MSR_IA32_FRED_RSP0 0x1cc/* Level 0 stack pointer */ +#define MSR_IA32_FRED_RSP1 0x1cd/* Level 1 stack pointer */ +#define MSR_IA32_FRED_RSP2 0x1ce/* Level 2 stack pointer */ +#define MSR_IA32_FRED_RSP3 0x1cf/* Level 3 stack pointer */ +#define MSR_IA32_FRED_STKLVLS 0x1d0/* Exception stack levels */ +#define MSR_IA32_FRED_SSP0 MSR_IA32_PL0_SSP/* Level 0 shadow stack pointer */ +#define MSR_IA32_FRED_SSP1 0x1d1/* Level 1 shadow stack pointer */ +#define MSR_IA32_FRED_SSP2 0x1d2/* Level 2 shadow stack pointer */ +#define MSR_IA32_FRED_SSP3 0x1d3/* Level 3 shadow stack pointer */ +#define MSR_IA32_FRED_CONFIG 0x1d4/* Entrypoint and interrupt stack level */ +/* Intel MSRs. Some also available on other CPUs */ #define MSR_TEST_CTRL 0x00000033#define MSR_TEST_CTRL_SPLIT_LOCK_DETECT_BIT 29#define MSR_TEST_CTRL_SPLIT_LOCK_DETECT BIT(MSR_TEST_CTRL_SPLIT_LOCK_DETECT_BIT)
Signature · 1 line
2.34.1
PATCH v5 13/34 x86/fred: header file for event types
In reply to Xin LiFRED inherits the Intel VT-x enhancement of classified events with a two-level event dispatch logic. The first-level dispatch is on the event type, not the event vector as used in the IDT architecture. This also means that vectors in different event types are orthogonal, e.g., vectors 0x10-0x1f become available as hardware interrupts.
Add a header file for event types, and also use it in <asm/vmx.h>.
--- arch/x86/include/asm/event-type.h | 17 +++++++++++++++++ arch/x86/include/asm/vmx.h | 17 +++++++++-------- 2 files changed, 26 insertions(+), 8 deletions(-) create mode 100644 arch/x86/include/asm/event-type.h
arch/x86/include/asm/event-type.h new file +17 -0
@@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_EVENT_TYPE_H +#define _ASM_X86_EVENT_TYPE_H + +/* +* Event type codes: these are the same that are used by VTx. +*/ +#define EVENT_TYPE_HWINT 0/* Maskable external interrupt */ +#define EVENT_TYPE_RESERVED 1 +#define EVENT_TYPE_NMI 2/* Non-maskable interrupt */ +#define EVENT_TYPE_HWFAULT 3/* Hardware exceptions (e.g., page fault) */ +#define EVENT_TYPE_SWINT 4/* Software interrupt (INT n) */ +#define EVENT_TYPE_PRIVSW 5/* INT1 (ICEBP) */ +#define EVENT_TYPE_SWFAULT 6/* Software exception (INT3 or INTO) */ +#define EVENT_TYPE_OTHER 7/* FRED: SYSCALL/SYSENTER */ + +#endif
arch/x86/include/asm/vmx.h +9 -8
@@ -15,6 +15,7 @@#include <linux/bitops.h> #include <linux/types.h> #include <uapi/asm/vmx.h> +#include <asm/event-type.h> #include <asm/vmxfeatures.h> #define VMCS_CONTROL_BIT (x) BIT(VMX_FEATURE_##x & 0x1f)
@@ -372,14 +373,14 @@ enum vmcs_field {#define VECTORING_INFO_DELIVER_CODE_MASK INTR_INFO_DELIVER_CODE_MASK#define VECTORING_INFO_VALID_MASK INTR_INFO_VALID_MASK -#define INTR_TYPE_EXT_INTR (0 << 8)/* external interrupt */ -#define INTR_TYPE_RESERVED (1 << 8)/* reserved */ -#define INTR_TYPE_NMI_INTR (2 << 8)/* NMI */ -#define INTR_TYPE_HARD_EXCEPTION (3 << 8)/* processor exception */ -#define INTR_TYPE_SOFT_INTR (4 << 8)/* software interrupt */ -#define INTR_TYPE_PRIV_SW_EXCEPTION (5 << 8)/* ICE breakpoint - undocumented */ -#define INTR_TYPE_SOFT_EXCEPTION (6 << 8)/* software exception */ -#define INTR_TYPE_OTHER_EVENT (7 << 8)/* other event */ +#define INTR_TYPE_EXT_INTR (EVENT_TYPE_HWINT << 8)/* external interrupt */ +#define INTR_TYPE_RESERVED (EVENT_TYPE_RESERVED << 8)/* reserved */ +#define INTR_TYPE_NMI_INTR (EVENT_TYPE_NMI << 8)/* NMI */ +#define INTR_TYPE_HARD_EXCEPTION (EVENT_TYPE_HWFAULT << 8)/* processor exception */ +#define INTR_TYPE_SOFT_INTR (EVENT_TYPE_SWINT << 8)/* software interrupt */ +#define INTR_TYPE_PRIV_SW_EXCEPTION (EVENT_TYPE_PRIVSW << 8)/* ICE breakpoint - undocumented */ +#define INTR_TYPE_SOFT_EXCEPTION (EVENT_TYPE_SWFAULT << 8)/* software exception */ +#define INTR_TYPE_OTHER_EVENT (EVENT_TYPE_OTHER << 8)/* other event */ /* GUEST_INTERRUPTIBILITY_INFO flags. */ #define GUEST_INTR_STATE_STI 0x00000001
Signature · 1 line
2.34.1
PATCH v5 14/34 x86/fred: header file with FRED definitions
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Add a header file for FRED prototypes and definitions.
--- arch/x86/include/asm/fred.h | 101 ++++++++++++++++++++++++++++++++++++ 1 file changed, 101 insertions(+) create mode 100644 arch/x86/include/asm/fred.h
arch/x86/include/asm/fred.h new file +101 -0
@@ -0,0 +1,101 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* arch/x86/include/asm/fred.h +* +* Macros for Flexible Return and Event Delivery (FRED) +*/ + +#ifndef ASM_X86_FRED_H +#define ASM_X86_FRED_H + +#ifdef CONFIG_X86_FRED + +#include <linux/const.h> +#include <asm/asm.h> + +/* +* FRED return instructions +* +* Replace with "ERETS"/"ERETU" once binutils support FRED return instructions. +* The binutils version supporting FRED instructions is still TBD, and will +* update once we have it. +*/ +#define ERETS _ASM_BYTES(0xf2,0x0f,0x01,0xca) +#define ERETU _ASM_BYTES(0xf3,0x0f,0x01,0xca) + +/* +* Event stack level macro for the FRED_STKLVLS MSR. +* Usage example: FRED_STKLVL(X86_TRAP_DF, 3) +* Multiple values can be ORd together. +*/ +#define FRED_STKLVL (v,l) (_AT(unsigned long, l) << (2*(v))) + +/* FRED_CONFIG MSR */ +#define FRED_CONFIG_CSL_MASK 0x3 +#define FRED_CONFIG_SHADOW_STACK_SPACE _BITUL(3) +#define FRED_CONFIG_REDZONE (b) __ALIGN_KERNEL_MASK((b), _UL(0x3f)) +#define FRED_CONFIG_INT_STKLVL (l) (_AT(unsigned long, l) << 9) +#define FRED_CONFIG_ENTRYPOINT (p) _AT(unsigned long, (p)) + +/* FRED event type and vector bit width and counts */ +#define FRED_EVENT_TYPE_BITS 3/* only 3 bits used in FRED 3.0 */ +#define FRED_EVENT_TYPE_COUNT _BITUL(FRED_EVENT_TYPE_BITS) +#define FRED_EVENT_VECTOR_BITS 8 +#define FRED_EVENT_VECTOR_COUNT _BITUL(FRED_EVENT_VECTOR_BITS) + +/* FRED EVENT_TYPE_OTHER vector numbers */ +#define FRED_SYSCALL 1 +#define FRED_SYSENTER 2 + +/* Flags above the CS selector (regs->csx) */ +#define FRED_CSL_ENABLE_NMI _BITUL(28) +#define FRED_CSL_ALLOW_SINGLE_STEP _BITUL(25) +#define FRED_CSL_INTERRUPT_SHADOW _BITUL(24) + +#ifndef __ASSEMBLY__ + +#include <linux/kernel.h> +#include <asm/ptrace.h> + +/* FRED stack frame information */ +struct fred_info { +unsigned long edata;/* Event data: CR2, DR6, ... */ +unsigned long resv; +}; + +/* Full format of the FRED stack frame */ +struct fred_frame { +struct pt_regs regs; +struct fred_info info; +}; + +/* Getting the FRED frame information from a pt_regs pointer */ +static __always_inline struct fred_info *fred_info(struct pt_regs *regs) +{ + return &container_of(regs, struct fred_frame, regs)->info; +} + +static __always_inline unsigned long fred_event_data (struct pt_regs *regs) +{ +return fred_info (regs)->edata; +} + +/* +* How FRED event handlers are called. +* +* FRED event delivery establishes the full supervisor context +* by pushing everything related to the event being delivered +* to the FRED stack frame, e.g., the faulting linear address +* of a #PF is pushed as event data of the FRED #PF stack frame. +* Thus a struct pt_regs has everything needed and it's the only +* input parameter required for a FRED event handler. +*/ +#define DECLARE_FRED_HANDLER (f) void f (struct pt_regs *regs) +#define DEFINE_FRED_HANDLER (f) noinstr DECLARE_FRED_HANDLER(f) +typedef DECLARE_FRED_HANDLER ((*fred_handler )); + +#endif /* __ASSEMBLY__ */ + +#endif /* CONFIG_X86_FRED */ + +#endif /* ASM_X86_FRED_H */
Signature · 1 line
2.34.1
PATCH v5 15/34 x86/fred: make unions for the cs and ss fields in struct pt_regs
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Make the cs and ss fields in struct pt_regs unions between the actual selector and the unsigned long stack slot. FRED uses this space to store additional flags.
The printk changes are simply due to the cs and ss fields changed to unsigned short from unsigned long.
---
Changes since v3:
* Rename csl/ssl of the pt_regs structure to csx/ssx (x for extended) (Andrew Cooper).
--- arch/x86/entry/vsyscall/vsyscall_64.c | 2 +- arch/x86/include/asm/ptrace.h | 36 ++++++++++++++++++++++++--- arch/x86/kernel/process_64.c | 2 +- 3 files changed, 34 insertions(+), 6 deletions(-)
arch/x86/entry/vsyscall/vsyscall_64.c +1 -1
@@ -76,7 +76,7 @@ static void warn_bad_vsyscall(const char *level, struct pt_regs *regs,if (!show_unhandled_signals)return ; -printk_ratelimited ("%s%s[%d] %s ip:%lx cs:%lx sp:%lx ax:%lx si:%lx di:%lx\n" , +printk_ratelimited ("%s%s[%d] %s ip:%lx cs:%x sp:%lx ax:%lx si:%lx di:%lx\n" , level, current->comm,task_pid_nr (current), message, regs->ip, regs->cs, regs->sp, regs->ax, regs->si, regs->di);
arch/x86/include/asm/ptrace.h +32 -4
@@ -82,13 +82,41 @@ struct pt_regs {* On hw interrupt, it's IRQ number: */ unsigned long orig_ax; -/* Return frame for iretq */ + +/* Return frame for iretq/eretu/erets */ unsigned long ip; -unsigned long cs; +union { +unsigned long csx;/* cs extended: CS + any fields above it */ +struct __attribute__((__packed__ )) { +unsigned short cs;/* CS selector proper */ +unsigned int current_stack_level:2 ; +unsigned int __csx_resv1 :6 ; +unsigned int interrupt_shadowed :1 ; +unsigned int software_initiated :1 ; +unsigned int __csx_resv2 :2 ; +unsigned int nmi :1 ; +unsigned int __csx_resv3 :3 ; +unsigned int __csx_resv4 :32 ; + }; + };unsigned long flags;unsigned long sp; -unsigned long ss; -/* top of stack page */ +union { +unsigned long ssx;/* ss extended: SS + any fields above it */ +struct __attribute__((__packed__ )) { +unsigned short ss;/* SS selector proper */ +unsigned int __ssx_resv1 :16 ; +unsigned int vector :8 ; +unsigned int __ssx_resv2 :8 ; +unsigned int type :4 ; +unsigned int __ssx_resv3 :4 ; +unsigned int enclv :1 ; +unsigned int long_mode :1 ; +unsigned int nested :1 ; +unsigned int __ssx_resv4 :1 ; +unsigned int instr_len :4 ; + }; + }; };#endif /* !__i386__ */
arch/x86/kernel/process_64.c +1 -1
@@ -116,7 +116,7 @@ void __show_regs(struct pt_regs *regs, enum show_regs_mode mode,printk ("%sFS: %016lx(%04x) GS:%016lx(%04x) knlGS:%016lx\n" , log_lvl, fs, fsindex, gs, gsindex, shadowgs); -printk ("%sCS: %04lx DS: %04x ES: %04x CR0: %016lx\n" , +printk ("%sCS: %04x DS: %04x ES: %04x CR0: %016lx\n" , log_lvl, regs->cs, ds, es, cr0);printk ("%sCR2: %016lx CR3: %016lx CR4: %016lx\n" , log_lvl, cr2, cr3, cr4);
Signature · 1 line
2.34.1
PATCH v5 16/34 x86/fred: reserve space for the FRED stack frame
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
When using FRED, reserve space at the top of the stack frame, just like i386 does. A future version of FRED might have dynamic frame sizes, though, in which case it might be necessary to make TOP_OF_KERNEL_STACK_PADDING a variable instead of a constant.
--- arch/x86/include/asm/thread_info.h | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
arch/x86/include/asm/thread_info.h +9 -3
@@ -31,7 +31,9 @@* In vm86 mode, the hardware frame is much longer still, so add 16 * bytes to make room for the real-mode segments. * -* x86_64 has a fixed-length stack frame. +* x86-64 has a fixed-length stack frame, but it depends on whether +* or not FRED is enabled. Future versions of FRED might make this +* dynamic, but for now it is always 2 words longer. */ #ifdef CONFIG_X86_32 # ifdef CONFIG_VM86
@@ -39,8 +41,12 @@# else # define TOP_OF_KERNEL_STACK_PADDING 8# endif -#else -# define TOP_OF_KERNEL_STACK_PADDING 0 +#else /* x86-64 */ +# ifdef CONFIG_X86_FRED +# define TOP_OF_KERNEL_STACK_PADDING (2*8) +# else +# define TOP_OF_KERNEL_STACK_PADDING 0 +# endif #endif /*
Signature · 1 line
2.34.1
PATCH v5 17/34 x86/fred: add a page fault entry stub for FRED
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Add a page fault entry stub for FRED.
On a FRED system, the faulting address (CR2) is passed on the stack, to avoid the problem of transient state. Thus we get the page fault address from the stack instead of CR2.
--- arch/x86/include/asm/fred.h | 2 ++ arch/x86/mm/fault.c | 20 ++++++++++++++++++-- 2 files changed, 20 insertions(+), 2 deletions(-)
arch/x86/include/asm/fred.h +2 -0
@@ -94,6 +94,8 @@ static __always_inline unsigned long fred_event_data(struct pt_regs *regs)#define DEFINE_FRED_HANDLER (f) noinstr DECLARE_FRED_HANDLER(f)typedef DECLARE_FRED_HANDLER ((*fred_handler )); +DECLARE_FRED_HANDLER (fred_exc_page_fault); +#endif /* __ASSEMBLY__ */ #endif /* CONFIG_X86_FRED */
arch/x86/mm/fault.c +18 -2
@@ -33,6 +33,7 @@#include <asm/kvm_para.h> /* kvm_handle_async_pf */ #include <asm/vdso.h> /* fixup_vdso_exception() */ #include <asm/irq_stack.h> +#include <asm/fred.h> /* fred_event_data() */ #define CREATE_TRACE_POINTS #include <asm/trace/exceptions.h>
@@ -1507,9 +1508,10 @@ handle_page_fault(struct pt_regs *regs, unsigned long error_code, } } -DEFINE_IDTENTRY_RAW_ERRORCODE (exc_page_fault) +static __always_inline void page_fault_common(struct pt_regs *regs, +unsigned int error_code, +unsigned long address) { -unsigned long address =read_cr2 ();irqentry_state_t state;prefetchw (¤t->mm->mmap_lock);
@@ -1556,3 +1558,17 @@ DEFINE_IDTENTRY_RAW_ERRORCODE(exc_page_fault)irqentry_exit (regs, state); } + +DEFINE_IDTENTRY_RAW_ERRORCODE (exc_page_fault) +{ +page_fault_common (regs, error_code,read_cr2 ()); +} + +#ifdef CONFIG_X86_FRED + +DEFINE_FRED_HANDLER (fred_exc_page_fault) +{ +page_fault_common (regs, regs->orig_ax,fred_event_data (regs)); +} + +#endif /* CONFIG_X86_FRED */
Signature · 1 line
2.34.1
PATCH v5 18/34 x86/fred: add a debug fault entry stub for FRED
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Add a debug fault entry stub for FRED.
On a FRED system, the debug trap status information (DR6) is passed on the stack, to avoid the problem of transient state. Furthermore, FRED transitions avoid a lot of ugly corner cases the handling of which can, and should be, skipped.
The FRED debug trap status information saved on the stack differs from DR6 in both stickiness and polarity; it is exactly what debug_read_clear_dr6() returns, and exc_debug_user()/exc_debug_kernel() expect.
---
Changes since v1:
* call irqentry_nmi_{enter,exit}() in both IDT and FRED debug fault kernel handler (Peter Zijlstra).
--- arch/x86/include/asm/fred.h | 1 + arch/x86/kernel/traps.c | 56 +++++++++++++++++++++++++++---------- 2 files changed, 42 insertions(+), 15 deletions(-)
arch/x86/include/asm/fred.h +1 -0
@@ -94,6 +94,7 @@ static __always_inline unsigned long fred_event_data(struct pt_regs *regs)#define DEFINE_FRED_HANDLER (f) noinstr DECLARE_FRED_HANDLER(f)typedef DECLARE_FRED_HANDLER ((*fred_handler )); +DECLARE_FRED_HANDLER (fred_exc_debug);DECLARE_FRED_HANDLER (fred_exc_page_fault);#endif /* __ASSEMBLY__ */
arch/x86/kernel/traps.c +41 -15
88 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v5 19/34 x86/fred: add a NMI entry stub for FRED
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
On a FRED system, NMIs nest both with themselves and faults, transient information is saved into the stack frame, and NMI unblocking only happens when the stack frame indicates that so should happen.
Thus, the NMI entry stub for FRED is really quite small...
--- arch/x86/include/asm/fred.h | 1 + arch/x86/kernel/nmi.c | 28 ++++++++++++++++++++++++++++ 2 files changed, 29 insertions(+)
arch/x86/include/asm/fred.h +1 -0
@@ -94,6 +94,7 @@ static __always_inline unsigned long fred_event_data(struct pt_regs *regs)#define DEFINE_FRED_HANDLER (f) noinstr DECLARE_FRED_HANDLER(f)typedef DECLARE_FRED_HANDLER ((*fred_handler )); +DECLARE_FRED_HANDLER (fred_exc_nmi);DECLARE_FRED_HANDLER (fred_exc_debug);DECLARE_FRED_HANDLER (fred_exc_page_fault);
arch/x86/kernel/nmi.c +28 -0
@@ -34,6 +34,7 @@#include <asm/cache.h> #include <asm/nospec-branch.h> #include <asm/sev.h> +#include <asm/fred.h> #define CREATE_TRACE_POINTS #include <trace/events/nmi.h>
@@ -643,6 +644,33 @@ void nmi_backtrace_stall_check(const struct cpumask *btp)#endif +#ifdef CONFIG_X86_FRED +DEFINE_FRED_HANDLER (fred_exc_nmi) +{ +/* +* With FRED, CR2 and DR6 are pushed atomically on faults, +* so we don't have to worry about saving and restoring them. +* Breakpoint faults nest, so assume it is OK to leave DR7 +* enabled. +*/ +irqentry_state_t irq_state =irqentry_nmi_enter (regs); + +/* +* VM exits induced by NMIs keep NMI blocked, and we do +* "int $2" to reinject the NMI w/ NMI kept being blocked. +* However "int $2" doesn't set the nmi bit in the FRED +* stack frame, so we explicitly set it to make sure a +* later ERETS will unblock NMI immediately. +*/ + regs->nmi =1 ; + +inc_irq_stat (__nmi_count); +default_do_nmi (regs); + +irqentry_nmi_exit (regs, irq_state); +} +#endif +void stop_nmi (void ) { ignore_nmis++;
Signature · 1 line
2.34.1
PATCH v5 20/34 x86/fred: add a machine check entry stub for FRED
In reply to Xin LiAdd a machine check entry stub for FRED.
Unlike IDT, no need to save/restore dr7 in FRED machine check handler.
--- arch/x86/include/asm/fred.h | 1 + arch/x86/kernel/cpu/mce/core.c | 11 +++++++++++ 2 files changed, 12 insertions(+)
arch/x86/include/asm/fred.h +1 -0
@@ -97,6 +97,7 @@ typedef DECLARE_FRED_HANDLER((*fred_handler));DECLARE_FRED_HANDLER (fred_exc_nmi);DECLARE_FRED_HANDLER (fred_exc_debug);DECLARE_FRED_HANDLER (fred_exc_page_fault); +DECLARE_FRED_HANDLER (fred_exc_machine_check);#endif /* __ASSEMBLY__ */
arch/x86/kernel/cpu/mce/core.c +11 -0
@@ -52,6 +52,7 @@#include <asm/mce.h> #include <asm/msr.h> #include <asm/reboot.h> +#include <asm/fred.h> #include "internal.h"
@@ -2111,6 +2112,16 @@ DEFINE_IDTENTRY_MCE_USER(exc_machine_check)exc_machine_check_user (regs); local_db_restore(dr7); } + +#ifdef CONFIG_X86_FRED +DEFINE_FRED_HANDLER (fred_exc_machine_check) +{ +if (user_mode (regs)) +exc_machine_check_user (regs); +else +exc_machine_check_kernel (regs); +} +#endif #else /* 32bit unified entry point */ DEFINE_IDTENTRY_RAW (exc_machine_check)
Signature · 1 line
2.34.1
Peter Zijlstra
maintainerRe: [PATCH v5 20/34] x86/fred: add a machine check entry stub for FRED
Quoted 3 lines · Xin Li
Add a machine check entry stub for FRED.
Unlike IDT, no need to save/restore dr7 in FRED machine check handler.
Given how fragile MCE is, the question should be, do we ever want hw breakpoints to happen while it is running?
If the hw-breakpoint handler trips on the same memory fail that got us into the mce the first time, we're dead.
RE: [PATCH v5 20/34] x86/fred: add a machine check entry stub for FRED
Quoted 4 lines
Quoted 1 line
Unlike IDT, no need to save/restore dr7 in FRED machine check handler.
Given how fragile MCE is, the question should be, do we ever want hw breakpoints to happen while it is running?
HW breakpoints still work if they are properly configured.
Quoted 2 lines
If the hw-breakpoint handler trips on the same memory fail that got us into the mce the first time, we're dead.
Right.
Unless the MCIP bit is turned off any subsequent #MC goes to shutdown ("machine is screwed").
It's the kernel debugger's responsibility to decide how to proceed in such cases. But if the kernel debugger itself is in a screwed memory region, we are soooooo dead.
Thanks!
Xin
Peter Zijlstra
maintainerRe: [PATCH v5 20/34] x86/fred: add a machine check entry stub for FRED
Quoted 10 lines · On Tue, Mar 21, 2023 at 12:04:47AM +0000, Li, Xin3 wrote:
Quoted 4 lines
Quoted 1 line
Unlike IDT, no need to save/restore dr7 in FRED machine check handler.
Given how fragile MCE is, the question should be, do we ever want hw breakpoints to happen while it is running?
HW breakpoints still work if they are properly configured.
Quoted 2 lines
If the hw-breakpoint handler trips on the same memory fail that got us into the mce the first time, we're dead.
Quoted 8 lines
Right.
Unless the MCIP bit is turned off any subsequent #MC goes to shutdown ("machine is screwed").
It's the kernel debugger's responsibility to decide how to proceed in such cases. But if the kernel debugger itself is in a screwed memory region, we are soooooo dead.
Yeah, so I would much prefer, for robustness sake, to start out with not allowing #DB in MCE -- much like today.
RE: [PATCH v5 20/34] x86/fred: add a machine check entry stub for FRED
Quoted 16 lines · On Tue, Mar 21, 2023 at 12:04:47AM +0000, Li, Xin3 wrote:
Quoted 15 lines
Quoted 4 lines
Quoted 1 line
Unlike IDT, no need to save/restore dr7 in FRED machine check handler.
Given how fragile MCE is, the question should be, do we ever want hw breakpoints to happen while it is running?
HW breakpoints still work if they are properly configured.
Quoted 2 lines
If the hw-breakpoint handler trips on the same memory fail that got us into the mce the first time, we're dead.
Right.
Unless the MCIP bit is turned off any subsequent #MC goes to shutdown ("machine is screwed").
Quoted 6 lines
Quoted 3 lines
It's the kernel debugger's responsibility to decide how to proceed in such cases. But if the kernel debugger itself is in a screwed memory region, we are soooooo dead.
Yeah, so I would much prefer, for robustness sake, to start out with not allowing #DB in MCE -- much like today.
Will disable #DB inside #MCE then.
PATCH v5 21/34 x86/fred: FRED entry/exit and dispatch code
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
The code to actually handle kernel and event entry/exit using FRED. It is split up into two files thus:
- entry_64_fred.S contains the actual entrypoints and exit code, and saves and restores registers.
- entry_fred.c contains the two-level event dispatch code for FRED. The first-level dispatch is on the event type, and the second-level is on the event vector.
---
Changes since v1:
* Initialize a FRED exception handler to fred_bad_event() instead of NULL if no FRED handler defined for an exception vector (Peter Zijlstra).
* Push calling irqentry_{enter,exit}() and instrumentation_{begin,end}() down into individual FRED exception handlers, instead of in the dispatch framework (Peter Zijlstra).
--- arch/x86/entry/Makefile | 5 +- arch/x86/entry/entry_64_fred.S | 55 ++++++++ arch/x86/entry/entry_fred.c | 232 ++++++++++++++++++++++++++++++++ arch/x86/include/asm/idtentry.h | 8 ++ 4 files changed, 299 insertions(+), 1 deletion(-) create mode 100644 arch/x86/entry/entry_64_fred.S create mode 100644 arch/x86/entry/entry_fred.c
arch/x86/entry/Makefile +4 -1
@@ -18,6 +18,9 @@ obj-y += vdso/ obj-y += vsyscall/ obj-$(CONFIG_PREEMPTION) += thunk_$(BITS).o +CFLAGS_entry_fred.o += -fno-stack-protector +CFLAGS_REMOVE_entry_fred.o += -pg $(CC_FLAGS_FTRACE) +obj-$(CONFIG_X86_FRED) += entry_64_fred.o entry_fred.o + obj-$(CONFIG_IA32_EMULATION) += entry_64_compat.o syscall_32.o obj-$(CONFIG_X86_X32_ABI) += syscall_x32.o -
arch/x86/entry/entry_64_fred.S new file +55 -0
55 lines not on this page · raw message
arch/x86/entry/entry_fred.c new file +232 -0
232 lines not on this page · raw message
arch/x86/include/asm/idtentry.h +8 -0
@@ -82,6 +82,7 @@ static __always_inline void __##func(struct pt_regs *regs)#define DECLARE_IDTENTRY_ERRORCODE (vector, func) \ asmlinkage void asm_##func(void); \ asmlinkage void xen_asm_##func(void); \ + __visible void fred_##func(struct pt_regs *regs); \ __visible void func(struct pt_regs *regs, unsigned long error_code)/**
@@ -106,6 +107,11 @@ __visible noinstr void func(struct pt_regs *regs, \irqentry_exit (regs, state); \ } \ \ +__visible noinstr void fred_##func(struct pt_regs *regs) \ +{ \ + func (regs, regs->orig_ax); \ +} \ + \ static __always_inline void __##func(struct pt_regs *regs, \ unsigned long error_code)
@@ -622,6 +628,8 @@ DECLARE_IDTENTRY_RAW(X86_TRAP_MC, exc_machine_check);#ifdef CONFIG_XEN_PV DECLARE_IDTENTRY_RAW (X86_TRAP_MC , xenpv_exc_machine_check);#endif +#else +#define fred_exc_machine_check fred_bad_event#endif /* NMI */
Signature · 1 line
2.34.1
PATCH v5 22/34 x86/fred: FRED initialization code
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
The code to initialize FRED when it's available and _not_ disabled.
cpu_init_fred_exceptions() is the core function to initialize FRED, which
1. Sets up FRED entrypoints for events happening in ring 0 and 3.
2. Sets up a default stack for event handling.
3. Sets up dedicated event stacks for DB/NMI/MC/DF, equivalent to the IDT IST stacks.
4. Forces 32-bit system calls to use "int $0x80" only.
5. Enables FRED and invalidtes IDT.
When the FRED is used, cpu_init_exception_handling() initializes FRED through calling cpu_init_fred_exceptions(), otherwise it sets up TSS IST and loads IDT.
As FRED uses the ring 3 FRED entrypoint for SYSCALL and SYSENTER, it skips setting up SYSCALL/SYSENTER related MSRs, e.g., MSR_LSTAR.
--- arch/x86/include/asm/fred.h | 14 +++++++ arch/x86/include/asm/traps.h | 2 + arch/x86/kernel/Makefile | 1 + arch/x86/kernel/cpu/common.c | 74 +++++++++++++++++++++++------------- arch/x86/kernel/fred.c | 73 +++++++++++++++++++++++++++++++++++ arch/x86/kernel/irqinit.c | 7 +++- arch/x86/kernel/traps.c | 16 +++++++- 7 files changed, 157 insertions(+), 30 deletions(-) create mode 100644 arch/x86/kernel/fred.c
arch/x86/include/asm/fred.h +14 -0
@@ -99,8 +99,22 @@ DECLARE_FRED_HANDLER(fred_exc_debug);DECLARE_FRED_HANDLER (fred_exc_page_fault);DECLARE_FRED_HANDLER (fred_exc_machine_check); +/* +* The actual assembly entry and exit points +*/ +extern __visible void fred_entrypoint_user(void); + +/* +* Initialization +*/ +void cpu_init_fred_exceptions (void ); +void fred_setup_apic (void ); +#endif /* __ASSEMBLY__ */ +#else +#define cpu_init_fred_exceptions () BUG() +#define fred_setup_apic () BUG()#endif /* CONFIG_X86_FRED */ #endif /* ASM_X86_FRED_H */
arch/x86/include/asm/traps.h +2 -0
@@ -56,6 +56,8 @@ void __noreturn handle_stack_overflow(struct pt_regs *regs, void f (struct pt_regs *regs) typedef DECLARE_SYSTEM_INTERRUPT_HANDLER((*system_interrupt_handler)); +system_interrupt_handler get_system_interrupt_handler (unsigned int i); +int external_interrupt (struct pt_regs *regs,unsigned int vector);#endif /* _ASM_X86_TRAPS_H */
arch/x86/kernel/Makefile +1 -0
@@ -47,6 +47,7 @@ obj-y += platform-quirks.o obj-y += process_$(BITS).o signal.o signal_$(BITS).o obj-y += traps.o idt.o irq.o irq_$(BITS).o dumpstack_$(BITS).o obj-y += time.o ioport.o dumpstack.o nmi.o +obj-$(CONFIG_X86_FRED) += fred.o obj-$(CONFIG_MODIFY_LDT_SYSCALL) += ldt.o obj-y += setup.o x86_init.o i8259.o irqinit.o obj-$(CONFIG_JUMP_LABEL) += jump_label.o
arch/x86/kernel/cpu/common.c +47 -27
105 lines not on this page · raw message
arch/x86/kernel/fred.c new file +73 -0
73 lines not on this page · raw message
arch/x86/kernel/irqinit.c +6 -1
@@ -28,6 +28,7 @@#include <asm/setup.h> #include <asm/i8259.h> #include <asm/traps.h> +#include <asm/fred.h> #include <asm/prom.h> /*
@@ -96,7 +97,11 @@ void __init native_init_IRQ(void)/* Execute any quirks before the call gates are initialised: */ x86_init.irqs.pre_vector_init (); -idt_setup_apic_and_irq_gates (); +if (cpu_feature_enabled (X86_FEATURE_FRED )) +fred_setup_apic (); +else +idt_setup_apic_and_irq_gates (); +lapic_assign_system_vectors ();if (!acpi_ioapic && !of_ioapic &&nr_legacy_irqs ()) {
arch/x86/kernel/traps.c +14 -2
@@ -1517,12 +1517,21 @@ static system_interrupt_handler system_interrupt_handlers[NR_SYSTEM_VECTORS] = {#undef SYSV +system_interrupt_handler get_system_interrupt_handler (unsigned int i) +{ +if (i >=NR_SYSTEM_VECTORS ) +return NULL ; + +return system_interrupt_handlers[i]; +} +void __init install_system_interrupt_handler (unsigned int n,const void *asm_addr,const void *addr) {BUG_ON (n <FIRST_SYSTEM_VECTOR ); system_interrupt_handlers[n -FIRST_SYSTEM_VECTOR ] = (system_interrupt_handler )addr; -alloc_intr_gate (n, asm_addr); +if (!cpu_feature_enabled (X86_FEATURE_FRED )) +alloc_intr_gate (n, asm_addr); }#ifndef CONFIG_X86_LOCAL_APIC
@@ -1590,7 +1599,10 @@ void __init trap_init(void)/* Initialize TSS before setting up traps so ISTs work */ cpu_init_exception_handling (); +/* Setup traps as cpu_init() might #GP */ -idt_setup_traps (); +if (!cpu_feature_enabled (X86_FEATURE_FRED )) +idt_setup_traps (); +cpu_init (); }
Signature · 1 line
2.34.1
Re: [PATCH v5 22/34] x86/fred: FRED initialization code
Hello
Comments in cpu_init_fred_exceptions() seem scarce for understanding.
arch/x86/kernel/fred.c line 17
Quoted 10 lines · Xin Li
arch/x86/kernel/fred.c new file +10
-0+/* +* Initialize FRED on this CPU. This cannot be __init as it is called +* during CPU hotplug. +*/ +void cpu_init_fred_exceptions (void ) +{ +wrmsrl (MSR_IA32_FRED_CONFIG , +FRED_CONFIG_ENTRYPOINT (fred_entrypoint_user) | +FRED_CONFIG_REDZONE (8 ) |/* Reserve for CALL emulation */ +FRED_CONFIG_INT_STKLVL (0 ));
What is it about "Reserve for CALL emulation"?
I guess it relates to X86_TRAP_BP. In entry_64.S:
.if \vector == X86_TRAP_BP
/*
* If coming from kernel space, create a 6-word gap to allow the
* int3 handler to emulate a call instruction.
*/arch/x86/kernel/fred.c line 23
Quoted 6 lines
arch/x86/kernel/fred.c new file +6
-0+ +wrmsrl (MSR_IA32_FRED_STKLVLS , +FRED_STKLVL (X86_TRAP_DB ,1 ) | +FRED_STKLVL (X86_TRAP_NMI ,2 ) | +FRED_STKLVL (X86_TRAP_MC ,2 ) | +FRED_STKLVL (X86_TRAP_DF ,3 ));
Why each exception here needs a stack level > 0?
Especially for X86_TRAP_DB and X86_TRAP_NMI.
Why does or why does not X86_TRAP_VE have a stack level > 0?
X86_TRAP_DF is the highest stack level, is it accidental or deliberate?
Thanks
Lai
H. Peter Anvin
maintainerRe: [PATCH v5 22/34] x86/fred: FRED initialization code
Quoted 38 lines · Lai Jiangshan
Hello
Comments in
cpu_init_fred_exceptions()seem scarce for understanding.arch/x86/kernel/fred.c line 17
Quoted 10 lines · Xin Li
arch/x86/kernel/fred.c new file +10
-0+/* +* Initialize FRED on this CPU. This cannot be __init as it is called +* during CPU hotplug. +*/ +void cpu_init_fred_exceptions (void ) +{ +wrmsrl (MSR_IA32_FRED_CONFIG , +FRED_CONFIG_ENTRYPOINT (fred_entrypoint_user) | +FRED_CONFIG_REDZONE (8 ) |/* Reserve for CALL emulation */ +FRED_CONFIG_INT_STKLVL (0 ));What is it about "Reserve for CALL emulation"?
I guess it relates to
X86_TRAP_BP. Inentry_64.S:.if \vector == X86_TRAP_BP /* * If coming from kernel space, create a 6-word gap to allow the * int3 handler to emulate a call instruction. */arch/x86/kernel/fred.c line 23
Quoted 6 lines
arch/x86/kernel/fred.c new file +6
-0+ +wrmsrl (MSR_IA32_FRED_STKLVLS , +FRED_STKLVL (X86_TRAP_DB ,1 ) | +FRED_STKLVL (X86_TRAP_NMI ,2 ) | +FRED_STKLVL (X86_TRAP_MC ,2 ) | +FRED_STKLVL (X86_TRAP_DF ,3 ));Why each exception here needs a stack level > 0?
Especially forX86_TRAP_DBandX86_TRAP_NMI.
Quoted 8 lines
Why does or why does not
X86_TRAP_VEhave a stack level > 0?
X86_TRAP_DFis the highest stack level, is it accidental or deliberate?Thanks
Lai
Yes, the extra redzone space is there to allow for the call emulation without having to adjust the stack frame "manually".
In theory we could enable it only while code patching is in progress, but that would probably just result in stack overflows becoming utterly impossible to debug as we have to consider the worst case.
The purpose of separate stacks for NMI, #DB and #MC *in the kernel* (remember that user space faults are always taken on stack level 0) is to avoid overflowing the kernel stack. #DB in the kernel would imply the use of a kernel debugger.
#DF is the highest level because a #DF means "something went wrong *while delivering an exception*." The number of cases for which that can happen with FRED is drastically reduced and basically amount to "the stack you pointed me to is broken."
Thus, you basically always want to change stacks on #DF, which means it should be at the highest level.
Re: [PATCH v5 22/34] x86/fred: FRED initialization code
Quoted 49 lines · H. Peter Anvin
Quoted 38 lines · Lai Jiangshan
Hello
Comments in
cpu_init_fred_exceptions()seem scarce for understanding.arch/x86/kernel/fred.c line 17
Quoted 10 lines · Xin Li
arch/x86/kernel/fred.c new file +10
-0+/* +* Initialize FRED on this CPU. This cannot be __init as it is called +* during CPU hotplug. +*/ +void cpu_init_fred_exceptions (void ) +{ +wrmsrl (MSR_IA32_FRED_CONFIG , +FRED_CONFIG_ENTRYPOINT (fred_entrypoint_user) | +FRED_CONFIG_REDZONE (8 ) |/* Reserve for CALL emulation */ +FRED_CONFIG_INT_STKLVL (0 ));What is it about "Reserve for CALL emulation"?
I guess it relates to
X86_TRAP_BP. Inentry_64.S:.if \vector == X86_TRAP_BP /* * If coming from kernel space, create a 6-word gap to allow the * int3 handler to emulate a call instruction. */arch/x86/kernel/fred.c line 23
Quoted 6 lines
arch/x86/kernel/fred.c new file +6
-0+ +wrmsrl (MSR_IA32_FRED_STKLVLS , +FRED_STKLVL (X86_TRAP_DB ,1 ) | +FRED_STKLVL (X86_TRAP_NMI ,2 ) | +FRED_STKLVL (X86_TRAP_MC ,2 ) | +FRED_STKLVL (X86_TRAP_DF ,3 ));Why each exception here needs a stack level > 0?
Especially forX86_TRAP_DBandX86_TRAP_NMI.Quoted 8 lines
Why does or why does not
X86_TRAP_VEhave a stack level > 0?
X86_TRAP_DFis the highest stack level, is it accidental or deliberate?Thanks
Lai
Quoted 5 lines
Yes, the extra redzone space is there to allow for the call emulation without having to adjust the stack frame "manually".
In theory we could enable it only while code patching is in progress, but that would probably just result in stack overflows becoming utterly impossible to debug as we have to consider the worst case.
The purpose of separate stacks for NMI, #DB and #MC *in the kernel* (remember that user space faults are always taken on stack level 0) is to avoid overflowing the kernel stack. #DB in the kernel would imply the use of a kernel debugger.
Could you add it to the code, please? I think it can help other reviewers.
If there is no other concrete reason other than overflowing for assigning NMI and #DB with a stack level > 0, #VE should also be assigned with a stack level > 0, and #BP too. #VE can happen anytime and anywhere, so it is subject to overflowing too.
Quoted 5 lines
#DF is the highest level because a #DF means "something went wrong *while delivering an exception*." The number of cases for which that can happen with FRED is drastically reduced and basically amount to "the stack you pointed me to is broken."
Thus, you basically always want to change stacks on #DF, which means it should be at the highest level.
Peter Zijlstra
maintainerRe: [PATCH v5 22/34] x86/fred: FRED initialization code
Quoted 4 lines · Lai Jiangshan
If there is no other concrete reason other than overflowing for assigning NMI and #DB with a stack level > 0, #VE should also be assigned with a stack level > 0, and #BP too. #VE can happen anytime and anywhere, so it is subject to overflowing too.
So #BP needs the stack-gap (redzone) for text_poke_bp().
#BP can end up in kprobes which can then end up in ftrace/perf, depending on how it's all wired up.
#VE is currently a trainwreck vs NMI/MCE, but I think FRED solves the worst of that. I'm not exactly sure how deep the #VE handler goes.
RE: [PATCH v5 22/34] x86/fred: FRED initialization code
Quoted 5 lines
Quoted 4 lines
If there is no other concrete reason other than overflowing for assigning NMI and #DB with a stack level > 0, #VE should also be assigned with a stack level > 0, and #BP too. #VE can happen anytime and anywhere, so it is subject to overflowing too.
Quoted 8 lines
So #BP needs the stack-gap (redzone) for
text_poke_bp().#BP can end up in kprobes which can then end up in ftrace/perf, depending on how it's all wired up.
#VE is currently a trainwreck vs NMI/MCE, but I think FRED solves the worst of that. I'm not exactly sure how deep the #VE handler goes.
VE under IDT is *not* using an IST, we need some solid rationales here.
Thanks!
Xin
Re: [PATCH v5 22/34] x86/fred: FRED initialization code
Quoted 6 lines · On 21/03/2023 12:12 am, Li, Xin3 wrote:
Quoted 6 lines
Quoted 4 lines
If there is no other concrete reason other than overflowing for assigning NMI and #DB with a stack level > 0, #VE should also be assigned with a stack level > 0, and #BP too. #VE can happen anytime and anywhere, so it is subject to overflowing too.
So #BP needs the stack-gap (redzone) for
text_poke_bp().
Quoted 7 lines
Quoted 6 lines
#BP can end up in kprobes which can then end up in ftrace/perf, depending on how it's all wired up.
#VE is currently a trainwreck vs NMI/MCE, but I think FRED solves the worst of that. I'm not exactly sure how deep the #VE handler goes.
VE under IDT is *not* using an IST, we need some solid rationales here.
#VE, and #VC on AMD, are borderline unusable. Both under IDT and FRED.
The reason #VE is not IST is because there are plenty of real cases where a non-malicious outer hypervisor could create reentrant faults that lose program state. e.g. hitting an IO instruction, then hitting an emulated MSR.
There are fewer cases where a non-IST #VE ends up in a re-entrant fault (IIRC, you can still manage it by unmapping the entry stack), but you're still trusting the outer hypervisor to not e.g. unmap the SYSCALL entry point.
FRED gets rid of the "reentrant fault overwriting it on the stack" case, and removes the syscall gap case, replacing them instead with a stack overflow in the worst case because there is still no upper bound to how many times #VE can actually be delivered in the course of servicing a single #VE.
~Andrew
P.S. While I hate to cite myself, if you haven't read
https://docs.google.com/document/d/1hWejnyDkjRRAW-JEsRjA5c9CKLOPc6VKJQsuvODlQEI/edit?usp=sharing
yet, do so. It did feed into some of the FRED design.
RE: [PATCH v5 22/34] x86/fred: FRED initialization code
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
If there is no other concrete reason other than overflowing for assigning NMI and #DB with a stack level > 0, #VE should also be assigned with a stack level > 0, and #BP too. #VE can happen anytime and anywhere, so it is subject to overflowing too.
So #BP needs the stack-gap (redzone) for
text_poke_bp().#BP can end up in kprobes which can then end up in ftrace/perf, depending
on
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
how it's all wired up.
#VE is currently a trainwreck vs NMI/MCE, but I think FRED solves the worst of that. I'm not exactly sure how deep the #VE handler goes.
VE under IDT is *not* using an IST, we need some solid rationales here.
#VE, and #VC on AMD, are borderline unusable. Both under IDT and FRED.
Oops!
Quoted 10 lines
The reason #VE is not IST is because there are plenty of real cases where a non-malicious outer hypervisor could create reentrant faults that lose program state. e.g. hitting an IO instruction, then hitting an emulated MSR.
There are fewer cases where a non-IST #VE ends up in a re-entrant fault (IIRC, you can still manage it by unmapping the entry stack), but you're still trusting the outer hypervisor to not e.g. unmap the SYSCALL entry point.
Quoted 5 lines
FRED gets rid of the "reentrant fault overwriting it on the stack" case, and removes the syscall gap case, replacing them instead with a stack overflow in the worst case because there is still no upper bound to how many times #VE can actually be delivered in the course of servicing a single #VE.
Exactly, FRED stack levels can make use of the whole regular stack space.
I guess you don't seem to support #VE on a higher stack level?
Quoted 6 lines
~Andrew
P.S. While I hate to cite myself, if you haven't read
https://docs.google.com/document/d/1hWejnyDkjRRAW-
JEsRjA5c9CKLOPc6VKJQsuvODlQEI/edit?usp=sharing yet, do so. It did feed into some of the FRED design.
Re: [PATCH v5 22/34] x86/fred: FRED initialization code
In reply to [email protected]Quoted 4 lines · On 3/20/23 18:02, [email protected] wrote:
There are fewer cases where a non-IST #VE ends up in a re-entrant fault (IIRC, you can still manage it by unmapping the entry stack), but you're still trusting the outer hypervisor to not e.g. unmap the SYSCALL entry point.
This is a general weakness of #VE. But, the current Linux TDX guest implementation is not vulnerable to it. If the host unmaps something unexpectedly, the guest will just die because of ATTR_SEPT_VE_DISABLE. No #VE:
RE: [PATCH v5 22/34] x86/fred: FRED initialization code
In reply to Lai JiangshanQuoted 3 lines
If there is no other concrete reason other than overflowing for assigning NMI and #DB with a stack level > 0, #VE should also be assigned with a stack level > 0, and #BP too. #VE can happen anytime and anywhere, so it is subject to overflowing too.
With IDT, both #VE and #BP do not use IST, but NMI, #DB, #MC and #DF do.
Let's keep this "secret" logic for now, i.e., not change the stack levels for #VE and #BP at this point. We can do "optimization", i.e., change them later :).
Re: [PATCH v5 22/34] x86/fred: FRED initialization code
Quoted 8 lines · On 3/21/23 19:22, Li, Xin3 wrote:
Quoted 3 lines
If there is no other concrete reason other than overflowing for assigning NMI and #DB with a stack level > 0, #VE should also be assigned with a stack level > 0, and #BP too. #VE can happen anytime and anywhere, so it is subject to overflowing too.
With IDT, both #VE and #BP do not use IST, but NMI, #DB, #MC and #DF do.
Let's keep this "secret" logic for now, i.e., not change the stack levels for #VE and #BP at this point. We can do "optimization", i.e., change them later 😄.
#VE also can't happen anywhere. There is some documentation about it in here:
https://docs.kernel.org/x86/tdx.html#linux-ve-handler
But, basically, the only halfway sane thing a guest might do to hit a #VE is touch some "MMIO". The host can *not* cause them in arbitrary places because of the SEPT_VE_DISABLE attribute.
#VE's also can't nest until after the guest retrieves the "VE info". That means that the #VE handler at _least_ reaches C code before it's subject to another #VE and that second one would still need to be induced by something the guest does explicitly.
RE: [PATCH v5 22/34] x86/fred: FRED initialization code
Quoted 17 lines
Quoted 1 line
Quoted 1 line
If there is no other concrete reason other than overflowing for assigning NMI
and
Quoted 1 line
Quoted 1 line
#DB with a stack level > 0, #VE should also be assigned with a stack level > 0,
and
Quoted 1 line
Quoted 1 line
#BP too. #VE can happen anytime and anywhere, so it is subject to
overflowing too.
Quoted 5 lines
With IDT, both #VE and #BP do not use IST, but NMI, #DB, #MC and #DF do.
Let's keep this "secret" logic for now, i.e., not change the stack levels for #VE and #BP at this point. We can do "optimization", i.e., change them later 😄.
#VE also can't happen anywhere. There is some documentation about it in here:
Quoted 8 lines
But, basically, the only halfway sane thing a guest might do to hit a #VE is touch some "MMIO". The host can *not* cause them in arbitrary places because of the
SEPT_VE_DISABLEattribute.#VE's also can't nest until after the guest retrieves the "VE info". That means that the #VE handler at _least_ reaches C code before it's subject to another #VE and that second one would still need to be induced by something the guest does explicitly.
Thanks a lot for the detailed background!
Re: [PATCH v5 22/34] x86/fred: FRED initialization code
In reply to Li, Xin3Quoted 8 lines · On 22/03/2023 2:22 am, Li, Xin3 wrote:
Quoted 3 lines
If there is no other concrete reason other than overflowing for assigning NMI and #DB with a stack level > 0, #VE should also be assigned with a stack level > 0, and #BP too. #VE can happen anytime and anywhere, so it is subject to overflowing too.
With IDT, both #VE and #BP do not use IST, but NMI, #DB, #MC and #DF do.
Let's keep this "secret" logic for now, i.e., not change the stack levels for #VE and #BP at this point. We can do "optimization", i.e., change them later :).
Fun fact. #BP used to be IST, and used to share the same IST as #DF.
This was spoiled by CVE-2018-8897 and a MovSS-delayed breakpoint over INT3, at which point hardware queued both a #BP and #DB on the same IST stack and lost program state.
There's no need specific need for #BP to be IST to begin with, hence why making it not-IST was the security fix.
~Andrew
Peter Zijlstra
maintainerRe: [PATCH v5 22/34] x86/fred: FRED initialization code
In reply to H. Peter AnvinQuoted 4 lines · H. Peter Anvin
The purpose of separate stacks for NMI, #DB and #MC *in the kernel* (remember that user space faults are always taken on stack level 0) is to avoid overflowing the kernel stack. #DB in the kernel would imply the use of a kernel debugger.
Perf (and through it bpf) also has access to #DB. They can set breakpoints on kernel instructions/memory just fine provided permission etc.
RE: [PATCH v5 22/34] x86/fred: FRED initialization code
Quoted 8 lines
Quoted 4 lines
The purpose of separate stacks for NMI, #DB and #MC *in the kernel* (remember that user space faults are always taken on stack level 0) is to avoid overflowing the kernel stack. #DB in the kernel would imply the use of a kernel debugger.
Perf (and through it bpf) also has access to #DB. They can set breakpoints on kernel instructions/memory just fine provided permission etc.
So they are still *kernel* debuggers :)
PATCH v5 23/34 x86/fred: update MSR_IA32_FRED_RSP0 during task switch
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
MSR_IA32_FRED_RSP0 is used during ring 3 event delivery, and needs to be updated to point to the top of next task stack during task switch.
Update MSR_IA32_FRED_RSP0 with WRMSR instruction for now, and will use WRMSRNS/WRMSRLIST for performance once it gets upstreamed.
--- arch/x86/include/asm/switch_to.h | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
arch/x86/include/asm/switch_to.h +9 -2
@@ -68,9 +68,16 @@ static inline void update_task_stack(struct task_struct *task)#ifdef CONFIG_X86_32 this_cpu_write (cpu_tss_rw.x86_tss.sp1, task->thread.sp0);#else -/* Xen PV enters the kernel on the thread stack. */ -if (cpu_feature_enabled (X86_FEATURE_XENPV )) +if (cpu_feature_enabled (X86_FEATURE_FRED )) { +/* +* Will use WRMSRNS/WRMSRLIST for performance once it's upstreamed. +*/ +wrmsrl (MSR_IA32_FRED_RSP0 , +task_top_of_stack (task) +TOP_OF_KERNEL_STACK_PADDING ); + }else if (cpu_feature_enabled (X86_FEATURE_XENPV )) { +/* Xen PV enters the kernel on the thread stack. */ load_sp0 (task_top_of_stack (task)); + }#endif }
Signature · 1 line
2.34.1
Peter Zijlstra
maintainerRe: [PATCH v5 23/34] x86/fred: update MSR_IA32_FRED_RSP0 during task switch
Quoted 22 lines · Xin Li
From: "H. Peter Anvin (Intel)" <[email protected]>
MSR_IA32_FRED_RSP0is used during ring 3 event delivery, and needs to be updated to point to the top of next task stack during task switch.Update
MSR_IA32_FRED_RSP0with WRMSR instruction for now, and will use WRMSRNS/WRMSRLIST for performance once it gets upstreamed.Signed-off-by H. Peter Anvin (Intel) <[email protected]>Tested-by Shan Kang <[email protected]>Signed-off-by Xin Li <[email protected]>--- arch/x86/include/asm/switch_to.h | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)arch/x86/include/asm/switch_to.h +0
-0@@ -68,9 +68,16 @@ static inline void update_task_stack(struct task_struct *task)#ifdef CONFIG_X86_32 this_cpu_write (cpu_tss_rw.x86_tss.sp1, task->thread.sp0);
arch/x86/include/asm/switch_to.h static inline void update_task_stack(struct task_struct *task) · line 77
Quoted 10 lines
arch/x86/include/asm/switch_to.h +7
-2#else -/* Xen PV enters the kernel on the thread stack. */ -if (cpu_feature_enabled (X86_FEATURE_XENPV )) +if (cpu_feature_enabled (X86_FEATURE_FRED )) { +/* +* Will use WRMSRNS/WRMSRLIST for performance once it's upstreamed. +*/ +wrmsrl (MSR_IA32_FRED_RSP0 , +task_top_of_stack (task) +TOP_OF_KERNEL_STACK_PADDING ); + }else if (cpu_feature_enabled (X86_FEATURE_XENPV )) {
Whee, so hardware will really only ever look at this when RSP0? I don't need to worry about exceptions during context switch?
RE: [PATCH v5 23/34] x86/fred: update MSR_IA32_FRED_RSP0 during task switch
Quoted 5 lines
arch/x86/include/asm/switch_to.h static inline void update_task_stack(struct task_struct *task)
Quoted 4 lines
arch/x86/include/asm/switch_to.h +3
-1-if (cpu_feature_enabled (X86_FEATURE_XENPV )) +if (cpu_feature_enabled (X86_FEATURE_FRED )) { +/* +* Will use WRMSRNS/WRMSRLIST for performance once it's upstreamed.
Quoted 8 lines
arch/x86/include/asm/switch_to.h static inline void update_task_stack(struct task_struct *task)
Quoted 3 lines
arch/x86/include/asm/switch_to.h +3
-0+*/ + wrmsrl(MSR_IA32_FRED_RSP0, + task_top_of_stack(task) +
TOP_OF_KERNEL_STACK_PADDING);arch/x86/include/asm/switch_to.h static inline void update_task_stack(struct task_struct *task) · line 77
Quoted 1 line
arch/x86/include/asm/switch_to.h +1
-0+ }else if (cpu_feature_enabled (X86_FEATURE_XENPV )) {Whee, so hardware will really only ever look at this when RSP0? I don't need to worry about exceptions during context switch?
You're right, we don't.
RSP0 is only used in ring3. Exceptions from ring0 just keep using the current kernel stack unless a higher stack level needs to be used, e.g., RSP3 for #DF.
Thanks!
Xin
PATCH v5 24/34 x86/fred: let ret_from_fork() jmp to fred_exit_user when FRED is enabled
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Let ret_from_fork() jmp to fred_exit_user when FRED is enabled, otherwise the existing IDT code is chosen.
--- arch/x86/entry/entry_64.S | 5 +++++ 1 file changed, 5 insertions(+)
arch/x86/entry/entry_64.S +5 -0
@@ -299,7 +299,12 @@ SYM_CODE_START_NOALIGN(ret_from_fork) UNWIND_HINT_REGS movq %rsp, %rdi call syscall_exit_to_user_mode /* returns with IRQs disabled */ +#ifdef CONFIG_X86_FRED + ALTERNATIVE "jmp swapgs_restore_regs_and_return_to_usermode", \ + "jmp fred_exit_user", X86_FEATURE_FRED +#else jmp swapgs_restore_regs_and_return_to_usermode +#endif 1: /* kernel thread */
Signature · 1 line
2.34.1
PATCH v5 25/34 x86/fred: disallow the swapgs instruction when FRED is enabled
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
The FRED architecture establishes the full supervisor/user through:
1) FRED event delivery swaps the value of the GS base address and that of the IA32_KERNEL_GS_BASE MSR.
2) ERETU swaps the value of the GS base address and that of the IA32_KERNEL_GS_BASE MSR.
Thus, the swapgs instruction is disallowed when FRED is enabled, otherwise it cauess #UD.
--- arch/x86/kernel/process_64.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
arch/x86/kernel/process_64.c +4 -2
@@ -165,7 +165,8 @@ static noinstr unsigned long __rdgsbase_inactive(void)lockdep_assert_irqs_disabled (); -if (!cpu_feature_enabled (X86_FEATURE_XENPV )) { +if (!cpu_feature_enabled (X86_FEATURE_FRED ) && + !cpu_feature_enabled (X86_FEATURE_XENPV )) {native_swapgs (); gsbase =rdgsbase ();native_swapgs ();
@@ -190,7 +191,8 @@ static noinstr void __wrgsbase_inactive(unsigned long gsbase) {lockdep_assert_irqs_disabled (); -if (!cpu_feature_enabled (X86_FEATURE_XENPV )) { +if (!cpu_feature_enabled (X86_FEATURE_FRED ) && + !cpu_feature_enabled (X86_FEATURE_XENPV )) {native_swapgs ();wrgsbase (gsbase);native_swapgs ();
Signature · 1 line
2.34.1
Peter Zijlstra
maintainerRe: [PATCH v5 25/34] x86/fred: disallow the swapgs instruction when FRED is enabled
Quoted 9 lines · Xin Li
From: "H. Peter Anvin (Intel)" <[email protected]>
The FRED architecture establishes the full supervisor/user through:
1) FRED event delivery swaps the value of the GS base address and that of theIA32_KERNEL_GS_BASEMSR.
2) ERETU swaps the value of the GS base address and that of theIA32_KERNEL_GS_BASEMSR.
Thus, the swapgs instruction is disallowed when FRED is enabled, otherwise it cauess #UD.
^^^ --- new word :-)
RE: [PATCH v5 25/34] x86/fred: disallow the swapgs instruction when FRED is enabled
Quoted 8 lines
Quoted 7 lines
The FRED architecture establishes the full supervisor/user through:
1) FRED event delivery swaps the value of the GS base address and that of theIA32_KERNEL_GS_BASEMSR.
2) ERETU swaps the value of the GS base address and that of theIA32_KERNEL_GS_BASEMSR.
Thus, the swapgs instruction is disallowed when FRED is enabled, otherwise it cauess #UD.^^^ --- new word :-)
My stupid fingers...
PATCH v5 26/34 x86/fred: no ESPFIX needed when FRED is enabled
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Because FRED always restores the full value of %rsp, ESPFIX is no longer needed when it's enabled.
--- arch/x86/kernel/espfix_64.c | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/x86/kernel/espfix_64.c +8 -0
@@ -106,6 +106,10 @@ void __init init_espfix_bsp(void)pgd_t *pgd;p4d_t *p4d; +/* FRED systems don't need ESPFIX */ +if (cpu_feature_enabled (X86_FEATURE_FRED )) +return ; +/* Install the espfix pud into the kernel page directory */ pgd = &init_top_pgt[pgd_index (ESPFIX_BASE_ADDR )]; p4d =p4d_alloc (&init_mm, pgd,ESPFIX_BASE_ADDR );
@@ -129,6 +133,10 @@ void init_espfix_ap(int cpu)void *stack_page;pteval_t ptemask; +/* FRED systems don't need ESPFIX */ +if (cpu_feature_enabled (X86_FEATURE_FRED )) +return ; +/* We only have to do this once... */ if (likely (per_cpu (espfix_stack, cpu)))return ;/* Already initialized */
Signature · 1 line
2.34.1
PATCH v5 27/34 x86/fred: allow single-step trap and NMI when starting a new thread
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
Allow single-step trap and NMI when starting a new thread, thus once the new thread returns to ring3, single-step trap and NMI are both enabled immediately.
High-order 48 bits above the lowest 16 bit CS are discarded by the legacy IRET instruction, thus can be set unconditionally, even when FRED is not enabled.
--- arch/x86/include/asm/fred.h | 11 +++++++++++ arch/x86/kernel/process_64.c | 13 +++++++------ 2 files changed, 18 insertions(+), 6 deletions(-)
arch/x86/include/asm/fred.h +11 -0
@@ -52,6 +52,14 @@#define FRED_CSL_ALLOW_SINGLE_STEP _BITUL(25)#define FRED_CSL_INTERRUPT_SHADOW _BITUL(24) +/* +* High-order 48 bits above the lowest 16 bit CS are discarded by the +* legacy IRET instruction, thus can be set unconditionally, even when +* FRED is not enabled. +*/ +#define CSL_PROCESS_START \ + (FRED_CSL_ENABLE_NMI | FRED_CSL_ALLOW_SINGLE_STEP) +#ifndef __ASSEMBLY__#include <linux/kernel.h>
@@ -115,6 +123,9 @@ void fred_setup_apic(void);#else #define cpu_init_fred_exceptions () BUG()#define fred_setup_apic () BUG() + +#define CSL_PROCESS_START 0 +#endif /* CONFIG_X86_FRED */ #endif /* ASM_X86_FRED_H */
arch/x86/kernel/process_64.c +7 -6
@@ -55,6 +55,7 @@#include <asm/resctrl.h> #include <asm/unistd.h> #include <asm/fsgsbase.h> +#include <asm/fred.h> #ifdef CONFIG_IA32_EMULATION /* Not included via unistd.h */ #include <asm/unistd_32_ia32.h>
@@ -506,7 +507,7 @@ void x86_gsbase_write_task(struct task_struct *task, unsigned long gsbase)static void start_thread_common (struct pt_regs *regs,unsigned long new_ip,unsigned long new_sp, -unsigned int _cs,unsigned int _ss,unsigned int _ds) +u16 _cs,u16 _ss,u16 _ds) {WARN_ON_ONCE (regs !=current_pt_regs ());
@@ -521,11 +522,11 @@ start_thread_common(struct pt_regs *regs, unsigned long new_ip,loadsegment (ds, _ds);load_gs_index (0 ); - regs->ip = new_ip; - regs->sp = new_sp; - regs->cs = _cs; - regs->ss = _ss; - regs->flags =X86_EFLAGS_IF ; + regs->ip = new_ip; + regs->sp = new_sp; + regs->csx = _cs |CSL_PROCESS_START ; + regs->ssx = _ss; + regs->flags =X86_EFLAGS_IF |X86_EFLAGS_FIXED ; }void
Signature · 1 line
2.34.1
PATCH v5 28/34 x86/fred: fixup fault on ERETU by jumping to fred_entrypoint_user
In reply to Xin LiIf the stack frame contains an invalid user context (e.g. due to invalid SS, a non-canonical RIP, etc.) the ERETU instruction will trap (#SS or #GP).
From a Linux point of view, this really should be considered a user space failure, so use the standard fault fixup mechanism to intercept the fault, fix up the exception frame, and redirect execution to fred_entrypoint_user. The end result is that it appears just as if the hardware had taken the exception immediately after completing the transition to user space.
--- arch/x86/entry/entry_64_fred.S | 8 +++++-- arch/x86/include/asm/extable_fixup_types.h | 4 +++- arch/x86/mm/extable.c | 28 ++++++++++++++++++++++ 3 files changed, 37 insertions(+), 3 deletions(-)
arch/x86/entry/entry_64_fred.S +6 -2
@@ -5,8 +5,10 @@ * The actual FRED entry points. */ #include <linux/linkage.h> -#include <asm/errno.h> +#include <asm/asm.h> #include <asm/asm-offsets.h> +#include <asm/errno.h> +#include <asm/export.h> #include <asm/fred.h> #include "calling.h"
@@ -38,7 +40,9 @@ SYM_CODE_START_NOALIGN(fred_entrypoint_user) call fred_entry_from_user SYM_INNER_LABEL(fred_exit_user, SYM_L_GLOBAL) FRED_EXIT - ERETU +1: ERETU + + _ASM_EXTABLE_TYPE(1b, fred_entrypoint_user, EX_TYPE_ERETU) SYM_CODE_END(fred_entrypoint_user) /*
arch/x86/include/asm/extable_fixup_types.h +3 -1
@@ -64,6 +64,8 @@#define EX_TYPE_UCOPY_LEN4 (EX_TYPE_UCOPY_LEN | EX_DATA_IMM(4))#define EX_TYPE_UCOPY_LEN8 (EX_TYPE_UCOPY_LEN | EX_DATA_IMM(8)) -#define EX_TYPE_ZEROPAD 20/* longword load with zeropad on fault */ +#define EX_TYPE_ZEROPAD 20/* longword load with zeropad on fault */ + +#define EX_TYPE_ERETU 21#endif
arch/x86/mm/extable.c +28 -0
@@ -6,6 +6,7 @@#include <xen/xen.h> #include <asm/fpu/api.h> +#include <asm/fred.h> #include <asm/sev.h> #include <asm/traps.h> #include <asm/kdebug.h>
@@ -195,6 +196,29 @@ static bool ex_handler_ucopy_len(const struct exception_table_entry *fixup,return ex_handler_uaccess (fixup, regs, trapnr); } +#ifdef CONFIG_X86_FRED +static bool ex_handler_eretu (const struct exception_table_entry *fixup, +struct pt_regs *regs,unsigned long error_code) +{ +struct pt_regs *uregs = (struct pt_regs *)(regs->sp - offsetof(struct pt_regs , ip)); +unsigned short ss = uregs->ss; +unsigned short cs = uregs->cs; + +fred_info (uregs)->edata =fred_event_data (regs); + uregs->ssx = regs->ssx; + uregs->ss = ss; + uregs->csx = regs->csx; + uregs->current_stack_level =0 ; + uregs->cs = cs; + +/* Copy error code to uregs and adjust stack pointer accordingly */ + uregs->orig_ax = error_code; + regs->sp -=8 ; + +return ex_handler_default (fixup, regs); +} +#endif +int ex_get_fixup_type (unsigned long ip) {const struct exception_table_entry *e =search_exception_tables (ip);
@@ -272,6 +296,10 @@ int fixup_exception(struct pt_regs *regs, int trapnr, unsigned long error_code,return ex_handler_ucopy_len (e, regs, trapnr, reg, imm);case EX_TYPE_ZEROPAD :return ex_handler_zeropad (e, regs, fault_addr); +#ifdef CONFIG_X86_FRED +case EX_TYPE_ERETU : +return ex_handler_eretu (e, regs, error_code); +#endif }BUG (); }
Signature · 1 line
2.34.1
Re: [PATCH v5 28/34] x86/fred: fixup fault on ERETU by jumping to fred_entrypoint_user
Quoted 4 lines
arch/x86/mm/extable.c +4
-0+#ifdef CONFIG_X86_FRED +static bool ex_handler_eretu (const struct exception_table_entry *fixup, +struct pt_regs *regs,unsigned long error_code) +{
arch/x86/mm/extable.c static bool ex_handler_ucopy_len(const struct exception_table_entry *fixup, · line 212
Quoted 10 lines
arch/x86/mm/extable.c +10
-0+struct pt_regs *uregs = (struct pt_regs *)(regs->sp - offsetof(struct pt_regs , ip)); +unsigned short ss = uregs->ss; +unsigned short cs = uregs->cs; + +fred_info (uregs)->edata =fred_event_data (regs); + uregs->ssx = regs->ssx; + uregs->ss = ss; + uregs->csx = regs->csx; + uregs->current_stack_level =0 ; + uregs->cs = cs;
Hello
If the ERETU instruction had tried to return from NMI to ring3 and just faulted, is NMI still blocked?
We know that IRET unconditionally enables NMI, but I can't find any clue in the FRED's manual.
In the pseudocode of ERETU in the manual, it seems that NMI is only enabled when ERETU succeeds with bit28 in csx set. If so, this code will fail to reenable NMI if bit28 is not explicitly re-set in csx.
Thanks,
Lai
Quoted 7 lines
arch/x86/mm/extable.c +7
-0+ +/* Copy error code to uregs and adjust stack pointer accordingly */ + uregs->orig_ax = error_code; + regs->sp -=8 ; + +return ex_handler_default (fixup, regs); +}
Re: [PATCH v5 28/34] x86/fred: fixup fault on ERETU by jumping to fred_entrypoint_user
Quoted 19 lines · Lai Jiangshan
Quoted 4 lines
arch/x86/mm/extable.c +4
-0+#ifdef CONFIG_X86_FRED +static bool ex_handler_eretu (const struct exception_table_entry *fixup, +struct pt_regs *regs,unsigned long error_code) +{arch/x86/mm/extable.c static bool ex_handler_ucopy_len(const struct exception_table_entry *fixup, · line 212
Quoted 10 lines
arch/x86/mm/extable.c +10
-0+struct pt_regs *uregs = (struct pt_regs *)(regs->sp - offsetof(struct pt_regs , ip)); +unsigned short ss = uregs->ss; +unsigned short cs = uregs->cs; + +fred_info (uregs)->edata =fred_event_data (regs); + uregs->ssx = regs->ssx; + uregs->ss = ss; + uregs->csx = regs->csx; + uregs->current_stack_level =0 ; + uregs->cs = cs;Hello
If the ERETU instruction had tried to return from NMI to ring3 and just faulted, is NMI still blocked?
Quoted 6 lines
We know that IRET unconditionally enables NMI, but I can't find any clue in the FRED's manual.
In the pseudocode of ERETU in the manual, it seems that NMI is only enabled when ERETU succeeds with bit28 in csx set. If so, this code will fail to reenable NMI if bit28 is not explicitly re-set in csx.
IRET clearing NMI blocking is the source of an immense amount of grief, and ultimately the reason why Linux and others can't use supervisor shadow stacks at the moment.
Changing this property, so NMIs only get unblocked on successful execution of an ERET{S,U}, was a key demand of the FRED spec.
i.e. until you have successfully ERET*'d, you're still logically in the NMI handler and NMIs need to remain blocked even when handling the #GP from a bad ERET.
~Andrew
Re: [PATCH v5 28/34] x86/fred: fixup fault on ERETU by jumping to fred_entrypoint_user
Quoted 32 lines · On Fri, Mar 17, 2023 at 5:56 PM <[email protected]> wrote:
Quoted 19 lines · Lai Jiangshan
Quoted 4 lines
arch/x86/mm/extable.c +4
-0+#ifdef CONFIG_X86_FRED +static bool ex_handler_eretu (const struct exception_table_entry *fixup, +struct pt_regs *regs,unsigned long error_code) +{arch/x86/mm/extable.c static bool ex_handler_ucopy_len(const struct exception_table_entry *fixup, · line 212
Quoted 10 lines
arch/x86/mm/extable.c +10
-0+struct pt_regs *uregs = (struct pt_regs *)(regs->sp - offsetof(struct pt_regs , ip)); +unsigned short ss = uregs->ss; +unsigned short cs = uregs->cs; + +fred_info (uregs)->edata =fred_event_data (regs); + uregs->ssx = regs->ssx; + uregs->ss = ss; + uregs->csx = regs->csx; + uregs->current_stack_level =0 ; + uregs->cs = cs;Hello
If the ERETU instruction had tried to return from NMI to ring3 and just faulted, is NMI still blocked?
Quoted 6 lines
We know that IRET unconditionally enables NMI, but I can't find any clue in the FRED's manual.
In the pseudocode of ERETU in the manual, it seems that NMI is only enabled when ERETU succeeds with bit28 in csx set. If so, this code will fail to reenable NMI if bit28 is not explicitly re-set in csx.
IRET clearing NMI blocking is the source of an immense amount of grief, and ultimately the reason why Linux and others can't use supervisor shadow stacks at the moment.
Quoted 7 lines
Changing this property, so NMIs only get unblocked on successful execution of an ERET{S,U}, was a key demand of the FRED spec.
i.e. until you have successfully ERET*'d, you're still logically in the NMI handler and NMIs need to remain blocked even when handling the #GP from a bad ERET.
Handling of the #GP for a bad ERETU can be rescheduled. It is not OK to reschedule with NMI blocked.
I think "regs->nmi = 1;" (not uregs->nmi) can fix the problem.
H. Peter Anvin
maintainerRe: [PATCH v5 28/34] x86/fred: fixup fault on ERETU by jumping to fred_entrypoint_user
Quoted 41 lines · Lai Jiangshan
Quoted 32 lines · On Fri, Mar 17, 2023 at 5:56 PM <[email protected]> wrote:
Quoted 19 lines · Lai Jiangshan
Quoted 4 lines
arch/x86/mm/extable.c +4
-0+#ifdef CONFIG_X86_FRED +static bool ex_handler_eretu (const struct exception_table_entry *fixup, +struct pt_regs *regs,unsigned long error_code) +{arch/x86/mm/extable.c static bool ex_handler_ucopy_len(const struct exception_table_entry *fixup, · line 212
Quoted 10 lines
arch/x86/mm/extable.c +10
-0+struct pt_regs *uregs = (struct pt_regs *)(regs->sp - offsetof(struct pt_regs , ip)); +unsigned short ss = uregs->ss; +unsigned short cs = uregs->cs; + +fred_info (uregs)->edata =fred_event_data (regs); + uregs->ssx = regs->ssx; + uregs->ss = ss; + uregs->csx = regs->csx; + uregs->current_stack_level =0 ; + uregs->cs = cs;Hello
If the ERETU instruction had tried to return from NMI to ring3 and just faulted, is NMI still blocked?
Quoted 6 lines
We know that IRET unconditionally enables NMI, but I can't find any clue in the FRED's manual.
In the pseudocode of ERETU in the manual, it seems that NMI is only enabled when ERETU succeeds with bit28 in csx set. If so, this code will fail to reenable NMI if bit28 is not explicitly re-set in csx.
IRET clearing NMI blocking is the source of an immense amount of grief, and ultimately the reason why Linux and others can't use supervisor shadow stacks at the moment.
Quoted 7 lines
Changing this property, so NMIs only get unblocked on successful execution of an ERET{S,U}, was a key demand of the FRED spec.
i.e. until you have successfully ERET*'d, you're still logically in the NMI handler and NMIs need to remain blocked even when handling the #GP from a bad ERET.
Quoted 5 lines
Handling of the #GP for a bad ERETU can be rescheduled. It is not OK to reschedule with NMI blocked.
I think "
regs->nmi= 1;" (noturegs->nmi) can fix the problem.
You are quite correct, since what we want here is to emulate having taken the fault in user space – which meant that NMI would have been re-enabled by the never-executed return.
I think the "best" solution is:
regs->nmi = uregs->nmi;uregs->nmi = 0;
... as enabling NMI is expected to have a performance penalty (being the less common case, an implementation which has a performance difference at all would want to optimize the non-NMI case), and I believe the compiler should be able to at least mostly fold those operations into ones it is doing anyway.
H. Peter Anvin
maintainerRe: [PATCH v5 28/34] x86/fred: fixup fault on ERETU by jumping to fred_entrypoint_user
In reply to [email protected]Quoted 31 lines · On March 17, 2023 2:55:44 AM PDT, [email protected] wrote:
Quoted 19 lines · Lai Jiangshan
Quoted 4 lines
arch/x86/mm/extable.c +4
-0+#ifdef CONFIG_X86_FRED +static bool ex_handler_eretu (const struct exception_table_entry *fixup, +struct pt_regs *regs,unsigned long error_code) +{arch/x86/mm/extable.c static bool ex_handler_ucopy_len(const struct exception_table_entry *fixup, · line 212
Quoted 10 lines
arch/x86/mm/extable.c +10
-0+struct pt_regs *uregs = (struct pt_regs *)(regs->sp - offsetof(struct pt_regs , ip)); +unsigned short ss = uregs->ss; +unsigned short cs = uregs->cs; + +fred_info (uregs)->edata =fred_event_data (regs); + uregs->ssx = regs->ssx; + uregs->ss = ss; + uregs->csx = regs->csx; + uregs->current_stack_level =0 ; + uregs->cs = cs;Hello
If the ERETU instruction had tried to return from NMI to ring3 and just faulted, is NMI still blocked?
Quoted 6 lines
We know that IRET unconditionally enables NMI, but I can't find any clue in the FRED's manual.
In the pseudocode of ERETU in the manual, it seems that NMI is only enabled when ERETU succeeds with bit28 in csx set. If so, this code will fail to reenable NMI if bit28 is not explicitly re-set in csx.
IRET clearing NMI blocking is the source of an immense amount of grief, and ultimately the reason why Linux and others can't use supervisor shadow stacks at the moment.
Quoted 8 lines
Changing this property, so NMIs only get unblocked on successful execution of an ERET{S,U}, was a key demand of the FRED spec.
i.e. until you have successfully ERET*'d, you're still logically in the NMI handler and NMIs need to remain blocked even when handling the #GP from a bad ERET.
~Andrew
This is correct.
RE: [PATCH v5 28/34] x86/fred: fixup fault on ERETU by jumping to fred_entrypoint_user
In reply to Lai JiangshanQuoted 16 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+#ifdef CONFIG_X86_FRED +static bool ex_handler_eretu(const struct exception_table_entry *fixup, + struct pt_regs *regs, unsigned long +error_code) { + struct pt_regs *uregs = (struct pt_regs *)(regs->sp - offsetof(struct pt_regs,ip));
arch/x86/mm/extable.c static bool ex_handler_ucopy_len(const struct exception_table_entry *fixup, · line 212
Quoted 9 lines
arch/x86/mm/extable.c +9
-0+unsigned short ss = uregs->ss; +unsigned short cs = uregs->cs; + +fred_info (uregs)->edata =fred_event_data (regs); + uregs->ssx = regs->ssx; + uregs->ss = ss; + uregs->csx = regs->csx; + uregs->current_stack_level =0 ; + uregs->cs = cs;
Quoted 4 lines
Hello
If the ERETU instruction had tried to return from NMI to ring3 and just faulted, is NMI still blocked?
Do you mean the NMI FRED stack frame contains an invalid ring3 context?
PATCH v5 29/34 x86/ia32: do not modify the DPL bits for a null selector
In reply to Xin LiWhen a null selector is to be loaded into a segment register, reload_segments() sets its DPL bits to 3. Later when the IRET instruction loads it, it zeros the segment register. The two operations offset each other to actually effect a nop.
Unlike IRET, ERETU does not make any of DS, ES, FS, or GS null if it is found to have DPL < 3. It is expected that a FRED-enabled operating system will return to ring 3 (in compatibility mode) only when those segments all have DPL = 3.
Thus when FRED is enabled, we end up with having 3 in a segment register even when it is initially set to 0.
Fix it by not modifying the DPL bits for a null selector.
--- arch/x86/kernel/signal_32.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)
arch/x86/kernel/signal_32.c +13 -8
@@ -36,22 +36,27 @@#ifdef CONFIG_IA32_EMULATION #include <asm/ia32_unistd.h> +static inline u16 usrseg (u16 sel) +{ +return sel <=3 ? sel : sel |3 ; +} +static inline void reload_segments (struct sigcontext_32 *sc) {unsigned int cur;savesegment (gs, cur); -if ((sc->gs |0x03 ) != cur) -load_gs_index (sc->gs |0x03 ); +if (usrseg (sc->gs) != cur) +load_gs_index (usrseg (sc->gs));savesegment (fs, cur); -if ((sc->fs |0x03 ) != cur) -loadsegment (fs, sc->fs |0x03 ); +if (usrseg (sc->fs) != cur) +loadsegment (fs,usrseg (sc->fs));savesegment (ds, cur); -if ((sc->ds |0x03 ) != cur) -loadsegment (ds, sc->ds |0x03 ); +if (usrseg (sc->ds) != cur) +loadsegment (ds,usrseg (sc->ds));savesegment (es, cur); -if ((sc->es |0x03 ) != cur) -loadsegment (es, sc->es |0x03 ); +if (usrseg (sc->es) != cur) +loadsegment (es,usrseg (sc->es)); }#define sigset32_t compat_sigset_t
Signature · 1 line
2.34.1
PATCH v5 30/34 x86/fred: allow FRED systems to use interrupt vectors 0x10-0x1f
In reply to Xin LiFrom: "H. Peter Anvin (Intel)" <[email protected]>
FRED inherits the Intel VT-x enhancement of classified events with a two-level event dispatch logic. The first-level dispatch is on the event type, and the second-level is on the event vector. This also means that vectors in different event types are orthogonal, thus, vectors 0x10-0x1f become available as hardware interrupts.
Enable interrupt vectors 0x10-0x1f on FRED systems (interrupt 0x80 is already enabled.) Most of these changes are about removing the assumption that the lowest-priority vector is hard-wired to 0x20.
--- arch/x86/include/asm/idtentry.h | 4 ++-- arch/x86/include/asm/irq.h | 5 +++++ arch/x86/include/asm/irq_vectors.h | 15 +++++++++++---- arch/x86/kernel/apic/apic.c | 11 ++++++++--- arch/x86/kernel/apic/vector.c | 8 +++++++- arch/x86/kernel/fred.c | 4 ++-- arch/x86/kernel/idt.c | 6 +++--- arch/x86/kernel/irq.c | 2 +- arch/x86/kernel/traps.c | 2 ++ 9 files changed, 41 insertions(+), 16 deletions(-)
arch/x86/include/asm/idtentry.h +2 -2
@@ -546,8 +546,8 @@ __visible noinstr void func(struct pt_regs *regs, \*/ .align IDT_ALIGNSYM_CODE_START (irq_entries_start ) - vector=FIRST_EXTERNAL_VECTOR - .rept NR_EXTERNAL_VECTORS + vector=FIRST_EXTERNAL_VECTOR_IDT + .rept FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR_IDT UNWIND_HINT_IRET_REGS 0 : ENDBR
arch/x86/include/asm/irq.h +5 -0
@@ -11,6 +11,11 @@#include <asm/apicdef.h> #include <asm/irq_vectors.h> +/* +* The first available IRQ vector +*/ +extern unsigned int __ro_after_init first_external_vector; +/* * The irq entry code is in the noinstr section and the start/end of * __irqentry_text is emitted via labels. Make the build fail if
arch/x86/include/asm/irq_vectors.h +11 -4
@@ -31,15 +31,23 @@/* * IDT vectors usable for external interrupt sources start at 0x20. -* (0x80 is the syscall vector, 0x30-0x3f are for ISA) +* (0x80 is the syscall vector, 0x30-0x3f are for ISA). +* +* With FRED we can also use 0x10-0x1f even though those overlap +* exception vectors as FRED distinguishes exceptions and interrupts. +* Therefore, FIRST_EXTERNAL_VECTOR is no longer a constant. */ -#define FIRST_EXTERNAL_VECTOR 0x20 +#define FIRST_EXTERNAL_VECTOR_IDT 0x20 +#define FIRST_EXTERNAL_VECTOR_FRED 0x10 +#define FIRST_EXTERNAL_VECTOR first_external_vector/* * Reserve the lowest usable vector (and hence lowest priority) 0x20 for * triggering cleanup after irq migration. 0x21-0x2f will still be used * for device interrupts. */ +#define IRQ_MOVE_CLEANUP_VECTOR_IDT FIRST_EXTERNAL_VECTOR_IDT +#define IRQ_MOVE_CLEANUP_VECTOR_FRED FIRST_EXTERNAL_VECTOR_FRED#define IRQ_MOVE_CLEANUP_VECTOR FIRST_EXTERNAL_VECTOR#define IA32_SYSCALL_VECTOR 0x80
@@ -48,7 +56,7 @@* Vectors 0x30-0x3f are used for ISA interrupts. * round up to the next 16-vector boundary */ -#define ISA_IRQ_VECTOR (irq) (((FIRST_EXTERNAL_VECTOR + 16) & ~15) + irq) +#define ISA_IRQ_VECTOR (irq) (((FIRST_EXTERNAL_VECTOR_IDT + 16) & ~15) + irq)/* * Special IRQ vectors used by the SMP architecture, 0xf0-0xff
@@ -114,7 +122,6 @@#define FIRST_SYSTEM_VECTOR NR_VECTORS#endif -#define NR_EXTERNAL_VECTORS (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR)#define NR_SYSTEM_VECTORS (NR_VECTORS - FIRST_SYSTEM_VECTOR)/*
arch/x86/kernel/apic/apic.c +8 -3
@@ -1621,12 +1621,17 @@ static void setup_local_APIC(void)/* * Set Task Priority to 'accept all except vectors 0-31'. An APIC * vector in the 16-31 range could be delivered if TPR == 0, but we -* would think it's an exception and terrible things will happen. We -* never change this later on. +* would think it's an exception and terrible things will happen, +* unless we are using FRED in which case interrupts and +* exceptions are distinguished by type code. +* +* We never change this later on. */ +BUG_ON (!first_external_vector); + value =apic_read (APIC_TASKPRI ); value &= ~APIC_TPRI_MASK ; - value |=0x10 ; + value |= (first_external_vector -0x10 ) &APIC_TPRI_MASK ;apic_write (APIC_TASKPRI , value);/* Clear eventually stale ISR/IRR bits */
arch/x86/kernel/apic/vector.c +7 -1
@@ -46,6 +46,7 @@ static struct irq_matrix *vector_matrix;#ifdef CONFIG_SMP static DEFINE_PER_CPU(struct hlist_head, cleanup_list);#endif +unsigned int first_external_vector =FIRST_EXTERNAL_VECTOR_IDT ;void lock_vector_lock (void ) {
@@ -796,7 +797,12 @@ int __init arch_early_irq_init(void)* Allocate the vector matrix allocator data structure and limit the * search area. */ - vector_matrix =irq_alloc_matrix (NR_VECTORS ,FIRST_EXTERNAL_VECTOR , +if (cpu_feature_enabled (X86_FEATURE_FRED )) + first_external_vector =FIRST_EXTERNAL_VECTOR_FRED ; +else + first_external_vector =FIRST_EXTERNAL_VECTOR_IDT ; + + vector_matrix =irq_alloc_matrix (NR_VECTORS , first_external_vector,FIRST_SYSTEM_VECTOR );BUG_ON (!vector_matrix);
arch/x86/kernel/fred.c +2 -2
@@ -51,7 +51,7 @@ void __init fred_setup_apic(void) {int i; -for (i =0 ; i <FIRST_EXTERNAL_VECTOR ; i++) +for (i =0 ; i <FIRST_EXTERNAL_VECTOR_FRED ; i++)set_bit (i, system_vectors);/*
@@ -60,7 +60,7 @@ void __init fred_setup_apic(void)* /proc/interrupts. */ #ifdef CONFIG_SMP -set_bit (IRQ_MOVE_CLEANUP_VECTOR , system_vectors); +set_bit (IRQ_MOVE_CLEANUP_VECTOR_FRED , system_vectors);#endif for (i =0 ; i <NR_SYSTEM_VECTORS ; i++) {
arch/x86/kernel/idt.c +3 -3
@@ -131,7 +131,7 @@ static const __initconst struct idt_data apic_idts[] = {INTG (RESCHEDULE_VECTOR , asm_sysvec_reschedule_ipi),INTG (CALL_FUNCTION_VECTOR , asm_sysvec_call_function),INTG (CALL_FUNCTION_SINGLE_VECTOR , asm_sysvec_call_function_single), -INTG (IRQ_MOVE_CLEANUP_VECTOR , asm_sysvec_irq_move_cleanup), +INTG (IRQ_MOVE_CLEANUP_VECTOR_IDT , asm_sysvec_irq_move_cleanup), INTG(REBOOT_VECTOR, asm_sysvec_reboot), #endif
@@ -274,13 +274,13 @@ static void __init idt_map_in_cea(void)*/ void __init idt_setup_apic_and_irq_gates (void ) { -int i =FIRST_EXTERNAL_VECTOR ; +int i =FIRST_EXTERNAL_VECTOR_IDT ;void *entry;idt_setup_from_table (idt_table, apic_idts,ARRAY_SIZE (apic_idts), true);for_each_clear_bit_from (i, system_vectors,FIRST_SYSTEM_VECTOR ) { - entry = irq_entries_start +IDT_ALIGN * (i -FIRST_EXTERNAL_VECTOR ); + entry = irq_entries_start +IDT_ALIGN * (i -FIRST_EXTERNAL_VECTOR_IDT );set_intr_gate (i, entry); }
arch/x86/kernel/irq.c +1 -1
@@ -359,7 +359,7 @@ void fixup_irqs(void)* vector_lock because the cpu is already marked !online, so * nothing else will touch it. */ -for (vector =FIRST_EXTERNAL_VECTOR ; vector <NR_VECTORS ; vector++) { +for (vector = first_external_vector; vector <NR_VECTORS ; vector++) {if (IS_ERR_OR_NULL (__this_cpu_read (vector_irq[vector])))continue ;
arch/x86/kernel/traps.c +2 -0
@@ -1544,6 +1544,8 @@ DEFINE_IDTENTRY_IRQ(spurious_interrupt)pr_info ("Spurious interrupt (vector 0x%x) on CPU#%d, should never happen.\n" , vector, smp_processor_id()); } + +unsigned int first_external_vector =FIRST_EXTERNAL_VECTOR_IDT ;#endif /*
Signature · 1 line
2.34.1
PATCH v5 31/34 x86/fred: allow dynamic stack frame size
In reply to Xin LiA FRED stack frame could contain different amount of information for different event types, or perhaps even for different instances of the same event type. Thus we need to eliminate the need of any advance information of the stack frame size to allow dynamic stack frame size.
Implement it through:
1) add a new field user_pt_regs to thread_info, and initialize it with a pointer to a virtual pt_regs structure at the top of a thread stack.
2) save a pointer to the user-space pt_regs structure created by fred_entrypoint_user() to user_pt_regs in fred_entry_from_user().
3) initialize the init_thread_info's user_pt_regs with a pointer to a virtual pt_regs structure at the top of init stack.
This approach also works for IDT, thus we unify the code.
--- arch/x86/entry/entry_32.S | 2 +- arch/x86/entry/entry_fred.c | 2 ++ arch/x86/include/asm/entry-common.h | 3 +++ arch/x86/include/asm/processor.h | 12 +++------ arch/x86/include/asm/switch_to.h | 3 +-- arch/x86/include/asm/thread_info.h | 41 ++++------------------------- arch/x86/kernel/head_32.S | 3 +-- arch/x86/kernel/process.c | 5 ++++ kernel/fork.c | 6 +++++ 9 files changed, 27 insertions(+), 50 deletions(-)
arch/x86/entry/entry_32.S +1 -1
@@ -1244,7 +1244,7 @@ SYM_CODE_START(rewind_stack_and_make_dead) xorl %ebp, %ebp movl PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %esi - leal -TOP_OF_KERNEL_STACK_PADDING-PTREGS_SIZE(%esi), %esp + leal -PTREGS_SIZE(%esi), %esp call make_task_dead 1: jmp 1b
arch/x86/entry/entry_fred.c +2 -0
@@ -178,6 +178,8 @@ __visible noinstr void fred_entry_from_user(struct pt_regs *regs) [EVENT_TYPE_OTHER] = fred_syscall_slow }; + current->thread_info.user_pt_regs = regs; +/* * FRED employs a two-level event dispatch mechanism, with * the first-level on the type of an event and the second-level
arch/x86/include/asm/entry-common.h +3 -0
@@ -12,6 +12,9 @@/* Check that the stack and regs on entry from user mode are sane. */ static __always_inline void arch_enter_from_user_mode(struct pt_regs *regs) { +if (!cpu_feature_enabled (X86_FEATURE_FRED )) + current->thread_info.user_pt_regs = regs; +if (IS_ENABLED (CONFIG_DEBUG_ENTRY )) {/* * Make sure that the entry code gave us a sensible EFLAGS
arch/x86/include/asm/processor.h +3 -9
@@ -626,17 +626,11 @@ static inline void spin_lock_prefetch(const void *x)prefetchw (x); } -#define TOP_OF_INIT_STACK ((unsigned long)&init_stack + sizeof(init_stack) - \ - TOP_OF_KERNEL_STACK_PADDING) +#define TOP_OF_INIT_STACK ((unsigned long)&init_stack + sizeof(init_stack)) -#define task_top_of_stack (task) ((unsigned long)(task_pt_regs(task) + 1)) +#define task_top_of_stack (task) ((unsigned long)task_stack_page(task) + THREAD_SIZE) -#define task_pt_regs (task) \ -({ \ - unsigned long __ptr = (unsigned long)task_stack_page(task); \ - __ptr += THREAD_SIZE - TOP_OF_KERNEL_STACK_PADDING; \ - ((struct pt_regs *)__ptr) - 1; \ -}) +#define task_pt_regs (task) ((task)->thread_info.user_pt_regs)#ifdef CONFIG_X86_32 #define INIT_THREAD { \
arch/x86/include/asm/switch_to.h +1 -2
@@ -72,8 +72,7 @@ static inline void update_task_stack(struct task_struct *task)/* * Will use WRMSRNS/WRMSRLIST for performance once it's upstreamed. */ -wrmsrl (MSR_IA32_FRED_RSP0 , -task_top_of_stack (task) +TOP_OF_KERNEL_STACK_PADDING ); +wrmsrl (MSR_IA32_FRED_RSP0 ,task_top_of_stack (task)); }else if (cpu_feature_enabled (X86_FEATURE_XENPV )) {/* Xen PV enters the kernel on the thread stack. */ load_sp0 (task_top_of_stack (task));
arch/x86/include/asm/thread_info.h +5 -36
70 lines not on this page · raw message
arch/x86/kernel/head_32.S +1 -2
@@ -517,8 +517,7 @@ SYM_DATA_END(initial_page_table) * reliably detect the end of the stack. */ SYM_DATA(initial_stack, - .long init_thread_union + THREAD_SIZE - - SIZEOF_PTREGS - TOP_OF_KERNEL_STACK_PADDING) + .long init_thread_union + THREAD_SIZE - SIZEOF_PTREGS) __INITRODATA int_msg:
arch/x86/kernel/process.c +5 -0
@@ -98,6 +98,11 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)return 0 ; } +void arch_init_user_pt_regs (struct task_struct *tsk) +{ + tsk->thread_info.user_pt_regs = (struct pt_regs *)task_top_of_stack (tsk)-1 ; +} +#ifdef CONFIG_X86_64 void arch_release_task_struct (struct task_struct *tsk) {
kernel/fork.c +6 -0
@@ -958,6 +958,10 @@ int __weak arch_dup_task_struct(struct task_struct *dst,return 0 ; } +void __weak arch_init_user_pt_regs (struct task_struct *tsk) +{ +} +void set_task_stack_end_magic (struct task_struct *tsk) {unsigned long *stackend;
@@ -985,6 +989,8 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)if (err) goto free_tsk; +arch_init_user_pt_regs (tsk); +#ifdef CONFIG_THREAD_INFO_IN_TASK refcount_set (&tsk->stack_refcount,1 );#endif
Signature · 1 line
2.34.1
PATCH v5 32/34 x86/fred: disable FRED by default in its early stage
In reply to Xin LiDisable FRED by default in its early stage.
To enable FRED, a new kernel command line option "fred" needs to be added.
--- Documentation/admin-guide/kernel-parameters.txt | 4 ++++ arch/x86/kernel/cpu/common.c | 3 +++ 2 files changed, 7 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +4 -0
@@ -1498,6 +1498,10 @@ Warning: use of this parameter will taint the kernel and may cause unknown problems. + fred + Forcefully enable flexible return and event delivery, + which is otherwise disabled by default. + ftrace=[tracer] [FTRACE] will set and start the specified tracer as early as possible in order to facilitate early
arch/x86/kernel/cpu/common.c +3 -0
@@ -1467,6 +1467,9 @@ static void __init cpu_parse_early_param(void)char *argptr = arg, *opt;int arglen, taint =0 ; +if (!cmdline_find_option_bool (boot_command_line,"fred" )) +setup_clear_cpu_cap (X86_FEATURE_FRED ); +#ifdef CONFIG_X86_32 if (cmdline_find_option_bool (boot_command_line,"no387" )) #ifdef CONFIG_MATH_EMULATION
Signature · 1 line
2.34.1
PATCH v5 33/34 KVM: x86/vmx: call external_interrupt() to handle IRQ in IRQ caused VM exits
In reply to Xin LiWhen FRED is enabled, IDT is gone, thus call external_interrupt() to handle IRQ in IRQ caused VM exits.
Create an event return stack frame with the host context immediately after a VM exit for calling external_interrupt(). All other fields of the pt_regs structure are cleared to 0. Refer to the discussion about the register values in the pt_regs structure at:
https://lore.kernel.org/kvm/[email protected]/
---
Changes since v4:
*) Do NOT use the term "injection", which in the KVM context means to
reinject an event into the guest (Sean Christopherson).
*) Use cs/ss instead of csx/ssx when initializing the pt_regs structure
for calling external_interrupt(), otherwise it breaks i386 build.
--- arch/x86/kvm/vmx/vmx.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +21 -1
@@ -47,6 +47,7 @@#include <asm/mshyperv.h> #include <asm/mwait.h> #include <asm/spec-ctrl.h> +#include <asm/traps.h> #include <asm/virtext.h> #include <asm/vmx.h>
@@ -6923,7 +6924,26 @@ static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu)return ;kvm_before_interrupt (vcpu,KVM_HANDLING_IRQ ); -vmx_do_interrupt_irqoff (gate_offset (desc)); +if (cpu_feature_enabled (X86_FEATURE_FRED )) { +struct vcpu_vmx *vmx =to_vmx (vcpu); +struct pt_regs regs = {}; + +/* +* Create an event return stack frame with the +* host context immediately after a VM exit. +* +* All other fields of the pt_regs structure are +* cleared to 0. +*/ + regs.ss = __KERNEL_DS; + regs.sp = vmx->loaded_vmcs->host_state.rsp; + regs.flags =X86_EFLAGS_FIXED ; + regs.cs = __KERNEL_CS; + regs.ip = (unsigned long )vmx_vmexit; + +external_interrupt (®s, vector); + }else +vmx_do_interrupt_irqoff (gate_offset (desc));kvm_after_interrupt (vcpu); vcpu->arch.at_instruction_boundary = true;
Signature · 1 line
2.34.1
Sean Christopherson
maintainerRe: [PATCH v5 33/34] KVM: x86/vmx: call external_interrupt() to handle IRQ in IRQ caused VM exits
arch/x86/kvm/vmx/vmx.c static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu) · line 6933
Quoted 12 lines · Xin Li
arch/x86/kvm/vmx/vmx.c +7
-1@@ -6923,7 +6924,26 @@ static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu)return ;kvm_before_interrupt (vcpu,KVM_HANDLING_IRQ ); -vmx_do_interrupt_irqoff (gate_offset (desc)); +if (cpu_feature_enabled (X86_FEATURE_FRED )) { +struct vcpu_vmx *vmx =to_vmx (vcpu); +struct pt_regs regs = {}; + +/* +* Create an event return stack frame with the +* host context immediately after a VM exit.
Why snapshot the context immediately after VM-Exit? It diverges from what is done in the non-FRED path, and it seems quite misleading and maybe even dangerous. The RSP and RIP values are long since gone, e.g. if something explodes, the stack trace will be outright wrong.
arch/x86/kvm/vmx/vmx.c static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu) · line 6944
Quoted 11 lines
arch/x86/kvm/vmx/vmx.c +11
-0+* +* All other fields of the pt_regs structure are +* cleared to 0. +*/ + regs.ss = __KERNEL_DS; + regs.sp = vmx->loaded_vmcs->host_state.rsp; + regs.flags =X86_EFLAGS_FIXED ; + regs.cs = __KERNEL_CS; + regs.ip = (unsigned long )vmx_vmexit; + +external_interrupt (®s, vector);
I assume FRED still uses the stack, so why not do something similar to vmx_do_interrupt_irqoff() and build @regs after an explicit CALL? Might even be possible to share some/all of VMX_DO_EVENT_IRQOFF.
arch/x86/kvm/vmx/vmx.c static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu) · line 6945
Quoted 1 line
arch/x86/kvm/vmx/vmx.c +1
-0+ }else
Curly braces needed since the first half has 'em.
Quoted 7 lines
arch/x86/kvm/vmx/vmx.c +1
-0+vmx_do_interrupt_irqoff (gate_offset (desc));kvm_after_interrupt (vcpu); vcpu->arch.at_instruction_boundary = true;Signature · 1 line
2.34.1
PATCH v5 34/34 KVM: x86/vmx: execute "int $2" to handle NMI in NMI caused VM exits when FRED is enabled
In reply to Xin LiExecute "int $2" to handle NMI in NMI caused VM exits when FRED is enabled.
Like IRET for IDT, ERETS/ERETU are required to end the NMI handler for FRED to unblock NMI ASAP (w/ bit 28 of CS set). And there are 2 approaches to invoke the FRED NMI handler:
1) execute "int $2", let the h/w do the job.
2) create a FRED NMI stack frame on the current kernel stack with ASM, and then jump to fred_entrypoint_kernel in arch/x86/entry/entry_64_fred.S.
1) is preferred as we want less ASM.
---
Changes since v4:
*) Do NOT use the term "injection", which in the KVM context means to
reinject an event into the guest (Sean Christopherson).
*) Add the explanation of why to execute "int $2" to invoke the NMI handler
in NMI caused VM exits (Sean Christopherson).
--- arch/x86/kvm/vmx/vmx.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +10 -1
@@ -7229,7 +7229,16 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu,if ((u16 )vmx->exit_reason.basic ==EXIT_REASON_EXCEPTION_NMI &&is_nmi (vmx_get_intr_info (vcpu))) {kvm_before_interrupt (vcpu,KVM_HANDLING_NMI ); -vmx_do_nmi_irqoff (); +/* +* Like IRET for IDT, ERETS/ERETU are required to end the NMI +* handler for FRED to unblock NMI ASAP (w/ bit 28 of CS set). +* +* Invoke the FRED NMI handler through executing "int $2". +*/ +if (cpu_feature_enabled (X86_FEATURE_FRED )) + asmvolatile ("int $2" ); +else +vmx_do_nmi_irqoff ();kvm_after_interrupt (vcpu); }
Signature · 1 line
2.34.1
RE: [PATCH v5 34/34] KVM: x86/vmx: execute "int $2" to handle NMI in NMI caused VM exits when FRED is enabled
Quoted 13 lines
Execute "int $2" to handle NMI in NMI caused VM exits when FRED is enabled.
Like IRET for IDT, ERETS/ERETU are required to end the NMI handler for FRED to unblock NMI ASAP (w/ bit 28 of CS set). And there are 2 approaches to invoke the FRED NMI handler:
1) execute "int $2", let the h/w do the job.
2) create a FRED NMI stack frame on the current kernel stack with ASM, and then jump tofred_entrypoint_kernelinarch/x86/entry/entry_64_fred.S.1) is preferred as we want less ASM.
Tested-by Shan Kang <[email protected]>Signed-off-by Xin Li <[email protected]>
Quoted 7 lines
---
Changes since v4:
*) Do NOT use the term "injection", which in the KVM context means to
reinject an event into the guest (Sean Christopherson).
*) Add the explanation of why to execute "int $2" to invoke the NMI handler
in NMI caused VM exits (Sean Christopherson).
Sean,
Do you have any further issue with the last 2 VMX patches?
If not, would you ack them?
Thanks!
Xin
Quoted 29 lines
--- arch/x86/kvm/vmx/vmx.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)arch/x86/kvm/vmx/vmx.c +10
-2@@ -7229,7 +7229,16 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, if ((u16)vmx->exit_reason.basic == EXIT_REASON_EXCEPTION_NMI && is_nmi(vmx_get_intr_info(vcpu))) {kvm_before_interrupt (vcpu,KVM_HANDLING_NMI ); -vmx_do_nmi_irqoff (); +/* +* Like IRET for IDT, ERETS/ERETU are required to end the NMI +* handler for FRED to unblock NMI ASAP (w/ bit 28 of CS set). +* +* Invoke the FRED NMI handler through executing "int $2". +*/ +if (cpu_feature_enabled (X86_FEATURE_FRED )) + asmvolatile ("int $2" ); +else +vmx_do_nmi_irqoff ();kvm_after_interrupt (vcpu); } --2.34.1
Sean Christopherson
maintainerRe: [PATCH v5 34/34] KVM: x86/vmx: execute "int $2" to handle NMI in NMI caused VM exits when FRED is enabled
In reply to Xin LiQuoted 4 lines · Xin Li
Execute "int $2" to handle NMI in NMI caused VM exits when FRED is enabled.
Like IRET for IDT, ERETS/ERETU are required to end the NMI handler for FRED to unblock NMI ASAP (w/ bit 28 of CS set).
That's "CS" on the stack correct? Is bit 28 set manually by software, or is it set automatically by hardware? If it's set by hardware, does "int $2" actually set the bit since it's not a real NMI?
Quoted 7 lines
And there are 2 approaches to
invoke the FRED NMI handler:
1) execute "int $2", let the h/w do the job.
2) create a FRED NMI stack frame on the current kernel stack with ASM, and then jump tofred_entrypoint_kernelinarch/x86/entry/entry_64_fred.S.1) is preferred as we want less ASM.
Who is "we", and how much assembly are we talking about? E.g. I personally don't mind a trampoline in KVM if it's small and/or can share code with existing assembly subroutines.
Re: [PATCH v5 34/34] KVM: x86/vmx: execute "int $2" to handle NMI in NMI caused VM exits when FRED is enabled
Quoted 8 lines · Sean Christopherson
Quoted 4 lines · Xin Li
Execute "int $2" to handle NMI in NMI caused VM exits when FRED is enabled.
Like IRET for IDT, ERETS/ERETU are required to end the NMI handler for FRED to unblock NMI ASAP (w/ bit 28 of CS set).
That's "CS" on the stack correct? Is bit 28 set manually by software, or is it set automatically by hardware? If it's set by hardware, does "int $2" actually set the bit since it's not a real NMI?
int $2 had better not set it... This is the piece of state that is intended to cause everything which isn't a real NMI to nest properly inside a real NMI.
It is supposed to be set on delivery of an NMI, and act as the trigger for ERET{U,S} to drop the latch.
Software is can set it manually in a FRED-frame in order to explicitly unblock NMIs.
~Andrew
Sean Christopherson
maintainerRe: [PATCH v5 34/34] KVM: x86/vmx: execute "int $2" to handle NMI in NMI caused VM exits when FRED is enabled
Quoted 14 lines · On Wed, Mar 22, 2023, [email protected] wrote:
Quoted 8 lines · Sean Christopherson
Quoted 4 lines · Xin Li
Execute "int $2" to handle NMI in NMI caused VM exits when FRED is enabled.
Like IRET for IDT, ERETS/ERETU are required to end the NMI handler for FRED to unblock NMI ASAP (w/ bit 28 of CS set).
That's "CS" on the stack correct? Is bit 28 set manually by software, or is it set automatically by hardware? If it's set by hardware, does "int $2" actually set the bit since it's not a real NMI?
int $2 had better not set it...� This is the piece of state that is intended to cause everything which isn't a real NMI to nest properly inside a real NMI.
Quoted 5 lines
It is supposed to be set on delivery of an NMI, and act as the trigger for ERET{U,S} to drop the latch.
Software is can set it manually in a FRED-frame in order to explicitly unblock NMIs.
Ah, found this in patch 19. That hunk really belongs in this patch, because this patch is full of magic without that information.
+ /*
+ * VM exits induced by NMIs keep NMI blocked, and we do + * "int $2" to reinject the NMI w/ NMI kept being blocked. + * However "int $2" doesn't set the nmi bit in the FRED + * stack frame, so we explicitly set it to make sure a + * later ERETS will unblock NMI immediately. + */
+ regs->nmi = 1;
Organization aside, this seems to defeat the purpose of _not_ unconditionally unmasking NMIs on ERET since the kernel assumes any random "int $2" is coming from KVM after an NMI VM-Exit.
Eww, and "int $2" doesn't even go directly to fred_exc_nmi(), it trampolines through fred_sw_interrupt_kernel() first. Looks like "int $2" from userspace gets routed to a #GP, so at least that bit is handled.
I'm not dead set against the proposed approach, but IMO it's not obviously better than a bit of assembly to have a more direct call into the NMI handler.
RE: [PATCH v5 34/34] KVM: x86/vmx: execute "int $2" to handle NMI in NMI caused VM exits when FRED is enabled
Quoted 3 lines
Organization aside, this seems to defeat the purpose of _not_ unconditionally unmasking NMIs on ERET since the kernel assumes any random "int $2" is coming from KVM after an NMI VM-Exit.
I'm a bit confused. KVM VMX is the only component needing to execute "int $2" and it surely has NMI blocked after an NMI VM-exit.
Quoted 3 lines
Eww, and "int $2" doesn't even go directly to
fred_exc_nmi(), it trampolines throughfred_sw_interrupt_kernel()first. Looks like "int $2" from userspace gets routed to a #GP, so at least that bit is handled.
FRED does a 2-level dispatch, unless an event handler is on a hot path, we don't promote its handling. NMI seems not a frequent event.
Quoted 2 lines
I'm not dead set against the proposed approach, but IMO it's not obviously better than a bit of assembly to have a more direct call into the NMI handler.
I will give it a shot.
RE: [PATCH v5 34/34] KVM: x86/vmx: execute "int $2" to handle NMI in NMI caused VM exits when FRED is enabled
Quoted 5 lines
Quoted 2 lines
I'm not dead set against the proposed approach, but IMO it's not obviously better than a bit of assembly to have a more direct call into the NMI
handler.
I will give it a shot.
Hi Sean,
I got a working patch, before I resend the whole FRED patch set again, can you please check if this is what you're expecting?
When FRED is enabled, the x86 CPU always pushes an error code on the stack immediately after the return instruction address is pushed. To generate such a stack frame, call a trampoline function first to push the return instruction address on the stack, and the trampoline function then pushes an error code (0 for IRQ/NMI) and jump to fred_entrypoint_kernel.
I could have vmx_do_interrupt_trampoline jump to fred_entrypoint_kernel Instead of calling external_interrupt(), but that would reenter the noinstr text again (not a big problem but seems not preferred by Peter Z).
Thanks!
Xin
arch/x86/kvm/vmx/vmenter.S +83 -3
@@ -31,7 +31,7 @@ #define VCPU_R15 __VCPU_REGS_R15 * WORD_SIZE #endif -.macro VMX_DO_EVENT_IRQOFF call_insn call_target +.macro VMX_DO_EVENT_IRQOFF call_insn call_target fred=1 nmi=0 /* * Unconditionally create a stack frame, getting the correct RSP on the * stack (for x86-64) would take two instructions anyways, and RBP can
@@ -46,11 +46,34 @@ * creating the synthetic interrupt stack frame for the IRQ/NMI. */ and $-16, %rsp + + .if \fred + push $0 /* Reserved by FRED, must be 0 */ + push $0 /* FRED event data, 0 for NMI and external interrupts */ + + .if \nmi + mov $(2 << 32 | 2 << 48), %_ASM_AX /* NMI event type and vector */ + .else + mov %_ASM_ARG1, %_ASM_AX + shl $32, %_ASM_AX /* external interrupt vector */ + .endif + add $__KERNEL_DS, %_ASM_AX + bts $57, %_ASM_AX /* bit 57: 64-bit mode */ + push %_ASM_AX + .else push $__KERNEL_DS + .endif + push %rbp #endif pushf + .if \nmi + mov $__KERNEL_CS, %_ASM_AX + bts $28, %_ASM_AX /* set the NMI bit */ + push %_ASM_AX + .else push $__KERNEL_CS + .endif \call_insn \call_target /*
@@ -299,8 +322,19 @@ SYM_INNER_LABEL(vmx_vmexit, SYM_L_GLOBAL) SYM_FUNC_END(__vmx_vcpu_run) +SYM_FUNC_START(vmx_do_nmi_trampoline) +#ifdef CONFIG_X86_FRED + ALTERNATIVE "jmp .Lno_errorcode_push", "", X86_FEATURE_FRED + push $0 /* FRED error code, 0 for NMI */ + jmp fred_entrypoint_kernel +#endif + +.Lno_errorcode_push: + jmp asm_exc_nmi_kvm_vmx +SYM_FUNC_END(vmx_do_nmi_trampoline) + SYM_FUNC_START(vmx_do_nmi_irqoff) - VMX_DO_EVENT_IRQOFF call asm_exc_nmi_kvm_vmx + VMX_DO_EVENT_IRQOFF call vmx_do_nmi_trampoline nmi=1 SYM_FUNC_END(vmx_do_nmi_irqoff)
@@ -358,5 +392,51 @@ SYM_FUNC_END(vmread_error_trampoline) #endif SYM_FUNC_START(vmx_do_interrupt_irqoff) - VMX_DO_EVENT_IRQOFF CALL_NOSPEC _ASM_ARG1 + VMX_DO_EVENT_IRQOFF CALL_NOSPEC _ASM_ARG1 fred=0 SYM_FUNC_END(vmx_do_interrupt_irqoff) + +#ifdef CONFIG_X86_64 +SYM_FUNC_START(vmx_do_interrupt_trampoline) + push $0 /* FRED error code, 0 for NMI and external interrupts */ + push %rdi + push %rsi + push %rdx + push %rcx + push %rax + push %r8 + push %r9 + push %r10 + push %r11 + push %rbx + push %rbp + push %r12 + push %r13 + push %r14 + push %r15 + + movq %rsp, %rdi /* %rdi -> pt_regs */ + call external_interrupt + + pop %r15 + pop %r14 + pop %r13 + pop %r12 + pop %rbp + pop %rbx + pop %r11 + pop %r10 + pop %r9 + pop %r8 + pop %rax + pop %rcx + pop %rdx + pop %rsi + pop %rdi + addq $8,%rsp /* Drop FRED error code */ + RET +SYM_FUNC_END(vmx_do_interrupt_trampoline) + +SYM_FUNC_START(vmx_do_fred_interrupt_irqoff) + VMX_DO_EVENT_IRQOFF call vmx_do_interrupt_trampoline +SYM_FUNC_END(vmx_do_fred_interrupt_irqoff) +#endif
arch/x86/kvm/vmx/vmx.c +7 -1
@@ -6875,6 +6875,7 @@ static void vmx_apicv_post_state_restore(struct kvm_vcpu *vcpu) }void vmx_do_interrupt_irqoff (unsigned long entry); +void vmx_do_fred_interrupt_irqoff (unsigned int vector);void vmx_do_nmi_irqoff (void );static void handle_nm_fault_irqoff (struct kvm_vcpu *vcpu)
@@ -6923,7 +6924,12 @@ static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu)return ;kvm_before_interrupt (vcpu,KVM_HANDLING_IRQ ); -vmx_do_interrupt_irqoff (gate_offset (desc)); +#ifdef CONFIG_X86_64 +if (cpu_feature_enabled (X86_FEATURE_FRED )) +vmx_do_fred_interrupt_irqoff (vector); + else +#endif +vmx_do_interrupt_irqoff (gate_offset (desc));kvm_after_interrupt (vcpu); vcpu->arch.at_instruction_boundary = true;
RE: [PATCH v5 34/34] KVM: x86/vmx: execute "int $2" to handle NMI in NMI caused VM exits when FRED is enabled
In reply to Sean ChristophersonQuoted 6 lines
Quoted 2 lines
Like IRET for IDT, ERETS/ERETU are required to end the NMI handler for FRED to unblock NMI ASAP (w/ bit 28 of CS set).
That's "CS" on the stack correct? Is bit 28 set manually by software, or is it set automatically by hardware? If it's set by hardware, does "int $2" actually set the bit since it's not a real NMI?
Right, It's the "CS" on the stack. The bit 28 is set by the FRED NMI handler:
https://lore.kernel.org/lkml/[email protected]/
Upon a NMI delivery, the NMI bit is always set by H/W. However, "int $2" does NOT set it, thus we need to explicitly set it.
Quoted 11 lines
Quoted 7 lines
And there are 2 approaches to
invoke the FRED NMI handler:
1) execute "int $2", let the h/w do the job.
2) create a FRED NMI stack frame on the current kernel stack with ASM, and then jump tofred_entrypoint_kernelinarch/x86/entry/entry_64_fred.S.1) is preferred as we want less ASM.
Who is "we", and how much assembly are we talking about? E.g. I personally don't mind a trampoline in KVM if it's small and/or can share code with existing assembly subroutines.
I ever got such a comment:
https://lore.kernel.org/lkml/8735bpbhat.ffs@tglx/
However, if ASM is also okay, I can work on it. And I don't think the ASM code will be big.
We tested the v5 FRED patch set on the Intel Simics® Simulator and a machine with a 7th Intel(R) Core(TM) CPU.
Following are the Kselftest results on X86-64.
+--------------------------------------------+-------+-------+-------+-------+ | Config | Pass | Fail | Skip | Hang | +--------------------------------------------+-------+-------+-------+-------+ | the 7th Intel(R) Core(TM) CPU | 3078 | 458 | 734 | 5 | | 6.3.0-rc1+ | | | | | +--------------------------------------------+-------+-------+-------+-------+ | the 7th Intel(R) Core(TM) CPU | 3078 | 458 | 734 | 5 | | 6.3.0-rc1+ w/ FRED patch set | | | | | +--------------------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/o FRED model | 1888 | 271 | 2105 | 11 | | 6.3.0-rc1+ | | | | | +--------------------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/o FRED model | 1888 | 271 | 2105 | 11 | | 6.3.0-rc1+ w/ FRED patch set | | | | | +--------------------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/ FRED model | 1889 | 270 | 2105 | 11 | | 6.3.0-rc1+ | | | | | +--------------------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/ FRED model | 1889 | 270 | 2105 | 11 | | 6.3.0-rc1+ w/ FRED patch set FRED disabled | | | | | +--------------------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/ FRED model | 1888 | 270 | 2105 | 12 | | 6.3.0-rc1+ w/ FRED patch set | | | | | +--------------------------------------------+-------+-------+-------+-------+
The following issues are seen in this round of test.
+----------------+----------------+----------------+----------------+ | | x86:test_ | bpf:test_progs | x86:sysret | | | vsyscall_32 | | _rip_64 | +----------------+----------------+----------------+----------------+ | the 7th | | | | | Intel(R) | FAIL | FAIL | PASS | | Core(TM) CPU | | | | | 6.3.0-rc1+ | | | | +----------------+----------------+----------------+----------------+ | the 7th | | | | | Intel(R) | | | | | Core(TM) CPU | FAIL | FAIL | PASS | | 6.3.0-rc1+ w/ | | | | | FRED patch set | | | | +----------------+----------------+----------------+----------------+ | Intel Simics® | | | | | Simulator w/o | FAIL | FAIL | PASS | | FRED model | | | | | 6.3.0-rc1+ | | | | +----------------+----------------+----------------+----------------+ | Intel Simics® | | | | | Simulator w/o | | | | | FRED model | FAIL | FAIL | PASS | | 6.3.0-rc1+ w/ | | | | | FRED patch set | | | | +----------------+----------------+----------------+----------------+ | Intel Simics® | | | | | Simulator w/ | PASS | FAIL | PASS | | FRED model | | | | | 6.3.0-rc1+ | | | | +----------------+----------------+----------------+----------------+ | Intel Simics® | | | | | Simulator w/ | | | | | FRED model | PASS | FAIL | PASS | | 6.3.0-rc1+ w/ | | | | | FRED patch set | | | | | FRED disabled | | | | +----------------+----------------+----------------+----------------+ | Intel Simics® | | | | | Simulator w/ | | | | | FRED model | PASS | HANG | FAIL | | 6.3.0-rc1+ w/ | | | | | FRED patch set | | | | +----------------+----------------+----------------+----------------+
The test "x86:sysret_rip_64" is NOT a valid test on FRED, and there is a fix from Ammar Faizi after we discussed it in the LKML.
The test "bpf:test_progs" is still in investigation.
The "x86:test_vsyscall_32" is a regression since the v3 FRED patch set.
Thanks
--Shan
Quoted 173 lines · Xin Li
This patch set enables FRED for x86-64.
The Intel flexible return and event delivery (FRED) architecture defines simple
new transitions that change privilege level (ring transitions). The FRED architecture was designed with the following goals:
1) Improve overall performance and response time by replacing event delivery through the interrupt descriptor table (IDT event delivery) and event return by
the IRET instruction with lower latency transitions.
2) Improve software robustness by ensuring that event delivery establishes the full supervisor context and that event return establishes the full user context.The new transitions defined by the FRED architecture are FRED event delivery and,
for returning from events, two FRED return instructions. FRED event delivery can
effect a transition from ring 3 to ring 0, but it is used also to deliver events
incident to ring 0. One FRED instruction (ERETU) effects a return from ring 0 to
ring 3, while the other (ERETS) returns while remaining in ring 0.Search for the latest FRED spec in most search engines with this search pattern:
site:intel.com FRED (flexible return and event delivery) specification
As of now there is no publicly avaiable CPU supporting FRED, thus the Intel Simics® Simulator is used as software development and testing vehicles. And it can be downloaded from:
https://www.intel.com/content/www/us/en/developer/articles/tool/simics-simulator.html
To enable FRED, the Simics package 8112 QSP-CPU needs to be installed with CPU model configured as:
$cpu_comp_class = "x86-experimental-fred"Longer term, we should refactor common code shared by FRED and IDT into common shared files, and contain IDT code using a new config
CONFIG_X86_IDT.Changes since v4:
* Rebased against v6.3-rc1.
* Do NOT use the term "injection", which in the KVM context means to reinject an event into the guest (Sean Christopherson).
* Add the explanation of why to execute "int $2" to invoke the NMI handler in NMI caused VM exits (Sean Christopherson).
* Use cs/ss instead of csx/ssx when initializing thept_regsstructure for callingexternal_interrupt(), otherwise it breaks i386 build.Changes since v3:
* Callexternal_interrupt()to handle IRQ in IRQ caused VM exits.
* Execute "int $2" to handle NMI in NMI caused VM exits.
* Rename csl/ssl of thept_regsstructure to csx/ssx (x for extended) (Andrew Cooper).Changes since v2:
* Improve comments for changes inarch/x86/include/asm/idtentry.h.Changes since v1:
* call irqentry_nmi_{enter,exit}() in both IDT and FRED debug fault kernel handler (Peter Zijlstra).
* Initialize a FRED exception handler tofred_bad_event()instead of NULL if no FRED handler defined for an exception vector (Peter Zijlstra).
* Push calling irqentry_{enter,exit}() and instrumentation_{begin,end}() down into individual FRED exception handlers, instead of in the dispatch framework (Peter Zijlstra).H. Peter Anvin (Intel) (24):
x86/traps: letcommon_interrupt()handleIRQ_MOVE_CLEANUP_VECTORx86/traps: add a system interrupt table for system interrupt dispatch x86/traps: addexternal_interrupt()to dispatch external interrupts x86/cpufeature: add the cpu feature bit for FRED x86/opcode: add ERETU, ERETS instructions to x86-opcode-map x86/objtool: teach objtool about ERETU and ERETS
x86/cpu: addX86_CR4_FREDmacro
x86/fred: addKconfigoption for FRED (CONFIG_X86_FRED) x86/fred: ifCONFIG_X86_FREDis disabled, disable FRED support x86/cpu: add MSR numbers for FRED configuration
x86/fred: header file with FRED definitions
x86/fred: make unions for the cs and ss fields in structpt_regsx86/fred: reserve space for the FRED stack frame
x86/fred: add a page fault entry stub for FRED
x86/fred: add a debug fault entry stub for FRED
x86/fred: add a NMI entry stub for FRED
x86/fred: FRED entry/exit and dispatch code
x86/fred: FRED initialization code
x86/fred: updateMSR_IA32_FRED_RSP0during task switch x86/fred: letret_from_fork()jmp tofred_exit_userwhen FRED is
enabled
x86/fred: disallow the swapgs instruction when FRED is enabled x86/fred: no ESPFIX needed when FRED is enabled
x86/fred: allow single-step trap and NMI when starting a new thread x86/fred: allow FRED systems to use interrupt vectors 0x10-0x1fXin Li (10):
x86/traps: addinstall_system_interrupt_handler()x86/traps: exportexternal_interrupt()for VMX IRQ reinjection x86/fred: header file for event types
x86/fred: add a machine check entry stub for FRED x86/fred: fixup fault on ERETU by jumping tofred_entrypoint_userx86/ia32: do not modify the DPL bits for a null selector x86/fred: allow dynamic stack frame size
x86/fred: disable FRED by default in its early stage
KVM: x86/vmx: callexternal_interrupt()to handle IRQ in IRQ caused VM
exits
KVM: x86/vmx: execute "int $2" to handle NMI in NMI caused VM exits
when FRED is enabled.../admin-guide/kernel-parameters.txt | 4 + arch/x86/Kconfig | 9 + arch/x86/entry/Makefile | 5 +- arch/x86/entry/entry_32.S | 2 +- arch/x86/entry/entry_64.S | 5 + arch/x86/entry/entry_64_fred.S | 59 +++++ arch/x86/entry/entry_fred.c | 234 ++++++++++++++++++ arch/x86/entry/vsyscall/vsyscall_64.c | 2 +- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/entry-common.h | 3 + arch/x86/include/asm/event-type.h | 17 ++ arch/x86/include/asm/extable_fixup_types.h | 4 +- arch/x86/include/asm/fred.h | 131 ++++++++++ arch/x86/include/asm/idtentry.h | 76 +++++- arch/x86/include/asm/irq.h | 5 + arch/x86/include/asm/irq_vectors.h | 15 +- arch/x86/include/asm/msr-index.h | 13 +- arch/x86/include/asm/processor.h | 12 +- arch/x86/include/asm/ptrace.h | 36 ++- arch/x86/include/asm/switch_to.h | 10 +- arch/x86/include/asm/thread_info.h | 35 +-- arch/x86/include/asm/traps.h | 13 + arch/x86/include/asm/vmx.h | 17 +- arch/x86/include/uapi/asm/processor-flags.h | 2 + arch/x86/kernel/Makefile | 1 + arch/x86/kernel/apic/apic.c | 11 +- arch/x86/kernel/apic/vector.c | 8 +- arch/x86/kernel/cpu/acrn.c | 7 +- arch/x86/kernel/cpu/common.c | 88 ++++--- arch/x86/kernel/cpu/mce/core.c | 11 + arch/x86/kernel/cpu/mshyperv.c | 22 +- arch/x86/kernel/espfix_64.c | 8 + arch/x86/kernel/fred.c | 73 ++++++ arch/x86/kernel/head_32.S | 3 +- arch/x86/kernel/idt.c | 6 +- arch/x86/kernel/irq.c | 6 +- arch/x86/kernel/irqinit.c | 7 +- arch/x86/kernel/kvm.c | 4 +- arch/x86/kernel/nmi.c | 28 +++ arch/x86/kernel/process.c | 5 + arch/x86/kernel/process_64.c | 21 +- arch/x86/kernel/signal_32.c | 21 +- arch/x86/kernel/traps.c | 175 +++++++++++-- arch/x86/kvm/vmx/vmx.c | 33 ++- arch/x86/lib/x86-opcode-map.txt | 2 +- arch/x86/mm/extable.c | 28 +++ arch/x86/mm/fault.c | 20 +- drivers/xen/events/events_base.c | 5 +- kernel/fork.c | 6 + tools/arch/x86/include/asm/cpufeatures.h | 1 + .../arch/x86/include/asm/disabled-features.h | 8 +- tools/arch/x86/include/asm/msr-index.h | 13 +- tools/arch/x86/lib/x86-opcode-map.txt | 2 +- tools/objtool/arch/x86/decode.c | 19 +- 55 files changed, 1185 insertions(+), 175 deletions(-) create mode 100644 arch/x86/entry/entry_64_fred.S create mode 100644 arch/x86/entry/entry_fred.c create mode 100644 arch/x86/include/asm/event-type.h create mode 100644 arch/x86/include/asm/fred.h create mode 100644 arch/x86/kernel/fred.c
Quoted 46 lines
The following issues are seen in this round of test.
+----------------+----------------+----------------+----------------+ | | x86:test_ | bpf:test_progs | x86:sysret | | | vsyscall_32 | | _rip_64 | +----------------+----------------+----------------+----------------+ | the 7th | | | | | Intel(R) | FAIL | FAIL | PASS | | Core(TM) CPU | | | | | 6.3.0-rc1+ | | | | +----------------+----------------+----------------+----------------+ | the 7th | | | | | Intel(R) | | | | | Core(TM) CPU | FAIL | FAIL | PASS | | 6.3.0-rc1+ w/ | | | | | FRED patch set | | | | +----------------+----------------+----------------+----------------+ | Intel Simics® | | | | | Simulator w/o | FAIL | FAIL | PASS | | FRED model | | | | | 6.3.0-rc1+ | | | | +----------------+----------------+----------------+----------------+ | Intel Simics® | | | | | Simulator w/o | | | | | FRED model | FAIL | FAIL | PASS | | 6.3.0-rc1+ w/ | | | | | FRED patch set | | | | +----------------+----------------+----------------+----------------+ | Intel Simics® | | | | | Simulator w/ | PASS | FAIL | PASS | | FRED model | | | | | 6.3.0-rc1+ | | | | +----------------+----------------+----------------+----------------+ | Intel Simics® | | | | | Simulator w/ | | | | | FRED model | PASS | FAIL | PASS | | 6.3.0-rc1+ w/ | | | | | FRED patch set | | | | | FRED disabled | | | | +----------------+----------------+----------------+----------------+ | Intel Simics® | | | | | Simulator w/ | | | | | FRED model | PASS | HANG | FAIL | | 6.3.0-rc1+ w/ | | | | | FRED patch set | | | | +----------------+----------------+----------------+----------------+
Quoted 1 line
The "x86:test_vsyscall_32" is a regression since the v3 FRED patch set.
The "x86:test_vsyscall_32" test passes on the Simics FRED model, no matter whether FRED is enabled or not, because the Simics FRED model has the RDPID instruction support.
While on a Simics non-FRED model or the test bare metal machine, which don't have the RDPID instruction support, the test reads CPU ID from the GDT_ENTRY_CPUNODE entry, thus it fails due to:
https://lore.kernel.org/lkml/[email protected]/
Thanks!
Xin
We wanted to check whether there are KVM regressions with the v5 FRED patch set, and had a round of Kselftest on KVM guests.
Following are the results on X86-64.
+--------------------------------+-------+-------+-------+-------+ | Config | Pass | Fail | Skip | Hang | +--------------------------------+-------+-------+-------+-------+ | the 7th Intel(R) Core(TM) CPU | | | | | | | host 6.3.0-rc1+ w/ FRED | 2720 | 403 | 689 | 9 | | patch set FRED disabled | | | | | | | guest 6.3.0-rc1+ | | | | | +--------------------------------+-------+-------+-------+-------+ | the 7th Intel(R) Core(TM) CPU | | | | | | | host 6.3.0-rc1+ w/ FRED | 2720 | 403 | 689 | 9 | | patch set | guest 6.3.0-rc1+ | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/o | | | | | | FRED model | host 6.3.0-rc1+ | | 1403 | 277 | 2127 | 14 | | guest 6.3.0-rc1+ | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/o | | | | | | FRED model | host 6.3.0-rc1+ | | | | | | w/ FRED patch set | guest | 1403 | 277 | 2127 | 14 | | 6.3.0-rc1+ w/ FRED patch set | | | | | | FRED disabled | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/o | | | | | | FRED model | host 6.3.0-rc1+ | 1403 | 277 | 2127 | 14 | | w/ FRED patch set | guest | | | | | | 6.3.0-rc1+ w/ FRED patch set | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/ | | | | | | FRED model + host 6.3.0-rc1+ | | 1404 | 276 | 2127 | 14 | | guest 6.3.0-rc1+ w/ FRED patch | | | | | | set | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/ | | | | | | FRED model | host 6.3.0-rc1+ | 1404 | 276 | 2127 | 14 | | w/ FRED patch set FRED | | | | | | disabled | guest 6.3.0-rc1+ | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/ | | | | | | FRED model | host 6.3.0-rc1+ | | | | | | w/ FRED patch set FRED | 1404 | 276 | 2127 | 14 | | disabled | guest 6.3.0-rc1+ w/ | | | | | | FRED patch set | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/ | | | | | | FRED model | host 6.3.0-rc1+ | 1404 | 276 | 2127 | 14 | | w/ FRED patch set | guest | | | | | | 6.3.0-rc1+ | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/ | | | | | | FRED model | host 6.3.0-rc1+ | | | | | | w/ FRED patch set | guest | 1404 | 276 | 2127 | 14 | | 6.3.0-rc1+ w/ FRED patch set | | | | | | FRED disabled | | | | | +--------------------------------+-------+-------+-------+-------+ | Intel Simics® Simulator w/ | | | | | | FRED model | host 6.3.0-rc1+ | 1404 | 276 | 2127 | 14 | | w/ FRED patch set | guest | | | | | | 6.3.0-rc1+ w/ FRED patch set | | | | | +--------------------------------+-------+-------+-------+-------+
The Simics FRED model has one more passed case, which is x86:test_vsyscall_32, due to https://lore.kernel.org/lkml/[email protected]/.
Thanks
--Shan
Quoted 263 lines · Shan
We tested the v5 FRED patch set on the Intel Simics® Simulator and a machine with a 7th Intel(R) Core(TM) CPU.
Following are the Kselftest results on X86-64.
+--------------------------------------------+-------+-------+-------+-------+ > Config | Pass | Fail | Skip | Hang | +--------------------------------------------+-------+-------+-------+-------+ > the 7th Intel(R) Core(TM) CPU | 3078 | 458 | 734 | 5 | > 6.3.0-rc1+ | | | | | +--------------------------------------------+-------+-------+-------+-------+ > the 7th Intel(R) Core(TM) CPU | 3078 | 458 | 734 | 5 | > 6.3.0-rc1+ w/ FRED patch set | | | | | +--------------------------------------------+-------+-------+-------+-------+ > Intel Simics® Simulator w/o FRED model | 1888 | 271 | 2105 | 11 | > 6.3.0-rc1+ | | | | | +--------------------------------------------+-------+-------+-------+-------+ > Intel Simics® Simulator w/o FRED model | 1888 | 271 | 2105 | 11 | > 6.3.0-rc1+ w/ FRED patch set | | | | | +--------------------------------------------+-------+-------+-------+-------+ > Intel Simics® Simulator w/ FRED model | 1889 | 270 | 2105 | 11 | > 6.3.0-rc1+ | | | | | +--------------------------------------------+-------+-------+-------+-------+ > Intel Simics® Simulator w/ FRED model | 1889 | 270 | 2105 | 11 | > 6.3.0-rc1+ w/ FRED patch set FRED disabled | | | | | +--------------------------------------------+-------+-------+-------+-------+ > Intel Simics® Simulator w/ FRED model | 1888 | 270 | 2105 | 12 | > 6.3.0-rc1+ w/ FRED patch set | | | | | +--------------------------------------------+-------+-------+-------+-------+The following issues are seen in this round of test.
+----------------+----------------+----------------+----------------+ > | x86:test_ | bpf:test_progs | x86:sysret | > | vsyscall_32 | | _rip_64 | +----------------+----------------+----------------+----------------+ > the 7th | | | | > Intel(R) | FAIL | FAIL | PASS | > Core(TM) CPU | | | | > 6.3.0-rc1+ | | | | +----------------+----------------+----------------+----------------+ > the 7th | | | | > Intel(R) | | | | > Core(TM) CPU | FAIL | FAIL | PASS | > 6.3.0-rc1+ w/ | | | | > FRED patch set | | | | +----------------+----------------+----------------+----------------+ > Intel Simics® | | | | > Simulator w/o | FAIL | FAIL | PASS | > FRED model | | | | > 6.3.0-rc1+ | | | | +----------------+----------------+----------------+----------------+ > Intel Simics® | | | | > Simulator w/o | | | | > FRED model | FAIL | FAIL | PASS | > 6.3.0-rc1+ w/ | | | | > FRED patch set | | | | +----------------+----------------+----------------+----------------+ > Intel Simics® | | | | > Simulator w/ | PASS | FAIL | PASS | > FRED model | | | | > 6.3.0-rc1+ | | | | +----------------+----------------+----------------+----------------+ > Intel Simics® | | | | > Simulator w/ | | | | > FRED model | PASS | FAIL | PASS | > 6.3.0-rc1+ w/ | | | | > FRED patch set | | | | > FRED disabled | | | | +----------------+----------------+----------------+----------------+ > Intel Simics® | | | | > Simulator w/ | | | | > FRED model | PASS | HANG | FAIL | > 6.3.0-rc1+ w/ | | | | > FRED patch set | | | | +----------------+----------------+----------------+----------------+The test "x86:sysret_rip_64" is NOT a valid test on FRED, and there is a fix from Ammar Faizi after we discussed it in the LKML.
The test "bpf:test_progs" is still in investigation.
The "x86:test_vsyscall_32" is a regression since the v3 FRED patch set.
Thanks
--ShanQuoted 177 lines · Xin Li
This patch set enables FRED for x86-64.
The Intel flexible return and event delivery (FRED) architecture defines simple
new transitions that change privilege level (ring transitions). The FRED architecture was designed with the following goals:
1) Improve overall performance and response time by replacing event delivery through the interrupt descriptor table (IDT event delivery) and event return by
the IRET instruction with lower latency transitions.
2) Improve software robustness by ensuring that event delivery establishes the
full supervisor context and that event return establishes the full user context.The new transitions defined by the FRED architecture are FRED event delivery and,
for returning from events, two FRED return instructions. FRED event delivery can
effect a transition from ring 3 to ring 0, but it is used also to deliver events
incident to ring 0. One FRED instruction (ERETU) effects a return from ring 0
to
ring 3, while the other (ERETS) returns while remaining in ring 0.Search for the latest FRED spec in most search engines with this search pattern:
site:intel.com FRED (flexible return and event delivery) specification
As of now there is no publicly avaiable CPU supporting FRED, thus the Intel Simics® Simulator is used as software development and testing vehicles. And it can be downloaded from:
https://www.intel.com/content/www/us/en/developer/articles/tool/simics-simulator.html
To enable FRED, the Simics package 8112 QSP-CPU needs to be installed with CPU
model configured as:
$cpu_comp_class = "x86-experimental-fred"Longer term, we should refactor common code shared by FRED and IDT into common
shared files, and contain IDT code using a new configCONFIG_X86_IDT.Changes since v4:
* Rebased against v6.3-rc1.
* Do NOT use the term "injection", which in the KVM context means to reinject an event into the guest (Sean Christopherson).
* Add the explanation of why to execute "int $2" to invoke the NMI handler in NMI caused VM exits (Sean Christopherson).
* Use cs/ss instead of csx/ssx when initializing thept_regsstructure for callingexternal_interrupt(), otherwise it breaks i386 build.Changes since v3:
* Callexternal_interrupt()to handle IRQ in IRQ caused VM exits.
* Execute "int $2" to handle NMI in NMI caused VM exits.
* Rename csl/ssl of thept_regsstructure to csx/ssx (x for extended) (Andrew Cooper).Changes since v2:
* Improve comments for changes inarch/x86/include/asm/idtentry.h.Changes since v1:
* call irqentry_nmi_{enter,exit}() in both IDT and FRED debug fault kernel handler (Peter Zijlstra).
* Initialize a FRED exception handler tofred_bad_event()instead of NULL if no FRED handler defined for an exception vector (Peter Zijlstra).
* Push calling irqentry_{enter,exit}() and instrumentation_{begin,end}() down into individual FRED exception handlers, instead of in the dispatch framework (Peter Zijlstra).H. Peter Anvin (Intel) (24):
x86/traps: letcommon_interrupt()handleIRQ_MOVE_CLEANUP_VECTORx86/traps: add a system interrupt table for system interrupt dispatch x86/traps: addexternal_interrupt()to dispatch external interrupts x86/cpufeature: add the cpu feature bit for FRED x86/opcode: add ERETU, ERETS instructions to x86-opcode-map x86/objtool: teach objtool about ERETU and ERETS
x86/cpu: addX86_CR4_FREDmacro
x86/fred: addKconfigoption for FRED (CONFIG_X86_FRED) x86/fred: ifCONFIG_X86_FREDis disabled, disable FRED support x86/cpu: add MSR numbers for FRED configuration
x86/fred: header file with FRED definitions
x86/fred: make unions for the cs and ss fields in structpt_regsx86/fred: reserve space for the FRED stack frame
x86/fred: add a page fault entry stub for FRED
x86/fred: add a debug fault entry stub for FRED
x86/fred: add a NMI entry stub for FRED
x86/fred: FRED entry/exit and dispatch code
x86/fred: FRED initialization code
x86/fred: updateMSR_IA32_FRED_RSP0during task switch x86/fred: letret_from_fork()jmp tofred_exit_userwhen FRED is
enabled
x86/fred: disallow the swapgs instruction when FRED is enabled x86/fred: no ESPFIX needed when FRED is enabled
x86/fred: allow single-step trap and NMI when starting a new thread x86/fred: allow FRED systems to use interrupt vectors 0x10-0x1fXin Li (10):
x86/traps: addinstall_system_interrupt_handler()x86/traps: exportexternal_interrupt()for VMX IRQ reinjection x86/fred: header file for event types
x86/fred: add a machine check entry stub for FRED x86/fred: fixup fault on ERETU by jumping tofred_entrypoint_userx86/ia32: do not modify the DPL bits for a null selector x86/fred: allow dynamic stack frame size
x86/fred: disable FRED by default in its early stage
KVM: x86/vmx: callexternal_interrupt()to handle IRQ in IRQ caused VM
exits
KVM: x86/vmx: execute "int $2" to handle NMI in NMI caused VM exits
when FRED is enabled.../admin-guide/kernel-parameters.txt | 4 + arch/x86/Kconfig | 9 + arch/x86/entry/Makefile | 5 +- arch/x86/entry/entry_32.S | 2 +- arch/x86/entry/entry_64.S | 5 + arch/x86/entry/entry_64_fred.S | 59 +++++ arch/x86/entry/entry_fred.c | 234 ++++++++++++++++++ arch/x86/entry/vsyscall/vsyscall_64.c | 2 +- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/entry-common.h | 3 + arch/x86/include/asm/event-type.h | 17 ++ arch/x86/include/asm/extable_fixup_types.h | 4 +- arch/x86/include/asm/fred.h | 131 ++++++++++ arch/x86/include/asm/idtentry.h | 76 +++++- arch/x86/include/asm/irq.h | 5 + arch/x86/include/asm/irq_vectors.h | 15 +- arch/x86/include/asm/msr-index.h | 13 +- arch/x86/include/asm/processor.h | 12 +- arch/x86/include/asm/ptrace.h | 36 ++- arch/x86/include/asm/switch_to.h | 10 +- arch/x86/include/asm/thread_info.h | 35 +-- arch/x86/include/asm/traps.h | 13 + arch/x86/include/asm/vmx.h | 17 +- arch/x86/include/uapi/asm/processor-flags.h | 2 + arch/x86/kernel/Makefile | 1 + arch/x86/kernel/apic/apic.c | 11 +- arch/x86/kernel/apic/vector.c | 8 +- arch/x86/kernel/cpu/acrn.c | 7 +- arch/x86/kernel/cpu/common.c | 88 ++++--- arch/x86/kernel/cpu/mce/core.c | 11 + arch/x86/kernel/cpu/mshyperv.c | 22 +- arch/x86/kernel/espfix_64.c | 8 + arch/x86/kernel/fred.c | 73 ++++++ arch/x86/kernel/head_32.S | 3 +- arch/x86/kernel/idt.c | 6 +- arch/x86/kernel/irq.c | 6 +- arch/x86/kernel/irqinit.c | 7 +- arch/x86/kernel/kvm.c | 4 +- arch/x86/kernel/nmi.c | 28 +++ arch/x86/kernel/process.c | 5 + arch/x86/kernel/process_64.c | 21 +- arch/x86/kernel/signal_32.c | 21 +- arch/x86/kernel/traps.c | 175 +++++++++++-- arch/x86/kvm/vmx/vmx.c | 33 ++- arch/x86/lib/x86-opcode-map.txt | 2 +- arch/x86/mm/extable.c | 28 +++ arch/x86/mm/fault.c | 20 +- drivers/xen/events/events_base.c | 5 +- kernel/fork.c | 6 + tools/arch/x86/include/asm/cpufeatures.h | 1 + .../arch/x86/include/asm/disabled-features.h | 8 +- tools/arch/x86/include/asm/msr-index.h | 13 +- tools/arch/x86/lib/x86-opcode-map.txt | 2 +- tools/objtool/arch/x86/decode.c | 19 +- 55 files changed, 1185 insertions(+), 175 deletions(-) create mode 100644 arch/x86/entry/entry_64_fred.S create mode 100644 arch/x86/entry/entry_fred.c create mode 100644 arch/x86/include/asm/event-type.h create mode 100644 arch/x86/include/asm/fred.h create mode 100644 arch/x86/kernel/fred.c