x86: IST cleanup
cover x86: IST cleanup
Various bits of cleanup for !CONFIG_PV builds.
Andrew Cooper (3):
x86/S3: Use percpu_traps_init() rather than opencoding SYSCALL/SYSENTER restoration x86/boot: Don't enable EFER.SCE for !CONFIG_PV builds
x86/pv: Don't use IST for NMI/#MC/#DB in !CONFIG_PV builds
xen/arch/x86/acpi/suspend.c | 25 ++----------------------- xen/arch/x86/boot/trampoline.S | 4 ++-- xen/arch/x86/cpu/common.c | 8 +++++--- xen/arch/x86/efi/efi-boot.h | 3 ++- xen/arch/x86/hvm/vmx/vmx.c | 4 ++++ xen/arch/x86/x86_64/traps.c | 19 ++++++------------- xen/include/asm-x86/processor.h | 12 +++++++++++- 7 files changed, 32 insertions(+), 43 deletions(-)
Signature · 1 line
2.11.0
PATCH 1/3 x86/S3: Use percpu_traps_init() rather than opencoding SYSCALL/SYSENTER restoration
This make the S3 BSP path consistent with AP paths, and reduces the amount of state needing stashing specially. Also, it takes care of re-setting up Xen's LBR configuration if requested, which was missing previously.
---
CC: Jan Beulich <[email protected]>
CC: Wei Liu <[email protected]>
CC: Roger Pau Monné <[email protected]>
--- xen/arch/x86/acpi/suspend.c | 25 ++----------------------- 1 file changed, 2 insertions(+), 23 deletions(-)
xen/arch/x86/acpi/suspend.c +2 -23
@@ -15,8 +15,6 @@#include <asm/xstate.h> #include <xen/hypercall.h> -static unsigned long saved_lstar, saved_cstar; -static unsigned long saved_sysenter_esp, saved_sysenter_eip;static unsigned long saved_fs_base, saved_gs_base, saved_kernel_gs_base;static uint64_t saved_xcr0;
@@ -25,14 +23,6 @@ void save_rest_processor_state(void) saved_fs_base =rdfsbase (); saved_gs_base =rdgsbase ();rdmsrl (MSR_SHADOW_GS_BASE , saved_kernel_gs_base); -rdmsrl (MSR_CSTAR , saved_cstar); -rdmsrl (MSR_LSTAR , saved_lstar); - -if ( cpu_has_sep ) - { -rdmsrl (MSR_IA32_SYSENTER_ESP , saved_sysenter_esp); -rdmsrl (MSR_IA32_SYSENTER_EIP , saved_sysenter_eip); - }if ( cpu_has_xsave ) saved_xcr0 =get_xcr0 ();
@@ -46,24 +36,13 @@ void restore_rest_processor_state(void)/* Restore full CR4 (inc MCE) now that the IDT is in place. */ write_cr4 (mmu_cr4_features); -/* Recover syscall MSRs */ -wrmsrl (MSR_LSTAR , saved_lstar); -wrmsrl (MSR_CSTAR , saved_cstar); -wrmsrl (MSR_STAR ,XEN_MSR_STAR ); -wrmsrl (MSR_SYSCALL_MASK ,XEN_SYSCALL_MASK ); +/* (re)initialise SYSCALL/SYSENTER state, amongst other things. */ +percpu_traps_init ();wrfsbase (saved_fs_base);wrgsbase (saved_gs_base);wrmsrl (MSR_SHADOW_GS_BASE , saved_kernel_gs_base); -if ( cpu_has_sep ) - { -/* Recover sysenter MSRs */ -wrmsrl (MSR_IA32_SYSENTER_ESP , saved_sysenter_esp); -wrmsrl (MSR_IA32_SYSENTER_EIP , saved_sysenter_eip); -wrmsr (MSR_IA32_SYSENTER_CS , __HYPERVISOR_CS,0 ); - } -if ( cpu_has_xsave && !set_xcr0 (saved_xcr0) )BUG ();
Signature · 1 line
2.11.0
Re: [PATCH 1/3] x86/S3: Use percpu_traps_init() rather than opencoding SYSCALL/SYSENTER restoration
xen/arch/x86/acpi/suspend.c void restore_rest_processor_state(void) · line 40
Quoted 11 lines · Andrew Cooper
xen/arch/x86/acpi/suspend.c +2
-5@@ -46,24 +36,13 @@ void restore_rest_processor_state(void)/* Restore full CR4 (inc MCE) now that the IDT is in place. */ write_cr4 (mmu_cr4_features); -/* Recover syscall MSRs */ -wrmsrl (MSR_LSTAR , saved_lstar); -wrmsrl (MSR_CSTAR , saved_cstar); -wrmsrl (MSR_STAR ,XEN_MSR_STAR ); -wrmsrl (MSR_SYSCALL_MASK ,XEN_SYSCALL_MASK ); +/* (re)initialise SYSCALL/SYSENTER state, amongst other things. */ +percpu_traps_init ();
Without tweaks to subarch_percpu_traps_init() this assumes that, now and going forward, map_domain_page() will work reliably at this early point of resume. I'm not opposed, i.e.
but it would feel less fragile if the redundant re-writing of the stubs would be skipped in the BSP resume case (I didn't check whether it's also redundant for APs, but I suspect it's not, as the stub pages may get allocated anew).
Jan
Re: [PATCH 1/3] x86/S3: Use percpu_traps_init() rather than opencoding SYSCALL/SYSENTER restoration
Quoted 15 lines · Jan Beulich
Review void restore_rest_processor_state(void) · line 40
Quoted 11 lines · Andrew Cooper
(quoted diff) +2
-5@@ -46,24 +36,13 @@ void restore_rest_processor_state(void) /* Restore full CR4 (inc MCE) now that the IDT is in place. */ write_cr4(mmu_cr4_features); - /* Recover syscall MSRs */ - wrmsrl(MSR_LSTAR, saved_lstar); - wrmsrl(MSR_CSTAR, saved_cstar); - wrmsrl(MSR_STAR, XEN_MSR_STAR); - wrmsrl(MSR_SYSCALL_MASK, XEN_SYSCALL_MASK); + /* (re)initialise SYSCALL/SYSENTER state, amongst other things. */ + percpu_traps_init();Without tweaks to
subarch_percpu_traps_init()this assumes that, now and going forward,map_domain_page()will work reliably at this early point of resume. I'm not opposed, i.e.
Quoted 1 line
Acked-by Jan Beulich <[email protected]>
I think this reasonable to expect, and robust going forward.
We are properly in d[IDLE]v0 context when it comes to pagetables, and there is nothing interesting between this point and coming fully back online.
That said, I could easily move the call to later in the resume path for even more certainty.
xen/arch/x86/acpi/power.c +0 -0
@@ -297,6 +297,8 @@ static int enter_state(u32 state)
ci->spec_ctrl_flags |= (default_spec_ctrl_flags & SCF_ist_wrmsr); spec_ctrl_exit_idle(ci);
+ /* (re)initialise SYSCALL/SYSENTER state, amongst other things. */ + percpu_traps_init();
+
done:
spin_debug_enable();
local_irq_restore(flags);
In fact - I prefer this, because it works towards one low priority goal of deleting {save,restore}_rest_processor_state() which I've still got a pending series for.
Would your ack still stand if I tweak the patch in this way?
Quoted 4 lines
but it would feel less fragile if the redundant re-writing of the stubs would be skipped in the BSP resume case (I didn't check whether it's also redundant for APs, but I suspect it's not, as the stub pages may get allocated anew).
I don't really agree. Symmetry (even if it is expected to be redundant) is much more easy to reason about in terms of robustness. S3 is not a fastpath.
~Andrew
Re: [PATCH 1/3] x86/S3: Use percpu_traps_init() rather than opencoding SYSCALL/SYSENTER restoration
Quoted 42 lines · Andrew Cooper
Quoted 15 lines · Jan Beulich
Review void restore_rest_processor_state(void) · line 40
Quoted 11 lines · Andrew Cooper
(quoted diff) +2
-5@@ -46,24 +36,13 @@ void restore_rest_processor_state(void) /* Restore full CR4 (inc MCE) now that the IDT is in place. */ write_cr4(mmu_cr4_features); - /* Recover syscall MSRs */ - wrmsrl(MSR_LSTAR, saved_lstar); - wrmsrl(MSR_CSTAR, saved_cstar); - wrmsrl(MSR_STAR, XEN_MSR_STAR); - wrmsrl(MSR_SYSCALL_MASK, XEN_SYSCALL_MASK); + /* (re)initialise SYSCALL/SYSENTER state, amongst other things. */ + percpu_traps_init();Without tweaks to
subarch_percpu_traps_init()this assumes that, now and going forward,map_domain_page()will work reliably at this early point of resume. I'm not opposed, i.e.Quoted 1 line
Acked-by Jan Beulich <[email protected]>I think this reasonable to expect, and robust going forward.
We are properly in d[IDLE]v0 context when it comes to pagetables, and there is nothing interesting between this point and coming fully back online.
That said, I could easily move the call to later in the resume path for even more certainty.
xen/arch/x86/acpi/power.c +0
-0@@ -297,6 +297,8 @@ static int enter_state(u32 state) ci->spec_ctrl_flags |= (default_spec_ctrl_flags & SCF_ist_wrmsr); spec_ctrl_exit_idle(ci);
+ /* (re)initialise SYSCALL/SYSENTER state, amongst other things. */ +percpu_traps_init();
+
done:
spin_debug_enable();
local_irq_restore(flags);
Quoted 5 lines
In fact - I prefer this, because it works towards one low priority goal of deleting {save,restore}_rest_processor_state() which I've still got a pending series for.
Would your ack still stand if I tweak the patch in this way?
Yes.
Jan
PATCH 2/3 x86/boot: Don't enable EFER.SCE for !CONFIG_PV builds
In reply to Andrew CooperThis will cause all SYSCALL/SYSRET instructions to suffer #UD rather than following the MSR_{L,C}STAR pointers, allowing us to drop the star_enter() panic helper, allowing us to clean up the IST stacks in a subsequent patch.
Drop the now-dead conditional SYSENTER logic in the middle of subarch_percpu_traps_init().
In addition, vmx_restore_host_msrs() need not restore any host state. (Regarding the asymmetric changes, VT-x automatically restores SYSENTER state on vmexit, and SVM restores both SYSCALL/SYSENTER state with the VMSAVE/VMLOAD instructions.)
---
CC: Jan Beulich <[email protected]>
CC: Wei Liu <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Kevin Tian <[email protected]>
--- xen/arch/x86/boot/trampoline.S | 4 ++-- xen/arch/x86/efi/efi-boot.h | 3 ++- xen/arch/x86/hvm/vmx/vmx.c | 4 ++++ xen/arch/x86/x86_64/traps.c | 19 ++++++------------- 4 files changed, 14 insertions(+), 16 deletions(-)
xen/arch/x86/boot/trampoline.S +2 -2
@@ -140,9 +140,9 @@ gdt_48: GLOBAL(trampoline_misc_enable_off) .quad 0 -/* EFER OR-mask for boot paths. This gets adjusted with NX when available. */ +/* EFER OR-mask for boot paths. SCE conditional on PV support, NX added when available. */ GLOBAL(trampoline_efer) - .long EFER_LME | EFER_SCE + .long EFER_LME | (EFER_SCE * IS_ENABLED(CONFIG_PV)) GLOBAL(trampoline_xen_phys_start) .long 0
xen/arch/x86/efi/efi-boot.h +2 -1
@@ -238,7 +238,8 @@ static void __init noreturn efi_arch_post_exit_boot(void)/* Set system registers and transfer control. */ asmvolatile ("pushq $0\n\tpopfq" );rdmsrl (MSR_EFER , efer); - efer |=EFER_SCE ; +if (IS_ENABLED (CONFIG_PV ) ) + efer |=EFER_SCE ;if ( cpu_has_nx ) efer |=EFER_NX ;wrmsrl (MSR_EFER , efer);
xen/arch/x86/hvm/vmx/vmx.c +4 -0
@@ -496,6 +496,10 @@ static void vmx_vcpu_destroy(struct vcpu *v)*/ static void vmx_restore_host_msrs (void ) { +/* No PV guests? No need to set restore host SYSCALL infrastructure. */ +if ( !IS_ENABLED (CONFIG_PV ) ) +return ; +/* Relies on the SYSCALL trampoline being at the start of the stubs. */ wrmsrl (MSR_STAR ,XEN_MSR_STAR );wrmsrl (MSR_LSTAR ,this_cpu (stubs.addr));
xen/arch/x86/x86_64/traps.c +6 -13
@@ -299,17 +299,8 @@ static unsigned int write_stub_trampoline( DEFINE_PER_CPU(struct stubs, stubs); -#ifdef CONFIG_PV void lstar_enter (void );void cstar_enter (void ); -#else -static void __cold star_enter (void ) -{ -panic ("lstar/cstar\n" ); -} -#define lstar_enter star_enter -#define cstar_enter star_enter -#endif /* CONFIG_PV */ void subarch_percpu_traps_init (void ) {
@@ -321,6 +312,10 @@ void subarch_percpu_traps_init(void)/* IST_MAX IST pages + at least 1 guard page + primary stack. */ BUILD_BUG_ON ((IST_MAX +1 ) *PAGE_SIZE +PRIMARY_STACK_SIZE >STACK_SIZE ); +/* No PV guests? No need to set up SYSCALL/SYSENTER infrastructure. */ +if ( !IS_ENABLED (CONFIG_PV ) ) +return ; + stub_page =map_domain_page (_mfn (this_cpu (stubs.mfn)));/*
@@ -338,10 +333,8 @@ void subarch_percpu_traps_init(void) {/* SYSENTER entry. */ wrmsrl (MSR_IA32_SYSENTER_ESP , stack_bottom); -wrmsrl (MSR_IA32_SYSENTER_EIP , -IS_ENABLED (CONFIG_PV ) ? (unsigned long )sysenter_entry :0 ); -wrmsr (MSR_IA32_SYSENTER_CS , -IS_ENABLED (CONFIG_PV ) ? __HYPERVISOR_CS :0 ,0 ); +wrmsrl (MSR_IA32_SYSENTER_EIP , (unsigned long )sysenter_entry); +wrmsr (MSR_IA32_SYSENTER_CS , __HYPERVISOR_CS,0 ); }/* Trampoline for SYSCALL entry from compatibility mode. */
Signature · 1 line
2.11.0
Re: [PATCH 2/3] x86/boot: Don't enable EFER.SCE for !CONFIG_PV builds
xen/arch/x86/efi/efi-boot.h static void __init noreturn efi_arch_post_exit_boot(void) · line 245
Quoted 12 lines · Andrew Cooper
xen/arch/x86/efi/efi-boot.h +2
-1@@ -238,7 +238,8 @@ static void __init noreturn efi_arch_post_exit_boot(void)/* Set system registers and transfer control. */ asmvolatile ("pushq $0\n\tpopfq" );rdmsrl (MSR_EFER , efer); - efer |=EFER_SCE ; +if (IS_ENABLED (CONFIG_PV ) ) + efer |=EFER_SCE ;if ( cpu_has_nx ) efer |=EFER_NX ;wrmsrl (MSR_EFER , efer);
Switch to simply ORing in trampoline_efer here?
With or without the adjustment
Jan
RE: [PATCH 2/3] x86/boot: Don't enable EFER.SCE for !CONFIG_PV builds
In reply to Andrew CooperQuoted 10 lines
From: Andrew Cooper <[email protected]>
Sent: Monday, April 20, 2020 10:59 PMThis will cause all SYSCALL/SYSRET instructions to suffer #UD rather than following the MSR_{L,C}STAR pointers, allowing us to drop the
star_enter()panic helper, allowing us to clean up the IST stacks in a subsequent patch.Drop the now-dead conditional SYSENTER logic in the middle of
subarch_percpu_traps_init().
Quoted 7 lines
In addition,
vmx_restore_host_msrs()need not restore any host state. (Regarding the asymmetric changes, VT-x automatically restores SYSENTER state on vmexit, and SVM restores both SYSCALL/SYSENTER state with
the VMSAVE/VMLOAD instructions.)Signed-off-by Andrew Cooper <[email protected]>
PATCH 3/3 x86/pv: Don't use IST for NMI/#MC/#DB in !CONFIG_PV builds
In reply to Andrew CooperISTs are used to force a stack switch on CPL0=>0 interrupts/exceptions. They however come with a nasty corner case in the case of reentrancy where the outer exception frame gets clobbered.
When the SYSCALL/SYSRET instructions aren't used, there is no need to use IST for anything other than #DF, which reduces the number of corner cases.
---
CC: Jan Beulich <[email protected]>
CC: Wei Liu <[email protected]>
CC: Roger Pau Monné <[email protected]>
--- xen/arch/x86/cpu/common.c | 8 +++++--- xen/include/asm-x86/processor.h | 12 +++++++++++- 2 files changed, 16 insertions(+), 4 deletions(-)
xen/arch/x86/cpu/common.c +5 -3
@@ -732,15 +732,17 @@ void load_system_tables(void) .rsp2 = 0x8600111111111111ul,/* -* MCE, NMI and Double Fault handlers get their own stacks. +* #DF always uses a separate stack. NMI/#MC/#DB only need a +* separate stacks when PV guests are used. * All others poisoned. */ .ist = { - [IST_MCE - 1] = stack_top + IST_MCE * PAGE_SIZE, [IST_DF -1 ] = stack_top +IST_DF *PAGE_SIZE , +#ifdef CONFIG_PV [IST_NMI - 1] = stack_top + IST_NMI * PAGE_SIZE, + [IST_MCE - 1] = stack_top + IST_MCE * PAGE_SIZE, [IST_DB - 1] = stack_top + IST_DB * PAGE_SIZE, - +#endif [IST_MAX ... ARRAY_SIZE(tss->ist) - 1] = 0x8600111111111111ul, },
xen/include/asm-x86/processor.h +11 -1
@@ -441,12 +441,18 @@ struct tss_page { }; DECLARE_PER_CPU(struct tss_page, tss_page); +/* +* Interrupt Stack Tables. Used to force a stack switch on a CPL0=>0 +* interrupt/exception. #DF uses IST all the time to detect stack overflows +* cleanly. NMI/#MC/#DB only need IST to cover the SYSCALL gap, and therefore +* only necessary with PV guests. +*/ #define IST_NONE 0UL#define IST_DF 1UL#define IST_NMI 2UL#define IST_MCE 3UL#define IST_DB 4UL -#define IST_MAX 4UL +#define IST_MAX (IS_ENABLED(CONFIG_PV) ? 4ul : 1ul)/* Set the Interrupt Stack Table used by a particular IDT entry. */ static inline void set_ist (idt_entry_t *idt,unsigned int ist)
@@ -461,6 +467,8 @@ static inline void set_ist(idt_entry_t *idt, unsigned int ist)static inline void enable_each_ist (idt_entry_t *idt) {set_ist (&idt[TRAP_double_fault],IST_DF ); +if ( !IS_ENABLED (CONFIG_PV ) ) +return ;set_ist (&idt[TRAP_nmi],IST_NMI );set_ist (&idt[TRAP_machine_check],IST_MCE );set_ist (&idt[TRAP_debug],IST_DB );
@@ -469,6 +477,8 @@ static inline void enable_each_ist(idt_entry_t *idt)static inline void disable_each_ist (idt_entry_t *idt) {set_ist (&idt[TRAP_double_fault],IST_NONE ); +if ( !IS_ENABLED (CONFIG_PV ) ) +return ;set_ist (&idt[TRAP_nmi],IST_NONE );set_ist (&idt[TRAP_machine_check],IST_NONE );set_ist (&idt[TRAP_debug],IST_NONE );
Signature · 1 line
2.11.0
Re: [PATCH 3/3] x86/pv: Don't use IST for NMI/#MC/#DB in !CONFIG_PV builds
xen/include/asm-x86/processor.h struct tss_page { · line 449
Quoted 12 lines · Andrew Cooper
xen/include/asm-x86/processor.h +6
-0@@ -441,12 +441,18 @@ struct tss_page { }; DECLARE_PER_CPU(struct tss_page, tss_page); +/* +* Interrupt Stack Tables. Used to force a stack switch on a CPL0=>0 +* interrupt/exception. #DF uses IST all the time to detect stack overflows +* cleanly. NMI/#MC/#DB only need IST to cover the SYSCALL gap, and therefore +* only necessary with PV guests. +*/
Is it really only the SYSCALL gap that we mean to cover? In particular for #MC I'd suspect it is a good idea to switch stacks as well, to get onto a distinct memory page in case the #MC was stack related. With NMI it might as well be better to switch; I agree we don't need any switching for #DB.
I also think that the comment at the top of current.h would want updating with these adjustments (which I notice lacks the #DB part anyway).
Jan
Re: [PATCH 3/3] x86/pv: Don't use IST for NMI/#MC/#DB in !CONFIG_PV builds
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Andrew Cooper
xen/include/asm-x86/processor.h +0
-0@@ -441,12 +441,18 @@ struct tss_page { }; DECLARE_PER_CPU(struct tss_page, tss_page);
Quoted 9 lines
xen/include/asm-x86/processor.h struct tss_page { · line 449
Quoted 6 lines
xen/include/asm-x86/processor.h +6
-0+/* +* Interrupt Stack Tables. Used to force a stack switch on a CPL0=>0 +* interrupt/exception. #DF uses IST all the time to detect stack overflows +* cleanly. NMI/#MC/#DB only need IST to cover the SYSCALL gap, and therefore +* only necessary with PV guests. +*/ Is it really only the SYSCALL gap that we mean to cover? In particular for #MC I'd suspect it is a good idea to switch stacks as well, to get onto a distinct memory page in case the #MC was stack related.
If #MC occurs on your stack, you have already lost. The chances of only taking a single #MC are tiny because the next-line prefetcher will be doing its job (or it hits when the lines (plural) leave L3, which will be in guest context at some point in the future.)
The very best you can hope for is to cleanly print something and crash - even if you manage to run the handler, you've got no idea if the interrupted context had a spinlock held, and Xen has no support for changing to a different pcpu stack.
Quoted 1 line
With NMI it might as well be better to switch;
Why? There is no benefit (with no SYSCALL in the picture), and a downside which causes state loss.
Quoted 6 lines
I agree we don't need any
switching for #DB.I also think that the comment at the top of
current.hwould want updating with these adjustments (which I notice lacks the #DB part anyway).
Oops - I totally forgot that for the XSA-260 fix.
~Andrew
Jan Beulich
Reviewed-byRe: [PATCH 3/3] x86/pv: Don't use IST for NMI/#MC/#DB in !CONFIG_PV builds
Quoted 18 lines · Andrew Cooper
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Andrew Cooper
xen/include/asm-x86/processor.h +0
-0@@ -441,12 +441,18 @@ struct tss_page { }; DECLARE_PER_CPU(struct tss_page, tss_page);Quoted 9 lines
xen/include/asm-x86/processor.h struct tss_page { · line 449
Quoted 6 lines
xen/include/asm-x86/processor.h +6
-0+/* +* Interrupt Stack Tables. Used to force a stack switch on a CPL0=>0 +* interrupt/exception. #DF uses IST all the time to detect stack overflows +* cleanly. NMI/#MC/#DB only need IST to cover the SYSCALL gap, and therefore +* only necessary with PV guests. +*/ Is it really only the SYSCALL gap that we mean to cover? In particular for #MC I'd suspect it is a good idea to switch stacks as well, to get onto a distinct memory page in case the #MC was stack related.
Quoted 4 lines
If #MC occurs on your stack, you have already lost. The chances of only taking a single #MC are tiny because the next-line prefetcher will be doing its job (or it hits when the lines (plural) leave L3, which will be in guest context at some point in the future.)
It would seem fishy behavior to me for the hardware to continue issuing prefetches against a page that has just been noticed to cause issues when accessed. Surely hardware at least _could_ internally mark such a page #UC or some such, to prevent further prefetches to actually hit the bus.
Quoted 4 lines
The very best you can hope for is to cleanly print something and crash - even if you manage to run the handler, you've got no idea if the interrupted context had a spinlock held, and Xen has no support for changing to a different pcpu stack.
But getting something printed is magnitudes better that hanging or rebooting entirely silently.
Nevertheless, as long as you extend the description somewhat to express this reasoning in some way
(a little hesitantly though).
Jan