x86/context-tracking: Remove last remaining calls to exception_enter/exception_exit()
Subsystems (3)
cover x86/context-tracking: Remove last remaining calls to exception_enter/exception_exit()
Thanks to the cleanups from Andy Lutomirski over the years, those calls can now be removed. This will allow for nice things in the future for x86 support on full nohz:
* Remove TIF_NOHZ and use a per-cpu switch to enable, disable context tracking.
* Avoid context tracking on housekeepers.
* Dynamically enable/disable context tracking on CPU on runtime and therefore allow runtime enable/disable of nohz_full
* Make nohz_full a property of cpuset.
Frederic Weisbecker (2):
x86/context-tracking: Remove exception_enter/exit() from
do_page_fault()
x86/context-tracking: Remove exception_enter/exit() from
KVM_PV_REASON_PAGE_NOT_PRESENT async page fault
arch/x86/kernel/kvm.c | 4 ---- arch/x86/mm/fault.c | 39 ++++++++++++--------------------------- 2 files changed, 12 insertions(+), 31 deletions(-)
Signature · 1 line
2.23.0
PATCH 1/2 x86/context-tracking: Remove exception_enter/exit() from do_page_fault()
do_page_fault(), like other exceptions, is already covered by user_enter() and user_exit() when the exception triggers in userspace.
As explained in 8c84014f3bbb112d07e73f30a10ac8a3a72f8649 ("x86/entry: Remove exception_enter() from most trap handlers"), exception_enter/exit() only remained to handle possible page fault from kernel mode while context tracking is in CONTEXT_USER mode, ie: on kernel entry before we manage to call user_exit(). And the only known offender was do_fast_syscall_32() fetching EBP register from where vDSO stashed it.
Meanwhile this got fixed with 9999c8c01f34c918a57d6e5ba2f5d8b79aa04801 ("x86/entry: Call enter_from_user_mode() with IRQs off") that moved enter_from_user_mode() before the call to get_user().
So we can safely remove it now.
--- arch/x86/mm/fault.c | 39 ++++++++++++--------------------------- 1 file changed, 12 insertions(+), 27 deletions(-)
arch/x86/mm/fault.c +12 -27
@@ -1486,27 +1486,6 @@ void do_user_addr_fault(struct pt_regs *regs, }NOKPROBE_SYMBOL (do_user_addr_fault); -/* -* Explicitly marked noinline such that the function tracer sees this as the -* page_fault entry point. -*/ -static noinline void -__do_page_fault (struct pt_regs *regs,unsigned long hw_error_code, -unsigned long address) -{ -prefetchw (¤t->mm->mmap_sem); - -if (unlikely (kmmio_fault (regs, address))) -return ; - -/* Was the fault on kernel-controlled part of the address space? */ -if (unlikely (fault_in_kernel_space (address))) -do_kern_addr_fault (regs, hw_error_code, address); -else -do_user_addr_fault (regs, hw_error_code, address); -} -NOKPROBE_SYMBOL (__do_page_fault); - static __always_inline voidtrace_page_fault_entries (struct pt_regs *regs,unsigned long error_code,unsigned long address)
@@ -1521,13 +1500,19 @@ trace_page_fault_entries(struct pt_regs *regs, unsigned long error_code, } dotraplinkage void -do_page_fault (struct pt_regs *regs,unsigned long error_code,unsigned long address) +do_page_fault (struct pt_regs *regs,unsigned long hw_error_code, +unsigned long address) { -enum ctx_state prev_state; +prefetchw (¤t->mm->mmap_sem); +trace_page_fault_entries (regs, hw_error_code, address); - prev_state =exception_enter (); -trace_page_fault_entries (regs, error_code, address); -__do_page_fault (regs, error_code, address); -exception_exit (prev_state); +if (unlikely (kmmio_fault (regs, address))) +return ; + +/* Was the fault on kernel-controlled part of the address space? */ +if (unlikely (fault_in_kernel_space (address))) +do_kern_addr_fault (regs, hw_error_code, address); +else +do_user_addr_fault (regs, hw_error_code, address); }NOKPROBE_SYMBOL (do_page_fault);
Signature · 1 line
2.23.0
[tip: x86/asm] x86/context-tracking: Remove exception_enter/exit() from do_page_fault()
The following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: ee6352b2c47a24234398e06381edd93a8e965976 Gitweb: https://git.kernel.org/tip/ee6352b2c47a24234398e06381edd93a8e965976 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 27 Dec 2019 17:36:11 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 07 Jan 2020 08:11:23 +01:00
x86/context-tracking: Remove exception_enter/exit() from do_page_fault()
do_page_fault(), like other exceptions, is already covered by user_enter() and user_exit() when the exception triggers in userspace.
As explained in:
8c84014f3bbb11 ("x86/entry: Remove exception_enter() from most trap handlers")
exception_enter/exit() only remained to handle possible page fault from kernel mode while context tracking is in CONTEXT_USER mode, ie: on kernel entry before we manage to call user_exit(). The only known offender was do_fast_syscall_32() fetching EBP register from where vDSO stashed it.
Meanwhile this got fixed in:
9999c8c01f34c9 ("x86/entry: Call enter_from_user_mode() with IRQs off")
that moved enter_from_user_mode() before the call to get_user().
So we can safely remove it now.
--- arch/x86/mm/fault.c | 39 ++++++++++++--------------------------- 1 file changed, 12 insertions(+), 27 deletions(-)
arch/x86/mm/fault.c +12 -27
@@ -1486,27 +1486,6 @@ good_area: }NOKPROBE_SYMBOL (do_user_addr_fault); -/* -* Explicitly marked noinline such that the function tracer sees this as the -* page_fault entry point. -*/ -static noinline void -__do_page_fault (struct pt_regs *regs,unsigned long hw_error_code, -unsigned long address) -{ -prefetchw (¤t->mm->mmap_sem); - -if (unlikely (kmmio_fault (regs, address))) -return ; - -/* Was the fault on kernel-controlled part of the address space? */ -if (unlikely (fault_in_kernel_space (address))) -do_kern_addr_fault (regs, hw_error_code, address); -else -do_user_addr_fault (regs, hw_error_code, address); -} -NOKPROBE_SYMBOL (__do_page_fault); - static __always_inline voidtrace_page_fault_entries (struct pt_regs *regs,unsigned long error_code,unsigned long address)
@@ -1521,13 +1500,19 @@ trace_page_fault_entries(struct pt_regs *regs, unsigned long error_code, } dotraplinkage void -do_page_fault (struct pt_regs *regs,unsigned long error_code,unsigned long address) +do_page_fault (struct pt_regs *regs,unsigned long hw_error_code, +unsigned long address) { -enum ctx_state prev_state; +prefetchw (¤t->mm->mmap_sem); +trace_page_fault_entries (regs, hw_error_code, address); - prev_state =exception_enter (); -trace_page_fault_entries (regs, error_code, address); -__do_page_fault (regs, error_code, address); -exception_exit (prev_state); +if (unlikely (kmmio_fault (regs, address))) +return ; + +/* Was the fault on kernel-controlled part of the address space? */ +if (unlikely (fault_in_kernel_space (address))) +do_kern_addr_fault (regs, hw_error_code, address); +else +do_user_addr_fault (regs, hw_error_code, address); }NOKPROBE_SYMBOL (do_page_fault);
PATCH 2/2 x86/context-tracking: Remove exception_enter/exit() from KVM_PV_REASON_PAGE_NOT_PRESENT async page fault
In reply to Frederic WeisbeckerThis is a leftover. Page faults, just like most other exceptions, are protected inside user_exit() / user_enter() calls in x86 entry code when we fault from userspace. So this pair of calls is now useless.
--- arch/x86/kernel/kvm.c | 4 ---- 1 file changed, 4 deletions(-)
arch/x86/kernel/kvm.c +0 -4
@@ -245,17 +245,13 @@ NOKPROBE_SYMBOL(kvm_read_and_reset_pf_reason); dotraplinkage voiddo_async_page_fault (struct pt_regs *regs,unsigned long error_code,unsigned long address) { -enum ctx_state prev_state; -switch (kvm_read_and_reset_pf_reason ()) {default :do_page_fault (regs, error_code, address);break ;case KVM_PV_REASON_PAGE_NOT_PRESENT :/* page is swapped out by the host. */ - prev_state =exception_enter ();kvm_async_pf_task_wait ((u32 )address, !user_mode (regs)); -exception_exit (prev_state);break ;case KVM_PV_REASON_PAGE_READY :rcu_irq_enter ();
Signature · 1 line
2.23.0
[tip: x86/asm] x86/context-tracking: Remove exception_enter/exit() from KVM_PV_REASON_PAGE_NOT_PRESENT async page fault
The following commit has been merged into the x86/asm branch of tip:
The following commit has been merged into the x86/asm branch of tip:
Commit-ID: 50cc02e599ef1e049473358604dc07d32b751e2c Gitweb: https://git.kernel.org/tip/50cc02e599ef1e049473358604dc07d32b751e2c Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 27 Dec 2019 17:36:12 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 07 Jan 2020 08:11:23 +01:00
x86/context-tracking: Remove exception_enter/exit() from KVM_PV_REASON_PAGE_NOT_PRESENT async page fault
This is a leftover. Page faults, just like most other exceptions, are protected inside user_exit() / user_enter() calls in x86 entry code when we fault from userspace. So this pair of calls is now superfluous.
--- arch/x86/kernel/kvm.c | 4 ---- 1 file changed, 4 deletions(-)
arch/x86/kernel/kvm.c +0 -4
@@ -245,17 +245,13 @@ NOKPROBE_SYMBOL(kvm_read_and_reset_pf_reason); dotraplinkage voiddo_async_page_fault (struct pt_regs *regs,unsigned long error_code,unsigned long address) { -enum ctx_state prev_state; -switch (kvm_read_and_reset_pf_reason ()) {default :do_page_fault (regs, error_code, address);break ;case KVM_PV_REASON_PAGE_NOT_PRESENT :/* page is swapped out by the host. */ - prev_state =exception_enter ();kvm_async_pf_task_wait ((u32 )address, !user_mode (regs)); -exception_exit (prev_state);break ;case KVM_PV_REASON_PAGE_READY :rcu_irq_enter ();
Peter Zijlstra
maintainerQuoted 12 lines · Frederic Weisbecker
Thanks to the cleanups from Andy Lutomirski over the years, those calls can now be removed. This will allow for nice things in the future for x86 support on full nohz:
* Remove
TIF_NOHZand use a per-cpu switch to enable, disable context tracking.* Avoid context tracking on housekeepers.
* Dynamically enable/disable context tracking on CPU on runtime and therefore allow runtime enable/disable of
nohz_full
Quoted 7 lines
* Make
nohz_fulla property of cpuset.Frederic Weisbecker (2):
x86/context-tracking: Remove exception_enter/exit() fromdo_page_fault()x86/context-tracking: Remove exception_enter/exit() from
KVM_PV_REASON_PAGE_NOT_PRESENTasync page fault
Thanks, these look good to me.
Did I tell you about my 'TODO' item vs text_poke_sync() ? Basically, we can avoid sending the IPI to NOHZ_FULL user CPUs, provided we make their enter_from_user_mode() do the sync_core().
Quoted 21 lines · Peter Zijlstra
Quoted 12 lines · Frederic Weisbecker
Thanks to the cleanups from Andy Lutomirski over the years, those calls can now be removed. This will allow for nice things in the future for x86 support on full nohz:
* Remove
TIF_NOHZand use a per-cpu switch to enable, disable context tracking.* Avoid context tracking on housekeepers.
* Dynamically enable/disable context tracking on CPU on runtime and therefore allow runtime enable/disable of
nohz_fullQuoted 7 lines
* Make
nohz_fulla property of cpuset.Frederic Weisbecker (2):
x86/context-tracking: Remove exception_enter/exit() fromdo_page_fault()x86/context-tracking: Remove exception_enter/exit() from
KVM_PV_REASON_PAGE_NOT_PRESENTasync page fault
Quoted 5 lines
Thanks, these look good to me.
Did I tell you about my 'TODO' item vs
text_poke_sync()? Basically, we can avoid sending the IPI toNOHZ_FULLuser CPUs, provided we make theirenter_from_user_mode()do thesync_core().
Yeah I believe I saw some email about that somewhere. I hope I didn't miss a patch about that. Anyway, whether past or future patch, I'll happily review that :-)