ARM: Switch to generic entry
Subsystems (2)
v3 cover ARM: Switch to generic entry
This patch series converts a slew of ARM assembly into the corresponding C code, step by step moving the codebase closer to the expectations of the generic entry code, and as a last step switches ARM over to the generic entry code, and an RFC patch fixes a bunch of warnings from lockdep and the context tracker.
This was inspired by Jinjie Ruans similar work for ARM64.
The low-level assembly calls into arch/arm/kernel/syscall.c to invoke syscalls from userspace, and to the functions listed in arch/arm/kernel/entry.c for any other transitions to and from userspace. Looking at these functions and the call sites in the assembly on the final result should give a pretty good idea about how this works, and what the generic entry expects from an architecture.
This is still a work in progress but improved a bit since v2. Main things to be fixed: ARM v7m, some splat on Debian boot and discussion around the RFC patch.
There is a git branch you can pull in and test (v6.13-rc1 based):
https://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-integrator.git/log/?h=b4/arm-generic-entry-v6.13-rc1
Upsides:
- Same code paths as x86, S390, RISCV, Loongarch and probably soon ARM64 is used for the ARM systems. This includes some instrumentation stubs helping out with things we haven't even started to look at such as kmsan and live patching (!).
- By introducing the new callbacks to C, we can move away from the deprecated (and I think partly unmaintained) context tracking mechanism for RCU (user_exit_callable(), user_enter_callable()) in favor of what everyone else is using, i.e. calling rcu_irq_enter_check_tick() on IRQ entry. If we do not go with this patch set we can perhaps look into a separate patch just switching ARM32 to the new context tracking, as tests show the performance impact appears negligible for this.
- I think also lockdep is now behaving more according to expectations (the lockdep calls in ARM64 and generic entry seems different and more fine-granular from the ARM32 code) and the three warnings I see on Vexpress boots with mainline goes away after this patch set, but I am no expert in lockdep so I cannot really tell if this is a real improvement. The patches does make ARM lockdep clean.
Downsides:
- I had to remove the "fast syscall restart" from Al Viro. I don't know how much it will affect performance, but if this is something we must have, let's try to make the solution generic, i.e. add fast syscall restart in the generic entry code.
- The "superfast return to userspace" using just very small assembly snippets to get back to userspace on e.g. IRQs if and only if no instrumentation was compiled in, is no longer possible, since we unconditionally call into code written in C. I *think* this accounts for the majority of the ~3-4% performance impact (see measurements below).
TODO:
- ARM v7m, I don't even know how to get a system with this to start testing.
- Should the abort exceptions also go as "interrupts"?
Testing:
- Booted into Versatile Express QEMU (ARMv7), Ux500 full graphic UI (PostmarketOS Phosh, ARMv7 on hardware, Gemini ARMv4 on hardware. No special issues.
- Tested some ptrace/strace obviously, such as issuing several instances of "ptrace find /" and let this scroll by in the terminal over some 10 minutes or so.
- Turned on RCU torture tests and ran for a while. Seems stable and the test outputs look normal.
- Ran stress-ng, works fine.
- Booted with "lockdep" (CONFIG_PROVE_LOCKING). The ARM32 mainline produce 3 warnings at boot and those go away after these patches. I haven't looked closer at what it was that I inadvertedly fixed here, but I suspect the current context tracking has the same issues as what I fix in the RFC patch.
Caveat:
- Since we are not nested and not the idle task the following was triggered in rcu.c while in __und_usr, __dabt_usr or __pabt_usr:
/*
* If we're not in an interrupt, we must be in the idle task! */
WARN_ON_ONCE(!nesting && !is_idle_task(current));
What I think we see is that data/prefetch and undefined instruction exceptions on ARM are not really interrupts as they don't disable actual IRQs while being handled so we get nesting outside of the kernels control.
The last RFC patch in the patch set illustrates clearly what the problem is and provides a possible workaround / best effort. But we need to discuss this first.
Performance impact:
The changes were tested using the standard syscall overhead testing oneliner:
perf bench syscall all
This executes 10,000,000 getppid() in sequence and measures the time taken for this to complete. The numbers vary a bit but they are consistent.
In QEMU I tested with Vexpress and two CPU cores (-M vexpress-a15 -m 2G -smp cpus=2). DRM graphics and framebuffer was activated to give a bit of background IRQ activity (vsync interrupts).
I ran the perf command three times on each configuration, and picked the one iteration where the original code performed the best, and the one where the patches kernel performed the worst, to get a worst-case comparison.
v6.13-rc1 vexpress_defconfig, best invocation:
Total time: 149.380 [sec]
14.938072 usecs/op
66,943 ops/secv6.13-rc1 vexpress_defconfig, and this patch set, worst invocation:
Total time: 158.312 [sec]
15.831248 usecs/op
63,166 ops/secHere we see a performance degradation of around 6% operations/sec for a vexpress dualcore defconfig in the best vs worst case. (This isn't statistically correct, the effect is likely smaller.)
Debians stock kernel was noticably faster, so I investigated what causes this. It turns out that the big performance hog for syscalls is actually PAN, and I think Debian armhf simply turns this off. So I re-did the tests with CONFIG_ARM_PAN turned off to emulate the impact on stock Debian:
v6.13-rc1 vexpress_defconfig, !PAN, best invocation:
Total time: 36.882 [sec]
3.688294 usecs/op
271,128 ops/secv6.13-rc1 vexpress_defconfig, !PAN and this patch set, worst invocation:
Total time: 39.266 [sec]
3.926700 usecs/op
254,666 ops/secAgain we see around 6%.
To conclude if any of this was due to the new context tracking, at one point I tested to patch back the old context tracking on top of generic entry. This is hardly something that can be recommended, and anyway showed no noticeable overhead difference.
Open questions:
- Generic entry requires PTRACE_SYSEMU and PTRACE_SYSEMU_SINGLESTEP to be defined. I added them but don't even know what they do or if generic entry magically adds support for them (probably not) so I need help here.
- I need Al Viro's input on how to deal with the "fast syscall restart" that I bluntly deleted, if we need to reincarnate it in the generic entry or what we shall do here.
- I need to test with an OABI rootfs.
- I need to fix ARM v7m
- Should I skip the new entry.c and just put all new code into the good old trap.c?
---
Changes in v3:
- Rewrote the code in entry.c so the IRQ handler saves pt_regs
calls IRQ handler (including switching to IRQ stack!) and
restores pt_regs in one function instead of one entry and
one exit function. This is what every other arch using
generic entry is doing, and we should do it too.
- The rewrite solved the caveat warnings from the previous patch
set which was blatantly not SMP safe :/
- Rewrite the data abort and prefetch abort handlers in a
separate patch which we may squash in the end, but this makes
the patch set easier to review.
- Drop a pointless patch rewriting the NMI handlers in C, it's
better to just patch into the end result in the last patch,
as we're replacing handle_fiq_as_nmi().
- Syscall C invocations have to be tagged __ADDRESSABLE() in order
not to upset KCFI: the file is only referenced in both ends by
assembly so we need to point this out to the compiler.
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- Performance impact measurements have been provided.
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Linus Walleij (30):
ARM: Prepare includes for generic entry
ARM: ptrace: Split report_syscall()
ARM: entry: Skip ret_slow_syscall label
ARM: process: Rewrite ret_from_fork i C
ARM: process: Remove local restart
ARM: entry: Invoke syscalls using C
ARM: entry: Rewrite two asm calls in C
ARM: entry: Move trace entry to C function
ARM: entry: save the syscall sp in thread_info
ARM: entry: move all tracing invocation to C
ARM: entry: Merge the common and trace entry code
ARM: entry: Rename syscall invocation
ARM: entry: Create user_mode_enter/exit
ARM: entry: Drop trace argument from usr_entry macro
ARM: entry: Separate call path for syscall SWI entry
ARM: entry: Drop argument to asm_irqentry macros
ARM: entry: Implement syscall_exit_to_user_mode()
ARM: entry: Drop the superfast ret_fast_syscall
ARM: entry: Remove fast and offset register restore
ARM: entry: Untangle ret_fast_syscall/to_user
ARM: entry: Do not double-call exit functions
ARM: entry: Move work processing to C
ARM: entry: Stop exiting syscalls like IRQs
ARM: entry: Complete syscall and IRQ transition to C
ARM: entry: Create irqentry calls from kernel mode
ARM: entry: Move in-kernel hardirq tracing to C
ARM: irq: Add irqstack helpers
ARM: entry: Convert to generic entry
ARM: entry: Handle prefetch and data aborts as interrupts
RFC: ARM: entry: Block IRQs in early IRQ context
arch/arm/Kconfig | 1 + arch/arm/include/asm/entry-common.h | 66 ++++++++++++ arch/arm/include/asm/entry.h | 14 +++ arch/arm/include/asm/ptrace.h | 8 +- arch/arm/include/asm/signal.h | 4 - arch/arm/include/asm/stacktrace.h | 2 +- arch/arm/include/asm/switch_to.h | 4 + arch/arm/include/asm/syscall.h | 7 ++ arch/arm/include/asm/thread_info.h | 18 +--- arch/arm/include/asm/traps.h | 5 +- arch/arm/include/uapi/asm/ptrace.h | 2 + arch/arm/kernel/Makefile | 5 +- arch/arm/kernel/asm-offsets.c | 1 + arch/arm/kernel/entry-armv.S | 83 ++++----------- arch/arm/kernel/entry-common.S | 198 +++++++++++++----------------------- arch/arm/kernel/entry-header.S | 100 ++---------------- arch/arm/kernel/entry.c | 115 +++++++++++++++++++++ arch/arm/kernel/irq.c | 31 ++++++ arch/arm/kernel/irq.h | 3 + arch/arm/kernel/process.c | 25 ++++- arch/arm/kernel/ptrace.c | 76 -------------- arch/arm/kernel/signal.c | 57 ++--------- arch/arm/kernel/syscall.c | 32 ++++++ arch/arm/kernel/traps.c | 29 +----- arch/arm/mm/abort-ev4.S | 2 +- arch/arm/mm/abort-ev4t.S | 2 +- arch/arm/mm/abort-ev5t.S | 4 +- arch/arm/mm/abort-ev5tj.S | 6 +- arch/arm/mm/abort-ev6.S | 2 +- arch/arm/mm/abort-ev7.S | 2 +- arch/arm/mm/abort-lv4t.S | 36 +++---- arch/arm/mm/abort-macro.S | 2 +- arch/arm/mm/abort-nommu.S | 2 +- arch/arm/mm/fault.c | 4 +- arch/arm/mm/fault.h | 8 +- arch/arm/mm/pabort-legacy.S | 2 +- arch/arm/mm/pabort-v6.S | 2 +- arch/arm/mm/pabort-v7.S | 2 +- 38 files changed, 460 insertions(+), 502 deletions(-)
---
base-commit: 85d1ad82bc4a5215e54409c85b8d0b5b1f40a289
change-id: 20240903-arm-generic-entry-ada145378bbe
Best regards,
Signature · 1 line
Linus Walleij <[email protected]>
RFC PATCH v3 1/30 ARM: Prepare includes for generic entry
The generic entry code needs a static inline function for regs_irq_disabled() so add this, in further requires the signature of on_thread_stack() to be __always_inline so add this as well.
--- arch/arm/include/asm/ptrace.h | 5 +++++ arch/arm/include/asm/stacktrace.h | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-)
arch/arm/include/asm/ptrace.h +5 -0
@@ -52,6 +52,11 @@ struct svc_pt_regs {#define fast_interrupts_enabled (regs) \ (!((regs)->ARM_cpsr & PSR_F_BIT)) +static inline int regs_irqs_disabled (struct pt_regs *regs) +{ +return !interrupts_enabled (regs); +} +/* Are the current registers suitable for user mode? * (used to maintain security in signal handlers) */
arch/arm/include/asm/stacktrace.h +1 -1
@@ -26,7 +26,7 @@ struct stackframe {#endif }; -static inline bool on_thread_stack (void ) +static __always_inline bool on_thread_stack(void) {unsigned long delta = current_stack_pointer ^ (unsigned long )current->stack;
Signature · 1 line
2.47.1
RFC PATCH v3 2/30 ARM: ptrace: Split report_syscall()
In reply to Linus WalleijThe generic entry code requires that report_syscall() be provided in two parts: report_syscall_enter() and report_syscall_exit() so split the combined function in two.
--- arch/arm/kernel/ptrace.c | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-)
arch/arm/kernel/ptrace.c +22 -8
@@ -827,31 +827,45 @@ enum ptrace_syscall_dir { PTRACE_SYSCALL_EXIT, }; -static void report_syscall (struct pt_regs *regs,enum ptrace_syscall_dir dir) +static void report_syscall_enter (struct pt_regs *regs) {unsigned long ip;/* * IP is used to denote syscall entry/exit: -* IP = 0 -> entry, =1 -> exit +* IP = 0 -> entry */ ip = regs->ARM_ip; - regs->ARM_ip = dir; + regs->ARM_ip =PTRACE_SYSCALL_ENTER ; -if (dir ==PTRACE_SYSCALL_EXIT ) -ptrace_report_syscall_exit (regs,0 ); -else if (ptrace_report_syscall_entry (regs)) +if (ptrace_report_syscall_entry (regs))current_thread_info ()->abi_syscall =-1 ; regs->ARM_ip = ip; } +static void report_syscall_exit (struct pt_regs *regs) +{ +unsigned long ip; + +/* +* IP is used to denote syscall entry/exit: +* IP = 1 -> exit +*/ + ip = regs->ARM_ip; + regs->ARM_ip =PTRACE_SYSCALL_EXIT ; + +ptrace_report_syscall_exit (regs,0 ); + + regs->ARM_ip = ip; +} + asmlinkage int syscall_trace_enter(struct pt_regs *regs) {int scno;if (test_thread_flag (TIF_SYSCALL_TRACE )) -report_syscall (regs,PTRACE_SYSCALL_ENTER ); +report_syscall_enter (regs);/* Do seccomp after ptrace; syscall may have changed. */ #ifdef CONFIG_HAVE_ARCH_SECCOMP_FILTER
@@ -892,5 +906,5 @@ asmlinkage void syscall_trace_exit(struct pt_regs *regs)trace_sys_exit (regs,regs_return_value (regs));if (test_thread_flag (TIF_SYSCALL_TRACE )) -report_syscall (regs,PTRACE_SYSCALL_EXIT ); +report_syscall_exit (regs); }
Signature · 1 line
2.47.1
RFC PATCH v3 3/30 ARM: entry: Skip ret_slow_syscall label
In reply to Linus WalleijDon't introduce this extra label anymore now that we have a clearly defined entry point for ret_to_user.
--- arch/arm/kernel/entry-common.S | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
arch/arm/kernel/entry-common.S +3 -4
@@ -102,7 +102,6 @@ ENDPROC(ret_fast_syscall) * do_work_pending() will update this state if necessary. */ ENTRY(ret_to_user) -ret_slow_syscall: #if IS_ENABLED(CONFIG_DEBUG_RSEQ) /* do_rseq_syscall needs interrupts enabled. */ enable_irq_notrace @ enable interrupts
@@ -136,7 +135,7 @@ ENTRY(ret_from_fork) badrne lr, 1f retne r5 1: get_thread_info tsk - b ret_slow_syscall + b ret_to_user ENDPROC(ret_from_fork) /*=============================================================================
@@ -310,13 +309,13 @@ __sys_trace_return_nosave: enable_irq_notrace mov r0, sp bl syscall_trace_exit - b ret_slow_syscall + b ret_to_user __sys_trace_return: str r0, [sp, #S_R0 + S_OFF]! @ save returned r0 mov r0, sp bl syscall_trace_exit - b ret_slow_syscall + b ret_to_user .macro syscall_table_start, sym .equ __sys_nr, 0
Signature · 1 line
2.47.1
RFC PATCH v3 4/30 ARM: process: Rewrite ret_from_fork i C
In reply to Linus WalleijMove the direct assembly call to schedule_tail() and the call starting a kernel thread into a C version of ret_from_fork() in process.c and rename the old ret_from_fork to ret_from_fork_asm following the pattern in x86.
Leave a comment on what we will do when we get to switching to generic entry with this as a base.
--- arch/arm/include/asm/switch_to.h | 4 ++++ arch/arm/kernel/entry-common.S | 22 +++++++++++++++------- arch/arm/kernel/process.c | 24 ++++++++++++++++++++++-- 3 files changed, 41 insertions(+), 9 deletions(-)
arch/arm/include/asm/switch_to.h +4 -0
@@ -17,6 +17,10 @@#define __complete_pending_tlbi ()#endif +asmlinkage void ret_from_fork_asm(void); +__visible void ret_from_fork(struct task_struct *prev, struct pt_regs *regs, +int (*fn)(void *),void *fn_arg); +/* * switch_to(prev, next) should switch from task `prev' to `next' * `prev' will never be the same as `next'. schedule() itself
arch/arm/kernel/entry-common.S +15 -7
@@ -5,6 +5,7 @@ * Copyright (C) 2000 Russell King */ +#include <linux/cfi_types.h> #include <asm/assembler.h> #include <asm/unistd.h> #include <asm/ftrace.h>
@@ -128,15 +129,22 @@ ENDPROC(ret_to_user) /* * This is how we return from a fork. */ -ENTRY(ret_from_fork) - bl schedule_tail - cmp r5, #0 - movne r0, r4 - badrne lr, 1f - retne r5 +SYM_TYPED_FUNC_START(ret_from_fork_asm) + /* + * ret_from_fork() expects: + * r0: prev + * r1: regs + * r2: fn() for kernel process or NULL + * r3: fn_arg for kernel process or N/A + */ + mov r1, sp + mov r2, r5 + mov r3, r4 + bl ret_from_fork + mov r0, #0 1: get_thread_info tsk b ret_to_user -ENDPROC(ret_from_fork) +SYM_FUNC_END(ret_from_fork_asm) /*============================================================================= * SWI handler
arch/arm/kernel/process.c +22 -2
@@ -29,6 +29,7 @@#include <asm/processor.h> #include <asm/thread_notify.h> #include <asm/stacktrace.h> +#include <asm/switch_to.h> #include <asm/system_misc.h> #include <asm/mach/time.h> #include <asm/tls.h>
@@ -230,7 +231,26 @@ void flush_thread(void)thread_notify (THREAD_NOTIFY_FLUSH , thread); } -asmlinkage void ret_from_fork(void) __asm__("ret_from_fork"); +/* The assembly stub sets up the parameters and calls ret_from_fork() */ +asmlinkage void ret_from_fork_asm(void) __asm__("ret_from_fork_asm"); + +__visible void ret_from_fork(struct task_struct *prev, struct pt_regs *regs, +int (*fn)(void *),void *fn_arg) +{ +schedule_tail (prev); + +/* Is this a kernel thread? */ +if (unlikely (fn)) { +fn (fn_arg); +/* +* A kernel thread is allowed to return here after successfully +* calling kernel_execve(). Exit to userspace to complete the +* execve() syscall. +*/ + } + +/* syscall_exit_to_user_mode(regs); here once we switch to generic entry */ +}int copy_thread (struct task_struct *p,const struct kernel_clone_args *args) {
@@ -263,7 +283,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) thread->cpu_context.r5 = (unsigned long )args->fn; childregs->ARM_cpsr =SVC_MODE ; } - thread->cpu_context.pc = (unsigned long )ret_from_fork; + thread->cpu_context.pc = (unsigned long )ret_from_fork_asm; thread->cpu_context.sp = (unsigned long )childregs;clear_ptrace_hw_breakpoint (p);
Signature · 1 line
2.47.1
RFC PATCH v3 5/30 ARM: process: Remove local restart
In reply to Linus WalleijThe ARM kernel contains a quirk to handle syscall restarts inside the kernel without exiting to userspace. The generic entry cannot handle this.
Rename do_signal() to arch_do_signal_or_restart() to fit with the upcoming generic entry conversion.
This is essentially a revert of commit 81783786d5cf
"ARM: 7473/1: deal with handlerless restarts without leaving the kernel" from 2012.
Other solutions may be possible, such as checking the PC after do_work_pending and assume it is a local restart if that address is inside the kernel, or modifying the generic entry code to track local restarts like ARM does and pass that information back.
--- arch/arm/include/asm/signal.h | 3 +-- arch/arm/kernel/entry-common.S | 9 +-------- arch/arm/kernel/signal.c | 28 +++++++++------------------- 3 files changed, 11 insertions(+), 29 deletions(-)
arch/arm/include/asm/signal.h +1 -2
@@ -24,7 +24,6 @@ typedef struct {#include <asm/sigcontext.h> void do_rseq_syscall (struct pt_regs *regs); -int do_work_pending (struct pt_regs *regs,unsigned int thread_flags, -int syscall); +void do_work_pending (struct pt_regs *regs,unsigned int thread_flags);#endif
arch/arm/kernel/entry-common.S +1 -8
@@ -86,14 +86,8 @@ ENDPROC(ret_fast_syscall) bne __sys_trace_return_nosave slow_work_pending: mov r0, sp @ 'regs' - mov r2, why @ 'syscall' bl do_work_pending - cmp r0, #0 - beq no_work_pending - movlt scno, #(__NR_restart_syscall - __NR_SYSCALL_BASE) - str scno, [tsk, #TI_ABI_SYSCALL] @ make sure tracers see update - ldmia sp, {r0 - r6} @ have to reload r0 - r6 - b local_restart @ ... and off we go + b no_work_pending ENDPROC(ret_fast_syscall) /*
@@ -266,7 +260,6 @@ ENTRY(vector_swi) */ TRACE( ldmia sp, {r0 - r3} ) -local_restart: ldr r10, [tsk, #TI_FLAGS] @ check for syscall tracing stmdb sp!, {r4, r5} @ push fifth and sixth args
arch/arm/kernel/signal.c +9 -19
@@ -18,6 +18,7 @@#include <asm/traps.h> #include <asm/unistd.h> #include <asm/vfp.h> +#include <asm/syscall.h> #include <asm/syscalls.h> #include "signal.h"
@@ -534,9 +535,10 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs)* the kernel can handle, and then we build all the user-level signal handling * stack-frames in one go after that. */ -static int do_signal (struct pt_regs *regs,int syscall) +static void arch_do_signal_or_restart (struct pt_regs *regs) {unsigned int retval =0 , continue_addr =0 , restart_addr =0 ; +bool syscall = (syscall_get_nr (current, regs) !=-1 );struct ksignal ksig;int restart =0 ;
@@ -590,16 +592,14 @@ static int do_signal(struct pt_regs *regs, int syscall) }else {/* no handler */ restore_saved_sigmask (); -if (unlikely (restart) && regs->ARM_pc == restart_addr) { +if (unlikely (restart) && regs->ARM_pc == restart_addr) regs->ARM_pc = continue_addr; -return restart; - } } -return 0 ; +return ; } -asmlinkage int -do_work_pending (struct pt_regs *regs,unsigned int thread_flags,int syscall) +asmlinkage void +do_work_pending (struct pt_regs *regs,unsigned int thread_flags) {/* * The assembly code enters us with IRQs off, but it hasn't
@@ -612,19 +612,10 @@ do_work_pending(struct pt_regs *regs, unsigned int thread_flags, int syscall)schedule (); }else {if (unlikely (!user_mode (regs))) -return 0 ; +return ;local_irq_enable ();if (thread_flags & (_TIF_SIGPENDING | _TIF_NOTIFY_SIGNAL)) { -int restart =do_signal (regs, syscall); -if (unlikely (restart)) { -/* -* Restart without handlers. -* Deal with it without leaving -* the kernel space. -*/ -return restart; - } - syscall =0 ; +arch_do_signal_or_restart (regs); }else if (thread_flags & _TIF_UPROBE) {uprobe_notify_resume (regs); }else {
@@ -634,7 +625,6 @@ do_work_pending(struct pt_regs *regs, unsigned int thread_flags, int syscall)local_irq_disable (); thread_flags =read_thread_flags (); }while (thread_flags & _TIF_WORK_MASK); -return 0 ; }struct page *get_signal_page (void )
Signature · 1 line
2.47.1
RFC PATCH v3 6/30 ARM: entry: Invoke syscalls using C
In reply to Linus WalleijThis makes the assembly invoking syscalls switch over to doing this from C with a small assembly stub to actually jump into the syscall.
Split the syscall invocation into two paths: one for plain invocation and one for tracing (also known as "reload" as it was reloading the registers from regs). We rename this path with the infix "trace" as that code will be trace-specific as we move code over to C.
Some registers such as r1 and lr get cobbled during the C calls and need to be restored when we return.
Right now the part in C doesn't do much more than check the syscall number to be valid (a test previously done with a cmp r #NR_syscalls inside the invoke_syscall macro) but we will gradually factor over more assembly to C that can then be switched to the generic entry code so the exercise gets a point.
Since both functions are called from a C file that is in turn only called from assembly, the functions have to be tagged with __ADDRESSABLE() in order not to upset CFI.
Tested with a full system boot and by issuing some command line tools with strace to make sure the tracing path still works.
--- arch/arm/include/asm/syscall.h | 3 ++ arch/arm/kernel/Makefile | 3 +- arch/arm/kernel/entry-common.S | 65 ++++++++++++++++++++++++++++++++++++------ arch/arm/kernel/entry-header.S | 25 ---------------- arch/arm/kernel/syscall.c | 28 ++++++++++++++++++ 5 files changed, 90 insertions(+), 34 deletions(-)
arch/arm/include/asm/syscall.h +3 -0
@@ -19,6 +19,9 @@extern const unsigned long sys_call_table[]; +int invoke_syscall (void *table,struct pt_regs *regs,int scno,void *retp); +int invoke_syscall_trace (void *table,struct pt_regs *regs,int scno,void *retp); +static inline int syscall_get_nr (struct task_struct *task,struct pt_regs *regs) {
arch/arm/kernel/Makefile +2 -1
@@ -20,7 +20,8 @@ CFLAGS_REMOVE_return_address.o = -pg obj-y := elf.o entry-common.o irq.o opcodes.o \ process.o ptrace.o reboot.o io.o \ setup.o signal.o sigreturn_codes.o \ - stacktrace.o sys_arm.o time.o traps.o + stacktrace.o sys_arm.o time.o traps.o \ + syscall.o KASAN_SANITIZE_stacktrace.o := n KASAN_SANITIZE_traps.o := n
arch/arm/kernel/entry-common.S +57 -8
91 lines not on this page · raw message
arch/arm/kernel/entry-header.S +0 -25
@@ -389,31 +389,6 @@ ALT_UP_B(.L1_\@) #endif .endm - .macro invoke_syscall, table, nr, tmp, ret, reload=0 -#ifdef CONFIG_CPU_SPECTRE - mov \tmp, \nr - cmp \tmp, #NR_syscalls @ check upper syscall limit - movcs \tmp, #0 - csdb - badr lr, \ret @ return address - .if \reload - add r1, sp, #S_R0 + S_OFF @ pointer to regs - ldmiacc r1, {r0 - r6} @ reload r0-r6 - stmiacc sp, {r4, r5} @ update stack arguments - .endif - ldrcc pc, [\table, \tmp, lsl #2] @ call sys_* routine -#else - cmp \nr, #NR_syscalls @ check upper syscall limit - badr lr, \ret @ return address - .if \reload - add r1, sp, #S_R0 + S_OFF @ pointer to regs - ldmiacc r1, {r0 - r6} @ reload r0-r6 - stmiacc sp, {r4, r5} @ update stack arguments - .endif - ldrcc pc, [\table, \nr, lsl #2] @ call sys_* routine -#endif - .endm - /* * These are the registers used in the syscall handler, and allow us to * have in theory up to 7 arguments to a function - r0 to r6.
arch/arm/kernel/syscall.c new file +28 -0
@@ -0,0 +1,28 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <linux/syscalls.h> +#include <asm/syscall.h> + +int invoke_syscall_asm (void *table,struct pt_regs *regs,int scno,void *retp); +__ADDRESSABLE (invoke_syscall_asm); + +__visible int invoke_syscall(void *table, struct pt_regs *regs, int scno, void *retp) +{ +if (scno < NR_syscalls) +/* Doing this with return makes sure the stack gets pop:ed */ +return invoke_syscall_asm (table, regs, scno, retp); + +return 0 ; +} + +int invoke_syscall_trace_asm (void *table,struct pt_regs *regs,int scno,void *retp); +__ADDRESSABLE (invoke_syscall_trace_asm); + +__visible int invoke_syscall_trace(void *table, struct pt_regs *regs, int scno, void *retp) +{ +if (scno < NR_syscalls) +/* Doing this with return makes sure the stack gets pop:ed */ +return invoke_syscall_trace_asm (table, regs, scno, retp); + +return 0 ; +}
Signature · 1 line
2.47.1
RFC PATCH v3 7/30 ARM: entry: Rewrite two asm calls in C
In reply to Linus WalleijThe normal and trace entry code calls out to arm_syscall() and sys_ni_syscall() from assembly, but these calls can be moved over to the new C implementation.
--- arch/arm/include/asm/traps.h | 2 +- arch/arm/kernel/entry-common.S | 23 +++++------------------ arch/arm/kernel/syscall.c | 13 +++++++++++-- arch/arm/kernel/traps.c | 2 +- 4 files changed, 18 insertions(+), 22 deletions(-)
arch/arm/include/asm/traps.h +1 -1
@@ -40,7 +40,7 @@ asmlinkage void dump_backtrace_stm(u32 *stack, u32 instruction, const char *logl asmlinkage void do_undefinstr(struct pt_regs *regs); asmlinkage void handle_fiq_as_nmi(struct pt_regs *regs); asmlinkage void bad_mode(struct pt_regs *regs, int reason); -asmlinkage int arm_syscall(int no, struct pt_regs *regs); +int arm_syscall (int no,struct pt_regs *regs); asmlinkage void baddataabort(int code, unsigned long instr, struct pt_regs *regs); asmlinkage void __div0(void); asmlinkage void handle_bad_stack(struct pt_regs *regs);
arch/arm/kernel/entry-common.S +5 -18
@@ -264,18 +264,10 @@ ENTRY(vector_swi) mov r0, tbl /* r1 already contains regs */ mov r2, scno @ syscall number from r7 - badr r3, __ret_fast_syscall - bl invoke_syscall - - /* Restore regs into r1 and lr after C call */ + /* We return here no matter what, also pass this as an argument */ badr lr, __ret_fast_syscall - add r1, sp, #S_OFF - -2: cmp scno, #(__ARM_NR_BASE - __NR_SYSCALL_BASE) - eor r0, scno, #__NR_SYSCALL_BASE @ put OS number back - bcs arm_syscall - mov why, #0 @ no longer a real syscall - b sys_ni_syscall @ not private func + mov r3, lr + b invoke_syscall #if defined(CONFIG_OABI_COMPAT) || !defined(CONFIG_AEABI) /*
@@ -308,13 +300,8 @@ __sys_trace: mov r0, tbl badr r3, __sys_trace_return bl invoke_syscall_trace - - /* Restore regs into r1 and lr after C call */ - add r1, sp, #S_R0 + S_OFF @ pointer to regs - badr lr, __sys_trace_return - - cmp scno, #-1 @ skip the syscall? - bne 2b + cmp r0, #-1 + bne __sys_trace_return add sp, sp, #S_OFF @ restore stack __sys_trace_return_nosave:
arch/arm/kernel/syscall.c +11 -2
@@ -12,7 +12,10 @@ __visible int invoke_syscall(void *table, struct pt_regs *regs, int scno, void */* Doing this with return makes sure the stack gets pop:ed */ return invoke_syscall_asm (table, regs, scno, retp); -return 0 ; +if (scno >= __ARM_NR_BASE) +return arm_syscall (scno, regs); + +return sys_ni_syscall (); }int invoke_syscall_trace_asm (void *table,struct pt_regs *regs,int scno,void *retp);
@@ -20,9 +23,15 @@ __ADDRESSABLE(invoke_syscall_trace_asm); __visible int invoke_syscall_trace(void *table, struct pt_regs *regs, int scno, void *retp) { +if (scno ==-1 ) +return -1 ; +if (scno < NR_syscalls)/* Doing this with return makes sure the stack gets pop:ed */ return invoke_syscall_trace_asm (table, regs, scno, retp); -return 0 ; +if (scno >= __ARM_NR_BASE) +return arm_syscall (scno, regs); + +return sys_ni_syscall (); }
arch/arm/kernel/traps.c +1 -1
@@ -609,7 +609,7 @@ do_cache_op(unsigned long start, unsigned long end, int flags)* 0x9f0000 - 0x9fffff are some more esoteric system calls */ #define NR (x) ((__ARM_NR_##x) - __ARM_NR_BASE) -asmlinkage int arm_syscall(int no, struct pt_regs *regs) +int arm_syscall (int no,struct pt_regs *regs) {if ((no >>16 ) != (__ARM_NR_BASE>>16 ))return bad_syscall (no, regs);
Signature · 1 line
2.47.1
RFC PATCH v3 8/30 ARM: entry: Move trace entry to C function
In reply to Linus WalleijThis moves over the code entering into tracing into the C syscall invocation code. As syscall_trace_enter() will provide the current syscall number from the regs we need not provide the syscall number to the invocation function any more.
--- arch/arm/include/asm/syscall.h | 2 +- arch/arm/kernel/entry-common.S | 6 +----- arch/arm/kernel/syscall.c | 5 ++++- 3 files changed, 6 insertions(+), 7 deletions(-)
arch/arm/include/asm/syscall.h +1 -1
@@ -20,7 +20,7 @@extern const unsigned long sys_call_table[];int invoke_syscall (void *table,struct pt_regs *regs,int scno,void *retp); -int invoke_syscall_trace (void *table,struct pt_regs *regs,int scno,void *retp); +int invoke_syscall_trace (void *table,struct pt_regs *regs,void *retp);static inline int syscall_get_nr (struct task_struct *task,struct pt_regs *regs)
arch/arm/kernel/entry-common.S +1 -5
@@ -292,13 +292,9 @@ ENDPROC(vector_swi) * context switches, and waiting for our parent to respond. */ __sys_trace: - add r0, sp, #S_OFF - bl syscall_trace_enter - mov scno, r0 - mov r2, r0 @ scno into r2 add r1, sp, #S_R0 + S_OFF @ pointer to regs mov r0, tbl - badr r3, __sys_trace_return + badr r2, __sys_trace_return bl invoke_syscall_trace cmp r0, #-1 bne __sys_trace_return
arch/arm/kernel/syscall.c +4 -1
@@ -21,8 +21,11 @@ __visible int invoke_syscall(void *table, struct pt_regs *regs, int scno, void *int invoke_syscall_trace_asm (void *table,struct pt_regs *regs,int scno,void *retp);__ADDRESSABLE (invoke_syscall_trace_asm); -__visible int invoke_syscall_trace(void *table, struct pt_regs *regs, int scno, void *retp) +__visible int invoke_syscall_trace(void *table, struct pt_regs *regs, void *retp) { +int scno; + + scno =syscall_trace_enter (regs);if (scno ==-1 )return -1 ;
Signature · 1 line
2.47.1
RFC PATCH v3 9/30 ARM: entry: save the syscall sp in thread_info
In reply to Linus WalleijWe are going to rewrite the syscall handling in C, which means that the stack used by the call code is no longer predicatably 8 bytes (for syscall arguments r4 and r5) but a varying number of bytes depending on how nested the C code is.
However the current code is just assuming it can rewind the stack by adding 8 to sp if a syscall is interrupted by a sigreturn call.
Solve this by storing the entry sp in the per-task struct thread_info and use that in the sigreturn wrapper instead. We already have the thread info available in the SWI entry and sigreturn is probably not so common that retrieveing a pointer to thread_info should affect anything very much.
Storing this per-task in thread_info makes the solution SMP robust.
--- arch/arm/include/asm/thread_info.h | 1 + arch/arm/kernel/asm-offsets.c | 1 + arch/arm/kernel/entry-common.S | 8 ++++++-- 3 files changed, 8 insertions(+), 2 deletions(-)
arch/arm/include/asm/thread_info.h +1 -0
@@ -67,6 +67,7 @@ struct thread_info {__u32 cpu_domain;/* cpu domain */ struct cpu_context_save cpu_context;/* cpu context */ __u32 abi_syscall;/* ABI type and syscall nr */ +__u32 sp_syscall;/* SP when entering syscall */ unsigned long tp_value[2 ];/* TLS registers */ union fp_state fpstate __attribute__((aligned(8)));union vfp_state vfpstate;
arch/arm/kernel/asm-offsets.c +1 -0
@@ -49,6 +49,7 @@ int main(void)DEFINE (TI_CPU_DOMAIN , offsetof(struct thread_info , cpu_domain));DEFINE (TI_CPU_SAVE , offsetof(struct thread_info , cpu_context));DEFINE (TI_ABI_SYSCALL , offsetof(struct thread_info , abi_syscall)); +DEFINE (TI_SP_SYSCALL , offsetof(struct thread_info , sp_syscall));DEFINE (TI_TP_VALUE , offsetof(struct thread_info , tp_value));DEFINE (TI_FPSTATE , offsetof(struct thread_info , fpstate));#ifdef CONFIG_VFP
arch/arm/kernel/entry-common.S +6 -2
@@ -232,6 +232,8 @@ ENTRY(vector_swi) uaccess_disable tbl get_thread_info tsk + /* Save a per-task copy of SP for sigreturn */ + str sp, [tsk, #TI_SP_SYSCALL] adr tbl, sys_call_table @ load syscall table pointer
@@ -377,13 +379,15 @@ sys_syscall: ENDPROC(sys_syscall) sys_sigreturn_wrapper: - add r0, sp, #S_OFF + get_thread_info tsk + ldr r0, [tsk, #TI_SP_SYSCALL] @ read back SP mov why, #0 @ prevent syscall restart handling b sys_sigreturn ENDPROC(sys_sigreturn_wrapper) sys_rt_sigreturn_wrapper: - add r0, sp, #S_OFF + get_thread_info tsk + ldr r0, [tsk, #TI_SP_SYSCALL] @ read back SP mov why, #0 @ prevent syscall restart handling b sys_rt_sigreturn ENDPROC(sys_rt_sigreturn_wrapper)
Signature · 1 line
2.47.1
RFC PATCH v3 10/30 ARM: entry: move all tracing invocation to C
In reply to Linus WalleijThe traced invocation of syscall is rewritten in C, moving over also the call to syscall_trace_exit() to C.
To do this we cannot have invoke_syscall_trace_asm() be a leaf call, which is the mechanism we hithereto relied on to make sure the stack is pulled back to the state we were at before we called out to the code written in C.
Push the registers potentially used by C on the stack, and make a copy of the two stack-pushed syscall arguments on the top of the stack before invoking the syscall and then drop these copied arguments and pop back the registers before returning from invoke_syscall_trace_asm().
As we are now calling from and returning to C, we no longer need to pass the regs pointer around so drop it.
--- arch/arm/include/asm/syscall.h | 2 +- arch/arm/kernel/entry-common.S | 32 +++++++++++++++----------------- arch/arm/kernel/ptrace.c | 2 +- arch/arm/kernel/syscall.c | 30 +++++++++++++++++++++--------- 4 files changed, 38 insertions(+), 28 deletions(-)
arch/arm/include/asm/syscall.h +1 -1
@@ -20,7 +20,7 @@extern const unsigned long sys_call_table[];int invoke_syscall (void *table,struct pt_regs *regs,int scno,void *retp); -int invoke_syscall_trace (void *table,struct pt_regs *regs,void *retp); +void invoke_syscall_trace (void *table,struct pt_regs *regs);static inline int syscall_get_nr (struct task_struct *task,struct pt_regs *regs)
arch/arm/kernel/entry-common.S +15 -17
@@ -296,22 +296,11 @@ ENDPROC(vector_swi) __sys_trace: add r1, sp, #S_R0 + S_OFF @ pointer to regs mov r0, tbl - badr r2, __sys_trace_return bl invoke_syscall_trace - cmp r0, #-1 - bne __sys_trace_return - add sp, sp, #S_OFF @ restore stack - -__sys_trace_return_nosave: - enable_irq_notrace - mov r0, sp - bl syscall_trace_exit + add sp, sp, #S_OFF @ restore stack pointer b ret_to_user -__sys_trace_return: - str r0, [sp, #S_R0 + S_OFF]! @ save returned r0 - mov r0, sp - bl syscall_trace_exit +__sys_trace_return_nosave: b ret_to_user .macro syscall_table_start, sym
@@ -437,18 +426,27 @@ SYM_FUNC_END(invoke_syscall_asm) * r0: syscall table * r1: regs * r2: syscall number - * r3: pointer to return function */ SYM_TYPED_FUNC_START(invoke_syscall_trace_asm) #ifdef CONFIG_CPU_SPECTRE csdb #endif + /* Save registers because we are being called from C */ + push {r4 - r10, lr} mov tbl, r0 + /* Make space to copy the two syscall stack arguments */ + sub sp, sp, #S_OFF mov scno, r2 - mov lr, r3 @ return address - ldmia r1, {r0 - r6} @ reload r0-r6 - stmia sp, {r4, r5} @ update stack arguments + badr lr, __invoke_syscall_trace_ret @ return right here + ldmia r1, {r0 - r6} @ reload r0-r6 from regs + stmia sp, {r4, r5} @ copy stack arguments ldr pc, [tbl, scno, lsl #2] @ call sys_* routine +__invoke_syscall_trace_ret: + /* Drop the copied stack arguments */ + add sp, sp, #S_OFF + pop {r4 - r10, lr} + ARM( mov pc, lr ) + THUMB( bx lr ) SYM_FUNC_END(invoke_syscall_trace_asm) #ifdef CONFIG_OABI_COMPAT
arch/arm/kernel/ptrace.c +1 -1
@@ -888,7 +888,7 @@ asmlinkage int syscall_trace_enter(struct pt_regs *regs)return scno; } -asmlinkage void syscall_trace_exit(struct pt_regs *regs) +void syscall_trace_exit (struct pt_regs *regs) {/* * Audit the syscall before anything else, as a debugger may
arch/arm/kernel/syscall.c +21 -9
@@ -18,23 +18,35 @@ __visible int invoke_syscall(void *table, struct pt_regs *regs, int scno, void *return sys_ni_syscall (); } -int invoke_syscall_trace_asm (void *table,struct pt_regs *regs,int scno,void *retp); +int invoke_syscall_trace_asm (void *table,struct pt_regs *regs,int scno);__ADDRESSABLE (invoke_syscall_trace_asm); -__visible int invoke_syscall_trace(void *table, struct pt_regs *regs, void *retp) +__visible void invoke_syscall_trace(void *table, struct pt_regs *regs) {int scno; +int ret; scno =syscall_trace_enter (regs);if (scno ==-1 ) -return -1 ; + goto trace_exit_nosave; -if (scno < NR_syscalls) -/* Doing this with return makes sure the stack gets pop:ed */ -return invoke_syscall_trace_asm (table, regs, scno, retp); +if (scno < NR_syscalls) { + ret =invoke_syscall_trace_asm (table, regs, scno); + goto trace_exit_save; + } -if (scno >= __ARM_NR_BASE) -return arm_syscall (scno, regs); +if (scno >= __ARM_NR_BASE) { + ret =arm_syscall (scno, regs); + goto trace_exit_save; + } -return sys_ni_syscall (); + ret =sys_ni_syscall (); + +trace_exit_save: +/* Save return value from syscall */ + regs->ARM_r0 = ret; + +trace_exit_nosave: +local_irq_enable (); +syscall_trace_exit (regs); }
Signature · 1 line
2.47.1
RFC PATCH v3 11/30 ARM: entry: Merge the common and trace entry code
In reply to Linus WalleijThe trace entry code now can handle recursive and complex calls in C using stack.
Move the common code over to using that approach.
We now use the ret_fast_syscall return path also when tracing, which appears to work just fine.
--- arch/arm/include/asm/syscall.h | 3 +-- arch/arm/kernel/entry-common.S | 61 +++++++++--------------------------------- arch/arm/kernel/syscall.c | 35 +++++++++++------------- 3 files changed, 28 insertions(+), 71 deletions(-)
arch/arm/include/asm/syscall.h +1 -2
@@ -19,8 +19,7 @@extern const unsigned long sys_call_table[]; -int invoke_syscall (void *table,struct pt_regs *regs,int scno,void *retp); -void invoke_syscall_trace (void *table,struct pt_regs *regs); +int invoke_syscall_trace (void *table,struct pt_regs *regs,int scno);static inline int syscall_get_nr (struct task_struct *task,struct pt_regs *regs)
arch/arm/kernel/entry-common.S +12 -49
113 lines not on this page · raw message
arch/arm/kernel/syscall.c +15 -20
@@ -3,50 +3,45 @@#include <linux/syscalls.h> #include <asm/syscall.h> -int invoke_syscall_asm (void *table,struct pt_regs *regs,int scno,void *retp); -__ADDRESSABLE (invoke_syscall_asm); - -__visible int invoke_syscall(void *table, struct pt_regs *regs, int scno, void *retp) +static inline bool has_syscall_work (unsigned long flags) { -if (scno < NR_syscalls) -/* Doing this with return makes sure the stack gets pop:ed */ -return invoke_syscall_asm (table, regs, scno, retp); - -if (scno >= __ARM_NR_BASE) -return arm_syscall (scno, regs); - -return sys_ni_syscall (); +return unlikely (flags & _TIF_SYSCALL_WORK); }int invoke_syscall_trace_asm (void *table,struct pt_regs *regs,int scno);__ADDRESSABLE (invoke_syscall_trace_asm); -__visible void invoke_syscall_trace(void *table, struct pt_regs *regs) +__visible int invoke_syscall_trace(void *table, struct pt_regs *regs, int scno) { -int scno; +unsigned long flags =read_thread_flags ();int ret; - scno =syscall_trace_enter (regs); -if (scno ==-1 ) - goto trace_exit_nosave; +if (has_syscall_work (flags)) { + scno =syscall_trace_enter (regs); +if (scno ==-1 ) + goto trace_exit_nosave; + }if (scno < NR_syscalls) { ret =invoke_syscall_trace_asm (table, regs, scno); - goto trace_exit_save; + goto exit_save; }if (scno >= __ARM_NR_BASE) { ret =arm_syscall (scno, regs); - goto trace_exit_save; + goto exit_save; } ret =sys_ni_syscall (); -trace_exit_save: +exit_save:/* Save return value from syscall */ regs->ARM_r0 = ret; +if (!has_syscall_work (flags)) +return 0 ; trace_exit_nosave:local_irq_enable ();syscall_trace_exit (regs); +return 1 ; }
Signature · 1 line
2.47.1
RFC PATCH v3 12/30 ARM: entry: Rename syscall invocation
In reply to Linus WalleijNow that we do not have a separate trace syscall invocation path, rename the invocation functions to be generic.
This can be squashed into the previous patch, it is just done here for the previous patch to be easier to get a cleaner diff and be easier to read.
--- arch/arm/include/asm/syscall.h | 2 +- arch/arm/kernel/entry-common.S | 10 +++++----- arch/arm/kernel/syscall.c | 8 ++++---- 3 files changed, 10 insertions(+), 10 deletions(-)
arch/arm/include/asm/syscall.h +1 -1
@@ -19,7 +19,7 @@extern const unsigned long sys_call_table[]; -int invoke_syscall_trace (void *table,struct pt_regs *regs,int scno); +int invoke_syscall (void *table,struct pt_regs *regs,int scno);static inline int syscall_get_nr (struct task_struct *task,struct pt_regs *regs)
arch/arm/kernel/entry-common.S +5 -5
@@ -260,7 +260,7 @@ ENTRY(vector_swi) stmdb sp!, {r4, r5} @ push fifth and sixth args mov r0, tbl mov r2, scno @ syscall number from r7 - bl invoke_syscall_trace + bl invoke_syscall cmp r0, #0 beq ret_fast_syscall /* This path taken when tracing */
@@ -390,7 +390,7 @@ ENDPROC(sys_mmap2) * r1: regs * r2: syscall number */ -SYM_TYPED_FUNC_START(invoke_syscall_trace_asm) +SYM_TYPED_FUNC_START(invoke_syscall_asm) #ifdef CONFIG_CPU_SPECTRE csdb #endif
@@ -400,17 +400,17 @@ SYM_TYPED_FUNC_START(invoke_syscall_trace_asm) /* Make space to copy the two syscall stack arguments */ sub sp, sp, #S_OFF mov scno, r2 - badr lr, __invoke_syscall_trace_ret @ return right here + badr lr, __invoke_syscall_ret @ return right here ldmia r1, {r0 - r6} @ reload r0-r6 from regs stmia sp, {r4, r5} @ copy stack arguments ldr pc, [tbl, scno, lsl #2] @ call sys_* routine -__invoke_syscall_trace_ret: +__invoke_syscall_ret: /* Drop the copied stack arguments */ add sp, sp, #S_OFF pop {r4 - r10, lr} ARM( mov pc, lr ) THUMB( bx lr ) -SYM_FUNC_END(invoke_syscall_trace_asm) +SYM_FUNC_END(invoke_syscall_asm) #ifdef CONFIG_OABI_COMPAT
arch/arm/kernel/syscall.c +4 -4
@@ -8,10 +8,10 @@ static inline bool has_syscall_work(unsigned long flags)return unlikely (flags & _TIF_SYSCALL_WORK); } -int invoke_syscall_trace_asm (void *table,struct pt_regs *regs,int scno); -__ADDRESSABLE (invoke_syscall_trace_asm); +int invoke_syscall_asm (void *table,struct pt_regs *regs,int scno); +__ADDRESSABLE (invoke_syscall_asm); -__visible int invoke_syscall_trace(void *table, struct pt_regs *regs, int scno) +__visible int invoke_syscall(void *table, struct pt_regs *regs, int scno) {unsigned long flags =read_thread_flags ();int ret;
@@ -23,7 +23,7 @@ __visible int invoke_syscall_trace(void *table, struct pt_regs *regs, int scno) }if (scno < NR_syscalls) { - ret =invoke_syscall_trace_asm (table, regs, scno); + ret =invoke_syscall_asm (table, regs, scno); goto exit_save; }
Signature · 1 line
2.47.1
RFC PATCH v3 13/30 ARM: entry: Create user_mode_enter/exit
In reply to Linus WalleijThe callbacks to the context tracking will be repurposed for several uses that are needed on the IRQ transition to/from userspace.
Rename the macro, establish call sites in C calling into the context tracking following the corresponding generic entry function prototypes, despite the assembly macro names become a bit long this makes it clear to readers exactly what is going on and where this call will go.
Drop the ifdefs pertaining to context tracking from the macro. The C calls we will use have stubs that will compile these out anyway.
The inversion of the signature of the context tracking calls are especially confusing since the generic entry uses the reverse semantics: *enter from* user mode (to kernel mode) and *exit to* user mode (from kernel mode) instead of the other way around as the old context tracker code user_exit_callable() and user_enter_callable() which have inverted semantics.
--- arch/arm/include/asm/entry.h | 14 ++++++++++++++ arch/arm/kernel/Makefile | 2 +- arch/arm/kernel/entry-armv.S | 2 +- arch/arm/kernel/entry-common.S | 4 ++-- arch/arm/kernel/entry-header.S | 24 ++++++++++++------------ arch/arm/kernel/entry.c | 15 +++++++++++++++ 6 files changed, 45 insertions(+), 16 deletions(-)
arch/arm/include/asm/entry.h new file +14 -0
@@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __ASM_ENTRY_H__ +#define __ASM_ENTRY_H__ + +struct pt_regs ; + +/* +* These are copies of generic entry headers so we can transition +* to generic entry once they are semantically equivalent. +*/ +void irqentry_enter_from_user_mode (struct pt_regs *regs); +void irqentry_exit_to_user_mode (struct pt_regs *regs); + +#endif /* __ASM_ENTRY_H__ */
arch/arm/kernel/Makefile +1 -1
@@ -17,7 +17,7 @@ CFLAGS_REMOVE_return_address.o = -pg # Object file lists. -obj-y := elf.o entry-common.o irq.o opcodes.o \ +obj-y := elf.o entry.o entry-common.o irq.o opcodes.o \ process.o ptrace.o reboot.o io.o \ setup.o signal.o sigreturn_codes.o \ stacktrace.o sys_arm.o time.o traps.o \
arch/arm/kernel/entry-armv.S +1 -1
@@ -406,7 +406,7 @@ ENDPROC(__fiq_abt) #ifdef CONFIG_TRACE_IRQFLAGS bl trace_hardirqs_off #endif - ct_user_exit save = 0 + asm_irqentry_enter_from_user_mode save = 0 .endif .endm
arch/arm/kernel/entry-common.S +2 -2
@@ -111,7 +111,7 @@ ENTRY(ret_to_user_from_irq) no_work_pending: asm_trace_hardirqs_on save = 0 - ct_user_enter save = 0 + asm_irqentry_exit_to_user_mode save = 0 #ifdef CONFIG_GCC_PLUGIN_STACKLEAK bl stackleak_erase_on_task_stack
@@ -191,7 +191,7 @@ ENTRY(vector_swi) alignment_trap r10, ip, cr_alignment asm_trace_hardirqs_on save=0 enable_irq_notrace - ct_user_exit save=0 + asm_irqentry_enter_from_user_mode save = 0 /* * Get the system call number.
arch/arm/kernel/entry-header.S +12 -12
@@ -362,31 +362,31 @@ ALT_UP_B(.L1_\@) .endm /* - * Context tracking subsystem. Used to instrument transitions + * Context tracking and other mode transitions. Used to instrument transitions * between user and kernel mode. - */ - .macro ct_user_exit, save = 1 -#ifdef CONFIG_CONTEXT_TRACKING_USER +*/ + .macro asm_irqentry_enter_from_user_mode, save = 1 .if \save stmdb sp!, {r0-r3, ip, lr} - bl user_exit_callable + mov r0, sp @ regs + bl irqentry_enter_from_user_mode ldmia sp!, {r0-r3, ip, lr} .else - bl user_exit_callable + mov r0, sp @ regs + bl irqentry_enter_from_user_mode .endif -#endif .endm - .macro ct_user_enter, save = 1 -#ifdef CONFIG_CONTEXT_TRACKING_USER + .macro asm_irqentry_exit_to_user_mode, save = 1 .if \save stmdb sp!, {r0-r3, ip, lr} - bl user_enter_callable + mov r0, sp @ regs + bl irqentry_exit_to_user_mode ldmia sp!, {r0-r3, ip, lr} .else - bl user_enter_callable + mov r0, sp @ regs + bl irqentry_exit_to_user_mode .endif -#endif .endm /*
arch/arm/kernel/entry.c new file +15 -0
@@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <asm/entry.h> +#include <linux/context_tracking.h> + +noinstr void irqentry_enter_from_user_mode(struct pt_regs *regs) +{ +/* This context tracking call has inverse naming */ +user_exit_callable (); +} + +noinstr void irqentry_exit_to_user_mode(struct pt_regs *regs) +{ +/* This context tracking call has inverse naming */ +user_enter_callable (); +}
Signature · 1 line
2.47.1
RFC PATCH v3 14/30 ARM: entry: Drop trace argument from usr_entry macro
In reply to Linus WalleijThe trace argument select whether to look into tracing of the hardirqs for the exceptions from userspace, but also selects whether to call further down to establish context for RCU.
Split off a second macro that is called on everything except the FIQ since this is the only outlier.
--- arch/arm/kernel/entry-armv.S | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
arch/arm/kernel/entry-armv.S +9 -4
@@ -357,7 +357,7 @@ ENDPROC(__fiq_abt) #error "sizeof(struct pt_regs) must be a multiple of 8" #endif - .macro usr_entry, trace=1, uaccess=1 + .macro usr_entry, uaccess=1 UNWIND(.fnstart ) UNWIND(.cantunwind ) @ don't unwind the user space sub sp, sp, #PT_REGS_SIZE
@@ -401,13 +401,14 @@ ENDPROC(__fiq_abt) @ Clear FP to mark the first stack frame @ zero_fp + .endm - .if \trace + /* Called after usr_entry for everything except FIQ */ + .macro usr_entry_enter #ifdef CONFIG_TRACE_IRQFLAGS bl trace_hardirqs_off #endif asm_irqentry_enter_from_user_mode save = 0 - .endif .endm .macro kuser_cmpxchg_check
@@ -429,6 +430,7 @@ ENDPROC(__fiq_abt) .align 5 __dabt_usr: usr_entry uaccess=0 + usr_entry_enter kuser_cmpxchg_check mov r2, sp dabt_helper
@@ -439,6 +441,7 @@ ENDPROC(__dabt_usr) .align 5 __irq_usr: usr_entry + usr_entry_enter kuser_cmpxchg_check irq_handler from_user=1 get_thread_info tsk
@@ -452,6 +455,7 @@ ENDPROC(__irq_usr) .align 5 __und_usr: usr_entry uaccess=0 + usr_entry_enter @ IRQs must be enabled before attempting to read the instruction from @ user space since that could cause a page/translation fault if the
@@ -476,6 +480,7 @@ ENDPROC(__und_usr) .align 5 __pabt_usr: usr_entry + usr_entry_enter mov r2, sp @ regs pabt_helper UNWIND(.fnend )
@@ -495,7 +500,7 @@ ENDPROC(ret_from_exception) .align 5 __fiq_usr: - usr_entry trace=0 + usr_entry kuser_cmpxchg_check mov r0, sp @ struct pt_regs *regs bl handle_fiq_as_nmi
Signature · 1 line
2.47.1
RFC PATCH v3 15/30 ARM: entry: Separate call path for syscall SWI entry
In reply to Linus WalleijThe syscalls (SWIs, software interrupts) are deviating from how any other interrupts are handled as they enable the IRQs again while processing the syscall, while "hard" IRQs disable all interrupts until they are handled.
Break out syscall_enter_from_user_mode() into its own function and call it instead of irqentry_enter_from_user_mode().
As we are moving toward generic entry, we use the signature from the generic function.
As the generic function requires the syscall number to be determined, we move the call down below the code that figures out the syscall number, the only practical effect should be that interrupts are re-enabled a few instructions later.
As we move the trace_hardirqs_on/off() calls into C, we can just get rid of the helper macro usr_entry_enter again and call asm_irqentry_enter_from_user_mode directly.
--- arch/arm/include/asm/entry.h | 1 + arch/arm/kernel/entry-armv.S | 16 ++++------------ arch/arm/kernel/entry-common.S | 18 +++++++++++++----- arch/arm/kernel/entry.c | 14 ++++++++++++++ 4 files changed, 32 insertions(+), 17 deletions(-)
arch/arm/include/asm/entry.h +1 -0
@@ -8,6 +8,7 @@ struct pt_regs;* These are copies of generic entry headers so we can transition * to generic entry once they are semantically equivalent. */ +long syscall_enter_from_user_mode (struct pt_regs *regs,long );void irqentry_enter_from_user_mode (struct pt_regs *regs);void irqentry_exit_to_user_mode (struct pt_regs *regs);
arch/arm/kernel/entry-armv.S +4 -12
@@ -403,14 +403,6 @@ ENDPROC(__fiq_abt) zero_fp .endm - /* Called after usr_entry for everything except FIQ */ - .macro usr_entry_enter -#ifdef CONFIG_TRACE_IRQFLAGS - bl trace_hardirqs_off -#endif - asm_irqentry_enter_from_user_mode save = 0 - .endm - .macro kuser_cmpxchg_check #if !defined(CONFIG_CPU_32v6K) && defined(CONFIG_KUSER_HELPERS) #ifndef CONFIG_MMU
@@ -430,7 +422,7 @@ ENDPROC(__fiq_abt) .align 5 __dabt_usr: usr_entry uaccess=0 - usr_entry_enter + asm_irqentry_enter_from_user_mode save = 0 kuser_cmpxchg_check mov r2, sp dabt_helper
@@ -441,7 +433,7 @@ ENDPROC(__dabt_usr) .align 5 __irq_usr: usr_entry - usr_entry_enter + asm_irqentry_enter_from_user_mode save = 0 kuser_cmpxchg_check irq_handler from_user=1 get_thread_info tsk
@@ -455,7 +447,7 @@ ENDPROC(__irq_usr) .align 5 __und_usr: usr_entry uaccess=0 - usr_entry_enter + asm_irqentry_enter_from_user_mode save = 0 @ IRQs must be enabled before attempting to read the instruction from @ user space since that could cause a page/translation fault if the
@@ -480,7 +472,7 @@ ENDPROC(__und_usr) .align 5 __pabt_usr: usr_entry - usr_entry_enter + asm_irqentry_enter_from_user_mode save = 0 mov r2, sp @ regs pabt_helper UNWIND(.fnend )
arch/arm/kernel/entry-common.S +13 -5
@@ -109,8 +109,6 @@ ENTRY(ret_to_user_from_irq) movs r1, r1, lsl #16 bne slow_work_pending no_work_pending: - asm_trace_hardirqs_on save = 0 - asm_irqentry_exit_to_user_mode save = 0 #ifdef CONFIG_GCC_PLUGIN_STACKLEAK
@@ -189,9 +187,6 @@ ENTRY(vector_swi) reload_current r10, ip zero_fp alignment_trap r10, ip, cr_alignment - asm_trace_hardirqs_on save=0 - enable_irq_notrace - asm_irqentry_enter_from_user_mode save = 0 /* * Get the system call number.
@@ -256,6 +251,19 @@ ENTRY(vector_swi) #else str scno, [tsk, #TI_ABI_SYSCALL] #endif + + /* + * Calling out to C to be careful to save and restore registers. + * This call could modify the syscall number. scno is r7 so we + * do not save and restore r7. + */ + mov r0, sp @ regs + mov r1, scno + push {r4 - r6, r8 - r10, lr} + bl syscall_enter_from_user_mode + pop {r4 - r6, r8 - r10, lr} + mov scno, r0 + mov r1, sp @ put regs into r1 stmdb sp!, {r4, r5} @ push fifth and sixth args mov r0, tbl
arch/arm/kernel/entry.c +14 -0
@@ -1,15 +1,29 @@// SPDX-License-Identifier: GPL-2.0 #include <asm/entry.h> #include <linux/context_tracking.h> +#include <linux/irqflags.h> + +long syscall_enter_from_user_mode (struct pt_regs *regs,long syscall) +{ +trace_hardirqs_on (); +local_irq_enable (); +/* This context tracking call has inverse naming */ +user_exit_callable (); + +/* This will optionally be modified later */ +return syscall; +} noinstr void irqentry_enter_from_user_mode(struct pt_regs *regs) { +trace_hardirqs_off ();/* This context tracking call has inverse naming */ user_exit_callable (); } noinstr void irqentry_exit_to_user_mode(struct pt_regs *regs) { +trace_hardirqs_on ();/* This context tracking call has inverse naming */ user_enter_callable (); }
Signature · 1 line
2.47.1
RFC PATCH v3 16/30 ARM: entry: Drop argument to asm_irqentry macros
In reply to Linus Walleijasm_irqentry_enter_from_user_mode and asm_irqentry_exit_to_user_mode have a "save" argument that will save and restore registers before the call to the C function.
Now all invocations set this argument to 0 so drop the surplus code.
--- arch/arm/kernel/entry-armv.S | 8 ++++---- arch/arm/kernel/entry-common.S | 2 +- arch/arm/kernel/entry-header.S | 18 ++---------------- 3 files changed, 7 insertions(+), 21 deletions(-)
arch/arm/kernel/entry-armv.S +4 -4
@@ -422,7 +422,7 @@ ENDPROC(__fiq_abt) .align 5 __dabt_usr: usr_entry uaccess=0 - asm_irqentry_enter_from_user_mode save = 0 + asm_irqentry_enter_from_user_mode kuser_cmpxchg_check mov r2, sp dabt_helper
@@ -433,7 +433,7 @@ ENDPROC(__dabt_usr) .align 5 __irq_usr: usr_entry - asm_irqentry_enter_from_user_mode save = 0 + asm_irqentry_enter_from_user_mode kuser_cmpxchg_check irq_handler from_user=1 get_thread_info tsk
@@ -447,7 +447,7 @@ ENDPROC(__irq_usr) .align 5 __und_usr: usr_entry uaccess=0 - asm_irqentry_enter_from_user_mode save = 0 + asm_irqentry_enter_from_user_mode @ IRQs must be enabled before attempting to read the instruction from @ user space since that could cause a page/translation fault if the
@@ -472,7 +472,7 @@ ENDPROC(__und_usr) .align 5 __pabt_usr: usr_entry - asm_irqentry_enter_from_user_mode save = 0 + asm_irqentry_enter_from_user_mode mov r2, sp @ regs pabt_helper UNWIND(.fnend )
arch/arm/kernel/entry-common.S +1 -1
@@ -109,7 +109,7 @@ ENTRY(ret_to_user_from_irq) movs r1, r1, lsl #16 bne slow_work_pending no_work_pending: - asm_irqentry_exit_to_user_mode save = 0 + asm_irqentry_exit_to_user_mode #ifdef CONFIG_GCC_PLUGIN_STACKLEAK bl stackleak_erase_on_task_stack
arch/arm/kernel/entry-header.S +2 -16
@@ -365,28 +365,14 @@ ALT_UP_B(.L1_\@) * Context tracking and other mode transitions. Used to instrument transitions * between user and kernel mode. */ - .macro asm_irqentry_enter_from_user_mode, save = 1 - .if \save - stmdb sp!, {r0-r3, ip, lr} + .macro asm_irqentry_enter_from_user_mode mov r0, sp @ regs bl irqentry_enter_from_user_mode - ldmia sp!, {r0-r3, ip, lr} - .else - mov r0, sp @ regs - bl irqentry_enter_from_user_mode - .endif .endm - .macro asm_irqentry_exit_to_user_mode, save = 1 - .if \save - stmdb sp!, {r0-r3, ip, lr} + .macro asm_irqentry_exit_to_user_mode mov r0, sp @ regs bl irqentry_exit_to_user_mode - ldmia sp!, {r0-r3, ip, lr} - .else - mov r0, sp @ regs - bl irqentry_exit_to_user_mode - .endif .endm /*
Signature · 1 line
2.47.1
RFC PATCH v3 17/30 ARM: entry: Implement syscall_exit_to_user_mode()
In reply to Linus WalleijMirroring syscall_enter_from_user_mode() we create the sibling call syscall_exit_to_user_mode() and start to move some stuff over from assembly, first the rseq call and the interrupt disablement.
--- arch/arm/include/asm/entry.h | 1 + arch/arm/include/asm/signal.h | 1 - arch/arm/kernel/entry-common.S | 10 ++-------- arch/arm/kernel/entry.c | 8 ++++++++ arch/arm/kernel/signal.c | 7 ------- 5 files changed, 11 insertions(+), 16 deletions(-)
arch/arm/include/asm/entry.h +1 -0
@@ -9,6 +9,7 @@ struct pt_regs;* to generic entry once they are semantically equivalent. */ long syscall_enter_from_user_mode (struct pt_regs *regs,long ); +void syscall_exit_to_user_mode (struct pt_regs *regs);void irqentry_enter_from_user_mode (struct pt_regs *regs);void irqentry_exit_to_user_mode (struct pt_regs *regs);
arch/arm/include/asm/signal.h +0 -1
@@ -23,7 +23,6 @@ typedef struct {#include <asm/sigcontext.h> -void do_rseq_syscall (struct pt_regs *regs);void do_work_pending (struct pt_regs *regs,unsigned int thread_flags);#endif
arch/arm/kernel/entry-common.S +2 -8
@@ -65,12 +65,9 @@ ret_fast_syscall: UNWIND(.fnstart ) UNWIND(.cantunwind ) add sp, sp, #(S_R0 + S_OFF) -#if IS_ENABLED(CONFIG_DEBUG_RSEQ) /* do_rseq_syscall needs interrupts enabled. */ mov r0, sp @ 'regs' - bl do_rseq_syscall -#endif - disable_irq_notrace @ disable interrupts + bl syscall_exit_to_user_mode ldr r1, [tsk, #TI_FLAGS] @ re-check for syscall tracing movs r1, r1, lsl #16 beq no_work_pending
@@ -97,13 +94,10 @@ ENDPROC(ret_fast_syscall) * do_work_pending() will update this state if necessary. */ ENTRY(ret_to_user) -#if IS_ENABLED(CONFIG_DEBUG_RSEQ) /* do_rseq_syscall needs interrupts enabled. */ enable_irq_notrace @ enable interrupts mov r0, sp @ 'regs' - bl do_rseq_syscall -#endif - disable_irq_notrace @ disable interrupts + bl syscall_exit_to_user_mode ENTRY(ret_to_user_from_irq) ldr r1, [tsk, #TI_FLAGS] movs r1, r1, lsl #16
arch/arm/kernel/entry.c +8 -0
@@ -1,7 +1,9 @@// SPDX-License-Identifier: GPL-2.0 #include <asm/entry.h> +#include <asm/signal.h> #include <linux/context_tracking.h> #include <linux/irqflags.h> +#include <linux/rseq.h> long syscall_enter_from_user_mode (struct pt_regs *regs,long syscall) {
@@ -14,6 +16,12 @@ long syscall_enter_from_user_mode(struct pt_regs *regs, long syscall)return syscall; } +void syscall_exit_to_user_mode (struct pt_regs *regs) +{ +rseq_syscall (regs); +local_irq_disable (); +} + noinstr void irqentry_enter_from_user_mode(struct pt_regs *regs) {trace_hardirqs_off ();
arch/arm/kernel/signal.c +0 -7
@@ -659,13 +659,6 @@ struct page *get_signal_page(void)return page; } -#ifdef CONFIG_DEBUG_RSEQ -asmlinkage void do_rseq_syscall(struct pt_regs *regs) -{ -rseq_syscall (regs); -} -#endif -/* * Compile-time assertions for siginfo_t offsets. Check NSIG* as well, as * changes likely come with new fields that should be added below.
Signature · 1 line
2.47.1
RFC PATCH v3 18/30 ARM: entry: Drop the superfast ret_fast_syscall
In reply to Linus WalleijWhen not using any irqflags tracing, context tracking or rseq debugging, the ARM kernel uses a "superfast" ret_fast_syscall with a very few assembly instructions.
With generic entry, such local optimizations are not really possible, we need to call C unconditionally and take the (hopefully small) overhead.
With generic entry in C, most of the tracing (etc) calls are replaced by stubs and optimized out by the compiler.
--- arch/arm/kernel/entry-common.S | 29 +---------------------------- 1 file changed, 1 insertion(+), 28 deletions(-)
arch/arm/kernel/entry-common.S +1 -28
@@ -30,34 +30,8 @@ saved_pc .req lr .section .entry.text,"ax",%progbits .align 5 -#if !(IS_ENABLED(CONFIG_TRACE_IRQFLAGS) || IS_ENABLED(CONFIG_CONTEXT_TRACKING_USER) || \ - IS_ENABLED(CONFIG_DEBUG_RSEQ)) /* - * This is the fast syscall return path. We do as little as possible here, - * such as avoiding writing r0 to the stack. We only use this path if we - * have tracing, context tracking and rseq debug disabled - the overheads - * from those features make this path too inefficient. - */ -ret_fast_syscall: - UNWIND(.fnstart ) - UNWIND(.cantunwind ) - disable_irq_notrace @ disable interrupts - ldr r1, [tsk, #TI_FLAGS] @ re-check for syscall tracing - movs r1, r1, lsl #16 - bne fast_work_pending - - restore_user_regs fast = 0, offset = S_OFF - UNWIND(.fnend ) -ENDPROC(ret_fast_syscall) - - /* Ok, we need to do extra processing, enter the slow path. */ -fast_work_pending: - add sp, sp, #(S_R0 + S_OFF) - /* fall through to work_pending */ -#else -/* - * The "replacement" ret_fast_syscall for when tracing, context tracking, - * or rseq debug is enabled. As we will need to call out to some C functions, + * As we will need to call out to some C functions, * we save r0 first to avoid needing to save registers around each C function * call. */
@@ -75,7 +49,6 @@ ret_fast_syscall: ENDPROC(ret_fast_syscall) /* Slower path - fall through to work_pending */ -#endif tst r1, #_TIF_SYSCALL_WORK beq slow_work_pending
Signature · 1 line
2.47.1
RFC PATCH v3 19/30 ARM: entry: Remove fast and offset register restore
In reply to Linus WalleijNow all remaining invocations of restore_user_regs has fast = 0 and offset = 0 set, so drop these parameters to the macro.
--- arch/arm/kernel/entry-armv.S | 2 +- arch/arm/kernel/entry-common.S | 2 +- arch/arm/kernel/entry-header.S | 29 +++++++++-------------------- 3 files changed, 11 insertions(+), 22 deletions(-)
arch/arm/kernel/entry-armv.S +1 -1
@@ -497,7 +497,7 @@ __fiq_usr: mov r0, sp @ struct pt_regs *regs bl handle_fiq_as_nmi get_thread_info tsk - restore_user_regs fast = 0, offset = 0 + restore_user_regs UNWIND(.fnend ) ENDPROC(__fiq_usr)
arch/arm/kernel/entry-common.S +1 -1
@@ -81,7 +81,7 @@ no_work_pending: #ifdef CONFIG_GCC_PLUGIN_STACKLEAK bl stackleak_erase_on_task_stack #endif - restore_user_regs fast = 0, offset = 0 + restore_user_regs ENDPROC(ret_to_user_from_irq) ENDPROC(ret_to_user)
arch/arm/kernel/entry-header.S +9 -20
@@ -290,7 +290,7 @@ .endm - .macro restore_user_regs, fast = 0, offset = 0 + .macro restore_user_regs #if defined(CONFIG_CPU_32v6K) && \ (!defined(CONFIG_CPU_V6) || defined(CONFIG_SMP)) #ifdef CONFIG_CPU_V6
@@ -309,8 +309,8 @@ ALT_UP_B(.L1_\@) #ifndef CONFIG_THUMB2_KERNEL @ ARM mode restore mov r2, sp - ldr r1, [r2, #\offset + S_PSR] @ get calling cpsr - ldr lr, [r2, #\offset + S_PC]! @ get pc + ldr r1, [r2, S_PSR] @ get calling cpsr + ldr lr, [r2, S_PC]! @ get pc tst r1, #PSR_I_BIT | 0x0f bne 1f msr spsr_cxsf, r1 @ save in spsr_svc
@@ -318,31 +318,24 @@ ALT_UP_B(.L1_\@) @ We must avoid clrex due to Cortex-A15 erratum #830321 strex r1, r2, [r2] @ clear the exclusive monitor #endif - .if \fast - ldmdb r2, {r1 - lr}^ @ get calling r1 - lr - .else ldmdb r2, {r0 - lr}^ @ get calling r0 - lr - .endif mov r0, r0 @ ARMv5T and earlier require a nop @ after ldm {}^ - add sp, sp, #\offset + PT_REGS_SIZE + add sp, sp, #PT_REGS_SIZE movs pc, lr @ return & move spsr_svc into cpsr 1: bug "Returning to usermode but unexpected PSR bits set?", \@ #elif defined(CONFIG_CPU_V7M) @ V7M restore. @ Note that we don't need to do clrex here as clearing the local @ monitor is part of the exception entry and exit sequence. - .if \offset - add sp, #\offset - .endif - v7m_exception_slow_exit ret_r0 = \fast + v7m_exception_slow_exit ret_r0 = 0 #else @ Thumb mode restore mov r2, sp - load_user_sp_lr r2, r3, \offset + S_SP @ calling sp, lr - ldr r1, [sp, #\offset + S_PSR] @ get calling cpsr - ldr lr, [sp, #\offset + S_PC] @ get pc - add sp, sp, #\offset + S_SP + load_user_sp_lr r2, r3, S_SP @ calling sp, lr + ldr r1, [sp, S_PSR] @ get calling cpsr + ldr lr, [sp, S_PC] @ get pc + add sp, sp, S_SP tst r1, #PSR_I_BIT | 0x0f bne 1f msr spsr_cxsf, r1 @ save in spsr_svc
@@ -350,11 +343,7 @@ ALT_UP_B(.L1_\@) @ We must avoid clrex due to Cortex-A15 erratum #830321 strex r1, r2, [sp] @ clear the exclusive monitor - .if \fast - ldmdb sp, {r1 - r12} @ get calling r1 - r12 - .else ldmdb sp, {r0 - r12} @ get calling r0 - r12 - .endif add sp, sp, #PT_REGS_SIZE - S_SP movs pc, lr @ return & move spsr_svc into cpsr 1: bug "Returning to usermode but unexpected PSR bits set?", \@
Signature · 1 line
2.47.1
RFC PATCH v3 20/30 ARM: entry: Untangle ret_fast_syscall/to_user
In reply to Linus WalleijThe return to userspace is a bit hard to follow. To prepare the ground for further refactorings we rewrite the functions to ret_fast_syscall and ret_to_user are mostly stand-alone symbols.
--- arch/arm/kernel/entry-common.S | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-)
arch/arm/kernel/entry-common.S +15 -10
@@ -44,20 +44,23 @@ ret_fast_syscall: bl syscall_exit_to_user_mode ldr r1, [tsk, #TI_FLAGS] @ re-check for syscall tracing movs r1, r1, lsl #16 - beq no_work_pending - UNWIND(.fnend ) -ENDPROC(ret_fast_syscall) - - /* Slower path - fall through to work_pending */ + beq 2f tst r1, #_TIF_SYSCALL_WORK - beq slow_work_pending + beq 1f + b ret_to_user -slow_work_pending: - mov r0, sp @ 'regs' +1: mov r0, sp @ 'regs' bl do_work_pending - b no_work_pending + +2: asm_irqentry_exit_to_user_mode + +#ifdef CONFIG_GCC_PLUGIN_STACKLEAK + bl stackleak_erase_on_task_stack +#endif + restore_user_regs + UNWIND(.fnend ) ENDPROC(ret_fast_syscall) /*
@@ -74,7 +77,9 @@ ENTRY(ret_to_user) ENTRY(ret_to_user_from_irq) ldr r1, [tsk, #TI_FLAGS] movs r1, r1, lsl #16 - bne slow_work_pending + beq no_work_pending + mov r0, sp @ 'regs' + bl do_work_pending no_work_pending: asm_irqentry_exit_to_user_mode
Signature · 1 line
2.47.1
RFC PATCH v3 21/30 ARM: entry: Do not double-call exit functions
In reply to Linus WalleijThis is a semantic change to ret_fast_syscall: if there is no work pending the function used to jump to ret_to_user, instead we proceed to return from the syscall.
If we jump to ret_to_user, IRQs are enabled and we call syscall_exit_to_user_mode a second time; at this point a this just means a second call to rseq_syscall(), but it seems wrong and we want syscall_exit_to_user_mode to be called exactly once.
Apparently the ARM kernel has under some circumstances called rseq_syscall() twice on the exit to userspace without side effects.
ret_to_user also checks for pending work a second time under ret_to_user_from_irq, but this isn't even needed any more on the fast return path: this was needed because the syscall invocation enabled interrupts when doing syscall tracing, and all tracing invocation has been moved over to C in earlier patches.
--- arch/arm/kernel/entry-common.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm/kernel/entry-common.S +1 -1
@@ -49,7 +49,7 @@ ret_fast_syscall: tst r1, #_TIF_SYSCALL_WORK beq 1f - b ret_to_user + b 2f 1: mov r0, sp @ 'regs' bl do_work_pending
Signature · 1 line
2.47.1
RFC PATCH v3 22/30 ARM: entry: Move work processing to C
In reply to Linus WalleijAt the end of a syscall software interrupt and every time we exit an interrupt handler we check if there is work to do, i.e. if the current thread (userspace or kernel daemon) has set the _TIF_SYSCALL_WORK flag.
Move this processing over to the C entry code.
--- arch/arm/kernel/entry-common.S | 14 +------------- arch/arm/kernel/entry.c | 10 ++++++++++ 2 files changed, 11 insertions(+), 13 deletions(-)
arch/arm/kernel/entry-common.S +1 -13
@@ -42,19 +42,7 @@ ret_fast_syscall: /* do_rseq_syscall needs interrupts enabled. */ mov r0, sp @ 'regs' bl syscall_exit_to_user_mode - ldr r1, [tsk, #TI_FLAGS] @ re-check for syscall tracing - movs r1, r1, lsl #16 - beq 2f - - tst r1, #_TIF_SYSCALL_WORK - beq 1f - - b 2f - -1: mov r0, sp @ 'regs' - bl do_work_pending - -2: asm_irqentry_exit_to_user_mode + asm_irqentry_exit_to_user_mode #ifdef CONFIG_GCC_PLUGIN_STACKLEAK bl stackleak_erase_on_task_stack
arch/arm/kernel/entry.c +10 -0
@@ -1,10 +1,16 @@// SPDX-License-Identifier: GPL-2.0 #include <asm/entry.h> +#include <asm/ptrace.h> #include <asm/signal.h> #include <linux/context_tracking.h> #include <linux/irqflags.h> #include <linux/rseq.h> +static inline bool has_syscall_work (unsigned long flags) +{ +return unlikely (flags & _TIF_SYSCALL_WORK); +} +long syscall_enter_from_user_mode (struct pt_regs *regs,long syscall) {trace_hardirqs_on ();
@@ -18,8 +24,12 @@ long syscall_enter_from_user_mode(struct pt_regs *regs, long syscall)void syscall_exit_to_user_mode (struct pt_regs *regs) { +unsigned long flags =read_thread_flags (); +rseq_syscall (regs);local_irq_disable (); +if (has_syscall_work (flags)) +do_work_pending (regs, flags); } noinstr void irqentry_enter_from_user_mode(struct pt_regs *regs)
Signature · 1 line
2.47.1
RFC PATCH v3 23/30 ARM: entry: Stop exiting syscalls like IRQs
In reply to Linus WalleijThe syscalls are issued as software interrupts, and to this point they were sharing code with the other exception handlers, but this does not work with generic entry.
Make syscall_exit_to_user_mode do what irqentry_exit_to_user_mode does at it's tail, and drop the invocations of irqentry_exit_to_user_mode from the syscall path so that these are now exception-exclusive.
Split ret_to_user and ret_to_user_from_irq into two distinct execution paths.
--- arch/arm/kernel/entry-common.S | 14 ++++++++++++-- arch/arm/kernel/entry.c | 4 ++++ 2 files changed, 16 insertions(+), 2 deletions(-)
arch/arm/kernel/entry-common.S +12 -2
@@ -42,7 +42,6 @@ ret_fast_syscall: /* do_rseq_syscall needs interrupts enabled. */ mov r0, sp @ 'regs' bl syscall_exit_to_user_mode - asm_irqentry_exit_to_user_mode #ifdef CONFIG_GCC_PLUGIN_STACKLEAK bl stackleak_erase_on_task_stack
@@ -62,6 +61,18 @@ ENTRY(ret_to_user) enable_irq_notrace @ enable interrupts mov r0, sp @ 'regs' bl syscall_exit_to_user_mode + ldr r1, [tsk, #TI_FLAGS] + movs r1, r1, lsl #16 + beq 1f + mov r0, sp @ 'regs' + bl do_work_pending +1: +#ifdef CONFIG_GCC_PLUGIN_STACKLEAK + bl stackleak_erase_on_task_stack +#endif + restore_user_regs +ENDPROC(ret_to_user) + ENTRY(ret_to_user_from_irq) ldr r1, [tsk, #TI_FLAGS] movs r1, r1, lsl #16
@@ -76,7 +87,6 @@ no_work_pending: #endif restore_user_regs ENDPROC(ret_to_user_from_irq) -ENDPROC(ret_to_user) /* * This is how we return from a fork.
arch/arm/kernel/entry.c +4 -0
@@ -30,6 +30,10 @@ void syscall_exit_to_user_mode(struct pt_regs *regs)local_irq_disable ();if (has_syscall_work (flags))do_work_pending (regs, flags); + +trace_hardirqs_on (); +/* This context tracking call has inverse naming */ +user_enter_callable (); } noinstr void irqentry_enter_from_user_mode(struct pt_regs *regs)
Signature · 1 line
2.47.1
RFC PATCH v3 24/30 ARM: entry: Complete syscall and IRQ transition to C
In reply to Linus WalleijThis moves over the last few lines of assembly to C. The subtle change is that in return to userspace from syscall (SWI) or interrupt, we need to call do_work_pending() as soon as the thread flags are != 0, just checking for work with _TIF_SYSCALL_WORK is not enough (the machine will freeze if we do that).
This is because do_work_pending() does not just handle
work: it handles _TIF_NEED_RESCHED, _TIF_SIGPENDING, _TIF_NOTIFY_SIGNAL and _TIF_UPROBE as well.
--- arch/arm/kernel/entry-common.S | 13 +------------ arch/arm/kernel/entry.c | 19 +++++++++++++------ arch/arm/kernel/signal.c | 3 +-- 3 files changed, 15 insertions(+), 20 deletions(-)
arch/arm/kernel/entry-common.S +1 -12
@@ -61,12 +61,7 @@ ENTRY(ret_to_user) enable_irq_notrace @ enable interrupts mov r0, sp @ 'regs' bl syscall_exit_to_user_mode - ldr r1, [tsk, #TI_FLAGS] - movs r1, r1, lsl #16 - beq 1f - mov r0, sp @ 'regs' - bl do_work_pending -1: + #ifdef CONFIG_GCC_PLUGIN_STACKLEAK bl stackleak_erase_on_task_stack #endif
@@ -74,12 +69,6 @@ ENTRY(ret_to_user) ENDPROC(ret_to_user) ENTRY(ret_to_user_from_irq) - ldr r1, [tsk, #TI_FLAGS] - movs r1, r1, lsl #16 - beq no_work_pending - mov r0, sp @ 'regs' - bl do_work_pending -no_work_pending: asm_irqentry_exit_to_user_mode #ifdef CONFIG_GCC_PLUGIN_STACKLEAK
arch/arm/kernel/entry.c +13 -6
@@ -6,11 +6,6 @@#include <linux/irqflags.h> #include <linux/rseq.h> -static inline bool has_syscall_work (unsigned long flags) -{ -return unlikely (flags & _TIF_SYSCALL_WORK); -} -long syscall_enter_from_user_mode (struct pt_regs *regs,long syscall) {trace_hardirqs_on ();
@@ -28,7 +23,11 @@ void syscall_exit_to_user_mode(struct pt_regs *regs)rseq_syscall (regs);local_irq_disable (); -if (has_syscall_work (flags)) +/* +* It really matters that we check for flags != 0 and not +* just for pending work here! +*/ +if (flags)do_work_pending (regs, flags);trace_hardirqs_on ();
@@ -45,6 +44,14 @@ noinstr void irqentry_enter_from_user_mode(struct pt_regs *regs) noinstr void irqentry_exit_to_user_mode(struct pt_regs *regs) { +unsigned long flags =read_thread_flags (); + +/* +* It really matters that we check for flags != 0 and not +* just for pending work here! +*/ +if (flags) +do_work_pending (regs, flags);trace_hardirqs_on ();/* This context tracking call has inverse naming */ user_enter_callable ();
arch/arm/kernel/signal.c +1 -2
@@ -598,8 +598,7 @@ static void arch_do_signal_or_restart(struct pt_regs *regs)return ; } -asmlinkage void -do_work_pending (struct pt_regs *regs,unsigned int thread_flags) +void do_work_pending (struct pt_regs *regs,unsigned int thread_flags) {/* * The assembly code enters us with IRQs off, but it hasn't
Signature · 1 line
2.47.1
RFC PATCH v3 25/30 ARM: entry: Create irqentry calls from kernel mode
In reply to Linus WalleijThis adds a few calls to C when exceptions enter/exit from the kernel itself, so these can be used for generic entry.
--- arch/arm/include/asm/entry.h | 2 ++ arch/arm/kernel/entry-armv.S | 3 +++ arch/arm/kernel/entry-header.S | 5 +++++ arch/arm/kernel/entry.c | 8 ++++++++ 4 files changed, 18 insertions(+)
arch/arm/include/asm/entry.h +2 -0
@@ -12,5 +12,7 @@ long syscall_enter_from_user_mode(struct pt_regs *regs, long);void syscall_exit_to_user_mode (struct pt_regs *regs);void irqentry_enter_from_user_mode (struct pt_regs *regs);void irqentry_exit_to_user_mode (struct pt_regs *regs); +void irqentry_enter_from_kernel_mode (struct pt_regs *regs); +void irqentry_exit_to_kernel_mode (struct pt_regs *regs);#endif /* __ASM_ENTRY_H__ */
arch/arm/kernel/entry-armv.S +3 -0
@@ -205,6 +205,9 @@ ENDPROC(__und_invalid) get_thread_info tsk uaccess_entry tsk, r0, r1, r2, \uaccess + mov r0, sp @ 'regs' + bl irqentry_enter_from_kernel_mode + .if \trace #ifdef CONFIG_TRACE_IRQFLAGS bl trace_hardirqs_off
arch/arm/kernel/entry-header.S +5 -0
@@ -210,6 +210,7 @@ .else @ IRQs off again before pulling preserved data off the stack disable_irq_notrace + #ifdef CONFIG_TRACE_IRQFLAGS tst \rpsr, #PSR_I_BIT bleq trace_hardirqs_on
@@ -217,6 +218,10 @@ blne trace_hardirqs_off #endif .endif + + mov r0, sp @ 'regs' + bl irqentry_exit_to_kernel_mode + uaccess_exit tsk, r0, r1 #ifndef CONFIG_THUMB2_KERNEL
arch/arm/kernel/entry.c +8 -0
@@ -56,3 +56,11 @@ noinstr void irqentry_exit_to_user_mode(struct pt_regs *regs)/* This context tracking call has inverse naming */ user_enter_callable (); } + +noinstr void irqentry_enter_from_kernel_mode(struct pt_regs *regs) +{ +} + +noinstr void irqentry_exit_to_kernel_mode(struct pt_regs *regs) +{ +}
Signature · 1 line
2.47.1
RFC PATCH v3 26/30 ARM: entry: Move in-kernel hardirq tracing to C
In reply to Linus WalleijMove the code tracing hardirqs on/off into the C callbacks for irqentry_enter_from_kernel_mode() and irqentry_exit_to_kernel_mode().
The semantic difference occurred is that we alsways check the PSR_I_BIT to determine if (hard) interrupts were enabled or not. The assembly has a tweak to avoid this if we are exiting an IRQ since it is obvious that IRQs must have been enabled to get there, but for simplicity we just check it for all exceptions.
--- arch/arm/kernel/entry-armv.S | 13 ++++--------- arch/arm/kernel/entry-header.S | 19 ++----------------- arch/arm/kernel/entry.c | 5 +++++ 3 files changed, 11 insertions(+), 26 deletions(-)
arch/arm/kernel/entry-armv.S +4 -9
@@ -158,7 +158,7 @@ ENDPROC(__und_invalid) #define SPFIX(code...) #endif - .macro svc_entry, stack_hole=0, trace=1, uaccess=1, overflow_check=1 + .macro svc_entry, stack_hole=0 uaccess=1, overflow_check=1 UNWIND(.fnstart ) sub sp, sp, #(SVC_REGS_SIZE + \stack_hole) THUMB( add sp, r1 ) @ get SP in a GPR without
@@ -208,11 +208,6 @@ ENDPROC(__und_invalid) mov r0, sp @ 'regs' bl irqentry_enter_from_kernel_mode - .if \trace -#ifdef CONFIG_TRACE_IRQFLAGS - bl trace_hardirqs_off -#endif - .endif .endm .align 5
@@ -239,7 +234,7 @@ __irq_svc: blne svc_preempt #endif - svc_exit r5, irq = 1 @ return from exception + svc_exit r5 @ return from exception UNWIND(.fnend ) ENDPROC(__irq_svc)
@@ -303,7 +298,7 @@ ENDPROC(__pabt_svc) .align 5 __fiq_svc: - svc_entry trace=0 + svc_entry mov r0, sp @ struct pt_regs *regs bl handle_fiq_as_nmi svc_exit_via_fiq
@@ -321,7 +316,7 @@ ENDPROC(__fiq_svc) @ .align 5 __fiq_abt: - svc_entry trace=0 + svc_entry ARM( msr cpsr_c, #ABT_MODE | PSR_I_BIT | PSR_F_BIT ) THUMB( mov r0, #ABT_MODE | PSR_I_BIT | PSR_F_BIT )
arch/arm/kernel/entry-header.S +2 -17
@@ -199,26 +199,11 @@ .endm - .macro svc_exit, rpsr, irq = 0 - .if \irq != 0 - @ IRQs already off -#ifdef CONFIG_TRACE_IRQFLAGS - @ The parent context IRQs must have been enabled to get here in - @ the first place, so there's no point checking the PSR I bit. - bl trace_hardirqs_on -#endif - .else + .macro svc_exit, rpsr + @ IRQs off again before pulling preserved data off the stack disable_irq_notrace -#ifdef CONFIG_TRACE_IRQFLAGS - tst \rpsr, #PSR_I_BIT - bleq trace_hardirqs_on - tst \rpsr, #PSR_I_BIT - blne trace_hardirqs_off -#endif - .endif - mov r0, sp @ 'regs' bl irqentry_exit_to_kernel_mode
arch/arm/kernel/entry.c +5 -0
@@ -59,8 +59,13 @@ noinstr void irqentry_exit_to_user_mode(struct pt_regs *regs) noinstr void irqentry_enter_from_kernel_mode(struct pt_regs *regs) { +trace_hardirqs_off (); } noinstr void irqentry_exit_to_kernel_mode(struct pt_regs *regs) { +if (interrupts_enabled (regs)) +trace_hardirqs_on (); +else +trace_hardirqs_off (); }
Signature · 1 line
2.47.1
RFC PATCH v3 27/30 ARM: irq: Add irqstack helpers
In reply to Linus WalleijAdd two helpers that essentially re-implement some IRQ dispatch code in C: on_irq_stack() that checks if a regs context is on the IRQ stack, and call_on_irq_stack() so we can explicitly issue handle_irq() on the IRQ stack from a C program.
--- arch/arm/kernel/irq.c | 31 +++++++++++++++++++++++++++++++ arch/arm/kernel/irq.h | 3 +++ 2 files changed, 34 insertions(+)
arch/arm/kernel/irq.c +31 -0
@@ -43,6 +43,7 @@#include <asm/mach/irq.h> #include <asm/mach/time.h> +#include "irq.h" #include "reboot.h" unsigned long irq_err_count;
@@ -71,6 +72,36 @@ static void __init init_irq_stacks(void) } } +/* +* on_irq_stack() - check if a regs context is using the IRQ stack +* @regs: the context to check +* Returns true if regs is using the IRQ stack +*/ +bool on_irq_stack (struct pt_regs *regs) +{ +u8 *high =__this_cpu_read (irq_stack_ptr); +u8 *low = high -THREAD_SIZE ; +u8 *sp = (u8 *)regs->ARM_sp; +bool on_stack; + + on_stack = (low <= sp && sp <= high); + +if (on_stack &&IS_ENABLED (CONFIG_VMAP_STACK )) +/* +* Also check that SP is inside the linear kernel memory +* if using VMAP:ed stacks. +* TODO: Ask Ard to explain why we need to do this +*/ + on_stack = (sp > (u8 *)high_memory); + +return on_stack; +} + +void call_on_irq_stack (void (*fn)(void *),void *arg) +{ +call_with_stack (fn, arg,__this_cpu_read (irq_stack_ptr)); +} +#ifdef CONFIG_SOFTIRQ_ON_OWN_STACK static void ____do_softirq (void *arg) {
arch/arm/kernel/irq.h new file +3 -0
@@ -0,0 +1,3 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +bool on_irq_stack (struct pt_regs *regs); +void call_on_irq_stack (void (*fn)(void *),void *arg);
Signature · 1 line
2.47.1
RFC PATCH v3 28/30 ARM: entry: Convert to generic entry
In reply to Linus WalleijThis rewrites ARM to use the generic entry.
All of the irqentry_* callbacks are collapsed and made to call into the generic entry code that handle the interrupts storing regs context before entry and restoring it after.
The syscall_enter_from_user_mode(), syscall_exit_to_user_mode(), do_work_pending() constructs are replaced with generic code.
The syscall invocation now enables syscall tracing "work" using the generic callbacks and no extra actions are needed to enable/disable syscall tracing.
The old context tracking with user_exit_callable() user_enter_callable() is no longer used. Only CSKY and Xtensa use this deprecated mechanism now.
handle_fiq_as_nmi() is retired as the generic entry provides the same functionality in irqentry_nmi_enter() and irqentry_nmi_exit().
Like with the ARM64 proposed patch for generic entry, it is necessary to create a small call "arm_exit_to_user_mode()" that is used for things such as ret_from_fork, i.e. when we create a completely new execution context with a botched return from interrupt.
The elaborate checking and switching to IRQ stack in the irq_handler macro was reimplemented in C for this
exercise: this was easiest and there was already code making use of the IRQ stack from C for soft interrupts.
I have dropped the context tracking (i.e. calling irqentry_enter() and looking for trace conditions) on dabt/pabt i.e. do_DataAbort and do_PrefetchAbort code paths. These are addressed in a separate patch for development/review reasons.
Open questions:
- I had to add defines for PTRACE_SYSEMU and PTRACE_SYSEMU_SINGLESTEP and I have no idea what those are or why they are required. Sorry for my ignorance, I need help to figure out how we deal with this.
--- arch/arm/Kconfig | 1 + arch/arm/include/asm/entry-common.h | 66 +++++++++++++++++++++++++++ arch/arm/include/asm/entry.h | 13 ++---- arch/arm/include/asm/ptrace.h | 3 -- arch/arm/include/asm/signal.h | 2 - arch/arm/include/asm/syscall.h | 7 ++- arch/arm/include/asm/thread_info.h | 17 +------ arch/arm/include/asm/traps.h | 1 - arch/arm/include/uapi/asm/ptrace.h | 2 + arch/arm/kernel/entry-armv.S | 50 ++++---------------- arch/arm/kernel/entry-common.S | 34 +++----------- arch/arm/kernel/entry-header.S | 18 -------- arch/arm/kernel/entry.c | 91 ++++++++++++++++++------------------- arch/arm/kernel/process.c | 5 +- arch/arm/kernel/ptrace.c | 90 ------------------------------------ arch/arm/kernel/signal.c | 31 +------------ arch/arm/kernel/syscall.c | 27 +++-------- arch/arm/kernel/traps.c | 25 ---------- 18 files changed, 149 insertions(+), 334 deletions(-)
arch/arm/Kconfig +1 -0
@@ -66,6 +66,7 @@ config ARM select GENERIC_CPU_AUTOPROBE select GENERIC_CPU_DEVICES select GENERIC_EARLY_IOREMAP + select GENERIC_ENTRY select GENERIC_IDLE_POLL_SETUP select GENERIC_IRQ_MULTI_HANDLER select GENERIC_IRQ_PROBE
arch/arm/include/asm/entry-common.h new file +66 -0
66 lines not on this page · raw message
arch/arm/include/asm/entry.h +3 -10
@@ -4,15 +4,8 @@struct pt_regs ; -/* -* These are copies of generic entry headers so we can transition -* to generic entry once they are semantically equivalent. -*/ -long syscall_enter_from_user_mode (struct pt_regs *regs,long ); -void syscall_exit_to_user_mode (struct pt_regs *regs); -void irqentry_enter_from_user_mode (struct pt_regs *regs); -void irqentry_exit_to_user_mode (struct pt_regs *regs); -void irqentry_enter_from_kernel_mode (struct pt_regs *regs); -void irqentry_exit_to_kernel_mode (struct pt_regs *regs); +void arm_irq_handler (struct pt_regs *regs,int mode); +void arm_fiq_handler (struct pt_regs *regs); +void arm_exit_to_user_mode (struct pt_regs *regs);#endif /* __ASM_ENTRY_H__ */
arch/arm/include/asm/ptrace.h +0 -3
@@ -200,8 +200,5 @@ static inline unsigned long it_advance(unsigned long cpsr)return cpsr; } -int syscall_trace_enter (struct pt_regs *regs); -void syscall_trace_exit (struct pt_regs *regs); -#endif /* __ASSEMBLY__ */ #endif
arch/arm/include/asm/signal.h +0 -2
@@ -23,6 +23,4 @@ typedef struct {#include <asm/sigcontext.h> -void do_work_pending (struct pt_regs *regs,unsigned int thread_flags); -#endif
arch/arm/include/asm/syscall.h +6 -1
@@ -19,7 +19,12 @@extern const unsigned long sys_call_table[]; -int invoke_syscall (void *table,struct pt_regs *regs,int scno); +void invoke_syscall (void *table,struct pt_regs *regs,int scno); + +static inline bool arch_syscall_is_vdso_sigreturn (struct pt_regs *regs) +{ +return false; +}static inline int syscall_get_nr (struct task_struct *task,struct pt_regs *regs)
arch/arm/include/asm/thread_info.h +1 -16
@@ -74,6 +74,7 @@ struct thread_info {#ifdef CONFIG_ARM_THUMBEE unsigned long thumbee_state;/* ThumbEE Handler Base register */ #endif +unsigned long syscall_work;/* SYSCALL_WORK_ flags */ };#define INIT_THREAD_INFO (tsk) \
@@ -149,30 +150,14 @@ extern int vfp_restore_user_hwstate(struct user_vfp *,#define TIF_USING_IWMMXT 17#define TIF_MEMDIE 18/* is terminating due to OOM killer */ #define TIF_RESTORE_SIGMASK 19 -#define TIF_SYSCALL_TRACE 20/* syscall trace active */ -#define TIF_SYSCALL_AUDIT 21/* syscall auditing active */ -#define TIF_SYSCALL_TRACEPOINT 22/* syscall tracepoint instrumentation */ -#define TIF_SECCOMP 23/* seccomp syscall filtering active */ -#define _TIF_SIGPENDING (1 << TIF_SIGPENDING)#define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED)#define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME)#define _TIF_UPROBE (1 << TIF_UPROBE) -#define _TIF_SYSCALL_TRACE (1 << TIF_SYSCALL_TRACE) -#define _TIF_SYSCALL_AUDIT (1 << TIF_SYSCALL_AUDIT) -#define _TIF_SYSCALL_TRACEPOINT (1 << TIF_SYSCALL_TRACEPOINT) -#define _TIF_SECCOMP (1 << TIF_SECCOMP)#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL)#define _TIF_USING_IWMMXT (1 << TIF_USING_IWMMXT) -/* Checks for any syscall work in entry-common.S */ -#define _TIF_SYSCALL_WORK (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \ - _TIF_SYSCALL_TRACEPOINT | _TIF_SECCOMP) - -/* -* Change these and you break ASM code in entry-common.S -*/ #define _TIF_WORK_MASK (_TIF_NEED_RESCHED | _TIF_SIGPENDING | \ _TIF_NOTIFY_RESUME | _TIF_UPROBE | \ _TIF_NOTIFY_SIGNAL)
arch/arm/include/asm/traps.h +0 -1
@@ -38,7 +38,6 @@ extern void *vectors_page; asmlinkage void dump_backtrace_stm(u32 *stack, u32 instruction, const char *loglvl); asmlinkage void do_undefinstr(struct pt_regs *regs); -asmlinkage void handle_fiq_as_nmi(struct pt_regs *regs); asmlinkage void bad_mode(struct pt_regs *regs, int reason);int arm_syscall (int no,struct pt_regs *regs); asmlinkage void baddataabort(int code, unsigned long instr, struct pt_regs *regs);
arch/arm/include/uapi/asm/ptrace.h +2 -0
@@ -33,6 +33,8 @@#define PTRACE_GETHBPREGS 29#define PTRACE_SETHBPREGS 30#define PTRACE_GETFDPIC 31 +#define PTRACE_SYSEMU 32 +#define PTRACE_SYSEMU_SINGLESTEP 33#define PTRACE_GETFDPIC_EXEC 0#define PTRACE_GETFDPIC_INTERP 1
arch/arm/kernel/entry-armv.S +9 -41
111 lines not on this page · raw message
arch/arm/kernel/entry-common.S +6 -28
@@ -39,10 +39,6 @@ ret_fast_syscall: UNWIND(.fnstart ) UNWIND(.cantunwind ) add sp, sp, #(S_R0 + S_OFF) - /* do_rseq_syscall needs interrupts enabled. */ - mov r0, sp @ 'regs' - bl syscall_exit_to_user_mode - #ifdef CONFIG_GCC_PLUGIN_STACKLEAK bl stackleak_erase_on_task_stack #endif
@@ -60,7 +56,7 @@ ENTRY(ret_to_user) /* do_rseq_syscall needs interrupts enabled. */ enable_irq_notrace @ enable interrupts mov r0, sp @ 'regs' - bl syscall_exit_to_user_mode + bl arm_exit_to_user_mode #ifdef CONFIG_GCC_PLUGIN_STACKLEAK bl stackleak_erase_on_task_stack
@@ -69,8 +65,6 @@ ENTRY(ret_to_user) ENDPROC(ret_to_user) ENTRY(ret_to_user_from_irq) - asm_irqentry_exit_to_user_mode - #ifdef CONFIG_GCC_PLUGIN_STACKLEAK bl stackleak_erase_on_task_stack #endif
@@ -92,9 +86,10 @@ SYM_TYPED_FUNC_START(ret_from_fork_asm) mov r2, r5 mov r3, r4 bl ret_from_fork - mov r0, #0 -1: get_thread_info tsk - b ret_to_user +#ifdef CONFIG_GCC_PLUGIN_STACKLEAK + bl stackleak_erase_on_task_stack +#endif + restore_user_regs SYM_FUNC_END(ret_from_fork_asm) /*=============================================================================
@@ -210,29 +205,12 @@ ENTRY(vector_swi) #else str scno, [tsk, #TI_ABI_SYSCALL] #endif - - /* - * Calling out to C to be careful to save and restore registers. - * This call could modify the syscall number. scno is r7 so we - * do not save and restore r7. - */ - mov r0, sp @ regs - mov r1, scno - push {r4 - r6, r8 - r10, lr} - bl syscall_enter_from_user_mode - pop {r4 - r6, r8 - r10, lr} - mov scno, r0 - mov r1, sp @ put regs into r1 stmdb sp!, {r4, r5} @ push fifth and sixth args mov r0, tbl mov r2, scno @ syscall number from r7 bl invoke_syscall - cmp r0, #0 - beq ret_fast_syscall - /* This path taken when tracing */ - add sp, sp, #(S_R0 + S_OFF) - b ret_to_user + b ret_fast_syscall #if defined(CONFIG_OABI_COMPAT) || !defined(CONFIG_AEABI) /*
arch/arm/kernel/entry-header.S +0 -18
@@ -203,10 +203,6 @@ @ IRQs off again before pulling preserved data off the stack disable_irq_notrace - - mov r0, sp @ 'regs' - bl irqentry_exit_to_kernel_mode - uaccess_exit tsk, r0, r1 #ifndef CONFIG_THUMB2_KERNEL
@@ -340,20 +336,6 @@ ALT_UP_B(.L1_\@) #endif /* !CONFIG_THUMB2_KERNEL */ .endm -/* - * Context tracking and other mode transitions. Used to instrument transitions - * between user and kernel mode. -*/ - .macro asm_irqentry_enter_from_user_mode - mov r0, sp @ regs - bl irqentry_enter_from_user_mode - .endm - - .macro asm_irqentry_exit_to_user_mode - mov r0, sp @ regs - bl irqentry_exit_to_user_mode - .endm - /* * These are the registers used in the syscall handler, and allow us to * have in theory up to 7 arguments to a function - r0 to r6.
arch/arm/kernel/entry.c +44 -47
115 lines not on this page · raw message
arch/arm/kernel/process.c +3 -2
@@ -25,6 +25,7 @@#include <linux/random.h> #include <linux/hw_breakpoint.h> #include <linux/leds.h> +#include <linux/entry-common.h> #include <asm/processor.h> #include <asm/thread_notify.h>
@@ -248,8 +249,8 @@ __visible void ret_from_fork(struct task_struct *prev, struct pt_regs *regs,* execve() syscall. */ } - -/* syscall_exit_to_user_mode(regs); here once we switch to generic entry */ +local_irq_enable (); +syscall_exit_to_user_mode (regs); }int copy_thread (struct task_struct *p,const struct kernel_clone_args *args)
arch/arm/kernel/ptrace.c +0 -90
99 lines not on this page · raw message
arch/arm/kernel/signal.c +2 -29
@@ -4,6 +4,7 @@* * Copyright (C) 1995-2009 Russell King */ +#include <linux/entry-common.h> #include <linux/errno.h> #include <linux/random.h> #include <linux/signal.h>
@@ -535,7 +536,7 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs)* the kernel can handle, and then we build all the user-level signal handling * stack-frames in one go after that. */ -static void arch_do_signal_or_restart (struct pt_regs *regs) +void arch_do_signal_or_restart (struct pt_regs *regs) {unsigned int retval =0 , continue_addr =0 , restart_addr =0 ;bool syscall = (syscall_get_nr (current, regs) !=-1 );
@@ -598,34 +599,6 @@ static void arch_do_signal_or_restart(struct pt_regs *regs)return ; } -void do_work_pending (struct pt_regs *regs,unsigned int thread_flags) -{ -/* -* The assembly code enters us with IRQs off, but it hasn't -* informed the tracing code of that for efficiency reasons. -* Update the trace code with the current status. -*/ -trace_hardirqs_off (); -do { -if (likely (thread_flags & _TIF_NEED_RESCHED)) { -schedule (); - }else { -if (unlikely (!user_mode (regs))) -return ; -local_irq_enable (); -if (thread_flags & (_TIF_SIGPENDING | _TIF_NOTIFY_SIGNAL)) { -arch_do_signal_or_restart (regs); - }else if (thread_flags & _TIF_UPROBE) { -uprobe_notify_resume (regs); - }else { -resume_user_mode_work (regs); - } - } -local_irq_disable (); - thread_flags =read_thread_flags (); - }while (thread_flags & _TIF_WORK_MASK); -} -struct page *get_signal_page (void ) {unsigned long ptr;
arch/arm/kernel/syscall.c +6 -21
@@ -1,26 +1,17 @@// SPDX-License-Identifier: GPL-2.0 +#include <linux/entry-common.h> #include <linux/syscalls.h> #include <asm/syscall.h> -static inline bool has_syscall_work (unsigned long flags) -{ -return unlikely (flags & _TIF_SYSCALL_WORK); -} -int invoke_syscall_asm (void *table,struct pt_regs *regs,int scno);__ADDRESSABLE (invoke_syscall_asm); -__visible int invoke_syscall(void *table, struct pt_regs *regs, int scno) +__visible void invoke_syscall(void *table, struct pt_regs *regs, int scno) { -unsigned long flags =read_thread_flags ();int ret; -if (has_syscall_work (flags)) { - scno =syscall_trace_enter (regs); -if (scno ==-1 ) - goto trace_exit_nosave; - } + scno =syscall_enter_from_user_mode (regs, scno);if (scno < NR_syscalls) { ret =invoke_syscall_asm (table, regs, scno);
@@ -35,13 +26,7 @@ __visible int invoke_syscall(void *table, struct pt_regs *regs, int scno) ret =sys_ni_syscall (); exit_save: -/* Save return value from syscall */ - regs->ARM_r0 = ret; -if (!has_syscall_work (flags)) -return 0 ; - -trace_exit_nosave: -local_irq_enable (); -syscall_trace_exit (regs); -return 1 ; +syscall_set_return_value (current, regs,0 , ret); + +syscall_exit_to_user_mode (regs); }
arch/arm/kernel/traps.c +0 -25
@@ -502,31 +502,6 @@ asmlinkage void do_undefinstr(struct pt_regs *regs) }NOKPROBE_SYMBOL (do_undefinstr) -/* -* Handle FIQ similarly to NMI on x86 systems. -* -* The runtime environment for NMIs is extremely restrictive -* (NMIs can pre-empt critical sections meaning almost all locking is -* forbidden) meaning this default FIQ handling must only be used in -* circumstances where non-maskability improves robustness, such as -* watchdog or debug logic. -* -* This handler is not appropriate for general purpose use in drivers -* platform code and can be overrideen using set_fiq_handler. -*/ -asmlinkage void __exception_irq_entry handle_fiq_as_nmi(struct pt_regs *regs) -{ -struct pt_regs *old_regs =set_irq_regs (regs); - -nmi_enter (); - -/* nop. FIQ handlers for special arch/arm features can be added here. */ - -nmi_exit (); - -set_irq_regs (old_regs); -} -/* * bad_mode handles the impossible case in the vectors. If you see one of * these, then it's extremely serious, and could mean you have buggy hardware.
Signature · 1 line
2.47.1
RFC PATCH v3 29/30 ARM: entry: Handle prefetch and data aborts as interrupts
In reply to Linus WalleijWhile it isn't entirely intuitive, it appears that any kind of exception such as data or prefetch abort ("page faults") need to be handled as some kind of "interrupts" when using generic entry. At least this is what other platforms are doing.
The same goes for undefined instruction handling, i.e. floating point emulation in the kernel (und-exceptions).
This is necessary for the context checking to pass: without this patch, a whole slew of warnings start to trigger from syscall_exit_to_user_mode_prepare() CT_WARN_ON(ct_state() != CT_STATE_KERNEL), i.e. syscalls seems to exit from user mode to user mode (not good), because the page faults screws up the context tracker.
This patch restores the order.
If this seems like the previous patch introduces a regression that is then fixed in this patch, it can simply be squashed into the former: having this rewrite separately surely makes development and review easier.
--- arch/arm/include/asm/entry.h | 3 +++ arch/arm/include/asm/traps.h | 2 +- arch/arm/kernel/entry-armv.S | 7 +++---- arch/arm/kernel/entry.c | 29 +++++++++++++++++++++++++++++ arch/arm/kernel/traps.c | 2 +- arch/arm/mm/abort-ev4.S | 2 +- arch/arm/mm/abort-ev4t.S | 2 +- arch/arm/mm/abort-ev5t.S | 4 ++-- arch/arm/mm/abort-ev5tj.S | 6 +++--- arch/arm/mm/abort-ev6.S | 2 +- arch/arm/mm/abort-ev7.S | 2 +- arch/arm/mm/abort-lv4t.S | 36 ++++++++++++++++++------------------ arch/arm/mm/abort-macro.S | 2 +- arch/arm/mm/abort-nommu.S | 2 +- arch/arm/mm/fault.c | 4 ++-- arch/arm/mm/fault.h | 8 ++++---- arch/arm/mm/pabort-legacy.S | 2 +- arch/arm/mm/pabort-v6.S | 2 +- arch/arm/mm/pabort-v7.S | 2 +- 19 files changed, 75 insertions(+), 44 deletions(-)
arch/arm/include/asm/entry.h +3 -0
@@ -4,6 +4,9 @@struct pt_regs ; +void arm_und_handler (struct pt_regs *regs); +void arm_dabt_handler (unsigned long addr,unsigned int fsr,struct pt_regs *regs); +void arm_pabt_handler (unsigned long addr,unsigned int ifsr,struct pt_regs *regs);void arm_irq_handler (struct pt_regs *regs,int mode);void arm_fiq_handler (struct pt_regs *regs);void arm_exit_to_user_mode (struct pt_regs *regs);
arch/arm/include/asm/traps.h +1 -1
@@ -37,7 +37,7 @@ extern void ptrace_break(struct pt_regs *regs);extern void *vectors_page; asmlinkage void dump_backtrace_stm(u32 *stack, u32 instruction, const char *loglvl); -asmlinkage void do_undefinstr(struct pt_regs *regs); +void do_undefinstr (struct pt_regs *regs); asmlinkage void bad_mode(struct pt_regs *regs, int reason);int arm_syscall (int no,struct pt_regs *regs); asmlinkage void baddataabort(int code, unsigned long instr, struct pt_regs *regs);
arch/arm/kernel/entry-armv.S +3 -4
@@ -230,7 +230,7 @@ __und_fault: ldr r2, [r0, #S_PC] sub r2, r2, r1 str r2, [r0, #S_PC] - b do_undefinstr + b arm_und_handler ENDPROC(__und_fault) .align 5
@@ -449,9 +449,8 @@ __pabt_usr: ENTRY(ret_from_exception) UNWIND(.fnstart ) UNWIND(.cantunwind ) - get_thread_info tsk - mov why, #0 - b ret_to_user + disable_irq_notrace + b ret_to_user_from_irq UNWIND(.fnend ) ENDPROC(__pabt_usr) ENDPROC(ret_from_exception)
arch/arm/kernel/entry.c +29 -0
@@ -7,8 +7,37 @@#include <linux/irqflags.h> #include <linux/percpu.h> #include <linux/rseq.h> +#include <asm/traps.h> #include "irq.h" +#include "../mm/fault.h" + +noinstr asmlinkage void arm_und_handler(struct pt_regs *regs) +{ +irqentry_state_t state =irqentry_enter (regs); + +do_undefinstr (regs); + +irqentry_exit (regs, state); +} + +noinstr asmlinkage void arm_dabt_handler(unsigned long addr, unsigned int fsr, struct pt_regs *regs) +{ +irqentry_state_t state =irqentry_enter (regs); + +do_DataAbort (addr, fsr, regs); + +irqentry_exit (regs, state); +} + +noinstr asmlinkage void arm_pabt_handler(unsigned long addr, unsigned int ifsr, struct pt_regs *regs) +{ +irqentry_state_t state =irqentry_enter (regs); + +do_PrefetchAbort (addr, ifsr, regs); + +irqentry_exit (regs, state); +} static void noinstr handle_arm_irq(void *data) {
arch/arm/kernel/traps.c +1 -1
@@ -449,7 +449,7 @@ int call_undef_hook(struct pt_regs *regs, unsigned int instr)return fn ?fn (regs, instr) :1 ; } -asmlinkage void do_undefinstr(struct pt_regs *regs) +noinstr void do_undefinstr(struct pt_regs *regs) {unsigned int instr;void __user *pc;
arch/arm/mm/abort-ev4.S +1 -1
@@ -24,4 +24,4 @@ ENTRY(v4_early_abort) bic r1, r1, #1 << 11 | 1 << 10 @ clear bits 11 and 10 of FSR tst r3, #1 << 20 @ L = 1 -> write? orreq r1, r1, #1 << 11 @ yes. - b do_DataAbort + b arm_dabt_handler
arch/arm/mm/abort-ev4t.S +1 -1
@@ -25,4 +25,4 @@ ENTRY(v4t_early_abort) bic r1, r1, #1 << 11 | 1 << 10 @ clear bits 11 and 10 of FSR tst r3, #1 << 20 @ check write orreq r1, r1, #1 << 11 - b do_DataAbort + b arm_dabt_handler
arch/arm/mm/abort-ev5t.S +2 -2
@@ -25,7 +25,7 @@ ENTRY(v5t_early_abort) uaccess_disable ip @ disable user access bic r1, r1, #1 << 11 @ clear bits 11 of FSR teq_ldrd tmp=ip, insn=r3 @ insn was LDRD? - beq do_DataAbort @ yes + beq arm_dabt_handler @ yes tst r3, #1 << 20 @ check write orreq r1, r1, #1 << 11 - b do_DataAbort + b arm_dabt_handler
arch/arm/mm/abort-ev5tj.S +3 -3
@@ -22,12 +22,12 @@ ENTRY(v5tj_early_abort) mrc p15, 0, r0, c6, c0, 0 @ get FAR bic r1, r1, #1 << 11 | 1 << 10 @ clear bits 11 and 10 of FSR tst r5, #PSR_J_BIT @ Java? - bne do_DataAbort + bne arm_dabt_handler do_thumb_abort fsr=r1, pc=r4, psr=r5, tmp=r3 ldreq r3, [r4] @ read aborted ARM instruction uaccess_disable ip @ disable userspace access teq_ldrd tmp=ip, insn=r3 @ insn was LDRD? - beq do_DataAbort @ yes + beq arm_dabt_handler @ yes tst r3, #1 << 20 @ L = 0 -> write orreq r1, r1, #1 << 11 @ yes. - b do_DataAbort + b arm_dabt_handler
arch/arm/mm/abort-ev6.S +1 -1
@@ -42,4 +42,4 @@ ENTRY(v6_early_abort) orreq r1, r1, #1 << 11 @ yes. #endif 1: uaccess_disable ip @ disable userspace access - b do_DataAbort + b arm_dabt_handler
arch/arm/mm/abort-ev7.S +1 -1
@@ -18,5 +18,5 @@ ENTRY(v7_early_abort) mrc p15, 0, r1, c5, c0, 0 @ get FSR mrc p15, 0, r0, c6, c0, 0 @ get FAR uaccess_disable ip @ disable userspace access - b do_DataAbort + b arm_dabt_handler ENDPROC(v7_early_abort)
arch/arm/mm/abort-lv4t.S +18 -18
@@ -46,8 +46,8 @@ ENTRY(v4t_late_abort) /* 9 */ b .data_arm_ldmstm @ ldm*b rn, <rlist> /* a */ b .data_unknown /* b */ b .data_unknown -/* c */ b do_DataAbort @ ldc rd, [rn], #m @ Same as ldr rd, [rn], #m -/* d */ b do_DataAbort @ ldc rd, [rn, #m] +/* c */ b arm_dabt_handler @ ldc rd, [rn], #m @ Same as ldr rd, [rn], #m +/* d */ b arm_dabt_handler @ ldc rd, [rn, #m] /* e */ b .data_unknown /* f */ b .data_unknown
@@ -60,7 +60,7 @@ ENTRY(v4t_late_abort) .data_arm_ldmstm: tst r8, #1 << 21 @ check writeback bit - beq do_DataAbort @ no writeback -> no fixup + beq arm_dabt_handler @ no writeback -> no fixup str r9, [sp, #-4]! mov r7, #0x11 orr r7, r7, #0x1100
@@ -81,11 +81,11 @@ ENTRY(v4t_late_abort) addeq r7, r7, r6, lsl #2 @ Undo decrement str r7, [r2, r9, lsr #14] @ Put register 'Rn' ldr r9, [sp], #4 - b do_DataAbort + b arm_dabt_handler .data_arm_lateldrhpre: tst r8, #1 << 21 @ Check writeback bit - beq do_DataAbort @ No writeback -> no fixup + beq arm_dabt_handler @ No writeback -> no fixup .data_arm_lateldrhpost: str r9, [sp, #-4]! and r9, r8, #0x00f @ get Rm / low nibble of immediate value
@@ -101,14 +101,14 @@ ENTRY(v4t_late_abort) addeq r7, r7, r6 @ Undo decrement str r7, [r2, r9, lsr #14] @ Put register 'Rn' ldr r9, [sp], #4 - b do_DataAbort + b arm_dabt_handler .data_arm_lateldrpreconst: tst r8, #1 << 21 @ check writeback bit - beq do_DataAbort @ no writeback -> no fixup + beq arm_dabt_handler @ no writeback -> no fixup .data_arm_lateldrpostconst: movs r6, r8, lsl #20 @ Get offset - beq do_DataAbort @ zero -> no fixup + beq arm_dabt_handler @ zero -> no fixup str r9, [sp, #-4]! and r9, r8, #15 << 16 @ Extract 'n' from instruction ldr r7, [r2, r9, lsr #14] @ Get register 'Rn'
@@ -117,11 +117,11 @@ ENTRY(v4t_late_abort) addeq r7, r7, r6, lsr #20 @ Undo decrement str r7, [r2, r9, lsr #14] @ Put register 'Rn' ldr r9, [sp], #4 - b do_DataAbort + b arm_dabt_handler .data_arm_lateldrprereg: tst r8, #1 << 21 @ check writeback bit - beq do_DataAbort @ no writeback -> no fixup + beq arm_dabt_handler @ no writeback -> no fixup .data_arm_lateldrpostreg: and r7, r8, #15 @ Extract 'm' from instruction ldr r6, [r2, r7, lsl #2] @ Get register 'Rm'
@@ -180,10 +180,10 @@ ENTRY(v4t_late_abort) /* 3 */ b .data_unknown /* 4 */ b .data_unknown /* 5 */ b .data_thumb_reg -/* 6 */ b do_DataAbort -/* 7 */ b do_DataAbort -/* 8 */ b do_DataAbort -/* 9 */ b do_DataAbort +/* 6 */ b arm_dabt_handler +/* 7 */ b arm_dabt_handler +/* 8 */ b arm_dabt_handler +/* 9 */ b arm_dabt_handler /* A */ b .data_unknown /* B */ b .data_thumb_pushpop /* C */ b .data_thumb_ldmstm
@@ -193,10 +193,10 @@ ENTRY(v4t_late_abort) .data_thumb_reg: tst r8, #1 << 9 - beq do_DataAbort + beq arm_dabt_handler tst r8, #1 << 10 @ If 'S' (signed) bit is set movne r1, #0 @ it must be a load instr - b do_DataAbort + b arm_dabt_handler .data_thumb_pushpop: tst r8, #1 << 10
@@ -217,7 +217,7 @@ ENTRY(v4t_late_abort) subne r7, r7, r6, lsl #2 @ decrement SP if POP str r7, [r2, #13 << 2] ldr r9, [sp], #4 - b do_DataAbort + b arm_dabt_handler .data_thumb_ldmstm: str r9, [sp, #-4]!
@@ -234,4 +234,4 @@ ENTRY(v4t_late_abort) sub r7, r7, r6, lsl #2 @ always decrement str r7, [r2, r9, lsr #6] ldr r9, [sp], #4 - b do_DataAbort + b arm_dabt_handler
arch/arm/mm/abort-macro.S +1 -1
@@ -20,7 +20,7 @@ orreq \tmp, \tmp, #1 << 11 @ Set L-bit if yes tst \tmp, #1 << 11 @ L = 0 -> write orreq \fsr, \fsr, #1 << 11 @ yes. - b do_DataAbort + b arm_dabt_handler not_thumb: .endm
arch/arm/mm/abort-nommu.S +1 -1
@@ -17,5 +17,5 @@ ENTRY(nommu_early_abort) mov r0, #0 @ clear r0, r1 (no FSR/FAR) mov r1, #0 - b do_DataAbort + b arm_dabt_handler ENDPROC(nommu_early_abort)
arch/arm/mm/fault.c +2 -2
@@ -588,7 +588,7 @@ hook_fault_code(int nr, int (*fn)(unsigned long, unsigned int, struct pt_regs *)/* * Dispatch a data abort to the relevant handler. */ -asmlinkage void +void do_DataAbort (unsigned long addr,unsigned int fsr,struct pt_regs *regs) {const struct fsr_info *inf = fsr_info +fsr_fs (fsr);
@@ -618,7 +618,7 @@ hook_ifault_code(int nr, int (*fn)(unsigned long, unsigned int, struct pt_regs * ifsr_info[nr].name = name; } -asmlinkage void +void do_PrefetchAbort (unsigned long addr,unsigned int ifsr,struct pt_regs *regs) {const struct fsr_info *inf = ifsr_info +fsr_fs (ifsr);
arch/arm/mm/fault.h +4 -4
@@ -37,9 +37,9 @@ static inline int fsr_fs(unsigned int fsr)void do_bad_area (unsigned long addr,unsigned int fsr,struct pt_regs *regs);void early_abt_enable (void ); -asmlinkage void do_DataAbort(unsigned long addr, unsigned int fsr, -struct pt_regs *regs); -asmlinkage void do_PrefetchAbort(unsigned long addr, unsigned int ifsr, -struct pt_regs *regs); +void do_DataAbort (unsigned long addr,unsigned int fsr, +struct pt_regs *regs); +void do_PrefetchAbort (unsigned long addr,unsigned int ifsr, +struct pt_regs *regs);#endif /* __ARCH_ARM_FAULT_H */
arch/arm/mm/pabort-legacy.S +1 -1
@@ -18,5 +18,5 @@ ENTRY(legacy_pabort) mov r0, r4 mov r1, #5 - b do_PrefetchAbort + b arm_pabt_handler ENDPROC(legacy_pabort)
arch/arm/mm/pabort-v6.S +1 -1
@@ -18,5 +18,5 @@ ENTRY(v6_pabort) mov r0, r4 mrc p15, 0, r1, c5, c0, 1 @ get IFSR - b do_PrefetchAbort + b arm_pabt_handler ENDPROC(v6_pabort)
arch/arm/mm/pabort-v7.S +1 -1
@@ -18,5 +18,5 @@ ENTRY(v7_pabort) mrc p15, 0, r0, c6, c0, 2 @ get IFAR mrc p15, 0, r1, c5, c0, 1 @ get IFSR - b do_PrefetchAbort + b arm_pabt_handler ENDPROC(v7_pabort)
Signature · 1 line
2.47.1
RFC PATCH v3 30/30 RFC: ARM: entry: Block IRQs in early IRQ context
In reply to Linus WalleijWhen dabt, pabt or und exceptions occur on ARM, ordinary interrupts (IRQs) can still happen. This isn't nice for the kernels context tracker, which expect (when using generic entry at least) that any nested IRQs happens between irqentry_enter() and irqentry_exit(), else it thinks something is fishy.
This workaround blocks interrupts in the pabt, dabt and und exception paths until after context has been established with irqentry_enter() and before it is exited with irqentry_exit().
This seems like far from perfect: it seems an interrupt could still occur right before the first disable_irq_notrace or right after the userspace registers are restored in ret_from_exception. I would like to know if there is some way to set up these exceptions to inherently block IRQs when handled, until we explicitly allow them between irqentry_enter() and irqentry_exit() or if this is simply the best we can do on ARM for these exceptions to make the context tracker happy.
--- arch/arm/kernel/entry-armv.S | 7 ++----- arch/arm/kernel/entry.c | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+), 5 deletions(-)
arch/arm/kernel/entry-armv.S +2 -5
@@ -389,6 +389,7 @@ ENDPROC(__fiq_abt) .align 5 __dabt_usr: + disable_irq_notrace usr_entry uaccess=0 kuser_cmpxchg_check mov r2, sp
@@ -416,11 +417,6 @@ ENDPROC(__irq_usr) __und_usr: usr_entry uaccess=0 - @ IRQs must be enabled before attempting to read the instruction from - @ user space since that could cause a page/translation fault if the - @ page table was modified by another CPU. - enable_irq - tst r5, #PSR_T_BIT @ Thumb mode? mov r1, #2 @ set insn size to 2 for Thumb bne 0f @ handle as Thumb undef exception
@@ -438,6 +434,7 @@ ENDPROC(__und_usr) .align 5 __pabt_usr: + disable_irq_notrace usr_entry mov r2, sp @ regs pabt_helper
arch/arm/kernel/entry.c +18 -0
@@ -16,8 +16,18 @@ noinstr asmlinkage void arm_und_handler(struct pt_regs *regs) {irqentry_state_t state =irqentry_enter (regs); +/* +* IRQs must be enabled before attempting to read the instruction from +* user space since that could cause a page/translation fault if the +* page table was modified by another CPU. +*/ + +local_irq_enable (); +do_undefinstr (regs); +local_irq_disable (); +irqentry_exit (regs, state); }
@@ -25,8 +35,12 @@ noinstr asmlinkage void arm_dabt_handler(unsigned long addr, unsigned int fsr, s {irqentry_state_t state =irqentry_enter (regs); +local_irq_enable (); +do_DataAbort (addr, fsr, regs); +local_irq_disable (); +irqentry_exit (regs, state); }
@@ -34,8 +48,12 @@ noinstr asmlinkage void arm_pabt_handler(unsigned long addr, unsigned int ifsr, {irqentry_state_t state =irqentry_enter (regs); +local_irq_enable (); +do_PrefetchAbort (addr, ifsr, regs); +local_irq_disable (); +irqentry_exit (regs, state); }
Signature · 1 line
2.47.1