objtool: UACCESS validation v4
Subsystems (14)
Peter Zijlstra
maintainercover objtool: UACCESS validation v4
Teach objtool to validate the UACCESS (SMAP, PAN) rules which are currently unenforced and (therefore obviously) violated.
UACCESS sections should be small; we want to limit the amount of code that can touch userspace. Furthermore, UACCESS state isn't scheduled, this means that anything that directly calls into the scheduler will result in random code running with UACCESS enabled and possibly getting back into the UACCESS region with UACCESS disabled and causing faults.
Forbid any CALL/RET while UACCESS is enabled; but provide a few exceptions.
This builds x86_64-allmodconfig and lots of x86_64-randconfig clean.
Changes since -v3:
- removed a bunch of functions from the UACCESS-safe list due to the removal of CONFIG_KASAN_EXTRA=y.
- hopefully addressed all the feedback from Josh
- realized objtool doesn't cover x86_32
- some added additional annotations/fixes: kcov, signal
- retains the DF check for now, Linus, do you (still) think it is worth doing that DF check?
--- arch/x86/Kconfig | 2 + arch/x86/ia32/ia32_signal.c | 29 ++- arch/x86/include/asm/alternative-asm.h | 11 + arch/x86/include/asm/alternative.h | 10 + arch/x86/include/asm/asm.h | 24 -- arch/x86/include/asm/nospec-branch.h | 28 +- arch/x86/include/asm/smap.h | 37 ++- arch/x86/include/asm/uaccess.h | 5 +- arch/x86/include/asm/uaccess_64.h | 3 - arch/x86/include/asm/xen/hypercall.h | 24 +- arch/x86/kernel/signal.c | 29 ++- arch/x86/lib/copy_user_64.S | 48 ++++ arch/x86/lib/memcpy_64.S | 3 +- arch/x86/lib/usercopy_64.c | 20 -- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 6 +- include/linux/compiler.h | 2 +- include/linux/uaccess.h | 2 + kernel/Makefile | 1 + kernel/trace/trace_branch.c | 4 + lib/Makefile | 1 + lib/ubsan.c | 4 + mm/kasan/Makefile | 3 + mm/kasan/common.c | 10 + mm/kasan/report.c | 3 +- scripts/Makefile.build | 3 + tools/objtool/arch.h | 8 +- tools/objtool/arch/x86/decode.c | 21 +- tools/objtool/builtin-check.c | 4 +- tools/objtool/builtin.h | 2 +- tools/objtool/check.c | 400 ++++++++++++++++++++++------- tools/objtool/check.h | 4 +- tools/objtool/elf.c | 15 +- tools/objtool/elf.h | 3 +- tools/objtool/special.c | 18 ++ tools/objtool/special.h | 1 + tools/objtool/warn.h | 8 + 36 files changed, 584 insertions(+), 212 deletions(-)
Peter Zijlstra
maintainerPATCH 1/25 x86: Make SMAP 64-bit only
We rely on objtool to verify AC=1 doesn't escape. However there is no objtool support for x86_32, and thus we cannot guarantee the correctness of the 32bit code.
Also; if you're running 32bit kernels on hardware with SMAP (which all should have LM support afaik) you're doing it wrong anyway.
XXX: we could do the PUSHF/POPF thing in __switch_to_asm() on x86_32, but why bother.
--- arch/x86/Kconfig | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/Kconfig +2 -0
@@ -1853,6 +1853,8 @@ config ARCH_RANDOM config X86_SMAP def_bool y + # Note: we rely on objtool to validate AC=1 doesn't escape + depends on HAVE_STACK_VALIDATION prompt "Supervisor Mode Access Prevention" if EXPERT ---help--- Supervisor Mode Access Prevention (SMAP) is a security
Re: [PATCH 01/25] x86: Make SMAP 64-bit only
Quoted 4 lines · Peter Zijlstra
We rely on objtool to verify AC=1 doesn't escape. However there is no objtool support for
x86_32, and thus we cannot guarantee the correctness of the 32bit code.
Absolutely not.
This is just crazy. We had working SMAP long before objtool, and we will have it regardless of objtool.
This is like saying "ok, I don't have an oxygen sensor, so I don't know that the air I'm breathing is sufficient to maintain life, so I'll just stop breathing".
So no way in hell do we make SMAP go away on 32-bit for no sane reason what-so-ever.
Besides, the x86-64 objtool coverage will cover 99% of all 32-bit code too, simply because we share it. In fact, it will cover most of the code for other architectures too.
Linus
Peter Zijlstra
maintainerRe: [PATCH 01/25] x86: Make SMAP 64-bit only
Quoted 10 lines · Linus Torvalds
Quoted 4 lines · Peter Zijlstra
We rely on objtool to verify AC=1 doesn't escape. However there is no objtool support for
x86_32, and thus we cannot guarantee the correctness of the 32bit code.Absolutely not.
This is just crazy. We had working SMAP long before objtool, and we will have it regardless of objtool.
Well, 'working', because as shown with this work, there's been a number of bugs around this.
Anyway, since you mentioned it, I checked, about 16 months: Broadwell was Oct'14, objtool was Feb'16.
Quoted 3 lines
This is like saying "ok, I don't have an oxygen sensor, so I don't know that the air I'm breathing is sufficient to maintain life, so I'll just stop breathing".
With PTI, booting a 32bit kernel on a Broadwell or later will already complain about it being sub-optimal for missing out on PCID. But sure; if you want to enable this..
Quoted 2 lines
So no way in hell do we make SMAP go away on 32-bit for no sane reason what-so-ever.
then I'd sleep much better if we make 32bit context switch EFLAGS. Because, yes, x86_64 objtool validates a lot of the x86_32 code too, but unless we have 100% coverage, there's always the chance an AC=1 escapes.
--- arch/x86/entry/entry_32.S | 2 ++ arch/x86/include/asm/switch_to.h | 1 + arch/x86/kernel/process_32.c | 5 +++++ 3 files changed, 8 insertions(+)
arch/x86/entry/entry_32.S +2 -0
@@ -650,6 +650,7 @@ ENTRY(__switch_to_asm) pushl %ebx pushl %edi pushl %esi + pushfl /* switch stack */ movl %esp, TASK_threadsp(%eax)
@@ -672,6 +673,7 @@ ENTRY(__switch_to_asm) #endif /* restore callee-saved registers */ + popfl popl %esi popl %edi popl %ebx
arch/x86/include/asm/switch_to.h +1 -0
@@ -46,6 +46,7 @@ struct inactive_task_frame {unsigned long r13;unsigned long r12;#else +unsigned long flags;unsigned long si;unsigned long di;#endif
arch/x86/kernel/process_32.c +5 -0
@@ -127,6 +127,11 @@ int copy_thread_tls(unsigned long clone_flags, unsigned long sp,struct task_struct *tsk;int err; +/* +* We start a new task with the RESET value of EFLAGS, in particular we +* context switch with IRQs disabled. +*/ + frame->flags =X86_EFLAGS_FIXED ; frame->bp =0 ; frame->ret_addr = (unsigned long ) ret_from_fork; p->thread.sp = (unsigned long ) fork_frame;
Peter Zijlstra
maintainerRe: [PATCH 01/25] x86: Make SMAP 64-bit only
Quoted 3 lines · Peter Zijlstra
then I'd sleep much better if we make 32bit context switch EFLAGS. Because, yes,
x86_64objtool validates a lot of thex86_32code too, but unless we have 100% coverage, there's always the chance an AC=1 escapes.
How about I do a patch that schedules EFLAGS for both 32bit and 64bit, mark this for backporting to infinity.
And then at the end, after the objtool-ac bits land, I do a patch removing the EFLAGS scheduling for x86_64.
Re: [PATCH 01/25] x86: Make SMAP 64-bit only
Quoted 6 lines · Peter Zijlstra
How about I do a patch that schedules EFLAGS for both 32bit and 64bit, mark this for backporting to infinity.
And then at the end, after the objtool-ac bits land, I do a patch removing the EFLAGS scheduling for
x86_64.
Sounds sane to me.
And we can make it AC-conditional if it's actually shown to be visible from a performance standpoint.
But iirc pushf/popf isn't really that expensive - in fact I think it's pretty cheap when system flags don't change. Which would be the common case unless you'd also make the popf do the irq restore part and simply make finish_lock_switch() re-enable irq's by doing an irqrestore?
I think popf is like 20 cycles or something (and pushf is just single cycles). Probably not worth worrying about in the task switch context.
Linus
[email protected]
reviewerRe: [PATCH 01/25] x86: Make SMAP 64-bit only
Quoted 21 lines · Linus Torvalds
Quoted 2 lines · Peter Zijlstra
How about I do a patch that schedules EFLAGS for both 32bit and
64bit,
Quoted 4 lines
mark this for backporting to infinity.
And then at the end, after the objtool-ac bits land, I do a patch removing the EFLAGS scheduling for
x86_64.Sounds sane to me.
And we can make it AC-conditional if it's actually shown to be visible from a performance standpoint.
But iirc pushf/popf isn't really that expensive - in fact I think it's pretty cheap when system flags don't change. Which would be the common case unless you'd also make the popf do the irq restore part and simply make
finish_lock_switch()re-enable irq's by doing an irqrestore?
Quoted 4 lines
I think popf is like 20 cycles or something (and pushf is just single cycles). Probably not worth worrying about in the task switch context.
Linus
Yes, the problem isn't scheduling per se but the risk of hiding problems.
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: [PATCH 01/25] x86: Make SMAP 64-bit only
In reply to Linus TorvaldsQuoted 8 lines · Linus Torvalds
Quoted 6 lines · Peter Zijlstra
How about I do a patch that schedules EFLAGS for both 32bit and 64bit, mark this for backporting to infinity.
And then at the end, after the objtool-ac bits land, I do a patch removing the EFLAGS scheduling for
x86_64.
Quoted 7 lines
Sounds sane to me.
And we can make it AC-conditional if it's actually shown to be visible from a performance standpoint.
But iirc pushf/popf isn't really that expensive - in fact I think it's pretty cheap when system flags don't change.
I did not see evidence of this. In my testing,
POPF is always ~20 cycles, even if popped flags are identical to current state of flags.
[email protected]
reviewerRe: [PATCH 01/25] x86: Make SMAP 64-bit only
Quoted 17 lines · Denys Vlasenko
Quoted 1 line · Linus Torvalds
On Mon, Mar 18, 2019 at 10:51 AM Peter Zijlstra
Quoted 2 lines · <[email protected]> wrote:
Quoted 2 lines
How about I do a patch that schedules EFLAGS for both 32bit and
64bit,
Quoted 8 lines
Quoted 4 lines
mark this for backporting to infinity.
And then at the end, after the objtool-ac bits land, I do a patch removing the EFLAGS scheduling for
x86_64.Sounds sane to me.
And we can make it AC-conditional if it's actually shown to be
visible
Quoted 2 lines
from a performance standpoint.
Quoted 8 lines
Quoted 1 line
But iirc pushf/popf isn't really that expensive - in fact I think
it's
Quoted 1 line
pretty cheap when system flags don't change.
I did not see evidence of this. In my testing,
POPF is always ~20 cycles, even if popped flags are identical to current
state of flags.
I think you will find that if you change system flags it is much slower.
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Peter Zijlstra
maintainerRe: [PATCH 01/25] x86: Make SMAP 64-bit only
Quoted 7 lines · On Thu, Mar 21, 2019 at 11:18:05AM -0700, [email protected] wrote:
Quoted 3 lines · Denys Vlasenko
I did not see evidence of this. In my testing,
POPF is always ~20 cycles, even if popped flags are identical to current state of flags.I think you will find that if you change system flags it is much slower.
So with all the patches in this series applied, only x86_32 will suffer this, and I don't think anybody still considers that a performance critical platform.
That said, we could do something terrible like:
arch/x86/entry/entry_32.S +2 -1
@@ -673,7 +673,8 @@ ENTRY(__switch_to_asm) #endif /* restore callee-saved registers */ - popfl + ALTERNATIVE "popl %esi", \ + "popfl", X86_FEATURE_SMAP popl %esi popl %edi popl %ebx
And then you only pay the POPF penalty when you run a 32bit kernel on a SMAP enabled CPU, and we have a very good solution in that code: run a 64bit kernel.
Re: [PATCH 01/25] x86: Make SMAP 64-bit only
In reply to Denys VlasenkoQuoted 8 lines · Denys Vlasenko
Quoted 4 lines
But iirc pushf/popf isn't really that expensive - in fact I think it's pretty cheap when system flags don't change.
I did not see evidence of this. In my testing,
POPF is always ~20 cycles, even if popped flags are identical to current state of flags.
It may have been an artifact on just some older CPU's. I have this distinct memory of popf that changed IF being more expensive, but maybe that was the Pentium 4 days.
Or maybe it's just that my memory is garbage.
Linus
Peter Zijlstra
maintainerPATCH 1/25 sched/x86: Save [ER]FLAGS on context switch
In reply to Peter ZijlstraNew patch #1
---
Subject: sched/x86: Save [ER]FLAGS on context switch
From: Peter Zijlstra <[email protected]>
Date: Thu Feb 14 10:30:52 CET 2019
Effectively reverts commit:
2c7577a75837 ("sched/x86_64: Don't save flags on context switch")
Specifically because SMAP uses FLAGS.AC which invalidates the claim that the kernel has clean flags.
In particular; while preemption from interrupt return is fine (the IRET frame on the exception stack contains FLAGS) it breaks any code that does synchonous scheduling, including preempt_enable().
This has become a significant issue ever since commit:
5b24a7a2aa20 ("Add 'unsafe' user access functions for batched accesses")
provided for means of having 'normal' C code between STAC / CLAC, exposing the FLAGS.AC state. So far this hasn't led to trouble, however fix it before it comes apart.
--- arch/x86/entry/entry_32.S | 2 ++ arch/x86/entry/entry_64.S | 2 ++ arch/x86/include/asm/switch_to.h | 1 + arch/x86/kernel/process_32.c | 7 +++++++ arch/x86/kernel/process_64.c | 8 ++++++++ 5 files changed, 20 insertions(+)
arch/x86/entry/entry_32.S +2 -0
@@ -650,6 +650,7 @@ ENTRY(__switch_to_asm) pushl %ebx pushl %edi pushl %esi + pushfl /* switch stack */ movl %esp, TASK_threadsp(%eax)
@@ -672,6 +673,7 @@ ENTRY(__switch_to_asm) #endif /* restore callee-saved registers */ + popfl popl %esi popl %edi popl %ebx
arch/x86/entry/entry_64.S +2 -0
@@ -291,6 +291,7 @@ ENTRY(__switch_to_asm) pushq %r13 pushq %r14 pushq %r15 + pushfq /* switch stack */ movq %rsp, TASK_threadsp(%rdi)
@@ -313,6 +314,7 @@ ENTRY(__switch_to_asm) #endif /* restore callee-saved registers */ + popfq popq %r15 popq %r14 popq %r13
arch/x86/include/asm/switch_to.h +1 -0
@@ -40,6 +40,7 @@ asmlinkage void ret_from_fork(void);* order of the fields must match the code in __switch_to_asm(). */ struct inactive_task_frame { +unsigned long flags;#ifdef CONFIG_X86_64 unsigned long r15;unsigned long r14;
arch/x86/kernel/process_32.c +7 -0
@@ -127,6 +127,13 @@ int copy_thread_tls(unsigned long clone_struct task_struct *tsk;int err; +/* +* For a new task use the RESET flags value since there is no before. +* All the status flags are zero; DF and all the system flags must also +* be 0, specifically IF must be 0 because we context switch to the new +* task with interrupts disabled. +*/ + frame->flags =X86_EFLAGS_FIXED ; frame->bp =0 ; frame->ret_addr = (unsigned long ) ret_from_fork; p->thread.sp = (unsigned long ) fork_frame;
arch/x86/kernel/process_64.c +8 -0
@@ -392,6 +392,14 @@ int copy_thread_tls(unsigned long clone_ childregs =task_pt_regs (p); fork_frame = container_of(childregs, struct fork_frame, regs); frame = &fork_frame->frame; + +/* +* For a new task use the RESET flags value since there is no before. +* All the status flags are zero; DF and all the system flags must also +* be 0, specifically IF must be 0 because we context switch to the new +* task with interrupts disabled. +*/ + frame->flags =X86_EFLAGS_FIXED ; frame->bp =0 ; frame->ret_addr = (unsigned long ) ret_from_fork; p->thread.sp = (unsigned long ) fork_frame;
Peter Zijlstra
maintainerPATCH 2/25 tracing: Improve "if" macro code generation
In reply to Peter ZijlstraWith CONFIG_PROFILE_ALL_BRANCHES, the "if" macro converts the conditional to an array index. This can cause GCC to create horrible code. When there are nested ifs, the generated code uses register values to encode branching decisions.
Make it easier for GCC to optimize by keeping the conditional as a conditional rather than converting it to an integer. This shrinks the generated code quite a bit, and also makes the code sane enough for objtool to understand.
--- include/linux/compiler.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/compiler.h +1 -1
@@ -67,7 +67,7 @@ void ftrace_likely_update(struct ftrace_ .line = __LINE__, \ }; \ ______r = !!(cond); \ - ______f.miss_hit[______r]++; \ + ______r ? ______f.miss_hit[1 ]++ : ______f.miss_hit[0 ]++;\ ______r; \ }))#endif /* CONFIG_PROFILE_ALL_BRANCHES */
Re: [PATCH 02/25] tracing: Improve "if" macro code generation
Quoted 21 lines · On Mon, 18 Mar 2019 16:38:42 +0100 Peter Zijlstra <[email protected]> wrote:
With
CONFIG_PROFILE_ALL_BRANCHES, the "if" macro converts the conditional to an array index. This can cause GCC to create horrible code. When there are nested ifs, the generated code uses register values to encode branching decisions.Make it easier for GCC to optimize by keeping the conditional as a conditional rather than converting it to an integer. This shrinks the generated code quite a bit, and also makes the code sane enough for objtool to understand.
Quoted 8 lines
Reported-by Peter Zijlstra <[email protected]>Signed-off-by Josh Poimboeuf <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>
-- Steve
Quoted 17 lines
--- include/linux/compiler.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)include/linux/compiler.h +1
-1@@ -67,7 +67,7 @@ void ftrace_likely_update(struct ftrace_ .line = __LINE__, \ }; \ ______r = !!(cond); \ - ______f.miss_hit[______r]++; \ + ______r ? ______f.miss_hit[1 ]++ : ______f.miss_hit[0 ]++;\ ______r; \ }))#endif /* CONFIG_PROFILE_ALL_BRANCHES */
Re: [PATCH 02/25] tracing: Improve "if" macro code generation
In reply to Peter ZijlstraQuoted 22 lines · Peter Zijlstra
With
CONFIG_PROFILE_ALL_BRANCHES, the "if" macro converts the conditional to an array index. This can cause GCC to create horrible code. When there are nested ifs, the generated code uses register values to encode branching decisions.Make it easier for GCC to optimize by keeping the conditional as a conditional rather than converting it to an integer. This shrinks the generated code quite a bit, and also makes the code sane enough for objtool to understand.
Quoted 8 lines
Reported-by Peter Zijlstra <[email protected]>Signed-off-by Josh Poimboeuf <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Missing "From" me.
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH 02/25] tracing: Improve "if" macro code generation
Quoted 32 lines · Josh Poimboeuf
Quoted 22 lines · Peter Zijlstra
With
CONFIG_PROFILE_ALL_BRANCHES, the "if" macro converts the conditional to an array index. This can cause GCC to create horrible code. When there are nested ifs, the generated code uses register values to encode branching decisions.Make it easier for GCC to optimize by keeping the conditional as a conditional rather than converting it to an integer. This shrinks the generated code quite a bit, and also makes the code sane enough for objtool to understand.
Quoted 8 lines
Reported-by Peter Zijlstra <[email protected]>Signed-off-by Josh Poimboeuf <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Quoted 1 line
Missing "From" me.
Right, sorry, quilt eats that and I sometimes forget to fix up :/
RE: [PATCH 02/25] tracing: Improve "if" macro code generation
In reply to Peter ZijlstraFrom: > Peter Zijlstra
Quoted 11 lines
Sent: 18 March 2019 15:39
With
CONFIG_PROFILE_ALL_BRANCHES, the "if" macro converts the conditional to an array index. This can cause GCC to create horrible code. When there are nested ifs, the generated code uses register values to encode branching decisions.Make it easier for GCC to optimize by keeping the conditional as a conditional rather than converting it to an integer. This shrinks the generated code quite a bit, and also makes the code sane enough for objtool to understand.
...
include/linux/compiler.h void ftrace_likely_update(struct ftrace_ · line 69
Quoted 9 lines
include/linux/compiler.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)include/linux/compiler.h +0
-0@@ -67,7 +67,7 @@ void ftrace_likely_update(struct ftrace_ .line = __LINE__, \ }; \ ______r = !!(cond); \
Is that (or maybe just the !!) needed any more??
include/linux/compiler.h void ftrace_likely_update(struct ftrace_ · line 73
Quoted 6 lines
include/linux/compiler.h +1
-1- ______f.miss_hit[______r]++; \ + ______r ? ______f.miss_hit[1 ]++ : ______f.miss_hit[0 ]++;\ ______r; \ }))#endif /* CONFIG_PROFILE_ALL_BRANCHES */
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Re: [PATCH 02/25] tracing: Improve "if" macro code generation
Quoted 4 lines · David Laight
Quoted 2 lines
______r = !!(cond); \Is that (or maybe just the !!) needed any more??
It is, because the 'cond' expression might not be an int, it could be a test for a pointer being non-NULL, or an u64 being non-zero, and not having the "!!" would mean that you'd get a warning or drop bits when assigning to 'int'.
And you do need the new temporary variable to avoid double evaluation the way that code is written.
That said, I do think the code is really ugly. We could:
- avoid the temporary by just simplifying things.
- do the '!!' just once in the parent macro.
- Steven has this crazy model of "more underscores are better". They
aren't. They don't help if things nest anyway, but what does help is meaningful names. Both when things don't nest, and when looking at generated asm files.
- ,, and finally, what _is_ better is to chop things up so that they
are smaller and make each macro do only one thing
So maybe do the patch something like the attached instead? Completely untested, but it looks sane to me.
Linus
include/linux/compiler.h | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-)
include/linux/compiler.h +18 -17
@@ -53,23 +53,24 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,* "Define 'is'", Bill Clinton * "Define 'if'", Steven Rostedt */ -#define if(cond, ...) __trace_if( (cond , ## __VA_ARGS__) ) -#define __trace_if (cond) \ - if (__builtin_constant_p(!!(cond)) ? !!(cond) : \ - ({ \ - int ______r; \ - static struct ftrace_branch_data \ - __aligned(4) \ - __section("_ftrace_branch") \ - ______f = { \ - .func = __func__, \ - .file = __FILE__, \ - .line = __LINE__, \ - }; \ - ______r = !!(cond); \ - ______f.miss_hit[______r]++; \ - ______r; \ - })) +#define if(cond, ...) if ( __trace_if_var( !!(cond , ## __VA_ARGS__) ) ) + +#define __trace_if_var (cond) (__builtin_constant_p(cond) ? (cond) : __trace_if_value(cond)) + +#define __trace_if_value (cond) ({ \ + static struct ftrace_branch_data \ + __aligned(4) \ + __section("_ftrace_branch") \ + __if_trace = { \ + .func = __func__, \ + .file = __FILE__, \ + .line = __LINE__, \ + }; \ + (cond) ? \ + (__if_trace.miss_hit[1]++,1) : \ + (__if_trace.miss_hit[0]++,0); \ +}) +#endif /* CONFIG_PROFILE_ALL_BRANCHES */ #else
RE: [PATCH 02/25] tracing: Improve "if" macro code generation
From: Linus Torvalds
Quoted 8 lines
Sent: 20 March 2019 17:26
To: David LaightQuoted 4 lines · David Laight
Quoted 2 lines
______r = !!(cond); \Is that (or maybe just the !!) needed any more??
Quoted 7 lines
It is, because the 'cond' expression might not be an int, it could be a test for a pointer being non-NULL, or an u64 being non-zero, and not having the "!!" would mean that you'd get a warning or drop bits when assigning to 'int'.
And you do need the new temporary variable to avoid double evaluation the way that code is written.
As usual I'd opened my mouth before checking the full context :-)
Review
Quoted 4 lines
______r = !!(cond);(quoted diff) +1
-1- ______f.miss_hit[______r]++; \ + ______r ? ______f.miss_hit[1]++ : ______f.miss_hit[0]++;\ ______r; \
Actually you can avoid double evaluation by doing:
(cond) ? (______f.miss_hit[1]++, 1) : (______f.miss_hit[0]++, 0)
With luck the compiler will move the increment to after the branch target.
for (_____ = ____; _____ < ______; _____++) :-)
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Re: [PATCH 02/25] tracing: Improve "if" macro code generation
Quoted 5 lines · David Laight
Actually you can avoid double evaluation by doing:
(cond) ? (______f.miss_hit[1]++, 1) : (______f.miss_hit[0]++, 0)
I don't think you looked at the patch in my attachment of the email you replied to, did you?
That's exactly what it does, except it also gets rid of those completely pointless and insane underscores, and makes the code more legible in other ways too.
Linus
Re: [PATCH 02/25] tracing: Improve "if" macro code generation
In reply to Linus TorvaldsQuoted 4 lines · On Wed, 20 Mar 2019 10:26:17 -0700 Linus Torvalds…
- Steven has this crazy model of "more underscores are better". They
aren't. They don't help if things nest anyway, but what does help is meaningful names. Both when things don't nest, and when looking at generated asm files.
Note, at the time I had Ingo as a mentor ;-)
Quoted 34 lines
include/linux/compiler.h | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-)include/linux/compiler.h +6
-17@@ -53,23 +53,24 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,* "Define 'is'", Bill Clinton * "Define 'if'", Steven Rostedt */ -#define if(cond, ...) __trace_if( (cond , ## __VA_ARGS__) ) -#define __trace_if (cond) \ - if (__builtin_constant_p(!!(cond)) ? !!(cond) : \ - ({ \ - int ______r; \ - static struct ftrace_branch_data \ - __aligned(4) \ - __section("_ftrace_branch") \ - ______f = { \ - .func = __func__, \ - .file = __FILE__, \ - .line = __LINE__, \ - }; \ - ______r = !!(cond); \ - ______f.miss_hit[______r]++; \ - ______r; \ - })) +#define if(cond, ...) if ( __trace_if_var( !!(cond , ## __VA_ARGS__) ) ) + +#define __trace_if_var (cond) (__builtin_constant_p(cond) ? (cond) : __trace_if_value(cond)) + +#define __trace_if_value (cond) ({ \ + static struct ftrace_branch_data \
include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val, · line 71
Quoted 10 lines
include/linux/compiler.h +10
-0+ __aligned(4) \ + __section("_ftrace_branch") \ + __if_trace = { \ + .func = __func__, \ + .file = __FILE__, \ + .line = __LINE__, \ + }; \ + (cond) ? \ + (__if_trace.miss_hit[1 ]++,1 ) : \ + (__if_trace.miss_hit[0 ]++,0 ); \
Probably want to add a space between the comma and the numbers.
But other than that.
-- Steve
Quoted 5 lines
include/linux/compiler.h +2
-0+}) +#endif /* CONFIG_PROFILE_ALL_BRANCHES */ #else
Steven Rostedt
maintainerRe: [PATCH 02/25] tracing: Improve "if" macro code generation
In reply to Linus TorvaldsQuoted 25 lines · On Wed, 20 Mar 2019 10:26:17 -0700 Linus Torvalds…
Quoted 4 lines · David Laight
Quoted 2 lines
______r = !!(cond); \Is that (or maybe just the !!) needed any more??
It is, because the 'cond' expression might not be an int, it could be a test for a pointer being non-NULL, or an u64 being non-zero, and not having the "!!" would mean that you'd get a warning or drop bits when assigning to 'int'.
And you do need the new temporary variable to avoid double evaluation the way that code is written.
That said, I do think the code is really ugly. We could:
- avoid the temporary by just simplifying things.
- do the '!!' just once in the parent macro.
- Steven has this crazy model of "more underscores are better". They
aren't. They don't help if things nest anyway, but what does help is meaningful names. Both when things don't nest, and when looking at generated asm files.
Quoted 6 lines
- ,, and finally, what _is_ better is to chop things up so that they
are smaller and make each macro do only one thingSo maybe do the patch something like the attached instead? Completely untested, but it looks sane to me.
Linus,
This patch works. Can I get your Signed-off-by for it?
-- Steve
Re: [PATCH 02/25] tracing: Improve "if" macro code generation
Quoted 2 lines · Steven Rostedt
This patch works. Can I get your Signed-off-by for it?
Yes. Please write some kind of comprehensible commit log for it, but
for the patch itself.
Linus
Steven Rostedt
maintainerRe: [PATCH 02/25] tracing: Improve "if" macro code generation
Quoted 5 lines · On Thu, 9 May 2019 09:51:59 -0700 Linus Torvalds…
Quoted 2 lines · Steven Rostedt
This patch works. Can I get your Signed-off-by for it?
Yes. Please write some kind of comprehensible commit log for it, but
How's this:
"Peter Zijlstra noticed that with CONFIG_PROFILE_ALL_BRANCHES, the "if" macro converts the conditional to an array index. This can cause GCC to create horrible code. When there are nested ifs, the generated code uses register values to encode branching decisions.
Josh Poimboeuf found that replacing the define "if" macro from using the condition as an array index and incrementing the branch statics with an if statement itself, reduced the asm complexity and shrinks the generated code quite a bit.
But this can be simplified even further by replacing the internal if statement with a ternary operator.
"
Quoted 2 lines
Signed-off-by Linus Torvalds <[email protected]>
Thanks,
-- Steve
Quoted 3 lines
for the patch itself.
Re: [PATCH 02/25] tracing: Improve "if" macro code generation
Quoted 21 lines · Steven Rostedt
Quoted 5 lines · On Thu, 9 May 2019 09:51:59 -0700 Linus Torvalds…
Quoted 2 lines · Steven Rostedt
This patch works. Can I get your Signed-off-by for it?
Yes. Please write some kind of comprehensible commit log for it, but
How's this:
"Peter Zijlstra noticed that with
CONFIG_PROFILE_ALL_BRANCHES, the "if" macro converts the conditional to an array index. This can cause GCC to create horrible code. When there are nested ifs, the generated code uses register values to encode branching decisions.Josh Poimboeuf found that replacing the define "if" macro from using the condition as an array index and incrementing the branch statics with an if statement itself, reduced the asm complexity and shrinks the generated code quite a bit.
Quoted 5 lines
But this can be simplified even further by replacing the internal if statement with a ternary operator.
Reported-by Peter Zijlstra (Intel) <[email protected]>Reported-by Josh Poimboeuf <[email protected]>
Actually, my original fix already went in:
37686b1353cf ("tracing: Improve "if" macro code generation")
But it introduced a regression:
https://lkml.kernel.org/r/201905040509.iqQ2CrOU%[email protected]
which Linus' patch fixes for some reason.
Signature · 1 line
Josh
Re: [PATCH 02/25] tracing: Improve "if" macro code generation
Quoted 30 lines · Josh Poimboeuf
Quoted 21 lines · Steven Rostedt
Quoted 5 lines · On Thu, 9 May 2019 09:51:59 -0700 Linus Torvalds…
Quoted 2 lines · Steven Rostedt
This patch works. Can I get your Signed-off-by for it?
Yes. Please write some kind of comprehensible commit log for it, but
How's this:
"Peter Zijlstra noticed that with
CONFIG_PROFILE_ALL_BRANCHES, the "if" macro converts the conditional to an array index. This can cause GCC to create horrible code. When there are nested ifs, the generated code uses register values to encode branching decisions.Josh Poimboeuf found that replacing the define "if" macro from using the condition as an array index and incrementing the branch statics with an if statement itself, reduced the asm complexity and shrinks the generated code quite a bit.
Quoted 5 lines
But this can be simplified even further by replacing the internal if statement with a ternary operator.
Reported-by Peter Zijlstra (Intel) <[email protected]>Reported-by Josh Poimboeuf <[email protected]>Actually, my original fix already went in:
Quoted 7 lines
37686b1353cf("tracing: Improve "if" macro code generation")But it introduced a regression:
https://lkml.kernel.org/r/201905040509.iqQ2CrOU%[email protected]
which Linus' patch fixes for some reason.
/me curses URL encoding
https://lkml.kernel.org/r/201905040509.iqQ2CrOU%[email protected]
Signature · 1 line
Josh
Re: [PATCH 02/25] tracing: Improve "if" macro code generation
Quoted 35 lines · Josh Poimboeuf
Quoted 34 lines · Josh Poimboeuf
Quoted 21 lines · Steven Rostedt
Quoted 5 lines · On Thu, 9 May 2019 09:51:59 -0700 Linus Torvalds…
Quoted 2 lines · Steven Rostedt
This patch works. Can I get your Signed-off-by for it?
Yes. Please write some kind of comprehensible commit log for it, but
How's this:
"Peter Zijlstra noticed that with
CONFIG_PROFILE_ALL_BRANCHES, the "if" macro converts the conditional to an array index. This can cause GCC to create horrible code. When there are nested ifs, the generated code uses register values to encode branching decisions.Josh Poimboeuf found that replacing the define "if" macro from using the condition as an array index and incrementing the branch statics with an if statement itself, reduced the asm complexity and shrinks the generated code quite a bit.
Quoted 5 lines
But this can be simplified even further by replacing the internal if statement with a ternary operator.
Reported-by Peter Zijlstra (Intel) <[email protected]>Reported-by Josh Poimboeuf <[email protected]>Actually, my original fix already went in:
37686b1353cf("tracing: Improve "if" macro code generation")But it introduced a regression:
Quoted 8 lines
Quoted 3 lines
https://lkml.kernel.org/r/201905040509.iqQ2CrOU%[email protected]
which Linus' patch fixes for some reason.
/me curses URL encoding
https://lkml.kernel.org/r/201905040509.iqQ2CrOU%[email protected]
Still fails for me.
Signature · 1 line
~Randy
Re: [PATCH 02/25] tracing: Improve "if" macro code generation
Quoted 38 lines · Randy Dunlap
Quoted 37 lines · Josh Poimboeuf
Quoted 36 lines · Josh Poimboeuf
Quoted 21 lines · Steven Rostedt
Quoted 5 lines · On Thu, 9 May 2019 09:51:59 -0700 Linus Torvalds…
Quoted 2 lines · Steven Rostedt
This patch works. Can I get your Signed-off-by for it?
Yes. Please write some kind of comprehensible commit log for it, but
How's this:
"Peter Zijlstra noticed that with
CONFIG_PROFILE_ALL_BRANCHES, the "if" macro converts the conditional to an array index. This can cause GCC to create horrible code. When there are nested ifs, the generated code uses register values to encode branching decisions.Josh Poimboeuf found that replacing the define "if" macro from using the condition as an array index and incrementing the branch statics with an if statement itself, reduced the asm complexity and shrinks the generated code quite a bit.
Quoted 5 lines
But this can be simplified even further by replacing the internal if statement with a ternary operator.
Reported-by Peter Zijlstra (Intel) <[email protected]>Reported-by Josh Poimboeuf <[email protected]>Actually, my original fix already went in:
37686b1353cf("tracing: Improve "if" macro code generation")But it introduced a regression:
https://lkml.kernel.org/r/201905040509.iqQ2CrOU%[email protected]
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
which Linus' patch fixes for some reason.
/me curses URL encoding
https://lkml.kernel.org/r/201905040509.iqQ2CrOU%[email protected]
Still fails for me.
Sorry, another URL fail. It wasn't on lkml. I should actually try clicking my own links.
https://lists.01.org/pipermail/kbuild-all/2019-May/060554.html
Signature · 1 line
Josh
Steven Rostedt
maintainerRe: [PATCH 02/25] tracing: Improve "if" macro code generation
In reply to Josh PoimboeufQuoted 9 lines · On Thu, 9 May 2019 13:45:31 -0500 Josh Poimboeuf <[email protected]> wrote:
Actually, my original fix already went in:
37686b1353cf("tracing: Improve "if" macro code generation")But it introduced a regression:
https://lkml.kernel.org/r/201905040509.iqQ2CrOU%[email protected]
which Linus' patch fixes for some reason.
Hmm, I'm still working on my pull request for the merge window, and if this already went in, I could just add this, and let it conflict. I'm sure Linus will have no problems in fixing up the conflicts.
I should change the subject, as it is the same ;-) Perhaps to:
tracing: Clean up "if" macro
But it would be good to find out why this fixes the issue you see. Perhaps its because we remove the internal if statement?
-- Steve
Steven Rostedt
maintainerRe: [PATCH 02/25] tracing: Improve "if" macro code generation
Quoted 10 lines · On Thu, 9 May 2019 15:06:44 -0400 Steven Rostedt <[email protected]> wrote:
Hmm, I'm still working on my pull request for the merge window, and if this already went in, I could just add this, and let it conflict. I'm sure Linus will have no problems in fixing up the conflicts.
I should change the subject, as it is the same ;-) Perhaps to:
tracing: Clean up "if" macro
But it would be good to find out why this fixes the issue you see. Perhaps its because we remove the internal if statement?
I'm adding this to my tree, if that's alright with everyone. It will conflict with your patch, but like I said, Linus should have no problem fixing up the conflicts.
But it probably would probably still be good to know why this fixes the issues you see.
-- Steve
From: Linus Torvalds <[email protected]>
Subject: [PATCH] tracing: Simplify "if" macro code
Peter Zijlstra noticed that with CONFIG_PROFILE_ALL_BRANCHES, the "if" macro converts the conditional to an array index. This can cause GCC to create horrible code. When there are nested ifs, the generated code uses register values to encode branching decisions.
Josh Poimboeuf found that replacing the define "if" macro from using the condition as an array index and incrementing the branch statics with an if statement itself, reduced the asm complexity and shrinks the generated code quite a bit.
But this can be simplified even further by replacing the internal if statement with a ternary operator.
--- include/linux/compiler.h | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-)
include/linux/compiler.h +18 -17
@@ -53,23 +53,24 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,* "Define 'is'", Bill Clinton * "Define 'if'", Steven Rostedt */ -#define if(cond, ...) __trace_if( (cond , ## __VA_ARGS__) ) -#define __trace_if (cond) \ - if (__builtin_constant_p(!!(cond)) ? !!(cond) : \ - ({ \ - int ______r; \ - static struct ftrace_branch_data \ - __aligned(4) \ - __section("_ftrace_branch") \ - ______f = { \ - .func = __func__, \ - .file = __FILE__, \ - .line = __LINE__, \ - }; \ - ______r = !!(cond); \ - ______f.miss_hit[______r]++; \ - ______r; \ - })) +#define if(cond, ...) if ( __trace_if_var( !!(cond , ## __VA_ARGS__) ) ) + +#define __trace_if_var (cond) (__builtin_constant_p(cond) ? (cond) : __trace_if_value(cond)) + +#define __trace_if_value (cond) ({ \ + static struct ftrace_branch_data \ + __aligned(4) \ + __section("_ftrace_branch") \ + __if_trace = { \ + .func = __func__, \ + .file = __FILE__, \ + .line = __LINE__, \ + }; \ + (cond) ? \ + (__if_trace.miss_hit[1]++,1) : \ + (__if_trace.miss_hit[0]++,0); \ +}) +#endif /* CONFIG_PROFILE_ALL_BRANCHES */ #else
Signature · 1 line
2.20.1
Re: [PATCH 02/25] tracing: Improve "if" macro code generation
Quoted 3 lines · Steven Rostedt
But it probably would probably still be good to know why this fixes the issues you see.
Looks like a compiler bug, plain and simple.
The simplified case really simplifies a lot for the internal IR, and has a single conditional that then increments a constant location and returns a constant value. That means that any branch following code in the compiler has a much simpler setup, and doesn't need to try to follow any chain of condirional branches with the same conditional to generate sane code.
But the code generation problem is clearly a compiler bug, although I'd not be surprised if it's also triggered by whatever horrid things ASAN does internally to gcc state.
Linus
Peter Zijlstra
maintainerPATCH 3/25 x86/ia32: Fix ia32_restore_sigcontext AC leak
In reply to Peter ZijlstraObjtool spotted that we call native_load_gs_index() with AC set. Re-arrange the code to avoid that.
---
arch/x86/ia32/ia32_signal.c +17 -12
@@ -61,9 +61,8 @@ } while (0)#define RELOAD_SEG (seg) { \ - unsigned int pre = GET_SEG(seg); \ + unsigned int pre = (seg) | 3; \ unsigned int cur = get_user_seg(seg); \ - pre |= 3; \ if (pre != cur) \ set_user_seg(seg, pre); \ }
@@ -72,6 +71,7 @@ static int ia32_restore_sigcontext(struct pt_regs *regs, struct sigcontext_32 __user *sc) {unsigned int tmpflags, err =0 ; +u16 gs, fs, es, ds;void __user *buf;u32 tmp;
@@ -79,16 +79,10 @@ static int ia32_restore_sigcontext(struct pt_regs *regs, current->restart_block.fn = do_no_restart_syscall; get_user_try { -/* -* Reload fs and gs if they have changed in the signal -* handler. This does not handle long fs/gs base changes in -* the handler, but does not clobber them at least in the -* normal case. -*/ -RELOAD_SEG (gs); -RELOAD_SEG (fs); -RELOAD_SEG (ds); -RELOAD_SEG (es); + gs =GET_SEG (gs); + fs =GET_SEG (fs); + ds =GET_SEG (ds); + es =GET_SEG (es);COPY (di);COPY (si);COPY (bp);COPY (sp);COPY (bx);COPY (dx);COPY (cx);COPY (ip);COPY (ax);
@@ -106,6 +100,17 @@ static int ia32_restore_sigcontext(struct pt_regs *regs, buf =compat_ptr (tmp); }get_user_catch (err); +/* +* Reload fs and gs if they have changed in the signal +* handler. This does not handle long fs/gs base changes in +* the handler, but does not clobber them at least in the +* normal case. +*/ +RELOAD_SEG (gs); +RELOAD_SEG (fs); +RELOAD_SEG (ds); +RELOAD_SEG (es); + err |=fpu__restore_sig (buf,1 );force_iret ();
Peter Zijlstra
maintainerPATCH 4/25 i915,uaccess: Fix redundant CLAC
In reply to Peter Zijlstradrivers/gpu/drm/i915/i915_gem_execbuffer.o: warning: objtool: .altinstr_replacement+0x3c: redundant UACCESS disable drivers/gpu/drm/i915/i915_gem_execbuffer.o: warning: objtool: .altinstr_replacement+0x66: redundant UACCESS disable
AKA. you don't need user_access_end() if user_access_begin() fails.
--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +4 -2
@@ -1606,6 +1606,7 @@ static int eb_copy_relocations(const str len)) { end_user:user_access_end (); +end:kvfree (relocs); err = -EFAULT ; goto err;
@@ -1625,7 +1626,7 @@ static int eb_copy_relocations(const str* relocations were valid. */ if (!user_access_begin (urelocs, size)) - goto end_user; + goto end;for (copied =0 ; copied < nreloc; copied++) unsafe_put_user(-1,
@@ -2616,7 +2617,7 @@ i915_gem_execbuffer2_ioctl(struct drm_de* when we did the "copy_from_user()" above. */ if (!user_access_begin (user_exec_list, count *sizeof (*user_exec_list))) - goto end_user; + goto end;for (i =0 ; i < args->buffer_count; i++) {if (!(exec2_list[i].offset &UPDATE ))
@@ -2630,6 +2631,7 @@ i915_gem_execbuffer2_ioctl(struct drm_de } end_user:user_access_end (); +end:; } args->flags &= ~__I915_EXEC_UNKNOWN_FLAGS;
Peter Zijlstra
maintainerPATCH 5/25 x86/uaccess: Move copy_user_handle_tail into asm
In reply to Peter ZijlstraBy writing the function in asm we avoid cross object code flow and objtool no longer gets confused about a 'stray' CLAC.
Also; the asm version is actually _simpler_.
--- arch/x86/include/asm/asm.h | 24 ------------------- arch/x86/include/asm/uaccess_64.h | 3 -- arch/x86/lib/copy_user_64.S | 48 ++++++++++++++++++++++++++++++++++++++ arch/x86/lib/usercopy_64.c | 20 --------------- 4 files changed, 48 insertions(+), 47 deletions(-)
arch/x86/include/asm/asm.h +0 -24
@@ -148,30 +148,6 @@_ASM_PTR (entry); \ .popsection -.macro ALIGN_DESTINATION -/* check for bad alignment of destination */ - movl %edi,%ecx - andl $7,%ecx - jz 102f /* already aligned */ - subl $8,%ecx - negl %ecx - subl %ecx,%edx -100: movb (%rsi),%al -101: movb %al,(%rdi) - incq %rsi - incq %rdi - decl %ecx - jnz 100b -102: - .section .fixup,"ax" -103: addl %ecx,%edx /* ecx is zerorest also */ - jmp copy_user_handle_tail - .previous - - _ASM_EXTABLE_UA(100b, 103b) - _ASM_EXTABLE_UA(101b, 103b) - .endm - #else# define _EXPAND_EXTABLE_HANDLE (x) #x# define _ASM_EXTABLE_HANDLE (from, to, handler) \
arch/x86/include/asm/uaccess_64.h +0 -3
@@ -208,9 +208,6 @@ __copy_from_user_flushcache(void *dst, c }unsigned long -copy_user_handle_tail (char *to,char *from,unsigned len); - -unsigned long mcsafe_handle_tail (char *to,char *from,unsigned len);#endif /* _ASM_X86_UACCESS_64_H */
arch/x86/lib/copy_user_64.S +48 -0
60 lines not on this page · raw message
arch/x86/lib/usercopy_64.c +0 -20
@@ -55,26 +55,6 @@ unsigned long clear_user(void __user *toEXPORT_SYMBOL (clear_user);/* -* Try to copy last bytes and clear the rest if needed. -* Since protection fault in copy_from/to_user is not a normal situation, -* it is not necessary to optimize tail handling. -*/ -__visible unsigned long -copy_user_handle_tail (char *to,char *from,unsigned len) -{ -for (; len; --len, to++) { -char c; - -if (__get_user_nocheck (c, from++,sizeof (char ))) -break ; -if (__put_user_nocheck (c, to,sizeof (char ))) -break ; - } -clac (); -return len; -} - -/* * Similar to copy_user_handle_tail, probe for the write fault point, * but reuse __memcpy_mcsafe in case a new read error is encountered. * clac() is handled in _copy_to_iter_mcsafe().
Peter Zijlstra
maintainerPATCH 6/25 x86/uaccess: Fix up the fixup
In reply to Peter Zijlstraarch/x86/lib/memcpy_64.o: warning: objtool: .fixup+0x7: return with UACCESS enabled
While the code isn't wrong; it is tedious (if at all possible) to figure out what function a particular chunk of .fixup belongs to.
This then confuses the objtool uaccess validation. Instead of returning directly from the .fixup, jump back into the right function.
--- arch/x86/lib/memcpy_64.S | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/lib/memcpy_64.S +2 -1
@@ -257,6 +257,7 @@ ENTRY(__memcpy_mcsafe) /* Copy successful. Return zero */ .L_done_memcpy_trap: xorl %eax, %eax +.L_done: ret ENDPROC(__memcpy_mcsafe) EXPORT_SYMBOL_GPL(__memcpy_mcsafe)
@@ -273,7 +274,7 @@ EXPORT_SYMBOL_GPL(__memcpy_mcsafe) addl %edx, %ecx .E_trailing_bytes: mov %ecx, %eax - ret + jmp .L_done /* * For write fault handling, given the destination is unaligned,
Peter Zijlstra
maintainerPATCH 7/25 x86/nospec,objtool: Introduce ANNOTATE_IGNORE_ALTERNATIVE
In reply to Peter ZijlstraTo facillitate other usage of ignoring alternatives; rename ANNOTATE_NOSPEC_IGNORE to ANNOTATE_IGNORE_ALTERNATIVE.
--- arch/x86/include/asm/alternative-asm.h | 11 +++++++++++ arch/x86/include/asm/alternative.h | 10 ++++++++++ arch/x86/include/asm/nospec-branch.h | 28 +++++++++------------------- tools/objtool/check.c | 8 ++++---- 4 files changed, 34 insertions(+), 23 deletions(-)
arch/x86/include/asm/alternative-asm.h +11 -0
@@ -20,6 +20,17 @@#endif /* +* objtool annotation to ignore the alternatives and only consider the original +* instruction(s). +*/ +.macro ANNOTATE_IGNORE_ALTERNATIVE + .Lannotate_\@: + .pushsection .discard.ignore_alts + .long .Lannotate_\@ - . + .popsection +.endm + +/* * Issue one struct alt_instr descriptor entry (need to put it into * the section .altinstructions, see below). This entry contains * enough information for the alternatives patching code to patch an
arch/x86/include/asm/alternative.h +10 -0
@@ -45,6 +45,16 @@#define LOCK_PREFIX ""#endif +/* +* objtool annotation to ignore the alternatives and only consider the original +* instruction(s). +*/ +#define ANNOTATE_IGNORE_ALTERNATIVE \ + "999:\n\t" \ + ".pushsection .discard.ignore_alts\n\t" \ + ".long 999b - .\n\t" \ + ".popsection\n\t" +struct alt_instr {s32 instr_offset;/* original instruction */ s32 repl_offset;/* offset to replacement instruction */
arch/x86/include/asm/nospec-branch.h +9 -19
@@ -11,6 +11,15 @@#include <asm/msr-index.h> /* +* This should be used immediately before a retpoline alternative. It tells +* objtool where the retpolines are so that it can make sense of the control +* flow by just reading the original instruction(s) and ignoring the +* alternatives. +*/ +#define ANNOTATE_NOSPEC_ALTERNATIVE \ + ANNOTATE_IGNORE_ALTERNATIVE + +/* * Fill the CPU return stack buffer. * * Each entry in the RSB, if used for a speculative 'ret', contains an
@@ -57,19 +66,6 @@#ifdef __ASSEMBLY__/* -* This should be used immediately before a retpoline alternative. It tells -* objtool where the retpolines are so that it can make sense of the control -* flow by just reading the original instruction(s) and ignoring the -* alternatives. -*/ -.macro ANNOTATE_NOSPEC_ALTERNATIVE - .Lannotate_\@: - .pushsection .discard.nospec - .long .Lannotate_\@ - . - .popsection -.endm - -/* * This should be used immediately before an indirect jump/call. It tells * objtool the subsequent indirect jump/call is vouched safe for retpoline * builds.
@@ -152,12 +148,6 @@#else /* __ASSEMBLY__ */ -#define ANNOTATE_NOSPEC_ALTERNATIVE \ - "999:\n\t" \ - ".pushsection .discard.nospec\n\t" \ - ".long 999b - .\n\t" \ - ".popsection\n\t" -#define ANNOTATE_RETPOLINE_SAFE \ "999:\n\t" \ ".pushsection .discard.retpoline_safe\n\t" \
tools/objtool/check.c +4 -4
@@ -457,13 +457,13 @@ static void add_ignores(struct objtool_f* But it at least allows objtool to understand the control flow *around* the * retpoline. */ -static int add_nospec_ignores (struct objtool_file *file) +static int add_ignore_alternatives (struct objtool_file *file) {struct section *sec;struct rela *rela;struct instruction *insn; - sec =find_section_by_name (file->elf,".rela.discard.nospec" ); + sec =find_section_by_name (file->elf,".rela.discard.ignore_alts" );if (!sec)return 0 ;
@@ -475,7 +475,7 @@ static int add_nospec_ignores(struct obj insn =find_insn (file, rela->sym->sec, rela->addend);if (!insn) { -WARN ("bad .discard.nospec entry" ); +WARN ("bad .discard.ignore_alts entry" );return -1 ; }
@@ -1239,7 +1239,7 @@ static int decode_sections(struct objtooadd_ignores (file); - ret =add_nospec_ignores (file); + ret =add_ignore_alternatives (file);if (ret)return ret;
Peter Zijlstra
maintainerPATCH 8/25 x86/uaccess,xen: Suppress SMAP warnings
In reply to Peter Zijlstradrivers/xen/privcmd.o: warning: objtool: privcmd_ioctl()+0x1414: call to hypercall_page() with UACCESS enabled
Some Xen hypercalls allow parameter buffers in user land, so they need to set AC=1. Avoid the warning for those cases.
--- arch/x86/include/asm/xen/hypercall.h | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-)
arch/x86/include/asm/xen/hypercall.h +20 -4
@@ -214,6 +214,22 @@ xen_single_call(unsigned int call,return (long )__res; } +static __always_inline void __xen_stac(void) +{ +/* +* Suppress objtool seeing the STAC/CLAC and getting confused about it +* calling random code with AC=1. +*/ + asm volatile(ANNOTATE_IGNORE_ALTERNATIVE + ASM_STAC ::: "memory", "flags"); +} + +static __always_inline void __xen_clac(void) +{ + asm volatile(ANNOTATE_IGNORE_ALTERNATIVE + ASM_CLAC ::: "memory", "flags"); +} + static inline long privcmd_call(unsigned int call, unsigned long a1, unsigned long a2,
@@ -222,9 +238,9 @@ privcmd_call(unsigned int call, {long res; -stac (); +__xen_stac (); res =xen_single_call (call, a1, a2, a3, a4, a5); -clac (); +__xen_clac ();return res; }
@@ -421,9 +437,9 @@ HYPERVISOR_dm_op( domid_t dom, unsigned int nr_bufs, struct xen_dm_op_buf *bufs) { int ret; -stac (); +__xen_stac (); ret =_hypercall3 (int, dm_op, dom, nr_bufs, bufs); -clac (); +__xen_clac ();return ret; }
Peter Zijlstra
maintainerPATCH 9/25 x86/uaccess: Always inline user_access_begin()
In reply to Peter ZijlstraIf GCC out-of-lines it, the STAC and CLAC are in different fuctions and objtool gets upset.
--- arch/x86/include/asm/uaccess.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/uaccess.h +1 -1
@@ -706,7 +706,7 @@ extern struct movsl_mask {* checking before using them, but you have to surround them with the * user_access_begin/end() pair. */ -static __must_check inline bool user_access_begin (const void __user *ptr,size_t len) +static __must_check __always_inline bool user_access_begin(const void __user *ptr, size_t len) {if (unlikely (!access_ok (ptr,len)))return 0 ;
Peter Zijlstra
maintainerPATCH 10/25 x86/uaccess,signal: Fix AC=1 bloat
In reply to Peter ZijlstraOccasionally GCC is less agressive with inlining and the following is observed:
arch/x86/kernel/signal.o: warning: objtool: restore_sigcontext()+0x3cc: call to force_valid_ss.isra.5() with UACCESS enabled arch/x86/kernel/signal.o: warning: objtool: do_signal()+0x384: call to frame_uc_flags.isra.0() with UACCESS enabled
Cure this by moving this code out of the AC=1 region, since it really isn't needed for the user access.
--- arch/x86/kernel/signal.c | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-)
arch/x86/kernel/signal.c +17 -12
@@ -132,16 +132,6 @@ static int restore_sigcontext(struct pt_COPY_SEG_CPL3 (cs);COPY_SEG_CPL3 (ss); -#ifdef CONFIG_X86_64 -/* -* Fix up SS if needed for the benefit of old DOSEMU and -* CRIU. -*/ -if (unlikely (!(uc_flags &UC_STRICT_RESTORE_SS ) && -user_64bit_mode (regs))) -force_valid_ss (regs); -#endif -get_user_ex (tmpflags, &sc->flags); regs->flags = (regs->flags & ~FIX_EFLAGS ) | (tmpflags &FIX_EFLAGS ); regs->orig_ax =-1 ;/* disable syscall checks */
@@ -150,6 +140,15 @@ static int restore_sigcontext(struct pt_ buf = (void __user *)buf_val; }get_user_catch (err); +#ifdef CONFIG_X86_64 +/* +* Fix up SS if needed for the benefit of old DOSEMU and +* CRIU. +*/ +if (unlikely (!(uc_flags &UC_STRICT_RESTORE_SS ) &&user_64bit_mode (regs))) +force_valid_ss (regs); +#endif + err |=fpu__restore_sig (buf,IS_ENABLED (CONFIG_X86_32 ));force_iret ();
@@ -461,6 +460,7 @@ static int __setup_rt_frame(int sig, str {struct rt_sigframe __user *frame;void __user *fp =NULL ; +unsigned long uc_flags;int err =0 ; frame =get_sigframe (&ksig->ka, regs,sizeof (struct rt_sigframe ), &fp);
@@ -473,9 +473,11 @@ static int __setup_rt_frame(int sig, strreturn -EFAULT ; } + uc_flags =frame_uc_flags (regs); + put_user_try {/* Create the ucontext. */ -put_user_ex (frame_uc_flags (regs), &frame->uc.uc_flags); +put_user_ex (uc_flags, &frame->uc.uc_flags);put_user_ex (0 , &frame->uc.uc_link);save_altstack_ex (&frame->uc.uc_stack, regs->sp);
@@ -541,6 +543,7 @@ static int x32_setup_rt_frame(struct ksi {#ifdef CONFIG_X86_X32_ABI struct rt_sigframe_x32 __user *frame; +unsigned long uc_flags;void __user *restorer;int err =0 ;void __user *fpstate =NULL ;
@@ -555,9 +558,11 @@ static int x32_setup_rt_frame(struct ksireturn -EFAULT ; } + uc_flags =frame_uc_flags (regs); + put_user_try {/* Create the ucontext. */ -put_user_ex (frame_uc_flags (regs), &frame->uc.uc_flags); +put_user_ex (uc_flags, &frame->uc.uc_flags);put_user_ex (0 , &frame->uc.uc_link);compat_save_altstack_ex (&frame->uc.uc_stack, regs->sp);put_user_ex (0 , &frame->uc.uc__pad0);
Peter Zijlstra
maintainerPATCH 11/25 x86/uaccess: Introduce user_access_{save,restore}()
In reply to Peter ZijlstraIntroduce common helpers for when we need to safely suspend a uaccess section; for instance to generate a {KA,UB}SAN report.
--- arch/x86/include/asm/smap.h | 20 ++++++++++++++++++++ arch/x86/include/asm/uaccess.h | 3 +++ include/linux/uaccess.h | 2 ++ 3 files changed, 25 insertions(+)
arch/x86/include/asm/smap.h +20 -0
@@ -58,6 +58,23 @@ static __always_inline void stac(void)alternative ("" ,__stringify (__ASM_STAC),X86_FEATURE_SMAP ); } +static __always_inline unsigned long smap_save (void ) +{ +unsigned long flags; + + asm volatile (ALTERNATIVE("", "pushf; pop %0; " __stringify(__ASM_CLAC), + X86_FEATURE_SMAP) + : "=rm" (flags) : : "memory", "cc"); + +return flags; +} + +static __always_inline void smap_restore(unsigned long flags) +{ + asm volatile (ALTERNATIVE("", "push %0; popf", X86_FEATURE_SMAP) + : : "g" (flags) : "memory", "cc"); +} +/* These macros can be used in asm() statements */ #define ASM_CLAC \ ALTERNATIVE("", __stringify(__ASM_CLAC), X86_FEATURE_SMAP)
@@ -69,6 +86,9 @@ static __always_inline void stac(void)static inline void clac (void ) { }static inline void stac (void ) { } +static inline unsigned long smap_save (void ) {return 0 ; } +static inline void smap_restore (unsigned long flags) { } +#define ASM_CLAC #define ASM_STAC
arch/x86/include/asm/uaccess.h +3 -0
@@ -716,6 +716,9 @@ static __must_check __always_inline bool#define user_access_begin (a,b) user_access_begin(a,b)#define user_access_end () __uaccess_end() +#define user_access_save () smap_save() +#define user_access_restore (x) smap_restore(x) +#define unsafe_put_user (x, ptr, label) \ __put_user_size((__typeof__(*(ptr)))(x), (ptr), sizeof(*(ptr)), label)
include/linux/uaccess.h +2 -0
@@ -268,6 +268,8 @@ extern long strncpy_from_unsafe(char *ds#define user_access_end () do { } while (0)#define unsafe_get_user (x, ptr, err) do { if (unlikely(__get_user(x, ptr))) goto err; } while (0)#define unsafe_put_user (x, ptr, err) do { if (unlikely(__put_user(x, ptr))) goto err; } while (0) +static inline unsigned long user_access_save (void ) {return 0UL ; } +static void user_access_restore (unsigned long flags) { }#endif #ifdef CONFIG_HARDENED_USERCOPY
Peter Zijlstra
maintainerPATCH 12/25 x86/smap: Ditch __stringify()
In reply to Peter ZijlstraLinus noticed all users of __ASM_STAC/__ASM_CLAC are with __stringify(). Just make them a string.
--- arch/x86/include/asm/smap.h | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-)
arch/x86/include/asm/smap.h +9 -10
@@ -13,13 +13,12 @@#ifndef _ASM_X86_SMAP_H#define _ASM_X86_SMAP_H -#include <linux/stringify.h> #include <asm/nops.h> #include <asm/cpufeatures.h> /* "Raw" instruction opcodes */ -#define __ASM_CLAC .byte 0x0f,0x01,0xca -#define __ASM_STAC .byte 0x0f,0x01,0xcb +#define __ASM_CLAC ".byte 0x0f,0x01,0xca" +#define __ASM_STAC ".byte 0x0f,0x01,0xcb"#ifdef __ASSEMBLY__
@@ -28,10 +27,10 @@#ifdef CONFIG_X86_SMAP #define ASM_CLAC \ - ALTERNATIVE "", __stringify(__ASM_CLAC), X86_FEATURE_SMAP + ALTERNATIVE "", __ASM_CLAC, X86_FEATURE_SMAP#define ASM_STAC \ - ALTERNATIVE "", __stringify(__ASM_STAC), X86_FEATURE_SMAP + ALTERNATIVE "", __ASM_STAC, X86_FEATURE_SMAP#else /* CONFIG_X86_SMAP */
@@ -49,20 +48,20 @@ static __always_inline void clac(void) {/* Note: a barrier is implicit in alternative() */ -alternative ("" ,__stringify (__ASM_CLAC),X86_FEATURE_SMAP ); +alternative ("" , __ASM_CLAC,X86_FEATURE_SMAP ); } static __always_inline void stac(void) {/* Note: a barrier is implicit in alternative() */ -alternative ("" ,__stringify (__ASM_STAC),X86_FEATURE_SMAP ); +alternative ("" , __ASM_STAC,X86_FEATURE_SMAP ); }static __always_inline unsigned long smap_save (void ) {unsigned long flags; - asm volatile (ALTERNATIVE("", "pushf; pop %0; " __stringify(__ASM_CLAC), + asm volatile (ALTERNATIVE("", "pushf; pop %0; " __ASM_CLAC, X86_FEATURE_SMAP) : "=rm" (flags) : : "memory", "cc");
@@ -77,9 +76,9 @@ static __always_inline void smap_restore/* These macros can be used in asm() statements */ #define ASM_CLAC \ - ALTERNATIVE("", __stringify(__ASM_CLAC), X86_FEATURE_SMAP) + ALTERNATIVE("", __ASM_CLAC, X86_FEATURE_SMAP)#define ASM_STAC \ - ALTERNATIVE("", __stringify(__ASM_STAC), X86_FEATURE_SMAP) + ALTERNATIVE("", __ASM_STAC, X86_FEATURE_SMAP)#else /* CONFIG_X86_SMAP */
Peter Zijlstra
maintainerPATCH 13/25 x86/uaccess,kasan: Fix KASAN vs SMAP
In reply to Peter ZijlstraKASAN inserts extra code for every LOAD/STORE emitted by te compiler. Much of this code is simple and safe to run with AC=1, however the kasan_report() function, called on error, is most certainly not safe to call with AC=1.
Therefore wrap kasan_report() in user_access_{save,restore}; which for x86 SMAP, saves/restores EFLAGS and clears AC before calling the real function.
Also ensure all the functions are without __fentry__ hook. The function tracer is also not safe.
--- mm/kasan/Makefile | 3 +++ mm/kasan/common.c | 10 ++++++++++ mm/kasan/report.c | 3 +-- 3 files changed, 14 insertions(+), 2 deletions(-)
mm/kasan/Makefile +3 -0
@@ -2,11 +2,13 @@ KASAN_SANITIZE := n UBSAN_SANITIZE_common.o := n UBSAN_SANITIZE_generic.o := n +UBSAN_SANITIZE_generic_report.o := n UBSAN_SANITIZE_tags.o := n KCOV_INSTRUMENT := n CFLAGS_REMOVE_common.o = -pg CFLAGS_REMOVE_generic.o = -pg +CFLAGS_REMOVE_generic_report.o = -pg CFLAGS_REMOVE_tags.o = -pg # Function splitter causes unnecessary splits in __asan_load1/__asan_store1
@@ -14,6 +16,7 @@ CFLAGS_REMOVE_tags.o = -pg CFLAGS_common.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) CFLAGS_generic.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) +CFLAGS_generic_report.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) CFLAGS_tags.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) obj-$(CONFIG_KASAN) := common.o init.o report.o
mm/kasan/common.c +10 -0
@@ -34,6 +34,7 @@#include <linux/types.h> #include <linux/vmalloc.h> #include <linux/bug.h> +#include <linux/uaccess.h> #include "kasan.h" #include "../slab.h"
@@ -612,6 +613,15 @@ void kasan_free_shadow(const struct vm_svfree (kasan_mem_to_shadow (vm->addr)); } +extern void __kasan_report (unsigned long addr,size_t size,bool is_write,unsigned long ip); + +void kasan_report (unsigned long addr,size_t size,bool is_write,unsigned long ip) +{ +unsigned long flags =user_access_save (); +__kasan_report (addr, size, is_write, ip); +user_access_restore (flags); +} +#ifdef CONFIG_MEMORY_HOTPLUG static bool shadow_mapped (unsigned long addr) {
mm/kasan/report.c +1 -2
@@ -281,8 +281,7 @@ void kasan_report_invalid_free(void *objend_report (&flags); } -void kasan_report (unsigned long addr,size_t size, -bool is_write,unsigned long ip) +void __kasan_report (unsigned long addr,size_t size,bool is_write,unsigned long ip) {struct kasan_access_info info;void *tagged_addr;
Peter Zijlstra
maintainerPATCH 14/25 x86/uaccess,ubsan: Fix UBSAN vs SMAP
In reply to Peter ZijlstraUBSAN can insert extra code in random locations; including AC=1 sections. Typically this code is not safe and needs wrapping.
So far, only __ubsan_handle_type_mismatch* have been observed in AC=1 sections and therefore only those are annotated.
--- lib/Makefile | 1 + lib/ubsan.c | 4 ++++ 2 files changed, 5 insertions(+)
lib/Makefile +1 -0
@@ -263,6 +263,7 @@ obj-$(CONFIG_UCS2_STRING) += ucs2_string obj-$(CONFIG_UBSAN) += ubsan.o UBSAN_SANITIZE_ubsan.o := n +CFLAGS_ubsan.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) obj-$(CONFIG_SBITMAP) += sbitmap.o
lib/ubsan.c +4 -0
@@ -17,6 +17,7 @@#include <linux/kernel.h> #include <linux/types.h> #include <linux/sched.h> +#include <linux/uaccess.h> #include "ubsan.h"
@@ -313,6 +314,7 @@ static void handle_object_size_mismatch(static void ubsan_type_mismatch_common (struct type_mismatch_data_common *data,unsigned long ptr) { +unsigned long flags =user_access_save ();if (!ptr)handle_null_ptr_deref (data);
@@ -320,6 +322,8 @@ static void ubsan_type_mismatch_common(shandle_misaligned_access (data, ptr); elsehandle_object_size_mismatch (data ,ptr ); + +user_access_restore (flags); } void __ubsan_handle_type_mismatch(struct type_mismatch_data *data,
Peter Zijlstra
maintainerPATCH 15/25 x86/uaccess,ftrace: Fix ftrace_likely_update() vs SMAP
In reply to Peter ZijlstraFor CONFIG_TRACE_BRANCH_PROFILING the likely/unlikely things get overloaded and generate callouts to this code, and thus also when AC=1.
Make it safe.
--- kernel/trace/trace_branch.c | 4 ++++ 1 file changed, 4 insertions(+)
kernel/trace/trace_branch.c +4 -0
@@ -205,6 +205,8 @@ void trace_likely_condition(struct ftracvoid ftrace_likely_update (struct ftrace_likely_data *f,int val,int expect,int is_constant) { +unsigned long flags =user_access_save (); +/* A constant is always correct */ if (is_constant) { f->constant++;
@@ -223,6 +225,8 @@ void ftrace_likely_update(struct ftrace_ f->data.correct++; else f->data.incorrect++; + +user_access_restore (flags); }EXPORT_SYMBOL (ftrace_likely_update);
Peter Zijlstra
maintainerPATCH 16/25 x86/uaccess,kcov: Disable stack protector
In reply to Peter Zijlstrakernel/kcov.o: warning: objtool: write_comp_data()+0x138: call to __stack_chk_fail() with UACCESS enabled kernel/kcov.o: warning: objtool: __sanitizer_cov_trace_pc()+0xd9: call to __stack_chk_fail() with UACCESS enabled
All the other instrumentation (KASAN,UBSAN) also have stack protector disabled.
--- kernel/Makefile | 1 + 1 file changed, 1 insertion(+)
kernel/Makefile +1 -0
@@ -30,6 +30,7 @@ KCOV_INSTRUMENT_extable.o := n # Don't self-instrument. KCOV_INSTRUMENT_kcov.o := n KASAN_SANITIZE_kcov.o := n +CFLAGS_kcov.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) # cond_syscall is currently not LTO compatible CFLAGS_sys_ni.o = $(DISABLE_LTO)
Peter Zijlstra
maintainerPATCH 17/25 objtool: Set insn->func for alternatives
In reply to Peter ZijlstraIn preparation of function attributes, we need each instruction to have a valid link back to its function.
Therefore make sure we set the function association for alternative instruction sequences; they are, after all, still part of the function.
--- tools/objtool/check.c | 1 + 1 file changed, 1 insertion(+)
tools/objtool/check.c +1 -0
@@ -695,6 +695,7 @@ static int handle_group_alt(struct objto last_new_insn = insn; insn->ignore = orig_insn->ignore_alts; + insn->func = orig_insn->func;if (insn->type !=INSN_JUMP_CONDITIONAL && insn->type !=INSN_JUMP_UNCONDITIONAL )
Peter Zijlstra
maintainerPATCH 18/25 objtool: Handle function aliases
In reply to Peter ZijlstraFunction aliases result in different symbols for the same set of instructions; track a canonical symbol so there is a unique point of access.
This again prepares the way for function attributes. And in particular the need for aliases comes from how KASAN uses them.
--- tools/objtool/elf.c | 15 +++++++++++---- tools/objtool/elf.h | 2 +- 2 files changed, 12 insertions(+), 5 deletions(-)
tools/objtool/elf.c +11 -4
@@ -219,7 +219,7 @@ static int read_sections(struct elf *elfstatic int read_symbols (struct elf *elf) {struct section *symtab, *sec; -struct symbol *sym, *pfunc; +struct symbol *sym, *pfunc, *alias;struct list_head *entry, *tmp;int symbols_nr, i;char *coldstr;
@@ -239,6 +239,7 @@ static int read_symbols(struct elf *elf)return -1 ; }memset (sym,0 ,sizeof (*sym)); + alias = sym; sym->idx = i;
@@ -288,11 +289,17 @@ static int read_symbols(struct elf *elf)break ; } -if (sym->offset == s->offset && sym->len >= s->len) { - entry = tmp; -break ; +if (sym->offset == s->offset) { +if (sym->len == s->len && alias == sym) + alias = s; + +if (sym->len >= s->len) { + entry = tmp; +break ; + } } } + sym->alias = alias;list_add (&sym->list, entry);hash_add (sym->sec->symbol_hash, &sym->hash, sym->idx); }
tools/objtool/elf.h +1 -1
@@ -61,7 +61,7 @@ struct symbol {unsigned char bind, type;unsigned long offset;unsigned int len; -struct symbol *pfunc, *cfunc; +struct symbol *pfunc, *cfunc, *alias; };struct rela {
Peter Zijlstra
maintainerPATCH 19/25 objtool: Rewrite add_ignores()
In reply to Peter ZijlstraThe whole add_ignores() thing was wildly weird; rewrite it according to 'modern' ways.
--- tools/objtool/check.c | 51 +++++++++++++++++++------------------------------- tools/objtool/check.h | 1 2 files changed, 20 insertions(+), 32 deletions(-)
tools/objtool/check.c +20 -31
@@ -105,29 +105,6 @@ static struct instruction *next_insn_sam insn = next_insn_same_sec(file, insn))/* -* Check if the function has been manually whitelisted with the -* STACK_FRAME_NON_STANDARD macro, or if it should be automatically whitelisted -* due to its use of a context switching instruction. -*/ -static bool ignore_func (struct objtool_file *file,struct symbol *func) -{ -struct rela *rela; - -/* check for STACK_FRAME_NON_STANDARD */ -if (file->whitelist && file->whitelist->rela) -list_for_each_entry (rela, &file->whitelist->rela->rela_list, list) { -if (rela->sym->type ==STT_SECTION && - rela->sym->sec == func->sec && - rela->addend == func->offset) -return true; -if (rela->sym->type ==STT_FUNC && rela->sym == func) -return true; - } - -return false; -} - -/* * This checks to see if the given function is a "noreturn" function. * * For global functions which are outside the scope of this object file, we
@@ -436,18 +413,31 @@ static void add_ignores(struct objtool_fstruct instruction *insn;struct section *sec;struct symbol *func; +struct rela *rela; -for_each_sec (file, sec) { -list_for_each_entry (func, &sec->symbol_list, list) { -if (func->type !=STT_FUNC ) -continue ; + sec =find_section_by_name (file->elf,".rela.discard.func_stack_frame_non_standard" ); +if (!sec) +return ; -if (!ignore_func (file, func)) +list_for_each_entry (rela, &sec->rela_list, list) { +switch (rela->sym->type) { +case STT_FUNC : + func = rela->sym; +break ; + +case STT_SECTION : + func =find_symbol_by_offset (rela->sym->sec, rela->addend); +if (!func || func->type !=STT_FUNC )continue ; +break ; -func_for_each_insn_all (file, func, insn) - insn->ignore = true; +default : +WARN ("unexpected relocation symbol type in %s: %d" , sec->name, rela->sym->type); +continue ; } + +func_for_each_insn_all (file, func, insn) + insn->ignore = true; } }
@@ -2198,7 +2188,6 @@ int check(const char *_objname, bool orcINIT_LIST_HEAD (&file.insn_list);hash_init (file.insn_hash); - file.whitelist =find_section_by_name (file.elf,".discard.func_stack_frame_non_standard" ); file.c_file =find_section_by_name (file.elf,".comment" ); file.ignore_unreachables = no_unreachable; file.hints = false;
tools/objtool/check.h +0 -1
@@ -60,7 +60,6 @@ struct objtool_file {struct elf *elf;struct list_head insn_list;DECLARE_HASHTABLE (insn_hash,16 ); -struct section *whitelist;bool ignore_unreachables, c_file, hints, rodata; };
Peter Zijlstra
maintainerPATCH 20/25 objtool: Add --backtrace support
In reply to Peter ZijlstraFor when you want to know the path that reached your fail state.
$ ./objtool check --no-fp --backtrace arch/x86/lib/usercopy_64.o arch/x86/lib/usercopy_64.o: warning: objtool: .altinstr_replacement+0x3: UACCESS disable without MEMOPs: __clear_user() arch/x86/lib/usercopy_64.o: warning: objtool: __clear_user()+0x3a: (alt) arch/x86/lib/usercopy_64.o: warning: objtool: __clear_user()+0x2e: (branch) arch/x86/lib/usercopy_64.o: warning: objtool: __clear_user()+0x18: (branch) arch/x86/lib/usercopy_64.o: warning: objtool: .altinstr_replacement+0xffffffffffffffff: (branch) arch/x86/lib/usercopy_64.o: warning: objtool: __clear_user()+0x5: (alt) arch/x86/lib/usercopy_64.o: warning: objtool: __clear_user()+0x0: <=== (func)
0000000000000000 <__clear_user>:
0: e8 00 00 00 00 callq 5 <__clear_user+0x5>
1: R_X86_64_PLT32 __fentry__-0x4 5: 90 nop 6: 90 nop 7: 90 nop 8: 48 89 f0 mov %rsi,%rax b: 48 c1 ee 03 shr $0x3,%rsi f: 83 e0 07 and $0x7,%eax 12: 48 89 f1 mov %rsi,%rcx 15: 48 85 c9 test %rcx,%rcx 18: 74 0f je 29 <__clear_user+0x29> 1a: 48 c7 07 00 00 00 00 movq $0x0,(%rdi) 21: 48 83 c7 08 add $0x8,%rdi 25: ff c9 dec %ecx 27: 75 f1 jne 1a <__clear_user+0x1a> 29: 48 89 c1 mov %rax,%rcx 2c: 85 c9 test %ecx,%ecx 2e: 74 0a je 3a <__clear_user+0x3a> 30: c6 07 00 movb $0x0,(%rdi) 33: 48 ff c7 inc %rdi 36: ff c9 dec %ecx 38: 75 f6 jne 30 <__clear_user+0x30> 3a: 90 nop 3b: 90 nop 3c: 90 nop 3d: 48 89 c8 mov %rcx,%rax 40: c3 retq
--- tools/objtool/builtin-check.c | 3 ++- tools/objtool/builtin.h | 2 +- tools/objtool/check.c | 18 ++++++++++++++---- tools/objtool/warn.h | 8 ++++++++ 4 files changed, 25 insertions(+), 6 deletions(-)
tools/objtool/builtin-check.c +2 -1
@@ -29,7 +29,7 @@#include "builtin.h" #include "check.h" -bool no_fp, no_unreachable, retpoline, module; +bool no_fp, no_unreachable, retpoline, module, backtrace;static const char *const check_usage[] = {"objtool check [<options>] file.o" ,
@@ -41,6 +41,7 @@ const struct option check_options[] = {OPT_BOOLEAN ('u' ,"no-unreachable" , &no_unreachable,"Skip 'unreachable instruction' warnings" ),OPT_BOOLEAN ('r' ,"retpoline" , &retpoline,"Validate retpoline assumptions" ),OPT_BOOLEAN ('m' ,"module" , &module,"Indicates the object will be part of a kernel module" ), +OPT_BOOLEAN ('b' ,"backtrace" , &backtrace,"unwind on error" ), OPT_END(), };
tools/objtool/builtin.h +1 -1
@@ -20,7 +20,7 @@#include <subcmd/parse-options.h> extern const struct option check_options[]; -extern bool no_fp, no_unreachable, retpoline, module; +extern bool no_fp, no_unreachable, retpoline, module, backtrace;extern int cmd_check (int argc,const char **argv);extern int cmd_orc (int argc,const char **argv);
tools/objtool/check.c +14 -4
@@ -1885,8 +1885,11 @@ static int validate_branch(struct objtooif (!insn->ignore_alts) {list_for_each_entry (alt, &insn->alts, list) { ret =validate_branch (file, alt->insn, state); -if (ret) -return 1 ; +if (ret) { +if (backtrace) +BT_FUNC ("(alt)" , insn); +return ret; + } } }
@@ -1933,8 +1936,11 @@ static int validate_branch(struct objtoo insn->jump_dest->func->pfunc == func)) { ret =validate_branch (file, insn->jump_dest, state); -if (ret) -return 1 ; +if (ret) { +if (backtrace) +BT_FUNC ("(branch)" , insn); +return ret; + } } else if (func && has_modified_stack_frame(&state)) { WARN_FUNC("sibling call from callable instruction with modified stack frame",
@@ -2005,6 +2011,8 @@ static int validate_unwind_hints(structfor_each_insn (file, insn) {if (insn->hint && !insn->visited) { ret =validate_branch (file, insn, state); +if (ret && backtrace) +BT_FUNC ("<=== (hint)" , insn); warnings += ret; } }
@@ -2133,6 +2141,8 @@ static int validate_functions(struct objcontinue ; ret =validate_branch (file, insn, state); +if (ret && backtrace) +BT_FUNC ("<=== (func)" , insn); warnings += ret; } }
tools/objtool/warn.h +8 -0
@@ -64,6 +64,14 @@ static inline char *offstr(struct sectiofree (_str); \ }) +#define BT_FUNC (format, insn, ...) \ +({ \ + struct instruction *_insn = (insn); \ + char *_str = offstr(_insn->sec, _insn->offset); \ + WARN(" %s: " format, _str, ##__VA_ARGS__); \ + free(_str); \ +}) +#define WARN_ELF (format, ...) \ WARN(format ": %s", ##__VA_ARGS__, elf_errmsg(-1))
Peter Zijlstra
maintainerPATCH 21/25 objtool: Rewrite alt->skip_orig
In reply to Peter ZijlstraReally skip the original instruction flow, instead of letting it continue with NOPs.
Since the alternative code flow already continues after the original instructions, only the alt-original is skipped.
--- tools/objtool/check.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
tools/objtool/check.c +10 -6
@@ -31,6 +31,7 @@struct alternative {struct list_head list;struct instruction *insn; +bool skip_orig; };const char *objname;
@@ -623,9 +624,6 @@ static int add_call_destinations(struct* conditionally jumps to the _end_ of the entry. We have to modify these * jumps' destinations to point back to .text rather than the end of the * entry in .altinstr_replacement. -* -* 4. It has been requested that we don't validate the !POPCNT feature path -* which is a "very very small percentage of machines". */ static int handle_group_alt(struct objtool_file *file, struct special_alt *special_alt,
@@ -641,9 +639,6 @@ static int handle_group_alt(struct objtoif (insn->offset >= special_alt->orig_off + special_alt->orig_len)break ; -if (special_alt->skip_orig) - insn->type =INSN_NOP ; - insn->alt_group = true; last_orig_insn = insn; }
@@ -808,6 +803,7 @@ static int add_special_section_alts(stru } alt->insn = new_insn; + alt->skip_orig = special_alt->skip_orig;list_add_tail (&alt->list, &orig_insn->alts);list_del (&special_alt->list);
@@ -1883,7 +1879,12 @@ static int validate_branch(struct objtoo insn->visited = true;if (!insn->ignore_alts) { +bool skip_orig = false; +list_for_each_entry (alt, &insn->alts, list) { +if (alt->skip_orig) + skip_orig = true; + ret =validate_branch (file, alt->insn, state);if (ret) {if (backtrace)
@@ -1891,6 +1892,9 @@ static int validate_branch(struct objtooreturn ret; } } + +if (skip_orig) +return 0 ; }switch (insn->type) {
Peter Zijlstra
maintainerPATCH 22/25 objtool: Fix sibling call detection
In reply to Peter ZijlstraIt turned out that we failed to detect some sibling calls; specifically those without relocation records; like:
$ ./objdump-func.sh defconfig-build/mm/kasan/generic.o __asan_loadN 0000 0000000000000840 <__asan_loadN>: 0000 840: 48 8b 0c 24 mov (%rsp),%rcx 0004 844: 31 d2 xor %edx,%edx 0006 846: e9 45 fe ff ff jmpq 690 <check_memory_region>
So extend the cross-function jump to also consider those that are not between known (or newly detected) parent/child functions, as sibling-cals when they jump to the start of the function.
The second part of that condition is to deal with random jumps to the middle of other function, as can be found in arch/x86/lib/copy_user_64.S for example.
This then (with later patches applied) makes the above recognise the sibling call:
mm/kasan/generic.o: warning: objtool: __asan_loadN()+0x6: call to check_memory_region() with UACCESS enabled
Also make sure to set insn->call_dest for sibling calls so we can know who we're calling. This is useful information when printing validation warnings later.
--- tools/objtool/check.c | 86 +++++++++++++++++++++++++++++++------------------- 1 file changed, 55 insertions(+), 31 deletions(-)
tools/objtool/check.c +55 -31
@@ -515,7 +515,8 @@ static int add_jump_destinations(structcontinue ; }else {/* sibling call */ - insn->jump_dest =0 ; + insn->call_dest = rela->sym; + insn->jump_dest =NULL ;continue ; }
@@ -537,25 +538,38 @@ static int add_jump_destinations(struct }/* -* For GCC 8+, create parent/child links for any cold -* subfunctions. This is _mostly_ redundant with a similar -* initialization in read_symbols(). -* -* If a function has aliases, we want the *first* such function -* in the symbol table to be the subfunction's parent. In that -* case we overwrite the initialization done in read_symbols(). -* -* However this code can't completely replace the -* read_symbols() code because this doesn't detect the case -* where the parent function's only reference to a subfunction -* is through a switch table. +* Cross-function jump. */ if (insn->func && insn->jump_dest->func && - insn->func != insn->jump_dest->func && - !strstr (insn->func->name,".cold." ) && -strstr (insn->jump_dest->func->name,".cold." )) { - insn->func->cfunc = insn->jump_dest->func; - insn->jump_dest->func->pfunc = insn->func; + insn->func != insn->jump_dest->func) { + +/* +* For GCC 8+, create parent/child links for any cold +* subfunctions. This is _mostly_ redundant with a +* similar initialization in read_symbols(). +* +* If a function has aliases, we want the *first* such +* function in the symbol table to be the subfunction's +* parent. In that case we overwrite the +* initialization done in read_symbols(). +* +* However this code can't completely replace the +* read_symbols() code because this doesn't detect the +* case where the parent function's only reference to a +* subfunction is through a switch table. +*/ +if (!strstr (insn->func->name,".cold." ) && +strstr (insn->jump_dest->func->name,".cold." )) { + insn->func->cfunc = insn->jump_dest->func; + insn->jump_dest->func->pfunc = insn->func; + + }else if (insn->jump_dest->func->pfunc != insn->func->pfunc && + insn->jump_dest->offset == insn->jump_dest->func->offset) { + +/* sibling class */ + insn->call_dest = insn->jump_dest->func; + insn->jump_dest =NULL ; + } } }
@@ -1785,6 +1799,17 @@ static bool insn_state_match(struct instreturn false; } +static int validate_sibling_call (struct instruction *insn,struct insn_state *state) +{ +if (has_modified_stack_frame (state)) { +WARN_FUNC ("sibling call from callable instruction with modified stack frame" , + insn->sec, insn->offset); +return 1 ; + } + +return 0 ; +} +/* * Follow the branch starting at the given instruction, and recursively follow * any other branches (jumps). Meanwhile, track the frame pointer state at
@@ -1935,9 +1960,14 @@ static int validate_branch(struct objtoocase INSN_JUMP_CONDITIONAL :case INSN_JUMP_UNCONDITIONAL : -if (insn->jump_dest && - (!func || !insn->jump_dest->func || - insn->jump_dest->func->pfunc == func)) { +if (func && !insn->jump_dest) { + ret =validate_sibling_call (insn, &state); +if (ret) +return ret; + + }else if (insn->jump_dest && + (!func || !insn->jump_dest->func || + insn->jump_dest->func->pfunc == func)) { ret =validate_branch (file, insn->jump_dest, state);if (ret) {
@@ -1945,11 +1975,6 @@ static int validate_branch(struct objtooBT_FUNC ("(branch)" , insn);return ret; } - - }else if (func &&has_modified_stack_frame (&state)) { -WARN_FUNC ("sibling call from callable instruction with modified stack frame" , - sec, insn->offset); -return 1 ; }if (insn->type ==INSN_JUMP_UNCONDITIONAL )
@@ -1958,11 +1983,10 @@ static int validate_branch(struct objtoobreak ;case INSN_JUMP_DYNAMIC : -if (func &&list_empty (&insn->alts) && -has_modified_stack_frame (&state)) { -WARN_FUNC ("sibling call from callable instruction with modified stack frame" , - sec, insn->offset); -return 1 ; +if (func &&list_empty (&insn->alts)) { + ret =validate_sibling_call (insn, &state); +if (ret) +return ret; }return 0 ;
Peter Zijlstra
maintainerPATCH 23/25 objtool: Add UACCESS validation
In reply to Peter ZijlstraIt is important that UACCESS regions are as small as possible; furthermore the UACCESS state is not scheduled, so doing anything that might directly call into the scheduler will cause random code to be ran with UACCESS enabled.
Teach objtool too track UACCESS state and warn about any CALL made while UACCESS is enabled. This very much includes the __fentry__() and __preempt_schedule() calls.
Note that exceptions _do_ save/restore the UACCESS state, and therefore they can drive preemption. This also means that all exception handlers must have an otherwise redundant UACCESS disable instruction; therefore ignore this warning for !STT_FUNC code (exception handlers are not normal functions).
--- scripts/Makefile.build | 3 tools/objtool/arch.h | 4 tools/objtool/arch/x86/decode.c | 9 +- tools/objtool/builtin-check.c | 3 tools/objtool/builtin.h | 2 tools/objtool/check.c | 169 +++++++++++++++++++++++++++++++++++++--- tools/objtool/check.h | 2 tools/objtool/elf.h | 1 tools/objtool/special.c | 18 ++++ tools/objtool/special.h | 1 10 files changed, 196 insertions(+), 16 deletions(-)
scripts/Makefile.build +3 -0
@@ -225,6 +225,9 @@ endif ifdef CONFIG_RETPOLINE objtool_args += --retpoline endif +ifdef CONFIG_X86_SMAP + objtool_args += --uaccess +endif # 'OBJECT_FILES_NON_STANDARD := y': skip objtool checking for a directory # 'OBJECT_FILES_NON_STANDARD_foo.o := 'y': skip objtool checking for a file
tools/objtool/arch.h +3 -1
@@ -33,7 +33,9 @@#define INSN_STACK 8#define INSN_BUG 9#define INSN_NOP 10 -#define INSN_OTHER 11 +#define INSN_STAC 11 +#define INSN_CLAC 12 +#define INSN_OTHER 13#define INSN_LAST INSN_OTHERenum op_dest_type {
tools/objtool/arch/x86/decode.c +8 -1
@@ -369,7 +369,14 @@ int arch_decode_instruction(struct elf *case 0x0f : -if (op2 >=0x80 && op2 <=0x8f ) { +if (op2 ==0x01 ) { + +if (modrm ==0xca ) + *type =INSN_CLAC ; +else if (modrm ==0xcb ) + *type =INSN_STAC ; + + }else if (op2 >=0x80 && op2 <=0x8f ) { *type =INSN_JUMP_CONDITIONAL ;
tools/objtool/builtin-check.c +2 -1
@@ -29,7 +29,7 @@#include "builtin.h" #include "check.h" -bool no_fp, no_unreachable, retpoline, module, backtrace; +bool no_fp, no_unreachable, retpoline, module, backtrace, uaccess;static const char *const check_usage[] = {"objtool check [<options>] file.o" ,
@@ -42,6 +42,7 @@ const struct option check_options[] = {OPT_BOOLEAN ('r' ,"retpoline" , &retpoline,"Validate retpoline assumptions" ),OPT_BOOLEAN ('m' ,"module" , &module,"Indicates the object will be part of a kernel module" ),OPT_BOOLEAN ('b' ,"backtrace" , &backtrace,"unwind on error" ), +OPT_BOOLEAN ('a' ,"uaccess" , &uaccess,"enable uaccess checking" ), OPT_END(), };
tools/objtool/builtin.h +1 -1
@@ -20,7 +20,7 @@#include <subcmd/parse-options.h> extern const struct option check_options[]; -extern bool no_fp, no_unreachable, retpoline, module, backtrace; +extern bool no_fp, no_unreachable, retpoline, module, backtrace, uaccess;extern int cmd_check (int argc,const char **argv);extern int cmd_orc (int argc,const char **argv);
tools/objtool/check.c +158 -11
231 lines not on this page · raw message
tools/objtool/check.h +1 -1
@@ -31,7 +31,7 @@ struct insn_state {int stack_size;unsigned char type;bool bp_scratch; -bool drap, end; +bool drap, end, uaccess;int drap_reg, drap_offset;struct cfi_reg vals[CFI_NUM_REGS ]; };
tools/objtool/elf.h +1 -0
@@ -62,6 +62,7 @@ struct symbol {unsigned long offset;unsigned int len;struct symbol *pfunc, *cfunc, *alias; +bool uaccess_safe; };struct rela {
tools/objtool/special.c +18 -0
@@ -23,6 +23,7 @@#include <stdlib.h> #include <string.h> +#include "builtin.h" #include "special.h" #include "warn.h"
@@ -42,6 +43,7 @@#define ALT_NEW_LEN_OFFSET 11#define X86_FEATURE_POPCNT (4*32+23) +#define X86_FEATURE_SMAP (9*32+20)struct special_entry {const char *sec;
@@ -110,6 +112,22 @@ static int get_alt_entry(struct elf *elf*/ if (feature ==X86_FEATURE_POPCNT ) alt->skip_orig = true; + +/* +* If UACCESS validation is enabled; force that alternative; +* otherwise force it the other way. +* +* What we want to avoid is having both the original and the +* alternative code flow at the same time, in that case we can +* find paths that see the STAC but take the NOP instead of +* CLAC and the other way around. +*/ +if (feature ==X86_FEATURE_SMAP ) { +if (uaccess) + alt->skip_orig = true; +else + alt->skip_alt = true; + } } orig_rela =find_rela_by_dest (sec, offset + entry->orig);
tools/objtool/special.h +1 -0
@@ -26,6 +26,7 @@ struct special_alt {bool group;bool skip_orig; +bool skip_alt;bool jump_or_nop;struct section *orig_sec;
Re: [PATCH 23/25] objtool: Add UACCESS validation
tools/objtool/check.c static void add_ignores(struct objtool_f · line 446
Quoted 6 lines · Peter Zijlstra
+++
b/tools/objtool/check.ctools/objtool/check.c +1
-0@@ -443,6 +443,82 @@ static void add_ignores(struct objtool_f }/* +* This is a whitelist of functions that is allowed to be called with AC set.
s/is/are/
Signature · 1 line
Josh
Peter Zijlstra
maintainerRe: [PATCH 23/25] objtool: Add UACCESS validation
In reply to Peter ZijlstraQuoted 20 lines · Peter Zijlstra
tools/objtool/check.c +20
-0+static const char *uaccess_safe_builtin[] = { +/* KASAN */ +"kasan_report" , +"check_memory_region" , +/* KASAN out-of-line */ +"__asan_loadN_noabort" , +"__asan_load1_noabort" , +"__asan_load2_noabort" , +"__asan_load4_noabort" , +"__asan_load8_noabort" , +"__asan_load16_noabort" , +"__asan_storeN_noabort" , +"__asan_store1_noabort" , +"__asan_store2_noabort" , +"__asan_store4_noabort" , +"__asan_store8_noabort" , +"__asan_store16_noabort" , +/* KASAN in-line */ +"__asan_report_load_n_noabort" , +"__asan_report_load1_noabort" ,
tools/objtool/check.c static void add_ignores(struct objtool_f · line 481
Quoted 10 lines
tools/objtool/check.c +10
-0+"__asan_report_load2_noabort" , +"__asan_report_load4_noabort" , +"__asan_report_load8_noabort" , +"__asan_report_load16_noabort" , +"__asan_report_store_n_noabort" , +"__asan_report_store1_noabort" , +"__asan_report_store2_noabort" , +"__asan_report_store4_noabort" , +"__asan_report_store8_noabort" , + "__asan_report_store16_noabort",
So I was building the kernel with GCC-4.9 (for another issue) and that got me:
arch/x86/kernel/signal.o: warning: objtool: restore_sigcontext()+0x69: call to __asan_store8() with UACCESS enabled arch/x86/kernel/signal.o: warning: objtool: setup_sigcontext()+0x3d: call to __asan_load8() with UACCESS enabled arch/x86/kernel/signal.o: warning: objtool: __setup_rt_frame()+0x308: call to __asan_load8() with UACCESS enabled lib/strncpy_from_user.o: warning: objtool: strncpy_from_user()+0x33d: call to __asan_store1() with UACCESS enabled lib/ubsan.o: warning: objtool: __ubsan_handle_type_mismatch_v1()+0x19: call to __asan_load8() with UACCESS enabled lib/ubsan.o: warning: objtool: __ubsan_handle_type_mismatch()+0x19: call to __asan_load8() with UACCESS enabled
Can any of the KASAN folks enlighten me on this? Should I also whitelist those symbols, and do I understand things correct that the difference between * and *_noabort is absolutely nothing?
Peter Zijlstra
maintainerPATCH 24/25 objtool: uaccess PUSHF/POPF support
In reply to Peter ZijlstraAdd PUSHF / POPF state.uaccess restore logic. This makes user_access_save() / user_access_restore() 'work' with objtool.
XXX: should be merged with the previous patch such that KASAN doesn't explode in between. Split for review.
--- tools/objtool/arch.h | 2 ++ tools/objtool/arch/x86/decode.c | 4 ++-- tools/objtool/check.c | 30 ++++++++++++++++++++++++++---- tools/objtool/check.h | 1 + 4 files changed, 31 insertions(+), 6 deletions(-)
tools/objtool/arch.h +2 -0
@@ -43,6 +43,7 @@ enum op_dest_type {OP_DEST_REG_INDIRECT ,OP_DEST_MEM ,OP_DEST_PUSH , +OP_DEST_PUSHF , OP_DEST_LEAVE, };
@@ -57,6 +58,7 @@ enum op_src_type {OP_SRC_REG_INDIRECT ,OP_SRC_CONST ,OP_SRC_POP , +OP_SRC_POPF ,OP_SRC_ADD , OP_SRC_AND, };
tools/objtool/arch/x86/decode.c +2 -2
@@ -357,13 +357,13 @@ int arch_decode_instruction(struct elf */* pushf */ *type =INSN_STACK ; op->src.type =OP_SRC_CONST ; - op->dest.type =OP_DEST_PUSH ; + op->dest.type =OP_DEST_PUSHF ;break ;case 0x9d :/* popf */ *type =INSN_STACK ; - op->src.type =OP_SRC_POP ; + op->src.type =OP_SRC_POPF ; op->dest.type =OP_DEST_MEM ;break ;
tools/objtool/check.c +26 -4
@@ -1392,11 +1392,11 @@ static int update_insn_state_regs(structreturn 0 ;/* push */ -if (op->dest.type ==OP_DEST_PUSH ) +if (op->dest.type ==OP_DEST_PUSH || op->dest.type ==OP_DEST_PUSHF ) cfa->offset +=8 ;/* pop */ -if (op->src.type ==OP_SRC_POP ) +if (op->src.type ==OP_SRC_POP || op->src.type ==OP_SRC_POPF ) cfa->offset -=8 ;/* add immediate to sp */
@@ -1653,6 +1653,7 @@ static int update_insn_state(struct instbreak ;case OP_SRC_POP : +case OP_SRC_POPF :if (!state->drap && op->dest.type ==OP_DEST_REG && op->dest.reg == cfa->base) {
@@ -1717,6 +1718,7 @@ static int update_insn_state(struct instbreak ;case OP_DEST_PUSH : +case OP_DEST_PUSHF : state->stack_size +=8 ;if (cfa->base ==CFI_SP ) cfa->offset +=8 ;
@@ -1807,7 +1809,7 @@ static int update_insn_state(struct instbreak ;case OP_DEST_MEM : -if (op->src.type !=OP_SRC_POP ) { +if (op->src.type !=OP_SRC_POP && op->src.type !=OP_SRC_POPF ) {WARN_FUNC ("unknown stack-related memory operation" , insn->sec, insn->offset);return -1 ;
@@ -2109,6 +2111,26 @@ static int validate_branch(struct objtooif (update_insn_state (insn, &state))return 1 ; +if (insn->stack_op.dest.type ==OP_DEST_PUSHF ) { +if (!state.uaccess_stack) { + state.uaccess_stack =1 ; + }else if (state.uaccess_stack >>31 ) { +WARN_FUNC ("PUSHF stack exhausted" , sec, insn->offset); +return 1 ; + } + state.uaccess_stack <<=1 ; + state.uaccess_stack |= state.uaccess; + } + +if (insn->stack_op.src.type ==OP_SRC_POPF ) { +if (state.uaccess_stack) { + state.uaccess = state.uaccess_stack &1 ; + state.uaccess_stack >>=1 ; +if (state.uaccess_stack ==1 ) + state.uaccess_stack =0 ; + } + } +break ;case INSN_STAC :
@@ -2126,7 +2148,7 @@ static int validate_branch(struct objtooreturn 1 ; } -if (func_uaccess_safe (func)) { +if (func_uaccess_safe (func) && !state.uaccess_stack) {WARN_FUNC ("UACCESS-safe disables UACCESS" , sec, insn->offset);return 1 ; }
tools/objtool/check.h +1 -0
@@ -32,6 +32,7 @@ struct insn_state {unsigned char type;bool bp_scratch;bool drap, end, uaccess; +unsigned int uaccess_stack;int drap_reg, drap_offset;struct cfi_reg vals[CFI_NUM_REGS ]; };
Peter Zijlstra
maintainerPATCH 25/25 objtool: Add Direction Flag validation
In reply to Peter ZijlstraHaving DF escape is BAD(tm).
Linus; you suggested this one, but since DF really is only used from ASM and the failure case is fairly obvious, do we really need this? OTOH the patch is fairly small and simple.
--- tools/objtool/arch.h | 4 +++- tools/objtool/arch/x86/decode.c | 8 ++++++++ tools/objtool/check.c | 25 +++++++++++++++++++++++++ tools/objtool/check.h | 2 +- 4 files changed, 37 insertions(+), 2 deletions(-)
tools/objtool/arch.h +3 -1
@@ -35,7 +35,9 @@#define INSN_NOP 10#define INSN_STAC 11#define INSN_CLAC 12 -#define INSN_OTHER 13 +#define INSN_STD 13 +#define INSN_CLD 14 +#define INSN_OTHER 15#define INSN_LAST INSN_OTHERenum op_dest_type {
tools/objtool/arch/x86/decode.c +8 -0
@@ -451,6 +451,14 @@ int arch_decode_instruction(struct elf * *type =INSN_CALL ;break ; +case 0xfc : + *type =INSN_CLD ; +break ; + +case 0xfd : + *type =INSN_STD ; +break ; +case 0xff :if (modrm_reg ==2 || modrm_reg ==3 )
tools/objtool/check.c +25 -0
@@ -1903,6 +1903,12 @@ static int validate_call(struct instructreturn 1 ; } +if (state->df) { +WARN_FUNC ("call to %s() with DF set" , + insn->sec, insn->offset,insn_dest_name (insn)); +return 1 ; + } +return 0 ; }
@@ -2044,6 +2050,11 @@ static int validate_branch(struct objtooreturn 1 ; } +if (state.df) { +WARN_FUNC ("return with DF set" , sec, insn->offset); +return 1 ; + } +if (func &&has_modified_stack_frame (&state)) {WARN_FUNC ("return with modified stack frame" , sec, insn->offset);
@@ -2172,6 +2183,20 @@ static int validate_branch(struct objtoo state.uaccess = false;break ; +case INSN_STD : +if (state.df) +WARN_FUNC ("recursive STD" , sec, insn->offset); + + state.df = true; +break ; + +case INSN_CLD : +if (!state.df && insn->func) +WARN_FUNC ("redundant CLD" , sec, insn->offset); + + state.df = false; +break ; +default :break ; }
tools/objtool/check.h +1 -1
@@ -31,7 +31,7 @@ struct insn_state {int stack_size;unsigned char type;bool bp_scratch; -bool drap, end, uaccess; +bool drap, end, uaccess, df;unsigned int uaccess_stack;int drap_reg, drap_offset;struct cfi_reg vals[CFI_NUM_REGS ];
Quoted 18 lines · Peter Zijlstra
Teach objtool to validate the UACCESS (SMAP, PAN) rules which are currently unenforced and (therefore obviously) violated.
UACCESS sections should be small; we want to limit the amount of code that can touch userspace. Furthermore, UACCESS state isn't scheduled, this means that anything that directly calls into the scheduler will result in random code running with UACCESS enabled and possibly getting back into the UACCESS region with UACCESS disabled and causing faults.
Forbid any CALL/RET while UACCESS is enabled; but provide a few exceptions.
This builds x86_64-allmodconfig and lots of x86_64-randconfig clean.
Changes since -v3:
- removed a bunch of functions from the UACCESS-safe list due to the removal of
CONFIG_KASAN_EXTRA=y.
Quoted 8 lines
- hopefully addressed all the feedback from Josh
- realized objtool doesn't cover
x86_32- some added additional annotations/fixes: kcov, signal
- retains the DF check for now, Linus, do you (still) think it is worth doing that DF check?
I'm still not crazy about the DF thing, but otherwise everything looks great.
For the objtool bits:
Signature · 1 line
Josh
Peter Zijlstra
maintainerQuoted 3 lines · Josh Poimboeuf
For the objtool bits:
Acked-by Josh Poimboeuf <[email protected]>
Thanks!
I'll go fold the PUSHF/POPF bits into the AC patch then.
Peter Zijlstra
maintainer[PATCH 26/25] sched/x86_64: Don't save flags on context switch
In reply to Peter ZijlstraSubject: sched/x86_64: Don't save flags on context switch
From: Peter Zijlstra <[email protected]>
Date: Tue Mar 19 11:35:46 CET 2019
Now that we have objtool validating AC=1 state for all x86_64 code, we can once again guarantee clean flags on schedule.
--- arch/x86/entry/entry_64.S | 2 -- arch/x86/include/asm/switch_to.h | 2 +- arch/x86/kernel/process_64.c | 7 ------- 3 files changed, 1 insertion(+), 10 deletions(-)
arch/x86/entry/entry_64.S +0 -2
@@ -291,7 +291,6 @@ ENTRY(__switch_to_asm) pushq %r13 pushq %r14 pushq %r15 - pushfq /* switch stack */ movq %rsp, TASK_threadsp(%rdi)
@@ -314,7 +313,6 @@ ENTRY(__switch_to_asm) #endif /* restore callee-saved registers */ - popfq popq %r15 popq %r14 popq %r13
arch/x86/include/asm/switch_to.h +1 -1
@@ -40,13 +40,13 @@ asmlinkage void ret_from_fork(void);* order of the fields must match the code in __switch_to_asm(). */ struct inactive_task_frame { -unsigned long flags;#ifdef CONFIG_X86_64 unsigned long r15;unsigned long r14;unsigned long r13;unsigned long r12;#else +unsigned long flags;unsigned long si;unsigned long di;#endif
arch/x86/kernel/process_64.c +0 -7
@@ -393,13 +393,6 @@ int copy_thread_tls(unsigned long clone_ fork_frame = container_of(childregs, struct fork_frame, regs); frame = &fork_frame->frame; -/* -* For a new task use the RESET flags value since there is no before. -* All the status flags are zero; DF and all the system flags must also -* be 0, specifically IF must be 0 because we context switch to the new -* task with interrupts disabled. -*/ - frame->flags =X86_EFLAGS_FIXED ; frame->bp =0 ; frame->ret_addr = (unsigned long ) ret_from_fork; p->thread.sp = (unsigned long ) fork_frame;