WARN_ON: userstacktrace on irq events
Subsystems (2)
Juri reported this from the -rt kernel, but I can easily trigger it in mainline. By simply doing:
# cd /sys/kernel/tracing # echo 1 > options/userstacktrace # echo 1 > events/irq/enable
With CONFIG_DEBUG_ATOMIC_SLEEP enabled, we get a WARNING splat of:
WARNING: CPU: 1 PID: 1688 at arch/x86/kernel/stacktrace.c:180 save_stack_trace_user+0x12b/0x140 Modules linked in: iptable_mangle xt_CHECKSUM tun CPU: 1 PID: 1688 Comm: sshd Not tainted 5.1.0-rc3-test+ #14 Hardware name: Hewlett-Packard HP Compaq Pro 6300 SFF/339A, BIOS K01 v03.03 07/14/2016 RIP: 0010:save_stack_trace_user+0x12b/0x140 Code: 44 8d 58 01 44 89 1f 48 89 34 c2 8b 07 89 c2 8b 77 04 49 39 ca 74 ad 4c 89 d1 39 f0 0f 82 37 ff ff ff c3 89 c2 8b 77 04 eb 9a <0f> 0b e9 3b ff ff ff 31 f6 e9 70 ff ff ff 8b 17 8b 77 04 eb 85 0f RSP: 0000:ffff8880d0687e40 EFLAGS: 00010006 RAX: 0000000000000001 RBX: 1ffff1101a0d0fc9 RCX: 00007fff28a44f12 RDX: 0000000000010000 RSI: 00007faa5f81f098 RDI: ffff8880d0687e68 RBP: ffff8880cc058368 R08: ffff888085ae7f58 R09: 0000000000000000 R10: ffffed100f207c97 R11: ffff88807903e4bb R12: ffff88807903e468 R13: ffff88807903e46c R14: 0000000000000698 R15: ffff88807903e46c FS: 00007faa5f43cdc0(0000) GS:ffff8880d0680000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000081 CR3: 0000000085296004 CR4: 00000000001606e0 Call Trace: <IRQ> ftrace_trace_userstack+0x17e/0x1f0 ? __trace_stack+0x80/0x80 trace_event_buffer_commit+0xd0/0x300 ? trace_event_buffer_reserve+0x107/0x130 trace_event_raw_event_x86_irq_vector+0xb8/0x120 ? __bpf_trace_vector_setup+0x10/0x10 ? irqtime_account_irq+0x46/0xe0 ? __bpf_trace_vector_setup+0x10/0x10 smp_apic_timer_interrupt+0x208/0x270 apic_timer_interrupt+0xf/0x20 </IRQ> RIP: 0033:0x7faa5f81f098 Code: 89 d1 40 0f b6 c6 48 89 fa f3 aa 48 89 d0 c3 0f 1f 00 f3 0f 1e fa 48 39 d1 0f 82 43 80 06 00 0f 1f 00 f3 0f 1e fa 66 0f 6e c6 <48> 89 f8 66 0f 60 c0 66 0f 61 c0 66 0f 70 c0 00 48 83 fa 10 0f 82 RSP: 002b:00007fff28a44ef8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 RAX: 0000000000000000 RBX: 00005649f9d12830 RCX: 0000000000000001 RDX: 0000000000000100 RSI: 0000000000000000 RDI: 00005649f9d36240 RBP: 00007fff28a44f12 R08: 0000000000000000 R09: 0000000000000100 R10: 0000000000000008 R11: 00007fff28a40f90 R12: 0000000000000000 R13: 00007fff28a44f58 R14: 00005649f9d12260 R15: 00005649f9d0ab40 ---[ end trace cd684bbd8c344b2a ]--- hrtimer: interrupt took 3268750 ns ------------[ cut here ]------------
This is simply caused by the irq trace events doing a user stack trace:
ftrace_trace_userstack {
save_stack_trace_user {
__save_stack_trace_user {
copy_stack_frame {
access_ok {
WARN_ON_IN_IRQ()BOOM! Warn on.
Can we make that access_ok() call in the copy_stack_frame not trigger the warning just if we are in an interrupt?
-- Steve
Quoted 15 lines · Steven Rostedt
Juri reported this from the -rt kernel, but I can easily trigger it in mainline. By simply doing:
# cd /sys/kernel/tracing # echo 1 > options/userstacktrace # echo 1 > events/irq/enable This is simply caused by the irq trace events doing a user stack trace:
ftrace_trace_userstack{save_stack_trace_user { __save_stack_trace_user { copy_stack_frame { access_ok { WARN_ON_IN_IRQ()
Quoted 4 lines
BOOM! Warn on.
Can we make that
access_ok()call in thecopy_stack_framenot trigger the warning just if we are in an interrupt?
You really want to have access_ok_atomic() or such which does not have the WARN and use that in copy_stack_frame(). That's fine here because the actual copy is inside a pagefault disabled region.
Thanks,
tglx
Quoted 8 lines · On Fri, 5 Apr 2019 10:12:27 +0200 (CEST) Thomas Gleixner <[email protected]>…
Quoted 4 lines
BOOM! Warn on.
Can we make that
access_ok()call in thecopy_stack_framenot trigger the warning just if we are in an interrupt?You really want to have
access_ok_atomic()or such which does not have the WARN and use that incopy_stack_frame(). That's fine here because the actual copy is inside a pagefault disabled region.
I was thinking the same.
Masami, did you post patches to do something like this? "access_ok_inatomic()" or something?
-- Steve
Quoted 8 lines · On Fri, 5 Apr 2019 09:32:09 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 5 lines · On Fri, 5 Apr 2019 10:12:27 +0200 (CEST) Thomas Gleixner <[email protected]>…
Quoted 4 lines
BOOM! Warn on.
Can we make that
access_ok()call in thecopy_stack_framenot trigger the warning just if we are in an interrupt?
Quoted 8 lines
Quoted 3 lines
You really want to have
access_ok_atomic()or such which does not have the WARN and use that incopy_stack_frame(). That's fine here because the actual copy is inside a pagefault disabled region.I was thinking the same.
Masami, did you post patches to do something like this? "
access_ok_inatomic()" or something?
Yeah, last month I sent
"x86/uaccess: Allow access_ok() in irq context if pagefault_disabled"
If you correctly disables the pagefault, access_ok() shouldn't warn it. Ah, I see.
copy_stack_frame(const void __user *fp, struct stack_frame_user *frame)
{
int ret;
if (!access_ok(fp, sizeof(*frame))) <== this is out of pagefault_disable()!
return 0;
ret = 1;
pagefault_disable();
if (__copy_from_user_inatomic(frame, fp, sizeof(*frame)))
ret = 0;
pagefault_enable();
return ret;
}How is below patch?
---
arch/x86/kernel/stacktrace.c +3 -6
@@ -98,14 +98,11 @@ struct stack_frame_user {static int copy_stack_frame (const void __user *fp,struct stack_frame_user *frame) { -int ret; +int ret =1 ; -if (!access_ok (fp,sizeof (*frame))) -return 0 ; - - ret =1 ;pagefault_disable (); -if (__copy_from_user_inatomic (frame, fp,sizeof (*frame))) +if (!access_ok (fp,sizeof (*frame)) || +__copy_from_user_inatomic (frame, fp,sizeof (*frame))) ret =0 ;pagefault_enable ();
Signature · 1 line
Masami Hiramatsu <[email protected]>
cover x86/stacktrace: Fix userstacktrace access_ok() WARNING in irq events
Hello
I also hit the same WARNING previously repored by Juri.
Hiramatsu san's patch looks good to me but I found that perf and oprofile code do the similar thing by just directly calling __range_not_ok().
perf: perf_callchain_user()@arch/x86/events/core.c
oprofile: dump_user_backtrace()@arch/x86/oprofile/backtrace.c
So for simplicity, I wrote a patch to fix the warning as other codes do.
Ideally, we should merge these similar stacktrace codes(perf, ftrace, oprofile) into one, but this time I made the minimum fix.
Eiichi Tsukata (1):
x86/stacktrace: Fix userstacktrace access_ok() WARNING in irq events
arch/x86/kernel/stacktrace.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Signature · 1 line
2.21.0
PATCH 1/1 x86/stacktrace: Fix userstacktrace access_ok() WARNING in irq events
When arch_stack_walk_user() is called from irq context, access_ok() can trigger the following WARNING if compiled with CONFIG_DEBUG_ATOMIC_SLEEP=y.
Reproducer:
// CONFIG_DEBUG_ATOMIC_SLEEP=y
# cd /sys/kernel/debug/tracing # echo 1 > options/userstacktrace # echo 1 > events/irq/irq_handler_entry/enable
WARNING:
WARNING: CPU: 0 PID: 2649 at arch/x86/kernel/stacktrace.c:103 arch_stack_walk_user+0x6e/0xf6 Modules linked in: CPU: 0 PID: 2649 Comm: bash Not tainted 5.3.0-rc1+ #99 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-2.fc30 04/01/2014 RIP: 0010:arch_stack_walk_user+0x6e/0xf6 Code: 00 48 89 45 c8 48 89 da 49 89 c7 49 89 c5 65 8b 05 5f 3f 3c 72 a9 00 01 1f 00 74 10 48 8b 45 c8 8b 80 58 16 00 00 85 c0 75 02 <0f> 0b 49 8b 85 18 17 00 00 48 83 e8 10 48 39 c2 77 32 41 83 85 58 RSP: 0018:ffff888068a09bc0 EFLAGS: 00010046 RAX: 0000000000000000 RBX: 00005567f28dc6a0 RCX: ffffffff8ddf6b71 RDX: 00005567f28dc6a0 RSI: 00007f3fcf7d20f8 RDI: ffff888068475048 RBP: ffff888068a09bf8 R08: ffffffff8ddf6b4b R09: ffffed100ced26f1 R10: ffffed100ced26f0 R11: ffff888067693787 R12: ffff88807c1bff58 R13: ffff888067693780 R14: ffff888068a09c28 R15: ffff888067693780 FS: 00007f3fcf6e3740(0000) GS:ffff888068a00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000055cd64646630 CR3: 000000005e230004 CR4: 0000000000160ef0 Call Trace: <IRQ> ? stack_trace_save+0xc0/0xc0 stack_trace_save_user+0x10a/0x16d ? stack_trace_save_tsk_reliable+0x1c0/0x1c0 ? __kasan_check_read+0x11/0x20 trace_buffer_unlock_commit_regs+0x185/0x240 trace_event_buffer_commit+0xec/0x330 trace_event_raw_event_irq_handler_entry+0x159/0x1e0 ? perf_trace_softirq+0x250/0x250 ? check_chain_key+0x1da/0x2d0 ? perf_trace_softirq+0x250/0x250 __handle_irq_event_percpu+0x22d/0x440 handle_irq_event_percpu+0x70/0x100 ? __handle_irq_event_percpu+0x440/0x440 ? __kasan_check_read+0x11/0x20 ? preempt_count_sub+0x1a/0x120 handle_irq_event+0x5a/0x8b handle_edge_irq+0x12f/0x3f0 handle_irq+0x34/0x40 do_IRQ+0xa6/0x1f0 common_interrupt+0xf/0xf </IRQ>
Fix it by calling __range_not_ok() directly instead of access_ok() as copy_from_user_nmi() does. This is fine here because the actual copy is inside a pagefault disabled region.
--- arch/x86/kernel/stacktrace.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/stacktrace.c +1 -1
@@ -100,7 +100,7 @@ copy_stack_frame(const void __user *fp, struct stack_frame_user *frame) {int ret; -if (!access_ok (fp,sizeof (*frame))) +if (__range_not_ok (fp,sizeof (*frame),TASK_SIZE ))return 0 ; ret =1 ;
Signature · 1 line
2.21.0
tip-bot for Eiichi Tsukata
merge bot[tip:x86/urgent] x86/stacktrace: Prevent access_ok() warnings in arch_stack_walk_user()
x86/stacktrace: Prevent access_ok() warnings in arch_stack_walk_user()
Commit-ID: 2af7c85714d8cafadf925d55441458eae312cd6b Gitweb: https://git.kernel.org/tip/2af7c85714d8cafadf925d55441458eae312cd6b Author: Eiichi Tsukata <[email protected]> AuthorDate: Mon, 22 Jul 2019 17:32:16 +0900 Committer: Thomas Gleixner <[email protected]> CommitDate: Mon, 22 Jul 2019 10:42:36 +0200
x86/stacktrace: Prevent access_ok() warnings in arch_stack_walk_user()
When arch_stack_walk_user() is called from atomic contexts, access_ok() can trigger the following warning if compiled with CONFIG_DEBUG_ATOMIC_SLEEP=y.
Reproducer:
// CONFIG_DEBUG_ATOMIC_SLEEP=y
# cd /sys/kernel/debug/tracing # echo 1 > options/userstacktrace # echo 1 > events/irq/irq_handler_entry/enable
WARNING: CPU: 0 PID: 2649 at arch/x86/kernel/stacktrace.c:103 arch_stack_walk_user+0x6e/0xf6 CPU: 0 PID: 2649 Comm: bash Not tainted 5.3.0-rc1+ #99 RIP: 0010:arch_stack_walk_user+0x6e/0xf6 Call Trace: <IRQ> stack_trace_save_user+0x10a/0x16d trace_buffer_unlock_commit_regs+0x185/0x240 trace_event_buffer_commit+0xec/0x330 trace_event_raw_event_irq_handler_entry+0x159/0x1e0 __handle_irq_event_percpu+0x22d/0x440 handle_irq_event_percpu+0x70/0x100 handle_irq_event+0x5a/0x8b handle_edge_irq+0x12f/0x3f0 handle_irq+0x34/0x40 do_IRQ+0xa6/0x1f0 common_interrupt+0xf/0xf </IRQ>
Fix it by calling __range_not_ok() directly instead of access_ok() as copy_from_user_nmi() does. This is fine here because the actual copy is inside a pagefault disabled region.
--- arch/x86/kernel/stacktrace.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/stacktrace.c +1 -1
@@ -100,7 +100,7 @@ copy_stack_frame(const void __user *fp, struct stack_frame_user *frame) {int ret; -if (!access_ok (fp,sizeof (*frame))) +if (__range_not_ok (fp,sizeof (*frame),TASK_SIZE ))return 0 ; ret =1 ;