suspicious RCU usage (perf)
Another day, another rcu backtrace..
This says rc6, but it's pretty darn close to rc7, I think it was running a build from Friday.
[260431.854524] ===============================
[260431.855485] [ INFO: suspicious RCU usage. ]
[260431.856437] 3.11.0-rc6+ #12 Not tainted
[260431.857377] -------------------------------
[260431.858318] include/linux/rcupdate.h:779 rcu_read_lock() used illegally while idle!
[260431.859278] other info that might help us debug this:
[260431.862144] RCU used illegally from idle CPU!
rcu_scheduler_active = 1, debug_locks = 0
[260431.864990] RCU used illegally from extended quiescent state!
[260431.865949] 1 lock held by trinity-kid3/15917:
[260431.866901] #0: (rcu_read_lock){.+.+..}, at: [<ffffffff81146d50>] __perf_event_overflow+0x100/0x320
[260431.867896] stack backtrace:
[260431.869815] CPU: 0 PID: 15917 Comm: trinity-kid3 Not tainted 3.11.0-rc6+ #12
[260431.871885] 0000000000000000 ffff8801095d3ad0 ffffffff816f9c6f ffff88023ce86950
[260431.872912] ffff8801095d3b00 ffffffff810be6a7 ffff880079695cd0 0000000000000000
[260431.873910] ffff8801095d3c00 0000000000000000 ffff8801095d3b80 ffffffff81146ef4
[260431.874909] Call Trace:
[260431.875883] [<ffffffff816f9c6f>] dump_stack+0x54/0x74
[260431.876869] [<ffffffff810be6a7>] lockdep_rcu_suspicious+0xe7/0x120
[260431.877852] [<ffffffff81146ef4>] __perf_event_overflow+0x2a4/0x320
[260431.878834] [<ffffffff81146d50>] ? __perf_event_overflow+0x100/0x320
[260431.879811] [<ffffffff81146e0c>] ? __perf_event_overflow+0x1bc/0x320
[260431.880784] [<ffffffff8170ce00>] ? ftrace_call+0x5/0x2f
[260431.881758] [<ffffffff81147101>] perf_swevent_overflow+0x51/0xe0
[260431.882728] [<ffffffff811471ef>] perf_swevent_event+0x5f/0x90
[260431.883685] [<ffffffff81147329>] perf_tp_event+0x109/0x4f0
[260431.884637] [<ffffffff81147542>] ? perf_tp_event+0x322/0x4f0
[260431.885586] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420
[260431.886539] [<ffffffff8107a320>] ? task_work_run+0xe0/0xe0
[260431.887486] [<ffffffff81135910>] perf_ftrace_function_call+0xc0/0xd0
[260431.888417] [<ffffffff81114d37>] ? ftrace_ops_control_func+0xe7/0x110
[260431.889353] [<ffffffff8107a320>] ? task_work_run+0xe0/0xe0
[260431.890281] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420
[260431.891191] [<ffffffff81114d37>] ftrace_ops_control_func+0xe7/0x110
[260431.892088] [<ffffffff8170ce00>] ftrace_call+0x5/0x2f
[260431.892975] [<ffffffff81114cbb>] ? ftrace_ops_control_func+0x6b/0x110
[260431.893862] [<ffffffff8170ce00>] ? ftrace_call+0x5/0x2f
[260431.894745] [<ffffffff81097def>] ? local_clock+0x3f/0x50
[260431.895631] [<ffffffff8107a325>] ? debug_lockdep_rcu_enabled+0x5/0x40
[260431.896516] [<ffffffff8107a325>] ? debug_lockdep_rcu_enabled+0x5/0x40
[260431.897387] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420
[260431.898264] [<ffffffff81107e24>] rcu_eqs_enter+0x64/0xa0
[260431.899133] [<ffffffff8110c673>] rcu_user_enter+0x13/0x20
[260431.899999] [<ffffffff8114e67a>] user_enter+0x6a/0xd0
[260431.900859] [<ffffffff81010208>] syscall_trace_leave+0x78/0x150
[260431.901712] [<ffffffff8170d4af>] int_check_syscall_exit_work+0x34/0x3dQuoted 3 lines · Dave Jones
Another day, another rcu backtrace..
This says rc6, but it's pretty darn close to rc7, I think it was running a build from Friday.
Could you please send your .config? Also, were you running ftrace, perf, RCU event tracing, or what?
Looks like you are running ftrace, but I though Steven had set that up so that it could be called from an extended quiescent state.
Thanx, Paul
Quoted 47 lines
[260431.854524] ===============================
[260431.855485] [ INFO: suspicious RCU usage. ]
[260431.856437] 3.11.0-rc6+ #12 Not tainted
[260431.857377] -------------------------------
[260431.858318] include/linux/rcupdate.h:779 rcu_read_lock() used illegally while idle!
[260431.859278] other info that might help us debug this:
[260431.862144] RCU used illegally from idle CPU!
rcu_scheduler_active = 1, debug_locks = 0
[260431.864990] RCU used illegally from extended quiescent state!
[260431.865949] 1 lock held by trinity-kid3/15917:
[260431.866901] #0: (rcu_read_lock){.+.+..}, at: [<ffffffff81146d50>] __perf_event_overflow+0x100/0x320
[260431.867896] stack backtrace:
[260431.869815] CPU: 0 PID: 15917 Comm: trinity-kid3 Not tainted 3.11.0-rc6+ #12
[260431.871885] 0000000000000000 ffff8801095d3ad0 ffffffff816f9c6f ffff88023ce86950
[260431.872912] ffff8801095d3b00 ffffffff810be6a7 ffff880079695cd0 0000000000000000
[260431.873910] ffff8801095d3c00 0000000000000000 ffff8801095d3b80 ffffffff81146ef4
[260431.874909] Call Trace:
[260431.875883] [<ffffffff816f9c6f>] dump_stack+0x54/0x74
[260431.876869] [<ffffffff810be6a7>] lockdep_rcu_suspicious+0xe7/0x120
[260431.877852] [<ffffffff81146ef4>] __perf_event_overflow+0x2a4/0x320
[260431.878834] [<ffffffff81146d50>] ? __perf_event_overflow+0x100/0x320
[260431.879811] [<ffffffff81146e0c>] ? __perf_event_overflow+0x1bc/0x320
[260431.880784] [<ffffffff8170ce00>] ? ftrace_call+0x5/0x2f
[260431.881758] [<ffffffff81147101>] perf_swevent_overflow+0x51/0xe0
[260431.882728] [<ffffffff811471ef>] perf_swevent_event+0x5f/0x90
[260431.883685] [<ffffffff81147329>] perf_tp_event+0x109/0x4f0
[260431.884637] [<ffffffff81147542>] ? perf_tp_event+0x322/0x4f0
[260431.885586] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420
[260431.886539] [<ffffffff8107a320>] ? task_work_run+0xe0/0xe0
[260431.887486] [<ffffffff81135910>] perf_ftrace_function_call+0xc0/0xd0
[260431.888417] [<ffffffff81114d37>] ? ftrace_ops_control_func+0xe7/0x110
[260431.889353] [<ffffffff8107a320>] ? task_work_run+0xe0/0xe0
[260431.890281] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420
[260431.891191] [<ffffffff81114d37>] ftrace_ops_control_func+0xe7/0x110
[260431.892088] [<ffffffff8170ce00>] ftrace_call+0x5/0x2f
[260431.892975] [<ffffffff81114cbb>] ? ftrace_ops_control_func+0x6b/0x110
[260431.893862] [<ffffffff8170ce00>] ? ftrace_call+0x5/0x2f
[260431.894745] [<ffffffff81097def>] ? local_clock+0x3f/0x50
[260431.895631] [<ffffffff8107a325>] ? debug_lockdep_rcu_enabled+0x5/0x40
[260431.896516] [<ffffffff8107a325>] ? debug_lockdep_rcu_enabled+0x5/0x40
[260431.897387] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420
[260431.898264] [<ffffffff81107e24>] rcu_eqs_enter+0x64/0xa0
[260431.899133] [<ffffffff8110c673>] rcu_user_enter+0x13/0x20
[260431.899999] [<ffffffff8114e67a>] user_enter+0x6a/0xd0
[260431.900859] [<ffffffff81010208>] syscall_trace_leave+0x78/0x150
[260431.901712] [<ffffffff8170d4af>] int_check_syscall_exit_work+0x34/0x3dQuoted 11 lines · On Mon, 26 Aug 2013 09:29:28 -0700 "Paul E. McKenney"…
Quoted 3 lines · Dave Jones
Another day, another rcu backtrace..
This says rc6, but it's pretty darn close to rc7, I think it was running a build from Friday.Could you please send your .config? Also, were you running ftrace, perf, RCU event tracing, or what?
Looks like you are running ftrace, but I though Steven had set that up so that it could be called from an extended quiescent state.
I know exactly what the issue is. Yes ftrace is safe to call even from these extended quiescent states, the problem is that ftrace is also the infrastructure of other users, where some of those users are not safe. Namely, perf.
Right now perf is not safe to trace all functions, as some of those functions have this issue. I was going to add something like:
FTRACE_NON_SAFE(rcu_eqs_enter);
where it will record locations that are not safe for all users, such that unless a function registers to ftrace with a flag of "FTRACE_FL_NON_SAFE_OK", anything that is on the non safe list (from the macro) will not be traced.
Now, how urgent is this fix? perf can only trace functions as root, and there's no reason for perf to be tracing all functions at the moment. But yes, a root user could run that and get this warning. Because I was going to implement this for 3.12 and not for this release.
-- Steve
Quoted 27 lines · Steven Rostedt
Quoted 11 lines · Dave Jones
Quoted 3 lines
Another day, another rcu backtrace..
This says rc6, but it's pretty darn close to rc7, I think it was running a build from Friday.Could you please send your .config? Also, were you running ftrace, perf, RCU event tracing, or what?
Looks like you are running ftrace, but I though Steven had set that up so that it could be called from an extended quiescent state.
I know exactly what the issue is. Yes ftrace is safe to call even from these extended quiescent states, the problem is that ftrace is also the infrastructure of other users, where some of those users are not safe. Namely, perf.
Right now perf is not safe to trace all functions, as some of those functions have this issue. I was going to add something like:
FTRACE_NON_SAFE(rcu_eqs_enter);where it will record locations that are not safe for all users, such that unless a function registers to ftrace with a flag of "
FTRACE_FL_NON_SAFE_OK", anything that is on the non safe list (from the macro) will not be traced.
Quoted 4 lines
Now, how urgent is this fix? perf can only trace functions as root, and there's no reason for perf to be tracing all functions at the moment. But yes, a root user could run that and get this warning. Because I was going to implement this for 3.12 and not for this release.
This was triggered as a regular user fwiw.
I had not been running perf, or any other tracing. It was just left fuzzing over the weekend with no interaction at all.
Dave
Quoted 4 lines · On Mon, 26 Aug 2013 13:50:12 -0400 Dave Jones <[email protected]> wrote:
This was triggered as a regular user fwiw.
I had not been running perf, or any other tracing. It was just left fuzzing over the weekend with no interaction at all.
So you are telling me that ftrace was enabled by a regular user? If so, that's a huge issue.
Quoted 5 lines
[260431.875883] [<ffffffff816f9c6f>] dump_stack+0x54/0x74 [260431.876869] [<ffffffff810be6a7>] lockdep_rcu_suspicious+0xe7/0x120 [260431.877852] [<ffffffff81146ef4>] __perf_event_overflow+0x2a4/0x320 [260431.878834] [<ffffffff81146d50>] ? __perf_event_overflow+0x100/0x320 [260431.879811] [<ffffffff81146e0c>] ? __perf_event_overflow+0x1bc/0x320
Quoted 8 lines
[260431.880784] [<ffffffff8170ce00>] ? ftrace_call+0x5/0x2f [260431.881758] [<ffffffff81147101>] perf_swevent_overflow+0x51/0xe0 [260431.882728] [<ffffffff811471ef>] perf_swevent_event+0x5f/0x90 [260431.883685] [<ffffffff81147329>] perf_tp_event+0x109/0x4f0 [260431.884637] [<ffffffff81147542>] ? perf_tp_event+0x322/0x4f0 [260431.885586] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420 [260431.886539] [<ffffffff8107a320>] ? task_work_run+0xe0/0xe0 [260431.887486] [<ffffffff81135910>] perf_ftrace_function_call+0xc0/0xd0
This is the perf function tracing call.
Quoted 5 lines
[260431.888417] [<ffffffff81114d37>] ? ftrace_ops_control_func+0xe7/0x110 [260431.889353] [<ffffffff8107a320>] ? task_work_run+0xe0/0xe0 [260431.890281] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420 [260431.891191] [<ffffffff81114d37>] ftrace_ops_control_func+0xe7/0x110 [260431.892088] [<ffffffff8170ce00>] ftrace_call+0x5/0x2f
ftrace_call is the mcount trampoline. The only way to get there is via function tracing, and function tracing should only be enabled by root.
Quoted 11 lines
[260431.892975] [<ffffffff81114cbb>] ? ftrace_ops_control_func+0x6b/0x110 [260431.893862] [<ffffffff8170ce00>] ? ftrace_call+0x5/0x2f [260431.894745] [<ffffffff81097def>] ? local_clock+0x3f/0x50 [260431.895631] [<ffffffff8107a325>] ? debug_lockdep_rcu_enabled+0x5/0x40 [260431.896516] [<ffffffff8107a325>] ? debug_lockdep_rcu_enabled+0x5/0x40 [260431.897387] [<ffffffff811079fb>] ? rcu_eqs_enter_common+0x5b/0x420 [260431.898264] [<ffffffff81107e24>] rcu_eqs_enter+0x64/0xa0 [260431.899133] [<ffffffff8110c673>] rcu_user_enter+0x13/0x20 [260431.899999] [<ffffffff8114e67a>] user_enter+0x6a/0xd0 [260431.900859] [<ffffffff81010208>] syscall_trace_leave+0x78/0x150 [260431.901712] [<ffffffff8170d4af>] int_check_syscall_exit_work+0x34/0x3d
So my question to you. If you were not running perf or any other tracing, and this is all just non-root user. How the hell did perf function tracing get started on your box????
-- Steve
Quoted 9 lines · Steven Rostedt
Quoted 4 lines · On Mon, 26 Aug 2013 13:50:12 -0400 Dave Jones <[email protected]> wrote:
This was triggered as a regular user fwiw.
I had not been running perf, or any other tracing. It was just left fuzzing over the weekend with no interaction at all.So you are telling me that ftrace was enabled by a regular user? If so, that's a huge issue.
quite.
Quoted 3 lines
So my question to you. If you were not running perf or any other tracing, and this is all just non-root user. How the hell did perf function tracing get started on your box????
What mechanisms are available that would trigger it being enabled ?
Is there some path through sys_perf_open_event that might be missing a capability check perhaps ?
Dave
Quoted 13 lines · On Mon, 26 Aug 2013 14:29:07 -0400 Dave Jones <[email protected]> wrote:
Quoted 9 lines · Steven Rostedt
Quoted 4 lines · On Mon, 26 Aug 2013 13:50:12 -0400 Dave Jones <[email protected]> wrote:
This was triggered as a regular user fwiw.
I had not been running perf, or any other tracing. It was just left fuzzing over the weekend with no interaction at all.So you are telling me that ftrace was enabled by a regular user? If so, that's a huge issue.
quite.
Quoted 9 lines
Quoted 3 lines
So my question to you. If you were not running perf or any other tracing, and this is all just non-root user. How the hell did perf function tracing get started on your box????
What mechanisms are available that would trigger it being enabled ?
Is there some path through
sys_perf_open_eventthat might be missing a capability check perhaps ?
That's a question for Ingo, Peter or Jiri.
-- Steve
Quoted 5 lines · Steven Rostedt
Quoted 3 lines
Is there some path through
sys_perf_open_eventthat might be missing a capability check perhaps ?That's a question for Ingo, Peter or Jiri.
Its not something I've looked at recently, git blames Jiri and fweisbec for most of that code.
Permission checks appear to live in kernel/trace/trace_event_perf.c:perf_trace_event_perm().
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Steven Rostedt
Quoted 3 lines
Is there some path through
sys_perf_open_eventthat might be missing a capability check perhaps ?
Quoted 7 lines
Quoted 1 line
That's a question for Ingo, Peter or Jiri.
Its not something I've looked at recently, git blames Jiri and fweisbec for most of that code.
Permission checks appear to live in kernel/trace/trace_event_perf.c:perf_trace_event_perm().
Well, it also depends on sysctl kernel.perf_event_paranoid settings.
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Steven Rostedt
Quoted 3 lines
Is there some path through
sys_perf_open_eventthat might be missing a capability check perhaps ?
Quoted 7 lines
Quoted 1 line
That's a question for Ingo, Peter or Jiri.
Its not something I've looked at recently, git blames Jiri and fweisbec for most of that code.
Permission checks appear to live in kernel/trace/trace_event_perf.c:perf_trace_event_perm().
Actually the following condition is weird:
/* The ftrace function trace is allowed only for root. */
if (ftrace_event_is_function(tp_event) &&
perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN))
return -EPERM;We probably intended to do:
/* The ftrace function trace is allowed only for root. */ if (ftrace_event_is_function(tp_event) ||
perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN))
return -EPERM;Can somebody confirm?
Quoted 14 lines · Frederic Weisbecker
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Steven Rostedt
Quoted 3 lines
Is there some path through
sys_perf_open_eventthat might be missing a capability check perhaps ?Quoted 7 lines
Quoted 1 line
That's a question for Ingo, Peter or Jiri.
Its not something I've looked at recently, git blames Jiri and fweisbec for most of that code.
Permission checks appear to live in kernel/trace/trace_event_perf.c:perf_trace_event_perm().
Quoted 7 lines
Actually the following condition is weird:
/* The ftrace function trace is allowed only for root. */ if (ftrace_event_is_function(tp_event) && perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN)) return -EPERM;
That says: If its its a function-event and we're paranoid but we don't have root, bail.
Quoted 8 lines
We probably intended to do:
/* The ftrace function trace is allowed only for root. */ if (ftrace_event_is_function(tp_event) || perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN)) return -EPERM;Can somebody confirm?
That would always disallow function-events, no?
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Frederic Weisbecker
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Steven Rostedt
Quoted 3 lines
Is there some path through
sys_perf_open_eventthat might be missing a capability check perhaps ?Quoted 7 lines
Quoted 1 line
That's a question for Ingo, Peter or Jiri.
Its not something I've looked at recently, git blames Jiri and fweisbec for most of that code.
Permission checks appear to live in kernel/trace/trace_event_perf.c:perf_trace_event_perm().
Actually the following condition is weird:
Quoted 8 lines
Quoted 5 lines
/* The ftrace function trace is allowed only for root. */ if (ftrace_event_is_function(tp_event) && perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN)) return -EPERM;That says: If its its a function-event and we're paranoid but we don't have root, bail.
Right, I misunderstood and thought we messed up general tracepoint permissions with function events.
Quoted 11 lines
Quoted 9 lines
We probably intended to do:
/* The ftrace function trace is allowed only for root. */ if (ftrace_event_is_function(tp_event) || perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN)) return -EPERM;Can somebody confirm?
That would always disallow function-events, no?
Yeah, sorry, returning from holidays require more dense coffee.
Quoted 13 lines · Dave Jones
Quoted 9 lines · Steven Rostedt
Quoted 4 lines · On Mon, 26 Aug 2013 13:50:12 -0400 Dave Jones <[email protected]> wrote:
This was triggered as a regular user fwiw.
I had not been running perf, or any other tracing. It was just left fuzzing over the weekend with no interaction at all.So you are telling me that ftrace was enabled by a regular user? If so, that's a huge issue.
quite.
Quoted 8 lines
Quoted 3 lines
So my question to you. If you were not running perf or any other tracing, and this is all just non-root user. How the hell did perf function tracing get started on your box????
What mechanisms are available that would trigger it being enabled ?
Is there some path through
sys_perf_open_eventthat might be missing a capability check perhaps ?
Do you have /sys/kernel/debug with access permissions?
David
Quoted 18 lines · David Ahern
Quoted 17 lines · Dave Jones
Quoted 9 lines · Steven Rostedt
Quoted 4 lines · On Mon, 26 Aug 2013 13:50:12 -0400 Dave Jones <[email protected]> wrote:
This was triggered as a regular user fwiw.
I had not been running perf, or any other tracing. It was just left fuzzing over the weekend with no interaction at all.So you are telling me that ftrace was enabled by a regular user? If so, that's a huge issue.
quite.
Quoted 3 lines
So my question to you. If you were not running perf or any other tracing, and this is all just non-root user. How the hell did perf function tracing get started on your box????
Quoted 6 lines
Quoted 4 lines
What mechanisms are available that would trigger it being enabled ?
Is there some path through
sys_perf_open_eventthat might be missing a capability check perhaps ?Do you have
/sys/kernel/debugwith access permissions?
Ah, yeah, that'll be it. Good catch.
Sorry for the false alarm Steve ;)
Dave
Quoted 5 lines · On Mon, 26 Aug 2013 15:49:24 -0400 Dave Jones <[email protected]> wrote:
Quoted 1 line
Do you have
/sys/kernel/debugwith access permissions?Ah, yeah, that'll be it. Good catch.
Sorry for the false alarm Steve ;)
Yeah, but it still does not explain how perf got started. Perf requires a sys_perf_event_open() call to run.
-- Steve
Quoted 4 lines · Steven Rostedt
On Mon, 26 Aug 2013 15:49:24 -0400
Dave Jones <[email protected]> wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Do you have
/sys/kernel/debugwith access permissions?Ah, yeah, that'll be it. Good catch.
Sorry for the false alarm Steve ;)
Yeah, but it still does not explain how perf got started. Perf requires a
sys_perf_event_open()call to run.
Per Peter's response another option is the paranoia level:
# perf event paranoia level: # -1 - not paranoid at all # 0 - disallow raw tracepoint access for unpriv # 1 - disallow cpu events for unpriv # 2 - disallow kernel profiling for unpriv kernel.perf_event_paranoid = -1
I keep my dev box at -1 and set /sys/kernel/debug to 755 to run perf as non-root.
David
Quoted 14 lines · David Ahern
Quoted 4 lines · Steven Rostedt
On Mon, 26 Aug 2013 15:49:24 -0400
Dave Jones <[email protected]> wrote:Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Do you have
/sys/kernel/debugwith access permissions?Ah, yeah, that'll be it. Good catch.
Sorry for the false alarm Steve ;)
Yeah, but it still does not explain how perf got started. Perf requires a
sys_perf_event_open()call to run.
Quoted 8 lines
Per Peter's response another option is the paranoia level:
# perf event paranoia level: # -1 - not paranoid at all # 0 - disallow raw tracepoint access for unpriv # 1 - disallow cpu events for unpriv # 2 - disallow kernel profiling for unpriv kernel.perf_event_paranoid = -1
This I have set to 1 on that box. (The default)
Dave