function_graph trace causes hang when using sleepgraph (4.15.0-rc1 and newer)
I've discovered an issue in the kernel which is causing sleepgraph (analyse_suspend) to hang the system when it tries to get a callgraph trace over suspend/resume. It happens on kernels 4.15.0-rc1 and newer.
I've whittled it down to 3 items which have to be set for the hang to occur:
1) enable function_graph trace
2) enable global clock
3) enable trace points device_pm_callback_start or device_pm_callback_end
These things all occur when sleepgraph is run with -f. It needs to collect a function graph trace with global clock in tandem with the the device_pm_callback trace events.
This script reproduces the error:
echo "global" > /sys/kernel/debug/tracing/trace_clock
echo "function_graph" > /sys/kernel/debug/tracing/current_tracer echo 1 > /sys/kernel/debug/tracing/events/power/device_pm_callback_start/enable echo 1 > /sys/kernel/debug/tracing/events/power/device_pm_callback_end/enable echo 1 > /sys/kernel/debug/tracing/tracing_on
echo "mem" > /sys/power/state
# hang occurs here: system has to be rebooted echo 0 > /sys/kernel/debug/tracing/tracing_on
I performed a bisect between 4.14.0 and 4.15.0-rc1 and it landed on an ftrace patch that was merged in for 4.15.0-rc1. I'm still debugging it, but I just wanted to get this out there ASAP so it can be hopefully be fixed before 4.15.0. I'm not sure if the issue is in ftrace or in the pm subsystem device_pm_callback code (though nothing has changed in the latter).
-- [BISECT LOG START] --
git bisect start
# bad: [4fbd8d194f06c8a3fd2af1ce560ddb31f7ec8323] Linux 4.15-rc1 git bisect bad 4fbd8d194f06c8a3fd2af1ce560ddb31f7ec8323
# good: [bebc6082da0a9f5d47a1ea2edc099bf671058bd4] Linux 4.14 git bisect good bebc6082da0a9f5d47a1ea2edc099bf671058bd4
# good: [1be2172e96e33bfa22a5c7a651f768ef30ce3984] Merge tag 'modules- for-v4.15' of git://git.kernel.org/pub/scm/linux/kernel/git/jeyu/linux
git bisect good 1be2172e96e33bfa22a5c7a651f768ef30ce3984
# good: [8c609698569578913ad40bb160b97c3f6cfa15ec] Merge tag 'armsoc- soc' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git bisect good 8c609698569578913ad40bb160b97c3f6cfa15ec
# good: [f6705bf959efac87bca76d40050d342f1d212587] Merge tag 'drm-for- v4.15-amd-dc' of git://people.freedesktop.org/~airlied/linux
git bisect good f6705bf959efac87bca76d40050d342f1d212587
# bad: [c633e898bde8990a34907c91b7d5245cab866c6e] Merge tag 'devicetree-fixes-for-4.15' ofgit://git.kernel.org/pub/scm/linux/kernel/git/robh/linux
git bisect bad c633e898bde8990a34907c91b7d5245cab866c6e
# bad: [fc35c1966e1372a21a88f6655279361e2f92713f] Merge tag 'clk-for- linus' of git://git.kernel.org/pub/scm/linux/kernel/git/clk/linux
git bisect bad fc35c1966e1372a21a88f6655279361e2f92713f
# bad: [09bd7c75e55cbaa6c731b0c3a5512ad89159f26f] Merge tag 'kbuild- v4.15' ofgit://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild
git bisect bad 09bd7c75e55cbaa6c731b0c3a5512ad89159f26f
# good: [5eb9e8ac9a8f8ae98ae4386357683a0b5684bb48] arch/ia64/include/asm/topology.h: remove unused parent_node() macro git bisect good 5eb9e8ac9a8f8ae98ae4386357683a0b5684bb48
# bad: [2dcd9c71c1ffa9a036e09047f60e08383bb0abb6] Merge tag 'trace- v4.15' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-
trace
git bisect bad 2dcd9c71c1ffa9a036e09047f60e08383bb0abb6
# bad: [466c81c45b650deca213fda3d0ec4761667379a9] perf/ftrace: Fix function trace events
git bisect bad 466c81c45b650deca213fda3d0ec4761667379a9
# bad: [3e234289f86b12985ef8909cd34525fcb66c4efb] ftrace: Allow module init functions to be traced
git bisect bad 3e234289f86b12985ef8909cd34525fcb66c4efb
# bad: [83c07ecc4203728e85fc4a2ce6fdf25d16ea118e] tracing: Remove lookups from tracing_map hitcount
git bisect bad 83c07ecc4203728e85fc4a2ce6fdf25d16ea118e
# bad: [12ecef0cb12102d8c034770173d2d1363cb97d52] tracing: Reverse the order of trace_types_lock and event_mutex
git bisect bad 12ecef0cb12102d8c034770173d2d1363cb97d52
# good: [6e7a2398114a2597a0995f96f44d908741ae5035] tracing: Remove redundant unread variable ret
git bisect good 6e7a2398114a2597a0995f96f44d908741ae5035
# first bad commit: [12ecef0cb12102d8c034770173d2d1363cb97d52] tracing: Reverse the order of trace_types_lock and event_mutex
-- [BISECT LOG END] --
This is the commit it landed on.
12ecef0cb12102d8c034770173d2d1363cb97d52 is the first bad commit commit 12ecef0cb12102d8c034770173d2d1363cb97d52
Author: Steven Rostedt (VMware) <[email protected]> Date: Thu Sep 21 16:22:49 2017 -0400
tracing: Reverse the order of trace_types_lock and event_mutex
In order to make future changes where we need to call tracing_set_clock() from within an event command, the order of trace_types_lock and event_mutex must be reversed, as the event command
will hold event_mutex and the trace_types_lock is taken from within tracing_set_clock().
l.home
Stephen, the problem is reversed by removing the following two commits, the one the bisect showed and the very next. So the problem is here:
commit 1a149d7d3f45d311da1f63473736c05f30ae8a75
Author: Steven Rostedt (VMware) <[email protected]> Date: Fri Sep 22 16:59:02 2017 -0400
ring-buffer: Rewrite trace_recursive_(un)lock() to be simpler
The current method to prevent the ring buffer from entering into a recursize
loop is to use a bitmask and set the bit that maps to the current context
(normal, softirq, irq or NMI), and if that bit was already set, it is
considered a recursive loop.
New code is being added that may require the ring buffer to be entered a
second time in the current context. The recursive locking prevents that from
happening. Instead of mapping a bitmask to the current context, just allow 4
levels of nesting in the ring buffer. This matches the 4 context levels that
it can already nest. It is highly unlikely to have more than two levels,
thus it should be fine when we add the second entry into the ring buffer. If
that proves to be a problem, we can always up the number to 8.
An added benefit is that reading preempt_count() to get the current level
adds a very slight but noticeable overhead. This removes that need.
commit 12ecef0cb12102d8c034770173d2d1363cb97d52
Author: Steven Rostedt (VMware) <[email protected]> Date: Thu Sep 21 16:22:49 2017 -0400
tracing: Reverse the order of trace_types_lock and event_mutex
In order to make future changes where we need to call tracing_set_clock() from within an event command, the order of trace_types_lock and event_mutex must be reversed, as the event command
will hold event_mutex and the trace_types_lock is taken from within tracing_set_clock().
l.home
Quoted 127 lines · Todd Brandt
I've discovered an issue in the kernel which is causing sleepgraph (
analyse_suspend) to hang the system when it tries to get a callgraph trace over suspend/resume. It happens on kernels 4.15.0-rc1 and newer.I've whittled it down to 3 items which have to be set for the hang to occur:
1) enable
function_graphtrace
2) enable global clock
3) enable trace pointsdevice_pm_callback_startordevice_pm_callback_endThese things all occur when sleepgraph is run with -f. It needs to collect a function graph trace with global clock in tandem with the the
device_pm_callbacktrace events.This script reproduces the error:
echo "global" >
/sys/kernel/debug/tracing/trace_clock
echo "function_graph" >/sys/kernel/debug/tracing/current_tracerecho 1 >/sys/kernel/debug/tracing/events/power/device_pm_callback_start/enable
echo 1 >/sys/kernel/debug/tracing/events/power/device_pm_callback_end/enableecho 1 > /sys/kernel/debug/tracing/tracing_on
echo "mem" >/sys/power/state# hang occurs here: system has to be rebooted echo 0 > /sys/kernel/debug/tracing/tracing_onI performed a bisect between 4.14.0 and 4.15.0-rc1 and it landed on an
ftrace patch that was merged in for 4.15.0-rc1. I'm still debugging it,
but I just wanted to get this out there ASAP so it can be hopefully be
fixed before 4.15.0. I'm not sure if the issue is in ftrace or in the pm subsystemdevice_pm_callbackcode (though nothing has changed in the
latter).-- [BISECT LOG START] --
git bisect start
# bad: [4fbd8d194f06c8a3fd2af1ce560ddb31f7ec8323] Linux 4.15-rc1 git bisect bad4fbd8d194f06c8a3fd2af1ce560ddb31f7ec8323
# good: [bebc6082da0a9f5d47a1ea2edc099bf671058bd4] Linux 4.14 git bisect goodbebc6082da0a9f5d47a1ea2edc099bf671058bd4
# good: [1be2172e96e33bfa22a5c7a651f768ef30ce3984] Merge tag 'modules-
for-v4.15' ofgit://git.kernel.org/pub/scm/linux/kernel/git/jeyu/linux
git bisect good1be2172e96e33bfa22a5c7a651f768ef30ce3984
# good: [8c609698569578913ad40bb160b97c3f6cfa15ec] Merge tag 'armsoc- soc' ofgit://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git bisect good8c609698569578913ad40bb160b97c3f6cfa15ec
# good: [f6705bf959efac87bca76d40050d342f1d212587] Merge tag 'drm- for-
v4.15-amd-dc' ofgit://people.freedesktop.org/~airlied/linux
git bisect goodf6705bf959efac87bca76d40050d342f1d212587
# bad: [c633e898bde8990a34907c91b7d5245cab866c6e] Merge tag 'devicetree-fixes-for-4.15' ofgit://git.kernel.org/pub/scm/linux/kernel/git/robh/linux
git bisect badc633e898bde8990a34907c91b7d5245cab866c6e
# bad: [fc35c1966e1372a21a88f6655279361e2f92713f] Merge tag 'clk-for- linus' ofgit://git.kernel.org/pub/scm/linux/kernel/git/clk/linux
git bisect badfc35c1966e1372a21a88f6655279361e2f92713f
# bad: [09bd7c75e55cbaa6c731b0c3a5512ad89159f26f] Merge tag 'kbuild- v4.15' ofgit://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild
git bisect bad09bd7c75e55cbaa6c731b0c3a5512ad89159f26f
# good: [5eb9e8ac9a8f8ae98ae4386357683a0b5684bb48]arch/ia64/include/asm/topology.h: remove unusedparent_node()macro git bisect good5eb9e8ac9a8f8ae98ae4386357683a0b5684bb48
# bad: [2dcd9c71c1ffa9a036e09047f60e08383bb0abb6] Merge tag 'trace- v4.15' ofgit://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-
trace
git bisect bad2dcd9c71c1ffa9a036e09047f60e08383bb0abb6
# bad: [466c81c45b650deca213fda3d0ec4761667379a9] perf/ftrace: Fix function trace events
git bisect bad466c81c45b650deca213fda3d0ec4761667379a9
# bad: [3e234289f86b12985ef8909cd34525fcb66c4efb] ftrace: Allow module
init functions to be traced
git bisect bad3e234289f86b12985ef8909cd34525fcb66c4efb
# bad: [83c07ecc4203728e85fc4a2ce6fdf25d16ea118e] tracing: Remove lookups fromtracing_maphitcount
git bisect bad83c07ecc4203728e85fc4a2ce6fdf25d16ea118e
# bad: [12ecef0cb12102d8c034770173d2d1363cb97d52] tracing: Reverse the
order oftrace_types_lockandevent_mutex
git bisect bad12ecef0cb12102d8c034770173d2d1363cb97d52
# good: [6e7a2398114a2597a0995f96f44d908741ae5035] tracing: Remove redundant unread variable ret
git bisect good6e7a2398114a2597a0995f96f44d908741ae5035# first bad commit: [12ecef0cb12102d8c034770173d2d1363cb97d52] tracing: Reverse the order of trace_types_lock and event_mutex-- [BISECT LOG END] --
This is the commit it landed on.
12ecef0cb12102d8c034770173d2d1363cb97d52is the first bad commit commit12ecef0cb12102d8c034770173d2d1363cb97d52
Author: Steven Rostedt (VMware) <[email protected]> Date: Thu Sep 21 16:22:49 2017 -0400tracing: Reverse the order of
trace_types_lockandevent_mutex
In order to make future changes where we need to calltracing_set_clock()from within an event command, the order oftrace_types_lockandevent_mutexmust be reversed, as the event command
will holdevent_mutexand thetrace_types_lockis taken from within
tracing_set_clock().ca
l.homeRequested-by Tom Zanussi <[email protected]>Signed-off-by Steven Rostedt (VMware) <[email protected]>
Quoted 8 lines · On Mon, 08 Jan 2018 17:02:29 -0800 Todd Brandt <[email protected]>…
Stephen, the problem is reversed by removing the following two commits, the one the bisect showed and the very next. So the problem is here:
commit
1a149d7d3f45d311da1f63473736c05f30ae8a75
Author: Steven Rostedt (VMware) <[email protected]> Date: Fri Sep 22 16:59:02 2017 -0400ring-buffer: Rewrite trace_recursive_(un)lock() to be simpler
This one still doesn't make sense, for why it would cause the hang.
Quoted 5 lines
commit
12ecef0cb12102d8c034770173d2d1363cb97d52
Author: Steven Rostedt (VMware) <[email protected]> Date: Thu Sep 21 16:22:49 2017 -0400tracing: Reverse the order of
trace_types_lockandevent_mutex
This one does.
Can you run lockdep when you do this and see if lockdep catches anything? If it does, it should point directly to where the inversed locking happened.
-- Steve
Quoted 21 lines · Steven Rostedt
Quoted 10 lines · On Mon, 08 Jan 2018 17:02:29 -0800 Todd Brandt <[email protected]>…
Stephen, the problem is reversed by removing the following two commits,
the one the bisect showed and the very next. So the problem is here:commit
1a149d7d3f45d311da1f63473736c05f30ae8a75
Author: Steven Rostedt (VMware) <[email protected]> Date: Fri Sep 22 16:59:02 2017 -0400ring-buffer: Rewrite trace_recursive_(un)lock() to be simpler
This one still doesn't make sense, for why it would cause the hang.
Quoted 4 lines
commit
12ecef0cb12102d8c034770173d2d1363cb97d52
Author: Steven Rostedt (VMware) <[email protected]> Date: Thu Sep 21 16:22:49 2017 -0400
Quoted 7 lines
Quoted 1 line
tracing: Reverse the order of
trace_types_lockandevent_mutexThis one does.
Can you run lockdep when you do this and see if lockdep catches anything? If it does, it should point directly to where the inversed locking happened.
Can you reproduce the issue there? I just want to be sure it's not something local to our machines here, as long as you have CONFIG_PM enabled it should work the same hopefully.
I'll give lockdep a try here.
Quoted 2 lines
-- Steve
Quoted 32 lines · Todd Brandt
Quoted 24 lines · Steven Rostedt
Quoted 10 lines · On Mon, 08 Jan 2018 17:02:29 -0800 Todd Brandt <[email protected]>…
Stephen, the problem is reversed by removing the following two commits,
the one the bisect showed and the very next. So the problem is here:commit
1a149d7d3f45d311da1f63473736c05f30ae8a75
Author: Steven Rostedt (VMware) <[email protected]> Date: Fri Sep 22 16:59:02 2017 -0400ring-buffer: Rewrite trace_recursive_(un)lock() to be simpler
This one still doesn't make sense, for why it would cause the hang.
Quoted 6 lines
commit
12ecef0cb12102d8c034770173d2d1363cb97d52
Author: Steven Rostedt (VMware) <[email protected]> Date: Thu Sep 21 16:22:49 2017 -0400tracing: Reverse the order of
trace_types_lockandevent_mutexQuoted 6 lines
This one does.
Can you run lockdep when you do this and see if lockdep catches anything? If it does, it should point directly to where the inversed
locking happened.
Quoted 5 lines
Can you reproduce the issue there? I just want to be sure it's not something local to our machines here, as long as you have
CONFIG_PMenabled it should work the same hopefully.I'll give lockdep a try here.
I tried lockdep here (/proc/lockdep*) but I can't get any useful data after the hang beyond the last printk. Is there a way I can enclose the device_pm_callback function in some kind of debug harness to prevent the hang (sorry, I'm not terribly familiar with lockdep for kernel level hangs)?
Quoted 3 lines
Quoted 3 lines
-- Steve
Quoted 3 lines · On Mon, 08 Jan 2018 17:25:15 -0800 Todd Brandt <[email protected]>…
Can you reproduce the issue there? I just want to be sure it's not something local to our machines here, as long as you have
CONFIG_PMenabled it should work the same hopefully.
Just to let you know, I'm currently traveling, so my testing ability is limited at the time being.
-- Steve
Quoted 10 lines · Steven Rostedt
Quoted 3 lines · On Mon, 08 Jan 2018 17:25:15 -0800 Todd Brandt <[email protected]>…
Can you reproduce the issue there? I just want to be sure it's not something local to our machines here, as long as you have
CONFIG_PMenabled it should work the same hopefully.Just to let you know, I'm currently traveling, so my testing ability is
limited at the time being.
ok, it's not an emergency, we're using a patched version of the kernel with those two commits removed in the interim. I tried running kgdb over serial console, but unfortunately it doesn't work over a USB-to- Serial dongle (even Greg K-H says it won't work, :P). This is the best output I could get, considering:
# run the test script here [ 545.733458] PM: suspend entry (deep) [ 545.734535] PM: Syncing filesystems ... done. [ 546.580551] Freezing user space processes ... (elapsed 0.013 seconds) done. [ 546.594846] OOM killer disabled. [ 546.595002] Freezing remaining freezable tasks ... (elapsed 0.005 seconds) done. [ 557.808719] NMI watchdog: Watchdog detected hard LOCKUP on cpu 0
So it's definitely a hard lockup, I just can't get the stack trace without a system with a built in serial console.
Quoted 2 lines
-- Steve
Quoted 7 lines · Steven Rostedt
Quoted 3 lines · On Mon, 08 Jan 2018 17:25:15 -0800 Todd Brandt <[email protected]>…
Can you reproduce the issue there? I just want to be sure it's not something local to our machines here, as long as you have
CONFIG_PMenabled it should work the same hopefully.
Quoted 5 lines
Just to let you know, I'm currently traveling, so my testing ability is
limited at the time being.-- Steve
I just tested 4.15.0-rc8 (with the ring_buffer changes) and it appears to work correctly now, thanks!