Skip to contentSkip to conversation

Exception in function_graph tracing

PATCH In review Danilo Cezar Zanella 3 participants 5 messages Sep 13, 2022 – Oct 5, 2022 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi

I'm getting the following error message when I set the current_tracer to function_graph. Procedures:

cd / sys/kernel/tracing
echo function_graph > current_tracer

Then I saw the following message in the console.

Unable to handle kernel paging request at virtual address fffffffc
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at kernel/sched/core.c:7270 migrate_disable+0x104/0x114
Modules linked in: rtc_armada38x ts7800_v2_wdt spi_orion spi_gpio
i2c_gpio i2c_algo_bit
CPU: 0 PID: 0 Comm:  Not tainted 4.19.249-rt111-llama-rt+ #36
Hardware name: Marvell Armada 380/385 (Device Tree)
[<c011424c>] (unwind_backtrace) from [<c010e318>] (show_stack+0x18/0x1c)
[<c010e318>] (show_stack) from [<c09da8ac>] (dump_stack+0x84/0x98)
[<c09da8ac>] (dump_stack) from [<c0124bc8>] (__warn+0x100/0x118)
[<c0124bc8>] (__warn) from [<c0124d0c>] (warn_slowpath_null+0x48/0x50)

[<c0124d0c>] (warn_slowpath_null) from [<c015163c>] (migrate_disable+0x104/0x114)
[<c015163c>] (migrate_disable) from [<c0182920>] (vprintk_emit+0x10c/0x1ec) [<c0182920>] (vprintk_emit) from [<c0182bc0>] (vprintk_default+0x28/0x30) [<c0182bc0>] (vprintk_default) from [<c09d5100>] (printk+0x38/0x5c) [<c09d5100>] (printk) from [<c09d4dc8>] (__do_kernel_fault.part.0+0x4c/0x7c) [<c09d4dc8>] (__do_kernel_fault.part.0) from [<c0118d84>] (do_page_fault+0x380/0x394)
[<c0118d84>] (do_page_fault) from [<c0118f6c>] (do_DataAbort+0x50/0xf0)

---[ end trace 0000000000000002 ]---
pgd = 721da14e
[fffffffc] *pgd=2fffd861, *pte=00000000, *ppte=00000000
Internal error: Oops: 37 [#1] PREEMPT SMP ARM
Modules linked in: rtc_armada38x ts7800_v2_wdt spi_orion spi_gpio
i2c_gpio i2c_algo_bit
CPU: 0 PID: 0 Comm:  Tainted: G        W         4.19.249-rt111-llama-rt+ #36
Hardware name: Marvell Armada 380/385 (Device Tree)
PC is at prepare_ftrace_return+0x20/0x48
LR is at ftrace_graph_caller+0x18/0x20
pc : [<c0112884>]    lr : [<c0112540>]    psr: 600f0193

sp : ef12a090 ip : 00000051 fp : 00000000
r10: e9c61d7c r9 : ef12a000 r8 : 00000001
r7 : ef12a19c r6 : ffffffff r5 : 600f0193 r4 : c0112884 r3 : 00000000 r2 : 00000000 r1 : c0118f28 r0 : fffffffc
Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none
Control: 10c5387d Table: 29c7004a DAC: 00000051 Process (pid: 0, stack limit = 0x6b08bd12)

Stack: (0xef12a090 to 0xef07651c)
[<c0112884>] (prepare_ftrace_return) from [<c0112540>]
(ftrace_graph_caller+0x18/0x20)
Code: e5933010 e593366c e3530000 18bd8070 (e5905000)
---[ end trace 0000000000000003 ]---
Kernel panic - not syncing: Attempted to kill the idle task!

SMP: failed to stop secondary CPUs

What is going on?
Am I doing something wrong?
Thanks

Danilo

#1 lore raw
Quoted 1 line · Danilo Cezar Zanella

Hi

Hi,

Quoted 9 lines

I'm getting the following error message when I set the current_tracer to function_graph. Procedures:

cd / sys/kernel/tracing
echo function_graph > current_tracer

Then I saw the following message in the console.

Unable to handle kernel paging request at virtual address fffffffc

…

Quoted 2 lines

What is going on?
Am I doing something wrong?

No, not as far I can tell. It looks like the function_graph tracer is busted in v4.19 even without the PREEMPT_RT patch. The function tracer seems to work.

Quoted 3 lines

Thanks

Danilo

Sebastian

A message that is not in the archive (sent off-list or not yet received): CAAxUTH4xawCFpnwpRkcaZ5fdzMEbZZmPOyVDoqdMrPrf+G64dg@mail.gmail.com

#3 lore raw
Quoted 1 line · Danilo Cezar Zanella

Hi Sebastian,

Hi Danilo,

please don't lose the list.

Quoted 10 lines
Quoted 1 line

No, not as far I can tell.

It sounds good. ;-)

Quoted 1 line

It looks like the function_graph tracer is busted in v4.19 even without the PREEMPT_RT patch.

It sounds bad... :-(

I'm new in LKM development and even more in Kernel tracing. I've tested 'nop' and 'function' tracing, both seem to work. Do you know when this issue was solved? Perhaps could I do a backport of it(if it is easy, of course)?

I managed to reproduce it on v4.19 but on v6.0-rcX. So it must have been fixed somehow in between. I browsed the history but I did not notice anything obvious.
So someone needs to dig deeper…

Quoted 3 lines

Thank you again

Danilo

Sebastian

#4 lore raw

PATCH 1/1 ARM: fix function graph tracer and unwinder dependencies

Danilo Cezar Zanella reported broken function graph tracer in the v4.19 tree. This was caused by the backport of commit
f9b58e8c7d031 ("ARM: 8800/1: use choice for kernel unwinders")

which ended in the v4.19-stable tree as of v4.19.222. It is also part of v4.14-stable since v4.14.259. It is also part of v4.9-stable since v4.9.299.

------->8---------

From: Russell King <[email protected]>
Date: Tue, 23 Apr 2019 17:09:38 +0100
Subject: [PATCH] ARM: fix function graph tracer and unwinder dependencies

Upstream commit 503621628b32782a07b2318e4112bd4372aa3401

Naresh Kamboju recently reported that the function-graph tracer crashes on ARM. The function-graph tracer assumes that the kernel is built with frame pointers.

We explicitly disabled the function-graph tracer when building Thumb2, since the Thumb2 ABI doesn't have frame pointers.

We recently changed the way the unwinder method was selected, which seems to have made it more likely that we can end up with the function- graph tracer enabled but without the kernel built with frame pointers.

Fix up the function graph tracer dependencies so the option is not available when we have no possibility of having frame pointers, and adjust the dependencies on the unwinder option to hide the non-frame pointer unwinder options if the function-graph tracer is enabled.

---
 arch/arm/Kconfig       | 2 +-
 arch/arm/Kconfig.debug | 6 +++---
 2 files changed, 4 insertions(+), 4 deletions(-)
arch/arm/Kconfig +1 -1
index d89d013f586cb..fce7e85f3ef57 100644
@@ -68,7 +68,7 @@ config ARM
 	select HAVE_EFFICIENT_UNALIGNED_ACCESS if (CPU_V6 || CPU_V6K || CPU_V7) && MMU
 	select HAVE_EXIT_THREAD
 	select HAVE_FTRACE_MCOUNT_RECORD if (!XIP_KERNEL)
-	select HAVE_FUNCTION_GRAPH_TRACER if (!THUMB2_KERNEL)
+	select HAVE_FUNCTION_GRAPH_TRACER if (!THUMB2_KERNEL && !CC_IS_CLANG)
 	select HAVE_FUNCTION_TRACER if (!XIP_KERNEL)
 	select HAVE_FUTEX_CMPXCHG if FUTEX
 	select HAVE_GCC_PLUGINS
arch/arm/Kconfig.debug +3 -3
index 01c760929c9e4..b931fac129a1b 100644
@@ -47,8 +47,8 @@ config DEBUG_WX
 
 choice
 	prompt "Choose kernel unwinder"
-	default UNWINDER_ARM if AEABI && !FUNCTION_GRAPH_TRACER
-	default UNWINDER_FRAME_POINTER if !AEABI || FUNCTION_GRAPH_TRACER
+	default UNWINDER_ARM if AEABI
+	default UNWINDER_FRAME_POINTER if !AEABI
 	help
 	  This determines which method will be used for unwinding kernel stack
 	  traces for panics, oopses, bugs, warnings, perf, /proc/<pid>/stack,
@@ -65,7 +65,7 @@ config UNWINDER_FRAME_POINTER
 
 config UNWINDER_ARM
 	bool "ARM EABI stack unwinder"
-	depends on AEABI
+	depends on AEABI && !FUNCTION_GRAPH_TRACER
 	select ARM_UNWIND
 	help
 	  This option enables stack unwinding support in the kernel
Signature · 1 line
2.37.2
maintainer
#5 lore raw

Re: [PATCH] ARM: fix function graph tracer and unwinder dependencies

Quoted 8 lines · Sebastian Andrzej Siewior

Danilo Cezar Zanella reported broken function graph tracer in the v4.19 tree. This was caused by the backport of commit
f9b58e8c7d031 ("ARM: 8800/1: use choice for kernel unwinders")

which ended in the v4.19-stable tree as of v4.19.222. It is also part of v4.14-stable since v4.14.259. It is also part of v4.9-stable since v4.9.299.

Now queued up, thanks.

greg k-h