Skip to contentSkip to conversation

ftrace kernel self test failure on RISC-V on 5.8, regression from 5.4.0

Bug 209317 Steven Rostedt 4 participants 5 messages Sep 28, 2020 – Oct 5, 2020 2 messages not in the archive RISC-V

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]/

A message that is not in the archive (sent off-list or not yet received): [email protected]/

#3 lore raw
Quoted 9 lines · Steven Rostedt
Quoted 5 lines · On Sat, 26 Sep 2020 22:02:35 +0000 [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=209317

--- Comment #4 from Colin Ian King ([email protected]) --- Issue still in 5.9-rc6

Quoted 6 lines

Atish,

As the issues bisects down to your commit, care to take a look at this.
(And take ownership of this bug)

Yes. I am already looking into this. Colin informed me about the bug over the weekend.

I couldn't change the ownership as I am not part of the editbugs group. I have sent an email to [email protected] for access.

Quoted 1 line

-- Steve

Signature · 2 lines
Regards,
Atish
#4 lore raw

Hi Alan and Zong,
I initially suspected ftrace is broken between v5.6 & v5.7 as Kolin pointed out. I couldn't find any reason how the HSM patch is related. Zong's ftrace patching code was also merged in that release.
However, I was able to reproduce the issue in the older kernel(v5.4) as well on both Qemu & Unleashed hardware.
Here are the steps:

mount -t debugfs none /sys/kernel/debug/
cd /sys/kernel/debug/tracing
echo function_graph > current_tracer
echo function > current_tracer

It works for the first time with function_graph but writing any other tracer crashes immediately.
Can you take a look to check if the bug is in ftrace infrastructure code ?

Quoted 33 lines · Atish Patra
Quoted 9 lines · Steven Rostedt
Quoted 5 lines · On Sat, 26 Sep 2020 22:02:35 +0000 [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=209317

--- Comment #4 from Colin Ian King ([email protected]) --- Issue still in 5.9-rc6

Quoted 6 lines

Atish,

As the issues bisects down to your commit, care to take a look at this.
(And take ownership of this bug)

Yes. I am already looking into this. Colin informed me about the bug over the weekend.

I couldn't change the ownership as I am not part of the editbugs group. I have sent an email to [email protected] for access.

Quoted 1 line

-- Steve

--
Regards,
Atish
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

Signature · 2 lines
Regards,
Atish
#5 lore raw

Hi Atish,

I can take out some time to take a look at it together, if anyone here fixes it or has ideas, please share the information, thanks.

Quoted 58 lines · Atish Patra

Hi Alan and Zong,
I initially suspected ftrace is broken between v5.6 & v5.7 as Kolin pointed out. I couldn't find any reason how the HSM patch is related. Zong's ftrace patching code was also merged in that release.
However, I was able to reproduce the issue in the older kernel(v5.4) as well on both Qemu & Unleashed hardware.
Here are the steps:

mount -t debugfs none /sys/kernel/debug/
cd /sys/kernel/debug/tracing
echo function_graph > current_tracer
echo function > current_tracer

It works for the first time with function_graph but writing any other tracer crashes immediately.
Can you take a look to check if the bug is in ftrace infrastructure code ?

Quoted 26 lines · Atish Patra
Quoted 9 lines · Steven Rostedt
Quoted 5 lines · On Sat, 26 Sep 2020 22:02:35 +0000 [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=209317

--- Comment #4 from Colin Ian King ([email protected]) --- Issue still in 5.9-rc6

Quoted 6 lines

Atish,

As the issues bisects down to your commit, care to take a look at this.
(And take ownership of this bug)

Yes. I am already looking into this. Colin informed me about the bug over the weekend.

I couldn't change the ownership as I am not part of the editbugs group. I have sent an email to [email protected] for access.

Quoted 1 line

-- Steve

Quoted 7 lines

--
Regards,
Atish
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

--
Regards,
Atish

#6 lore raw
Quoted 6 lines · Zong Li

Hi Atish,

I can take out some time to take a look at it together, if anyone here fixes it or has ideas, please share the information, thanks.

Thanks. I observed this in case it helps.

Across kernels, the panic trace seems to point out the one of the first two functions after patching is corrupted. rcu_momentary_dyntick_idle or stop_machine_yield[1]

[1]https://elixir.bootlin.com/linux/v5.9-rc7/source/kernel/stop_machine.c#L213

I am suspecting nop was not replaced with the correct auipc+jalr pair?

Quoted 59 lines · Atish Patra
Quoted 58 lines

Hi Alan and Zong,
I initially suspected ftrace is broken between v5.6 & v5.7 as Kolin pointed out. I couldn't find any reason how the HSM patch is related. Zong's ftrace patching code was also merged in that release.
However, I was able to reproduce the issue in the older kernel(v5.4) as well on both Qemu & Unleashed hardware.
Here are the steps:

mount -t debugfs none /sys/kernel/debug/
cd /sys/kernel/debug/tracing
echo function_graph > current_tracer
echo function > current_tracer

It works for the first time with function_graph but writing any other tracer crashes immediately.
Can you take a look to check if the bug is in ftrace infrastructure code ?

Quoted 26 lines · Atish Patra
Quoted 9 lines · Steven Rostedt
Quoted 5 lines · On Sat, 26 Sep 2020 22:02:35 +0000 [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=209317

--- Comment #4 from Colin Ian King ([email protected]) --- Issue still in 5.9-rc6

Quoted 6 lines

Atish,

As the issues bisects down to your commit, care to take a look at this.
(And take ownership of this bug)

Yes. I am already looking into this. Colin informed me about the bug over the weekend.

I couldn't change the ownership as I am not part of the editbugs group. I have sent an email to [email protected] for access.

Quoted 1 line

-- Steve

Quoted 7 lines

--
Regards,
Atish
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

--
Regards,
Atish

Signature · 2 lines
Regards,
Atish