Skip to contentSkip to conversation

printk: Make console tracepoint safe in NMI() context

PATCH Superseded Petr Mladek 4 participants 7 messages Jul 15, 2022

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Petr Mladek

author maintainer
#0 lore raw

PATCH 1/1 printk: Make console tracepoint safe in NMI() context

The commit 701850dc0c31bfadf75a0 ("printk, tracing: fix console tracepoint") moved the tracepoint from console_unlock() to vprintk_store(). As a result, it might be called in any context and triggered the following warning:

  WARNING: CPU: 1 PID: 16462 at include/trace/events/printk.h:10 printk_sprint+0x81/0xda
  Modules linked in: ppdev parport_pc parport
  CPU: 1 PID: 16462 Comm: event_benchmark Not tainted 5.19.0-rc5-test+ #5
  Hardware name: MSI MS-7823/CSM-H87M-G43 (MS-7823), BIOS V1.6 02/22/2014
  EIP: printk_sprint+0x81/0xda
  Code: 89 d8 e8 88 fc 33 00 e9 02 00 00 00 eb 6b 64 a1 a4 b8 91 c1 e8 fd d6 ff ff 84 c0 74 5c 64 a1 14 08 92 c1 a9 00 00 f0 00 74 02 <0f> 0b 64 ff 05 14 08 92 c1 b8 e0 c4 6b c1 e8 a5 dc 00 00 89 c7 e8
  EAX: 80110001 EBX: c20a52f8 ECX: 0000000c EDX: 6d203036
  ESI: 3df6004c EDI: 00000000 EBP: c61fbd7c ESP: c61fbd70
  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00010006
  CR0: 80050033 CR2: b7efc000 CR3: 05b80000 CR4: 001506f0
  Call Trace:
   vprintk_store+0x24b/0x2ff
   vprintk+0x37/0x4d
   _printk+0x14/0x16
   nmi_handle+0x1ef/0x24e
   ? find_next_bit.part.0+0x13/0x13
   ? find_next_bit.part.0+0x13/0x13
   ? function_trace_call+0xd8/0xd9
   default_do_nmi+0x57/0x1af
   ? trace_hardirqs_off_finish+0x2a/0xd9
   ? to_kthread+0xf/0xf
   exc_nmi+0x9b/0xf4
   asm_exc_nmi+0xae/0x29c

It comes from:

#define __DO_TRACE(name, args, cond, rcuidle) \
[...]

		/* srcu can't be used from NMI */	\
		WARN_ON_ONCE(rcuidle && in_nmi());	\

It might be possible to make srcu working in NMI. But it would be slower on some architectures. It is not worth doing it just because of this tracepoint.

It would be possible to disable this tracepoint in NMI or in rcuidle context. Where the rcuidle context looks more rare and thus more acceptable to be ignored.

Alternative solution would be to move the tracepoint back to console code. But the location is less reliable by definition. Also the synchronization against other tracing messages is much worse.

Let's ignore the tracepoint in rcuidle context as the least evil solution.

---
 include/trace/events/printk.h | 11 ++++++++++-
 kernel/printk/printk.c        |  2 +-
 2 files changed, 11 insertions(+), 2 deletions(-)
include/trace/events/printk.h +10 -1
index 13d405b2fd8b..a3ee720f41b5 100644
@@ -7,11 +7,20 @@
 
 #include <linux/tracepoint.h>
 
-TRACE_EVENT(console,
+TRACE_EVENT_CONDITION(console,
 	TP_PROTO(const char *text, size_t len),
 
 	TP_ARGS(text, len),
 
+	/*
+	 * trace_console_rcuidle() is not working in NMI. printk()
+	 * is used more often in NMI than in rcuidle context.
+	 * Choose the less evil solution here.
+	 *
+	 * raw_smp_processor_id() is reliable in rcuidle context.
+	 */
+	TP_CONDITION(!rcu_is_idle_cpu(raw_smp_processor_id())),
+
 	TP_STRUCT__entry(
 		__dynamic_array(char, msg, len + 1)
 	),
kernel/printk/printk.c +1 -1
index b49c6ff6dca0..bd76a45ecc7f 100644
@@ -2108,7 +2108,7 @@ static u16 printk_sprint(char *text, u16 size, int facility,
 		}
 	}
 
-	trace_console_rcuidle(text, text_len);
+	trace_console(text, text_len);
 
 	return text_len;
 }
Signature · 1 line
2.35.3
#1 lore raw
Quoted 75 lines · Petr Mladek

The commit 701850dc0c31bfadf75a0 ("printk, tracing: fix console tracepoint") moved the tracepoint from console_unlock() to vprintk_store(). As a result, it might be called in any context and triggered the following warning:

  WARNING: CPU: 1 PID: 16462 at include/trace/events/printk.h:10 printk_sprint+0x81/0xda
  Modules linked in: ppdev parport_pc parport
  CPU: 1 PID: 16462 Comm: event_benchmark Not tainted 5.19.0-rc5-test+ #5
  Hardware name: MSI MS-7823/CSM-H87M-G43 (MS-7823), BIOS V1.6 02/22/2014
  EIP: printk_sprint+0x81/0xda
  Code: 89 d8 e8 88 fc 33 00 e9 02 00 00 00 eb 6b 64 a1 a4 b8 91 c1 e8 fd d6 ff ff 84 c0 74 5c 64 a1 14 08 92 c1 a9 00 00 f0 00 74 02 <0f> 0b 64 ff 05 14 08 92 c1 b8 e0 c4 6b c1 e8 a5 dc 00 00 89 c7 e8
  EAX: 80110001 EBX: c20a52f8 ECX: 0000000c EDX: 6d203036
  ESI: 3df6004c EDI: 00000000 EBP: c61fbd7c ESP: c61fbd70
  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00010006
  CR0: 80050033 CR2: b7efc000 CR3: 05b80000 CR4: 001506f0
  Call Trace:
   vprintk_store+0x24b/0x2ff
   vprintk+0x37/0x4d
   _printk+0x14/0x16
   nmi_handle+0x1ef/0x24e
   ? find_next_bit.part.0+0x13/0x13
   ? find_next_bit.part.0+0x13/0x13
   ? function_trace_call+0xd8/0xd9
   default_do_nmi+0x57/0x1af
   ? trace_hardirqs_off_finish+0x2a/0xd9
   ? to_kthread+0xf/0xf
   exc_nmi+0x9b/0xf4
   asm_exc_nmi+0xae/0x29c

It comes from:

  #define __DO_TRACE(name, args, cond, rcuidle) \
  [...]
                /* srcu can't be used from NMI */       \
                WARN_ON_ONCE(rcuidle && in_nmi());      \

It might be possible to make srcu working in NMI. But it would be slower on some architectures. It is not worth doing it just because of this tracepoint.

It would be possible to disable this tracepoint in NMI or in rcuidle context. Where the rcuidle context looks more rare and thus more acceptable to be ignored.

Alternative solution would be to move the tracepoint back to console code. But the location is less reliable by definition. Also the synchronization against other tracing messages is much worse.

Let's ignore the tracepoint in rcuidle context as the least evil solution.

---
 include/trace/events/printk.h | 11 ++++++++++-
 kernel/printk/printk.c        |  2 +-
 2 files changed, 11 insertions(+), 2 deletions(-)
include/trace/events/printk.h +1 -1
index 13d405b2fd8b..a3ee720f41b5 100644
@@ -7,11 +7,20 @@
 
 #include <linux/tracepoint.h>
 
-TRACE_EVENT(console,
+TRACE_EVENT_CONDITION(console,
        TP_PROTO(const char *text, size_t len),
 
        TP_ARGS(text, len),

include/trace/events/printk.h line 23

Quoted 10 lines
include/trace/events/printk.h +9 -0
 
+       /*
+        * trace_console_rcuidle() is not working in NMI. printk()
+        * is used more often in NMI than in rcuidle context.
+        * Choose the less evil solution here.
+        *
+        * raw_smp_processor_id() is reliable in rcuidle context.
+        */
+       TP_CONDITION(!rcu_is_idle_cpu(raw_smp_processor_id())),
+

Couldn't this just use rcu_is_watching()?

| * rcu_is_watching - see if RCU thinks that the current CPU is not idle

Thanks,
-- Marco

maintainer
#2 lore raw
Quoted 3 lines · On Fri, 15 Jul 2022 14:39:52 +0200 Marco Elver <[email protected]> wrote:

Couldn't this just use rcu_is_watching()?

| * rcu_is_watching - see if RCU thinks that the current CPU is not idle

Maybe, but I was thinking that Petr had a way to hit the issue that we worry about. But since the non _rcuide() call requires rcu watching, prehaps that is better to use.

-- Steve

#3 lore raw
Quoted 10 lines · Steven Rostedt
Quoted 3 lines · On Fri, 15 Jul 2022 14:39:52 +0200 Marco Elver <[email protected]> wrote:

Couldn't this just use rcu_is_watching()?

| * rcu_is_watching - see if RCU thinks that the current CPU is not idle

Maybe, but I was thinking that Petr had a way to hit the issue that we worry about. But since the non _rcuide() call requires rcu watching, prehaps that is better to use.

In case this helps... ;-)

The rcu_is_watching() function is designed to be used from the current CPU, so it dispenses with memory ordering. However, it explicitly disables preemption in order to avoid weird preemption patterns.

The formulation that Marco used is designed to be used from a remote CPU, and so it includes explicit memory ordering that is not needed in this case. But it does not disable preemption.

So if preemption is enabled at that point in tracing, you really want to be using rcu_is_watching().

Thanx, Paul

maintainer
#4 lore raw
Quoted 2 lines · On Fri, 15 Jul 2022 08:10:00 -0700 "Paul E. McKenney" <[email protected]>…

So if preemption is enabled at that point in tracing, you really want to be using rcu_is_watching().

And yes, at that point in tracing, preemption is still enabled if the tracepoint was called with preemption enabled.

Thus, we really need to convert that to rcu_is_watching().

-- Steve

author maintainer
#5 lore raw
In reply to Paul E. McKenney
Quoted 11 lines · Paul E. McKenney
Quoted 10 lines · Steven Rostedt
Quoted 3 lines · On Fri, 15 Jul 2022 14:39:52 +0200 Marco Elver <[email protected]> wrote:

Couldn't this just use rcu_is_watching()?

| * rcu_is_watching - see if RCU thinks that the current CPU is not idle

Maybe, but I was thinking that Petr had a way to hit the issue that we worry about. But since the non _rcuide() call requires rcu watching, prehaps that is better to use.

I actually saw the warning even with simple sysrq+l. I wonder why I have missed it during testing. It was probably well hidden within the other backtraces.

I was not aware that rcu_is_watching() and rcu_is_idle_cpu() did basically the same. I used rcu_is_idle_cpu() because of the "idle" in the name and the function description ;-)

Quoted 6 lines

In case this helps... ;-)

The rcu_is_watching() function is designed to be used from the current CPU, so it dispenses with memory ordering. However, it explicitly disables preemption in order to avoid weird preemption patterns.

Quoted 6 lines

The formulation that Marco used is designed to be used from a remote CPU, and so it includes explicit memory ordering that is not needed in this case. But it does not disable preemption.

So if preemption is enabled at that point in tracing, you really want to be using rcu_is_watching().

rcu_is_watching() is the right variant then. I am going to send v2.

Thanks a lot for the detailed explanation.

Best Regards,
Petr

maintainer
#6 lore raw
In reply to Petr Mladek

include/trace/events/printk.h line 23

Quoted 9 lines · On Fri, 15 Jul 2022 14:01:52 +0200 Petr Mladek <[email protected]> wrote:
include/trace/events/printk.h +9 -0
+	/*
+	 * trace_console_rcuidle() is not working in NMI. printk()
+	 * is used more often in NMI than in rcuidle context.
+	 * Choose the less evil solution here.
+	 *
+	 * raw_smp_processor_id() is reliable in rcuidle context.
+	 */
+	TP_CONDITION(!rcu_is_idle_cpu(raw_smp_processor_id())),
+

As Marco mentioned in the other thread, would a check for 'rcu_is_watching()' be better?

-- Steve

Quoted 3 lines
include/trace/events/printk.h +0 -0
 	TP_STRUCT__entry(
 		__dynamic_array(char, msg, len + 1)
 	),