Skip to contentSkip to conversation

tracing: Last minute fixes for 6.2

PATCH In review Steven Rostedt 2 participants 13 messages Dec 14, 2022

Subsystems (6)
Stable branch x86 Tracing x86 MM +2 more subsystems
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

cover tracing: Last minute fixes for 6.2

git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace.git
trace/for-next

Head SHA1: 9a2179ff02e179a866375bd6916f761381749b5b

Guilherme G. Piccoli (1):
tracing: Improve panic/die notifiers

Steven Rostedt (Google) (5):
x86/mm/kmmio: Remove redundant preempt_disable()
tracing: Have trigger filter parsing errors show up in error_log
tracing: Remove pointer (asterisk) and brackets from cpumask_t field
tracing: Do not synchronize freeing of trigger filter on boot up
tracing: Fix cpumask() example typo

Yang Jihong (1):
tracing: Fix infinite loop in tracing_read_pipe on overflowed print_trace_line

[email protected] (1):
ftrace: Prevent RCU stall on PREEMPT_VOLUNTARY kernels

----

 arch/x86/mm/kmmio.c                        | 13 +++---
 include/trace/stages/stage4_event_fields.h | 10 ++++-
 kernel/trace/ftrace.c                      |  1 +
 kernel/trace/trace.c                       | 70 +++++++++++++++++-------------
 kernel/trace/trace_events_trigger.c        | 19 ++++++--
 samples/trace_events/trace-events-sample.h |  2 +-
 6 files changed, 72 insertions(+), 43 deletions(-)
author maintainer
#1 lore raw

PATCH 1/8 tracing: Fix infinite loop in tracing_read_pipe on overflowed print_trace_line

From: Yang Jihong <[email protected]>

print_trace_line may overflow seq_file buffer. If the event is not consumed, the while loop keeps peeking this event, causing a infinite loop.

Cc Masami Hiramatsu <[email protected]>
Fixes 088b1e427dbba ("ftrace: pipe fixes")
---
 kernel/trace/trace.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)
kernel/trace/trace.c +14 -1
index 664619b3f1e1..548890c7c0f5 100644
@@ -6802,7 +6802,20 @@ tracing_read_pipe(struct file *filp, char __user *ubuf,
 
 		ret = print_trace_line(iter);
 		if (ret == TRACE_TYPE_PARTIAL_LINE) {
-			/* don't print partial lines */
+			/*
+			 * If one print_trace_line() fills entire trace_seq in one shot,
+			 * trace_seq_to_user() will returns -EBUSY because save_len == 0,
+			 * In this case, we need to consume it, otherwise, loop will peek
+			 * this event next time, resulting in an infinite loop.
+			 */
+			if (save_len == 0) {
+				iter->seq.full = 0;
+				trace_seq_puts(&iter->seq, "[LINE TOO BIG]\n");
+				trace_consume(iter);
+				break;
+			}
+
+			/* In other cases, don't print partial lines */
 			iter->seq.seq.len = save_len;
 			break;
 		}
Signature · 1 line
2.35.1
author maintainer
#2 lore raw

PATCH 2/8 x86/mm/kmmio: Remove redundant preempt_disable()

In reply to Steven Rostedt

From: "Steven Rostedt (Google)" <[email protected]>

Now that kmmio uses rcu_read_lock_sched_notrace() there's no reason to call preempt_disable() as the read_lock_sched_notrace() already does that and is redundant.

This also removes the preempt_enable_no_resched() as the "no_resched()" portion was bogus as there's no reason to do that.

Cc Masami Hiramatsu <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Karol Herbst <[email protected]>
Cc Pekka Paalanen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 arch/x86/mm/kmmio.c | 13 +++++--------
 1 file changed, 5 insertions(+), 8 deletions(-)
arch/x86/mm/kmmio.c +5 -8
index 853c49877c16..9f82019179e1 100644
@@ -246,14 +246,13 @@ int kmmio_handler(struct pt_regs *regs, unsigned long addr)
 	page_base &= page_level_mask(l);
 
 	/*
-	 * Preemption is now disabled to prevent process switch during
-	 * single stepping. We can only handle one active kmmio trace
+	 * Hold the RCU read lock over single stepping to avoid looking
+	 * up the probe and kmmio_fault_page again. The rcu_read_lock_sched()
+	 * also disables preemption and prevents process switch during
+	 * the single stepping. We can only handle one active kmmio trace
 	 * per cpu, so ensure that we finish it before something else
-	 * gets to run. We also hold the RCU read lock over single
-	 * stepping to avoid looking up the probe and kmmio_fault_page
-	 * again.
+	 * gets to run.
 	 */
-	preempt_disable();
 	rcu_read_lock_sched_notrace();
 
 	faultpage = get_kmmio_fault_page(page_base);
@@ -324,7 +323,6 @@ int kmmio_handler(struct pt_regs *regs, unsigned long addr)
 
 no_kmmio:
 	rcu_read_unlock_sched_notrace();
-	preempt_enable_no_resched();
 	return ret;
 }
 
@@ -364,7 +362,6 @@ static int post_kmmio_handler(unsigned long condition, struct pt_regs *regs)
 	ctx->active--;
 	BUG_ON(ctx->active);
 	rcu_read_unlock_sched_notrace();
-	preempt_enable_no_resched();
 
 	/*
 	 * if somebody else is singlestepping across a probe point, flags
Signature · 1 line
2.35.1
author maintainer
#3 lore raw

PATCH 3/8 tracing: Have trigger filter parsing errors show up in error_log

In reply to Steven Rostedt

From: "Steven Rostedt (Google)" <[email protected]>

It is annoying that the filter parsing of triggers do not show up in the error_log. Trying to figure out what is incorrect in the input is difficult when it fails for a typo.

Have the errors of filter parsing show up in error_log as well.

---
 kernel/trace/trace_events_trigger.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/trace/trace_events_trigger.c +8 -1
index 918730d74932..19ce9d22bfd7 100644
@@ -1067,7 +1067,14 @@ int set_trigger_filter(char *filter_str,
 
 	/* The filter is for the 'trigger' event, not the triggered event */
 	ret = create_event_filter(file->tr, file->event_call,
-				  filter_str, false, &filter);
+				  filter_str, true, &filter);
+
+	/* Only enabled set_str for error handling */
+	if (filter) {
+		kfree(filter->filter_string);
+		filter->filter_string = NULL;
+	}
+
 	/*
 	 * If create_event_filter() fails, filter still needs to be freed.
 	 * Which the calling code will do with data->filter.
Signature · 1 line
2.35.1
author maintainer
#4 lore raw

PATCH 4/8 tracing: Remove pointer (asterisk) and brackets from cpumask_t field

In reply to Steven Rostedt

From: "Steven Rostedt (Google)" <[email protected]>

To differentiate between long arrays and cpumasks, the __cpumask() field was created. Part of the TRACE_EVENT() macros test if the type is signed or not by using the is_signed_type() macro. The __cpumask() field used the __dynamic_array() helper but because cpumask_t is a structure, it could not be used in the is_signed_type() macro as that would fail to build, so instead it passed in the pointer to cpumask_t.

Unfortunately, that creates in the format file:

  field:__data_loc cpumask_t *[] mask;    offset:36;      size:4; signed:0;

Which looks like an array of pointers to cpumask_t and not a cpumask_t type, which is misleading to user space parsers.

Douglas Raillard pointed out that the "[]" are also misleading, as cpumask_t is not an array.

Since cpumask() hasn't been created yet, and the parsers currently fail on it (but will still produce the raw output), make it be:

field:__data_loc cpumask_t mask; offset:36; size:4; signed:0;

Which is the correct type of the field.

Then the parsers can be updated to handle this.

Cc Masami Hiramatsu <[email protected]>
Cc Valentin Schneider <[email protected]>
Cc Andrew Morton <[email protected]>
Fixes 8230f27b1ccc ("tracing: Add __cpumask to denote a trace event field that is a cpumask_t")
---
 include/trace/stages/stage4_event_fields.h | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
include/trace/stages/stage4_event_fields.h +8 -2
index f2990d22313c..affd541fd25e 100644
@@ -47,7 +47,10 @@
 #define __bitmask(item, nr_bits) __dynamic_array(unsigned long, item, -1)
 
 #undef __cpumask
-#define __cpumask(item) __dynamic_array(cpumask_t *, item, -1)
+#define __cpumask(item) {						\
+	.type = "__data_loc cpumask_t", .name = #item,			\
+	.size = 4, .align = 4,						\
+	.is_signed = 0, .filter_type = FILTER_OTHER },
 
 #undef __sockaddr
 #define __sockaddr(field, len) __dynamic_array(u8, field, len)
@@ -68,7 +71,10 @@
 #define __rel_bitmask(item, nr_bits) __rel_dynamic_array(unsigned long, item, -1)
 
 #undef __rel_cpumask
-#define __rel_cpumask(item) __rel_dynamic_array(cpumask_t *, item, -1)
+#define __rel_cpumask(item) {						\
+	.type = "__rel_loc cpumask_t", .name = #item,			\
+	.size = 4, .align = 4,						\
+	.is_signed = 0, .filter_type = FILTER_OTHER },
 
 #undef __rel_sockaddr
 #define __rel_sockaddr(field, len) __rel_dynamic_array(u8, field, len)
Signature · 1 line
2.35.1
author maintainer
#5 lore raw

PATCH 5/8 tracing: Do not synchronize freeing of trigger filter on boot up

In reply to Steven Rostedt

From: "Steven Rostedt (Google)" <[email protected]>

If a trigger filter on the kernel command line fails to apply (due to syntax error), it will be freed. The freeing will call tracepoint_synchronize_unregister(), but this is not needed during early boot up, and will even trigger a lockdep splat.

Avoid calling the synchronization function when system_state is SYSTEM_BOOTING.

---
 kernel/trace/trace_events_trigger.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
kernel/trace/trace_events_trigger.c +8 -2
index 19ce9d22bfd7..e535959939d3 100644
@@ -1085,8 +1085,14 @@ int set_trigger_filter(char *filter_str,
 	rcu_assign_pointer(data->filter, filter);
 
 	if (tmp) {
-		/* Make sure the call is done with the filter */
-		tracepoint_synchronize_unregister();
+		/*
+		 * Make sure the call is done with the filter.
+		 * It is possible that a filter could fail at boot up,
+		 * and then this path will be called. Avoid the synchronization
+		 * in that case.
+		 */
+		if (system_state != SYSTEM_BOOTING)
+			tracepoint_synchronize_unregister();
 		free_event_filter(tmp);
 	}
 
Signature · 1 line
2.35.1
author maintainer
#6 lore raw

PATCH 6/8 ftrace: Prevent RCU stall on PREEMPT_VOLUNTARY kernels

In reply to Steven Rostedt

From: "[email protected]" <[email protected]>

The function match_records() may take a while due to a large number of string comparisons, so when in PREEMPT_VOLUNTARY kernels we could face RCU stalls due to that.

Add a cond_resched() to prevent that.

---
 kernel/trace/ftrace.c | 1 +
 1 file changed, 1 insertion(+)
kernel/trace/ftrace.c +1 -0
index d04552c0c275..b8e374a372e5 100644
@@ -4204,6 +4204,7 @@ match_records(struct ftrace_hash *hash, char *func, int len, char *mod)
 			}
 			found = 1;
 		}
+		cond_resched();
 	} while_for_each_ftrace_rec();
  out_unlock:
 	mutex_unlock(&ftrace_lock);
Signature · 1 line
2.35.1
#7 lore raw

Re: [for-next][PATCH 6/8] ftrace: Prevent RCU stall on PREEMPT_VOLUNTARY kernels

Quoted 1 line · Steven Rostedt

From: "[email protected]" <[email protected]>

Hi Steve, would be possible to use my full name here, like: Guilherme G. Piccoli <[email protected]> ?

Thanks,

Guilherme

Quoted 31 lines

The function match_records() may take a while due to a large number of string comparisons, so when in PREEMPT_VOLUNTARY kernels we could face RCU stalls due to that.

Add a cond_resched() to prevent that.

---
 kernel/trace/ftrace.c | 1 +
 1 file changed, 1 insertion(+)
kernel/trace/ftrace.c +1 -0
index d04552c0c275..b8e374a372e5 100644
@@ -4204,6 +4204,7 @@ match_records(struct ftrace_hash *hash, char *func, int len, char *mod)
 			}
 			found = 1;
 		}
+		cond_resched();
 	} while_for_each_ftrace_rec();
  out_unlock:
 	mutex_unlock(&ftrace_lock);
author maintainer
#8 lore raw

Re: [for-next][PATCH 6/8] ftrace: Prevent RCU stall on PREEMPT_VOLUNTARY kernels

Quoted 5 lines · On Wed, 14 Dec 2022 12:53:04 -0300 "Guilherme G. Piccoli"…
Quoted 1 line · Steven Rostedt

From: "[email protected]" <[email protected]>

Hi Steve, would be possible to use my full name here, like: Guilherme G. Piccoli <[email protected]> ?

Sure, I haven't pushed to the next repo yet.

I found the reason it didn't go into my patchwork, and did a resend via a "redirect", and that must have changed the from address :-/

-- Steve

author maintainer Acked-by Paul E. McKenney Acked-by Masami Hiramatsu (Google)
#9 lore raw

Re: [for-next][PATCH 6/8] ftrace: Prevent RCU stall on PREEMPT_VOLUNTARY kernels

Quoted 6 lines · On Wed, 14 Dec 2022 11:15:44 -0500 Steven Rostedt <[email protected]> wrote:
Quoted 3 lines · On Wed, 14 Dec 2022 12:53:04 -0300 "Guilherme G. Piccoli"…
Quoted 1 line · Steven Rostedt

From: "[email protected]" <[email protected]>

Quoted 8 lines
Quoted 2 lines

Hi Steve, would be possible to use my full name here, like: Guilherme G. Piccoli <[email protected]> ?

Sure, I haven't pushed to the next repo yet.

I found the reason it didn't go into my patchwork, and did a resend via a "redirect", and that must have changed the from address :-/

Here's the new update:

-- Steve

From d0b24b4e91fcb8408c4979888547f86be514e337 Mon Sep 17 00:00:00 2001
From: "Guilherme G. Piccoli" <[email protected]>
Date: Tue, 15 Nov 2022 17:48:47 -0300
Subject: [PATCH] ftrace: Prevent RCU stall on PREEMPT_VOLUNTARY kernels

The function match_records() may take a while due to a large number of string comparisons, so when in PREEMPT_VOLUNTARY kernels we could face RCU stalls due to that.

Add a cond_resched() to prevent that.

---
 kernel/trace/ftrace.c | 1 +
 1 file changed, 1 insertion(+)
kernel/trace/ftrace.c +1 -0
index d04552c0c275..b8e374a372e5 100644
@@ -4204,6 +4204,7 @@ match_records(struct ftrace_hash *hash, char *func, int len, char *mod)
 			}
 			found = 1;
 		}
+		cond_resched();
 	} while_for_each_ftrace_rec();
  out_unlock:
 	mutex_unlock(&ftrace_lock);
Signature · 1 line
2.35.1
#10 lore raw

Re: [for-next][PATCH 6/8] ftrace: Prevent RCU stall on PREEMPT_VOLUNTARY kernels

In reply to Steven Rostedt
Quoted 9 lines · Steven Rostedt
Quoted 5 lines · On Wed, 14 Dec 2022 12:53:04 -0300 "Guilherme G. Piccoli"…
Quoted 1 line · Steven Rostedt

From: "[email protected]" <[email protected]>

Hi Steve, would be possible to use my full name here, like: Guilherme G. Piccoli <[email protected]> ?

Quoted 6 lines

Sure, I haven't pushed to the next repo yet.

I found the reason it didn't go into my patchwork, and did a resend via a "redirect", and that must have changed the from address :-/

-- Steve

Perfect, thanks a lot for your prompt response!!

author maintainer
#11 lore raw

PATCH 7/8 tracing: Improve panic/die notifiers

In reply to Steven Rostedt

From: "Guilherme G. Piccoli" <[email protected]>

Currently the tracing dump_on_oops feature is implemented through separate notifiers, one for die/oops and the other for panic; given they have the same functionality, let's unify them.

Also improve the function comment and change the priority of the notifier to make it execute earlier, avoiding showing useless trace data (like the callback names for the other notifiers); finally, we also removed an unnecessary header inclusion.

---
 kernel/trace/trace.c | 55 ++++++++++++++++++++++----------------------
 1 file changed, 27 insertions(+), 28 deletions(-)
kernel/trace/trace.c +27 -28
index 548890c7c0f5..6d7ef130f57e 100644
@@ -19,7 +19,6 @@
 #include <linux/kallsyms.h>
 #include <linux/security.h>
 #include <linux/seq_file.h>
-#include <linux/notifier.h>
 #include <linux/irqflags.h>
 #include <linux/debugfs.h>
 #include <linux/tracefs.h>
@@ -9876,40 +9875,40 @@ static __init int tracer_init_tracefs(void)
 
 fs_initcall(tracer_init_tracefs);
 
-static int trace_panic_handler(struct notifier_block *this,
-			       unsigned long event, void *unused)
-{
-	if (ftrace_dump_on_oops)
-		ftrace_dump(ftrace_dump_on_oops);
-	return NOTIFY_OK;
-}
+static int trace_die_panic_handler(struct notifier_block *self,
+				unsigned long ev, void *unused);
 
 static struct notifier_block trace_panic_notifier = {
-	.notifier_call  = trace_panic_handler,
-	.next           = NULL,
-	.priority       = 150   /* priority: INT_MAX >= x >= 0 */
+	.notifier_call = trace_die_panic_handler,
+	.priority = INT_MAX - 1,
 };
 
-static int trace_die_handler(struct notifier_block *self,
-			     unsigned long val,
-			     void *data)
-{
-	switch (val) {
-	case DIE_OOPS:
-		if (ftrace_dump_on_oops)
-			ftrace_dump(ftrace_dump_on_oops);
-		break;
-	default:
-		break;
-	}
-	return NOTIFY_OK;
-}
-
 static struct notifier_block trace_die_notifier = {
-	.notifier_call = trace_die_handler,
-	.priority = 200
+	.notifier_call = trace_die_panic_handler,
+	.priority = INT_MAX - 1,
 };
 
+/*
+ * The idea is to execute the following die/panic callback early, in order
+ * to avoid showing irrelevant information in the trace (like other panic
+ * notifier functions); we are the 2nd to run, after hung_task/rcu_stall
+ * warnings get disabled (to prevent potential log flooding).
+ */
+static int trace_die_panic_handler(struct notifier_block *self,
+				unsigned long ev, void *unused)
+{
+	if (!ftrace_dump_on_oops)
+		return NOTIFY_DONE;
+
+	/* The die notifier requires DIE_OOPS to trigger */
+	if (self == &trace_die_notifier && ev != DIE_OOPS)
+		return NOTIFY_DONE;
+
+	ftrace_dump(ftrace_dump_on_oops);
+
+	return NOTIFY_DONE;
+}
+
 /*
  * printk is set to max of 1024, we really don't need it that big.
  * Nothing should be printing 1000 characters anyway.
Signature · 1 line
2.35.1
author maintainer
#12 lore raw

PATCH 8/8 tracing: Fix cpumask() example typo

In reply to Steven Rostedt

From: "Steven Rostedt (Google)" <[email protected]>

The sample code for using cpumask used the wrong field for the __get_cpumask() helper. It used "cpus" which is the bitmask (but would still give a proper example) instead of the "cpum" that was there to be used.

Although it produces the same output, fix it, because it's an example and is confusing in how to properly use the cpumask() macro.

---
 samples/trace_events/trace-events-sample.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
samples/trace_events/trace-events-sample.h +1 -1
index fb4548a44153..1c6b843b8c4e 100644
@@ -359,7 +359,7 @@ TRACE_EVENT(foo_bar,
 		  __print_array(__get_dynamic_array(list),
 				__get_dynamic_array_len(list) / sizeof(int),
 				sizeof(int)),
-		  __get_str(str), __get_bitmask(cpus), __get_cpumask(cpus),
+		  __get_str(str), __get_bitmask(cpus), __get_cpumask(cpum),
 		  __get_str(vstr))
 );
 
Signature · 1 line
2.35.1