Skip to contentSkip to conversation

few fixes for synthetic trace events

PATCH Superseded Sven Schnelle 2 participants 7 messages Aug 10, 2023 – Sep 8, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v3 cover few fixes for synthetic trace events

Hi Steven,

I'm now sending these patches in one patchset, because the second patch has a dependeny on the union vs. cast fix.

Changes in v3:
- remove superfluous struct around union trace_synth_field

Changes in v2:
- cosmetic changes
- add struct trace_dynamic_info to include/linux/trace_events.h

Sven Schnelle (3):
tracing/synthetic: use union instead of casts tracing/synthetic: skip first entry for stack traces tracing/synthetic: allocate one additional element for size

 include/linux/trace_events.h      |  11 ++++
 kernel/trace/trace.h              |   8 +++
 kernel/trace/trace_events_synth.c | 103 ++++++++++++------------------
 3 files changed, 60 insertions(+), 62 deletions(-)
Signature · 1 line
2.39.2
#1 lore raw

PATCH v3 1/3 tracing/synthetic: use union instead of casts

The current code uses a lot of casts to access the fields member in struct synth_trace_events with different sizes. This makes the code hard to read, and had already introduced an endianness bug. Use a union and struct instead.

---
 include/linux/trace_events.h      | 11 ++++
 kernel/trace/trace.h              |  8 +++
 kernel/trace/trace_events_synth.c | 87 +++++++++++++------------------
 3 files changed, 56 insertions(+), 50 deletions(-)
include/linux/trace_events.h +11 -0
index 3930e676436c..1e8bbdb8da90 100644
@@ -59,6 +59,17 @@ int trace_raw_output_prep(struct trace_iterator *iter,
 extern __printf(2, 3)
 void trace_event_printf(struct trace_iterator *iter, const char *fmt, ...);
 
+/* Used to find the offset and length of dynamic fields in trace events */
+struct trace_dynamic_info {
+#ifdef CONFIG_CPU_BIG_ENDIAN
+	u16	offset;
+	u16	len;
+#else
+	u16	len;
+	u16	offset;
+#endif
+};
+
 /*
  * The trace entry - the most basic unit of tracing. This is what
  * is printed in the end as a single line in the trace output, such as:
kernel/trace/trace.h +8 -0
index e1edc2197fc8..95956f75bea5 100644
@@ -1295,6 +1295,14 @@ static inline void trace_branch_disable(void)
 /* set ring buffers to default size if not already done so */
 int tracing_update_buffers(void);
 
+union trace_synth_field {
+	u8				as_u8;
+	u16				as_u16;
+	u32				as_u32;
+	u64				as_u64;
+	struct trace_dynamic_info	as_dynamic;
+};
+
 struct ftrace_event_field {
 	struct list_head	link;
 	const char		*name;
kernel/trace/trace_events_synth.c +37 -50

223 lines not on this page · raw message

Signature · 1 line
2.39.2
maintainer
#2 lore raw

Re: [PATCH v3 1/3] tracing/synthetic: use union instead of casts

Quoted 7 lines · On Thu, 10 Aug 2023 08:05:36 +0200 Sven Schnelle <[email protected]> wrote:
include/linux/trace_events.h +1 -0
@@ -59,6 +59,17 @@ int trace_raw_output_prep(struct trace_iterator *iter,
 extern __printf(2, 3)
 void trace_event_printf(struct trace_iterator *iter, const char *fmt, ...);
 
+/* Used to find the offset and length of dynamic fields in trace events */

include/linux/trace_events.h int trace_raw_output_prep(struct trace_iterator *iter, · line 72

Quoted 10 lines
include/linux/trace_events.h +10 -0
+struct trace_dynamic_info {
+#ifdef CONFIG_CPU_BIG_ENDIAN
+	u16	offset;
+	u16	len;
+#else
+	u16	len;
+	u16	offset;
+#endif
+};
+

So this patch broke synthetic events. I found that it works with:

include/linux/trace_events.h +2 -2
index 12f875e9e69a..4a98e23b0e77 100644
@@ -65,10 +65,10 @@ struct trace_dynamic_info {
 	u16	offset;
 	u16	len;
 #else
-	u16	len;
 	u16	offset;
+	u16	len;
 #endif
-};
+} __packed;
 
 /*
  * The trace entry - the most basic unit of tracing. This is what

That is, I had to swap the order of len vs offset, and to also be safe, I added "__packed".

I'm guessing that the BIG ENDIAN is broken too? So I'm going to swap that as well.

It causes the following test to fail:

./ftracetest test.d/trigger/inter-event/trigger-synthetic-event-dynstring.tc

Which I didn't notice because I found that test to be broken, and needs this patch:

tools/testing/selftests/ftrace/test.d/trigger/inter-event/trigger-synthetic-event-dynstring.tc +1 -1
index 213d890ed188..174376ddbc6c 100644
@@ -1,7 +1,7 @@
 #!/bin/sh
 # SPDX-License-Identifier: GPL-2.0
 # description: event trigger - test inter-event histogram trigger trace action with dynamic string param
-# requires: set_event synthetic_events events/sched/sched_process_exec/hist "char name[]' >> synthetic_events":README ping:program
+# requires: set_event synthetic_events events/sched/sched_process_exec/hist "' >> synthetic_events":README ping:program
 
 fail() { #msg
     echo $1

-- Steve

#3 lore raw

PATCH v3 2/3 tracing/synthetic: skip first entry for stack traces

In reply to Sven Schnelle

While debugging another issue i noticed that the stack trace output contains the number of entries on top:

         <idle>-0       [000] d..4.   203.322502: wake_lat: pid=0 delta=2268270616 stack=STACK:

=> 0x10
=> __schedule+0xac6/0x1a98
=> schedule+0x126/0x2c0
=> schedule_timeout+0x242/0x2c0
=> __wait_for_common+0x434/0x680
=> __wait_rcu_gp+0x198/0x3e0
=> synchronize_rcu+0x112/0x138
=> ring_buffer_reset_online_cpus+0x140/0x2e0
=> tracing_reset_online_cpus+0x15c/0x1d0
=> tracing_set_clock+0x180/0x1d8
=> hist_register_trigger+0x486/0x670
=> event_hist_trigger_parse+0x494/0x1318
=> trigger_process_regex+0x1d4/0x258
=> event_trigger_write+0xb4/0x170
=> vfs_write+0x210/0xad0
=> ksys_write+0x122/0x208

Fix this by skipping the first element. Also replace the pointer logic with an index variable which is easier to read.

Fixes 00cf3d672a9d ("tracing: Allow synthetic events to pass around stacktraces")
---
 kernel/trace/trace_events_synth.c | 17 ++++-------------
 1 file changed, 4 insertions(+), 13 deletions(-)
kernel/trace/trace_events_synth.c +4 -13
index 7fff8235075f..80a2a832f857 100644
@@ -350,7 +350,7 @@ static enum print_line_t print_synth_event(struct trace_iterator *iter,
 	struct trace_seq *s = &iter->seq;
 	struct synth_trace_event *entry;
 	struct synth_event *se;
-	unsigned int i, n_u64;
+	unsigned int i, j, n_u64;
 	char print_fmt[32];
 	const char *fmt;
 
@@ -389,18 +389,13 @@ static enum print_line_t print_synth_event(struct trace_iterator *iter,
 				n_u64 += STR_VAR_LEN_MAX / sizeof(u64);
 			}
 		} else if (se->fields[i]->is_stack) {
-			unsigned long *p, *end;
 			union trace_synth_field *data = &entry->fields[n_u64];
-
-			p = (void *)entry + data->as_dynamic.offset;
-			end = (void *)p + data->as_dynamic.len - (sizeof(long) - 1);
+			unsigned long *p = (void *)entry + data->as_dynamic.offset;
 
 			trace_seq_printf(s, "%s=STACK:\n", se->fields[i]->name);
-
-			for (; *p && p < end; p++)
-				trace_seq_printf(s, "=> %pS\n", (void *)*p);
+			for (j = 1; j < data->as_dynamic.len / sizeof(long); j++)
+				trace_seq_printf(s, "=> %pS\n", (void *)p[j]);
 			n_u64++;
-
 		} else {
 			struct trace_print_flags __flags[] = {
 			    __def_gfpflag_names, {-1, NULL} };
@@ -490,10 +485,6 @@ static unsigned int trace_stack(struct synth_trace_event *entry,
 			break;
 	}
 
-	/* Include the zero'd element if it fits */
-	if (len < HIST_STACKTRACE_DEPTH)
-		len++;
-
 	len *= sizeof(long);
 
 	/* Find the dynamic section to copy the stack into. */
Signature · 1 line
2.39.2
#4 lore raw

PATCH v3 3/3 tracing/synthetic: allocate one additional element for size

In reply to Sven Schnelle

While debugging another issue i noticed that the stack trace contains one invalid entry at the end:

<idle>-0 [008] d..4. 26.484201: wake_lat: pid=0 delta=2629976084 000000009cc24024 stack=STACK:
=> __schedule+0xac6/0x1a98
=> schedule+0x126/0x2c0
=> schedule_timeout+0x150/0x2c0
=> kcompactd+0x9ca/0xc20
=> kthread+0x2f6/0x3d8
=> __ret_from_fork+0x8a/0xe8
=> 0x6b6b6b6b6b6b6b6b

This is because the code failed to add the one element containing the number of entries to field_size.

Fixes 00cf3d672a9d ("tracing: Allow synthetic events to pass around stacktraces")
---
 kernel/trace/trace_events_synth.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/trace/trace_events_synth.c +2 -1
index 80a2a832f857..9897d0bfcab7 100644
@@ -528,7 +528,8 @@ static notrace void trace_event_raw_event_synth(void *__data,
 		str_val = (char *)(long)var_ref_vals[val_idx];
 
 		if (event->dynamic_fields[i]->is_stack) {
-			len = *((unsigned long *)str_val);
+			/* reserve one extra element for size */
+			len = *((unsigned long *)str_val) + 1;
 			len *= sizeof(unsigned long);
 		} else {
 			len = fetch_store_strlen((unsigned long)str_val);
Signature · 1 line
2.39.2
#5 lore raw
In reply to Sven Schnelle
Quoted 17 lines · Sven Schnelle

Hi Steven,

I'm now sending these patches in one patchset, because the second patch has a dependeny on the union vs. cast fix.

Changes in v3:
- remove superfluous struct around union trace_synth_field

Changes in v2:
- cosmetic changes
- add struct trace_dynamic_info to include/linux/trace_events.h

Sven Schnelle (3):
tracing/synthetic: use union instead of casts tracing/synthetic: skip first entry for stack traces tracing/synthetic: allocate one additional element for size

Quoted 4 lines
 include/linux/trace_events.h      |  11 ++++
 kernel/trace/trace.h              |   8 +++
 kernel/trace/trace_events_synth.c | 103 ++++++++++++------------------
 3 files changed, 60 insertions(+), 62 deletions(-)

Gentle ping? We're still seeing a lot of KASAN warnings.

maintainer
#6 lore raw
Quoted 19 lines · On Wed, 16 Aug 2023 11:08:52 +0200 Sven Schnelle <[email protected]> wrote:
Quoted 17 lines · Sven Schnelle

Hi Steven,

I'm now sending these patches in one patchset, because the second patch has a dependeny on the union vs. cast fix.

Changes in v3:
- remove superfluous struct around union trace_synth_field

Changes in v2:
- cosmetic changes
- add struct trace_dynamic_info to include/linux/trace_events.h

Sven Schnelle (3):
tracing/synthetic: use union instead of casts tracing/synthetic: skip first entry for stack traces tracing/synthetic: allocate one additional element for size

Quoted 6 lines
Quoted 4 lines
 include/linux/trace_events.h      |  11 ++++
 kernel/trace/trace.h              |   8 +++
 kernel/trace/trace_events_synth.c | 103 ++++++++++++------------------
 3 files changed, 60 insertions(+), 62 deletions(-)  

Gentle ping? We're still seeing a lot of KASAN warnings.

Can you resend and include a Cc to [email protected]. That will put it into patchwork:

https://patchwork.kernel.org/project/linux-trace-kernel/list/

Also, include everyone in the MAINTAINERS file too please.

That's what I use to know what to pull in for tracing. I have a personal patchwork on my INBOX that this did go into, but that's always considered lower priority.

-- Steve