Skip to contentSkip to conversation

few fixes for synthetic trace events

PATCH RESEND In review Sven Schnelle 3 participants 6 messages Aug 16, 2023 – Aug 25, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v3 cover few fixes for synthetic trace events

Hi Steven,

I'm now sending these patches in one patchset, because the second patch has a dependeny on the union vs. cast fix.

Changes in v3:
- remove superfluous struct around union trace_synth_field

Changes in v2:
- cosmetic changes
- add struct trace_dynamic_info to include/linux/trace_events.h

Sven Schnelle (3):
tracing/synthetic: use union instead of casts tracing/synthetic: skip first entry for stack traces tracing/synthetic: allocate one additional element for size

 include/linux/trace_events.h      |  11 ++++
 kernel/trace/trace.h              |   8 +++
 kernel/trace/trace_events_synth.c | 103 ++++++++++++------------------
 3 files changed, 60 insertions(+), 62 deletions(-)
Signature · 1 line
2.39.2
#1 lore raw

PATCH RESEND v3 1/3 tracing/synthetic: use union instead of casts

The current code uses a lot of casts to access the fields member in struct synth_trace_events with different sizes. This makes the code hard to read, and had already introduced an endianness bug. Use a union and struct instead.

---
 include/linux/trace_events.h      | 11 ++++
 kernel/trace/trace.h              |  8 +++
 kernel/trace/trace_events_synth.c | 87 +++++++++++++------------------
 3 files changed, 56 insertions(+), 50 deletions(-)
include/linux/trace_events.h +11 -0
index 3930e676436c..1e8bbdb8da90 100644
@@ -59,6 +59,17 @@ int trace_raw_output_prep(struct trace_iterator *iter,
 extern __printf(2, 3)
 void trace_event_printf(struct trace_iterator *iter, const char *fmt, ...);
 
+/* Used to find the offset and length of dynamic fields in trace events */
+struct trace_dynamic_info {
+#ifdef CONFIG_CPU_BIG_ENDIAN
+	u16	offset;
+	u16	len;
+#else
+	u16	len;
+	u16	offset;
+#endif
+};
+
 /*
  * The trace entry - the most basic unit of tracing. This is what
  * is printed in the end as a single line in the trace output, such as:
kernel/trace/trace.h +8 -0
index e1edc2197fc8..95956f75bea5 100644
@@ -1295,6 +1295,14 @@ static inline void trace_branch_disable(void)
 /* set ring buffers to default size if not already done so */
 int tracing_update_buffers(void);
 
+union trace_synth_field {
+	u8				as_u8;
+	u16				as_u16;
+	u32				as_u32;
+	u64				as_u64;
+	struct trace_dynamic_info	as_dynamic;
+};
+
 struct ftrace_event_field {
 	struct list_head	link;
 	const char		*name;
kernel/trace/trace_events_synth.c +37 -50

223 lines not on this page · raw message

Signature · 1 line
2.39.2
#2 lore raw

Re: [PATCH RESEND v3 1/3] tracing/synthetic: use union instead of casts

kernel test robot noticed "BUG:unable_to_handle_page_fault_for_address" on:

Hello,

kernel test robot noticed "BUG:unable_to_handle_page_fault_for_address" on:

commit: e2c9745169808b98f235c09d1366cf8b53ce0d3c ("[PATCH RESEND v3 1/3] tracing/synthetic: use union instead of casts")
url: https://github.com/intel-lab-lkp/linux/commits/Sven-Schnelle/tracing-synthetic-use-union-instead-of-casts/20230817-002758
base: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git 4853c74bd7ab7fdb83f319bd9ace8a08c031e9b6 patch link: https://lore.kernel.org/all/[email protected]/
patch subject: [PATCH RESEND v3 1/3] tracing/synthetic: use union instead of casts

in testcase: boot

compiler: clang-16
test machine: qemu-system-i386 -enable-kvm -cpu SandyBridge -smp 2 -m 4G

(please refer to attached dmesg/kmsg for entire log/backtrace)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]

[   64.204176][   T48] Dumping ftrace buffer:
[   64.204632][   T48] ---------------------------------
[   64.205082][   T48] BUG: unable to handle page fault for address: 001c24ca
[   64.205641][   T48] #PF: supervisor read access in kernel mode
[   64.206115][   T48] #PF: error_code(0x0000) - not-present page
[   64.206588][   T48] *pde = 00000000
[   64.206897][   T48] Oops: 0000 [#1] SMP
[   64.207221][   T48] CPU: 0 PID: 48 Comm: rcu_scale_write Tainted: G                T  6.5.0-rc6-00037-ge2c974516980 #1
[   64.208074][   T48] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
[   64.208892][   T48] EIP: string+0x98/0x100
[   64.209237][   T48] Code: 94 c2 75 b4 a1 8c c2 75 b4 40 89 44 24 18 31 d2 eb 0e 47 89 3d 94 c2 75 b4 42 39 54 24 0c 74 32 8b 44 24 08 8d 0c 10 8b 45 08 <0f> b6 04
 10 84 c0 74 2c 8b 74 24 18 01 d6 89 35 8c c2 75 b4 3b 0c
[   64.210761][   T48] EAX: 001c24ca EBX: 000405c2 ECX: b516e406 EDX: 00000000
[   64.211325][   T48] ESI: ffff0a00 EDI: 00005af2 EBP: b600bcd4 ESP: b600bca8
[   64.211876][   T48] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00010046
[   64.212468][   T48] CR0: 80050033 CR2: 001c24ca CR3: 08d2a000 CR4: 000406d0
[   64.213022][   T48] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[   64.213577][   T48] DR6: fffe0ff0 DR7: 00000400
[   64.213945][   T48] Call Trace:
[   64.214208][   T48]  ? __die_body+0x69/0xc0
[   64.214546][   T48]  ? __die+0x7e/0x90
[   64.214863][   T48]  ? page_fault_oops+0x22d/0x2b0
[   64.215242][   T48]  ? is_prefetch+0x4a/0x220
[   64.215605][   T48]  ? kernelmode_fixup_or_oops+0xfa/0x140
[   64.216051][   T48]  ? __bad_area_nosemaphore+0x64/0x2a0
[   64.216485][   T48]  ? bad_area_nosemaphore+0x12/0x20
[   64.216899][   T48]  ? do_user_addr_fault+0x650/0x7e0
[   64.217313][   T48]  ? pvclock_clocksource_read_nowd+0x7c/0x230
[   64.217800][   T48]  ? exc_page_fault+0xc5/0x358
[   64.218183][   T48]  ? pvclock_clocksource_read_nowd+0x230/0x230
[   64.218666][   T48]  ? handle_exception+0x14b/0x14b
[   64.219059][   T48]  ? number+0x9b/0x630
[   64.219378][   T48]  ? pvclock_clocksource_read_nowd+0x230/0x230
[   64.219866][   T48]  ? string+0x98/0x100
[   64.220187][   T48]  ? pvclock_clocksource_read_nowd+0x230/0x230
[   64.220656][   T48]  ? string+0x98/0x100
[   64.220986][   T48]  vsnprintf+0x420/0x580
[   64.221323][   T48]  ? vsnprintf+0x3e7/0x580
[   64.221669][   T48]  seq_buf_vprintf+0x79/0xc0
[   64.222029][   T48]  trace_seq_printf+0x35/0xa0
[   64.222400][   T48]  print_synth_event+0x26f/0x300
[   64.222805][   T48]  ? trace_event_raw_event_synth+0x410/0x410
[   64.223272][   T48]  print_trace_fmt+0xfe/0x170
[   64.223651][   T48]  print_trace_line+0x10d/0x1c0
[   64.224046][   T48]  ftrace_dump+0x2fa/0x410
[   64.224407][   T48]  rcu_scale_writer+0x59c/0x640
[   64.224808][   T48]  kthread+0x158/0x170
[   64.225146][   T48]  ? rcu_scale_reader+0x180/0x180
[   64.225559][   T48]  ? kthread_unuse_mm+0x160/0x160
[   64.225968][   T48]  ? kthread_unuse_mm+0x160/0x160
[   64.226379][   T48]  ret_from_fork+0x43/0x70
[   64.226742][   T48]  ret_from_fork_asm+0x12/0x1c
[   64.227130][   T48]  entry_INT80_32+0x108/0x108
[   64.227516][   T48] Modules linked in:
[   64.227832][   T48] CR2: 00000000001c24ca
[   64.228166][   T48] ---[ end trace 0000000000000000 ]---
[   64.228586][   T48] EIP: string+0x98/0x100
[   64.228921][   T48] Code: 94 c2 75 b4 a1 8c c2 75 b4 40 89 44 24 18 31 d2 eb 0e 47 89 3d 94 c2 75 b4 42 39 54 24 0c 74 32 8b 44 24 08 8d 0c 10 8b 45 08 <0f> b6 04
 10 84 c0 74 2c 8b 74 24 18 01 d6 89 35 8c c2 75 b4 3b 0c
[   64.230401][   T48] EAX: 001c24ca EBX: 000405c2 ECX: b516e406 EDX: 00000000
[   64.230918][   T48] ESI: ffff0a00 EDI: 00005af2 EBP: b600bcd4 ESP: b600bca8
[   64.231432][   T48] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00010046
[   64.231993][   T48] CR0: 80050033 CR2: 001c24ca CR3: 08d2a000 CR4: 000406d0
[   64.232527][   T48] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[   64.233094][   T48] DR6: fffe0ff0 DR7: 00000400
[   64.233474][   T48] Kernel panic - not syncing: Fatal exception
[   64.234050][   T48] Kernel Offset: disabled

The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20230825/[email protected]

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#3 lore raw

PATCH RESEND v3 2/3 tracing/synthetic: skip first entry for stack traces

In reply to Sven Schnelle

While debugging another issue i noticed that the stack trace output contains the number of entries on top:

         <idle>-0       [000] d..4.   203.322502: wake_lat: pid=0 delta=2268270616 stack=STACK:

=> 0x10
=> __schedule+0xac6/0x1a98
=> schedule+0x126/0x2c0
=> schedule_timeout+0x242/0x2c0
=> __wait_for_common+0x434/0x680
=> __wait_rcu_gp+0x198/0x3e0
=> synchronize_rcu+0x112/0x138
=> ring_buffer_reset_online_cpus+0x140/0x2e0
=> tracing_reset_online_cpus+0x15c/0x1d0
=> tracing_set_clock+0x180/0x1d8
=> hist_register_trigger+0x486/0x670
=> event_hist_trigger_parse+0x494/0x1318
=> trigger_process_regex+0x1d4/0x258
=> event_trigger_write+0xb4/0x170
=> vfs_write+0x210/0xad0
=> ksys_write+0x122/0x208

Fix this by skipping the first element. Also replace the pointer logic with an index variable which is easier to read.

Fixes 00cf3d672a9d ("tracing: Allow synthetic events to pass around stacktraces")
---
 kernel/trace/trace_events_synth.c | 17 ++++-------------
 1 file changed, 4 insertions(+), 13 deletions(-)
kernel/trace/trace_events_synth.c +4 -13
index 7fff8235075f..80a2a832f857 100644
@@ -350,7 +350,7 @@ static enum print_line_t print_synth_event(struct trace_iterator *iter,
 	struct trace_seq *s = &iter->seq;
 	struct synth_trace_event *entry;
 	struct synth_event *se;
-	unsigned int i, n_u64;
+	unsigned int i, j, n_u64;
 	char print_fmt[32];
 	const char *fmt;
 
@@ -389,18 +389,13 @@ static enum print_line_t print_synth_event(struct trace_iterator *iter,
 				n_u64 += STR_VAR_LEN_MAX / sizeof(u64);
 			}
 		} else if (se->fields[i]->is_stack) {
-			unsigned long *p, *end;
 			union trace_synth_field *data = &entry->fields[n_u64];
-
-			p = (void *)entry + data->as_dynamic.offset;
-			end = (void *)p + data->as_dynamic.len - (sizeof(long) - 1);
+			unsigned long *p = (void *)entry + data->as_dynamic.offset;
 
 			trace_seq_printf(s, "%s=STACK:\n", se->fields[i]->name);
-
-			for (; *p && p < end; p++)
-				trace_seq_printf(s, "=> %pS\n", (void *)*p);
+			for (j = 1; j < data->as_dynamic.len / sizeof(long); j++)
+				trace_seq_printf(s, "=> %pS\n", (void *)p[j]);
 			n_u64++;
-
 		} else {
 			struct trace_print_flags __flags[] = {
 			    __def_gfpflag_names, {-1, NULL} };
@@ -490,10 +485,6 @@ static unsigned int trace_stack(struct synth_trace_event *entry,
 			break;
 	}
 
-	/* Include the zero'd element if it fits */
-	if (len < HIST_STACKTRACE_DEPTH)
-		len++;
-
 	len *= sizeof(long);
 
 	/* Find the dynamic section to copy the stack into. */
Signature · 1 line
2.39.2
#4 lore raw

PATCH RESEND v3 3/3 tracing/synthetic: allocate one additional element for size

In reply to Sven Schnelle

While debugging another issue i noticed that the stack trace contains one invalid entry at the end:

<idle>-0 [008] d..4. 26.484201: wake_lat: pid=0 delta=2629976084 000000009cc24024 stack=STACK:
=> __schedule+0xac6/0x1a98
=> schedule+0x126/0x2c0
=> schedule_timeout+0x150/0x2c0
=> kcompactd+0x9ca/0xc20
=> kthread+0x2f6/0x3d8
=> __ret_from_fork+0x8a/0xe8
=> 0x6b6b6b6b6b6b6b6b

This is because the code failed to add the one element containing the number of entries to field_size.

Fixes 00cf3d672a9d ("tracing: Allow synthetic events to pass around stacktraces")
---
 kernel/trace/trace_events_synth.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/trace/trace_events_synth.c +2 -1
index 80a2a832f857..9897d0bfcab7 100644
@@ -528,7 +528,8 @@ static notrace void trace_event_raw_event_synth(void *__data,
 		str_val = (char *)(long)var_ref_vals[val_idx];
 
 		if (event->dynamic_fields[i]->is_stack) {
-			len = *((unsigned long *)str_val);
+			/* reserve one extra element for size */
+			len = *((unsigned long *)str_val) + 1;
 			len *= sizeof(unsigned long);
 		} else {
 			len = fetch_store_strlen((unsigned long)str_val);
Signature · 1 line
2.39.2
maintainer
#5 lore raw
In reply to Sven Schnelle
Quoted 4 lines · On Wed, 16 Aug 2023 17:49:25 +0200 Sven Schnelle <[email protected]> wrote:

Hi Steven,

I'm now sending these patches in one patchset, because the second patch has a dependeny on the union vs. cast fix.

Thanks. I'm currently waiting on some other fixes before running them all through my tests before sending them off to Linus. If they are not ready by tomorrow, I'll just kick off my tests without them.

-- Steve

Quoted 18 lines

Changes in v3:
- remove superfluous struct around union trace_synth_field

Changes in v2:
- cosmetic changes
- add struct trace_dynamic_info to include/linux/trace_events.h

Sven Schnelle (3):
tracing/synthetic: use union instead of casts tracing/synthetic: skip first entry for stack traces tracing/synthetic: allocate one additional element for size

 include/linux/trace_events.h      |  11 ++++
 kernel/trace/trace.h              |   8 +++
 kernel/trace/trace_events_synth.c | 103 ++++++++++++------------------
 3 files changed, 60 insertions(+), 62 deletions(-)