Skip to contentSkip to conversation

tracing: Minor fixes for 6.6

PATCH New Steven Rostedt 1 participant 12 messages Sep 2, 2023

Subsystems (5)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

cover tracing: Minor fixes for 6.6

Tracing fixes and clean ups for 6.6:

- Replace strlcpy() with strscpy()

- Initialize the pipe cpumask to zero on allocation

- Use within_module() instead of open coding it

- Remove extra space in hwlat_detectory/mode output

- Use LIST_HEAD() instead of open coding it

- A bunch of clean ups and fixes for the cpumask filter

- Set local da_mon_##name to static

- Fix race in snapshot buffer between cpu write and swap

git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace.git
trace/urgent

Head SHA1: cbb557ba92f08b945e2cb20b7ab37ef49ab53cdd

Azeem Shaikh (1):
tracing: Replace strlcpy with strscpy in trace/events/task.h

Brian Foster (1):
tracing: Zero the pipe cpumask on alloc to avoid spurious -EBUSY

Levi Yun (1):

      ftrace: Use within_module to check rec->ip within specified module.

Mikhail Kobuk (1):
tracing: Remove extra space at the end of hwlat_detector/mode

Ruan Jinjie (1):
ftrace: Use LIST_HEAD to initialize clear_hash

Valentin Schneider (4):
tracing/filters: Fix error-handling of cpulist parsing buffer tracing/filters: Fix double-free of struct filter_pred.mask tracing/filters: Change parse_pred() cpulist ternary into an if block tracing/filters: Fix coding style issues

Yu Liao (1):
rv: Set variable 'da_mon_##name' to static

Zheng Yejian (1):
tracing: Fix race issue between cpu buffer write and swap

----

 include/rv/da_monitor.h            |  2 +-
 include/trace/events/task.h        |  2 +-
 kernel/trace/ftrace.c              | 10 +++-------
 kernel/trace/trace.c               | 21 ++++++++++++++-------
 kernel/trace/trace_events_filter.c | 25 +++++++++++++++++++------
 kernel/trace/trace_hwlat.c         |  2 +-
 6 files changed, 39 insertions(+), 23 deletions(-)
author maintainer
#1 lore raw

PATCH 1/11 Set variable da_mon_##name to static

From: Yu Liao <[email protected]>

gcc with W=1 reports
kernel/trace/rv/monitors/wip/wip.c:20:1: sparse: sparse: symbol 'da_mon_wip' was not declared. Should it be static?

The per-cpu variable 'da_mon_##name' is only used in its defining file, so it should be static.

---
 include/rv/da_monitor.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/rv/da_monitor.h +1 -1
index 9eb75683e012..9705b2a98e49 100644
@@ -262,7 +262,7 @@ static inline void da_monitor_destroy_##name(void)						\
 /*												\
  * per-cpu monitor variables									\
  */												\
-DEFINE_PER_CPU(struct da_monitor, da_mon_##name);						\
+static DEFINE_PER_CPU(struct da_monitor, da_mon_##name);					\
 												\
 /*												\
  * da_get_monitor_##name - return current CPU monitor address					\
Signature · 1 line
2.40.1
author maintainer
#2 lore raw

PATCH 2/11 tracing: Remove extra space at the end of hwlat_detector/mode

In reply to Steven Rostedt

From: Mikhail Kobuk <[email protected]>

Space is printed after each mode value including the last one:

$ echo \"$(sudo cat /sys/kernel/tracing/hwlat_detector/mode)\"
"none [round-robin] per-cpu "

Found by Linux Verification Center (linuxtesting.org) with SVACE.

Cc Masami Hiramatsu <[email protected]>
Fixes 8fa826b7344d ("trace/hwlat: Implement the mode config option")
---
 kernel/trace/trace_hwlat.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/trace/trace_hwlat.c +1 -1
index 2f37a6e68aa9..b791524a6536 100644
@@ -635,7 +635,7 @@ static int s_mode_show(struct seq_file *s, void *v)
 	else
 		seq_printf(s, "%s", thread_mode_str[mode]);
 
-	if (mode != MODE_MAX)
+	if (mode < MODE_MAX - 1) /* if mode is any but last */
 		seq_puts(s, " ");
 
 	return 0;
Signature · 1 line
2.40.1
author maintainer
#3 lore raw

PATCH 3/11 tracing: Fix race issue between cpu buffer write and swap

In reply to Steven Rostedt

From: Zheng Yejian <[email protected]>

Warning happened in rb_end_commit() at code:

	if (RB_WARN_ON(cpu_buffer, !local_read(&cpu_buffer->committing)))
  WARNING: CPU: 0 PID: 139 at kernel/trace/ring_buffer.c:3142
	rb_commit+0x402/0x4a0
  Call Trace:
   ring_buffer_unlock_commit+0x42/0x250
   trace_buffer_unlock_commit_regs+0x3b/0x250
   trace_event_buffer_commit+0xe5/0x440
   trace_event_buffer_reserve+0x11c/0x150
   trace_event_raw_event_sched_switch+0x23c/0x2c0
   __traceiter_sched_switch+0x59/0x80
   __schedule+0x72b/0x1580
   schedule+0x92/0x120
   worker_thread+0xa0/0x6f0

It is because the race between writing event into cpu buffer and swapping cpu buffer through file per_cpu/cpu0/snapshot:

Write on CPU 0 Swap buffer by per_cpu/cpu0/snapshot on CPU 1
-------- --------

                             tracing_snapshot_write()
                               [...]
  ring_buffer_lock_reserve()
    cpu_buffer = buffer->buffers[cpu]; // 1. Suppose find 'cpu_buffer_a';
    [...]
    rb_reserve_next_event()
      [...]
                               ring_buffer_swap_cpu()
                                 if (local_read(&cpu_buffer_a->committing))
                                     goto out_dec;
                                 if (local_read(&cpu_buffer_b->committing))
                                     goto out_dec;
                                 buffer_a->buffers[cpu] = cpu_buffer_b;
                                 buffer_b->buffers[cpu] = cpu_buffer_a;
                                 // 2. cpu_buffer has swapped here.

      rb_start_commit(cpu_buffer);
      if (unlikely(READ_ONCE(cpu_buffer->buffer)
          != buffer)) { // 3. This check passed due to 'cpu_buffer->buffer'
        [...]           //    has not changed here.
        return NULL;
      }
                                 cpu_buffer_b->buffer = buffer_a;
                                 cpu_buffer_a->buffer = buffer_b;
                                 [...]

      // 4. Reserve event from 'cpu_buffer_a'.

ring_buffer_unlock_commit()

    [...]
    cpu_buffer = buffer->buffers[cpu]; // 5. Now find 'cpu_buffer_b' !!!
    rb_commit(cpu_buffer)
      rb_end_commit()  // 6. WARN for the wrong 'committing' state !!!

Based on above analysis, we can easily reproduce by following testcase:

  ``` bash
  #!/bin/bash

  dmesg -n 7
  sysctl -w kernel.panic_on_warn=1
  TR=/sys/kernel/tracing
  echo 7 > ${TR}/buffer_size_kb
  echo "sched:sched_switch" > ${TR}/set_event
  while [ true ]; do
          echo 1 > ${TR}/per_cpu/cpu0/snapshot
  done &
  while [ true ]; do
          echo 1 > ${TR}/per_cpu/cpu0/snapshot
  done &
  while [ true ]; do
          echo 1 > ${TR}/per_cpu/cpu0/snapshot
  done &
  ```

To fix it, IIUC, we can use smp_call_function_single() to do the swap on the target cpu where the buffer is located, so that above race would be avoided.

Fixes f1affcaaa861 ("tracing: Add snapshot in the per_cpu trace directories")
---
 kernel/trace/trace.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
kernel/trace/trace.c +12 -5
index 3e55375f47e0..23579fba1a57 100644
@@ -7599,6 +7599,11 @@ static int tracing_snapshot_open(struct inode *inode, struct file *file)
 	return ret;
 }
 
+static void tracing_swap_cpu_buffer(void *tr)
+{
+	update_max_tr_single((struct trace_array *)tr, current, smp_processor_id());
+}
+
 static ssize_t
 tracing_snapshot_write(struct file *filp, const char __user *ubuf, size_t cnt,
 		       loff_t *ppos)
@@ -7657,13 +7662,15 @@ tracing_snapshot_write(struct file *filp, const char __user *ubuf, size_t cnt,
 			ret = tracing_alloc_snapshot_instance(tr);
 		if (ret < 0)
 			break;
-		local_irq_disable();
 		/* Now, we're going to swap */
-		if (iter->cpu_file == RING_BUFFER_ALL_CPUS)
+		if (iter->cpu_file == RING_BUFFER_ALL_CPUS) {
+			local_irq_disable();
 			update_max_tr(tr, current, smp_processor_id(), NULL);
-		else
-			update_max_tr_single(tr, current, iter->cpu_file);
-		local_irq_enable();
+			local_irq_enable();
+		} else {
+			smp_call_function_single(iter->cpu_file, tracing_swap_cpu_buffer,
+						 (void *)tr, 1);
+		}
 		break;
 	default:
 		if (tr->allocated_snapshot) {
Signature · 1 line
2.40.1
author maintainer
#4 lore raw

PATCH 4/11 tracing: Replace strlcpy with strscpy in trace/events/task.h

In reply to Steven Rostedt

From: Azeem Shaikh <[email protected]>

strlcpy() reads the entire source buffer first. This read may exceed the destination size limit. This is both inefficient and can lead to linear read overflows if a source string is not NUL-terminated [1]. In an effort to remove strlcpy() completely [2], replace strlcpy() here with strscpy().

No return values were used, so direct replacement is safe.

[1] https://www.kernel.org/doc/html/latest/process/deprecated.html#strlcpy
[2] https://github.com/KSPP/linux/issues/89

---
 include/trace/events/task.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/trace/events/task.h +1 -1
index 64d160930b0d..47b527464d1a 100644
@@ -47,7 +47,7 @@ TRACE_EVENT(task_rename,
 	TP_fast_assign(
 		__entry->pid = task->pid;
 		memcpy(entry->oldcomm, task->comm, TASK_COMM_LEN);
-		strlcpy(entry->newcomm, comm, TASK_COMM_LEN);
+		strscpy(entry->newcomm, comm, TASK_COMM_LEN);
 		__entry->oom_score_adj = task->signal->oom_score_adj;
 	),
 
Signature · 1 line
2.40.1
author maintainer
#5 lore raw

PATCH 5/11 ftrace: Use within_module to check rec->ip within specified module.

In reply to Steven Rostedt

From: Levi Yun <[email protected]>

within_module_core && within_module_init condition is same to within module but it's more readable.

Use within_module instead of former condition to check rec->ip within specified module area or not.

---
 kernel/trace/ftrace.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
kernel/trace/ftrace.c +2 -4
index 05c0024815bf..c46dd6d97afe 100644
@@ -6779,8 +6779,7 @@ void ftrace_release_mod(struct module *mod)
 	last_pg = &ftrace_pages_start;
 	for (pg = ftrace_pages_start; pg; pg = *last_pg) {
 		rec = &pg->records[0];
-		if (within_module_core(rec->ip, mod) ||
-		    within_module_init(rec->ip, mod)) {
+		if (within_module(rec->ip, mod)) {
 			/*
 			 * As core pages are first, the first
 			 * page should never be a module page.
@@ -6852,8 +6851,7 @@ void ftrace_module_enable(struct module *mod)
 		 * not part of this module, then skip this pg,
 		 * which the "break" will do.
 		 */
-		if (!within_module_core(rec->ip, mod) &&
-		    !within_module_init(rec->ip, mod))
+		if (!within_module(rec->ip, mod))
 			break;
 
 		/* Weak functions should still be ignored */
Signature · 1 line
2.40.1
author maintainer
#6 lore raw

PATCH 6/11 ftrace: Use LIST_HEAD to initialize clear_hash

In reply to Steven Rostedt

From: Ruan Jinjie <[email protected]>

Use LIST_HEAD() to initialize clear_hash instead of open-coding it.

---
 kernel/trace/ftrace.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
kernel/trace/ftrace.c +1 -3
index c46dd6d97afe..8de8bec5f366 100644
@@ -7140,9 +7140,7 @@ void ftrace_free_mem(struct module *mod, void *start_ptr, void *end_ptr)
 	struct dyn_ftrace key;
 	struct ftrace_mod_map *mod_map = NULL;
 	struct ftrace_init_func *func, *func_next;
-	struct list_head clear_hash;
-
-	INIT_LIST_HEAD(&clear_hash);
+	LIST_HEAD(clear_hash);
 
 	key.ip = start;
 	key.flags = end;	/* overload flags, as it is unsigned long */
Signature · 1 line
2.40.1
author maintainer
#7 lore raw

PATCH 7/11 tracing: Zero the pipe cpumask on alloc to avoid spurious -EBUSY

In reply to Steven Rostedt

From: Brian Foster <[email protected]>

The pipe cpumask used to serialize opens between the main and percpu trace pipes is not zeroed or initialized. This can result in spurious -EBUSY returns if underlying memory is not fully zeroed. This has been observed by immediate failure to read the main trace_pipe file on an otherwise newly booted and idle system:

 # cat /sys/kernel/debug/tracing/trace_pipe
 cat: /sys/kernel/debug/tracing/trace_pipe: Device or resource busy

Zero the allocation of pipe_cpumask to avoid the problem.

Fixes c2489bb7e6be ("tracing: Introduce pipe_cpumask to avoid race on trace_pipes")
---
 kernel/trace/trace.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/trace/trace.c +2 -2
index 23579fba1a57..35783a7baf15 100644
@@ -9474,7 +9474,7 @@ static struct trace_array *trace_array_create(const char *name)
 	if (!alloc_cpumask_var(&tr->tracing_cpumask, GFP_KERNEL))
 		goto out_free_tr;
 
-	if (!alloc_cpumask_var(&tr->pipe_cpumask, GFP_KERNEL))
+	if (!zalloc_cpumask_var(&tr->pipe_cpumask, GFP_KERNEL))
 		goto out_free_tr;
 
 	tr->trace_flags = global_trace.trace_flags & ~ZEROED_TRACE_FLAGS;
@@ -10419,7 +10419,7 @@ __init static int tracer_alloc_buffers(void)
 	if (trace_create_savedcmd() < 0)
 		goto out_free_temp_buffer;
 
-	if (!alloc_cpumask_var(&global_trace.pipe_cpumask, GFP_KERNEL))
+	if (!zalloc_cpumask_var(&global_trace.pipe_cpumask, GFP_KERNEL))
 		goto out_free_savedcmd;
 
 	/* TODO: make the number of buffers hot pluggable with CPUS */
Signature · 1 line
2.40.1
author maintainer
#8 lore raw

PATCH 8/11 tracing/filters: Fix error-handling of cpulist parsing buffer

In reply to Steven Rostedt

From: Valentin Schneider <[email protected]>

parse_pred() allocates a string buffer to parse the user-provided cpulist, but doesn't check the allocation result nor does it free the buffer once it is no longer needed.

Add an allocation check, and free the buffer as soon as it is no longer needed.

---
 kernel/trace/trace_events_filter.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
kernel/trace/trace_events_filter.c +8 -2
index 3a529214a21b..c06e1d596f4b 100644
@@ -1744,17 +1744,23 @@ static int parse_pred(const char *str, void *data,
 
 		/* Copy the cpulist between { and } */
 		tmp = kmalloc((i - maskstart) + 1, GFP_KERNEL);
-		strscpy(tmp, str + maskstart, (i - maskstart) + 1);
+		if (!tmp)
+			goto err_mem;
 
+		strscpy(tmp, str + maskstart, (i - maskstart) + 1);
 		pred->mask = kzalloc(cpumask_size(), GFP_KERNEL);
-		if (!pred->mask)
+		if (!pred->mask) {
+			kfree(tmp);
 			goto err_mem;
+		}
 
 		/* Now parse it */
 		if (cpulist_parse(tmp, pred->mask)) {
+			kfree(tmp);
 			parse_error(pe, FILT_ERR_INVALID_CPULIST, pos + i);
 			goto err_free;
 		}
+		kfree(tmp);
 
 		/* Move along */
 		i++;
Signature · 1 line
2.40.1
author maintainer
#9 lore raw

PATCH 9/11 tracing/filters: Fix double-free of struct filter_pred.mask

In reply to Steven Rostedt

From: Valentin Schneider <[email protected]>

When a cpulist filter is found to contain a single CPU, that CPU is saved as a scalar and the backing cpumask storage is freed.

Also NULL the mask to avoid a double-free once we get down to free_predicate().

---
 kernel/trace/trace_events_filter.c | 1 +
 1 file changed, 1 insertion(+)
kernel/trace/trace_events_filter.c +1 -0
index c06e1d596f4b..eb331e8b00b6 100644
@@ -1773,6 +1773,7 @@ static int parse_pred(const char *str, void *data,
 		if (single) {
 			pred->val = cpumask_first(pred->mask);
 			kfree(pred->mask);
+			pred->mask = NULL;
 		}
 
 		if (field->filter_type == FILTER_CPUMASK) {
Signature · 1 line
2.40.1
author maintainer
#10 lore raw

PATCH 10/11 tracing/filters: Change parse_pred() cpulist ternary into an if block

In reply to Steven Rostedt

From: Valentin Schneider <[email protected]>

Review comments noted that an if block would be clearer than a ternary, so swap it out.

No change in behaviour intended

---
 kernel/trace/trace_events_filter.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
kernel/trace/trace_events_filter.c +6 -2
index eb331e8b00b6..09b4733a2933 100644
@@ -1782,13 +1782,17 @@ static int parse_pred(const char *str, void *data,
 				FILTER_PRED_FN_CPUMASK;
 		} else if (field->filter_type == FILTER_CPU) {
 			if (single) {
-				pred->op = pred->op == OP_BAND ? OP_EQ : pred->op;
+				if (pred->op == OP_BAND)
+					pred->op = OP_EQ;
+
 				pred->fn_num = FILTER_PRED_FN_CPU;
 			} else {
 				pred->fn_num = FILTER_PRED_FN_CPU_CPUMASK;
 			}
 		} else if (single) {
-			pred->op = pred->op == OP_BAND ? OP_EQ : pred->op;
+			if (pred->op == OP_BAND)
+				pred->op = OP_EQ;
+
 			pred->fn_num = select_comparison_fn(pred->op, field->size, false);
 			if (pred->op == OP_NE)
 				pred->not = 1;
Signature · 1 line
2.40.1
author maintainer
#11 lore raw

PATCH 11/11 tracing/filters: Fix coding style issues

In reply to Steven Rostedt

From: Valentin Schneider <[email protected]>

Recent commits have introduced some coding style issues, fix those up.

---
 kernel/trace/trace_events_filter.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/trace/trace_events_filter.c +4 -2
index 09b4733a2933..33264e510d16 100644
@@ -1360,7 +1360,7 @@ int filter_assign_type(const char *type)
 			return FILTER_DYN_STRING;
 		if (strstr(type, "cpumask_t"))
 			return FILTER_CPUMASK;
-		}
+	}
 
 	if (strstr(type, "__rel_loc") && strstr(type, "char"))
 		return FILTER_RDYN_STRING;
@@ -1731,7 +1731,9 @@ static int parse_pred(const char *str, void *data,
 		maskstart = i;
 
 		/* Walk the cpulist until closing } */
-		for (; str[i] && str[i] != '}'; i++);
+		for (; str[i] && str[i] != '}'; i++)
+			;
+
 		if (str[i] != '}') {
 			parse_error(pe, FILT_ERR_MISSING_BRACE_CLOSE, pos + i);
 			goto err_free;
Signature · 1 line
2.40.1