Skip to contentSkip to conversation

tracing: of: Boot time tracing using devicetree

RFC PATCH In review Masami Hiramatsu 2 participants 19 messages Jul 15, 2019 – Jul 22, 2019

Subsystems (3)
lore
Share & tools
lore

Masami Hiramatsu

author maintainer
#0 lore raw

v2 cover tracing: of: Boot time tracing using devicetree

Hello,

Here is the 2nd version of RFC series to add boot-time tracing using devicetree. Previous thread is here.

https://lkml.kernel.org/r/156113387975.28344.16009584175308192243.stgit@devnote2

In this version, I moved the ftrace node under /chosen/linux,ftrace and remove compatible property, because it must be in fixed place. Also this version has following features;

- Introduce "instance" node, which can have events nodes for setting events filters and actions for the trace instance.
- Introduce "cpumask" property
- Introduce "ftrace-filters" and "ftrace-notraces"
- Introduce "fgraph-filters", "fgraph-notraces" and "fgraph-max-depth"

At this moment, this feature is only available on the architecutre which supports devicetree. For x86, we can use it on qemu with --dtb option, or apply below patch on grub to add devicetree support on grub-x86.

https://github.com/mhiramat/grub/commit/644c35bfd2d18c772cc353b74215344f8264923a

Note that the devicetree for x86 must contain the nodes only under /chosen/, or it may cause a problem if it conflicts with ACPI. (Maybe we need to disable the FDT nodes except for nodes under /chosen
on boot if ACPI exists.)

This series can be applied on Steve's tracing tree (ftrace/core) or available on below

https://github.com/mhiramat/linux.git ftrace-devicetree-v2

Usage
======
With this series, we can setup new kprobe and synthetic events, more complicated event filters and trigger actions including histogram via devicetree.

For example, following kernel parameters

trace_options=sym-addr trace_event=initcall:* tp_printk trace_buf_size=1M ftrace=function ftrace_filter="vfs*"

it can be written in devicetree like below.

/{

chosen {
	...
	ftrace {
		options = "sym-addr";
		events = "initcall:*";
		tp-printk;
		buffer-size-kb = <0x400>;	// 1024KB == 1MB
		ftrace-filters = "vfs*";
	};

Moreover, now we can expand it to add filters for events, kprobe events, and synthetic events with histogram like below.

	ftrace {
		...
		event0 {
			event = "task:task_newtask";
			filter = "pid < 128";	// adding filters
			enable;
		};
		event1 {
			event = "kprobes:vfs_read";
			probes = "vfs_read $arg1 $arg2"; // add kprobes
			filter = "common_pid < 200";
			enable;
		};
		event2 {
			event = "initcall_latency";	// add synth event
			fields = "unsigned long func", "u64 lat";
			// with histogram
			actions = "hist:keys=func.sym,lat:vals=lat:sort=lat";
		};
		// and synthetic event callers
		event3 {
			event = "initcall:initcall_start";
			actions = "hist:keys=func:ts0=common_timestamp.usecs";
		};
		event4 {
			event = "initcall:initcall_finish";
			actions = "hist:keys=func:lat=common_timestamp.usecs-$ts0:onmatch(initcall.initcall_start).initcall_latency(func,$lat)";
		};
	};

Also, this version supports "instance" node, which allows us to run several tracers for different purpose at once. For example, one tracer is for tracing functions in module alpha, and others tracing module beta, you can write followings.

	ftrace {
		instance0 {
			instance = "foo";
			tracer = "function";
			ftrace-filters = "*:mod:alpha";
		};
		instance1 {
			instance = "bar";
			tracer = "function";
			ftrace-filters = "*:mod:beta";
			
		};
	};

The instance node also accepts event nodes so that each instance can customize its event tracing.

Discussion
=====
On the previous thread, we discussed that the this devicetree usage itself was acceptable or not. Fortunately, I had a chance to discuss it in a F2F meeting with Frank and Tim last week.

I think the advantages of using devicetree are,

- reuse devicetree's structured syntax for complicated tracefs settings
- reuse OF-APIs in linux kernel to accept and parse it
- reuse dtc complier to compile it and validate syntax. (with yaml schema, we can enhance it)
- reuse current bootloader (and qemu) to load it

And we talked about some other ideas to avoid using devicetree.

- expand kernel command line (ascii command strings)
- expand kernel command line with base64 encoded comressed ascii command strings
- load (compressed) ascii command strings to somewhere on memory and pass the address via kernel cmdline
- load (compressed) ascii command strings to somewhere on memory and pass the address via /chosen node (as same as initrd)
- load binary C data and point it from kernel cmdline
- load binary C data and point it from /chosen node (as same as initrd)
- load binary C data as a section of kernel image

The first 2 ideas expand the kernel's cmdline to pass some "magic" command to setup ftrace. In both case, the problems are the maximal size of cmdline and the issues related to the complexity of commands.
My example showed that the ftrace settings becomes long even if making one histogram, which can be longer than 256 bytes. The long and complex data can easily lead mis-typing, but cmdline has no syntax validator, it just ignores the mis-typed commands.
(Of course even with the devicetree, it must be smaller than 2 pages)

Next 2 ideas are similar, but load the commands on some other memory area and pass only address via cmdline. This solves the size limitation issue, but still no syntax validation. Of course we can make a new structured syntax validator similar to (or just forked from) dt-validate. The problem (or disadvantage) of these (and following) ideas, is to change the kernel and boot loaders to load another binary blobs on memory.

Maybe if we introduce a generic structured kernel boot arguments, which is a kind of /chosen node of devicetree. (But if there is already such hook, why we make another one...?)
Also, this "GSKBA" may introduce a parser and access APIs which will be very similar to OF-APIs. This also seems redundant to me.

So the last 3 ideas will avoid introducing new parser and APIs, we just compile the data as C data and point it from cmdline or somewhere else. With these ideas, we still need to expand boot loaders to support loading new binary blobs. (And the last one requires to add elf header parser/modifier to boot loader too)

Quoted 1 line

From the above reasons, I think using devicetree's /chosen node is

the least intrusive way to introduce this boot-time tracing feature.

Any suggestions, thoughts?

Thank you,

---

Masami Hiramatsu (15):
tracing: Apply soft-disabled and filter to tracepoints printk
tracing: kprobes: Output kprobe event to printk buffer
tracing: Expose EXPORT_SYMBOL_GPL symbol
tracing: kprobes: Register to dynevent earlier stage
tracing: Accept different type for synthetic event fields
tracing: Add NULL trace-array check in print_synth_event()
dt-bindings: tracing: Add ftrace binding document
tracing: of: Add setup tracing by devicetree support
tracing: of: Add trace event settings
tracing: of: Add kprobe event support
tracing: of: Add synthetic event support
tracing: of: Add instance node support
tracing: of: Add cpumask property support
tracing: of: Add function tracer filter properties
tracing: of: Add function-graph tracer option properties

 .../devicetree/bindings/chosen/linux,ftrace.yaml   |  306 ++++++++++++
 include/linux/trace_events.h                       |    1 
 kernel/trace/Kconfig                               |   10 
 kernel/trace/Makefile                              |    1 
 kernel/trace/ftrace.c                              |   85 ++-
 kernel/trace/trace.c                               |   90 ++--
 kernel/trace/trace_events.c                        |    3 
 kernel/trace/trace_events_hist.c                   |   14 -
 kernel/trace/trace_events_trigger.c                |    2 
 kernel/trace/trace_kprobe.c                        |   81 ++-
 kernel/trace/trace_of.c                            |  507 ++++++++++++++++++++
 11 files changed, 1004 insertions(+), 96 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/chosen/linux,ftrace.yaml
 create mode 100644 kernel/trace/trace_of.c

--
Masami Hiramatsu (Linaro) <[email protected]>

author maintainer
#1 lore raw

RFC PATCH v2 1/15 tracing: Apply soft-disabled and filter to tracepoints printk

Apply soft-disabled and the filter rule of the trace events to the printk output of tracepoints (a.k.a. tp_printk kernel parameter) as same as trace buffer output.

---
 kernel/trace/trace.c |    7 +++++++
 1 file changed, 7 insertions(+)
kernel/trace/trace.c +7 -0
index 6b62e1718548..047f37816bf1 100644
@@ -2502,6 +2502,7 @@ static DEFINE_MUTEX(tracepoint_printk_mutex);
 static void output_printk(struct trace_event_buffer *fbuffer)
 {
 	struct trace_event_call *event_call;
+	struct trace_event_file *file;
 	struct trace_event *event;
 	unsigned long flags;
 	struct trace_iterator *iter = tracepoint_print_iter;
@@ -2515,6 +2516,12 @@ static void output_printk(struct trace_event_buffer *fbuffer)
 	    !event_call->event.funcs->trace)
 		return;
 
+	file = fbuffer->trace_file;
+	if (test_bit(EVENT_FILE_FL_SOFT_DISABLED_BIT, &file->flags) ||
+	    (unlikely(file->flags & EVENT_FILE_FL_FILTERED) &&
+	     !filter_match_preds(file->filter, fbuffer->entry)))
+		return;
+
 	event = &fbuffer->trace_file->event_call->event;
 
 	spin_lock_irqsave(&tracepoint_iter_lock, flags);
author maintainer
#2 lore raw

RFC PATCH v2 2/15 tracing: kprobes: Output kprobe event to printk buffer

In reply to Masami Hiramatsu

Since kprobe-events use event_trigger_unlock_commit_regs() directly, that events doesn't show up in printk buffer if "tp_printk" is set.

Use trace_event_buffer_commit() in kprobe events so that it can invoke output_printk() as same as other trace events.

---
 include/linux/trace_events.h |    1 +
 kernel/trace/trace.c         |    4 +--
 kernel/trace/trace_events.c  |    1 +
 kernel/trace/trace_kprobe.c  |   57 +++++++++++++++++++++---------------------
 4 files changed, 32 insertions(+), 31 deletions(-)
include/linux/trace_events.h +1 -0
index 5150436783e8..8912ccdb3d4b 100644
@@ -211,6 +211,7 @@ struct trace_event_buffer {
 	void				*entry;
 	unsigned long			flags;
 	int				pc;
+	struct pt_regs			*regs;
 };
 
 void *trace_event_buffer_reserve(struct trace_event_buffer *fbuffer,
kernel/trace/trace.c +2 -2
index 047f37816bf1..30f09c058f79 100644
@@ -2572,9 +2572,9 @@ void trace_event_buffer_commit(struct trace_event_buffer *fbuffer)
 	if (static_key_false(&tracepoint_printk_key.key))
 		output_printk(fbuffer);
 
-	event_trigger_unlock_commit(fbuffer->trace_file, fbuffer->buffer,
+	event_trigger_unlock_commit_regs(fbuffer->trace_file, fbuffer->buffer,
 				    fbuffer->event, fbuffer->entry,
-				    fbuffer->flags, fbuffer->pc);
+				    fbuffer->flags, fbuffer->pc, fbuffer->regs);
 }
 EXPORT_SYMBOL_GPL(trace_event_buffer_commit);
 
kernel/trace/trace_events.c +1 -0
index c7506bc81b75..22cf08bd2317 100644
@@ -271,6 +271,7 @@ void *trace_event_buffer_reserve(struct trace_event_buffer *fbuffer,
 	if (!fbuffer->event)
 		return NULL;
 
+	fbuffer->regs = NULL;
 	fbuffer->entry = ring_buffer_event_data(fbuffer->event);
 	return fbuffer->entry;
 }
kernel/trace/trace_kprobe.c +28 -29

97 lines not on this page · raw message

author maintainer
#3 lore raw

RFC PATCH v2 3/15 tracing: Expose EXPORT_SYMBOL_GPL symbol

In reply to Masami Hiramatsu

Since ftrace_set_clr_event is already exported by EXPORT_SYMBOL_GPL, it should not be static.

---
 kernel/trace/trace_events.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/trace/trace_events.c +1 -1
index 22cf08bd2317..c2d38048edae 100644
@@ -788,7 +788,7 @@ static int __ftrace_set_clr_event(struct trace_array *tr, const char *match,
 	return ret;
 }
 
-static int ftrace_set_clr_event(struct trace_array *tr, char *buf, int set)
+int ftrace_set_clr_event(struct trace_array *tr, char *buf, int set)
 {
 	char *event = NULL, *sub = NULL, *match;
 	int ret;
author maintainer
#4 lore raw

RFC PATCH v2 4/15 tracing: kprobes: Register to dynevent earlier stage

In reply to Masami Hiramatsu

Register kprobe event to dynevent in subsys_initcall level. This will allow kernel to register new kprobe events in fs_initcall level via trace_run_command.

---
 kernel/trace/trace_kprobe.c |   19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)
kernel/trace/trace_kprobe.c +15 -4
index 6c5145525f90..5135c07b6557 100644
@@ -1484,11 +1484,12 @@ static __init void setup_boot_kprobe_events(void)
 	enable_boot_kprobe_events();
 }
 
-/* Make a tracefs interface for controlling probe points */
-static __init int init_kprobe_trace(void)
+/*
+ * Register dynevent at subsys_initcall. This allows kernel to setup kprobe
+ * events in fs_initcall without tracefs.
+ */
+static __init int init_kprobe_trace_early(void)
 {
-	struct dentry *d_tracer;
-	struct dentry *entry;
 	int ret;
 
 	ret = dyn_event_register(&trace_kprobe_ops);
@@ -1498,6 +1499,16 @@ static __init int init_kprobe_trace(void)
 	if (register_module_notifier(&trace_kprobe_module_nb))
 		return -EINVAL;
 
+	return 0;
+}
+subsys_initcall(init_kprobe_trace_early);
+
+/* Make a tracefs interface for controlling probe points */
+static __init int init_kprobe_trace(void)
+{
+	struct dentry *d_tracer;
+	struct dentry *entry;
+
 	d_tracer = tracing_init_dentry();
 	if (IS_ERR(d_tracer))
 		return 0;
author maintainer
#5 lore raw

RFC PATCH v2 5/15 tracing: Accept different type for synthetic event fields

In reply to Masami Hiramatsu

Make the synthetic event accepts a different type field to record. However, the size and signed flag must be same.

---
 kernel/trace/trace_events_hist.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
kernel/trace/trace_events_hist.c +5 -2
index ca6b0dff60c5..a7f447195143 100644
@@ -4063,8 +4063,11 @@ static int check_synth_field(struct synth_event *event,
 
 	field = event->fields[field_pos];
 
-	if (strcmp(field->type, hist_field->type) != 0)
-		return -EINVAL;
+	if (strcmp(field->type, hist_field->type) != 0) {
+		if (field->size != hist_field->size ||
+		    field->is_signed != hist_field->is_signed)
+			return -EINVAL;
+	}
 
 	return 0;
 }
author maintainer
#6 lore raw

RFC PATCH v2 6/15 tracing: Add NULL trace-array check in print_synth_event()

In reply to Masami Hiramatsu

Add NULL trace-array check in print_synth_event(), because if we enable tp_printk option, iter->tr can be NULL.

---
 kernel/trace/trace_events_hist.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/trace/trace_events_hist.c +1 -1
index a7f447195143..db973928e580 100644
@@ -822,7 +822,7 @@ static enum print_line_t print_synth_event(struct trace_iterator *iter,
 		fmt = synth_field_fmt(se->fields[i]->type);
 
 		/* parameter types */
-		if (tr->trace_flags & TRACE_ITER_VERBOSE)
+		if (tr && tr->trace_flags & TRACE_ITER_VERBOSE)
 			trace_seq_printf(s, "%s ", fmt);
 
 		snprintf(print_fmt, sizeof(print_fmt), "%%s=%s%%s", fmt);
author maintainer
#7 lore raw

RFC PATCH v2 7/15 dt-bindings: tracing: Add ftrace binding document

In reply to Masami Hiramatsu

Add a devicetree binding document for ftrace node.

---
Changes in v2:
- Add cpumask, ftrace-filters, ftrace-notraces, fgraph-filters, fgraph-notraces, fgraph-max-depth and instance node.
- Remove compatible and move file to bindings/chosen/linux,ftrace.yaml

---
 .../devicetree/bindings/chosen/linux,ftrace.yaml   |  306 ++++++++++++++++++++
 1 file changed, 306 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/chosen/linux,ftrace.yaml
Documentation/devicetree/bindings/chosen/linux,ftrace.yaml new file +306 -0
new file mode 100644
index 000000000000..a5c60ac2f66d
@@ -0,0 +1,306 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+# Copyright 2019 Linaro Ltd.
+%YAML 1.2
+---
+$id: "http://devicetree.org/schemas/tracing/ftrace.yaml#"
+$schema: "http://devicetree.org/meta-schemas/core.yaml#"
+
+title: Ftrace setting devicetree binding
+
+maintainers:
+  - Masami Hiramatsu <[email protected]>
+
+description: |
+  Boot-time ftrace tracing setting via devicetree. Users can use devicetree node
+  for programming ftrace boot-time tracing. This node must be placed at
+  /chosen/linux,ftrace.
+
+properties:
+  dump-on-oops:
+    allOf:
+      - $ref: /schemas/types.yaml#/definitions/uint32
+      - enum: [0, 1, 2]
+    description: |
+      A neumerical flag to enable ftrace dump on Kernel Oops. 0 means no dump,
+      1 means dump on the origin cpu of the oops, and means dump on all cpus.
+
+  traceoff-on-warning:
+    $ref: /schemas/types.yaml#/definitions/flag
+    description: A flag to stop tracing on warning.
+
+  tp-printk:
+    $ref: /schemas/types.yaml#/definitions/flag
+    description: A flag to send tracing output to printk buffer too.
+
+  alloc-snapshot:
+    $ref: /schemas/types.yaml#/definitions/flag
+    description: |
+      A flag to allocate snapshot buffer at boot. This will be required if you
+      use "snapshot" action on some events.
+
+  trace-clock:
+    allOf:
+      - $ref: /schemas/types.yaml#/definitions/string
+      - enum: [ global, local, counter, uptime, perf, mono, mono_raw, boot, ppc-tb, x86-tsc ]
+    description: Specify which clock method is used for trace-clock.
+
+  buffer-size-kb:
+    allOf:
+      - $ref: /schemas/types.yaml#/definitions/uint32
+      - minimum: 1
+    description: |
+      The size of per-cpu tracing buffer in KByte. Note that the size must be
+      larger than page size, and total size of buffers depends on the number
+      of CPUs.
+
+  events:
+    minItems: 1
+    $ref: /schemas/types.yaml#/definitions/string-array
+    description: |
+      A string array of enabling events (including wildcard patterns).
+      See Documentation/trace/events.rst for detail.
+
+  options:
+    minItems: 1
+    $ref: /schemas/types.yaml#/definitions/string-array
+    description: |
+      A string array of trace options for ftrace to control what gets printed
+      in the trace output or manipulate the tracers.
+      See Documentation/trace/ftrace.rst#trace_options for detail.
+
+  tracer:
+    default: nop
+    $ref: /schemas/types.yaml#/definitions/string
+    description: A string of the tracer to start up.
+
+  cpumask:
+    $ref: /schemas/types.yaml#/definitions/string
+    description: |
+      A hexadecimal number of the cpu-mask value which is given as a string.
+      This is because the number of cores can be bigger than 64.
+
+  ftrace-filters:
+    minItems: 1
+    $ref: /schemas/types.yaml#/definitions/string-array
+    description: |
+        A string array of the functions traced by the function tracer at boot
+        up. The function can be given with wildcards. This list can be
+        specified in each instance.
+
+  ftrace-notraces:
+    minItems: 1
+    $ref: /schemas/types.yaml#/definitions/string-array
+    description: |
+        A string array of the functions NOT traced by the function tracer at
+        boot up. The function can be given with wildcards. This list can be
+        specified in each instance.
+
+  fgraph-filters:
+    minItems: 1
+    $ref: /schemas/types.yaml#/definitions/string-array
+    description: |
+        A string array of the top level callers functions traced by the
+        function graph tracer at boot up. The function can be given with
+        wildcards. This list is not available in instance node.
+
+  fgraph-notraces:
+    minItems: 1
+    $ref: /schemas/types.yaml#/definitions/string-array
+    description: |
+        A string array of the top level callers functions NOT traced by the
+        function graph tracer at boot up. The function can be given with
+        wildcards. This list is not available in instance node.
+
+  fgraph-max-depth:
+    default: 0
+    $ref: /schemas/types.yaml#/definitions/uint32
+    description: |
+        This is the max depth, the function graph tracer will trace into
+        a function. 0 means no limit for the trace depth.
+
+patternProperties:
+   "event[0-9a-fA-F]+$":
+     type: object
+
+     description: |
+       event* properties are child nodes for per-event settings. It is also
+       able to define new kprobe events and synthetic events. Note that you
+       can not define both "probes" and "fields" properties on same event.
+
+     properties:
+       event:
+         $ref: /schemas/types.yaml#/definitions/string
+         description: |
+           Event name string formatted as "GROUP:EVENT". For synthetic event,
+           you must use "synthetic" for group name. For kprobe and synthetic
+           event, you can ommit the group name.
+
+       probes:
+         minItems: 1
+         $ref: /schemas/types.yaml#/definitions/string-array
+         description: |
+           A string array of kprobe event definitions, including location
+           (symbol+offset) and event arguments.
+           See Documentation/trace/kprobetrace.rst for detail.
+
+       fields:
+         minItems: 1
+         $ref: /schemas/types.yaml#/definitions/string-array
+         description: |
+           A string of synthetic event's fields definition. Note that you
+           don't need to repeat event name.
+
+       filter:
+         $ref: /schemas/types.yaml#/definitions/string
+         description: A string of event filter rule
+
+       actions:
+         minItems: 1
+         $ref: /schemas/types.yaml#/definitions/string-array
+         description: A string array of event trigger actions.
+
+       enable:
+         type: boolean
+         description: |
+           A flag to enable event. Note that the event is not enabled by
+           default. (But actions will set the event soft-disabled)
+
+     oneOf:
+       - required:
+         - event
+       - required:
+         - event
+         - probes
+       - required:
+         - event
+         - fields
+
+   "instance[0-9a-fA-F]+$":
+     type: object
+
+     description: |
+       instance* properties are child nodes for per-instance settings.
+       It is also able to have event nodes as linux,ftrace node does.
+
+     properties:
+       instance:
+         $ref: /schemas/types.yaml#/definitions/string
+         description: The name of new instance.
+
+       trace-clock:
+         allOf:
+           - $ref: /schemas/types.yaml#/definitions/string
+           - enum: [ global, local, counter, uptime, perf, mono, mono_raw, boot, ppc-tb, x86-tsc ]
+         description: Specify which clock method is used for trace-clock.
+
+       buffer-size-kb:
+         allOf:
+           - $ref: /schemas/types.yaml#/definitions/uint32
+           - minimum: 1
+         description: |
+           The size of per-cpu tracing buffer in KByte. Note that the size must be
+           larger than page size, and total size of buffers depends on the number
+           of CPUs.
+
+       events:
+         minItems: 1
+         $ref: /schemas/types.yaml#/definitions/string-array
+         description: |
+           A string array of enabling events (including wildcard patterns).
+           See Documentation/trace/events.rst for detail.
+
+       options:
+         minItems: 1
+         $ref: /schemas/types.yaml#/definitions/string-array
+         description: |
+           A string array of trace options for ftrace to control what gets printed
+           in the trace output or manipulate the tracers.
+           See Documentation/trace/ftrace.rst#trace_options for detail.
+
+       tracer:
+         default: nop
+         $ref: /schemas/types.yaml#/definitions/string
+         description: A string of the tracer to start up.
+
+       cpumask:
+         $ref: /schemas/types.yaml#/definitions/string
+         description: |
+           A hexadecimal number of the cpu-mask value which is given as a string.
+           This is because the number of cores can be bigger than 64.
+
+       ftrace-filters:
+         minItems: 1
+         $ref: /schemas/types.yaml#/definitions/string-array
+         description: |
+             A string array of the functions traced by the function tracer at boot
+             up. The function can be given with wildcards. This list can be
+             specified in each instance.
+
+       ftrace-notraces:
+         minItems: 1
+         $ref: /schemas/types.yaml#/definitions/string-array
+         description: |
+             A string array of the functions NOT traced by the function tracer at
+             boot up. The function can be given with wildcards. This list can be
+             specified in each instance.
+
+       fgraph-filters:
+         minItems: 1
+         $ref: /schemas/types.yaml#/definitions/string-array
+         description: |
+             A string array of the top level callers functions traced by the
+             function graph tracer at boot up. The function can be given with
+             wildcards. This list is not available in instance node.
+
+       fgraph-notraces:
+         minItems: 1
+         $ref: /schemas/types.yaml#/definitions/string-array
+         description: |
+             A string array of the top level callers functions NOT traced by the
+             function graph tracer at boot up. The function can be given with
+             wildcards. This list is not available in instance node.
+
+       fgraph-max-depth:
+         default: 0
+         $ref: /schemas/types.yaml#/definitions/uint32
+         description: |
+             This is the max depth, the function graph tracer will trace into
+             a function. 0 means no limit for the trace depth.
+
+     required:
+      - instance
+
+examples:
+  - |
+    ftrace {
+          events = "initcall:*";
+          tp-printk;
+          buffer-size-kb = <0x400>;
+          event1 {
+                event = "kprobes:vfs_read";
+                probes = "vfs_read $arg1 $arg2";
+                filter = "common_pid < 200";
+          };
+          event2 {
+                event = "initcall_latency";
+                fields = "unsigned long func", "u64 lat";
+                actions = "hist:keys=func.sym,lat:vals=lat:sort=lat";
+          };
+          event3 {
+                event = "initcall:initcall_start";
+                actions = "hist:keys=func:ts0=common_timestamp.usecs";
+          };
+          event4 {
+                event = "initcall:initcall_finish";
+                actions = "hist:keys=func:lat=common_timestamp.usecs-$ts0:onmatch(initcall.initcall_start).initcall_latency(func,$lat)";
+          };
+          instance0 {
+                buffer-size-kb = <0x100>;
+                event5 {
+                      event = "task:task_newtask";
+                      filter = "pid < 128";
+                      enable;
+                };
+          };
+
+    };
author maintainer
#8 lore raw

RFC PATCH v2 8/15 tracing: of: Add setup tracing by devicetree support

In reply to Masami Hiramatsu

Setup tracing options by devicetree instead of kernel parameters.

Since the kernel parameter is limited length, sometimes there is no space to setup the tracing options. This will read the tracing options from devicetree "ftrace" node and setup tracers at boot.

Note that this is not replacing the kernel parameters, because this devicetree based setting is later than that. If you want to trace earlier boot events, you still need kernel parameters.

---
Changes in v2:
- Move ftrace node under chosen node.
- Make events property available only when CONFIG_EVENT_TRACING=y

---
 kernel/trace/Kconfig    |   10 +++
 kernel/trace/Makefile   |    1 
 kernel/trace/trace.c    |   38 ++++++++----
 kernel/trace/trace_of.c |  150 +++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 186 insertions(+), 13 deletions(-)
 create mode 100644 kernel/trace/trace_of.c
kernel/trace/Kconfig +10 -0
index 07d22c61b634..025198bb2764 100644
@@ -796,6 +796,16 @@ config GCOV_PROFILE_FTRACE
 	  Note that on a kernel compiled with this config, ftrace will
 	  run significantly slower.
 
+config OF_TRACING
+	bool "Boot Trace Programming by Devicetree"
+	depends on OF && TRACING
+	select OF_EARLY_FLATTREE
+	default y
+	help
+	  Enable developer to setup ftrace subsystem via devicetree
+	  at boot time for debugging (tracing) driver initialization
+	  and boot process.
+
 endif # FTRACE
 
 endif # TRACING_SUPPORT
kernel/trace/Makefile +1 -0
index c2b2148bb1d2..6f68271f5c56 100644
@@ -82,6 +82,7 @@ endif
 obj-$(CONFIG_DYNAMIC_EVENTS) += trace_dynevent.o
 obj-$(CONFIG_PROBE_EVENTS) += trace_probe.o
 obj-$(CONFIG_UPROBE_EVENTS) += trace_uprobe.o
+obj-$(CONFIG_OF_TRACING) += trace_of.o
 
 obj-$(CONFIG_TRACEPOINT_BENCHMARK) += trace_benchmark.o
 
kernel/trace/trace.c +25 -13
index 30f09c058f79..6f50c58a3ad4 100644
@@ -158,7 +158,7 @@ union trace_eval_map_item {
 static union trace_eval_map_item *trace_eval_maps;
 #endif /* CONFIG_TRACE_EVAL_MAP_FILE */
 
-static int tracing_set_tracer(struct trace_array *tr, const char *buf);
+int tracing_set_tracer(struct trace_array *tr, const char *buf);
 static void ftrace_trace_userstack(struct ring_buffer *buffer,
 				   unsigned long flags, int pc);
 
@@ -178,6 +178,11 @@ static int __init set_cmdline_ftrace(char *str)
 }
 __setup("ftrace=", set_cmdline_ftrace);
 
+void __init trace_init_dump_on_oops(int mode)
+{
+	ftrace_dump_on_oops = mode;
+}
+
 static int __init set_ftrace_dump_on_oops(char *str)
 {
 	if (*str++ != '=' || !*str) {
@@ -4614,7 +4619,7 @@ int set_tracer_flag(struct trace_array *tr, unsigned int mask, int enabled)
 	return 0;
 }
 
-static int trace_set_options(struct trace_array *tr, char *option)
+int trace_set_options(struct trace_array *tr, char *option)
 {
 	char *cmp;
 	int neg = 0;
@@ -5505,8 +5510,8 @@ static int __tracing_resize_ring_buffer(struct trace_array *tr,
 	return ret;
 }
 
-static ssize_t tracing_resize_ring_buffer(struct trace_array *tr,
-					  unsigned long size, int cpu_id)
+ssize_t tracing_resize_ring_buffer(struct trace_array *tr,
+				  unsigned long size, int cpu_id)
 {
 	int ret = size;
 
@@ -5585,7 +5590,7 @@ static void add_tracer_options(struct trace_array *tr, struct tracer *t)
 	create_trace_option_files(tr, t);
 }
 
-static int tracing_set_tracer(struct trace_array *tr, const char *buf)
+int tracing_set_tracer(struct trace_array *tr, const char *buf)
 {
 	struct tracer *t;
 #ifdef CONFIG_TRACER_MAX_TRACE
@@ -9160,16 +9165,23 @@ __init static int tracer_alloc_buffers(void)
 	return ret;
 }
 
+void __init trace_init_tracepoint_printk(void)
+{
+	tracepoint_printk = 1;
+
+	tracepoint_print_iter =
+		kmalloc(sizeof(*tracepoint_print_iter), GFP_KERNEL);
+	if (WARN_ON(!tracepoint_print_iter))
+		tracepoint_printk = 0;
+	else
+		static_key_enable(&tracepoint_printk_key.key);
+}
+
 void __init early_trace_init(void)
 {
-	if (tracepoint_printk) {
-		tracepoint_print_iter =
-			kmalloc(sizeof(*tracepoint_print_iter), GFP_KERNEL);
-		if (WARN_ON(!tracepoint_print_iter))
-			tracepoint_printk = 0;
-		else
-			static_key_enable(&tracepoint_printk_key.key);
-	}
+	if (tracepoint_printk)
+		trace_init_tracepoint_printk();
+
 	tracer_alloc_buffers();
 }
 
kernel/trace/trace_of.c new file +150 -0

150 lines not on this page · raw message

author maintainer
#9 lore raw

RFC PATCH v2 9/15 tracing: of: Add trace event settings

In reply to Masami Hiramatsu

Add per-event settings, which includes filter and actions.

---
Changes in v2:
- Make event node available only if CONFIG_EVENT_TRACING=y

---
 kernel/trace/trace_events_trigger.c |    2 -
 kernel/trace/trace_of.c             |   82 +++++++++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+), 1 deletion(-)
kernel/trace/trace_events_trigger.c +1 -1
index 2a2912cb4533..74a19c18219f 100644
@@ -208,7 +208,7 @@ static int event_trigger_regex_open(struct inode *inode, struct file *file)
 	return ret;
 }
 
-static int trigger_process_regex(struct trace_event_file *file, char *buff)
+int trigger_process_regex(struct trace_event_file *file, char *buff)
 {
 	char *command, *next = buff;
 	struct event_command *p;
kernel/trace/trace_of.c +82 -0

102 lines not on this page · raw message

author maintainer
#10 lore raw

RFC PATCH v2 10/15 tracing: of: Add kprobe event support

In reply to Masami Hiramatsu

Add kprobe event support in event node. User can add probe definitions by "probes" property as a string array.

---
Changes in v2:
- Make probes property avaiable if CONFIG_KPROBE_EVENTS=y.

---
 kernel/trace/trace_kprobe.c |    5 +++
 kernel/trace/trace_of.c     |   76 +++++++++++++++++++++++++++++++++++++------
 2 files changed, 71 insertions(+), 10 deletions(-)
kernel/trace/trace_kprobe.c +5 -0
index 5135c07b6557..03ce60928c18 100644
@@ -728,6 +728,11 @@ static int create_or_delete_trace_kprobe(int argc, char **argv)
 	return ret == -ECANCELED ? -EINVAL : ret;
 }
 
+int trace_kprobe_run_command(const char *command)
+{
+	return trace_run_command(command, create_or_delete_trace_kprobe);
+}
+
 static int trace_kprobe_release(struct dyn_event *ev)
 {
 	struct trace_kprobe *tk = to_trace_kprobe(ev);
kernel/trace/trace_of.c +66 -10

98 lines not on this page · raw message

author maintainer
#11 lore raw

RFC PATCH v2 11/15 tracing: of: Add synthetic event support

In reply to Masami Hiramatsu

Add synthetic event node support. The synthetic event node must be a child node of ftrace node, and the node must start with "synth@" prefix. The synth node requires fields string (not string array), which defines the fields as same as tracing/synth_events interface.

---
Changes in v2:
- Make synth property available only if CONFIG_HIST_TRIGGERS=y

---
 kernel/trace/trace_events_hist.c |    5 +++
 kernel/trace/trace_of.c          |   64 ++++++++++++++++++++++++++++++++++++--
 2 files changed, 66 insertions(+), 3 deletions(-)
kernel/trace/trace_events_hist.c +5 -0
index db973928e580..e7f5d0a353e2 100644
@@ -1343,6 +1343,11 @@ static int create_or_delete_synth_event(int argc, char **argv)
 	return ret == -ECANCELED ? -EINVAL : ret;
 }
 
+int synth_event_run_command(const char *command)
+{
+	return trace_run_command(command, create_or_delete_synth_event);
+}
+
 static int synth_event_create(int argc, const char **argv)
 {
 	const char *name = argv[0];
kernel/trace/trace_of.c +61 -3

82 lines not on this page · raw message

author maintainer
#12 lore raw

RFC PATCH v2 12/15 tracing: of: Add instance node support

In reply to Masami Hiramatsu

Add instance node support to devicetree ftrace binding. User can set some options and event nodes in instance node.

---
 kernel/trace/trace_of.c |   67 +++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 55 insertions(+), 12 deletions(-)
kernel/trace/trace_of.c +55 -12
index e9142c63ece1..7fe81c25dc59 100644
@@ -21,9 +21,10 @@ extern int tracing_set_tracer(struct trace_array *tr, const char *buf);
 extern void __init trace_init_tracepoint_printk(void);
 extern ssize_t tracing_resize_ring_buffer(struct trace_array *tr,
 					  unsigned long size, int cpu_id);
+extern struct trace_array *trace_array_create(const char *name);
 
 static void __init
-trace_of_set_ftrace_options(struct trace_array *tr, struct device_node *node)
+trace_of_set_instance_options(struct trace_array *tr, struct device_node *node)
 {
 	struct property *prop;
 	const char *p;
@@ -48,6 +49,24 @@ trace_of_set_ftrace_options(struct trace_array *tr, struct device_node *node)
 			pr_err("Failed to set trace clock: %s\n", p);
 	}
 
+	/* This accepts per-cpu buffer size in KB */
+	err = of_property_read_u32_index(node, "buffer-size-kb", 0, &v);
+	if (!err) {
+		v <<= 10;	/* KB to Byte */
+		if (v < PAGE_SIZE)
+			pr_err("Buffer size is too small: %d KB\n", v >> 10);
+		if (tracing_resize_ring_buffer(tr, v, RING_BUFFER_ALL_CPUS) < 0)
+			pr_err("Failed to resize trace buffer to %d KB\n",
+				v >> 10);
+	}
+}
+
+static void __init
+trace_of_set_ftrace_options(struct trace_array *tr, struct device_node *node)
+{
+	u32 v = 0;
+	int err;
+
 	/* Command line boot options */
 	if (of_find_property(node, "dump-on-oops", NULL)) {
 		err = of_property_read_u32_index(node, "dump-on-oops", 0, &v);
@@ -63,20 +82,11 @@ trace_of_set_ftrace_options(struct trace_array *tr, struct device_node *node)
 	if (of_find_property(node, "tp-printk", NULL))
 		trace_init_tracepoint_printk();
 
-	/* This accepts per-cpu buffer size in KB */
-	err = of_property_read_u32_index(node, "buffer-size-kb", 0, &v);
-	if (!err) {
-		v <<= 10;	/* KB to Byte */
-		if (v < PAGE_SIZE)
-			pr_err("Buffer size is too small: %d KB\n", v >> 10);
-		if (tracing_resize_ring_buffer(tr, v, RING_BUFFER_ALL_CPUS) < 0)
-			pr_err("Failed to resize trace buffer to %d KB\n",
-				v >> 10);
-	}
-
 	if (of_find_property(node, "alloc-snapshot", NULL))
 		if (tracing_alloc_snapshot() < 0)
 			pr_err("Failed to allocate snapshot buffer\n");
+
+	trace_of_set_instance_options(tr, node);
 }
 
 #ifdef CONFIG_EVENT_TRACING
@@ -310,6 +320,38 @@ trace_of_enable_tracer(struct trace_array *tr, struct device_node *node)
 	}
 }
 
+static void __init
+trace_of_init_instances(struct device_node *__node)
+{
+	struct device_node *node;
+	struct trace_array *tr;
+	const char *p;
+	int err;
+
+	for_each_child_of_node(__node, node) {
+		if (!of_node_name_prefix(node, "instance"))
+			continue;
+
+		err = of_property_read_string(node, "instance", &p);
+		if (err) {
+			pr_err("Failed to get instance name on %s\n",
+				of_node_full_name(node));
+			continue;
+		}
+
+		tr = trace_array_create(p);
+		if (IS_ERR(tr)) {
+			pr_err("Failed to create instance %s\n", p);
+			continue;
+		}
+
+		trace_of_set_instance_options(tr, node);
+		trace_of_init_events(tr, node);
+		trace_of_enable_events(tr, node);
+		trace_of_enable_tracer(tr, node);
+	}
+}
+
 static struct device_node * __init trace_of_find_ftrace_node(void)
 {
 	if (!of_chosen)
@@ -337,6 +379,7 @@ static int __init trace_of_init(void)
 	trace_of_init_events(tr, trace_node);
 	trace_of_enable_events(tr, trace_node);
 	trace_of_enable_tracer(tr, trace_node);
+	trace_of_init_instances(trace_node);
 
 end:
 	of_node_put(trace_node);
author maintainer
#13 lore raw

RFC PATCH v2 13/15 tracing: of: Add cpumask property support

In reply to Masami Hiramatsu

Add cpumask property for tracing. Note that this property accepts a string cpumask, not a digit number.

---
 kernel/trace/trace.c    |   41 ++++++++++++++++++++++++++++-------------
 kernel/trace/trace_of.c |   14 ++++++++++++++
 2 files changed, 42 insertions(+), 13 deletions(-)
kernel/trace/trace.c +28 -13
index 6f50c58a3ad4..251d9de14b12 100644
@@ -4437,20 +4437,12 @@ tracing_cpumask_read(struct file *filp, char __user *ubuf,
 	return count;
 }
 
-static ssize_t
-tracing_cpumask_write(struct file *filp, const char __user *ubuf,
-		      size_t count, loff_t *ppos)
+int tracing_set_cpumask(struct trace_array *tr, cpumask_var_t tracing_cpumask_new)
 {
-	struct trace_array *tr = file_inode(filp)->i_private;
-	cpumask_var_t tracing_cpumask_new;
-	int err, cpu;
-
-	if (!alloc_cpumask_var(&tracing_cpumask_new, GFP_KERNEL))
-		return -ENOMEM;
+	int cpu;
 
-	err = cpumask_parse_user(ubuf, count, tracing_cpumask_new);
-	if (err)
-		goto err_unlock;
+	if (!tr)
+		return -EINVAL;
 
 	local_irq_disable();
 	arch_spin_lock(&tr->max_lock);
@@ -4474,11 +4466,34 @@ tracing_cpumask_write(struct file *filp, const char __user *ubuf,
 	local_irq_enable();
 
 	cpumask_copy(tr->tracing_cpumask, tracing_cpumask_new);
+
+	return 0;
+}
+
+static ssize_t
+tracing_cpumask_write(struct file *filp, const char __user *ubuf,
+		      size_t count, loff_t *ppos)
+{
+	struct trace_array *tr = file_inode(filp)->i_private;
+	cpumask_var_t tracing_cpumask_new;
+	int err;
+
+	if (!alloc_cpumask_var(&tracing_cpumask_new, GFP_KERNEL))
+		return -ENOMEM;
+
+	err = cpumask_parse_user(ubuf, count, tracing_cpumask_new);
+	if (err)
+		goto err_free;
+
+	err = tracing_set_cpumask(tr, tracing_cpumask_new);
+	if (err)
+		goto err_free;
+
 	free_cpumask_var(tracing_cpumask_new);
 
 	return count;
 
-err_unlock:
+err_free:
 	free_cpumask_var(tracing_cpumask_new);
 
 	return err;
kernel/trace/trace_of.c +14 -0
index 7fe81c25dc59..14d6c9a8fd81 100644
@@ -22,6 +22,8 @@ extern void __init trace_init_tracepoint_printk(void);
 extern ssize_t tracing_resize_ring_buffer(struct trace_array *tr,
 					  unsigned long size, int cpu_id);
 extern struct trace_array *trace_array_create(const char *name);
+extern int tracing_set_cpumask(struct trace_array *tr,
+			       cpumask_var_t tracing_cpumask_new);
 
 static void __init
 trace_of_set_instance_options(struct trace_array *tr, struct device_node *node)
@@ -59,6 +61,18 @@ trace_of_set_instance_options(struct trace_array *tr, struct device_node *node)
 			pr_err("Failed to resize trace buffer to %d KB\n",
 				v >> 10);
 	}
+
+	err = of_property_read_string(node, "cpumask", &p);
+	if (!err) {
+		cpumask_var_t new_mask;
+
+		if (alloc_cpumask_var(&new_mask, GFP_KERNEL)) {
+			if (cpumask_parse(p, new_mask) < 0 ||
+			    tracing_set_cpumask(tr, new_mask) < 0)
+				pr_err("Failed to set new CPU mask %s\n", p);
+			free_cpumask_var(new_mask);
+		}
+	}
 }
 
 static void __init
author maintainer
#14 lore raw

RFC PATCH v2 14/15 tracing: of: Add function tracer filter properties

In reply to Masami Hiramatsu

Add function tracer filter properties which includes
- ftrace-filters : string array of filter rules
- ftrace-notraces : string array of notrace rules
These properties are available on ftrace root node and instance node.

---
 kernel/trace/trace_of.c |   36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
kernel/trace/trace_of.c +36 -0
index 14d6c9a8fd81..1ee6fab918f5 100644
@@ -322,11 +322,47 @@ trace_of_init_events(struct trace_array *tr, struct device_node *node)
 #define trace_of_init_events(tr, node) do {} while (0)
 #endif
 
+#ifdef CONFIG_FUNCTION_TRACER
+extern bool ftrace_filter_param __initdata;
+extern int ftrace_set_filter(struct ftrace_ops *ops, unsigned char *buf,
+			     int len, int reset);
+extern int ftrace_set_notrace(struct ftrace_ops *ops, unsigned char *buf,
+			     int len, int reset);
+
+static void __init
+trace_of_set_ftrace_filter(struct ftrace_ops *ops, const char *property,
+			   struct device_node *node)
+{
+	struct property *prop;
+	const char *p;
+	char buf[MAX_BUF_LEN];
+	int err;
+
+	of_property_for_each_string(node, property, prop, p) {
+		if (strlcpy(buf, p, ARRAY_SIZE(buf)) >= ARRAY_SIZE(buf)) {
+			pr_err("filter string is too long: %s\n", p);
+			return;
+		}
+		err = ftrace_set_filter(ops, buf, strlen(buf), 0);
+		if (err) {
+			pr_err("Failed to add %s: %s\n", property, buf);
+			return;
+		}
+		ftrace_filter_param = true;
+	}
+}
+#else
+#define trace_of_set_ftrace_filter(ops, prop, node) do {} while (0)
+#endif
+
 static void __init
 trace_of_enable_tracer(struct trace_array *tr, struct device_node *node)
 {
 	const char *p;
 
+	trace_of_set_ftrace_filter(tr->ops, "ftrace-filters", node);
+	trace_of_set_ftrace_filter(tr->ops, "ftrace-notraces", node);
+
 	if (!of_property_read_string(node, "tracer", &p)) {
 		if (tracing_set_tracer(tr, p) < 0)
 			pr_err("Failed to set given tracer: %s\n", p);
author maintainer
#15 lore raw

RFC PATCH v2 15/15 tracing: of: Add function-graph tracer option properties

In reply to Masami Hiramatsu

Add following function-graph tracer related options
- fgraph-filters : string array of filter
- fgraph-notraces : string array of notrace-filter
- fgraph-max-depth : u32 value of max depth

Note that these properties are available on ftrace root node only, because these filters are globally applied.

---
 kernel/trace/ftrace.c   |   85 ++++++++++++++++++++++++++++++-----------------
 kernel/trace/trace_of.c |   68 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 123 insertions(+), 30 deletions(-)
kernel/trace/ftrace.c +55 -30
index 5c3eadb143ed..7bc60ac4583e 100644
@@ -1241,7 +1241,7 @@ static void clear_ftrace_mod_list(struct list_head *head)
 	mutex_unlock(&ftrace_lock);
 }
 
-static void free_ftrace_hash(struct ftrace_hash *hash)
+void free_ftrace_hash(struct ftrace_hash *hash)
 {
 	if (!hash || hash == EMPTY_HASH)
 		return;
@@ -4895,7 +4895,7 @@ __setup("ftrace_filter=", set_ftrace_filter);
 #ifdef CONFIG_FUNCTION_GRAPH_TRACER
 static char ftrace_graph_buf[FTRACE_FILTER_SIZE] __initdata;
 static char ftrace_graph_notrace_buf[FTRACE_FILTER_SIZE] __initdata;
-static int ftrace_graph_set_hash(struct ftrace_hash *hash, char *buffer);
+int ftrace_graph_set_hash(struct ftrace_hash *hash, char *buffer);
 
 static int __init set_graph_function(char *str)
 {
@@ -5224,6 +5224,26 @@ __ftrace_graph_open(struct inode *inode, struct file *file,
 	return ret;
 }
 
+struct ftrace_hash *ftrace_graph_copy_hash(bool enable)
+{
+	struct ftrace_hash *hash;
+
+	mutex_lock(&graph_lock);
+
+	if (enable)
+		hash = rcu_dereference_protected(ftrace_graph_hash,
+					lockdep_is_held(&graph_lock));
+	else
+		hash = rcu_dereference_protected(ftrace_graph_notrace_hash,
+					lockdep_is_held(&graph_lock));
+
+	hash = alloc_and_copy_ftrace_hash(FTRACE_HASH_DEFAULT_BITS, hash);
+
+	mutex_unlock(&graph_lock);
+
+	return hash;
+}
+
 static int
 ftrace_graph_open(struct inode *inode, struct file *file)
 {
@@ -5280,11 +5300,40 @@ ftrace_graph_notrace_open(struct inode *inode, struct file *file)
 	return ret;
 }
 
+int ftrace_graph_apply_hash(struct ftrace_hash *hash, bool enable)
+{
+	struct ftrace_hash *old_hash, *new_hash;
+
+	new_hash = __ftrace_hash_move(hash);
+	if (!new_hash)
+		return -ENOMEM;
+
+	mutex_lock(&graph_lock);
+
+	if (enable) {
+		old_hash = rcu_dereference_protected(ftrace_graph_hash,
+				lockdep_is_held(&graph_lock));
+		rcu_assign_pointer(ftrace_graph_hash, new_hash);
+	} else {
+		old_hash = rcu_dereference_protected(ftrace_graph_notrace_hash,
+				lockdep_is_held(&graph_lock));
+		rcu_assign_pointer(ftrace_graph_notrace_hash, new_hash);
+	}
+
+	mutex_unlock(&graph_lock);
+
+	/* Wait till all users are no longer using the old hash */
+	synchronize_rcu();
+
+	free_ftrace_hash(old_hash);
+
+	return 0;
+}
+
 static int
 ftrace_graph_release(struct inode *inode, struct file *file)
 {
 	struct ftrace_graph_data *fgd;
-	struct ftrace_hash *old_hash, *new_hash;
 	struct trace_parser *parser;
 	int ret = 0;
 
@@ -5309,41 +5358,17 @@ ftrace_graph_release(struct inode *inode, struct file *file)
 
 		trace_parser_put(parser);
 
-		new_hash = __ftrace_hash_move(fgd->new_hash);
-		if (!new_hash) {
-			ret = -ENOMEM;
-			goto out;
-		}
-
-		mutex_lock(&graph_lock);
-
-		if (fgd->type == GRAPH_FILTER_FUNCTION) {
-			old_hash = rcu_dereference_protected(ftrace_graph_hash,
-					lockdep_is_held(&graph_lock));
-			rcu_assign_pointer(ftrace_graph_hash, new_hash);
-		} else {
-			old_hash = rcu_dereference_protected(ftrace_graph_notrace_hash,
-					lockdep_is_held(&graph_lock));
-			rcu_assign_pointer(ftrace_graph_notrace_hash, new_hash);
-		}
-
-		mutex_unlock(&graph_lock);
-
-		/* Wait till all users are no longer using the old hash */
-		synchronize_rcu();
-
-		free_ftrace_hash(old_hash);
+		ret = ftrace_graph_apply_hash(fgd->new_hash,
+					fgd->type == GRAPH_FILTER_FUNCTION);
 	}
 
- out:
 	free_ftrace_hash(fgd->new_hash);
 	kfree(fgd);
 
 	return ret;
 }
 
-static int
-ftrace_graph_set_hash(struct ftrace_hash *hash, char *buffer)
+int ftrace_graph_set_hash(struct ftrace_hash *hash, char *buffer)
 {
 	struct ftrace_glob func_g;
 	struct dyn_ftrace *rec;
kernel/trace/trace_of.c +68 -0

80 lines not on this page · raw message

#16 lore raw
In reply to Masami Hiramatsu

Hi Masami,

After receiving this email, I replied to one email on the v1 thread, so there will be a little bit of overlap in the ordering of the two threads. Feel free to reply to my comments in the v1 thread in this thread instead.

More comments below.

On 7/14/19 10:11 PM, Masami Hiramatsu wrote:> Hello,

Quoted 107 lines

Here is the 2nd version of RFC series to add boot-time tracing using devicetree. Previous thread is here.

https://lkml.kernel.org/r/156113387975.28344.16009584175308192243.stgit@devnote2

In this version, I moved the ftrace node under /chosen/linux,ftrace and remove compatible property, because it must be in fixed place. Also this version has following features;

- Introduce "instance" node, which can have events nodes for setting events filters and actions for the trace instance.
- Introduce "cpumask" property
- Introduce "ftrace-filters" and "ftrace-notraces"
- Introduce "fgraph-filters", "fgraph-notraces" and "fgraph-max-depth"

At this moment, this feature is only available on the architecutre which supports devicetree. For x86, we can use it on qemu with --dtb option, or apply below patch on grub to add devicetree support on grub-x86.

https://github.com/mhiramat/grub/commit/644c35bfd2d18c772cc353b74215344f8264923a

Note that the devicetree for x86 must contain the nodes only under /chosen/, or it may cause a problem if it conflicts with ACPI. (Maybe we need to disable the FDT nodes except for nodes under /chosen
on boot if ACPI exists.)

This series can be applied on Steve's tracing tree (ftrace/core) or available on below

https://github.com/mhiramat/linux.git ftrace-devicetree-v2

Usage
======
With this series, we can setup new kprobe and synthetic events, more complicated event filters and trigger actions including histogram via devicetree.

For example, following kernel parameters

trace_options=sym-addr trace_event=initcall:* tp_printk trace_buf_size=1M ftrace=function ftrace_filter="vfs*"

it can be written in devicetree like below.

/{

chosen {
	...
	ftrace {
		options = "sym-addr";
		events = "initcall:*";
		tp-printk;
		buffer-size-kb = <0x400>;	// 1024KB == 1MB
		ftrace-filters = "vfs*";
	};

Moreover, now we can expand it to add filters for events, kprobe events, and synthetic events with histogram like below.

	ftrace {
		...
		event0 {
			event = "task:task_newtask";
			filter = "pid < 128";	// adding filters
			enable;
		};
		event1 {
			event = "kprobes:vfs_read";
			probes = "vfs_read $arg1 $arg2"; // add kprobes
			filter = "common_pid < 200";
			enable;
		};
		event2 {
			event = "initcall_latency";	// add synth event
			fields = "unsigned long func", "u64 lat";
			// with histogram
			actions = "hist:keys=func.sym,lat:vals=lat:sort=lat";
		};
		// and synthetic event callers
		event3 {
			event = "initcall:initcall_start";
			actions = "hist:keys=func:ts0=common_timestamp.usecs";
		};
		event4 {
			event = "initcall:initcall_finish";
			actions = "hist:keys=func:lat=common_timestamp.usecs-$ts0:onmatch(initcall.initcall_start).initcall_latency(func,$lat)";
		};
	};

Also, this version supports "instance" node, which allows us to run several tracers for different purpose at once. For example, one tracer is for tracing functions in module alpha, and others tracing module beta, you can write followings.

	ftrace {
		instance0 {
			instance = "foo";
			tracer = "function";
			ftrace-filters = "*:mod:alpha";
		};
		instance1 {
			instance = "bar";
			tracer = "function";
			ftrace-filters = "*:mod:beta";
			
		};
	};
Quoted 8 lines

The instance node also accepts event nodes so that each instance can customize its event tracing.

Discussion
=====
On the previous thread, we discussed that the this devicetree usage itself was acceptable or not. Fortunately, I had a chance to discuss it in a F2F meeting with Frank and Tim last week.

Thanks for writing up some of what we discussed.

Let me add a problem statement and use case. I'll probably get it at least a little bit wrong, so please update as needed.

(1) You feel the ftrace kernel command line syntax is not sufficiently user friendly.

(2) The kernel command line is too small to contain the full set of desired ftrace commands and options.

(3) There is a desire to change the boot time ftrace commands and options without re-compiling or re-linking the Linux kernel.

Quoted 8 lines

I think the advantages of using devicetree are,

- reuse devicetree's structured syntax for complicated tracefs settings
- reuse OF-APIs in linux kernel to accept and parse it
- reuse dtc complier to compile it and validate syntax. (with yaml schema, we can enhance it)
- reuse current bootloader (and qemu) to load it

Devicetree is not a universal data structure and communication channel.

Devicetree is a description of the hardware and also conveys bootloader specific information that is need by the kernel to boot.

Quoted 5 lines

And we talked about some other ideas to avoid using devicetree.

- expand kernel command line (ascii command strings)
- expand kernel command line with base64 encoded comressed ascii command strings

Base64 being one of possibly many ways to convert arbitrary binary data to ascii safe data _if_ you want to transfer the ftrace options and commands in a binary format.

Quoted 2 lines

- load (compressed) ascii command strings to somewhere on memory and pass the address via kernel cmdline

Similar to the way initrd is handled, if I understand correctly. (I am not up to date on how initrd location is passed to the kernel for a non-devicetree kernel.)

Compressed or not compressed would be an ftrace design choice.

Quoted 2 lines

- load (compressed) ascii command strings to somewhere on memory and pass the address via /chosen node (as same as initrd)

Compressed or not compressed would be an ftrace design choice.

Quoted 3 lines

- load binary C data and point it from kernel cmdline
- load binary C data and point it from /chosen node (as same as initrd)
- load binary C data as a section of kernel image

For the three options above:

Binary data if ftrace prefers structured data.
A list of strings if ftrace wants to use the existing kernel command line syntax.

For the third of the above three options, the linker would provide the start and end address of the ftrace options and commands section.

Quoted 3 lines

The first 2 ideas expand the kernel's cmdline to pass some "magic" command to setup ftrace. In both case, the problems are the maximal size of cmdline and the issues related to the complexity of commands.

Not a "magic" command. Either continue using the existing ftrace syntax or add something like: ftrace_cmd="whatever format ftrace desires".

Why can the maximum size of the cmdline not be increased?

Quoted 4 lines

My example showed that the ftrace settings becomes long even if making one histogram, which can be longer than 256 bytes. The long and complex data can easily lead mis-typing, but cmdline has no syntax validator, it just ignores the mis-typed commands.

Hand typing a kernel command line is already not a fun exercise, even before adding ftrace commands. If you are hand typing kernel command lines then I suggest you improve your tools (eg bootloader or whatever is not allowing you to edit and store command lines).

Quoted 9 lines

(Of course even with the devicetree, it must be smaller than 2 pages)

Next 2 ideas are similar, but load the commands on some other memory area and pass only address via cmdline. This solves the size limitation issue, but still no syntax validation. Of course we can make a new structured syntax validator similar to (or just forked from) dt-validate. The problem (or disadvantage) of these (and following) ideas, is to change the kernel and boot loaders to load another binary blobs on memory.

Quoted 3 lines

Maybe if we introduce a generic structured kernel boot arguments, which is a kind of /chosen node of devicetree. (But if there is already such hook, why we make another one...?)

I got lost in the next sentence, so for my benefit:
GSKBA == generic structured kernel boot arguments

Quoted 6 lines

Also, this "GSKBA" may introduce a parser and access APIs which will be very similar to OF-APIs. This also seems redundant to me.

So the last 3 ideas will avoid introducing new parser and APIs, we just compile the data as C data and point it from cmdline or somewhere else.

Or if in a kernel data section then the linker can provide the begin and end address of the blob. This is already implemented for some other data structures.

Quoted 3 lines

With these ideas, we still need to expand boot loaders to support loading new binary blobs. (And the last one requires to add elf header parser/modifier to boot loader too)

Why would the boot loader need to access the elf header? The linker can provide the location of the new kernel data section via kernel variables.

Quoted 3 lines
Quoted 2 lines

From the above reasons, I think using devicetree's /chosen node is

the least intrusive way to introduce this boot-time tracing feature.

This is still mis-use of the devicetree data structure. This data does not belong in the devicetree.

Quoted 43 lines

Any suggestions, thoughts?

Thank you,

---

Masami Hiramatsu (15):
tracing: Apply soft-disabled and filter to tracepoints printk
tracing: kprobes: Output kprobe event to printk buffer
tracing: Expose EXPORT_SYMBOL_GPL symbol
tracing: kprobes: Register to dynevent earlier stage
tracing: Accept different type for synthetic event fields
tracing: Add NULL trace-array check in print_synth_event()
dt-bindings: tracing: Add ftrace binding document
tracing: of: Add setup tracing by devicetree support
tracing: of: Add trace event settings
tracing: of: Add kprobe event support
tracing: of: Add synthetic event support
tracing: of: Add instance node support
tracing: of: Add cpumask property support
tracing: of: Add function tracer filter properties
tracing: of: Add function-graph tracer option properties

 .../devicetree/bindings/chosen/linux,ftrace.yaml   |  306 ++++++++++++
 include/linux/trace_events.h                       |    1 
 kernel/trace/Kconfig                               |   10 
 kernel/trace/Makefile                              |    1 
 kernel/trace/ftrace.c                              |   85 ++-
 kernel/trace/trace.c                               |   90 ++--
 kernel/trace/trace_events.c                        |    3 
 kernel/trace/trace_events_hist.c                   |   14 -
 kernel/trace/trace_events_trigger.c                |    2 
 kernel/trace/trace_kprobe.c                        |   81 ++-
 kernel/trace/trace_of.c                            |  507 ++++++++++++++++++++
 11 files changed, 1004 insertions(+), 96 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/chosen/linux,ftrace.yaml
 create mode 100644 kernel/trace/trace_of.c

--
Masami Hiramatsu (Linaro) <[email protected]>

author maintainer
#17 lore raw

Hi Frank,

Quoted 6 lines · On Mon, 15 Jul 2019 07:21:27 -0700 Frank Rowand <[email protected]> wrote:

Hi Masami,

After receiving this email, I replied to one email on the v1 thread, so there will be a little bit of overlap in the ordering of the two threads. Feel free to reply to my comments in the v1 thread in this thread instead.

OK, thanks for the notice :)

Quoted 4 lines

More comments below.

On 7/14/19 10:11 PM, Masami Hiramatsu wrote:> Hello,

[...]

Quoted 12 lines
Quoted 6 lines

Discussion
=====
On the previous thread, we discussed that the this devicetree usage itself was acceptable or not. Fortunately, I had a chance to discuss it in a F2F meeting with Frank and Tim last week.

Thanks for writing up some of what we discussed.

Let me add a problem statement and use case. I'll probably get it at least a little bit wrong, so please update as needed.

Quoted 8 lines

(1) You feel the ftrace kernel command line syntax is not sufficiently user friendly.

(2) The kernel command line is too small to contain the full set of desired ftrace commands and options.

(3) There is a desire to change the boot time ftrace commands and options without re-compiling or re-linking the Linux kernel.

Thank you for covering these items :) Yes, these are what I'm thinking.

Quoted 9 lines
Quoted 8 lines

I think the advantages of using devicetree are,

- reuse devicetree's structured syntax for complicated tracefs settings
- reuse OF-APIs in linux kernel to accept and parse it
- reuse dtc complier to compile it and validate syntax. (with yaml schema, we can enhance it)
- reuse current bootloader (and qemu) to load it

Quoted 4 lines

Devicetree is not a universal data structure and communication channel.

Devicetree is a description of the hardware and also conveys bootloader specific information that is need by the kernel to boot.

Yes, I see. But I think there is a room to contain a small communication channel under /chosen, from bootloader.

Quoted 10 lines
Quoted 6 lines

And we talked about some other ideas to avoid using devicetree.

- expand kernel command line (ascii command strings)
- expand kernel command line with base64 encoded comressed ascii command strings

Base64 being one of possibly many ways to convert arbitrary binary data to ascii safe data _if_ you want to transfer the ftrace options and commands in a binary format.

I actually don't want it :( but if the ascii commands can be compressed (maybe not so efficient), it is a possible way.

Quoted 6 lines
Quoted 2 lines

- load (compressed) ascii command strings to somewhere on memory and pass the address via kernel cmdline

Similar to the way initrd is handled, if I understand correctly. (I am not up to date on how initrd location is passed to the kernel for a non-devicetree kernel.)

Initrd is not passed via kernel cmdline, it is loaded and passed via architecture dependent way. As far as I know, x86 and arm (without DT) uses own data structure, arm64 (and arm with DT) uses devicetree /chosen node.

Quoted 8 lines

Compressed or not compressed would be an ftrace design choice.

Quoted 2 lines

- load (compressed) ascii command strings to somewhere on memory and pass the address via /chosen node (as same as initrd)

Compressed or not compressed would be an ftrace design choice.

Yes, it is optional.

Quoted 10 lines
Quoted 4 lines

- load binary C data and point it from kernel cmdline
- load binary C data and point it from /chosen node (as same as initrd)
- load binary C data as a section of kernel image

For the three options above:

Binary data if ftrace prefers structured data.
A list of strings if ftrace wants to use the existing kernel command line syntax.

Yes, any data which doesn't need complex parser is OK.

Quoted 3 lines

For the third of the above three options, the linker would provide the start and end address of the ftrace options and commands section.

But that means we need to fill the data structure when we build the kernel, isn't it?

Quoted 8 lines
Quoted 3 lines

The first 2 ideas expand the kernel's cmdline to pass some "magic" command to setup ftrace. In both case, the problems are the maximal size of cmdline and the issues related to the complexity of commands.

Not a "magic" command. Either continue using the existing ftrace syntax or add something like: ftrace_cmd="whatever format ftrace desires".

Why can the maximum size of the cmdline not be increased?

We can, but we also has to change bootloaders.

Quoted 9 lines
Quoted 4 lines

My example showed that the ftrace settings becomes long even if making one histogram, which can be longer than 256 bytes. The long and complex data can easily lead mis-typing, but cmdline has no syntax validator, it just ignores the mis-typed commands.

Hand typing a kernel command line is already not a fun exercise, even before adding ftrace commands. If you are hand typing kernel command lines then I suggest you improve your tools (eg bootloader or whatever is not allowing you to edit and store command lines).

Indeed, if we extend kernel cmdline to support it, such tool we have to introduce (like dtc)

Quoted 9 lines
Quoted 9 lines

(Of course even with the devicetree, it must be smaller than 2 pages)

Next 2 ideas are similar, but load the commands on some other memory area and pass only address via cmdline. This solves the size limitation issue, but still no syntax validation. Of course we can make a new structured syntax validator similar to (or just forked from) dt-validate. The problem (or disadvantage) of these (and following) ideas, is to change the kernel and boot loaders to load another binary blobs on memory.

Quoted 6 lines
Quoted 3 lines

Maybe if we introduce a generic structured kernel boot arguments, which is a kind of /chosen node of devicetree. (But if there is already such hook, why we make another one...?)

I got lost in the next sentence, so for my benefit:
GSKBA == generic structured kernel boot arguments

Oh, sorry, that's my bad.

Quoted 10 lines
Quoted 6 lines

Also, this "GSKBA" may introduce a parser and access APIs which will be very similar to OF-APIs. This also seems redundant to me.

So the last 3 ideas will avoid introducing new parser and APIs, we just compile the data as C data and point it from cmdline or somewhere else.

Or if in a kernel data section then the linker can provide the begin and end address of the blob. This is already implemented for some other data structures.

Yeah, but does that mean we have to rebuild kernel image?
In some cases, (e.g. debugging distro kernel) we can not modify kernel image also, I don't like to replace entire image. I would like to choose a tracing command file from boot loader.

Quoted 7 lines
Quoted 3 lines

With these ideas, we still need to expand boot loaders to support loading new binary blobs. (And the last one requires to add elf header parser/modifier to boot loader too)

Why would the boot loader need to access the elf header? The linker can provide the location of the new kernel data section via kernel variables.

Oh, I thought you meant that the new data was added boot time by boot loader.

Quoted 5 lines
Quoted 2 lines
Quoted 1 line

From the above reasons, I think using devicetree's /chosen node is

the least intrusive way to introduce this boot-time tracing feature.

This is still mis-use of the devicetree data structure. This data does not belong in the devicetree.

I think if the boot loader supports overlay file, we can choose the overlay file when booting for /chosen node. That can be a part of boot loader choice, isn't it? :)

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
author maintainer
#18 lore raw

Hello,

I discussed with Frank and other kernel developers last week at OSSJ 2019. Eventually, I decided to leave from devicetree, because it can unstabilize current devicetree desgin and policy. Instead, aim to introduce a new generic structured kernel cmdline, something like "configtree".

I thought JSON or other generic data format, but they look a bit bloated for my purpose. I just need something like "extended hierarchical kernel cmdline". For example,

ftrace {
options = "sym-addr"
events = "initcall:*"
tp-printk
event.0 {
name = "tasl:task_newtask"
filter = "pid < 128"
}
}

Which can be written as

ftrace.options="sym-addr" ftrace.events="initcall:*" ftrace.tp-printk ftrace.event.0.name="tasl:task_newtask" ftrace.event.0.filter="pid < 128"

on current kernel cmdline.

So, the parameters are linearly extended from current kernel cmdline. Kernel internal APIs must be able to handle both of current cmdline key-values and configtree key-values.

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>