Skip to contentSkip to conversation

tracing: skc: Boot-time tracing and Supplemental Kernel Cmdline

RFC PATCH In review Masami Hiramatsu 2 participants 22 messages Aug 26, 2019 – Aug 27, 2019

Subsystems (6)
lore
Share & tools
lore

Masami Hiramatsu

author maintainer
#0 lore raw

v3 cover tracing: skc: Boot-time tracing and Supplemental Kernel Cmdline

Hello,

This is the 3rd version of RFC series for the boot-time tracing.

Previous thread is here.

https://lkml.kernel.org/r/156316746861.23477.5815110570539190650.stgit@devnote2

On that thread and offline talk with Frank Rowand at OSS Japan2019 last month, we agreed that the devicetree is only for hardware description and should not expand it for software settings.

Thus, in this version, I introduced a new extended command line feature called Supplemental Kernel Cmdline (SKC) instead of devicetree, and rewrote the boot-time tracing part based on that.

Supplemental Kernel Cmdline
===========================

Supplemental kernel command line (SKC) allows admin to pass a tree-structured supplemental kernel commandline file (SKC file) when boot up kernel. This expands the kernel command line in efficient way.

Each key is described as a dot-jointed-words. And user can write the key-words in tree stlye.
For example,

feature.option.foo = 1;
feature.option.bar = 2;

can be also written in

 feature.option {
    foo = 1;
    bar = 2;

}

(Note that in both style, the same words are merged automatically
and make a single tree)
All values are treated as a string, or array of strings, e.g.

feature.options = "foo", "bar";

User can see the loaded SKC key-value list via /proc/skc. The size of SKC is limited upto 32KB and 512 key-words and values in total.

SKC and Bootloader
==================

To play with SKC, you need to use patched qemu or patched grub. Please checkout below for x86 support. (for PoC, I implemented it only for x86, but it is not so hard to do same on other archs)

https://github.com/mhiramat/qemu.git skc

https://github.com/mhiramat/grub.git grub-x86-skc

User can pass an skc file when boot the qemu machine with "-skc PATH" option. Or specify the skc file on grub console by "skc PATH".

Boot-time Tracing
=================

Boot-time tracing side has been updated for SKC, but the difference is small, because SKC has similar tree-based interfaces to OF APIs. Currently, it supports following SKC options. Please read Documentation/trace/boottime-trace.rst for details.

- ftrace.options = OPT1[,OPT2...];
- ftrace.trace_clock = CLOCK;
- ftrace.dump_on_oops [= MODE];
- ftrace.traceoff_on_warning;
- ftrace.tp_printk;
- ftrace.buffer_size = SIZE;
- ftrace.alloc_snapshot;
- ftrace.events = EVENT[, EVENT2...];
- ftrace.tracer = TRACER;
- ftrace.event.GROUP.EVENT.filter = FILTER;
- ftrace.event.GROUP.EVENT.actions = ACTION[, ACTION2...];
- ftrace.event.GROUP.EVENT.enable;
- ftrace.event.kprobes.EVENT.probes = PROBE[, PROBE2...];
- ftrace.event.synthetic.EVENT.fields = FIELD[, FIELD2...];
- ftrace.[instance.INSTANCE.]cpumask = CPUMASK;
- ftrace.[instance.INSTANCE.]ftrace.filters
- ftrace.[instance.INSTANCE.]ftrace.notraces
- ftrace.fgraph.filters = FILTER[, FILTER2...];
- ftrace.fgraph.notraces = FILTER[, FILTER2...];
- ftrace.fgraph.max_depth = MAX_DEPTH;

This series can be applied on Steve's tracing tree (ftrace/core) or available on below

https://github.com/mhiramat/linux.git ftrace-boottrace-v3

Usage
=====

With this series, we can setup new kprobe and synthetic events, more complicated event filters and trigger actions including histogram via supplemental kernel cmdline.

We can add filter and actions for each event, define kprobe events, and synthetic events with histogram like below.

ftrace.event {
	task.task_newtask {
		filter = "pid < 128";
		enable;
	}
	kprobes.vfs_read {
		probes = "vfs_read $arg1 $arg2";
		filter = "common_pid < 200";
		enable;
	}
	synthetic.initcall_latency {
		fields = "unsigned long func", "u64 lat";
		actions = "hist:keys=func.sym,lat:vals=lat:sort=lat";
	}
	initcall.initcall_start {
		actions = "hist:keys=func:ts0=common_timestamp.usecs";
	}
	initcall.initcall_finish {
		actions = "hist:keys=func:lat=common_timestamp.usecs-$ts0:onmatch(initcall.initcall_start).initcall_latency(func,$lat)";
	}
}

Also, this supports "instance" node, which allows us to run several tracers for different purpose at once. For example, one tracer is for tracing functions in module alpha, and others tracing module beta, you can write followings.

ftrace.instance {
	foo {
		tracer = "function";
		ftrace-filters = "*:mod:alpha";
	}
	bar {
		tracer = "function";
		ftrace-filters = "*:mod:beta";
	}
}

The instance node also accepts event nodes so that each instance can customize its event tracing.

This boot-time trace also supports ftrace kernel parameters. For example, following kernel parameters

trace_options=sym-addr trace_event=initcall:* tp_printk trace_buf_size=1M ftrace=function ftrace_filter="vfs*"

it can be written in SKC like below.

ftrace {
	options = sym-addr;
	events = "initcall:*";
	tp-printk;
	buffer-size = 1MB;
	ftrace-filters = "vfs*";
}

However, since the initialization timing is different, if you need to trace very early boot, please use normal kernel parameters.

Some Notes
==========

- For specifying the skc data address, there are 2 ways to pass it to kernel. One is expanding arch-specific entries (e.g. setup_data on x86), another is passing it via kernel cmdline. The former may need to expand devicetree anyway (we need /chosen/skc_addr property for some arch.) The latter is more generic, but it may not work if the space on cmdline shorts.
According to Frank's suggestion, I introduced "skc=PADDR,SIZE" option to kernel cmdline, boot loaders must support it. This is generic, and I still have some concerns, like memory reservation. In this version, I used early_param() to reserve SKC memory. If it is too late, we'd better considering to use arch specific data passing.

- Currently, supplemental kernel cmdline doesn't support __setup() routine to avoid confusion, but I think it is possible to support it.

- For saving memory consuming after boot, all variables and APIs are __init and __initdata. Thus runtime memory footprint becomes minimum. But if some loadable modules wants to use it. We have to keep at least query interface and tree nodes on memory.

- As you can see, the EVENT.actions value is a bit ugly. Maybe we can introduce sub-nodes under action node, something like below.

ftrace.event.initcall.initcall_finish.action.hist {
	keys = "func";
	lat = "common_timestamp.usecs-$ts0";
	onmatch = "initcall.initcall_start";
        call = "initcall_latency(func,$lat)";
}

Any suggestions, thoughts?

Thank you,

---

Masami Hiramatsu (19):
skc: Add supplemental kernel cmdline support
skc: Add /proc/sup_cmdline to show SKC key-value list
skc: Add a boot setup routine from cmdline
Documentation: skc: Add a doc for supplemental kernel cmdline
tracing: Apply soft-disabled and filter to tracepoints printk
tracing: kprobes: Output kprobe event to printk buffer
tracing: Expose EXPORT_SYMBOL_GPL symbol
tracing: kprobes: Register to dynevent earlier stage
tracing: Accept different type for synthetic event fields
tracing: Add NULL trace-array check in print_synth_event() tracing/boot: Add boot-time tracing by supplemental kernel cmdline tracing/boot: Add per-event settings
tracing/boot Add kprobe event support
tracing/boot: Add synthetic event support tracing/boot: Add instance node support
tracing/boot: Add cpu_mask option support tracing/boot: Add function tracer filter options tracing/boot: Add function-graph tracer options
Documentation: tracing: Add boot-time tracing document

 Documentation/admin-guide/index.rst             |    1 
 Documentation/admin-guide/kernel-parameters.txt |    6 
 Documentation/admin-guide/skc.rst               |  123 ++++
 Documentation/trace/boottime-trace.rst          |  185 ++++++
 MAINTAINERS                                     |    8 
 arch/Kconfig                                    |    9 
 fs/proc/Makefile                                |    1 
 fs/proc/sup_cmdline.c                           |  106 ++++
 include/linux/skc.h                             |  205 +++++++
 include/linux/trace_events.h                    |    1 
 init/main.c                                     |   54 ++
 kernel/trace/Kconfig                            |    9 
 kernel/trace/Makefile                           |    1 
 kernel/trace/ftrace.c                           |   85 ++-
 kernel/trace/trace.c                            |   90 ++-
 kernel/trace/trace_boot.c                       |  457 +++++++++++++++
 kernel/trace/trace_events.c                     |    3 
 kernel/trace/trace_events_hist.c                |   14 
 kernel/trace/trace_events_trigger.c             |    2 
 kernel/trace/trace_kprobe.c                     |   81 ++-
 lib/Kconfig                                     |    3 
 lib/Makefile                                    |    2 
 lib/skc.c                                       |  694 +++++++++++++++++++++++
 23 files changed, 2044 insertions(+), 96 deletions(-)
 create mode 100644 Documentation/admin-guide/skc.rst
 create mode 100644 Documentation/trace/boottime-trace.rst
 create mode 100644 fs/proc/sup_cmdline.c
 create mode 100644 include/linux/skc.h
 create mode 100644 kernel/trace/trace_boot.c
 create mode 100644 lib/skc.c

--
Masami Hiramatsu (Linaro) <[email protected]>

author maintainer
#1 lore raw

RFC PATCH v3 1/19 skc: Add supplemental kernel cmdline support

Supplemental kernel command line (SKC) allows admin to pass a tree-structured supplemental kernel commandline file (SKC file) when boot up kernel. This expands the kernel command line in efficient way.

SKC file will contain some key-value commands, e.g.

key.word = value1;
another.key.word = value2;

It can fold same keys with braces, also you can write array data. For example,

key {

   word1 {
      setting1 = data;
      setting2;

}
word2.array = "val1", "val2";
}

User can access these key-value pair and tree structure via SKC APIs.

---
 MAINTAINERS         |    6 
 arch/Kconfig        |    9 +
 include/linux/skc.h |  205 +++++++++++++++
 lib/Kconfig         |    3 
 lib/Makefile        |    2 
 lib/skc.c           |  694 +++++++++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 919 insertions(+)
 create mode 100644 include/linux/skc.h
 create mode 100644 lib/skc.c
MAINTAINERS +6 -0
index 035ffc1e16a3..67590c0e37c5 100644
@@ -15362,6 +15362,12 @@ W:	http://www.stlinux.com
 S:	Supported
 F:	drivers/net/ethernet/stmicro/stmmac/
 
+SUPPLEMENTAL KERNEL CMDLINE
+M:	Masami Hiramatsu <[email protected]>
+S:	Maintained
+F:	lib/skc.c
+F:	include/linux/skc.h
+
 SUN3/3X
 M:	Sam Creasey <[email protected]>
 W:	http://sammy.net/sun3/
arch/Kconfig +9 -0
index a7b57dd42c26..14d709ef0396 100644
@@ -24,6 +24,15 @@ config HAVE_IMA_KEXEC
 config HOTPLUG_SMT
 	bool
 
+config SKC
+	bool "Supplemental kernel cmdline"
+	select LIBSKC
+	default y
+	help
+	 Supplemental kernel command line allows system admin to pass a
+	 text file as complemental extension of kernel cmdline when boot.
+	 The bootloader needs to support loading the SKC file.
+
 config OPROFILE
 	tristate "OProfile system profiling"
 	depends on PROFILING
include/linux/skc.h new file +205 -0

205 lines not on this page · raw message

lib/Kconfig +3 -0
index f33d66fc0e86..97a53f400a7c 100644
@@ -568,6 +568,9 @@ config DIMLIB
 config LIBFDT
 	bool
 
+config LIBSKC
+	bool
+
 config OID_REGISTRY
 	tristate
 	help
lib/Makefile +2 -0
index 29c02a924973..52132d0d8ff2 100644
@@ -229,6 +229,8 @@ $(foreach file, $(libfdt_files), \
 	$(eval CFLAGS_$(file) = -I $(srctree)/scripts/dtc/libfdt))
 lib-$(CONFIG_LIBFDT) += $(libfdt_files)
 
+lib-$(CONFIG_LIBSKC) += skc.o
+
 obj-$(CONFIG_RBTREE_TEST) += rbtree_test.o
 obj-$(CONFIG_INTERVAL_TREE_TEST) += interval_tree_test.o
 
lib/skc.c new file +694 -0

694 lines not on this page · raw message

#2 lore raw

Re: [RFC PATCH v3 01/19] skc: Add supplemental kernel cmdline support

Quoted 14 lines · Masami Hiramatsu

Supplemental kernel command line (SKC) allows admin to pass a tree-structured supplemental kernel commandline file (SKC file) when boot up kernel. This expands the kernel command line in efficient way.

SKC file will contain some key-value commands, e.g.

key.word = value1;
another.key.word = value2;

It can fold same keys with braces, also you can write array data. For example,

Quoted 7 lines
key {
   word1 {
      setting1 = data;
      setting2;
   }
   word2.array = "val1", "val2";
}

Why invent a custom file format? You could use YAML (or JSON):

key:
word1:
setting1: data
setting2: true
word2:
- val1
- val2

That would allow you to define a schema for defined options and can easily be manipulated with python (or any language with dictionaries and lists). That does imply adding a YAML parser to the kernel which I'm not sure is a great idea. There is a C parser lib, but working with YAML in C is not that great compared to python.

Another option would be using the DTS format, but as a separate file. That's not unprecedented as u-boot FIT image is a DTB. Then the kernel already has the parser. And you could still have schema now.

A new interface will take a lot of bootloader work to make it easy to use given the user has to manually load some file in the bootloader and know a good address to load it to. Between that and rebuilding the kernel with the configuration, I'd pick rebuilding the kernel. Perhaps this version will highlight that the original proposal was not so bad.

Another thought, maybe you could process the configuration file that's in a readable/editable format into a flat representation that could simply be added to the kernel command line:

key.word1.setting1=data key.word1.setting2 key.word2=val1,val2

That would then use an existing interface and probably simplify the kernel parsing.

Rob

author maintainer
#3 lore raw

Re: [RFC PATCH v3 01/19] skc: Add supplemental kernel cmdline support

Hi Rob,

Thank you for your comment!

Quoted 23 lines · On Mon, 26 Aug 2019 08:27:48 -0500 Rob Herring <[email protected]> wrote:
Quoted 14 lines · Masami Hiramatsu

Supplemental kernel command line (SKC) allows admin to pass a tree-structured supplemental kernel commandline file (SKC file) when boot up kernel. This expands the kernel command line in efficient way.

SKC file will contain some key-value commands, e.g.

key.word = value1;
another.key.word = value2;

It can fold same keys with braces, also you can write array data. For example,

Quoted 7 lines
key {
   word1 {
      setting1 = data;
      setting2;
   }
   word2.array = "val1", "val2";
}
Quoted 1 line

Why invent a custom file format? You could use YAML (or JSON):

Yeah, actually my early idea was using JSON, since it is widely used and many good tools. However, I thought that is not human friendly format :(. I would like to give an easy to read/write but structured interface.

Quoted 8 lines

key:
word1:
setting1: data
setting2: true
word2:
- val1
- val2

(Ah, in above example "array" is just a part of key, and is not a reserved word.)

Quoted 5 lines

That would allow you to define a schema for defined options and can easily be manipulated with python (or any language with dictionaries and lists). That does imply adding a YAML parser to the kernel which I'm not sure is a great idea. There is a C parser lib, but working with YAML in C is not that great compared to python.

Yes, using plain YAML maybe requires user-space coverter to some other format.

Quoted 4 lines

Another option would be using the DTS format, but as a separate file. That's not unprecedented as u-boot FIT image is a DTB. Then the kernel already has the parser. And you could still have schema now.

Yeah, that is what I consider at first. I discussed it with Frank at OSSJ, but he suggested to not use DTS, nor touch current parser in kernel. So I finally convinced not using DTS.

Quoted 3 lines

A new interface will take a lot of bootloader work to make it easy to use given the user has to manually load some file in the bootloader and know a good address to load it to.

Right, that is what I have to do next if this is accepted. As I shown, I modified Qemu and Grub. (Since U-Boot is very flexible, it is easy to load skc file and modify bootargs by manual.)
What I found was, since the bootloaders already supported loading DTB, it would not be so hard to add loading another file :) (curiously, the most complicated part was modifying kernel cmdline)

Quoted 3 lines

Between that and rebuilding the
kernel with the configuration, I'd pick rebuilding the kernel. Perhaps this version will highlight that the original proposal was not so bad.

Maybe for embedded, yes. For admins who use vendor kernel, no.

Quoted 3 lines

Another thought, maybe you could process the configuration file that's in a readable/editable format into a flat representation that could simply be added to the kernel command line:

(BTW, it is easy to make a flat representation data as you can see in /proc/sup_cmdline, which is added by [2/19])

Quoted 5 lines

key.word1.setting1=data key.word1.setting2 key.word2=val1,val2

That would then use an existing interface and probably simplify the kernel parsing.

Hmm, if it is just for passing extended arguments, that will be enough (that was my first version of SKC, here
https://github.com/mhiramat/skc/tree/5f0429c244d1c9f8f84711bc33e1e6f90df62df8 )

But I found that was not enough flexible for my usage. For expressing complex ftrace settings (e.g. nesting options, some options related to other options etc.), I need tree-structured data, something like Devicetree.

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
author maintainer
#4 lore raw

RFC PATCH v3 2/19 skc: Add /proc/sup_cmdline to show SKC key-value list

In reply to Masami Hiramatsu

Add /proc/sup_cmdline which shows the list of key-value pairs in SKC data. Since after boot, all SKC data and tree are removed, this interface just keep a copy of key-value pairs in text.

---
 MAINTAINERS           |    1 
 fs/proc/Makefile      |    1 
 fs/proc/sup_cmdline.c |  106 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 108 insertions(+)
 create mode 100644 fs/proc/sup_cmdline.c
MAINTAINERS +1 -0
index 67590c0e37c5..10dd38311d96 100644
@@ -15366,6 +15366,7 @@ SUPPLEMENTAL KERNEL CMDLINE
 M:	Masami Hiramatsu <[email protected]>
 S:	Maintained
 F:	lib/skc.c
+F:	fs/proc/sup_cmdline.c
 F:	include/linux/skc.h
 
 SUN3/3X
fs/proc/Makefile +1 -0
index ead487e80510..a5d018f9422c 100644
@@ -33,3 +33,4 @@ proc-$(CONFIG_PROC_KCORE)	+= kcore.o
 proc-$(CONFIG_PROC_VMCORE)	+= vmcore.o
 proc-$(CONFIG_PRINTK)	+= kmsg.o
 proc-$(CONFIG_PROC_PAGE_MONITOR)	+= page.o
+proc-$(CONFIG_SKC)	+= sup_cmdline.o
fs/proc/sup_cmdline.c new file +106 -0

106 lines not on this page · raw message

author maintainer
#5 lore raw

RFC PATCH v3 3/19 skc: Add a boot setup routine from cmdline

In reply to Masami Hiramatsu

Add a boot setup routine from cmdline option "skc=ADDR,SIZE". Bootloader has to setup this cmdline option when it loads the skc file on memory. The ADDR must be a physical address.

---
 Documentation/admin-guide/kernel-parameters.txt |    5 ++
 init/main.c                                     |   54 +++++++++++++++++++++++
 2 files changed, 59 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +5 -0
index 47d981a86e2f..9a955b1bd1bf 100644
@@ -4273,6 +4273,11 @@
 	simeth=		[IA-64]
 	simscsi=
 
+	skc=paddr,size	[SKC]
+			Pass the physical memory address and size of loaded
+			supplemental kernel cmdline (SKC) text. This will
+			be treated by bootloader which loads the SKC file.
+
 	slram=		[HW,MTD]
 
 	slab_nomerge	[MM]
init/main.c +54 -0

72 lines not on this page · raw message

author maintainer
#6 lore raw

RFC PATCH v3 4/19 Documentation: skc: Add a doc for supplemental kernel cmdline

In reply to Masami Hiramatsu

Add a documentation for supplemental kernel cmdline under admin-guide, since it is including the syntax of SKC file.

---
 Documentation/admin-guide/index.rst             |    1 
 Documentation/admin-guide/kernel-parameters.txt |    1 
 Documentation/admin-guide/skc.rst               |  123 +++++++++++++++++++++++
 MAINTAINERS                                     |    1 
 4 files changed, 126 insertions(+)
 create mode 100644 Documentation/admin-guide/skc.rst
Documentation/admin-guide/index.rst +1 -0
index 33feab2f4084..44b4cb61003a 100644
@@ -106,6 +106,7 @@ configure specific aspects of kernel behavior to your liking.
    rtc
    svga
    video-output
+   skc
 
 .. only::  subproject and html
 
Documentation/admin-guide/kernel-parameters.txt +1 -0
index 9a955b1bd1bf..334ce59de23a 100644
@@ -4277,6 +4277,7 @@
 			Pass the physical memory address and size of loaded
 			supplemental kernel cmdline (SKC) text. This will
 			be treated by bootloader which loads the SKC file.
+			See Documentation/admin-guide/skc.rst.
 
 	slram=		[HW,MTD]
 
Documentation/admin-guide/skc.rst new file +123 -0

123 lines not on this page · raw message

MAINTAINERS +1 -0
index 10dd38311d96..9c0b97643515 100644
@@ -15368,6 +15368,7 @@ S:	Maintained
 F:	lib/skc.c
 F:	fs/proc/sup_cmdline.c
 F:	include/linux/skc.h
+F:	Documentation/admin-guide/skc.rst
 
 SUN3/3X
 M:	Sam Creasey <[email protected]>
author maintainer
#7 lore raw

RFC PATCH v3 5/19 tracing: Apply soft-disabled and filter to tracepoints printk

In reply to Masami Hiramatsu

Apply soft-disabled and the filter rule of the trace events to the printk output of tracepoints (a.k.a. tp_printk kernel parameter) as same as trace buffer output.

---
 kernel/trace/trace.c |    7 +++++++
 1 file changed, 7 insertions(+)
kernel/trace/trace.c +7 -0
index 525a97fbbc60..ead3a9faa584 100644
@@ -2502,6 +2502,7 @@ static DEFINE_MUTEX(tracepoint_printk_mutex);
 static void output_printk(struct trace_event_buffer *fbuffer)
 {
 	struct trace_event_call *event_call;
+	struct trace_event_file *file;
 	struct trace_event *event;
 	unsigned long flags;
 	struct trace_iterator *iter = tracepoint_print_iter;
@@ -2515,6 +2516,12 @@ static void output_printk(struct trace_event_buffer *fbuffer)
 	    !event_call->event.funcs->trace)
 		return;
 
+	file = fbuffer->trace_file;
+	if (test_bit(EVENT_FILE_FL_SOFT_DISABLED_BIT, &file->flags) ||
+	    (unlikely(file->flags & EVENT_FILE_FL_FILTERED) &&
+	     !filter_match_preds(file->filter, fbuffer->entry)))
+		return;
+
 	event = &fbuffer->trace_file->event_call->event;
 
 	spin_lock_irqsave(&tracepoint_iter_lock, flags);
author maintainer
#8 lore raw

RFC PATCH v3 6/19 tracing: kprobes: Output kprobe event to printk buffer

In reply to Masami Hiramatsu

Since kprobe-events use event_trigger_unlock_commit_regs() directly, that events doesn't show up in printk buffer if "tp_printk" is set.

Use trace_event_buffer_commit() in kprobe events so that it can invoke output_printk() as same as other trace events.

---
 include/linux/trace_events.h |    1 +
 kernel/trace/trace.c         |    4 +--
 kernel/trace/trace_events.c  |    1 +
 kernel/trace/trace_kprobe.c  |   57 +++++++++++++++++++++---------------------
 4 files changed, 32 insertions(+), 31 deletions(-)
include/linux/trace_events.h +1 -0
index 5150436783e8..8912ccdb3d4b 100644
@@ -211,6 +211,7 @@ struct trace_event_buffer {
 	void				*entry;
 	unsigned long			flags;
 	int				pc;
+	struct pt_regs			*regs;
 };
 
 void *trace_event_buffer_reserve(struct trace_event_buffer *fbuffer,
kernel/trace/trace.c +2 -2
index ead3a9faa584..605faf584164 100644
@@ -2572,9 +2572,9 @@ void trace_event_buffer_commit(struct trace_event_buffer *fbuffer)
 	if (static_key_false(&tracepoint_printk_key.key))
 		output_printk(fbuffer);
 
-	event_trigger_unlock_commit(fbuffer->trace_file, fbuffer->buffer,
+	event_trigger_unlock_commit_regs(fbuffer->trace_file, fbuffer->buffer,
 				    fbuffer->event, fbuffer->entry,
-				    fbuffer->flags, fbuffer->pc);
+				    fbuffer->flags, fbuffer->pc, fbuffer->regs);
 }
 EXPORT_SYMBOL_GPL(trace_event_buffer_commit);
 
kernel/trace/trace_events.c +1 -0
index c7506bc81b75..22cf08bd2317 100644
@@ -271,6 +271,7 @@ void *trace_event_buffer_reserve(struct trace_event_buffer *fbuffer,
 	if (!fbuffer->event)
 		return NULL;
 
+	fbuffer->regs = NULL;
 	fbuffer->entry = ring_buffer_event_data(fbuffer->event);
 	return fbuffer->entry;
 }
kernel/trace/trace_kprobe.c +28 -29

97 lines not on this page · raw message

author maintainer
#9 lore raw

RFC PATCH v3 7/19 tracing: Expose EXPORT_SYMBOL_GPL symbol

In reply to Masami Hiramatsu

Since ftrace_set_clr_event is already exported by EXPORT_SYMBOL_GPL, it should not be static.

---
 kernel/trace/trace_events.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/trace/trace_events.c +1 -1
index 22cf08bd2317..c2d38048edae 100644
@@ -788,7 +788,7 @@ static int __ftrace_set_clr_event(struct trace_array *tr, const char *match,
 	return ret;
 }
 
-static int ftrace_set_clr_event(struct trace_array *tr, char *buf, int set)
+int ftrace_set_clr_event(struct trace_array *tr, char *buf, int set)
 {
 	char *event = NULL, *sub = NULL, *match;
 	int ret;
author maintainer
#10 lore raw

RFC PATCH v3 8/19 tracing: kprobes: Register to dynevent earlier stage

In reply to Masami Hiramatsu

Register kprobe event to dynevent in subsys_initcall level. This will allow kernel to register new kprobe events in fs_initcall level via trace_run_command.

---
 kernel/trace/trace_kprobe.c |   19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)
kernel/trace/trace_kprobe.c +15 -4
index 6c5145525f90..5135c07b6557 100644
@@ -1484,11 +1484,12 @@ static __init void setup_boot_kprobe_events(void)
 	enable_boot_kprobe_events();
 }
 
-/* Make a tracefs interface for controlling probe points */
-static __init int init_kprobe_trace(void)
+/*
+ * Register dynevent at subsys_initcall. This allows kernel to setup kprobe
+ * events in fs_initcall without tracefs.
+ */
+static __init int init_kprobe_trace_early(void)
 {
-	struct dentry *d_tracer;
-	struct dentry *entry;
 	int ret;
 
 	ret = dyn_event_register(&trace_kprobe_ops);
@@ -1498,6 +1499,16 @@ static __init int init_kprobe_trace(void)
 	if (register_module_notifier(&trace_kprobe_module_nb))
 		return -EINVAL;
 
+	return 0;
+}
+subsys_initcall(init_kprobe_trace_early);
+
+/* Make a tracefs interface for controlling probe points */
+static __init int init_kprobe_trace(void)
+{
+	struct dentry *d_tracer;
+	struct dentry *entry;
+
 	d_tracer = tracing_init_dentry();
 	if (IS_ERR(d_tracer))
 		return 0;
author maintainer
#11 lore raw

RFC PATCH v3 9/19 tracing: Accept different type for synthetic event fields

In reply to Masami Hiramatsu

Make the synthetic event accepts a different type field to record. However, the size and signed flag must be same.

---
 kernel/trace/trace_events_hist.c |    7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
kernel/trace/trace_events_hist.c +5 -2
index ca6b0dff60c5..a7f447195143 100644
@@ -4063,8 +4063,11 @@ static int check_synth_field(struct synth_event *event,
 
 	field = event->fields[field_pos];
 
-	if (strcmp(field->type, hist_field->type) != 0)
-		return -EINVAL;
+	if (strcmp(field->type, hist_field->type) != 0) {
+		if (field->size != hist_field->size ||
+		    field->is_signed != hist_field->is_signed)
+			return -EINVAL;
+	}
 
 	return 0;
 }
author maintainer
#12 lore raw

RFC PATCH v3 10/19 tracing: Add NULL trace-array check in print_synth_event()

In reply to Masami Hiramatsu

Add NULL trace-array check in print_synth_event(), because if we enable tp_printk option, iter->tr can be NULL.

---
 kernel/trace/trace_events_hist.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/trace/trace_events_hist.c +1 -1
index a7f447195143..db973928e580 100644
@@ -822,7 +822,7 @@ static enum print_line_t print_synth_event(struct trace_iterator *iter,
 		fmt = synth_field_fmt(se->fields[i]->type);
 
 		/* parameter types */
-		if (tr->trace_flags & TRACE_ITER_VERBOSE)
+		if (tr && tr->trace_flags & TRACE_ITER_VERBOSE)
 			trace_seq_printf(s, "%s ", fmt);
 
 		snprintf(print_fmt, sizeof(print_fmt), "%%s=%s%%s", fmt);
author maintainer
#13 lore raw

RFC PATCH v3 11/19 tracing/boot: Add boot-time tracing by supplemental kernel cmdline

In reply to Masami Hiramatsu

Setup tracing options by supplemental kernel cmdline (skc) in addition to kernel parameters. In this patch, add following commands support

- ftrace.options = OPT1[,OPT2...];
Enable given ftrace options.

- ftrace.trace_clock = CLOCK;
Set given CLOCK to ftrace's trace_clock.

- ftrace.dump_on_oops [= MODE];
Dump ftrace on Oops. If MODE = 1 or omitted, dump trace buffer on all CPUs. If MODE = 2, dump a buffer on a CPU which kicks Oops.

- ftrace.traceoff_on_warning;
Stop tracing if WARN_ON() occurs.

- ftrace.tp_printk;
Output trace-event data on printk buffer too.

- ftrace.buffer_size = SIZE;
Configure ftrace buffer size to SIZE. You can use "KB" or "MB" for that SIZE.

- ftrace.alloc_snapshot;
Allocate snapshot buffer.

- ftrace.events = EVENT[, EVENT2...];
Enable given events on boot. You can use a wild card in EVENT.

- ftrace.tracer = TRACER;
Set TRACER to current tracer on boot. (e.g. function)

Since the kernel parameter is limited length, sometimes there is no space to setup the tracing options. This will read the tracing options from skc "ftrace" prefix and setup tracers at boot.

Note that this is not replacing the kernel parameters, because this skc based setting is later than that. If you want to trace earlier boot events, you still need kernel parameters.

---
 kernel/trace/Kconfig      |    9 +++
 kernel/trace/Makefile     |    1 
 kernel/trace/trace.c      |   38 ++++++++----
 kernel/trace/trace_boot.c |  137 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 172 insertions(+), 13 deletions(-)
 create mode 100644 kernel/trace/trace_boot.c
kernel/trace/Kconfig +9 -0
index 98da8998c25c..0f831adb4e4a 100644
@@ -797,6 +797,15 @@ config GCOV_PROFILE_FTRACE
 	  Note that on a kernel compiled with this config, ftrace will
 	  run significantly slower.
 
+config BOOTTIME_TRACING
+	bool "Boot-time Tracing support"
+	depends on SKC && TRACING
+	default y
+	help
+	  Enable developer to setup ftrace subsystem via supplemental
+	  kernel cmdline at boot time for debugging (tracing) driver
+	  initialization and boot process.
+
 endif # FTRACE
 
 endif # TRACING_SUPPORT
kernel/trace/Makefile +1 -0
index c2b2148bb1d2..f9d3c2c72fb5 100644
@@ -82,6 +82,7 @@ endif
 obj-$(CONFIG_DYNAMIC_EVENTS) += trace_dynevent.o
 obj-$(CONFIG_PROBE_EVENTS) += trace_probe.o
 obj-$(CONFIG_UPROBE_EVENTS) += trace_uprobe.o
+obj-$(CONFIG_BOOTTIME_TRACING) += trace_boot.o
 
 obj-$(CONFIG_TRACEPOINT_BENCHMARK) += trace_benchmark.o
 
kernel/trace/trace.c +25 -13
index 605faf584164..69400a87e48f 100644
@@ -158,7 +158,7 @@ union trace_eval_map_item {
 static union trace_eval_map_item *trace_eval_maps;
 #endif /* CONFIG_TRACE_EVAL_MAP_FILE */
 
-static int tracing_set_tracer(struct trace_array *tr, const char *buf);
+int tracing_set_tracer(struct trace_array *tr, const char *buf);
 static void ftrace_trace_userstack(struct ring_buffer *buffer,
 				   unsigned long flags, int pc);
 
@@ -178,6 +178,11 @@ static int __init set_cmdline_ftrace(char *str)
 }
 __setup("ftrace=", set_cmdline_ftrace);
 
+void __init trace_init_dump_on_oops(int mode)
+{
+	ftrace_dump_on_oops = mode;
+}
+
 static int __init set_ftrace_dump_on_oops(char *str)
 {
 	if (*str++ != '=' || !*str) {
@@ -4614,7 +4619,7 @@ int set_tracer_flag(struct trace_array *tr, unsigned int mask, int enabled)
 	return 0;
 }
 
-static int trace_set_options(struct trace_array *tr, char *option)
+int trace_set_options(struct trace_array *tr, char *option)
 {
 	char *cmp;
 	int neg = 0;
@@ -5505,8 +5510,8 @@ static int __tracing_resize_ring_buffer(struct trace_array *tr,
 	return ret;
 }
 
-static ssize_t tracing_resize_ring_buffer(struct trace_array *tr,
-					  unsigned long size, int cpu_id)
+ssize_t tracing_resize_ring_buffer(struct trace_array *tr,
+				  unsigned long size, int cpu_id)
 {
 	int ret = size;
 
@@ -5585,7 +5590,7 @@ static void add_tracer_options(struct trace_array *tr, struct tracer *t)
 	create_trace_option_files(tr, t);
 }
 
-static int tracing_set_tracer(struct trace_array *tr, const char *buf)
+int tracing_set_tracer(struct trace_array *tr, const char *buf)
 {
 	struct tracer *t;
 #ifdef CONFIG_TRACER_MAX_TRACE
@@ -9170,16 +9175,23 @@ __init static int tracer_alloc_buffers(void)
 	return ret;
 }
 
+void __init trace_init_tracepoint_printk(void)
+{
+	tracepoint_printk = 1;
+
+	tracepoint_print_iter =
+		kmalloc(sizeof(*tracepoint_print_iter), GFP_KERNEL);
+	if (WARN_ON(!tracepoint_print_iter))
+		tracepoint_printk = 0;
+	else
+		static_key_enable(&tracepoint_printk_key.key);
+}
+
 void __init early_trace_init(void)
 {
-	if (tracepoint_printk) {
-		tracepoint_print_iter =
-			kmalloc(sizeof(*tracepoint_print_iter), GFP_KERNEL);
-		if (WARN_ON(!tracepoint_print_iter))
-			tracepoint_printk = 0;
-		else
-			static_key_enable(&tracepoint_printk_key.key);
-	}
+	if (tracepoint_printk)
+		trace_init_tracepoint_printk();
+
 	tracer_alloc_buffers();
 }
 
kernel/trace/trace_boot.c new file +137 -0

137 lines not on this page · raw message

author maintainer
#14 lore raw

RFC PATCH v3 12/19 tracing/boot: Add per-event settings

In reply to Masami Hiramatsu

Add per-event settings for boottime tracing. User can set filter, actions and enable on each event on boot. The event entries are under ftrace.event.GROUP.EVENT node (note that the option key includes event's group name and event name.) This supports below commands.

- ftrace.event.GROUP.EVENT.enable;
Enables GROUP:EVENT tracing.

- ftrace.event.GROUP.EVENT.filter = FILTER;
Set FILTER rule to the GROUP:EVENT.

- ftrace.event.GROUP.EVENT.actions = ACTION[, ACTION2...]; Set ACTIONs to the GROUP:EVENT.

For example,

  ftrace.event.sched.sched_process_exec {
                filter = "pid < 128";
		enable;

}

this will enable tracing "sched:sched_process_exec" event with "pid < 128" filter.

---
 kernel/trace/trace_boot.c           |   60 +++++++++++++++++++++++++++++++++++
 kernel/trace/trace_events_trigger.c |    2 +
 2 files changed, 61 insertions(+), 1 deletion(-)
kernel/trace/trace_boot.c +60 -0
index cc5e81368065..cfd628c16761 100644
@@ -80,6 +80,7 @@ trace_boot_set_ftrace_options(struct trace_array *tr, struct skc_node *node)
 
 #ifdef CONFIG_EVENT_TRACING
 extern int ftrace_set_clr_event(struct trace_array *tr, char *buf, int set);
+extern int trigger_process_regex(struct trace_event_file *file, char *buff);
 
 static void __init
 trace_boot_enable_events(struct trace_array *tr, struct skc_node *node)
@@ -98,8 +99,66 @@ trace_boot_enable_events(struct trace_array *tr, struct skc_node *node)
 			pr_err("Failed to enable event: %s\n", p);
 	}
 }
+
+static void __init
+trace_boot_init_one_event(struct trace_array *tr, struct skc_node *gnode,
+			  struct skc_node *enode)
+{
+	struct trace_event_file *file;
+	struct skc_node *anode;
+	char buf[MAX_BUF_LEN];
+	const char *p, *group, *event;
+
+	group = skc_node_get_data(gnode);
+	event = skc_node_get_data(enode);
+
+	mutex_lock(&event_mutex);
+	file = find_event_file(tr, group, event);
+	if (!file) {
+		pr_err("Failed to find event: %s:%s\n", group, event);
+		goto out;
+	}
+
+	p = skc_node_find_value(enode, "filter", NULL);
+	if (p && *p != '\0') {
+		if (strlcpy(buf, p, ARRAY_SIZE(buf)) >= ARRAY_SIZE(buf))
+			pr_err("filter string is too long: %s\n", p);
+		else if (apply_event_filter(file, buf) < 0)
+			pr_err("Failed to apply filter: %s\n", buf);
+	}
+
+	skc_node_for_each_array_value(enode, "actions", anode, p) {
+		if (strlcpy(buf, p, ARRAY_SIZE(buf)) >= ARRAY_SIZE(buf))
+			pr_err("action string is too long: %s\n", p);
+		else if (trigger_process_regex(file, buf) < 0)
+			pr_err("Failed to apply an action: %s\n", buf);
+	}
+
+	if (skc_node_find_value(enode, "enable", NULL)) {
+		if (trace_event_enable_disable(file, 1, 0) < 0)
+			pr_err("Failed to enable event node: %s:%s\n",
+				group, event);
+	}
+out:
+	mutex_unlock(&event_mutex);
+}
+
+static void __init
+trace_boot_init_events(struct trace_array *tr, struct skc_node *node)
+{
+	struct skc_node *gnode, *enode;
+
+	node = skc_node_find_child(node, "event");
+	if (!node)
+		return;
+	/* per-event key starts with "event.GROUP.EVENT" */
+	skc_node_for_each_child(node, gnode)
+		skc_node_for_each_child(gnode, enode)
+			trace_boot_init_one_event(tr, gnode, enode);
+}
 #else
 #define trace_boot_enable_events(tr, node) do {} while (0)
+#define trace_boot_init_events(tr, node) do {} while (0)
 #endif
 
 static void __init
@@ -128,6 +187,7 @@ static int __init trace_boot_init(void)
 		return 0;
 
 	trace_boot_set_ftrace_options(tr, trace_node);
+	trace_boot_init_events(tr, trace_node);
 	trace_boot_enable_events(tr, trace_node);
 	trace_boot_enable_tracer(tr, trace_node);
 
kernel/trace/trace_events_trigger.c +1 -1
index 2a2912cb4533..74a19c18219f 100644
@@ -208,7 +208,7 @@ static int event_trigger_regex_open(struct inode *inode, struct file *file)
 	return ret;
 }
 
-static int trigger_process_regex(struct trace_event_file *file, char *buff)
+int trigger_process_regex(struct trace_event_file *file, char *buff)
 {
 	char *command, *next = buff;
 	struct event_command *p;
author maintainer
#15 lore raw

RFC PATCH v3 13/19 tracing/boot Add kprobe event support

In reply to Masami Hiramatsu

Add kprobe event support on event node. If the group name of event is "kprobes", boottime trace defines new probe event according to "probes" values.

- ftrace.event.kprobes.EVENT.probes = PROBE[, PROBE2...]; Defines new kprobe event based on PROBEs. It is able to define multiple probes on one event, but those must have same type of arguments.

For example,

 ftrace.events.kprobes.myevent {
	probes = "vfs_read $arg1 $arg2";
	enable;

}

This will add kprobes:myevent on vfs_read with the 1st and the 2nd arguments.

---
 kernel/trace/trace_boot.c   |   46 +++++++++++++++++++++++++++++++++++++++++++
 kernel/trace/trace_kprobe.c |    5 +++++
 2 files changed, 51 insertions(+)
kernel/trace/trace_boot.c +46 -0
index cfd628c16761..40c89c7ceee0 100644
@@ -100,6 +100,48 @@ trace_boot_enable_events(struct trace_array *tr, struct skc_node *node)
 	}
 }
 
+#ifdef CONFIG_KPROBE_EVENTS
+extern int trace_kprobe_run_command(const char *command);
+
+static int __init
+trace_boot_add_kprobe_event(struct skc_node *node, const char *event)
+{
+	struct skc_node *anode;
+	char buf[MAX_BUF_LEN];
+	const char *val;
+	char *p;
+	int len;
+
+	len = snprintf(buf, ARRAY_SIZE(buf) - 1, "p:kprobes/%s ", event);
+	if (len >= ARRAY_SIZE(buf)) {
+		pr_err("Event name is too long: %s\n", event);
+		return -E2BIG;
+	}
+	p = buf + len;
+	len = ARRAY_SIZE(buf) - len;
+
+	skc_node_for_each_array_value(node, "probes", anode, val) {
+		if (strlcpy(p, val, len) >= len) {
+			pr_err("Probe definition is too long: %s\n", val);
+			return -E2BIG;
+		}
+		if (trace_kprobe_run_command(buf) < 0) {
+			pr_err("Failed to add probe: %s\n", buf);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+#else
+static inline int __init
+trace_boot_add_kprobe_event(struct skc_node *node, const char *event)
+{
+	pr_err("Kprobe event is not supported.\n");
+	return -ENOTSUPP;
+}
+#endif
+
 static void __init
 trace_boot_init_one_event(struct trace_array *tr, struct skc_node *gnode,
 			  struct skc_node *enode)
@@ -112,6 +154,10 @@ trace_boot_init_one_event(struct trace_array *tr, struct skc_node *gnode,
 	group = skc_node_get_data(gnode);
 	event = skc_node_get_data(enode);
 
+	if (!strcmp(group, "kprobes"))
+		if (trace_boot_add_kprobe_event(enode, event) < 0)
+			return;
+
 	mutex_lock(&event_mutex);
 	file = find_event_file(tr, group, event);
 	if (!file) {
kernel/trace/trace_kprobe.c +5 -0
index 5135c07b6557..03ce60928c18 100644
@@ -728,6 +728,11 @@ static int create_or_delete_trace_kprobe(int argc, char **argv)
 	return ret == -ECANCELED ? -EINVAL : ret;
 }
 
+int trace_kprobe_run_command(const char *command)
+{
+	return trace_run_command(command, create_or_delete_trace_kprobe);
+}
+
 static int trace_kprobe_release(struct dyn_event *ev)
 {
 	struct trace_kprobe *tk = to_trace_kprobe(ev);
author maintainer
#16 lore raw

RFC PATCH v3 14/19 tracing/boot: Add synthetic event support

In reply to Masami Hiramatsu

Add synthetic event node support. The synthetic event is a kind of event node, but the group name is "synthetic".

- ftrace.event.synthetic.EVENT.fields = FIELD[, FIELD2...]; Defines new synthetic event with FIELDs. Each field should be "type varname".

The synthetic node requires "fields" string arraies, which defines the fields as same as tracing/synth_events interface.

---
 kernel/trace/trace_boot.c        |   47 ++++++++++++++++++++++++++++++++++++++
 kernel/trace/trace_events_hist.c |    5 ++++
 2 files changed, 52 insertions(+)
kernel/trace/trace_boot.c +47 -0
index 40c89c7ceee0..2e9fddff660f 100644
@@ -142,6 +142,50 @@ trace_boot_add_kprobe_event(struct skc_node *node, const char *event)
 }
 #endif
 
+#ifdef CONFIG_HIST_TRIGGERS
+extern int synth_event_run_command(const char *command);
+
+static int __init
+trace_boot_add_synth_event(struct skc_node *node, const char *event)
+{
+	struct skc_node *anode;
+	char buf[MAX_BUF_LEN], *q;
+	const char *p;
+	int len, delta, ret;
+
+	len = ARRAY_SIZE(buf);
+	delta = snprintf(buf, len, "%s", event);
+	if (delta >= len) {
+		pr_err("Event name is too long: %s\n", event);
+		return -E2BIG;
+	}
+	len -= delta; q = buf + delta;
+
+	skc_node_for_each_array_value(node, "fields", anode, p) {
+		delta = snprintf(q, len, " %s;", p);
+		if (delta >= len) {
+			pr_err("fields string is too long: %s\n", p);
+			return -E2BIG;
+		}
+		len -= delta; q += delta;
+	}
+
+	ret = synth_event_run_command(buf);
+	if (ret < 0)
+		pr_err("Failed to add synthetic event: %s\n", buf);
+
+
+	return ret;
+}
+#else
+static inline int __init
+trace_boot_add_synth_event(struct skc_node *node, const char *event)
+{
+	pr_err("Synthetic event is not supported.\n");
+	return -ENOTSUPP;
+}
+#endif
+
 static void __init
 trace_boot_init_one_event(struct trace_array *tr, struct skc_node *gnode,
 			  struct skc_node *enode)
@@ -157,6 +201,9 @@ trace_boot_init_one_event(struct trace_array *tr, struct skc_node *gnode,
 	if (!strcmp(group, "kprobes"))
 		if (trace_boot_add_kprobe_event(enode, event) < 0)
 			return;
+	if (!strcmp(group, "synthetic"))
+		if (trace_boot_add_synth_event(enode, event) < 0)
+			return;
 
 	mutex_lock(&event_mutex);
 	file = find_event_file(tr, group, event);
kernel/trace/trace_events_hist.c +5 -0
index db973928e580..e7f5d0a353e2 100644
@@ -1343,6 +1343,11 @@ static int create_or_delete_synth_event(int argc, char **argv)
 	return ret == -ECANCELED ? -EINVAL : ret;
 }
 
+int synth_event_run_command(const char *command)
+{
+	return trace_run_command(command, create_or_delete_synth_event);
+}
+
 static int synth_event_create(int argc, const char **argv)
 {
 	const char *name = argv[0];
author maintainer
#17 lore raw

RFC PATCH v3 15/19 tracing/boot: Add instance node support

In reply to Masami Hiramatsu

Add instance node support to boottime tracing. User can set some options and event nodes under instance node.

- ftrace.instance.INSTANCE[...];
Add new INSTANCE instance. Some options and event nodes are acceptable for instance node.

---
 kernel/trace/trace_boot.c |   72 ++++++++++++++++++++++++++++++++++++---------
 1 file changed, 57 insertions(+), 15 deletions(-)
kernel/trace/trace_boot.c +57 -15
index 2e9fddff660f..74cffecd12a4 100644
@@ -21,15 +21,15 @@ extern int tracing_set_tracer(struct trace_array *tr, const char *buf);
 extern void __init trace_init_tracepoint_printk(void);
 extern ssize_t tracing_resize_ring_buffer(struct trace_array *tr,
 					  unsigned long size, int cpu_id);
+extern struct trace_array *trace_array_create(const char *name);
 
 static void __init
-trace_boot_set_ftrace_options(struct trace_array *tr, struct skc_node *node)
+trace_boot_set_instance_options(struct trace_array *tr, struct skc_node *node)
 {
 	struct skc_node *anode;
 	const char *p;
 	char buf[MAX_BUF_LEN];
 	unsigned long v = 0;
-	int err;
 
 	/* Common ftrace options */
 	skc_node_for_each_array_value(node, "options", anode, p) {
@@ -48,6 +48,23 @@ trace_boot_set_ftrace_options(struct trace_array *tr, struct skc_node *node)
 			pr_err("Failed to set trace clock: %s\n", p);
 	}
 
+	p = skc_node_find_value(node, "buffer_size", NULL);
+	if (p && *p != '\0') {
+		v = memparse(p, NULL);
+		if (v < PAGE_SIZE)
+			pr_err("Buffer size is too small: %s\n", p);
+		if (tracing_resize_ring_buffer(tr, v, RING_BUFFER_ALL_CPUS) < 0)
+			pr_err("Failed to resize trace buffer to %s\n", p);
+	}
+}
+
+static void __init
+trace_boot_set_global_options(struct trace_array *tr, struct skc_node *node)
+{
+	unsigned long v = 0;
+	const char *p;
+	int err;
+
 	/* Command line boot options */
 	p = skc_node_find_value(node, "dump_on_oops", NULL);
 	if (p) {
@@ -64,15 +81,6 @@ trace_boot_set_ftrace_options(struct trace_array *tr, struct skc_node *node)
 	if (skc_node_find_value(node, "tp_printk", NULL))
 		trace_init_tracepoint_printk();
 
-	p = skc_node_find_value(node, "buffer_size", NULL);
-	if (p && *p != '\0') {
-		v = memparse(p, NULL);
-		if (v < PAGE_SIZE)
-			pr_err("Buffer size is too small: %s\n", p);
-		if (tracing_resize_ring_buffer(tr, v, RING_BUFFER_ALL_CPUS) < 0)
-			pr_err("Failed to resize trace buffer to %s\n", p);
-	}
-
 	if (skc_node_find_value(node, "alloc_snapshot", NULL))
 		if (tracing_alloc_snapshot() < 0)
 			pr_err("Failed to allocate snapshot buffer\n");
@@ -266,6 +274,40 @@ trace_boot_enable_tracer(struct trace_array *tr, struct skc_node *node)
 	}
 }
 
+static void __init
+trace_boot_init_one_instance(struct trace_array *tr, struct skc_node *node)
+{
+	trace_boot_set_instance_options(tr, node);
+	trace_boot_init_events(tr, node);
+	trace_boot_enable_events(tr, node);
+	trace_boot_enable_tracer(tr, node);
+}
+
+static void __init
+trace_boot_init_instances(struct skc_node *node)
+{
+	struct skc_node *inode;
+	struct trace_array *tr;
+	const char *p;
+
+	node = skc_node_find_child(node, "instance");
+	if (!node)
+		return;
+
+	skc_node_for_each_child(node, inode) {
+		p = skc_node_get_data(inode);
+		if (!p || *p == '\0')
+			continue;
+
+		tr = trace_array_create(p);
+		if (IS_ERR(tr)) {
+			pr_err("Failed to create instance %s\n", p);
+			continue;
+		}
+		trace_boot_init_one_instance(tr, inode);
+	}
+}
+
 static int __init trace_boot_init(void)
 {
 	struct skc_node *trace_node;
@@ -279,10 +321,10 @@ static int __init trace_boot_init(void)
 	if (!tr)
 		return 0;
 
-	trace_boot_set_ftrace_options(tr, trace_node);
-	trace_boot_init_events(tr, trace_node);
-	trace_boot_enable_events(tr, trace_node);
-	trace_boot_enable_tracer(tr, trace_node);
+	trace_boot_set_global_options(tr, trace_node);
+	/* Global trace array is also one instance */
+	trace_boot_init_one_instance(tr, trace_node);
+	trace_boot_init_instances(trace_node);
 
 	return 0;
 }
author maintainer
#18 lore raw

RFC PATCH v3 16/19 tracing/boot: Add cpu_mask option support

In reply to Masami Hiramatsu

Add ftrace.cpumask option support for setting trace cpumask.

- ftrace.[instance.INSTANCE.]cpumask = CPUMASK;
Set the trace cpumask. Note that the CPUMASK should be a string which <tracefs>/tracing_cpumask can accepts.

---
 kernel/trace/trace.c      |   41 ++++++++++++++++++++++++++++-------------
 kernel/trace/trace_boot.c |   14 ++++++++++++++
 2 files changed, 42 insertions(+), 13 deletions(-)
kernel/trace/trace.c +28 -13
index 69400a87e48f..bfe513e472d2 100644
@@ -4437,20 +4437,12 @@ tracing_cpumask_read(struct file *filp, char __user *ubuf,
 	return count;
 }
 
-static ssize_t
-tracing_cpumask_write(struct file *filp, const char __user *ubuf,
-		      size_t count, loff_t *ppos)
+int tracing_set_cpumask(struct trace_array *tr, cpumask_var_t tracing_cpumask_new)
 {
-	struct trace_array *tr = file_inode(filp)->i_private;
-	cpumask_var_t tracing_cpumask_new;
-	int err, cpu;
-
-	if (!alloc_cpumask_var(&tracing_cpumask_new, GFP_KERNEL))
-		return -ENOMEM;
+	int cpu;
 
-	err = cpumask_parse_user(ubuf, count, tracing_cpumask_new);
-	if (err)
-		goto err_unlock;
+	if (!tr)
+		return -EINVAL;
 
 	local_irq_disable();
 	arch_spin_lock(&tr->max_lock);
@@ -4474,11 +4466,34 @@ tracing_cpumask_write(struct file *filp, const char __user *ubuf,
 	local_irq_enable();
 
 	cpumask_copy(tr->tracing_cpumask, tracing_cpumask_new);
+
+	return 0;
+}
+
+static ssize_t
+tracing_cpumask_write(struct file *filp, const char __user *ubuf,
+		      size_t count, loff_t *ppos)
+{
+	struct trace_array *tr = file_inode(filp)->i_private;
+	cpumask_var_t tracing_cpumask_new;
+	int err;
+
+	if (!alloc_cpumask_var(&tracing_cpumask_new, GFP_KERNEL))
+		return -ENOMEM;
+
+	err = cpumask_parse_user(ubuf, count, tracing_cpumask_new);
+	if (err)
+		goto err_free;
+
+	err = tracing_set_cpumask(tr, tracing_cpumask_new);
+	if (err)
+		goto err_free;
+
 	free_cpumask_var(tracing_cpumask_new);
 
 	return count;
 
-err_unlock:
+err_free:
 	free_cpumask_var(tracing_cpumask_new);
 
 	return err;
kernel/trace/trace_boot.c +14 -0
index 74cffecd12a4..755a2040aebf 100644
@@ -22,6 +22,8 @@ extern void __init trace_init_tracepoint_printk(void);
 extern ssize_t tracing_resize_ring_buffer(struct trace_array *tr,
 					  unsigned long size, int cpu_id);
 extern struct trace_array *trace_array_create(const char *name);
+extern int tracing_set_cpumask(struct trace_array *tr,
+				cpumask_var_t tracing_cpumask_new);
 
 static void __init
 trace_boot_set_instance_options(struct trace_array *tr, struct skc_node *node)
@@ -56,6 +58,18 @@ trace_boot_set_instance_options(struct trace_array *tr, struct skc_node *node)
 		if (tracing_resize_ring_buffer(tr, v, RING_BUFFER_ALL_CPUS) < 0)
 			pr_err("Failed to resize trace buffer to %s\n", p);
 	}
+
+	p = skc_node_find_value(node, "cpumask", NULL);
+	if (p && *p != '\0') {
+		cpumask_var_t new_mask;
+
+		if (alloc_cpumask_var(&new_mask, GFP_KERNEL)) {
+			if (cpumask_parse(p, new_mask) < 0 ||
+			    tracing_set_cpumask(tr, new_mask) < 0)
+				pr_err("Failed to set new CPU mask %s\n", p);
+			free_cpumask_var(new_mask);
+		}
+	}
 }
 
 static void __init
author maintainer
#19 lore raw

RFC PATCH v3 17/19 tracing/boot: Add function tracer filter options

In reply to Masami Hiramatsu

Add below function-tracer filter options to boottime tracer.

- ftrace.[instance.INSTANCE.]ftrace.filters
This will take an array of tracing function filter rules

- ftrace.[instance.INSTANCE.]ftrace.notraces
This will take an array of NON-tracing function filter rules

---
 kernel/trace/trace_boot.c |   40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
kernel/trace/trace_boot.c +40 -0
index 755a2040aebf..942ca8d3fcc8 100644
@@ -276,11 +276,51 @@ trace_boot_init_events(struct trace_array *tr, struct skc_node *node)
 #define trace_boot_init_events(tr, node) do {} while (0)
 #endif
 
+#ifdef CONFIG_FUNCTION_TRACER
+extern bool ftrace_filter_param __initdata;
+extern int ftrace_set_filter(struct ftrace_ops *ops, unsigned char *buf,
+			     int len, int reset);
+extern int ftrace_set_notrace(struct ftrace_ops *ops, unsigned char *buf,
+			      int len, int reset);
+static void __init
+trace_boot_set_ftrace_filter(struct trace_array *tr, struct skc_node *node)
+{
+	struct skc_node *anode;
+	const char *p;
+	char *q;
+
+	skc_node_for_each_array_value(node, "ftrace.filters", anode, p) {
+		q = kstrdup(p, GFP_KERNEL);
+		if (!q)
+			return;
+		if (ftrace_set_filter(tr->ops, q, strlen(q), 0) < 0)
+			pr_err("Failed to add %s to ftrace filter\n", p);
+		else
+			ftrace_filter_param = true;
+		kfree(q);
+	}
+	skc_node_for_each_array_value(node, "ftrace.notraces", anode, p) {
+		q = kstrdup(p, GFP_KERNEL);
+		if (!q)
+			return;
+		if (ftrace_set_notrace(tr->ops, q, strlen(q), 0) < 0)
+			pr_err("Failed to add %s to ftrace filter\n", p);
+		else
+			ftrace_filter_param = true;
+		kfree(q);
+	}
+}
+#else
+#define trace_boot_set_ftrace_filter(tr, node) do {} while (0)
+#endif
+
 static void __init
 trace_boot_enable_tracer(struct trace_array *tr, struct skc_node *node)
 {
 	const char *p;
 
+	trace_boot_set_ftrace_filter(tr, node);
+
 	p = skc_node_find_value(node, "tracer", NULL);
 	if (p && *p != '\0') {
 		if (tracing_set_tracer(tr, p) < 0)
author maintainer
#20 lore raw

RFC PATCH v3 18/19 tracing/boot: Add function-graph tracer options

In reply to Masami Hiramatsu

Add following function-graph tracer related options

- ftrace.fgraph.filters = FILTER[, FILTER2...];
Add fgraph tracing function filters.

- ftrace.fgraph.notraces = FILTER[, FILTER2...];
Add fgraph non tracing function filters.

- ftrace.fgraph.max_depth = MAX_DEPTH;
Set MAX_DEPTH to maximum depth of fgraph tracer.

Note that these properties are available on ftrace global node only, because these filters are globally applied.

---
 kernel/trace/ftrace.c     |   85 +++++++++++++++++++++++++++++----------------
 kernel/trace/trace_boot.c |   71 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 126 insertions(+), 30 deletions(-)
kernel/trace/ftrace.c +55 -30
index eca34503f178..e016ce12e60a 100644
@@ -1241,7 +1241,7 @@ static void clear_ftrace_mod_list(struct list_head *head)
 	mutex_unlock(&ftrace_lock);
 }
 
-static void free_ftrace_hash(struct ftrace_hash *hash)
+void free_ftrace_hash(struct ftrace_hash *hash)
 {
 	if (!hash || hash == EMPTY_HASH)
 		return;
@@ -4897,7 +4897,7 @@ __setup("ftrace_filter=", set_ftrace_filter);
 #ifdef CONFIG_FUNCTION_GRAPH_TRACER
 static char ftrace_graph_buf[FTRACE_FILTER_SIZE] __initdata;
 static char ftrace_graph_notrace_buf[FTRACE_FILTER_SIZE] __initdata;
-static int ftrace_graph_set_hash(struct ftrace_hash *hash, char *buffer);
+int ftrace_graph_set_hash(struct ftrace_hash *hash, char *buffer);
 
 static int __init set_graph_function(char *str)
 {
@@ -5226,6 +5226,26 @@ __ftrace_graph_open(struct inode *inode, struct file *file,
 	return ret;
 }
 
+struct ftrace_hash *ftrace_graph_copy_hash(bool enable)
+{
+	struct ftrace_hash *hash;
+
+	mutex_lock(&graph_lock);
+
+	if (enable)
+		hash = rcu_dereference_protected(ftrace_graph_hash,
+					lockdep_is_held(&graph_lock));
+	else
+		hash = rcu_dereference_protected(ftrace_graph_notrace_hash,
+					lockdep_is_held(&graph_lock));
+
+	hash = alloc_and_copy_ftrace_hash(FTRACE_HASH_DEFAULT_BITS, hash);
+
+	mutex_unlock(&graph_lock);
+
+	return hash;
+}
+
 static int
 ftrace_graph_open(struct inode *inode, struct file *file)
 {
@@ -5282,11 +5302,40 @@ ftrace_graph_notrace_open(struct inode *inode, struct file *file)
 	return ret;
 }
 
+int ftrace_graph_apply_hash(struct ftrace_hash *hash, bool enable)
+{
+	struct ftrace_hash *old_hash, *new_hash;
+
+	new_hash = __ftrace_hash_move(hash);
+	if (!new_hash)
+		return -ENOMEM;
+
+	mutex_lock(&graph_lock);
+
+	if (enable) {
+		old_hash = rcu_dereference_protected(ftrace_graph_hash,
+				lockdep_is_held(&graph_lock));
+		rcu_assign_pointer(ftrace_graph_hash, new_hash);
+	} else {
+		old_hash = rcu_dereference_protected(ftrace_graph_notrace_hash,
+				lockdep_is_held(&graph_lock));
+		rcu_assign_pointer(ftrace_graph_notrace_hash, new_hash);
+	}
+
+	mutex_unlock(&graph_lock);
+
+	/* Wait till all users are no longer using the old hash */
+	synchronize_rcu();
+
+	free_ftrace_hash(old_hash);
+
+	return 0;
+}
+
 static int
 ftrace_graph_release(struct inode *inode, struct file *file)
 {
 	struct ftrace_graph_data *fgd;
-	struct ftrace_hash *old_hash, *new_hash;
 	struct trace_parser *parser;
 	int ret = 0;
 
@@ -5311,41 +5360,17 @@ ftrace_graph_release(struct inode *inode, struct file *file)
 
 		trace_parser_put(parser);
 
-		new_hash = __ftrace_hash_move(fgd->new_hash);
-		if (!new_hash) {
-			ret = -ENOMEM;
-			goto out;
-		}
-
-		mutex_lock(&graph_lock);
-
-		if (fgd->type == GRAPH_FILTER_FUNCTION) {
-			old_hash = rcu_dereference_protected(ftrace_graph_hash,
-					lockdep_is_held(&graph_lock));
-			rcu_assign_pointer(ftrace_graph_hash, new_hash);
-		} else {
-			old_hash = rcu_dereference_protected(ftrace_graph_notrace_hash,
-					lockdep_is_held(&graph_lock));
-			rcu_assign_pointer(ftrace_graph_notrace_hash, new_hash);
-		}
-
-		mutex_unlock(&graph_lock);
-
-		/* Wait till all users are no longer using the old hash */
-		synchronize_rcu();
-
-		free_ftrace_hash(old_hash);
+		ret = ftrace_graph_apply_hash(fgd->new_hash,
+					fgd->type == GRAPH_FILTER_FUNCTION);
 	}
 
- out:
 	free_ftrace_hash(fgd->new_hash);
 	kfree(fgd);
 
 	return ret;
 }
 
-static int
-ftrace_graph_set_hash(struct ftrace_hash *hash, char *buffer)
+int ftrace_graph_set_hash(struct ftrace_hash *hash, char *buffer)
 {
 	struct ftrace_glob func_g;
 	struct dyn_ftrace *rec;
kernel/trace/trace_boot.c +71 -0

83 lines not on this page · raw message

author maintainer
#21 lore raw

RFC PATCH v3 19/19 Documentation: tracing: Add boot-time tracing document

In reply to Masami Hiramatsu

Add a documentation about boot-time tracing options for SKC file.

---
 Documentation/trace/boottime-trace.rst |  185 ++++++++++++++++++++++++++++++++
 1 file changed, 185 insertions(+)
 create mode 100644 Documentation/trace/boottime-trace.rst
Documentation/trace/boottime-trace.rst new file +185 -0
new file mode 100644
index 000000000000..fc074afd014e
@@ -0,0 +1,185 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=================
+Boot-time tracing
+=================
+
+:Author: Masami Hiramatsu <[email protected]>
+
+Overview
+========
+
+Boot-time tracing allows users to trace boot-time process including
+device initialization with full features of ftrace including per-event
+filter and actions, histograms, kprobe-events and synthetic-events,
+and trace instances.
+Since kernel cmdline is not enough to control these complex features,
+this uses supplemental kernel cmdline (SKC) to describe tracing
+feature programming.
+
+Options in Supplemental Kernel Cmdline
+======================================
+
+Here is the list of available options list for boot time tracing in
+supplemental kenrel cmdline file [1]_. All options are under "ftrace."
+prefix to isolate from other subsystems.
+
+.. [1] See Documentation/admin-guide/skc.rst for details.
+
+Ftrace Global Options
+---------------------
+
+These options are only for global ftrace node since these are globally
+applied.
+
+ftrace.tp_printk;
+   Output trace-event data on printk buffer too.
+
+ftrace.dump_on_oops [= MODE];
+   Dump ftrace on Oops. If MODE = 1 or omitted, dump trace buffer
+   on all CPUs. If MODE = 2, dump a buffer on a CPU which kicks Oops.
+
+ftrace.traceoff_on_warning;
+   Stop tracing if WARN_ON() occurs.
+
+ftrace.fgraph.filters = FILTER[, FILTER2...];
+   Add fgraph tracing function filters.
+
+ftrace.fgraph.notraces = FILTER[, FILTER2...];
+   Add fgraph non tracing function filters.
+
+ftrace.fgraph.max_depth = MAX_DEPTH;
+   Set MAX_DEPTH to maximum depth of fgraph tracer.
+
+
+Ftrace Per-instance Options
+---------------------------
+
+These options can be used for each instance including global ftrace node.
+
+ftrace.[instance.INSTANCE.]options = OPT1[, OPT2[...]];
+   Enable given ftrace options.
+
+ftrace.[instance.INSTANCE.]trace_clock = CLOCK;
+   Set given CLOCK to ftrace's trace_clock.
+
+ftrace.[instance.INSTANCE.]buffer_size = SIZE;
+   Configure ftrace buffer size to SIZE. You can use "KB" or "MB"
+   for that SIZE.
+
+ftrace.[instance.INSTANCE.]alloc_snapshot;
+   Allocate snapshot buffer.
+
+ftrace.[instance.INSTANCE.]events = EVENT[, EVENT2[...]];
+   Enable given events on boot. You can use a wild card in EVENT.
+
+ftrace.[instance.INSTANCE.]tracer = TRACER;
+   Set TRACER to current tracer on boot. (e.g. function)
+
+ftrace.[instance.INSTANCE.]ftrace.filters
+   This will take an array of tracing function filter rules
+
+ftrace.[instance.INSTANCE.]ftrace.notraces
+   This will take an array of NON-tracing function filter rules
+
+
+Ftrace Per-Event Options
+------------------------
+
+These options are setting per-event options.
+
+ftrace.[instance.INSTANCE.]event.GROUP.EVENT.enable;
+   Enables GROUP:EVENT tracing.
+
+ftrace.[instance.INSTANCE.]event.GROUP.EVENT.filter = FILTER;
+   Set FILTER rule to the GROUP:EVENT.
+
+ftrace.[instance.INSTANCE.]event.GROUP.EVENT.actions = ACTION[, ACTION2[...]];
+   Set ACTIONs to the GROUP:EVENT.
+
+ftrace.[instance.INSTANCE.]event.kprobes.EVENT.probes = PROBE[, PROBE2[...]];
+   Defines new kprobe event based on PROBEs. It is able to define
+   multiple probes on one event, but those must have same type of
+   arguments. This option is available only for the event which
+   group name is "kprobes".
+
+ftrace.[instance.INSTANCE.]event.synthetic.EVENT.fields = FIELD[, FIELD2[...]];
+   Defines new synthetic event with FIELDs. Each field should be
+   "type varname".
+
+Note that kprobe and synthetic event definitions can be written under
+instance node, but those are also visible from other instances. So please
+take care for event name conflict.
+
+Examples
+========
+
+For example, to add filter and actions for each event, define kprobe
+events, and synthetic events with histogram, write SKC like below.
+
+::
+
+  ftrace.event {
+        task.task_newtask {
+                filter = "pid < 128";
+                enable;
+        }
+        kprobes.vfs_read {
+                probes = "vfs_read $arg1 $arg2";
+                filter = "common_pid < 200";
+                enable;
+        }
+        synthetic.initcall_latency {
+                fields = "unsigned long func", "u64 lat";
+                actions = "hist:keys=func.sym,lat:vals=lat:sort=lat";
+        }
+        initcall.initcall_start {
+                actions = "hist:keys=func:ts0=common_timestamp.usecs";
+        }
+        initcall.initcall_finish {
+                actions = "hist:keys=func:lat=common_timestamp.usecs-$ts0:onmatch(initcall.initcall_start).initcall_latency(func,$lat)";
+        }
+  }
+
+Also, boottime tracing supports "instance" node, which allows us to run
+several tracers for different purpose at once. For example, one tracer
+is for tracing functions in module alpha, and others tracing module beta,
+you can write as below.
+
+::
+
+  ftrace.instance {
+        foo {
+                tracer = "function";
+                ftrace-filters = "*:mod:alpha";
+        }
+        bar {
+                tracer = "function";
+                ftrace-filters = "*:mod:beta";
+        }
+  }
+
+The instance node also accepts event nodes so that each instance
+can customize its event tracing.
+
+This boot-time trace also supports ftrace kernel parameters.
+For example, following kernel parameters
+
+::
+
+ trace_options=sym-addr trace_event=initcall:* tp_printk trace_buf_size=1M ftrace=function ftrace_filter="vfs*"
+
+This can be written in SKC like below.
+
+::
+
+  ftrace {
+        options = sym-addr;
+        events = "initcall:*";
+        tp-printk;
+        buffer-size = 1MB;
+        ftrace-filters = "vfs*";
+  }
+
+However, since the initialization timing is different, if you need
+to trace very early boot, please use normal kernel parameters.