Skip to contentSkip to conversation

tracing: Addition of tracing instances via kernel command line

PATCH New Steven Rostedt 1 participant 5 messages Feb 7, 2023

Subsystems (2)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

v2 cover tracing: Addition of tracing instances via kernel command line

Instance creation can happen with the bootconfig but there's currently no way to create and enable instances via the normal command line.

This series fixes that.

trace_instance=foo,sched_switch,sched_waking ftrace_boot_snapshot=foo

The above creates a "foo" instance at early start up and enables the sched_switch and sched_waking events. Then at the end of boot up a snapshot is taken so that user space does not overwrite the boot up data.

Changes since v1: https://lore.kernel.org/linux-trace-kernel/[email protected]/
- Fixed document grammar (Randy Dunlap)
- Removed extra workqueue.h include (Ross Zwisler)
- Use sizeof(boot_instance_info) than using COMMAND_LINE_SIZE)
- Remove unneeded @ip in kernel-doc (Ross Zwisler)
- Make stub function for do_allocate_snapshot() for !CONFIG_TRACER_MAX_TRACE (kernel test robot)
- Compare strstr() to NULL and not zero (kernel test robot)
- Use sizeof(boot_snapshot_info) instead of COMMAND_LINE_SIZE (Ross Zwisler)

Steven Rostedt (Google) (4):
tracing: Add creation of instances at boot command line
tracing: Add enabling of events to boot instances
tracing: Add trace_array_puts() to write into instance
tracing: Allow boot instances to have snapshot buffers

----

 Documentation/admin-guide/bootconfig.rst        |   6 +-
 Documentation/admin-guide/kernel-parameters.txt |  29 +++++
 include/linux/bootconfig.h                      |   2 +-
 include/linux/trace.h                           |  12 ++
 init/Kconfig                                    |  13 --
 init/main.c                                     |   2 +-
 kernel/trace/trace.c                            | 160 +++++++++++++++++++++---
 kernel/trace/trace.h                            |   2 +
 kernel/trace/trace_events.c                     |   9 +-
 9 files changed, 193 insertions(+), 42 deletions(-)
author maintainer
#1 lore raw

PATCH v2 1/4 tracing: Add creation of instances at boot command line

From: "Steven Rostedt (Google)" <[email protected]>

Add kernel command line to add tracing instances. This only creates instances at boot but still does not enable any events to them. Later changes will extend this command line to add enabling of events, filters, and triggers. As well as possibly redirecting trace_printk()!

---
Changes since v1: https://lkml.kernel.org/r/[email protected]
- Fixed document grammar (Randy Dunlap)
- Removed extra workqueue.h include (Ross Zwisler)
- Use sizeof(boot_instance_info) than using COMMAND_LINE_SIZE)

 .../admin-guide/kernel-parameters.txt         |  6 +++
 kernel/trace/trace.c                          | 50 +++++++++++++++++++
 2 files changed, 56 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +6 -0
index 6cfa6e3996cf..9545da5ed849 100644
@@ -6272,6 +6272,12 @@
 			comma-separated list of trace events to enable. See
 			also Documentation/trace/events.rst
 
+	trace_instance=[instance-info]
+			[FTRACE] Create a ring buffer instance early in boot up.
+			This will be listed in:
+
+				/sys/kernel/tracing/instances
+
 	trace_options=[option-list]
 			[FTRACE] Enable or disable tracer options at boot.
 			The option-list is a comma delimited list of options
kernel/trace/trace.c +50 -0

80 lines not on this page · raw message

Signature · 1 line
2.39.1
author maintainer
#2 lore raw

PATCH v2 2/4 tracing: Add enabling of events to boot instances

In reply to Steven Rostedt

From: "Steven Rostedt (Google)" <[email protected]>

Add the format of:

trace_instance=foo,sched:sched_switch,irq_handler_entry,initcall

That will create the "foo" instance and enable the sched_switch event (here were the "sched" system is explicitly specified), the irq_handler_entry event, and all events under the system initcall.

---
 Documentation/admin-guide/kernel-parameters.txt | 14 ++++++++++++++
 kernel/trace/trace.c                            |  4 ++++
 kernel/trace/trace.h                            |  2 ++
 kernel/trace/trace_events.c                     |  9 ++++-----
 4 files changed, 24 insertions(+), 5 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +14 -0
index 9545da5ed849..40304dc3d319 100644
@@ -6278,6 +6278,20 @@
 
 				/sys/kernel/tracing/instances
 
+			Events can be enabled at the time the instance is created
+			via:
+
+				trace_instance=<name>,<system1>:<event1>,<system2>:<event2>
+
+			Note, the "<system*>:" portion is optional if the event is
+			unique.
+
+				trace_instance=foo,sched:sched_switch,irq_handler_entry,initcall
+
+			will enable the "sched_switch" event (note, the "sched:" is optional, and
+			the same thing would happen if it was left off). The irq_handler_entry
+			event, and all events under the "initcall" system.
+
 	trace_options=[option-list]
 			[FTRACE] Enable or disable tracer options at boot.
 			The option-list is a comma delimited list of options
kernel/trace/trace.c +4 -0
index 863716fcca49..5b16af9e7d30 100644
@@ -10188,6 +10188,10 @@ __init static void enable_instances(void)
 		}
 		/* Allow user space to delete it */
 		trace_array_put(tr);
+
+		while ((tok = strsep(&curr_str, ","))) {
+			early_enable_events(tr, tok, true);
+		}
 	}
 }
 
kernel/trace/trace.h +2 -0
index d16929dd0f08..f10bf804dd2b 100644
@@ -1334,6 +1334,8 @@ DECLARE_PER_CPU(int, trace_buffered_event_cnt);
 void trace_buffered_event_disable(void);
 void trace_buffered_event_enable(void);
 
+void early_enable_events(struct trace_array *tr, char *buf, bool disable_first);
+
 static inline void
 __trace_event_discard_commit(struct trace_buffer *buffer,
 			     struct ring_buffer_event *event)
kernel/trace/trace_events.c +4 -5
index b07931ad97de..cf3fd74fa675 100644
@@ -3750,10 +3750,9 @@ static __init int event_trace_memsetup(void)
 	return 0;
 }
 
-static __init void
-early_enable_events(struct trace_array *tr, bool disable_first)
+__init void
+early_enable_events(struct trace_array *tr, char *buf, bool disable_first)
 {
-	char *buf = bootup_event_buf;
 	char *token;
 	int ret;
 
@@ -3806,7 +3805,7 @@ static __init int event_trace_enable(void)
 	 */
 	__trace_early_add_events(tr);
 
-	early_enable_events(tr, false);
+	early_enable_events(tr, bootup_event_buf, false);
 
 	trace_printk_start_comm();
 
@@ -3834,7 +3833,7 @@ static __init int event_trace_enable_again(void)
 	if (!tr)
 		return -ENODEV;
 
-	early_enable_events(tr, true);
+	early_enable_events(tr, bootup_event_buf, true);
 
 	return 0;
 }
Signature · 1 line
2.39.1
author maintainer
#3 lore raw

PATCH v2 3/4 tracing: Add trace_array_puts() to write into instance

In reply to Steven Rostedt

From: "Steven Rostedt (Google)" <[email protected]>

Add a generic trace_array_puts() that can be used to "trace_puts()" into an allocated trace_array instance. This is just another variant of trace_array_printk().

---
Changes since v1: https://lkml.kernel.org/r/[email protected]
- Remove unneeded @ip in kernel-doc (Ross Zwisler)

 include/linux/trace.h | 12 ++++++++++++
 kernel/trace/trace.c  | 27 +++++++++++++++++----------
 2 files changed, 29 insertions(+), 10 deletions(-)
include/linux/trace.h +12 -0
index 80ffda871749..2a70a447184c 100644
@@ -33,6 +33,18 @@ struct trace_array;
 int register_ftrace_export(struct trace_export *export);
 int unregister_ftrace_export(struct trace_export *export);
 
+/**
+ * trace_array_puts - write a constant string into the trace buffer.
+ * @tr:    The trace array to write to
+ * @str:   The constant string to write
+ */
+#define trace_array_puts(tr, str)					\
+	({								\
+		str ? __trace_array_puts(tr, _THIS_IP_, str, strlen(str)) : -1;	\
+	})
+int __trace_array_puts(struct trace_array *tr, unsigned long ip,
+		       const char *str, int size);
+
 void trace_printk_init_buffers(void);
 __printf(3, 4)
 int trace_array_printk(struct trace_array *tr, unsigned long ip,
kernel/trace/trace.c +17 -10
index 5b16af9e7d30..9918bf7441ed 100644
@@ -1023,13 +1023,8 @@ __buffer_unlock_commit(struct trace_buffer *buffer, struct ring_buffer_event *ev
 		ring_buffer_unlock_commit(buffer);
 }
 
-/**
- * __trace_puts - write a constant string into the trace buffer.
- * @ip:	   The address of the caller
- * @str:   The constant string to write
- * @size:  The size of the string.
- */
-int __trace_puts(unsigned long ip, const char *str, int size)
+int __trace_array_puts(struct trace_array *tr, unsigned long ip,
+		       const char *str, int size)
 {
 	struct ring_buffer_event *event;
 	struct trace_buffer *buffer;
@@ -1037,7 +1032,7 @@ int __trace_puts(unsigned long ip, const char *str, int size)
 	unsigned int trace_ctx;
 	int alloc;
 
-	if (!(global_trace.trace_flags & TRACE_ITER_PRINTK))
+	if (!(tr->trace_flags & TRACE_ITER_PRINTK))
 		return 0;
 
 	if (unlikely(tracing_selftest_running || tracing_disabled))
@@ -1046,7 +1041,7 @@ int __trace_puts(unsigned long ip, const char *str, int size)
 	alloc = sizeof(*entry) + size + 2; /* possible \n added */
 
 	trace_ctx = tracing_gen_ctx();
-	buffer = global_trace.array_buffer.buffer;
+	buffer = tr->array_buffer.buffer;
 	ring_buffer_nest_start(buffer);
 	event = __trace_buffer_lock_reserve(buffer, TRACE_PRINT, alloc,
 					    trace_ctx);
@@ -1068,11 +1063,23 @@ int __trace_puts(unsigned long ip, const char *str, int size)
 		entry->buf[size] = '\0';
 
 	__buffer_unlock_commit(buffer, event);
-	ftrace_trace_stack(&global_trace, buffer, trace_ctx, 4, NULL);
+	ftrace_trace_stack(tr, buffer, trace_ctx, 4, NULL);
  out:
 	ring_buffer_nest_end(buffer);
 	return size;
 }
+EXPORT_SYMBOL_GPL(__trace_array_puts);
+
+/**
+ * __trace_puts - write a constant string into the trace buffer.
+ * @ip:	   The address of the caller
+ * @str:   The constant string to write
+ * @size:  The size of the string.
+ */
+int __trace_puts(unsigned long ip, const char *str, int size)
+{
+	return __trace_array_puts(&global_trace, ip, str, size);
+}
 EXPORT_SYMBOL_GPL(__trace_puts);
 
 /**
Signature · 1 line
2.39.1
author maintainer
#4 lore raw

PATCH v2 4/4 tracing: Allow boot instances to have snapshot buffers

In reply to Steven Rostedt

From: "Steven Rostedt (Google)" <[email protected]>

Add to ftrace_boot_snapshot, "=<instance>" name, where the instance will get a snapshot buffer, and will take a snapshot at the end of boot (which will save the boot traces).

---
Changes since v1: https://lkml.kernel.org/r/[email protected]
- Make stub function for do_allocate_snapshot() for !CONFIG_TRACER_MAX_TRACE (kernel test robot)
- Compare strstr() to NULL and not zero (kernel test robot)
- Use sizeof(boot_snapshot_info) instead of COMMAND_LINE_SIZE (Ross Zwisler)

 .../admin-guide/kernel-parameters.txt         |  9 +++
 kernel/trace/trace.c                          | 79 +++++++++++++++++--
 2 files changed, 81 insertions(+), 7 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +9 -0
index 40304dc3d319..7fa1ee4ba910 100644
@@ -1532,6 +1532,15 @@
 			boot up that is likely to be overridden by user space
 			start up functionality.
 
+			Optionally, the snapshot can also be defined for a tracing
+			instance that was created by the trace_instance= command
+			line parameter.
+
+			trace_instance=foo,sched_switch ftrace_boot_snapshot=foo
+
+			The above will cause the "foo" tracing instance to trigger
+			a snapshot at the end of boot up.
+
 	ftrace_dump_on_oops[=orig_cpu]
 			[FTRACE] will dump the trace buffers on oops.
 			If no parameter is passed, ftrace will dump
kernel/trace/trace.c +72 -7

119 lines not on this page · raw message

Signature · 1 line
2.39.1