Skip to contentSkip to conversation

tracing: Rebase for update of v9 of RTLA tool

PATCH New Steven Rostedt 1 participant 15 messages Jan 12, 2022

Subsystems (2)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

cover tracing: Rebase for update of v9 of RTLA tool

Since I pulled in version 8 of the rtla tool instead of the most recent v9 (which was the one I last reviewed, but somehow pulled in the older version), I'm rebasing to v9. This is not an issue as none of the patches actually touch kernel code. Only tools and Documentation.

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git
for-next

Head SHA1: fa7e5a516e8ca0740ad1563b33d24b430dc31f77

Daniel Bristot de Oliveira (14):
rtla: Real-Time Linux Analysis tool
rtla: Helper functions for rtla
rtla: Add osnoise tool
rtla/osnoise: Add osnoise top mode rtla/osnoise: Add the hist mode
rtla: Add timerlat tool and timelart top mode rtla/timerlat: Add timerlat hist mode
rtla: Add Documentation
rtla: Add rtla osnoise man page
rtla: Add rtla osnoise top documentation
rtla: Add rtla osnoise hist documentation
rtla: Add rtla timerlat documentation
rtla: Add rtla timerlat top documentation
rtla: Add rtla timerlat hist documentation

----

 Documentation/tools/rtla/Makefile                  |  41 +
 Documentation/tools/rtla/common_appendix.rst       |  12 +
 Documentation/tools/rtla/common_hist_options.rst   |  23 +
 Documentation/tools/rtla/common_options.rst        |  28 +
 .../tools/rtla/common_osnoise_description.rst      |   8 +
 .../tools/rtla/common_osnoise_options.rst          |  17 +
 .../tools/rtla/common_timerlat_description.rst     |  10 +
 .../tools/rtla/common_timerlat_options.rst         |  16 +
 Documentation/tools/rtla/common_top_options.rst    |   3 +
 Documentation/tools/rtla/rtla-osnoise-hist.rst     |  66 ++
 Documentation/tools/rtla/rtla-osnoise-top.rst      |  61 ++
 Documentation/tools/rtla/rtla-osnoise.rst          |  59 ++
 Documentation/tools/rtla/rtla-timerlat-hist.rst    | 106 +++
 Documentation/tools/rtla/rtla-timerlat-top.rst     | 145 ++++
 Documentation/tools/rtla/rtla-timerlat.rst         |  57 ++
 Documentation/tools/rtla/rtla.rst                  |  48 ++
 tools/tracing/rtla/Makefile                        | 102 +++
 tools/tracing/rtla/README.txt                      |  36 +
 tools/tracing/rtla/src/osnoise.c                   | 875 +++++++++++++++++++++
 tools/tracing/rtla/src/osnoise.h                   |  91 +++
 tools/tracing/rtla/src/osnoise_hist.c              | 801 +++++++++++++++++++
 tools/tracing/rtla/src/osnoise_top.c               | 579 ++++++++++++++
 tools/tracing/rtla/src/rtla.c                      |  87 ++
 tools/tracing/rtla/src/timerlat.c                  |  72 ++
 tools/tracing/rtla/src/timerlat.h                  |   4 +
 tools/tracing/rtla/src/timerlat_hist.c             | 822 +++++++++++++++++++
 tools/tracing/rtla/src/timerlat_top.c              | 618 +++++++++++++++
 tools/tracing/rtla/src/trace.c                     | 192 +++++
 tools/tracing/rtla/src/trace.h                     |  27 +
 tools/tracing/rtla/src/utils.c                     | 433 ++++++++++
 tools/tracing/rtla/src/utils.h                     |  56 ++
 31 files changed, 5495 insertions(+)
 create mode 100644 Documentation/tools/rtla/Makefile
 create mode 100644 Documentation/tools/rtla/common_appendix.rst
 create mode 100644 Documentation/tools/rtla/common_hist_options.rst
 create mode 100644 Documentation/tools/rtla/common_options.rst
 create mode 100644 Documentation/tools/rtla/common_osnoise_description.rst
 create mode 100644 Documentation/tools/rtla/common_osnoise_options.rst
 create mode 100644 Documentation/tools/rtla/common_timerlat_description.rst
 create mode 100644 Documentation/tools/rtla/common_timerlat_options.rst
 create mode 100644 Documentation/tools/rtla/common_top_options.rst
 create mode 100644 Documentation/tools/rtla/rtla-osnoise-hist.rst
 create mode 100644 Documentation/tools/rtla/rtla-osnoise-top.rst
 create mode 100644 Documentation/tools/rtla/rtla-osnoise.rst
 create mode 100644 Documentation/tools/rtla/rtla-timerlat-hist.rst
 create mode 100644 Documentation/tools/rtla/rtla-timerlat-top.rst
 create mode 100644 Documentation/tools/rtla/rtla-timerlat.rst
 create mode 100644 Documentation/tools/rtla/rtla.rst
 create mode 100644 tools/tracing/rtla/Makefile
 create mode 100644 tools/tracing/rtla/README.txt
 create mode 100644 tools/tracing/rtla/src/osnoise.c
 create mode 100644 tools/tracing/rtla/src/osnoise.h
 create mode 100644 tools/tracing/rtla/src/osnoise_hist.c
 create mode 100644 tools/tracing/rtla/src/osnoise_top.c
 create mode 100644 tools/tracing/rtla/src/rtla.c
 create mode 100644 tools/tracing/rtla/src/timerlat.c
 create mode 100644 tools/tracing/rtla/src/timerlat.h
 create mode 100644 tools/tracing/rtla/src/timerlat_hist.c
 create mode 100644 tools/tracing/rtla/src/timerlat_top.c
 create mode 100644 tools/tracing/rtla/src/trace.c
 create mode 100644 tools/tracing/rtla/src/trace.h
 create mode 100644 tools/tracing/rtla/src/utils.c
 create mode 100644 tools/tracing/rtla/src/utils.h
author maintainer
#1 lore raw

PATCH 1/14 rtla: Real-Time Linux Analysis tool

From: Daniel Bristot de Oliveira <[email protected]>

The rtla is a meta-tool that includes a set of commands that aims to analyze the real-time properties of Linux. But instead of testing Linux as a black box, rtla leverages kernel tracing capabilities to provide precise information about the properties and root causes of unexpected results.

rtla --help works and provide information about the available options.

This is just the "main" and the Makefile, no function yet.

Cc Tao Zhou <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 tools/tracing/rtla/Makefile   | 76 +++++++++++++++++++++++++++++++++++
 tools/tracing/rtla/README.txt | 36 +++++++++++++++++
 tools/tracing/rtla/src/rtla.c | 72 +++++++++++++++++++++++++++++++++
 3 files changed, 184 insertions(+)
 create mode 100644 tools/tracing/rtla/Makefile
 create mode 100644 tools/tracing/rtla/README.txt
 create mode 100644 tools/tracing/rtla/src/rtla.c
tools/tracing/rtla/Makefile new file +76 -0
new file mode 100644
index 000000000000..56b1bdd60c96
@@ -0,0 +1,76 @@
+NAME	:=	rtla
+VERSION	:=	0.5
+
+# From libtracefs:
+# Makefiles suck: This macro sets a default value of $(2) for the
+# variable named by $(1), unless the variable has been set by
+# environment or command line. This is necessary for CC and AR
+# because make sets default values, so the simpler ?= approach
+# won't work as expected.
+define allow-override
+  $(if $(or $(findstring environment,$(origin $(1))),\
+            $(findstring command line,$(origin $(1)))),,\
+    $(eval $(1) = $(2)))
+endef
+
+# Allow setting CC and AR, or setting CROSS_COMPILE as a prefix.
+$(call allow-override,CC,$(CROSS_COMPILE)gcc)
+$(call allow-override,AR,$(CROSS_COMPILE)ar)
+$(call allow-override,STRIP,$(CROSS_COMPILE)strip)
+$(call allow-override,PKG_CONFIG,pkg-config)
+$(call allow-override,LD_SO_CONF_PATH,/etc/ld.so.conf.d/)
+$(call allow-override,LDCONFIG,ldconfig)
+
+INSTALL	=	install
+FOPTS	:=	-flto=auto -ffat-lto-objects -fexceptions -fstack-protector-strong \
+		-fasynchronous-unwind-tables -fstack-clash-protection
+WOPTS	:= 	-Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -Wno-maybe-uninitialized
+
+TRACEFS_HEADERS	:= $$($(PKG_CONFIG) --cflags libtracefs)
+
+CFLAGS	:=	-O -g -DVERSION=\"$(VERSION)\" $(FOPTS) $(MOPTS) $(WOPTS) $(TRACEFS_HEADERS)
+LDFLAGS	:=	-ggdb
+LIBS	:=	$$($(PKG_CONFIG) --libs libtracefs) -lprocps
+
+SRC	:=	$(wildcard src/*.c)
+HDR	:=	$(wildcard src/*.h)
+OBJ	:=	$(SRC:.c=.o)
+DIRS	:=	src
+FILES	:=	Makefile README.txt
+CEXT	:=	bz2
+TARBALL	:=	$(NAME)-$(VERSION).tar.$(CEXT)
+TAROPTS	:=	-cvjf $(TARBALL)
+BINDIR	:=	/usr/bin
+DATADIR	:=	/usr/share
+DOCDIR	:=	$(DATADIR)/doc
+MANDIR	:=	$(DATADIR)/man
+LICDIR	:=	$(DATADIR)/licenses
+
+.PHONY:	all
+all:	rtla
+
+rtla: $(OBJ)
+	$(CC) -o rtla $(LDFLAGS) $(OBJ) $(LIBS)
+
+static: $(OBJ)
+	$(CC) -o rtla-static $(LDFLAGS) --static $(OBJ) $(LIBS) -lpthread -ldl
+
+.PHONY: install
+install:
+	$(INSTALL) -d -m 755 $(DESTDIR)$(BINDIR)
+	$(INSTALL) rtla -m 755 $(DESTDIR)$(BINDIR)
+	$(STRIP) $(DESTDIR)$(BINDIR)/rtla
+
+.PHONY: clean tarball
+clean:
+	@test ! -f rtla || rm rtla
+	@test ! -f rtla-static || rm rtla-static
+	@test ! -f src/rtla.o || rm src/rtla.o
+	@test ! -f $(TARBALL) || rm -f $(TARBALL)
+	@rm -rf *~ $(OBJ) *.tar.$(CEXT)
+
+tarball:  clean
+	rm -rf $(NAME)-$(VERSION) && mkdir $(NAME)-$(VERSION)
+	cp -r $(DIRS) $(FILES) $(NAME)-$(VERSION)
+	tar $(TAROPTS) --exclude='*~' $(NAME)-$(VERSION)
+	rm -rf $(NAME)-$(VERSION)
tools/tracing/rtla/README.txt new file +36 -0
new file mode 100644
index 000000000000..6c88446f7e74
@@ -0,0 +1,36 @@
+RTLA: Real-Time Linux Analysis tools
+
+The rtla is a meta-tool that includes a set of commands that
+aims to analyze the real-time properties of Linux. But, instead of
+testing Linux as a black box, rtla leverages kernel tracing
+capabilities to provide precise information about the properties
+and root causes of unexpected results.
+
+Installing RTLA
+
+RTLA depends on some libraries and tools. More precisely, it depends on the
+following libraries:
+
+ - libtracefs
+ - libtraceevent
+ - procps
+
+It also depends on python3-docutils to compile man pages.
+
+For development, we suggest the following steps for compiling rtla:
+
+  $ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.git
+  $ cd libtraceevent/
+  $ make
+  $ sudo make install
+  $ cd ..
+  $ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtracefs.git
+  $ cd libtracefs/
+  $ make
+  $ sudo make install
+  $ cd ..
+  $ cd $rtla_src
+  $ make
+  $ sudo make install
+
+For further information, please refer to the rtla man page.
tools/tracing/rtla/src/rtla.c new file +72 -0

72 lines not on this page · raw message

Signature · 1 line
2.33.0
author maintainer
#2 lore raw

PATCH 2/14 rtla: Helper functions for rtla

In reply to Steven Rostedt

From: Daniel Bristot de Oliveira <[email protected]>

This is a set of utils and tracer helper functions. They are used by rtla mostly to parse config, display data and some trace operations that are not part of libtracefs (because they are only useful it for this case).

Cc Tao Zhou <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 tools/tracing/rtla/src/trace.c | 192 +++++++++++++++
 tools/tracing/rtla/src/trace.h |  27 ++
 tools/tracing/rtla/src/utils.c | 433 +++++++++++++++++++++++++++++++++
 tools/tracing/rtla/src/utils.h |  56 +++++
 4 files changed, 708 insertions(+)
 create mode 100644 tools/tracing/rtla/src/trace.c
 create mode 100644 tools/tracing/rtla/src/trace.h
 create mode 100644 tools/tracing/rtla/src/utils.c
 create mode 100644 tools/tracing/rtla/src/utils.h
tools/tracing/rtla/src/trace.c new file +192 -0
new file mode 100644
index 000000000000..107a0c6387f7
@@ -0,0 +1,192 @@
+// SPDX-License-Identifier: GPL-2.0
+#define _GNU_SOURCE
+#include <sys/sendfile.h>
+#include <tracefs.h>
+#include <signal.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <errno.h>
+
+#include "trace.h"
+#include "utils.h"
+
+/*
+ * enable_tracer_by_name - enable a tracer on the given instance
+ */
+int enable_tracer_by_name(struct tracefs_instance *inst, const char *tracer_name)
+{
+	enum tracefs_tracers tracer;
+	int retval;
+
+	tracer = TRACEFS_TRACER_CUSTOM;
+
+	debug_msg("enabling %s tracer\n", tracer_name);
+
+	retval = tracefs_tracer_set(inst, tracer, tracer_name);
+	if (retval < 0) {
+		if (errno == ENODEV)
+			err_msg("tracer %s not found!\n", tracer_name);
+
+		err_msg("failed to enable the tracer %s\n", tracer_name);
+		return -1;
+	}
+
+	return 0;
+}
+
+/*
+ * disable_tracer - set nop tracer to the insta
+ */
+void disable_tracer(struct tracefs_instance *inst)
+{
+	enum tracefs_tracers t = TRACEFS_TRACER_NOP;
+	int retval;
+
+	retval = tracefs_tracer_set(inst, t);
+	if (retval < 0)
+		err_msg("oops, error disabling tracer\n");
+}
+
+/*
+ * create_instance - create a trace instance with *instance_name
+ */
+struct tracefs_instance *create_instance(char *instance_name)
+{
+	return tracefs_instance_create(instance_name);
+}
+
+/*
+ * destroy_instance - remove a trace instance and free the data
+ */
+void destroy_instance(struct tracefs_instance *inst)
+{
+	tracefs_instance_destroy(inst);
+	tracefs_instance_free(inst);
+}
+
+/*
+ * save_trace_to_file - save the trace output of the instance to the file
+ */
+int save_trace_to_file(struct tracefs_instance *inst, const char *filename)
+{
+	const char *file = "trace";
+	mode_t mode = 0644;
+	char buffer[4096];
+	int out_fd, in_fd;
+	int retval = -1;
+
+	in_fd = tracefs_instance_file_open(inst, file, O_RDONLY);
+	if (in_fd < 0) {
+		err_msg("Failed to open trace file\n");
+		return -1;
+	}
+
+	out_fd = creat(filename, mode);
+	if (out_fd < 0) {
+		err_msg("Failed to create output file %s\n", filename);
+		goto out_close_in;
+	}
+
+	do {
+		retval = read(in_fd, buffer, sizeof(buffer));
+		if (retval <= 0)
+			goto out_close;
+
+		retval = write(out_fd, buffer, retval);
+		if (retval < 0)
+			goto out_close;
+	} while (retval > 0);
+
+	retval = 0;
+out_close:
+	close(out_fd);
+out_close_in:
+	close(in_fd);
+	return retval;
+}
+
+/*
+ * collect_registered_events - call the existing callback function for the event
+ *
+ * If an event has a registered callback function, call it.
+ * Otherwise, ignore the event.
+ */
+int
+collect_registered_events(struct tep_event *event, struct tep_record *record,
+			  int cpu, void *context)
+{
+	struct trace_instance *trace = context;
+	struct trace_seq *s = trace->seq;
+
+	if (!event->handler)
+		return 0;
+
+	event->handler(s, record, event, context);
+
+	return 0;
+}
+
+/*
+ * trace_instance_destroy - destroy and free a rtla trace instance
+ */
+void trace_instance_destroy(struct trace_instance *trace)
+{
+	if (trace->inst) {
+		disable_tracer(trace->inst);
+		destroy_instance(trace->inst);
+	}
+
+	if (trace->seq)
+		free(trace->seq);
+
+	if (trace->tep)
+		tep_free(trace->tep);
+}
+
+/*
+ * trace_instance_init - create an rtla trace instance
+ *
+ * It is more than the tracefs instance, as it contains other
+ * things required for the tracing, such as the local events and
+ * a seq file.
+ *
+ * Note that the trace instance is returned disabled. This allows
+ * the tool to apply some other configs, like setting priority
+ * to the kernel threads, before starting generating trace entries.
+ */
+int trace_instance_init(struct trace_instance *trace, char *tool_name)
+{
+	trace->seq = calloc(1, sizeof(*trace->seq));
+	if (!trace->seq)
+		goto out_err;
+
+	trace_seq_init(trace->seq);
+
+	trace->inst = create_instance(tool_name);
+	if (!trace->inst)
+		goto out_err;
+
+	trace->tep = tracefs_local_events(NULL);
+	if (!trace->tep)
+		goto out_err;
+
+	/*
+	 * Let the main enable the record after setting some other
+	 * things such as the priority of the tracer's threads.
+	 */
+	tracefs_trace_off(trace->inst);
+
+	return 0;
+
+out_err:
+	trace_instance_destroy(trace);
+	return 1;
+}
+
+/*
+ * trace_instance_start - start tracing a given rtla instance
+ */
+int trace_instance_start(struct trace_instance *trace)
+{
+	return tracefs_trace_on(trace->inst);
+}
tools/tracing/rtla/src/trace.h new file +27 -0
new file mode 100644
index 000000000000..0ea1df0ad9a7
@@ -0,0 +1,27 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <tracefs.h>
+#include <stddef.h>
+
+struct trace_instance {
+	struct tracefs_instance		*inst;
+	struct tep_handle		*tep;
+	struct trace_seq		*seq;
+};
+
+int trace_instance_init(struct trace_instance *trace, char *tool_name);
+int trace_instance_start(struct trace_instance *trace);
+void trace_instance_destroy(struct trace_instance *trace);
+
+struct trace_seq *get_trace_seq(void);
+int enable_tracer_by_name(struct tracefs_instance *inst, const char *tracer_name);
+void disable_tracer(struct tracefs_instance *inst);
+
+int enable_osnoise(struct trace_instance *trace);
+int enable_timerlat(struct trace_instance *trace);
+
+struct tracefs_instance *create_instance(char *instance_name);
+void destroy_instance(struct tracefs_instance *inst);
+
+int save_trace_to_file(struct tracefs_instance *inst, const char *filename);
+int collect_registered_events(struct tep_event *tep, struct tep_record *record,
+			      int cpu, void *context);
tools/tracing/rtla/src/utils.c new file +433 -0

433 lines not on this page · raw message

tools/tracing/rtla/src/utils.h new file +56 -0

56 lines not on this page · raw message

Signature · 1 line
2.33.0
author maintainer
#3 lore raw

PATCH 3/14 rtla: Add osnoise tool

In reply to Steven Rostedt

From: Daniel Bristot de Oliveira <[email protected]>

The osnoise tool is the interface for the osnoise tracer. The osnoise tool will have multiple "modes" with different outputs. At this point, no mode is included.

The osnoise.c includes the osnoise_context abstraction. It serves to read-save-change-restore the default values from tracing/osnoise/ directory. When the context is deleted, the default values are restored.

It also includes some other helper functions for managing osnoise tracer sessions.

With these bits and pieces in place, we can start adding some functionality to rtla.

Cc Tao Zhou <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 tools/tracing/rtla/Makefile      |   2 +
 tools/tracing/rtla/src/osnoise.c | 855 +++++++++++++++++++++++++++++++
 tools/tracing/rtla/src/osnoise.h |  89 ++++
 tools/tracing/rtla/src/rtla.c    |  10 +
 4 files changed, 956 insertions(+)
 create mode 100644 tools/tracing/rtla/src/osnoise.c
 create mode 100644 tools/tracing/rtla/src/osnoise.h
tools/tracing/rtla/Makefile +2 -0
index 56b1bdd60c96..d40b442fcd30 100644
@@ -60,6 +60,8 @@ install:
 	$(INSTALL) -d -m 755 $(DESTDIR)$(BINDIR)
 	$(INSTALL) rtla -m 755 $(DESTDIR)$(BINDIR)
 	$(STRIP) $(DESTDIR)$(BINDIR)/rtla
+	@test ! -f $(DESTDIR)$(BINDIR)/osnoise || rm $(DESTDIR)$(BINDIR)/osnoise
+	ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/osnoise
 
 .PHONY: clean tarball
 clean:
tools/tracing/rtla/src/osnoise.c new file +855 -0

855 lines not on this page · raw message

tools/tracing/rtla/src/osnoise.h new file +89 -0

89 lines not on this page · raw message

tools/tracing/rtla/src/rtla.c +10 -0
index 5ae2664ed47d..669b9750b3b3 100644
@@ -8,6 +8,8 @@
 #include <string.h>
 #include <stdio.h>
 
+#include "osnoise.h"
+
 /*
  * rtla_usage - print rtla usage
  */
@@ -22,6 +24,7 @@ static void rtla_usage(void)
 		"  usage: rtla COMMAND ...",
 		"",
 		"  commands:",
+		"     osnoise  - gives information about the operating system noise (osnoise)",
 		"",
 		NULL,
 	};
@@ -39,7 +42,14 @@ static void rtla_usage(void)
  */
 int run_command(int argc, char **argv, int start_position)
 {
+	if (strcmp(argv[start_position], "osnoise") == 0) {
+		osnoise_main(argc-start_position, &argv[start_position]);
+		goto ran;
+	}
+
 	return 0;
+ran:
+	return 1;
 }
 
 int main(int argc, char *argv[])
Signature · 1 line
2.33.0
author maintainer
#4 lore raw

PATCH 4/14 rtla/osnoise: Add osnoise top mode

In reply to Steven Rostedt

From: Daniel Bristot de Oliveira <[email protected]>

The rtla osnoise tool is an interface for the osnoise tracer. The osnoise tracer dispatches a kernel thread per-cpu. These threads read the time in a loop while with preemption, softirqs and IRQs enabled, thus allowing all the sources of osnoise during its execution. The osnoise threads take note of the entry and exit point of any source of interferences, increasing a per-cpu interference counter. The osnoise tracer also saves an interference counter for each source of interference.

The rtla osnoise top mode displays information about the periodic summary from the osnoise tracer.

One example of rtla osnoise top output is:

[root@alien ~]# rtla osnoise top -c 0-3 -d 1m -q -r 900000 -P F:1
Operating System Noise
duration: 0 00:01:00 | time is in us
CPU Period Runtime Noise % CPU Aval Max Noise Max Single HW NMI IRQ Softirq Thread

  0 #58         52200000         1031    99.99802          91           60           0            0        52285            0          101
  1 #59         53100000            5    99.99999           5            5           0            9        53122            0           18
  2 #59         53100000            7    99.99998           7            7           0            8        53115            0           18
  3 #59         53100000         8274    99.98441         277           23           0            9        53778            0          660

"rtla osnoise top --help" works and provide information about the available options.

Cc Tao Zhou <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 tools/tracing/rtla/src/osnoise.c     |  16 +
 tools/tracing/rtla/src/osnoise.h     |   1 +
 tools/tracing/rtla/src/osnoise_top.c | 579 +++++++++++++++++++++++++++
 3 files changed, 596 insertions(+)
 create mode 100644 tools/tracing/rtla/src/osnoise_top.c
tools/tracing/rtla/src/osnoise.c +16 -0
index e5254044e878..fbf4f9bfda02 100644
@@ -844,9 +844,25 @@ int osnoise_main(int argc, char *argv[])
 	if (argc == 0)
 		goto usage;
 
+	/*
+	 * if osnoise was called without any argument, run the
+	 * default cmdline.
+	 */
+	if (argc == 1) {
+		osnoise_top_main(argc, argv);
+		exit(0);
+	}
+
 	if ((strcmp(argv[1], "-h") == 0) || (strcmp(argv[1], "--help") == 0)) {
 		osnoise_usage();
 		exit(0);
+	} else if (strncmp(argv[1], "-", 1) == 0) {
+		/* the user skipped the tool, call the default one */
+		osnoise_top_main(argc, argv);
+		exit(0);
+	} else if (strcmp(argv[1], "top") == 0) {
+		osnoise_top_main(argc-1, &argv[1]);
+		exit(0);
 	}
 
 usage:
tools/tracing/rtla/src/osnoise.h +1 -0
index 1184fc66dd50..a4b114ed7d71 100644
@@ -86,4 +86,5 @@ void osnoise_destroy_tool(struct osnoise_tool *top);
 struct osnoise_tool *osnoise_init_tool(char *tool_name);
 struct osnoise_tool *osnoise_init_trace_tool(char *tracer);
 
+int osnoise_top_main(int argc, char **argv);
 int osnoise_main(int argc, char **argv);
tools/tracing/rtla/src/osnoise_top.c new file +579 -0

579 lines not on this page · raw message

Signature · 1 line
2.33.0
author maintainer
#5 lore raw

PATCH 5/14 rtla/osnoise: Add the hist mode

In reply to Steven Rostedt

From: Daniel Bristot de Oliveira <[email protected]>

The rtla osnoise hist tool collects all osnoise:sample_threshold occurrence in a histogram, displaying the results in a user-friendly way. The tool also allows many configurations of the osnoise tracer and the collection of the tracer output.

Here is one example of the rtla osnoise hist tool output:
---------- %< ----------
[root@f34 ~]# rtla osnoise hist --bucket-size 10 --entries 100 -c 0-8 -d 1M -r 9000 -P F:1 # RTLA osnoise histogram
# Time unit is microseconds (us)
# Duration: 0 00:01:00
Index CPU-000 CPU-001 CPU-002 CPU-003 CPU-004 CPU-005 CPU-006 CPU-007 CPU-008

 0           430       434       352       455       440       463       467       436       484
 10           88        88        92       141       120       100       126       166       100
 20           19         7        12        22         8         8        13        13        16
 30            6         0         2         0         1         2         2         1         0
 50            0         0         0         0         0         0         1         0         0
 over:         0         0         0         0         0         0         0         0         0
 count:      543       529       458       618       569       573       609       616       600
 min:          0         0         0         0         0         0         0         0         0
 avg:          0         0         0         0         0         0         0         0         0

max: 30 20 30 20 30 30 50 30 20
---------- >% ----------

Running
- rtla osnoise hist --help

provides information about the available options.

Cc Tao Zhou <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 tools/tracing/rtla/src/osnoise.c      |   6 +-
 tools/tracing/rtla/src/osnoise.h      |   1 +
 tools/tracing/rtla/src/osnoise_hist.c | 801 ++++++++++++++++++++++++++
 3 files changed, 807 insertions(+), 1 deletion(-)
 create mode 100644 tools/tracing/rtla/src/osnoise_hist.c
tools/tracing/rtla/src/osnoise.c +5 -1
index fbf4f9bfda02..7b73d1eccd0e 100644
@@ -828,7 +828,8 @@ static void osnoise_usage(void)
 		"  usage: [rtla] osnoise [MODE] ...",
 		"",
 		"  modes:",
-		"     top  - prints the summary from osnoise tracer",
+		"     top   - prints the summary from osnoise tracer",
+		"     hist  - prints a histogram of osnoise samples",
 		"",
 		"if no MODE is given, the top mode is called, passing the arguments",
 		NULL,
@@ -863,6 +864,9 @@ int osnoise_main(int argc, char *argv[])
 	} else if (strcmp(argv[1], "top") == 0) {
 		osnoise_top_main(argc-1, &argv[1]);
 		exit(0);
+	} else if (strcmp(argv[1], "hist") == 0) {
+		osnoise_hist_main(argc-1, &argv[1]);
+		exit(0);
 	}
 
 usage:
tools/tracing/rtla/src/osnoise.h +1 -0
index a4b114ed7d71..9e4b2e2a4559 100644
@@ -86,5 +86,6 @@ void osnoise_destroy_tool(struct osnoise_tool *top);
 struct osnoise_tool *osnoise_init_tool(char *tool_name);
 struct osnoise_tool *osnoise_init_trace_tool(char *tracer);
 
+int osnoise_hist_main(int argc, char *argv[]);
 int osnoise_top_main(int argc, char **argv);
 int osnoise_main(int argc, char **argv);
tools/tracing/rtla/src/osnoise_hist.c new file +801 -0

801 lines not on this page · raw message

Signature · 1 line
2.33.0
author maintainer
#6 lore raw

PATCH 6/14 rtla: Add timerlat tool and timelart top mode

In reply to Steven Rostedt

From: Daniel Bristot de Oliveira <[email protected]>

The rtla timerlat tool is an interface for the timerlat tracer. The timerlat tracer dispatches a kernel thread per-cpu. These threads set a periodic timer to wake themselves up and go back to sleep. After the wakeup, they collect and generate useful information for the debugging of operating system timer latency.

The timerlat tracer outputs information in two ways. It periodically prints the timer latency at the timer IRQ handler and the Thread handler. It also provides information for each noise via the osnoise tracepoints.

The rtla timerlat top mode displays a summary of the periodic output from the timerlat tracer.

Here is one example of the rtla timerlat tool output:
---------- %< ----------
[root@alien ~]# rtla timerlat top -c 0-3 -d 1m
Timer Latency

  0 00:01:00   |          IRQ Timer Latency (us)        |         Thread Timer Latency (us)
CPU COUNT      |      cur       min       avg       max |      cur       min       avg       max
  0 #60001     |        0         0         0         3 |        1         1         1         6
  1 #60001     |        0         0         0         3 |        2         1         1         5
  2 #60001     |        0         0         1         6 |        1         1         2         7
  3 #60001     |        0         0         0         7 |        1         1         1        11

---------- >% ----------

Running:

  # rtla timerlat --help
  # rtla timerlat top --help
provides information about the available options.
Cc Tao Zhou <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 tools/tracing/rtla/Makefile           |   2 +
 tools/tracing/rtla/src/rtla.c         |   5 +
 tools/tracing/rtla/src/timerlat.c     |  68 +++
 tools/tracing/rtla/src/timerlat.h     |   4 +
 tools/tracing/rtla/src/timerlat_top.c | 618 ++++++++++++++++++++++++++
 5 files changed, 697 insertions(+)
 create mode 100644 tools/tracing/rtla/src/timerlat.c
 create mode 100644 tools/tracing/rtla/src/timerlat.h
 create mode 100644 tools/tracing/rtla/src/timerlat_top.c
tools/tracing/rtla/Makefile +2 -0
index d40b442fcd30..7a7348faec32 100644
@@ -62,6 +62,8 @@ install:
 	$(STRIP) $(DESTDIR)$(BINDIR)/rtla
 	@test ! -f $(DESTDIR)$(BINDIR)/osnoise || rm $(DESTDIR)$(BINDIR)/osnoise
 	ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/osnoise
+	@test ! -f $(DESTDIR)$(BINDIR)/timerlat || rm $(DESTDIR)$(BINDIR)/timerlat
+	ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/timerlat
 
 .PHONY: clean tarball
 clean:
tools/tracing/rtla/src/rtla.c +5 -0
index 669b9750b3b3..09bd21b8af81 100644
@@ -9,6 +9,7 @@
 #include <stdio.h>
 
 #include "osnoise.h"
+#include "timerlat.h"
 
 /*
  * rtla_usage - print rtla usage
@@ -25,6 +26,7 @@ static void rtla_usage(void)
 		"",
 		"  commands:",
 		"     osnoise  - gives information about the operating system noise (osnoise)",
+		"     timerlat - measures the timer irq and thread latency",
 		"",
 		NULL,
 	};
@@ -45,6 +47,9 @@ int run_command(int argc, char **argv, int start_position)
 	if (strcmp(argv[start_position], "osnoise") == 0) {
 		osnoise_main(argc-start_position, &argv[start_position]);
 		goto ran;
+	} else if (strcmp(argv[start_position], "timerlat") == 0) {
+		timerlat_main(argc-start_position, &argv[start_position]);
+		goto ran;
 	}
 
 	return 0;
tools/tracing/rtla/src/timerlat.c new file +68 -0

68 lines not on this page · raw message

tools/tracing/rtla/src/timerlat.h new file +4 -0
new file mode 100644
index 000000000000..6f9d3925acd0
@@ -0,0 +1,4 @@
+// SPDX-License-Identifier: GPL-2.0
+
+int timerlat_top_main(int argc, char *argv[]);
+int timerlat_main(int argc, char *argv[]);
tools/tracing/rtla/src/timerlat_top.c new file +618 -0

618 lines not on this page · raw message

Signature · 1 line
2.33.0
author maintainer
#7 lore raw

PATCH 7/14 rtla/timerlat: Add timerlat hist mode

In reply to Steven Rostedt

From: Daniel Bristot de Oliveira <[email protected]>

The rtla hist hist mode displays a histogram of each tracer event occurrence, both for IRQ and timer latencies. The tool also allows many configurations of the timerlat tracer and the collection of the tracer output.

Here is one example of the rtla timerlat hist mode output:
---------- %< ----------
[root@alien ~]# rtla timerlat hist -c 0-3 -d 1M # RTLA timerlat histogram
# Time unit is microseconds (us)
# Duration: 0 00:01:00
Index IRQ-000 Thr-000 IRQ-001 Thr-001 IRQ-002 Thr-002 IRQ-003 Thr-003
0 58572 0 59373 0 58691 0 58895 0
1 1422 57021 628 57241 1310 56160 1102 56805

 2             6      2931         0      2695         0      3567         4      3031
 3             1        40         0        53         0       260         0       142
 4             0         7         0         5         0         6         0        17
 5             0         2         0         5         0         7         0         4
 6             0         0         0         2         0         1         0         1
 8             0         0         0         0         0         0         0         1
 over:         0         0         0         0         0         0         0         0
 count:    60001     60001     60001     60001     60001     60001     60001     60001
 min:          0         1         0         1         0         1         0         1
 avg:          0         1         0         1         0         1         0         1
 max:          3         5         1         6         1         6         2         8

---------- >% ----------

Running
- rtla timerlat hist --help
provides information about the available options.

Cc Tao Zhou <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 tools/tracing/rtla/src/timerlat.c      |   6 +-
 tools/tracing/rtla/src/timerlat.h      |   2 +-
 tools/tracing/rtla/src/timerlat_hist.c | 822 +++++++++++++++++++++++++
 3 files changed, 828 insertions(+), 2 deletions(-)
 create mode 100644 tools/tracing/rtla/src/timerlat_hist.c
tools/tracing/rtla/src/timerlat.c +5 -1
index 29a51c01b084..97abbf494fee 100644
@@ -25,7 +25,8 @@ static void timerlat_usage(void)
 		"  usage: [rtla] timerlat [MODE] ...",
 		"",
 		"  modes:",
-		"     top  - prints the summary from timerlat tracer",
+		"     top   - prints the summary from timerlat tracer",
+		"     hist  - prints a histogram of timer latencies",
 		"",
 		"if no MODE is given, the top mode is called, passing the arguments",
 		NULL,
@@ -60,6 +61,9 @@ int timerlat_main(int argc, char *argv[])
 	} else if (strcmp(argv[1], "top") == 0) {
 		timerlat_top_main(argc-1, &argv[1]);
 		exit(0);
+	} else if (strcmp(argv[1], "hist") == 0) {
+		timerlat_hist_main(argc-1, &argv[1]);
+		exit(0);
 	}
 
 usage:
tools/tracing/rtla/src/timerlat.h +1 -1
index 6f9d3925acd0..88561bfd14f3 100644
@@ -1,4 +1,4 @@
 // SPDX-License-Identifier: GPL-2.0
-
+int timerlat_hist_main(int argc, char *argv[]);
 int timerlat_top_main(int argc, char *argv[]);
 int timerlat_main(int argc, char *argv[]);
tools/tracing/rtla/src/timerlat_hist.c new file +822 -0

822 lines not on this page · raw message

Signature · 1 line
2.33.0
author maintainer
#8 lore raw

PATCH 8/14 rtla: Add Documentation

In reply to Steven Rostedt

From: Daniel Bristot de Oliveira <[email protected]>

Adds the basis for rtla documentation. This patch also includes the rtla(1) man page.

As suggested by Jonathan Corbet, we are placing these man pages at Documentation/tools/rtla, using rst format. It is not linked to the official documentation, though.

The Makefile is based on bpftool's Documentation one.

Cc Tao Zhou <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 Documentation/tools/rtla/Makefile            | 41 +++++++++++++++++
 Documentation/tools/rtla/common_appendix.rst | 12 +++++
 Documentation/tools/rtla/rtla.rst            | 48 ++++++++++++++++++++
 tools/tracing/rtla/Makefile                  | 30 ++++++++++--
 4 files changed, 127 insertions(+), 4 deletions(-)
 create mode 100644 Documentation/tools/rtla/Makefile
 create mode 100644 Documentation/tools/rtla/common_appendix.rst
 create mode 100644 Documentation/tools/rtla/rtla.rst
Documentation/tools/rtla/Makefile new file +41 -0
new file mode 100644
index 000000000000..9f2b84af1a6c
@@ -0,0 +1,41 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# Based on bpftool's Documentation Makefile
+
+INSTALL		?= install
+RM		?= rm -f
+RMDIR		?= rmdir --ignore-fail-on-non-empty
+
+PREFIX		?= /usr/share
+MANDIR		?= $(PREFIX)/man
+MAN1DIR		= $(MANDIR)/man1
+
+MAN1_RST	= $(wildcard rtla*.rst)
+
+_DOC_MAN1	= $(patsubst %.rst,%.1,$(MAN1_RST))
+DOC_MAN1	= $(addprefix $(OUTPUT),$(_DOC_MAN1))
+
+RST2MAN_DEP	:= $(shell command -v rst2man 2>/dev/null)
+RST2MAN_OPTS	+= --verbose
+
+$(OUTPUT)%.1: %.rst
+ifndef RST2MAN_DEP
+	$(error "rst2man not found, but required to generate man pages")
+endif
+	rst2man $(RST2MAN_OPTS) $< > $@
+
+man1: $(DOC_MAN1)
+man: man1
+
+clean:
+	$(RM) $(DOC_MAN1)
+
+install: man
+	$(INSTALL) -d -m 755 $(DESTDIR)$(MAN1DIR)
+	$(INSTALL) -m 644 $(DOC_MAN1) $(DESTDIR)$(MAN1DIR)
+
+uninstall:
+	$(RM) $(addprefix $(DESTDIR)$(MAN1DIR)/,$(_DOC_MAN1))
+	$(RMDIR) $(DESTDIR)$(MAN1DIR)
+
+.PHONY: man man1 clean install uninstall
+.DEFAULT_GOAL := man
Documentation/tools/rtla/common_appendix.rst new file +12 -0
new file mode 100644
index 000000000000..b494084acccd
@@ -0,0 +1,12 @@
+REPORTING BUGS
+==============
+Report bugs to <[email protected]>
+
+LICENSE
+=======
+**rtla** is Free Software licensed under the GNU GPLv2
+
+COPYING
+=======
+Copyright \(C) 2021 Red Hat, Inc. Free use of this software is granted under
+the terms of the GNU Public License (GPL).
Documentation/tools/rtla/rtla.rst new file +48 -0

48 lines not on this page · raw message

tools/tracing/rtla/Makefile +26 -4
index 7a7348faec32..2d52ff0bff7d 100644
@@ -45,18 +45,28 @@ DATADIR	:=	/usr/share
 DOCDIR	:=	$(DATADIR)/doc
 MANDIR	:=	$(DATADIR)/man
 LICDIR	:=	$(DATADIR)/licenses
+SRCTREE	:=	$(if $(BUILD_SRC),$(BUILD_SRC),$(CURDIR))
+
+# If running from the tarball, man pages are stored in the Documentation
+# dir. If running from the kernel source, man pages are stored in
+# Documentation/tools/rtla/.
+ifneq ($(wildcard Documentation/.*),)
+DOCSRC	=	Documentation/
+else
+DOCSRC	=	$(SRCTREE)/../../../Documentation/tools/rtla/
+endif
 
 .PHONY:	all
 all:	rtla
 
-rtla: $(OBJ)
+rtla: $(OBJ) doc
 	$(CC) -o rtla $(LDFLAGS) $(OBJ) $(LIBS)
 
 static: $(OBJ)
 	$(CC) -o rtla-static $(LDFLAGS) --static $(OBJ) $(LIBS) -lpthread -ldl
 
 .PHONY: install
-install:
+install: doc_install
 	$(INSTALL) -d -m 755 $(DESTDIR)$(BINDIR)
 	$(INSTALL) rtla -m 755 $(DESTDIR)$(BINDIR)
 	$(STRIP) $(DESTDIR)$(BINDIR)/rtla
@@ -66,15 +76,27 @@ install:
 	ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/timerlat
 
 .PHONY: clean tarball
-clean:
+clean: doc_clean
 	@test ! -f rtla || rm rtla
 	@test ! -f rtla-static || rm rtla-static
 	@test ! -f src/rtla.o || rm src/rtla.o
 	@test ! -f $(TARBALL) || rm -f $(TARBALL)
 	@rm -rf *~ $(OBJ) *.tar.$(CEXT)
 
-tarball:  clean
+tarball: clean
 	rm -rf $(NAME)-$(VERSION) && mkdir $(NAME)-$(VERSION)
 	cp -r $(DIRS) $(FILES) $(NAME)-$(VERSION)
+	mkdir $(NAME)-$(VERSION)/Documentation/
+	cp -rp $(SRCTREE)/../../../Documentation/tools/rtla/* $(NAME)-$(VERSION)/Documentation/
 	tar $(TAROPTS) --exclude='*~' $(NAME)-$(VERSION)
 	rm -rf $(NAME)-$(VERSION)
+
+.PHONY: doc doc_clean doc_install
+doc:
+	$(MAKE) -C $(DOCSRC)
+
+doc_clean:
+	$(MAKE) -C $(DOCSRC) clean
+
+doc_install:
+	$(MAKE) -C $(DOCSRC) install
Signature · 1 line
2.33.0
author maintainer
#9 lore raw

PATCH 9/14 rtla: Add rtla osnoise man page

In reply to Steven Rostedt

From: Daniel Bristot de Oliveira <[email protected]>

Man page for rtla osnoise command.

Cc Tao Zhou <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 .../tools/rtla/common_osnoise_description.rst |  8 +++
 Documentation/tools/rtla/rtla-osnoise.rst     | 59 +++++++++++++++++++
 2 files changed, 67 insertions(+)
 create mode 100644 Documentation/tools/rtla/common_osnoise_description.rst
 create mode 100644 Documentation/tools/rtla/rtla-osnoise.rst
Documentation/tools/rtla/common_osnoise_description.rst new file +8 -0
new file mode 100644
index 000000000000..8973c5df888f
@@ -0,0 +1,8 @@
+The **rtla osnoise** tool is an interface for the *osnoise* tracer. The
+*osnoise* tracer dispatches a kernel thread per-cpu. These threads read the
+time in a loop while with preemption, softirq and IRQs enabled, thus
+allowing all the sources of operating systme noise during its execution.
+The *osnoise*'s tracer threads take note of the delta between each time
+read, along with an interference counter of all sources of interference.
+At the end of each period, the *osnoise* tracer displays a summary of
+the results.
Documentation/tools/rtla/rtla-osnoise.rst new file +59 -0

59 lines not on this page · raw message

Signature · 1 line
2.33.0
author maintainer
#10 lore raw

PATCH 10/14 rtla: Add rtla osnoise top documentation

In reply to Steven Rostedt

From: Daniel Bristot de Oliveira <[email protected]>

Man page for rtla osnoise top mode.

Cc Tao Zhou <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 Documentation/tools/rtla/common_options.rst   | 28 +++++++++
 .../tools/rtla/common_osnoise_options.rst     | 17 ++++++
 .../tools/rtla/common_top_options.rst         |  3 +
 Documentation/tools/rtla/rtla-osnoise-top.rst | 61 +++++++++++++++++++
 4 files changed, 109 insertions(+)
 create mode 100644 Documentation/tools/rtla/common_options.rst
 create mode 100644 Documentation/tools/rtla/common_osnoise_options.rst
 create mode 100644 Documentation/tools/rtla/common_top_options.rst
 create mode 100644 Documentation/tools/rtla/rtla-osnoise-top.rst
Documentation/tools/rtla/common_options.rst new file +28 -0
new file mode 100644
index 000000000000..721790ad984e
@@ -0,0 +1,28 @@
+**-c**, **--cpus** *cpu-list*
+
+        Set the osnoise tracer to run the sample threads in the cpu-list.
+
+**-d**, **--duration** *time[s|m|h|d]*
+
+        Set the duration of the session.
+
+**-D**, **--debug**
+
+        Print debug info.
+
+**-t**, **--trace**\[*=file*]
+
+        Save the stopped trace to [*file|osnoise_trace.txt*].
+
+**-P**, **--priority** *o:prio|r:prio|f:prio|d:runtime:period*
+
+        Set scheduling parameters to the osnoise tracer threads, the format to set the priority are:
+
+        - *o:prio* - use SCHED_OTHER with *prio*;
+        - *r:prio* - use SCHED_RR with *prio*;
+        - *f:prio* - use SCHED_FIFO with *prio*;
+        - *d:runtime[us|ms|s]:period[us|ms|s]* - use SCHED_DEADLINE with *runtime* and *period* in nanoseconds.
+
+**-h**, **--help**
+
+        Print help menu.
Documentation/tools/rtla/common_osnoise_options.rst new file +17 -0
new file mode 100644
index 000000000000..d556883e4e26
@@ -0,0 +1,17 @@
+**-p**, **--period** *us*
+
+        Set the *osnoise* tracer period in microseconds.
+
+**-r**, **--runtime** *us*
+
+        Set the *osnoise* tracer runtime in microseconds.
+
+**-s**, **--stop** *us*
+
+        Stop the trace if a single sample is higher than the argument in microseconds.
+        If **-T** is set, it will also save the trace to the output.
+
+**-S**, **--stop-total** *us*
+
+        Stop the trace if the total sample is higher than the argument in microseconds.
+        If **-T** is set, it will also save the trace to the output.
Documentation/tools/rtla/common_top_options.rst new file +3 -0
new file mode 100644
index 000000000000..f48878938f84
@@ -0,0 +1,3 @@
+**-q**, **--quiet**
+
+        Print only a summary at the end of the session.
Documentation/tools/rtla/rtla-osnoise-top.rst new file +61 -0

61 lines not on this page · raw message

Signature · 1 line
2.33.0
author maintainer
#11 lore raw

PATCH 11/14 rtla: Add rtla osnoise hist documentation

In reply to Steven Rostedt

From: Daniel Bristot de Oliveira <[email protected]>

Man page for rtla osnoise hist mode.

Cc Tao Zhou <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 .../tools/rtla/common_hist_options.rst        | 23 +++++++
 .../tools/rtla/rtla-osnoise-hist.rst          | 66 +++++++++++++++++++
 2 files changed, 89 insertions(+)
 create mode 100644 Documentation/tools/rtla/common_hist_options.rst
 create mode 100644 Documentation/tools/rtla/rtla-osnoise-hist.rst
Documentation/tools/rtla/common_hist_options.rst new file +23 -0
new file mode 100644
index 000000000000..0266cd08a6c9
@@ -0,0 +1,23 @@
+**-b**, **--bucket-size** *N*
+
+        Set the histogram bucket size (default *1*).
+
+**-e**, **--entries** *N*
+
+        Set the number of entries of the histogram (default 256).
+
+**--no-header**
+
+        Do not print header.
+
+**--no-summary**
+
+        Do not print summary.
+
+**--no-index**
+
+        Do not print index.
+
+**--with-zeros**
+
+        Print zero only entries.
Documentation/tools/rtla/rtla-osnoise-hist.rst new file +66 -0

66 lines not on this page · raw message

Signature · 1 line
2.33.0
author maintainer
#12 lore raw

PATCH 12/14 rtla: Add rtla timerlat documentation

In reply to Steven Rostedt

From: Daniel Bristot de Oliveira <[email protected]>

Man page for rtla timerlat tool.

Cc Tao Zhou <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 .../rtla/common_timerlat_description.rst      | 10 ++++
 Documentation/tools/rtla/rtla-timerlat.rst    | 57 +++++++++++++++++++
 2 files changed, 67 insertions(+)
 create mode 100644 Documentation/tools/rtla/common_timerlat_description.rst
 create mode 100644 Documentation/tools/rtla/rtla-timerlat.rst
Documentation/tools/rtla/common_timerlat_description.rst new file +10 -0
new file mode 100644
index 000000000000..321201cb8597
@@ -0,0 +1,10 @@
+The **rtla timerlat** tool is an interface for the *timerlat* tracer. The
+*timerlat* tracer dispatches a kernel thread per-cpu. These threads
+set a periodic timer to wake themselves up and go back to sleep. After
+the wakeup, they collect and generate useful information for the
+debugging of operating system timer latency.
+
+The *timerlat* tracer outputs information in two ways. It periodically
+prints the timer latency at the timer *IRQ* handler and the *Thread*
+handler. It also enable the trace of the most relevant information via
+**osnoise:** tracepoints.
Documentation/tools/rtla/rtla-timerlat.rst new file +57 -0

57 lines not on this page · raw message

Signature · 1 line
2.33.0
author maintainer
#13 lore raw

PATCH 13/14 rtla: Add rtla timerlat top documentation

In reply to Steven Rostedt

From: Daniel Bristot de Oliveira <[email protected]>

Man page for rtla timerlat top mode.

Cc Tao Zhou <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 .../tools/rtla/common_timerlat_options.rst    |  16 ++
 .../tools/rtla/rtla-timerlat-top.rst          | 145 ++++++++++++++++++
 2 files changed, 161 insertions(+)
 create mode 100644 Documentation/tools/rtla/common_timerlat_options.rst
 create mode 100644 Documentation/tools/rtla/rtla-timerlat-top.rst
Documentation/tools/rtla/common_timerlat_options.rst new file +16 -0
new file mode 100644
index 000000000000..e9c1bfd55d48
@@ -0,0 +1,16 @@
+**-p**, **--period** *us*
+
+        Set the *timerlat* tracer period in microseconds.
+
+**-i**, **--irq** *us*
+
+        Stop trace if the *IRQ* latency is higher than the argument in us.
+
+**-T**, **--thread** *us*
+
+        Stop trace if the *Thread* latency is higher than the argument in us.
+
+**-s**, **--stack** *us*
+
+        Save the stack trace at the *IRQ* if a *Thread* latency is higher than the
+        argument in us.
Documentation/tools/rtla/rtla-timerlat-top.rst new file +145 -0

145 lines not on this page · raw message

Signature · 1 line
2.33.0
author maintainer
#14 lore raw

PATCH 14/14 rtla: Add rtla timerlat hist documentation

In reply to Steven Rostedt

From: Daniel Bristot de Oliveira <[email protected]>

Man page for rtla timerlat hist mode.

Cc Tao Zhou <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 .../tools/rtla/rtla-timerlat-hist.rst         | 106 ++++++++++++++++++
 1 file changed, 106 insertions(+)
 create mode 100644 Documentation/tools/rtla/rtla-timerlat-hist.rst
Documentation/tools/rtla/rtla-timerlat-hist.rst new file +106 -0
new file mode 100644
index 000000000000..e12eae1f3301
@@ -0,0 +1,106 @@
+=====================
+rtla-timerlat-hist
+=====================
+------------------------------------------------
+Histograms of the operating system timer latency
+------------------------------------------------
+
+:Manual section: 1
+
+SYNOPSIS
+========
+**rtla timerlat hist** [*OPTIONS*] ...
+
+DESCRIPTION
+===========
+
+.. include:: common_timerlat_description.rst
+
+The **rtla timerlat hist** displays a histogram of each tracer event
+occurrence. This tool uses the periodic information, and the
+**osnoise:** tracepoints are enabled when using the **-T** option.
+
+OPTIONS
+=======
+
+.. include:: common_timerlat_options.rst
+
+.. include:: common_hist_options.rst
+
+.. include:: common_options.rst
+
+EXAMPLE
+=======
+In the example below, **rtla timerlat hist** is set to run for *10* minutes,
+in the cpus *0-4*, *skipping zero* only lines. Moreover, **rtla timerlat
+hist** will change the priority of the *timelat* threads to run under
+*SCHED_DEADLINE* priority, with a *10us* runtime every *1ms* period. The
+*1ms* period is also passed to the *timerlat* tracer::
+
+  [root@alien ~]# timerlat hist -d 10m -c 0-4 -P d:100us:1ms -p 1ms
+  # RTLA timerlat histogram
+  # Time unit is microseconds (us)
+  # Duration:   0 00:10:00
+  Index   IRQ-000   Thr-000   IRQ-001   Thr-001   IRQ-002   Thr-002   IRQ-003   Thr-003   IRQ-004   Thr-004
+  0        276489         0    206089         0    466018         0    481102         0    205546         0
+  1        318327     35487    388149     30024     94531     48382     83082     71078    388026     55730
+  2          3282    122584      4019    126527     28231    109012     23311     89309      4568     98739
+  3           940     11815       837      9863      6209     16227      6895     17196       910      9780
+  4           444     17287       424     11574      2097     38443      2169     36736       462     13476
+  5           206     43291       255     25581      1223    101908      1304    101137       236     28913
+  6           132    101501        96     64584       635    213774       757    215471        99     73453
+  7            74    169347        65    124758       350     57466       441     53639        69    148573
+  8            53     85183        31    156751       229      9052       306      9026        39    139907
+  9            22     10387        12     42762       161      2554       225      2689        19     26192
+  10           13      1898         8      5770       114      1247       128      1405        13      3772
+  11            9       560         9       924        71       686        76       765         8       713
+  12            4       256         2       360        50       411        64       474         3       278
+  13            2       167         2       172        43       256        53       350         4       180
+  14            1        88         1       116        15       198        42       223         0       115
+  15            2        63         3        94        11       139        20       150         0        58
+  16            2        37         0        56         5        78        10       102         0        39
+  17            0        18         0        28         4        57         8        80         0        15
+  18            0         8         0        17         2        50         6        56         0        12
+  19            0         9         0         5         0        19         0        48         0        18
+  20            0         4         0         8         0        11         2        27         0         4
+  21            0         2         0         3         1         9         1        18         0         6
+  22            0         1         0         3         1         7         0         3         0         5
+  23            0         2         0         4         0         2         0         7         0         2
+  24            0         2         0         2         1         3         0         3         0         5
+  25            0         0         0         1         0         1         0         1         0         3
+  26            0         1         0         0         0         2         0         2         0         0
+  27            0         0         0         3         0         1         0         0         0         1
+  28            0         0         0         3         0         0         0         1         0         0
+  29            0         0         0         2         0         2         0         1         0         3
+  30            0         1         0         0         0         0         0         0         0         0
+  31            0         1         0         0         0         0         0         2         0         2
+  32            0         0         0         1         0         2         0         0         0         0
+  33            0         0         0         2         0         0         0         0         0         1
+  34            0         0         0         0         0         0         0         0         0         2
+  35            0         1         0         1         0         0         0         0         0         1
+  36            0         1         0         0         0         1         0         1         0         0
+  37            0         0         0         1         0         0         0         0         0         0
+  40            0         0         0         0         0         1         0         1         0         0
+  41            0         0         0         0         0         0         0         0         0         1
+  42            0         0         0         0         0         0         0         0         0         1
+  44            0         0         0         0         0         1         0         0         0         0
+  46            0         0         0         0         0         0         0         1         0         0
+  47            0         0         0         0         0         0         0         0         0         1
+  50            0         0         0         0         0         0         0         0         0         1
+  54            0         0         0         1         0         0         0         0         0         0
+  58            0         0         0         1         0         0         0         0         0         0
+  over:         0         0         0         0         0         0         0         0         0         0
+  count:   600002    600002    600002    600002    600002    600002    600002    600002    600002    600002
+  min:          0         1         0         1         0         1         0         1         0         1
+  avg:          0         5         0         5         0         4         0         4         0         5
+  max:         16        36        15        58        24        44        21        46        13        50
+
+SEE ALSO
+========
+**rtla-timerlat**\(1), **rtla-timerlat-top**\(1)
+
+*timerlat* tracer documentation: <https://www.kernel.org/doc/html/latest/trace/timerlat-tracer.html>
+
+AUTHOR
+======
+Written by Daniel Bristot de Oliveira <[email protected]>
Signature · 1 line
2.33.0