Skip to contentSkip to conversation

RTLA: An interface for osnoise/timerlat tracers

PATCH Superseded Daniel Bristot de Oliveira 3 participants 29 messages Nov 29, 2021 – Dec 9, 2021

Subsystems (2)

v8 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v8 cover RTLA: An interface for osnoise/timerlat tracers

The rtla(1) is a meta-tool that includes a set of commands that aims to analyze the real-time properties of Linux. But instead of testing Linux as a black box, rtla leverages kernel tracing capabilities to provide precise information about the properties and root causes of unexpected results.

To start, it presents an interface to the osnoise and timerlat tracers. In the future, it will also serve as home to the rtsl [1] and other latency/noise tracers.

If you just want to run it, you can download the tarball here:
- https://bristot.me/files/rtla/tarball/rtla-0.4.tar.bz2

To compile rtla on fedora you need:

  $ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.git
  $ cd libtraceevent/
  $ make
  $ sudo make install
  $ cd ..
  $ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtracefs.git
  $ cd libtracefs/
  $ make
  $ sudo make install
  $ cd ..
  $ sudo dnf install python3-docutils procps-devel
  $ cd $rtla_src
  $ make
  $ sudo make install

The tracing option (-t) depends some kernel patches that are available at [2].

This tool was be discussed at the RT-MC during LPC2021 [3]

[1] rtsl: https://github.com/bristot/rtsl/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://youtu.be/cZUzc0U1jJ4

Changes from V7:
- Add README.txt with information about how to compile the tool (Steven)
- Use $$($(PKG_CONFIG) --libs libtracefs) to find libtracefs on Makefile (Steven)
- Fix *buffer[4096] (using buffer[4096]) on save_trace_to_file() (Steven)
Changes from v6:
- Revisited osnoise option config functions
- Properly handles offline CPUs
- Some cleanups
- Fixed an histogram allocation problem (Tao Zhou)
- Revisited open()/read()/write() (Tao Zhou)
Changes from v5:
- Fix retval check in save_trace_to_file() (Tao Zhou)
- Fix goto logic in save_trace_to_file() (Tao Zhou)
- Removed unused save_trace_pipe_to_file() function
- Correctly handle an error on osnoise_set_* functions during "apply config" for all tools (Tao Zhou)
Changes from v3:
- Add cross-compile support (Ahmed S. Darwish)
- Move documentation to Documentation/tools/rtla (Jonathan Corbet)
- Use .rst format for documentation (Jonathan Corbet)
- Use include option from .rst to group common parts of the documentation
- Makefile (main and doc) cleanups
Changes from v2:
- Fix the miss conception of the "size" for kernel histograms (Steven/Tom)
- Change the --skip-zeros to --with-zeros option as the former is better for humans (and the latter for computers to plot charts).
- A lot of checkpatch fixes for the user-space part.
Changes from v1:
- Fixes -t options on osnoise tracers (-t means --trace for all tools now)
- Fixes --bucket-size references (not --bucket_size)

Daniel Bristot de Oliveira (14):
rtla: Real-Time Linux Analysis tool
rtla: Helper functions for rtla
rtla: Add osnoise tool
rtla/osnoise: Add osnoise top mode
rtla/osnoise: Add the hist mode
rtla: Add timerlat tool and timelart top mode
rtla/timerlat: Add timerlat hist mode
rtla: Add Documentation
rtla: Add rtla osnoise man page
rtla: Add rtla osnoise top documentation
rtla: Add rtla osnoise hist documentation
rtla: Add rtla timerlat documentation
rtla: Add rtla timerlat top documentation
rtla: Add rtla timerlat hist documentation

 Documentation/tools/rtla/Makefile             |   41 +
 Documentation/tools/rtla/common_appendix.rst  |   12 +
 .../tools/rtla/common_hist_options.rst        |   23 +
 Documentation/tools/rtla/common_options.rst   |   24 +
 .../tools/rtla/common_osnoise_description.rst |    8 +
 .../tools/rtla/common_osnoise_options.rst     |   17 +
 .../rtla/common_timerlat_description.rst      |   10 +
 .../tools/rtla/common_timerlat_options.rst    |   16 +
 .../tools/rtla/common_top_options.rst         |    3 +
 .../tools/rtla/rtla-osnoise-hist.rst          |   66 ++
 Documentation/tools/rtla/rtla-osnoise-top.rst |   61 +
 Documentation/tools/rtla/rtla-osnoise.rst     |   59 +
 .../tools/rtla/rtla-timerlat-hist.rst         |  106 ++
 .../tools/rtla/rtla-timerlat-top.rst          |  145 +++
 Documentation/tools/rtla/rtla-timerlat.rst    |   57 +
 Documentation/tools/rtla/rtla.rst             |   48 +
 tools/tracing/rtla/Makefile                   |  102 ++
 tools/tracing/rtla/README.txt                 |   36 +
 tools/tracing/rtla/src/osnoise.c              | 1017 +++++++++++++++++
 tools/tracing/rtla/src/osnoise.h              |   96 ++
 tools/tracing/rtla/src/osnoise_hist.c         |  799 +++++++++++++
 tools/tracing/rtla/src/osnoise_top.c          |  577 ++++++++++
 tools/tracing/rtla/src/rtla.c                 |   87 ++
 tools/tracing/rtla/src/timerlat.c             |   72 ++
 tools/tracing/rtla/src/timerlat.h             |    4 +
 tools/tracing/rtla/src/timerlat_hist.c        |  820 +++++++++++++
 tools/tracing/rtla/src/timerlat_top.c         |  615 ++++++++++
 tools/tracing/rtla/src/trace.c                |  192 ++++
 tools/tracing/rtla/src/trace.h                |   27 +
 tools/tracing/rtla/src/utils.c                |  433 +++++++
 tools/tracing/rtla/src/utils.h                |   56 +
 31 files changed, 5629 insertions(+)
 create mode 100644 Documentation/tools/rtla/Makefile
 create mode 100644 Documentation/tools/rtla/common_appendix.rst
 create mode 100644 Documentation/tools/rtla/common_hist_options.rst
 create mode 100644 Documentation/tools/rtla/common_options.rst
 create mode 100644 Documentation/tools/rtla/common_osnoise_description.rst
 create mode 100644 Documentation/tools/rtla/common_osnoise_options.rst
 create mode 100644 Documentation/tools/rtla/common_timerlat_description.rst
 create mode 100644 Documentation/tools/rtla/common_timerlat_options.rst
 create mode 100644 Documentation/tools/rtla/common_top_options.rst
 create mode 100644 Documentation/tools/rtla/rtla-osnoise-hist.rst
 create mode 100644 Documentation/tools/rtla/rtla-osnoise-top.rst
 create mode 100644 Documentation/tools/rtla/rtla-osnoise.rst
 create mode 100644 Documentation/tools/rtla/rtla-timerlat-hist.rst
 create mode 100644 Documentation/tools/rtla/rtla-timerlat-top.rst
 create mode 100644 Documentation/tools/rtla/rtla-timerlat.rst
 create mode 100644 Documentation/tools/rtla/rtla.rst
 create mode 100644 tools/tracing/rtla/Makefile
 create mode 100644 tools/tracing/rtla/README.txt
 create mode 100644 tools/tracing/rtla/src/osnoise.c
 create mode 100644 tools/tracing/rtla/src/osnoise.h
 create mode 100644 tools/tracing/rtla/src/osnoise_hist.c
 create mode 100644 tools/tracing/rtla/src/osnoise_top.c
 create mode 100644 tools/tracing/rtla/src/rtla.c
 create mode 100644 tools/tracing/rtla/src/timerlat.c
 create mode 100644 tools/tracing/rtla/src/timerlat.h
 create mode 100644 tools/tracing/rtla/src/timerlat_hist.c
 create mode 100644 tools/tracing/rtla/src/timerlat_top.c
 create mode 100644 tools/tracing/rtla/src/trace.c
 create mode 100644 tools/tracing/rtla/src/trace.h
 create mode 100644 tools/tracing/rtla/src/utils.c
 create mode 100644 tools/tracing/rtla/src/utils.h
Signature · 1 line
2.31.1
#1 lore raw

PATCH v8 1/14 rtla: Real-Time Linux Analysis tool

The rtla is a meta-tool that includes a set of commands that aims to analyze the real-time properties of Linux. But instead of testing Linux as a black box, rtla leverages kernel tracing capabilities to provide precise information about the properties and root causes of unexpected results.

rtla --help works and provide information about the available options.

This is just the "main" and the Makefile, no function yet.

Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 tools/tracing/rtla/Makefile   | 76 +++++++++++++++++++++++++++++++++++
 tools/tracing/rtla/README.txt | 36 +++++++++++++++++
 tools/tracing/rtla/src/rtla.c | 72 +++++++++++++++++++++++++++++++++
 3 files changed, 184 insertions(+)
 create mode 100644 tools/tracing/rtla/Makefile
 create mode 100644 tools/tracing/rtla/README.txt
 create mode 100644 tools/tracing/rtla/src/rtla.c
tools/tracing/rtla/Makefile new file +76 -0
new file mode 100644
index 000000000000..d99ea2d8b87e
@@ -0,0 +1,76 @@
+NAME	:=	rtla
+VERSION	:=	0.4
+
+# From libtracefs:
+# Makefiles suck: This macro sets a default value of $(2) for the
+# variable named by $(1), unless the variable has been set by
+# environment or command line. This is necessary for CC and AR
+# because make sets default values, so the simpler ?= approach
+# won't work as expected.
+define allow-override
+  $(if $(or $(findstring environment,$(origin $(1))),\
+            $(findstring command line,$(origin $(1)))),,\
+    $(eval $(1) = $(2)))
+endef
+
+# Allow setting CC and AR, or setting CROSS_COMPILE as a prefix.
+$(call allow-override,CC,$(CROSS_COMPILE)gcc)
+$(call allow-override,AR,$(CROSS_COMPILE)ar)
+$(call allow-override,STRIP,$(CROSS_COMPILE)strip)
+$(call allow-override,PKG_CONFIG,pkg-config)
+$(call allow-override,LD_SO_CONF_PATH,/etc/ld.so.conf.d/)
+$(call allow-override,LDCONFIG,ldconfig)
+
+INSTALL	=	install
+FOPTS	:=	-flto=auto -ffat-lto-objects -fexceptions -fstack-protector-strong \
+		-fasynchronous-unwind-tables -fstack-clash-protection
+WOPTS	:= 	-Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -Wno-maybe-uninitialized
+
+TRACEFS_HEADERS	:= $$($(PKG_CONFIG) --cflags libtracefs)
+
+CFLAGS	:=	-O -g -DVERSION=\"$(VERSION)\" $(FOPTS) $(MOPTS) $(WOPTS) $(TRACEFS_HEADERS)
+LDFLAGS	:=	-ggdb
+LIBS	:=	$$($(PKG_CONFIG) --libs libtracefs) -lprocps
+
+SRC	:=	$(wildcard src/*.c)
+HDR	:=	$(wildcard src/*.h)
+OBJ	:=	$(SRC:.c=.o)
+DIRS	:=	src
+FILES	:=	Makefile README.txt
+CEXT	:=	bz2
+TARBALL	:=	$(NAME)-$(VERSION).tar.$(CEXT)
+TAROPTS	:=	-cvjf $(TARBALL)
+BINDIR	:=	/usr/bin
+DATADIR	:=	/usr/share
+DOCDIR	:=	$(DATADIR)/doc
+MANDIR	:=	$(DATADIR)/man
+LICDIR	:=	$(DATADIR)/licenses
+
+.PHONY:	all
+all:	rtla
+
+rtla: $(OBJ)
+	$(CC) -o rtla $(LDFLAGS) $(OBJ) $(LIBS)
+
+static: $(OBJ)
+	$(CC) -o rtla-static $(LDFLAGS) --static $(OBJ) $(LIBS) -lpthread -ldl
+
+.PHONY: install
+install:
+	$(INSTALL) -d -m 755 $(DESTDIR)$(BINDIR)
+	$(INSTALL) rtla -m 755 $(DESTDIR)$(BINDIR)
+	$(STRIP) $(DESTDIR)$(BINDIR)/rtla
+
+.PHONY: clean tarball
+clean:
+	@test ! -f rtla || rm rtla
+	@test ! -f rtla-static || rm rtla-static
+	@test ! -f src/rtla.o || rm src/rtla.o
+	@test ! -f $(TARBALL) || rm -f $(TARBALL)
+	@rm -rf *~ $(OBJ) *.tar.$(CEXT)
+
+tarball:  clean
+	rm -rf $(NAME)-$(VERSION) && mkdir $(NAME)-$(VERSION)
+	cp -r $(DIRS) $(FILES) $(NAME)-$(VERSION)
+	tar $(TAROPTS) --exclude='*~' $(NAME)-$(VERSION)
+	rm -rf $(NAME)-$(VERSION)
tools/tracing/rtla/README.txt new file +36 -0
new file mode 100644
index 000000000000..6c88446f7e74
@@ -0,0 +1,36 @@
+RTLA: Real-Time Linux Analysis tools
+
+The rtla is a meta-tool that includes a set of commands that
+aims to analyze the real-time properties of Linux. But, instead of
+testing Linux as a black box, rtla leverages kernel tracing
+capabilities to provide precise information about the properties
+and root causes of unexpected results.
+
+Installing RTLA
+
+RTLA depends on some libraries and tools. More precisely, it depends on the
+following libraries:
+
+ - libtracefs
+ - libtraceevent
+ - procps
+
+It also depends on python3-docutils to compile man pages.
+
+For development, we suggest the following steps for compiling rtla:
+
+  $ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.git
+  $ cd libtraceevent/
+  $ make
+  $ sudo make install
+  $ cd ..
+  $ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtracefs.git
+  $ cd libtracefs/
+  $ make
+  $ sudo make install
+  $ cd ..
+  $ cd $rtla_src
+  $ make
+  $ sudo make install
+
+For further information, please refer to the rtla man page.
tools/tracing/rtla/src/rtla.c new file +72 -0

72 lines not on this page · raw message

Signature · 1 line
2.31.1
#2 lore raw

PATCH v8 2/14 rtla: Helper functions for rtla

In reply to Daniel Bristot de Oliveira

This is a set of utils and tracer helper functions. They are used by rtla mostly to parse config, display data and some trace operations that are not part of libtracefs (because they are only useful it for this case).

Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 tools/tracing/rtla/src/trace.c | 192 +++++++++++++++
 tools/tracing/rtla/src/trace.h |  27 ++
 tools/tracing/rtla/src/utils.c | 433 +++++++++++++++++++++++++++++++++
 tools/tracing/rtla/src/utils.h |  56 +++++
 4 files changed, 708 insertions(+)
 create mode 100644 tools/tracing/rtla/src/trace.c
 create mode 100644 tools/tracing/rtla/src/trace.h
 create mode 100644 tools/tracing/rtla/src/utils.c
 create mode 100644 tools/tracing/rtla/src/utils.h
tools/tracing/rtla/src/trace.c new file +192 -0
new file mode 100644
index 000000000000..a403cb4238bb
@@ -0,0 +1,192 @@
+// SPDX-License-Identifier: GPL-2.0
+#define _GNU_SOURCE
+#include <sys/sendfile.h>
+#include <tracefs.h>
+#include <signal.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <errno.h>
+
+#include "trace.h"
+#include "utils.h"
+
+/*
+ * enable_tracer_by_name - enable a tracer on the given instance
+ */
+int enable_tracer_by_name(struct tracefs_instance *inst, const char *tracer)
+{
+	enum tracefs_tracers t;
+	int retval;
+
+	t = TRACEFS_TRACER_CUSTOM;
+
+	debug_msg("enabling %s tracer\n", tracer);
+
+	retval = tracefs_tracer_set(inst, t, tracer);
+	if (retval < 0) {
+		if (errno == ENODEV)
+			err_msg("tracer %s not found!\n", tracer);
+
+		err_msg("failed to enable the tracer %s\n", tracer);
+		return -1;
+	}
+
+	return 0;
+}
+
+/*
+ * disable_tracer - set nop tracer to the insta
+ */
+void disable_tracer(struct tracefs_instance *inst)
+{
+	enum tracefs_tracers t = TRACEFS_TRACER_NOP;
+	int retval;
+
+	retval = tracefs_tracer_set(inst, t);
+	if (retval < 0)
+		err_msg("oops, error disabling tracer\n");
+}
+
+/*
+ * create_instance - create a trace instance with *instance_name
+ */
+struct tracefs_instance *create_instance(char *instance_name)
+{
+	return tracefs_instance_create(instance_name);
+}
+
+/*
+ * destroy_instance - remove a trace instance and free the data
+ */
+void destroy_instance(struct tracefs_instance *inst)
+{
+	tracefs_instance_destroy(inst);
+	tracefs_instance_free(inst);
+}
+
+/*
+ * save_trace_to_file - save the trace output of the instance to the file
+ */
+int save_trace_to_file(struct tracefs_instance *inst, const char *filename)
+{
+	const char *file = "trace";
+	mode_t mode = 0644;
+	char buffer[4096];
+	int out_fd, in_fd;
+	int retval = -1;
+
+	in_fd = tracefs_instance_file_open(inst, file, O_RDONLY);
+	if (in_fd < 0) {
+		err_msg("Failed to open trace file\n");
+		return -1;
+	}
+
+	out_fd = creat(filename, mode);
+	if (out_fd < 0) {
+		err_msg("Failed to create output file %s\n", filename);
+		goto out_close_in;
+	}
+
+	do {
+		retval = read(in_fd, buffer, sizeof(buffer));
+		if (retval <= 0)
+			goto out_close;
+
+		retval = write(out_fd, buffer, retval);
+		if (retval < 0)
+			goto out_close;
+	} while (retval > 0);
+
+	retval = 0;
+out_close:
+	close(out_fd);
+out_close_in:
+	close(in_fd);
+	return retval;
+}
+
+/*
+ * collect_registered_events - call the existing callback function for the event
+ *
+ * If an event has a registered callback function, call it.
+ * Otherwise, ignore the event.
+ */
+int
+collect_registered_events(struct tep_event *event, struct tep_record *record,
+			  int cpu, void *context)
+{
+	struct trace_instance *trace = context;
+	struct trace_seq *s = trace->seq;
+
+	if (!event->handler)
+		return 0;
+
+	event->handler(s, record, event, context);
+
+	return 0;
+}
+
+/*
+ * trace_instance_destroy - destroy and free a rtla trace instance
+ */
+void trace_instance_destroy(struct trace_instance *trace)
+{
+	if (trace->inst) {
+		disable_tracer(trace->inst);
+		destroy_instance(trace->inst);
+	}
+
+	if (trace->seq)
+		free(trace->seq);
+
+	if (trace->tep)
+		tep_free(trace->tep);
+}
+
+/*
+ * trace_instance_init - create an rtla trace instance
+ *
+ * It is more than the tracefs instance, as it contains other
+ * things required for the tracing, such as the local events and
+ * a seq file.
+ *
+ * Note that the trace instance is returned disabled. This allows
+ * the tool to apply some other configs, like setting priority
+ * to the kernel threads, before starting generating trace entries.
+ */
+int trace_instance_init(struct trace_instance *trace, char *tool_name)
+{
+	trace->seq = calloc(1, sizeof(*trace->seq));
+	if (!trace->seq)
+		goto out_err;
+
+	trace_seq_init(trace->seq);
+
+	trace->inst = create_instance(tool_name);
+	if (!trace->inst)
+		goto out_err;
+
+	trace->tep = tracefs_local_events(NULL);
+	if (!trace->tep)
+		goto out_err;
+
+	/*
+	 * Let the main enable the record after setting some other
+	 * things such as the priority of the tracer's threads.
+	 */
+	tracefs_trace_off(trace->inst);
+
+	return 0;
+
+out_err:
+	trace_instance_destroy(trace);
+	return 1;
+}
+
+/*
+ * trace_instance_start - start tracing a given rtla instance
+ */
+int trace_instance_start(struct trace_instance *trace)
+{
+	return tracefs_trace_on(trace->inst);
+}
tools/tracing/rtla/src/trace.h new file +27 -0
new file mode 100644
index 000000000000..7c3930909b70
@@ -0,0 +1,27 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <tracefs.h>
+#include <stddef.h>
+
+struct trace_instance {
+	struct tracefs_instance		*inst;
+	struct tep_handle		*tep;
+	struct trace_seq		*seq;
+};
+
+int trace_instance_init(struct trace_instance *trace, char *tool_name);
+int trace_instance_start(struct trace_instance *trace);
+void trace_instance_destroy(struct trace_instance *trace);
+
+struct trace_seq *get_trace_seq(void);
+int enable_tracer_by_name(struct tracefs_instance *inst, const char *tracer);
+void disable_tracer(struct tracefs_instance *inst);
+
+int enable_osnoise(struct trace_instance *trace);
+int enable_timerlat(struct trace_instance *trace);
+
+struct tracefs_instance *create_instance(char *instance_name);
+void destroy_instance(struct tracefs_instance *inst);
+
+int save_trace_to_file(struct tracefs_instance *inst, const char *filename);
+int collect_registered_events(struct tep_event *tep, struct tep_record *record,
+			      int cpu, void *context);
tools/tracing/rtla/src/utils.c new file +433 -0

433 lines not on this page · raw message

tools/tracing/rtla/src/utils.h new file +56 -0

56 lines not on this page · raw message

Signature · 1 line
2.31.1
#3 lore raw

Re: [PATCH V8 02/14] rtla: Helper functions for rtla

tools/tracing/rtla/src/trace.c line 25

Quoted 13 lines · Daniel Bristot de Oliveira
tools/tracing/rtla/src/trace.c new file +13 -0
+/*
+ * enable_tracer_by_name - enable a tracer on the given instance
+ */
+int enable_tracer_by_name(struct tracefs_instance *inst, const char *tracer)
+{ 
+	enum tracefs_tracers t;
+	int retval;
+
+	t = TRACEFS_TRACER_CUSTOM;
+
+	debug_msg("enabling %s tracer\n", tracer);
+
+	retval = tracefs_tracer_set(inst, t, tracer);

The comment of parameters used for tracefs_tracer_set() says:

'
* tracefs_set_tracer - function to set the tracer
* @instance: ftrace instance, can be NULL for top tracing instance
* @tracer: The tracer enum that defines the tracer to be set
* @t: A tracer name if TRACEFS_TRACER_CUSTOM is passed in for @tracer
'

tracefs_tracer_set(inst, tracer, t); @tracer is enum and @t is tracer name. this is not important.

Quoted 10 lines
tools/tracing/rtla/src/trace.c new file +10 -0
+	if (retval < 0) {
+		if (errno == ENODEV)
+			err_msg("tracer %s not found!\n", tracer);
+
+		err_msg("failed to enable the tracer %s\n", tracer);
+		return -1;
+	}
+
+	return 0;
+}
#4 lore raw

Re: [PATCH V8 02/14] rtla: Helper functions for rtla

Quoted 25 lines · Tao Zhou

tools/tracing/rtla/src/trace.c line 25

Quoted 13 lines · Daniel Bristot de Oliveira
tools/tracing/rtla/src/trace.c new file +13 -0
+/*
+ * enable_tracer_by_name - enable a tracer on the given instance
+ */
+int enable_tracer_by_name(struct tracefs_instance *inst, const char *tracer)
+{ 
+	enum tracefs_tracers t;
+	int retval;
+
+	t = TRACEFS_TRACER_CUSTOM;
+
+	debug_msg("enabling %s tracer\n", tracer);
+
+	retval = tracefs_tracer_set(inst, t, tracer);

The comment of parameters used for tracefs_tracer_set() says:

'
* tracefs_set_tracer - function to set the tracer
* @instance: ftrace instance, can be NULL for top tracing instance
* @tracer: The tracer enum that defines the tracer to be set
* @t: A tracer name if TRACEFS_TRACER_CUSTOM is passed in for @tracer
'

Quoted 3 lines

tracefs_tracer_set(inst, tracer, t); @tracer is enum and @t is tracer name. this is not important.

cosmetics.

Anyway: s/tracer/tracer_name/ ; s/t/tracer/

-- Daniel

#5 lore raw

PATCH v8 3/14 rtla: Add osnoise tool

In reply to Daniel Bristot de Oliveira

The osnoise tool is the interface for the osnoise tracer. The osnoise tool will have multiple "modes" with different outputs. At this point, no mode is included.

The osnoise.c includes the osnoise_context abstraction. It serves to read-save-change-restore the default values from tracing/osnoise/ directory. When the context is deleted, the default values are restored.

It also includes some other helper functions for managing osnoise tracer sessions.

With these bits and pieces in place, we can start adding some functionality to rtla.

Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 tools/tracing/rtla/Makefile      |    2 +
 tools/tracing/rtla/src/osnoise.c | 1013 ++++++++++++++++++++++++++++++
 tools/tracing/rtla/src/osnoise.h |   95 +++
 tools/tracing/rtla/src/rtla.c    |   10 +
 4 files changed, 1120 insertions(+)
 create mode 100644 tools/tracing/rtla/src/osnoise.c
 create mode 100644 tools/tracing/rtla/src/osnoise.h
tools/tracing/rtla/Makefile +2 -0
index d99ea2d8b87e..ba6f327e815a 100644
@@ -60,6 +60,8 @@ install:
 	$(INSTALL) -d -m 755 $(DESTDIR)$(BINDIR)
 	$(INSTALL) rtla -m 755 $(DESTDIR)$(BINDIR)
 	$(STRIP) $(DESTDIR)$(BINDIR)/rtla
+	@test ! -f $(DESTDIR)$(BINDIR)/osnoise || rm $(DESTDIR)$(BINDIR)/osnoise
+	ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/osnoise
 
 .PHONY: clean tarball
 clean:
tools/tracing/rtla/src/osnoise.c new file +1013 -0

1,013 lines not on this page · raw message

tools/tracing/rtla/src/osnoise.h new file +95 -0

95 lines not on this page · raw message

tools/tracing/rtla/src/rtla.c +10 -0
index 5ae2664ed47d..669b9750b3b3 100644
@@ -8,6 +8,8 @@
 #include <string.h>
 #include <stdio.h>
 
+#include "osnoise.h"
+
 /*
  * rtla_usage - print rtla usage
  */
@@ -22,6 +24,7 @@ static void rtla_usage(void)
 		"  usage: rtla COMMAND ...",
 		"",
 		"  commands:",
+		"     osnoise  - gives information about the operating system noise (osnoise)",
 		"",
 		NULL,
 	};
@@ -39,7 +42,14 @@ static void rtla_usage(void)
  */
 int run_command(int argc, char **argv, int start_position)
 {
+	if (strcmp(argv[start_position], "osnoise") == 0) {
+		osnoise_main(argc-start_position, &argv[start_position]);
+		goto ran;
+	}
+
 	return 0;
+ran:
+	return 1;
 }
 
 int main(int argc, char *argv[])
Signature · 1 line
2.31.1
#6 lore raw

Re: [PATCH V8 03/14] rtla: Add osnoise tool

Hi Daniel,

Quoted 4 lines · Daniel Bristot de Oliveira

+/*
+ * osnoise_restore_cpus - restore the original "osnoise/cpus"
+ *
+ * osnoise_set_cpus() saves the original data for the "osnoise/cpus"

osnoise_set_cpus() --> osnoise_restore_cpus()

Quoted 8 lines

+ * file. This function restore the original config it was previously
+ * modified.
+ */

tools/tracing/rtla/src/osnoise.c new file +5 -0
+void osnoise_restore_cpus(struct osnoise_context *context)
+{
+	int retval;
+
+	if (!context->orig_cpus)

tools/tracing/rtla/src/osnoise.c line 111

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+		return;
+
+	if (!context->curr_cpus)
+		return;
+
+	/* nothing to do? */
+	if (!strcmp(context->orig_cpus, context->curr_cpus))
+		goto out_done;
+
+	retval = write(context->cpus_fd, context->orig_cpus, strlen(context->orig_cpus));

'strlen(context->orig_cpus) + 1' for write size;

tools/tracing/rtla/src/osnoise.c line 112

Quoted 1 line
tools/tracing/rtla/src/osnoise.c new file +1 -0
+	if (retval < strlen(context->orig_cpus))

Same here. Check 'strlen(context->orig_cpus) + 1'

Quoted 85 lines
tools/tracing/rtla/src/osnoise.c new file +85 -0
+		err_msg("could not restore original osnoise cpus\n");
+
+out_done:
+	free(context->curr_cpus);
+	context->curr_cpus = NULL;
+}
+
+/*
+ * osnoise_get_runtime - return the original "osnoise/runtime_us" value
+ *
+ * It also saves the value to be restored.
+ */
+unsigned long long osnoise_get_runtime(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	long long runtime_us;
+	char *runtime_path;
+	int retval;
+
+	if (context->runtime_us != OSNOISE_TIME_INIT_VAL)
+		return context->runtime_us;
+
+	if (context->orig_runtime_us != OSNOISE_TIME_INIT_VAL)
+		return context->orig_runtime_us;
+
+	runtime_path = tracefs_get_tracing_file("osnoise/runtime_us");
+
+	context->runtime_fd = open(runtime_path, O_RDWR);
+	if (context->runtime_fd < 0)
+		goto out_err;
+
+	retval = read(context->runtime_fd, &buffer, sizeof(buffer));
+	if (retval <= 0)
+		goto out_close;
+
+	runtime_us = get_llong_from_str(buffer);
+	if (runtime_us < 0)
+		goto out_close;
+
+	tracefs_put_tracing_file(runtime_path);
+
+	context->orig_runtime_us = runtime_us;
+	return runtime_us;
+
+out_close:
+	close(context->runtime_fd);
+	context->runtime_fd = CLOSED_FD;
+out_err:
+	tracefs_put_tracing_file(runtime_path);
+	return 0;
+}
+/*
+ * osnoise_get_period - return the original "osnoise/period_us" value
+ *
+ * It also saves the value to be restored.
+ */
+unsigned long long osnoise_get_period(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	char *period_path;
+	long long period_us;
+	int retval;
+
+	if (context->period_us != OSNOISE_TIME_INIT_VAL)
+		return context->period_us;
+
+	if (context->orig_period_us != OSNOISE_TIME_INIT_VAL)
+		return context->orig_period_us;
+
+	period_path = tracefs_get_tracing_file("osnoise/period_us");
+
+	context->period_fd = open(period_path, O_RDWR);
+	if (context->period_fd < 0)
+		goto out_err;
+
+	retval = read(context->period_fd, &buffer, sizeof(buffer));
+	if (retval <= 0)
+		goto out_close;
+
+	period_us = get_llong_from_str(buffer);
+	if (period_us < 0)
+		goto out_close;
+
+	tracefs_put_tracing_file(period_path);
+

tools/tracing/rtla/src/osnoise.c line 224

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+	context->orig_period_us = period_us;
+	return period_us;
+
+out_close:
+	close(context->period_fd);
+	context->period_fd = CLOSED_FD;
+out_err:
+	tracefs_put_tracing_file(period_path);
+	return 0;
+}

osnoise_get_period() and osnoise_get_runtime() almost the same. Use macro to generate code. Some thing also not sure now. Shame

#define osnoise_get_period osnoise_get(period)
#define osnoise_get_runtime osnoise_get(runtime)

#define osnoise_get(x) \
unsigned long long osnoise_get_##x(struct osnoise_context *context) \

{              \
	char buffer[BUFF_U64_STR_SIZE];             \
	char * x##_path;             \ 
	long long x##_us;            \
	if (context->x##_us != OSNOISE_TIME_INIT_VAL)                   \
		return context->x##_us;           \
	if (context->orig_##x##_us != OSNOISE_TIME_INIT_VAL)            \
		return context->orig_##x##_us;          \
	x##_path = tracefs_get_tracing_file("osnoise/x##_us");        \
	context->x##_fd = open(x##_path, O_RDWR);               \
	if (context->x##_fd < 0)                        \
		goto out_err;                 \
	retval = read(context->x##_fd, &buffer, sizeof(buffer));        \
	if (retval <= 0)                  \
		goto out_close;               \
	x##_us = get_llong_from_str(buffer);            \
	if (x##_us < 0)                   \
		goto out_close;               \
	tracefs_put_tracing_file(x##_path);             \
	context->orig_##x##_us = x##_us;                \
	return x##_us;                    \
out_close:                            \
	close(context->x##_fd);           \
	context->x##_fd = CLOSED_FD;                    \
out_err:                              \
	tracefs_put_tracing_file(x##_path);             \
	return 0;                         \
}
Quoted 28 lines
tools/tracing/rtla/src/osnoise.c new file +28 -0
+
+static int __osnoise_write_runtime(struct osnoise_context *context,
+				   unsigned long long runtime)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	int retval;
+
+	if (context->orig_runtime_us == OSNOISE_TIME_INIT_VAL)
+		return -1;
+
+	snprintf(buffer, sizeof(buffer), "%llu\n", runtime);
+
+	retval = write(context->runtime_fd, buffer, strlen(buffer) + 1);
+	if (retval < (strlen(buffer) + 1))
+		return -1;
+
+	context->runtime_us = runtime;
+	return 0;
+}
+
+static int __osnoise_write_period(struct osnoise_context *context,
+				  unsigned long long period)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	int retval;
+
+	if (context->orig_period_us == OSNOISE_TIME_INIT_VAL)
+		return -1;

tools/tracing/rtla/src/osnoise.c line 262

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+
+	snprintf(buffer, sizeof(buffer), "%llu\n", period);
+
+	retval = write(context->period_fd, buffer, strlen(buffer) + 1);
+	if (retval < (strlen(buffer) + 1))
+		return -1;
+
+	context->period_us = period;
+	return 0;
+}

__osnoise_write_period() and __osnoise_write_runtime() share almost the same code. Macro also use in here. Not sure it is right. Shame.

#define __osnoise_write(x) \

static int __osnoise_write_##x(struct osnoise_context *context,    \
				  unsigned long long period)    \
{     \
	char buffer[BUFF_U64_STR_SIZE];     \
	int retval;     \
	if (context->orig_##x##_us == OSNOISE_TIME_INIT_VAL)   \
		return -1;    \
	snprintf(buffer, sizeof(buffer), "%llu\n", period);    \
	retval = write(context->x##_fd, buffer, strlen(buffer) + 1);   \
	if (retval < (strlen(buffer) + 1))    \
		return -1;    \
	context->x##_us = period;    \
	return 0;    \
}

#define __osnoise_write_period __osnoise_write(period) #define __osnoise_write_runtime __osnoise_write(runtime)

Quoted 15 lines
tools/tracing/rtla/src/osnoise.c new file +15 -0
+
+/*
+ * osnoise_set_runtime_period - set osnoise runtime and period
+ *
+ * Osnoise's runtime and period are related as runtime <= period.
+ * Thus, this function saves the original values, and then tries
+ * to set the runtime and period if they are != 0.
+ */
+int osnoise_set_runtime_period(struct osnoise_context *context,
+			       unsigned long long runtime,
+			       unsigned long long period)
+{
+	unsigned long long curr_runtime_us;
+	unsigned long long curr_period_us;
+	int retval;

tools/tracing/rtla/src/osnoise.c line 287

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+
+	if (!period && !runtime)
+		return 0;
+
+	curr_runtime_us = osnoise_get_runtime(context);
+	curr_period_us = osnoise_get_period(context);
+
+	/* error getting any value? */
+	if (curr_period_us == -1 || curr_runtime_us == -1)
+		return -1;

'curr_period_us' and 'curr_runtime_us' error value should be 0(OSNOISE_TIME_INIT_VAL).

Quoted 38 lines
tools/tracing/rtla/src/osnoise.c new file +38 -0
+
+	if (!period) {
+		if (runtime > curr_period_us)
+			return -1;
+		return __osnoise_write_runtime(context, runtime);
+	} else if (!runtime) {
+		if (period < curr_runtime_us)
+			return -1;
+		return __osnoise_write_period(context, period);
+	}
+
+	if (runtime > curr_period_us) {
+		retval = __osnoise_write_period(context, period);
+		if (retval)
+			return -1;
+		retval = __osnoise_write_runtime(context, runtime);
+		if (retval)
+			return -1;
+	} else {
+		retval = __osnoise_write_runtime(context, runtime);
+		if (retval)
+			return -1;
+		retval = __osnoise_write_period(context, period);
+		if (retval)
+			return -1;
+	}
+
+	return 0;
+}
+
+/*
+ * osnoise_get_timerlat_period_us - read and save the original "timerlat_period_us"
+ */
+static long long
+osnoise_get_timerlat_period_us(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	long long timerlat_period_us;

tools/tracing/rtla/src/osnoise.c line 381

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+	char *stop_path;
+	int retval;
+
+	if (context->timerlat_period_us != OSNOISE_TIME_INIT_VAL)
+		return context->timerlat_period_us;
+
+	if (context->orig_timerlat_period_us != OSNOISE_TIME_INIT_VAL)
+		return context->orig_timerlat_period_us;
+
+	stop_path = tracefs_get_tracing_file("osnoise/timerlat_period_us");

Using timerlat_period_path seems to be straightforward.

Quoted 14 lines
tools/tracing/rtla/src/osnoise.c new file +14 -0
+
+	context->timerlat_period_us_fd = open(stop_path, O_RDWR);
+	if (context->timerlat_period_us_fd < 0)
+		goto out_err;
+
+	retval = read(context->timerlat_period_us_fd, &buffer, sizeof(buffer));
+	if (retval <= 0)
+		goto out_close;
+
+	timerlat_period_us = get_llong_from_str(buffer);
+	if (timerlat_period_us < 0)
+		goto out_close;
+
+	tracefs_put_tracing_file(stop_path);

tools/tracing/rtla/src/osnoise.c line 405

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+
+	context->orig_timerlat_period_us = timerlat_period_us;
+	return timerlat_period_us;
+
+out_close:
+	close(context->timerlat_period_us_fd);
+	context->timerlat_period_us_fd = CLOSED_FD;
+out_err:
+	tracefs_put_tracing_file(stop_path);
+	return -1;

Should return 0 or OSNOISE_TIME_INIT_VAL and the return type can be unsigned long long.

Quoted 34 lines
tools/tracing/rtla/src/osnoise.c new file +34 -0
+}
+
+/*
+ * osnoise_get_stop_us - read and save the original "stop_tracing_us"
+ */
+static long long
+osnoise_get_stop_us(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	long long stop_us;
+	char *stop_path;
+	int retval;
+
+	if (context->stop_us != OSNOISE_OPTION_INIT_VAL)
+		return context->stop_us;
+
+	if (context->orig_stop_us != OSNOISE_OPTION_INIT_VAL)
+		return context->orig_stop_us;
+
+	stop_path = tracefs_get_tracing_file("osnoise/stop_tracing_us");
+
+	context->stop_us_fd = open(stop_path, O_RDWR);
+	if (context->stop_us_fd < 0)
+		goto out_err;
+
+	retval = read(context->stop_us_fd, &buffer, sizeof(buffer));
+	if (retval <= 0)
+		goto out_close;
+
+	stop_us = get_llong_from_str(buffer);
+	if (stop_us < 0)
+		goto out_close;
+
+	tracefs_put_tracing_file(stop_path);

tools/tracing/rtla/src/osnoise.c line 511

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+
+	context->orig_stop_us = stop_us;
+	return stop_us;
+
+out_close:
+	close(context->stop_us_fd);
+	context->stop_us_fd = CLOSED_FD;
+out_err:
+	tracefs_put_tracing_file(stop_path);
+	return -1;

The same.

tools/tracing/rtla/src/osnoise.c line 584

Quoted 11 lines
tools/tracing/rtla/src/osnoise.c new file +11 -0
+}
+
+/*
+ * osnoise_get_stop_total_us - read and save the original "stop_tracing_total_us"
+ */
+static long long
+osnoise_get_stop_total_us(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	long long stop_total_us;
+	char *stop_path;

Use stop_total_path to differentiate with stop_path used in osnoise_get_stop_us().

Quoted 23 lines
tools/tracing/rtla/src/osnoise.c new file +23 -0
+	int retval;
+
+	if (context->stop_total_us != OSNOISE_OPTION_INIT_VAL)
+		return context->stop_total_us;
+
+	if (context->orig_stop_total_us != OSNOISE_OPTION_INIT_VAL)
+		return context->orig_stop_total_us;
+
+	stop_path = tracefs_get_tracing_file("osnoise/stop_tracing_total_us");
+
+	context->stop_total_us_fd = open(stop_path, O_RDWR);
+	if (context->stop_total_us_fd < 0)
+		goto out_err;
+
+	retval = read(context->stop_total_us_fd, &buffer, sizeof(buffer));
+	if (retval <= 0)
+		goto out_close;
+
+	stop_total_us = get_llong_from_str(buffer);
+	if (stop_total_us < 0)
+		goto out_close;
+
+	tracefs_put_tracing_file(stop_path);

tools/tracing/rtla/src/osnoise.c line 617

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+
+	context->orig_stop_total_us = stop_total_us;
+	return stop_total_us;
+
+out_close:
+	close(context->stop_total_us_fd);
+	context->stop_total_us_fd = CLOSED_FD;
+out_err:
+	tracefs_put_tracing_file(stop_path);
+	return -1;

The same.

tools/tracing/rtla/src/osnoise.c line 690

Quoted 11 lines
tools/tracing/rtla/src/osnoise.c new file +11 -0
+}
+
+/*
+ * osnoise_get_print_stack - read and save the original "print_stack"
+ */
+static long long
+osnoise_get_print_stack(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	long long print_stack;
+	char *stop_path;

This is print_stack_path.

Quoted 23 lines
tools/tracing/rtla/src/osnoise.c new file +23 -0
+	int retval;
+
+	if (context->print_stack != OSNOISE_OPTION_INIT_VAL)
+		return context->print_stack;
+
+	if (context->orig_print_stack != OSNOISE_OPTION_INIT_VAL)
+		return context->orig_print_stack;
+
+	stop_path = tracefs_get_tracing_file("osnoise/print_stack");
+
+	context->print_stack_fd = open(stop_path, O_RDWR);
+	if (context->print_stack_fd < 0)
+		goto out_err;
+
+	retval = read(context->print_stack_fd, &buffer, sizeof(buffer));
+	if (retval <= 0)
+		goto out_close;
+
+	print_stack = get_llong_from_str(buffer);
+	if (print_stack < 0)
+		goto out_close;
+
+	tracefs_put_tracing_file(stop_path);

tools/tracing/rtla/src/osnoise.c line 723

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+
+	context->orig_print_stack = print_stack;
+	return print_stack;
+
+out_close:
+	close(context->print_stack_fd);
+	context->print_stack_fd = CLOSED_FD;
+out_err:
+	tracefs_put_tracing_file(stop_path);
+	return -1;

The same.

Quoted 28 lines
tools/tracing/rtla/src/osnoise.c new file +28 -0
+}
+/*
+ * osnoise_context_alloc - alloc an osnoise_context
+ *
+ * The osnoise context contains the information of the "osnoise/" configs.
+ * It is used to set and restore the config.
+ */
+struct osnoise_context *osnoise_context_alloc(void)
+{
+	struct osnoise_context *context;
+
+	context = calloc(1, sizeof(*context));
+	if (!context)
+		goto out_err;
+
+	context->cpus_fd 		= CLOSED_FD;
+	context->runtime_fd		= CLOSED_FD;
+	context->period_fd		= CLOSED_FD;
+	context->stop_us_fd		= CLOSED_FD;
+	context->stop_total_us_fd	= CLOSED_FD;
+	context->timerlat_period_us_fd	= CLOSED_FD;
+	context->print_stack_fd		= CLOSED_FD;
+
+	context->orig_stop_us		= OSNOISE_OPTION_INIT_VAL;
+	context->stop_us		= OSNOISE_OPTION_INIT_VAL;
+
+	context->orig_stop_total_us	= OSNOISE_OPTION_INIT_VAL;
+	context->stop_total_us		= OSNOISE_OPTION_INIT_VAL;

tools/tracing/rtla/src/osnoise.c line 862

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+
+	context->orig_print_stack	= OSNOISE_OPTION_INIT_VAL;
+	context->print_stack		= OSNOISE_OPTION_INIT_VAL;
+
+	osnoise_get_context(context);
+
+	return context;
+out_err:
+	if (context)
+		free(context);

context is NULL here, so no need the check. Just directly return NULL when 'if(!context)' is enough.

Review

Quoted 2 lines
(quoted diff) +2 -0
+	return NULL;
+}

Sorry for my slow and not complete reply.. and leave not sure here.

Thanks,
Tao

#7 lore raw

Re: [PATCH V8 03/14] rtla: Add osnoise tool

Hi Tao

Quoted 9 lines · Tao Zhou

Hi Daniel,

Quoted 4 lines · Daniel Bristot de Oliveira

+/*
+ * osnoise_restore_cpus - restore the original "osnoise/cpus"
+ *
+ * osnoise_set_cpus() saves the original data for the "osnoise/cpus"

osnoise_set_cpus() --> osnoise_restore_cpus()

This comment is correct... osnoise_set_cpus saves the data (in the tool), osnoise_restore_cpus() restores it.

Quoted 21 lines
Quoted 8 lines

+ * file. This function restore the original config it was previously
+ * modified.
+ */

tools/tracing/rtla/src/osnoise.c new file +5 -0
+void osnoise_restore_cpus(struct osnoise_context *context)
+{
+	int retval;
+
+	if (!context->orig_cpus)

tools/tracing/rtla/src/osnoise.c line 111

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+		return;
+
+	if (!context->curr_cpus)
+		return;
+
+	/* nothing to do? */
+	if (!strcmp(context->orig_cpus, context->curr_cpus))
+		goto out_done;
+
+	retval = write(context->cpus_fd, context->orig_cpus, strlen(context->orig_cpus));

'strlen(context->orig_cpus) + 1' for write size;

Quoted 3 lines

tools/tracing/rtla/src/osnoise.c line 112

Quoted 1 line
tools/tracing/rtla/src/osnoise.c new file +1 -0
+	if (retval < strlen(context->orig_cpus))

Same here. Check 'strlen(context->orig_cpus) + 1'

Fixed in v9.

Quoted 125 lines
Quoted 85 lines
tools/tracing/rtla/src/osnoise.c new file +85 -0
+		err_msg("could not restore original osnoise cpus\n");
+
+out_done:
+	free(context->curr_cpus);
+	context->curr_cpus = NULL;
+}
+
+/*
+ * osnoise_get_runtime - return the original "osnoise/runtime_us" value
+ *
+ * It also saves the value to be restored.
+ */
+unsigned long long osnoise_get_runtime(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	long long runtime_us;
+	char *runtime_path;
+	int retval;
+
+	if (context->runtime_us != OSNOISE_TIME_INIT_VAL)
+		return context->runtime_us;
+
+	if (context->orig_runtime_us != OSNOISE_TIME_INIT_VAL)
+		return context->orig_runtime_us;
+
+	runtime_path = tracefs_get_tracing_file("osnoise/runtime_us");
+
+	context->runtime_fd = open(runtime_path, O_RDWR);
+	if (context->runtime_fd < 0)
+		goto out_err;
+
+	retval = read(context->runtime_fd, &buffer, sizeof(buffer));
+	if (retval <= 0)
+		goto out_close;
+
+	runtime_us = get_llong_from_str(buffer);
+	if (runtime_us < 0)
+		goto out_close;
+
+	tracefs_put_tracing_file(runtime_path);
+
+	context->orig_runtime_us = runtime_us;
+	return runtime_us;
+
+out_close:
+	close(context->runtime_fd);
+	context->runtime_fd = CLOSED_FD;
+out_err:
+	tracefs_put_tracing_file(runtime_path);
+	return 0;
+}
+/*
+ * osnoise_get_period - return the original "osnoise/period_us" value
+ *
+ * It also saves the value to be restored.
+ */
+unsigned long long osnoise_get_period(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	char *period_path;
+	long long period_us;
+	int retval;
+
+	if (context->period_us != OSNOISE_TIME_INIT_VAL)
+		return context->period_us;
+
+	if (context->orig_period_us != OSNOISE_TIME_INIT_VAL)
+		return context->orig_period_us;
+
+	period_path = tracefs_get_tracing_file("osnoise/period_us");
+
+	context->period_fd = open(period_path, O_RDWR);
+	if (context->period_fd < 0)
+		goto out_err;
+
+	retval = read(context->period_fd, &buffer, sizeof(buffer));
+	if (retval <= 0)
+		goto out_close;
+
+	period_us = get_llong_from_str(buffer);
+	if (period_us < 0)
+		goto out_close;
+
+	tracefs_put_tracing_file(period_path);
+

tools/tracing/rtla/src/osnoise.c line 224

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+	context->orig_period_us = period_us;
+	return period_us;
+
+out_close:
+	close(context->period_fd);
+	context->period_fd = CLOSED_FD;
+out_err:
+	tracefs_put_tracing_file(period_path);
+	return 0;
+}

osnoise_get_period() and osnoise_get_runtime() almost the same. Use macro to generate code. Some thing also not sure now. Shame

#define osnoise_get_period osnoise_get(period)
#define osnoise_get_runtime osnoise_get(runtime)

#define osnoise_get(x) \
unsigned long long osnoise_get_##x(struct osnoise_context *context) \

{              \
	char buffer[BUFF_U64_STR_SIZE];             \
	char * x##_path;             \ 
	long long x##_us;            \
	if (context->x##_us != OSNOISE_TIME_INIT_VAL)                   \
		return context->x##_us;           \
	if (context->orig_##x##_us != OSNOISE_TIME_INIT_VAL)            \
		return context->orig_##x##_us;          \
	x##_path = tracefs_get_tracing_file("osnoise/x##_us");        \
	context->x##_fd = open(x##_path, O_RDWR);               \
	if (context->x##_fd < 0)                        \
		goto out_err;                 \
	retval = read(context->x##_fd, &buffer, sizeof(buffer));        \
	if (retval <= 0)                  \
		goto out_close;               \
	x##_us = get_llong_from_str(buffer);            \
	if (x##_us < 0)                   \
		goto out_close;               \
	tracefs_put_tracing_file(x##_path);             \
	context->orig_##x##_us = x##_us;                \
Quoted 8 lines
	return x##_us;                    \
out_close:                            \
	close(context->x##_fd);           \
	context->x##_fd = CLOSED_FD;                    \
out_err:                              \
	tracefs_put_tracing_file(x##_path);             \
	return 0;                         \
}

I am not sure if it is worth to trade the readability for just two functions. I will keep this as is foe now, and think about it in a second moment.

[...]

Quoted 20 lines
Quoted 20 lines
tools/tracing/rtla/src/osnoise.c new file +20 -0
+
+/*
+ * osnoise_set_runtime_period - set osnoise runtime and period
+ *
+ * Osnoise's runtime and period are related as runtime <= period.
+ * Thus, this function saves the original values, and then tries
+ * to set the runtime and period if they are != 0.
+ */
+int osnoise_set_runtime_period(struct osnoise_context *context,
+			       unsigned long long runtime,
+			       unsigned long long period)
+{
+	unsigned long long curr_runtime_us;
+	unsigned long long curr_period_us;
+	int retval;
+
+	if (!period && !runtime)
+		return 0;
+
+	curr_runtime_us = osnoise_get_runtime(context);
Quoted 9 lines

tools/tracing/rtla/src/osnoise.c line 287

Quoted 5 lines
tools/tracing/rtla/src/osnoise.c new file +5 -0
+	curr_period_us = osnoise_get_period(context);
+
+	/* error getting any value? */
+	if (curr_period_us == -1 || curr_runtime_us == -1)
+		return -1;

'curr_period_us' and 'curr_runtime_us' error value should be 0(OSNOISE_TIME_INIT_VAL).

Right, I am now (in v9) returning the *_INIT_VAL on all errors, and using the macro to check for errors.

[...]

Quoted 9 lines
Quoted 9 lines
tools/tracing/rtla/src/osnoise.c new file +9 -0
+static long long
+osnoise_get_timerlat_period_us(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	long long timerlat_period_us;
+	char *stop_path;
+	int retval;
+
+	if (context->timerlat_period_us != OSNOISE_TIME_INIT_VAL)
Quoted 9 lines

tools/tracing/rtla/src/osnoise.c line 381

Quoted 6 lines
tools/tracing/rtla/src/osnoise.c new file +6 -0
+		return context->timerlat_period_us;
+
+	if (context->orig_timerlat_period_us != OSNOISE_TIME_INIT_VAL)
+		return context->orig_timerlat_period_us;
+
+	stop_path = tracefs_get_tracing_file("osnoise/timerlat_period_us");

Using timerlat_period_path seems to be straightforward.

I am using config_path for all variables like this.

[...]

Quoted 41 lines
Quoted 27 lines
tools/tracing/rtla/src/osnoise.c new file +27 -0
+/*
+ * osnoise_context_alloc - alloc an osnoise_context
+ *
+ * The osnoise context contains the information of the "osnoise/" configs.
+ * It is used to set and restore the config.
+ */
+struct osnoise_context *osnoise_context_alloc(void)
+{
+	struct osnoise_context *context;
+
+	context = calloc(1, sizeof(*context));
+	if (!context)
+		goto out_err;
+
+	context->cpus_fd 		= CLOSED_FD;
+	context->runtime_fd		= CLOSED_FD;
+	context->period_fd		= CLOSED_FD;
+	context->stop_us_fd		= CLOSED_FD;
+	context->stop_total_us_fd	= CLOSED_FD;
+	context->timerlat_period_us_fd	= CLOSED_FD;
+	context->print_stack_fd		= CLOSED_FD;
+
+	context->orig_stop_us		= OSNOISE_OPTION_INIT_VAL;
+	context->stop_us		= OSNOISE_OPTION_INIT_VAL;
+
+	context->orig_stop_total_us	= OSNOISE_OPTION_INIT_VAL;
+	context->stop_total_us		= OSNOISE_OPTION_INIT_VAL;

tools/tracing/rtla/src/osnoise.c line 862

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+
+	context->orig_print_stack	= OSNOISE_OPTION_INIT_VAL;
+	context->print_stack		= OSNOISE_OPTION_INIT_VAL;
+
+	osnoise_get_context(context);
+
+	return context;
+out_err:
+	if (context)
+		free(context);

context is NULL here, so no need the check. Just directly return NULL when 'if(!context)' is enough.

Quoted 2 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+	return NULL;
+}

In v9, I am removing the goto, returning NULL if (!context).

Quoted 4 lines

Sorry for my slow and not complete reply.. and leave not sure here.

Thanks,
Tao

Thanks Tao
-- Daniel

maintainer
#8 lore raw

Re: [PATCH V8 03/14] rtla: Add osnoise tool

Quoted 7 lines · On Thu, 2 Dec 2021 16:18:53 +0100 Daniel Bristot de Oliveira…
Quoted 7 lines

tools/tracing/rtla/src/osnoise.c line 111

Quoted 4 lines
tools/tracing/rtla/src/osnoise.c new file +4 -0
+	if (!strcmp(context->orig_cpus, context->curr_cpus))
+		goto out_done;
+
+	retval = write(context->cpus_fd, context->orig_cpus, strlen(context->orig_cpus));  

'strlen(context->orig_cpus) + 1' for write size;

Quoted 5 lines
Quoted 3 lines

tools/tracing/rtla/src/osnoise.c line 112

Quoted 1 line
tools/tracing/rtla/src/osnoise.c new file +1 -0
+	if (retval < strlen(context->orig_cpus))  

Same here. Check 'strlen(context->orig_cpus) + 1'

Fixed in v9.

And if you used the tracefs_instance_file_write() function, you would not have had his bug ;-)

-- Steve

maintainer
#9 lore raw

Re: [PATCH V8 03/14] rtla: Add osnoise tool

In reply to Daniel Bristot de Oliveira
Quoted 93 lines · On Mon, 29 Nov 2021 12:07:41 +0100 Daniel Bristot de Oliveira…

The osnoise tool is the interface for the osnoise tracer. The osnoise tool will have multiple "modes" with different outputs. At this point, no mode is included.

The osnoise.c includes the osnoise_context abstraction. It serves to read-save-change-restore the default values from tracing/osnoise/ directory. When the context is deleted, the default values are restored.

It also includes some other helper functions for managing osnoise tracer sessions.

With these bits and pieces in place, we can start adding some functionality to rtla.

Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 tools/tracing/rtla/Makefile      |    2 +
 tools/tracing/rtla/src/osnoise.c | 1013 ++++++++++++++++++++++++++++++
 tools/tracing/rtla/src/osnoise.h |   95 +++
 tools/tracing/rtla/src/rtla.c    |   10 +
 4 files changed, 1120 insertions(+)
 create mode 100644 tools/tracing/rtla/src/osnoise.c
 create mode 100644 tools/tracing/rtla/src/osnoise.h
tools/tracing/rtla/Makefile +2 -0
index d99ea2d8b87e..ba6f327e815a 100644
@@ -60,6 +60,8 @@ install:
 	$(INSTALL) -d -m 755 $(DESTDIR)$(BINDIR)
 	$(INSTALL) rtla -m 755 $(DESTDIR)$(BINDIR)
 	$(STRIP) $(DESTDIR)$(BINDIR)/rtla
+	@test ! -f $(DESTDIR)$(BINDIR)/osnoise || rm $(DESTDIR)$(BINDIR)/osnoise
+	ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/osnoise
 
 .PHONY: clean tarball
 clean:
tools/tracing/rtla/src/osnoise.c new file +36 -0
new file mode 100644
index 000000000000..7ef686dddc09
@@ -0,0 +1,1013 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
+ */
+
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <pthread.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <stdio.h>
+
+#include "osnoise.h"
+#include "utils.h"
+
+/*
+ * osnoise_get_cpus - return the original "osnoise/cpus" content
+ *
+ * It also saves the value to be restored.
+ */
+char *osnoise_get_cpus(struct osnoise_context *context)
+{
+	char buffer[1024];
+	char *cpus_path;
+	int retval;
+
+	if (context->curr_cpus)
+		return context->curr_cpus;
+
+	if (context->orig_cpus)
+		return context->orig_cpus;
+
+	cpus_path = tracefs_get_tracing_file("osnoise/cpus");

tools/tracing/rtla/src/osnoise.c line 46

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+
+	context->cpus_fd = open(cpus_path, O_RDWR);
+	if (context->cpus_fd < 0)
+		goto out_err;
+
+	retval = read(context->cpus_fd, &buffer, sizeof(buffer));
+	if (retval <= 0)
+		goto out_close;
+
+	context->orig_cpus = strdup(buffer);

Or you could have done:

	context->orig_cpus = tracefs_instance_read(NULL, "osnoise/cpus");

as I doubt that reading and writing the cpus file you really care about keeping around the file descriptor. It's not something likely to be done where you care about "disrupting" the system. But if you really do care:

	context->cpus_fd = tracefs_instance_file_open(NULL, "osnoise/cpus",
				O_RDWR);
Quoted 24 lines
tools/tracing/rtla/src/osnoise.c new file +24 -0
+	if (!context->orig_cpus)
+		goto out_close;
+
+	tracefs_put_tracing_file(cpus_path);
+
+	return context->orig_cpus;
+
+out_close:
+	close(context->cpus_fd);
+out_err:
+	tracefs_put_tracing_file(cpus_path);
+	return NULL;
+}
+
+/*
+ * osnoise_set_cpus - configure osnoise to run on *cpus
+ *
+ * "osnoise/cpus" file is used to set the cpus in which osnoise/timerlat
+ * will run. This function opens this file, saves the current value,
+ * and set the cpus passed as argument.
+ */
+int osnoise_set_cpus(struct osnoise_context *context, char *cpus)
+{
+	char *orig_cpus = osnoise_get_cpus(context);

tools/tracing/rtla/src/osnoise.c line 80

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+	int retval;
+
+	if (!orig_cpus)
+		return -1;
+
+	context->curr_cpus = strdup(cpus);
+	if (!context->curr_cpus)
+		return -1;
+
+	retval = write(context->cpus_fd, cpus, strlen(cpus) + 1);
	retval = tracefs_instance_file_write(NULL, "osnoise/cpus", cpus);
Quoted 21 lines
tools/tracing/rtla/src/osnoise.c new file +21 -0
+	if (retval < (strlen(cpus) + 1)) {
+		free(context->curr_cpus);
+		context->curr_cpus = NULL;
+		return -1;
+	}
+
+	return 0;
+}
+
+/*
+ * osnoise_restore_cpus - restore the original "osnoise/cpus"
+ *
+ * osnoise_set_cpus() saves the original data for the "osnoise/cpus"
+ * file. This function restore the original config it was previously
+ * modified.
+ */
+void osnoise_restore_cpus(struct osnoise_context *context)
+{
+	int retval;
+
+	if (!context->orig_cpus)

tools/tracing/rtla/src/osnoise.c line 111

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+		return;
+
+	if (!context->curr_cpus)
+		return;
+
+	/* nothing to do? */
+	if (!strcmp(context->orig_cpus, context->curr_cpus))
+		goto out_done;
+
+	retval = write(context->cpus_fd, context->orig_cpus, strlen(context->orig_cpus));
	retval = tracefs_instance_file_write(NULL, "osnoise/cpus", context->orig_cpus);
Quoted 42 lines
tools/tracing/rtla/src/osnoise.c new file +42 -0
+	if (retval < strlen(context->orig_cpus))
+		err_msg("could not restore original osnoise cpus\n");
+
+out_done:
+	free(context->curr_cpus);
+	context->curr_cpus = NULL;
+}
+
+/*
+ * osnoise_put_cpus - restore cpus config and cleanup data
+ */
+void osnoise_put_cpus(struct osnoise_context *context)
+{
+	osnoise_restore_cpus(context);
+
+	if (!context->orig_cpus)
+		return;
+
+	free(context->orig_cpus);
+	context->orig_cpus = NULL;
+	close(context->cpus_fd);
+	context->cpus_fd = CLOSED_FD;
+}
+
+/*
+ * osnoise_get_runtime - return the original "osnoise/runtime_us" value
+ *
+ * It also saves the value to be restored.
+ */
+unsigned long long osnoise_get_runtime(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	long long runtime_us;
+	char *runtime_path;
+	int retval;
+
+	if (context->runtime_us != OSNOISE_TIME_INIT_VAL)
+		return context->runtime_us;
+
+	if (context->orig_runtime_us != OSNOISE_TIME_INIT_VAL)
+		return context->orig_runtime_us;
+

tools/tracing/rtla/src/osnoise.c line 163

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+	runtime_path = tracefs_get_tracing_file("osnoise/runtime_us");
+
+	context->runtime_fd = open(runtime_path, O_RDWR);
+	if (context->runtime_fd < 0)
+		goto out_err;
+
+	retval = read(context->runtime_fd, &buffer, sizeof(buffer));
+	if (retval <= 0)
+		goto out_close;
+
	char *buf;

	buf = tracefs_instance_file_read(NULL, "osnoise/runtime_us");

tools/tracing/rtla/src/osnoise.c line 174

Quoted 11 lines
tools/tracing/rtla/src/osnoise.c new file +11 -0
+	runtime_us = get_llong_from_str(buffer);
+	if (runtime_us < 0)
+		goto out_close;
+
+	tracefs_put_tracing_file(runtime_path);
+
+	context->orig_runtime_us = runtime_us;
+	return runtime_us;
+
+out_close:
+	close(context->runtime_fd);
	free(buf);
Quoted 16 lines
tools/tracing/rtla/src/osnoise.c new file +16 -0
+	context->runtime_fd = CLOSED_FD;
+out_err:
+	tracefs_put_tracing_file(runtime_path);
+	return 0;
+}
+
+/*
+ * osnoise_get_period - return the original "osnoise/period_us" value
+ *
+ * It also saves the value to be restored.
+ */
+unsigned long long osnoise_get_period(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	char *period_path;
+	long long period_us;

tools/tracing/rtla/src/osnoise.c line 200

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+	int retval;
+
+	if (context->period_us != OSNOISE_TIME_INIT_VAL)
+		return context->period_us;
+
+	if (context->orig_period_us != OSNOISE_TIME_INIT_VAL)
+		return context->orig_period_us;
+
+	period_path = tracefs_get_tracing_file("osnoise/period_us");
+

Here too.

Quoted 27 lines
tools/tracing/rtla/src/osnoise.c new file +27 -0
+	context->period_fd = open(period_path, O_RDWR);
+	if (context->period_fd < 0)
+		goto out_err;
+
+	retval = read(context->period_fd, &buffer, sizeof(buffer));
+	if (retval <= 0)
+		goto out_close;
+
+	period_us = get_llong_from_str(buffer);
+	if (period_us < 0)
+		goto out_close;
+
+	tracefs_put_tracing_file(period_path);
+
+	context->orig_period_us = period_us;
+	return period_us;
+
+out_close:
+	close(context->period_fd);
+	context->period_fd = CLOSED_FD;
+out_err:
+	tracefs_put_tracing_file(period_path);
+	return 0;
+}
+
+static int __osnoise_write_runtime(struct osnoise_context *context,
+				   unsigned long long runtime)

tools/tracing/rtla/src/osnoise.c line 237

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	int retval;
+
+	if (context->orig_runtime_us == OSNOISE_TIME_INIT_VAL)
+		return -1;
+
+	snprintf(buffer, sizeof(buffer), "%llu\n", runtime);
+
+	retval = write(context->runtime_fd, buffer, strlen(buffer) + 1);

Again, how important is it to have the fd?

Quoted 9 lines
tools/tracing/rtla/src/osnoise.c new file +9 -0
+	if (retval < (strlen(buffer) + 1))
+		return -1;
+
+	context->runtime_us = runtime;
+	return 0;
+}
+
+static int __osnoise_write_period(struct osnoise_context *context,
+				  unsigned long long period)

tools/tracing/rtla/src/osnoise.c line 256

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	int retval;
+
+	if (context->orig_period_us == OSNOISE_TIME_INIT_VAL)
+		return -1;
+
+	snprintf(buffer, sizeof(buffer), "%llu\n", period);
+
+	retval = write(context->period_fd, buffer, strlen(buffer) + 1);

Here too?

Quoted 121 lines
tools/tracing/rtla/src/osnoise.c new file +121 -0
+	if (retval < (strlen(buffer) + 1))
+		return -1;
+
+	context->period_us = period;
+	return 0;
+}
+
+/*
+ * osnoise_set_runtime_period - set osnoise runtime and period
+ *
+ * Osnoise's runtime and period are related as runtime <= period.
+ * Thus, this function saves the original values, and then tries
+ * to set the runtime and period if they are != 0.
+ */
+int osnoise_set_runtime_period(struct osnoise_context *context,
+			       unsigned long long runtime,
+			       unsigned long long period)
+{
+	unsigned long long curr_runtime_us;
+	unsigned long long curr_period_us;
+	int retval;
+
+	if (!period && !runtime)
+		return 0;
+
+	curr_runtime_us = osnoise_get_runtime(context);
+	curr_period_us = osnoise_get_period(context);
+
+	/* error getting any value? */
+	if (curr_period_us == -1 || curr_runtime_us == -1)
+		return -1;
+
+	if (!period) {
+		if (runtime > curr_period_us)
+			return -1;
+		return __osnoise_write_runtime(context, runtime);
+	} else if (!runtime) {
+		if (period < curr_runtime_us)
+			return -1;
+		return __osnoise_write_period(context, period);
+	}
+
+	if (runtime > curr_period_us) {
+		retval = __osnoise_write_period(context, period);
+		if (retval)
+			return -1;
+		retval = __osnoise_write_runtime(context, runtime);
+		if (retval)
+			return -1;
+	} else {
+		retval = __osnoise_write_runtime(context, runtime);
+		if (retval)
+			return -1;
+		retval = __osnoise_write_period(context, period);
+		if (retval)
+			return -1;
+	}
+
+	return 0;
+}
+
+/*
+ * osnoise_restore_runtime_period - restore the original runtime and period
+ */
+void osnoise_restore_runtime_period(struct osnoise_context *context)
+{
+	unsigned long long orig_runtime = context->orig_runtime_us;
+	unsigned long long orig_period = context->orig_period_us;
+	unsigned long long curr_runtime = context->runtime_us;
+	unsigned long long curr_period = context->period_us;
+	int retval;
+
+	if ((orig_runtime == OSNOISE_TIME_INIT_VAL) && (orig_period == OSNOISE_TIME_INIT_VAL))
+		return;
+
+	if ((orig_period == curr_period) && (orig_runtime == curr_runtime))
+		goto out_done;
+
+	retval = osnoise_set_runtime_period(context, orig_runtime, orig_period);
+	if (retval)
+		err_msg("Could not restore original osnoise runtime/period\n");
+
+out_done:
+	context->runtime_us = OSNOISE_TIME_INIT_VAL;
+	context->period_us = OSNOISE_TIME_INIT_VAL;
+}
+
+/*
+ * osnoise_put_runtime_period - restore original values and cleanup data
+ */
+void osnoise_put_runtime_period(struct osnoise_context *context)
+{
+	osnoise_restore_runtime_period(context);
+
+	if (context->orig_runtime_us != OSNOISE_TIME_INIT_VAL) {
+		close(context->runtime_fd);
+		context->runtime_fd = CLOSED_FD;
+		context->orig_runtime_us = OSNOISE_TIME_INIT_VAL;
+	}
+
+	if (context->orig_period_us != OSNOISE_TIME_INIT_VAL) {
+		close(context->period_fd);
+		context->period_fd = CLOSED_FD;
+		context->orig_period_us = OSNOISE_TIME_INIT_VAL;
+	}
+}
+
+/*
+ * osnoise_get_timerlat_period_us - read and save the original "timerlat_period_us"
+ */
+static long long
+osnoise_get_timerlat_period_us(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	long long timerlat_period_us;
+	char *stop_path;
+	int retval;
+
+	if (context->timerlat_period_us != OSNOISE_TIME_INIT_VAL)
+		return context->timerlat_period_us;
+

tools/tracing/rtla/src/osnoise.c line 387

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+	if (context->orig_timerlat_period_us != OSNOISE_TIME_INIT_VAL)
+		return context->orig_timerlat_period_us;
+
+	stop_path = tracefs_get_tracing_file("osnoise/timerlat_period_us");
+
+	context->timerlat_period_us_fd = open(stop_path, O_RDWR);
+	if (context->timerlat_period_us_fd < 0)
+		goto out_err;
+
+	retval = read(context->timerlat_period_us_fd, &buffer, sizeof(buffer));

Same here.

Quoted 25 lines
tools/tracing/rtla/src/osnoise.c new file +25 -0
+	if (retval <= 0)
+		goto out_close;
+
+	timerlat_period_us = get_llong_from_str(buffer);
+	if (timerlat_period_us < 0)
+		goto out_close;
+
+	tracefs_put_tracing_file(stop_path);
+
+	context->orig_timerlat_period_us = timerlat_period_us;
+	return timerlat_period_us;
+
+out_close:
+	close(context->timerlat_period_us_fd);
+	context->timerlat_period_us_fd = CLOSED_FD;
+out_err:
+	tracefs_put_tracing_file(stop_path);
+	return -1;
+}
+
+/*
+ * osnoise_set_timerlat_period_us - set "timerlat_period_us"
+ */
+int osnoise_set_timerlat_period_us(struct osnoise_context *context, long long timerlat_period_us)
+{

tools/tracing/rtla/src/osnoise.c line 422

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+	long long curr_timerlat_period_us = osnoise_get_timerlat_period_us(context);
+	char buffer[BUFF_U64_STR_SIZE];
+	int retval;
+
+	if (curr_timerlat_period_us == OSNOISE_TIME_INIT_VAL)
+		return -1;
+
+	snprintf(buffer, BUFF_U64_STR_SIZE, "%lld\n", timerlat_period_us);
+
+	retval = write(context->timerlat_period_us_fd, buffer, strlen(buffer) + 1);

And here.

Quoted 15 lines
tools/tracing/rtla/src/osnoise.c new file +15 -0
+	if (retval < (strlen(buffer) + 1))
+		return -1;
+
+	context->timerlat_period_us = timerlat_period_us;
+
+	return 0;
+}
+
+/*
+ * osnoise_restore_timerlat_period_us - restore "timerlat_period_us"
+ */
+void osnoise_restore_timerlat_period_us(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	int retval;

tools/tracing/rtla/src/osnoise.c line 447

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+
+	if (context->orig_timerlat_period_us == OSNOISE_TIME_INIT_VAL)
+		return;
+
+	if (context->orig_timerlat_period_us == context->timerlat_period_us)
+		goto out_done;
+
+	snprintf(buffer, BUFF_U64_STR_SIZE, "%lld\n", context->orig_timerlat_period_us);
+
+	retval = write(context->timerlat_period_us_fd, buffer, strlen(buffer) + 1);

And here.

Quoted 36 lines
tools/tracing/rtla/src/osnoise.c new file +36 -0
+	if (retval < (strlen(buffer) + 1))
+		err_msg("Could not restore original osnoise timerlat_period_us\n");
+
+out_done:
+	context->timerlat_period_us = OSNOISE_TIME_INIT_VAL;
+}
+
+/*
+ * osnoise_put_timerlat_period_us - restore original values and cleanup data
+ */
+void osnoise_put_timerlat_period_us(struct osnoise_context *context)
+{
+	osnoise_restore_timerlat_period_us(context);
+
+	if (context->orig_timerlat_period_us == OSNOISE_TIME_INIT_VAL)
+		return;
+
+	close(context->timerlat_period_us_fd);
+	context->timerlat_period_us_fd = CLOSED_FD;
+	context->orig_timerlat_period_us = OSNOISE_TIME_INIT_VAL;
+}
+
+/*
+ * osnoise_get_stop_us - read and save the original "stop_tracing_us"
+ */
+static long long
+osnoise_get_stop_us(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	long long stop_us;
+	char *stop_path;
+	int retval;
+
+	if (context->stop_us != OSNOISE_OPTION_INIT_VAL)
+		return context->stop_us;
+

tools/tracing/rtla/src/osnoise.c line 493

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+	if (context->orig_stop_us != OSNOISE_OPTION_INIT_VAL)
+		return context->orig_stop_us;
+
+	stop_path = tracefs_get_tracing_file("osnoise/stop_tracing_us");
+
+	context->stop_us_fd = open(stop_path, O_RDWR);
+	if (context->stop_us_fd < 0)
+		goto out_err;
+
+	retval = read(context->stop_us_fd, &buffer, sizeof(buffer));

And here.

Quoted 25 lines
tools/tracing/rtla/src/osnoise.c new file +25 -0
+	if (retval <= 0)
+		goto out_close;
+
+	stop_us = get_llong_from_str(buffer);
+	if (stop_us < 0)
+		goto out_close;
+
+	tracefs_put_tracing_file(stop_path);
+
+	context->orig_stop_us = stop_us;
+	return stop_us;
+
+out_close:
+	close(context->stop_us_fd);
+	context->stop_us_fd = CLOSED_FD;
+out_err:
+	tracefs_put_tracing_file(stop_path);
+	return -1;
+}
+
+/*
+ * osnoise_set_stop_us - set "stop_tracing_us"
+ */
+int osnoise_set_stop_us(struct osnoise_context *context, long long stop_us)
+{

tools/tracing/rtla/src/osnoise.c line 528

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+	long long curr_stop_us = osnoise_get_stop_us(context);
+	char buffer[BUFF_U64_STR_SIZE];
+	int retval;
+
+	if (curr_stop_us == OSNOISE_OPTION_INIT_VAL)
+		return -1;
+
+	snprintf(buffer, BUFF_U64_STR_SIZE, "%lld\n", stop_us);
+
+	retval = write(context->stop_us_fd, buffer, strlen(buffer) + 1);

And here.

Quoted 15 lines
tools/tracing/rtla/src/osnoise.c new file +15 -0
+	if (retval < (strlen(buffer) + 1))
+		return -1;
+
+	context->stop_us = stop_us;
+
+	return 0;
+}
+
+/*
+ * osnoise_restore_stop_us - restore the original "stop_tracing_us"
+ */
+void osnoise_restore_stop_us(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	int retval;

tools/tracing/rtla/src/osnoise.c line 553

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+
+	if (context->orig_stop_us == OSNOISE_OPTION_INIT_VAL)
+		return;
+
+	if (context->orig_stop_us == context->stop_us)
+		goto out_done;
+
+	snprintf(buffer, BUFF_U64_STR_SIZE, "%lld\n", context->orig_stop_us);
+
+	retval = write(context->stop_us_fd, buffer, strlen(buffer) + 1);

And here.

Quoted 36 lines
tools/tracing/rtla/src/osnoise.c new file +36 -0
+	if (retval < (strlen(buffer) + 1))
+		err_msg("Could not restore original osnoise stop_us\n");
+
+out_done:
+	context->stop_us = OSNOISE_OPTION_INIT_VAL;
+}
+
+/*
+ * osnoise_put_stop_us - restore original values and cleanup data
+ */
+void osnoise_put_stop_us(struct osnoise_context *context)
+{
+	osnoise_restore_stop_us(context);
+
+	if (context->orig_stop_us == OSNOISE_OPTION_INIT_VAL)
+		return;
+
+	close(context->stop_us_fd);
+	context->stop_us_fd = CLOSED_FD;
+	context->orig_stop_us = OSNOISE_OPTION_INIT_VAL;
+}
+
+/*
+ * osnoise_get_stop_total_us - read and save the original "stop_tracing_total_us"
+ */
+static long long
+osnoise_get_stop_total_us(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	long long stop_total_us;
+	char *stop_path;
+	int retval;
+
+	if (context->stop_total_us != OSNOISE_OPTION_INIT_VAL)
+		return context->stop_total_us;
+

tools/tracing/rtla/src/osnoise.c line 599

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+	if (context->orig_stop_total_us != OSNOISE_OPTION_INIT_VAL)
+		return context->orig_stop_total_us;
+
+	stop_path = tracefs_get_tracing_file("osnoise/stop_tracing_total_us");
+
+	context->stop_total_us_fd = open(stop_path, O_RDWR);
+	if (context->stop_total_us_fd < 0)
+		goto out_err;
+
+	retval = read(context->stop_total_us_fd, &buffer, sizeof(buffer));

And here.

Quoted 25 lines
tools/tracing/rtla/src/osnoise.c new file +25 -0
+	if (retval <= 0)
+		goto out_close;
+
+	stop_total_us = get_llong_from_str(buffer);
+	if (stop_total_us < 0)
+		goto out_close;
+
+	tracefs_put_tracing_file(stop_path);
+
+	context->orig_stop_total_us = stop_total_us;
+	return stop_total_us;
+
+out_close:
+	close(context->stop_total_us_fd);
+	context->stop_total_us_fd = CLOSED_FD;
+out_err:
+	tracefs_put_tracing_file(stop_path);
+	return -1;
+}
+
+/*
+ * osnoise_set_stop_total_us - set "stop_tracing_total_us"
+ */
+int osnoise_set_stop_total_us(struct osnoise_context *context, long long stop_total_us)
+{

tools/tracing/rtla/src/osnoise.c line 634

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+	long long curr_stop_total_us = osnoise_get_stop_total_us(context);
+	char buffer[BUFF_U64_STR_SIZE];
+	int retval;
+
+	if (curr_stop_total_us == OSNOISE_OPTION_INIT_VAL)
+		return -1;
+
+	snprintf(buffer, BUFF_U64_STR_SIZE, "%lld\n", stop_total_us);
+
+	retval = write(context->stop_total_us_fd, buffer, strlen(buffer) + 1);

And here.

Hmm, we should add a helper:

tracefs_instance_file_printf(instance, fmt, ...)

Where the above could be:

	tracefs_instance_file_printf(NULL, "%lld\n", stop_total_us);

Of course, for now, you could just add a helper function that does that.

Quoted 15 lines
tools/tracing/rtla/src/osnoise.c new file +15 -0
+	if (retval < (strlen(buffer) + 1))
+		return -1;
+
+	context->stop_total_us = stop_total_us;
+
+	return 0;
+}
+
+/*
+ * osnoise_restore_stop_total_us - restore the original "stop_tracing_total_us"
+ */
+void osnoise_restore_stop_total_us(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	int retval;

tools/tracing/rtla/src/osnoise.c line 659

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+
+	if (context->orig_stop_total_us == OSNOISE_OPTION_INIT_VAL)
+		return;
+
+	if (context->orig_stop_total_us == context->stop_total_us)
+		goto out_done;
+
+	snprintf(buffer, BUFF_U64_STR_SIZE, "%lld\n", context->orig_stop_total_us);
+
+	retval = write(context->stop_total_us_fd, buffer, strlen(buffer) + 1);

And here.

Quoted 38 lines
tools/tracing/rtla/src/osnoise.c new file +38 -0
+	if (retval < (strlen(buffer) + 1))
+		err_msg("Could not restore original osnoise stop_total_us\n");
+
+out_done:
+	context->stop_total_us = OSNOISE_OPTION_INIT_VAL;
+}
+
+/*
+ * osnoise_put_stop_total_us - restore original values and cleanup data
+ */
+void osnoise_put_stop_total_us(struct osnoise_context *context)
+{
+	osnoise_restore_stop_total_us(context);
+
+	if (context->orig_stop_total_us == OSNOISE_OPTION_INIT_VAL)
+		return;
+
+	close(context->stop_total_us_fd);
+	context->stop_total_us_fd = CLOSED_FD;
+	context->orig_stop_total_us = OSNOISE_OPTION_INIT_VAL;
+}
+
+/*
+ * osnoise_get_print_stack - read and save the original "print_stack"
+ */
+static long long
+osnoise_get_print_stack(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	long long print_stack;
+	char *stop_path;
+	int retval;
+
+	if (context->print_stack != OSNOISE_OPTION_INIT_VAL)
+		return context->print_stack;
+
+	if (context->orig_print_stack != OSNOISE_OPTION_INIT_VAL)
+		return context->orig_print_stack;

tools/tracing/rtla/src/osnoise.c line 707

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+
+	stop_path = tracefs_get_tracing_file("osnoise/print_stack");
+
+	context->print_stack_fd = open(stop_path, O_RDWR);
+	if (context->print_stack_fd < 0)
+		goto out_err;
+
+	retval = read(context->print_stack_fd, &buffer, sizeof(buffer));
+	if (retval <= 0)
+		goto out_close;

And here.

Quoted 23 lines
tools/tracing/rtla/src/osnoise.c new file +23 -0
+
+	print_stack = get_llong_from_str(buffer);
+	if (print_stack < 0)
+		goto out_close;
+
+	tracefs_put_tracing_file(stop_path);
+
+	context->orig_print_stack = print_stack;
+	return print_stack;
+
+out_close:
+	close(context->print_stack_fd);
+	context->print_stack_fd = CLOSED_FD;
+out_err:
+	tracefs_put_tracing_file(stop_path);
+	return -1;
+}
+
+/*
+ * osnoise_set_print_stack - set "print_stack"
+ */
+int osnoise_set_print_stack(struct osnoise_context *context, long long print_stack)
+{

tools/tracing/rtla/src/osnoise.c line 740

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+	long long curr_print_stack = osnoise_get_print_stack(context);
+	char buffer[BUFF_U64_STR_SIZE];
+	int retval;
+
+	if (curr_print_stack == OSNOISE_OPTION_INIT_VAL)
+		return -1;
+
+	snprintf(buffer, BUFF_U64_STR_SIZE, "%lld\n", print_stack);
+
+	retval = write(context->print_stack_fd, buffer, strlen(buffer) + 1);

And here.

Quoted 15 lines
tools/tracing/rtla/src/osnoise.c new file +15 -0
+	if (retval < (strlen(buffer) + 1))
+		return -1;
+
+	context->print_stack = print_stack;
+
+	return 0;
+}
+
+/*
+ * osnoise_restore_print_stack - restore the original "print_stack"
+ */
+void osnoise_restore_print_stack(struct osnoise_context *context)
+{
+	char buffer[BUFF_U64_STR_SIZE];
+	int retval;

tools/tracing/rtla/src/osnoise.c line 765

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+
+	if (context->orig_print_stack == OSNOISE_OPTION_INIT_VAL)
+		return;
+
+	if (context->orig_print_stack == context->print_stack)
+		goto out_done;
+
+	snprintf(buffer, BUFF_U64_STR_SIZE, "%lld\n", context->orig_print_stack);
+
+	retval = write(context->print_stack_fd, buffer, strlen(buffer) + 1);

And here.

Quoted 71 lines
tools/tracing/rtla/src/osnoise.c new file +71 -0
+	if (retval < (strlen(buffer) + 1))
+		err_msg("Could not restore original osnoise print_stack\n");
+
+out_done:
+	context->print_stack = OSNOISE_OPTION_INIT_VAL;
+}
+
+/*
+ * osnoise_put_print_stack - restore original values and cleanup data
+ */
+void osnoise_put_print_stack(struct osnoise_context *context)
+{
+	osnoise_restore_print_stack(context);
+
+	if (context->orig_print_stack == OSNOISE_OPTION_INIT_VAL)
+		return;
+
+	close(context->print_stack_fd);
+	context->print_stack_fd = CLOSED_FD;
+	context->orig_print_stack = OSNOISE_OPTION_INIT_VAL;
+}
+
+/*
+ * enable_osnoise - enable osnoise tracer in the trace_instance
+ */
+int enable_osnoise(struct trace_instance *trace)
+{
+	return enable_tracer_by_name(trace->inst, "osnoise");
+}
+
+/*
+ * enable_timerlat - enable timerlat tracer in the trace_instance
+ */
+int enable_timerlat(struct trace_instance *trace)
+{
+	return enable_tracer_by_name(trace->inst, "timerlat");
+}
+
+enum {
+	FLAG_CONTEXT_NEWLY_CREATED	= (1 << 0),
+	FLAG_CONTEXT_DELETED		= (1 << 1),
+};
+
+/*
+ * osnoise_get_context - increase the usage of a context and return it
+ */
+int osnoise_get_context(struct osnoise_context *context)
+{
+	int ret;
+
+	if (context->flags & FLAG_CONTEXT_DELETED) {
+		ret = -1;
+	} else {
+		context->ref++;
+		ret = 0;
+	}
+
+	return ret;
+}
+
+/*
+ * osnoise_context_alloc - alloc an osnoise_context
+ *
+ * The osnoise context contains the information of the "osnoise/" configs.
+ * It is used to set and restore the config.
+ */
+struct osnoise_context *osnoise_context_alloc(void)
+{
+	struct osnoise_context *context;
+
+	context = calloc(1, sizeof(*context));

tools/tracing/rtla/src/osnoise.c line 846

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+	if (!context)
+		goto out_err;
+
+	context->cpus_fd 		= CLOSED_FD;
+	context->runtime_fd		= CLOSED_FD;
+	context->period_fd		= CLOSED_FD;
+	context->stop_us_fd		= CLOSED_FD;
+	context->stop_total_us_fd	= CLOSED_FD;
+	context->timerlat_period_us_fd	= CLOSED_FD;
+	context->print_stack_fd		= CLOSED_FD;

You could save a lot of code by using the tracefs_instance_file_*() helpers. And do you really need to keep around the file descriptors?

-- Steve

Quoted 304 lines
tools/tracing/rtla/src/osnoise.c new file +167 -0
+
+	context->orig_stop_us		= OSNOISE_OPTION_INIT_VAL;
+	context->stop_us		= OSNOISE_OPTION_INIT_VAL;
+
+	context->orig_stop_total_us	= OSNOISE_OPTION_INIT_VAL;
+	context->stop_total_us		= OSNOISE_OPTION_INIT_VAL;
+
+	context->orig_print_stack	= OSNOISE_OPTION_INIT_VAL;
+	context->print_stack		= OSNOISE_OPTION_INIT_VAL;
+
+	osnoise_get_context(context);
+
+	return context;
+out_err:
+	if (context)
+		free(context);
+	return NULL;
+}
+
+/*
+ * osnoise_put_context - put the osnoise_put_context
+ *
+ * If there is no other user for the context, the original data
+ * is restored.
+ */
+void osnoise_put_context(struct osnoise_context *context)
+{
+	if (--context->ref < 1)
+		context->flags |= FLAG_CONTEXT_DELETED;
+
+	if (!(context->flags & FLAG_CONTEXT_DELETED))
+		return;
+
+	osnoise_put_cpus(context);
+	osnoise_put_runtime_period(context);
+	osnoise_put_stop_us(context);
+	osnoise_put_stop_total_us(context);
+	osnoise_put_timerlat_period_us(context);
+	osnoise_put_print_stack(context);
+
+	free(context);
+}
+
+/*
+ * osnoise_destroy_tool - disable trace, restore configs and free data
+ */
+void osnoise_destroy_tool(struct osnoise_tool *top)
+{
+	trace_instance_destroy(&top->trace);
+
+	if (top->context)
+		osnoise_put_context(top->context);
+
+	free(top);
+}
+
+/*
+ * osnoise_init_tool - init an osnoise tool
+ *
+ * It allocs data, create a context to store data and
+ * creates a new trace instance for the tool.
+ */
+struct osnoise_tool *osnoise_init_tool(char *tool_name)
+{
+	struct osnoise_tool *top;
+	int retval;
+
+	top = calloc(1, sizeof(*top));
+	if (!top)
+		return NULL;
+
+	top->context = osnoise_context_alloc();
+	if (!top->context)
+		goto out_err;
+
+	retval = trace_instance_init(&top->trace, tool_name);
+	if (retval)
+		goto out_err;
+
+	return top;
+out_err:
+	osnoise_destroy_tool(top);
+	return NULL;
+}
+
+/*
+ * osnoise_init_trace_tool - init a tracer instance to trace osnoise events
+ */
+struct osnoise_tool *osnoise_init_trace_tool(char *tracer)
+{
+	struct osnoise_tool *trace;
+	int retval;
+
+	trace = osnoise_init_tool("osnoise_trace");
+	if (!trace)
+		return NULL;
+
+	retval = tracefs_event_enable(trace->trace.inst, "osnoise", NULL);
+	if (retval < 0 && !errno) {
+		err_msg("Could not find osnoise events\n");
+		goto out_err;
+	}
+
+	retval = enable_tracer_by_name(trace->trace.inst, tracer);
+	if (retval) {
+		err_msg("Could not enable osnoiser tracer for tracing\n");
+		goto out_err;
+	}
+
+	return trace;
+out_err:
+	osnoise_destroy_tool(trace);
+	return NULL;
+}
+
+static void osnoise_usage(void)
+{
+	int i;
+
+	static const char *msg[] = {
+		"",
+		"osnoise version " VERSION,
+		"",
+		"  usage: [rtla] osnoise [MODE] ...",
+		"",
+		"  modes:",
+		"     top  - prints the summary from osnoise tracer",
+		"",
+		"if no MODE is given, the top mode is called, passing the arguments",
+		NULL,
+	};
+
+	for (i = 0; msg[i]; i++)
+		fprintf(stderr, "%s\n", msg[i]);
+	exit(1);
+}
+
+int osnoise_main(int argc, char *argv[])
+{
+	if (argc == 0)
+		goto usage;
+
+	/*
+	 * if osnoise was called without any argument, run the
+	 * default cmdline.
+	 */
+	if (argc == 1) {
+		osnoise_top_main(argc, argv);
+		exit(0);
+	}
+
+	if ((strcmp(argv[1], "-h") == 0) || (strcmp(argv[1], "--help") == 0)) {
+		osnoise_usage();
+		exit(0);
+	} else if (strncmp(argv[1], "-", 1) == 0) {
+		/* the user skipped the tool, call the default one */
+		osnoise_top_main(argc, argv);
+		exit(0);
+	} else if (strcmp(argv[1], "top") == 0) {
+		osnoise_top_main(argc-1, &argv[1]);
+		exit(0);
+	}
+
+usage:
+	osnoise_usage();
+	exit(1);
+}
tools/tracing/rtla/src/osnoise.h new file +95 -0
new file mode 100644
index 000000000000..e3e8f8359d99
@@ -0,0 +1,95 @@
+// SPDX-License-Identifier: GPL-2.0
+#include "trace.h"
+
+/*
+ * osnoise_context - read, store, write, restore osnoise configs.
+ */
+struct osnoise_context {
+	int			flags;
+	int			ref;
+
+	int			cpus_fd;
+	int			runtime_fd;
+	int			period_fd;
+	int			stop_us_fd;
+	int			stop_total_us_fd;
+	int			timerlat_period_us_fd;
+	int			print_stack_fd;
+
+	char			*curr_cpus;
+	char			*orig_cpus;
+
+	/* 0 as init value */
+	unsigned long long	orig_runtime_us;
+	unsigned long long	runtime_us;
+
+	/* 0 as init value */
+	unsigned long long	orig_period_us;
+	unsigned long long	period_us;
+
+	/* 0 as init value */
+	long long		orig_timerlat_period_us;
+	long long		timerlat_period_us;
+
+	/* -1 as init value because 0 is disabled */
+	long long		orig_stop_us;
+	long long		stop_us;
+
+	/* -1 as init value because 0 is disabled */
+	long long		orig_stop_total_us;
+	long long		stop_total_us;
+
+	/* -1 as init value because 0 is disabled */
+	long long		orig_print_stack;
+	long long		print_stack;
+};
+
+#define CLOSED_FD		(-1)
+#define OSNOISE_OPTION_INIT_VAL	(-1)
+#define OSNOISE_TIME_INIT_VAL	(0)
+
+struct osnoise_context *osnoise_context_alloc(void);
+int osnoise_get_context(struct osnoise_context *context);
+void osnoise_put_context(struct osnoise_context *context);
+
+int osnoise_set_cpus(struct osnoise_context *context, char *cpus);
+void osnoise_restore_cpus(struct osnoise_context *context);
+
+int osnoise_set_runtime_period(struct osnoise_context *context,
+			       unsigned long long runtime,
+			       unsigned long long period);
+void osnoise_restore_runtime_period(struct osnoise_context *context);
+
+int osnoise_set_stop_us(struct osnoise_context *context,
+			long long stop_us);
+void osnoise_restore_stop_us(struct osnoise_context *context);
+
+int osnoise_set_stop_total_us(struct osnoise_context *context,
+			      long long stop_total_us);
+void osnoise_restore_stop_total_us(struct osnoise_context *context);
+
+int osnoise_set_timerlat_period_us(struct osnoise_context *context,
+				   long long timerlat_period_us);
+void osnoise_restore_timerlat_period_us(struct osnoise_context *context);
+
+void osnoise_restore_print_stack(struct osnoise_context *context);
+int osnoise_set_print_stack(struct osnoise_context *context,
+			    long long print_stack);
+
+/*
+ * osnoise_tool -  osnoise based tool definition.
+ */
+struct osnoise_tool {
+	struct trace_instance		trace;
+	struct osnoise_context		*context;
+	void				*data;
+	void				*params;
+	time_t				start_time;
+};
+
+void osnoise_destroy_tool(struct osnoise_tool *top);
+struct osnoise_tool *osnoise_init_tool(char *tool_name);
+struct osnoise_tool *osnoise_init_trace_tool(char *tracer);
+
+int osnoise_top_main(int argc, char **argv);
+int osnoise_main(int argc, char **argv);
tools/tracing/rtla/src/rtla.c +10 -0
index 5ae2664ed47d..669b9750b3b3 100644
@@ -8,6 +8,8 @@
 #include <string.h>
 #include <stdio.h>
 
+#include "osnoise.h"
+
 /*
  * rtla_usage - print rtla usage
  */
@@ -22,6 +24,7 @@ static void rtla_usage(void)
 		"  usage: rtla COMMAND ...",
 		"",
 		"  commands:",
+		"     osnoise  - gives information about the operating system noise (osnoise)",
 		"",
 		NULL,
 	};
@@ -39,7 +42,14 @@ static void rtla_usage(void)
  */
 int run_command(int argc, char **argv, int start_position)
 {
+	if (strcmp(argv[start_position], "osnoise") == 0) {
+		osnoise_main(argc-start_position, &argv[start_position]);
+		goto ran;
+	}
+
 	return 0;
+ran:
+	return 1;
 }
 
 int main(int argc, char *argv[])
#10 lore raw

Re: [PATCH V8 03/14] rtla: Add osnoise tool

Quoted 110 lines · Steven Rostedt
Quoted 93 lines · On Mon, 29 Nov 2021 12:07:41 +0100 Daniel Bristot de Oliveira…

The osnoise tool is the interface for the osnoise tracer. The osnoise tool will have multiple "modes" with different outputs. At this point, no mode is included.

The osnoise.c includes the osnoise_context abstraction. It serves to read-save-change-restore the default values from tracing/osnoise/ directory. When the context is deleted, the default values are restored.

It also includes some other helper functions for managing osnoise tracer sessions.

With these bits and pieces in place, we can start adding some functionality to rtla.

Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 tools/tracing/rtla/Makefile      |    2 +
 tools/tracing/rtla/src/osnoise.c | 1013 ++++++++++++++++++++++++++++++
 tools/tracing/rtla/src/osnoise.h |   95 +++
 tools/tracing/rtla/src/rtla.c    |   10 +
 4 files changed, 1120 insertions(+)
 create mode 100644 tools/tracing/rtla/src/osnoise.c
 create mode 100644 tools/tracing/rtla/src/osnoise.h
tools/tracing/rtla/Makefile +2 -0
index d99ea2d8b87e..ba6f327e815a 100644
@@ -60,6 +60,8 @@ install:
 	$(INSTALL) -d -m 755 $(DESTDIR)$(BINDIR)
 	$(INSTALL) rtla -m 755 $(DESTDIR)$(BINDIR)
 	$(STRIP) $(DESTDIR)$(BINDIR)/rtla
+	@test ! -f $(DESTDIR)$(BINDIR)/osnoise || rm $(DESTDIR)$(BINDIR)/osnoise
+	ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/osnoise
 
 .PHONY: clean tarball
 clean:
tools/tracing/rtla/src/osnoise.c new file +36 -0
new file mode 100644
index 000000000000..7ef686dddc09
@@ -0,0 +1,1013 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
+ */
+
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <pthread.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <stdio.h>
+
+#include "osnoise.h"
+#include "utils.h"
+
+/*
+ * osnoise_get_cpus - return the original "osnoise/cpus" content
+ *
+ * It also saves the value to be restored.
+ */
+char *osnoise_get_cpus(struct osnoise_context *context)
+{
+	char buffer[1024];
+	char *cpus_path;
+	int retval;
+
+	if (context->curr_cpus)
+		return context->curr_cpus;
+
+	if (context->orig_cpus)
+		return context->orig_cpus;
+
+	cpus_path = tracefs_get_tracing_file("osnoise/cpus");

tools/tracing/rtla/src/osnoise.c line 46

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+
+	context->cpus_fd = open(cpus_path, O_RDWR);
+	if (context->cpus_fd < 0)
+		goto out_err;
+
+	retval = read(context->cpus_fd, &buffer, sizeof(buffer));
+	if (retval <= 0)
+		goto out_close;
+
+	context->orig_cpus = strdup(buffer);

Or you could have done:

Quoted 1 line
	context->orig_cpus = tracefs_instance_read(NULL, "osnoise/cpus");

Yep, that would be better.

Quoted 2 lines

as I doubt that reading and writing the cpus file you really care about keeping around the file descriptor.

Yep, I do not necessarly need it.... (do not ask me why I am not using, it is obviosly better... I might have just missed it).

It's not something likely to be done

Quoted 4 lines

where you care about "disrupting" the system. But if you really do care:

	context->cpus_fd = tracefs_instance_file_open(NULL, "osnoise/cpus",
				O_RDWR);

I will use tracefs helpers to read/write files, and remove the file descriptor variables.

[...]

Quoted 6 lines

tools/tracing/rtla/src/osnoise.c line 237

Quoted 3 lines
tools/tracing/rtla/src/osnoise.c new file +3 -0
+	snprintf(buffer, sizeof(buffer), "%llu\n", runtime);
+
+	retval = write(context->runtime_fd, buffer, strlen(buffer) + 1);

Again, how important is it to have the fd?

it is not...

[...]

Quoted 20 lines
Quoted 5 lines
tools/tracing/rtla/src/osnoise.c new file +5 -0
+/*
+ * osnoise_set_stop_total_us - set "stop_tracing_total_us"
+ */
+int osnoise_set_stop_total_us(struct osnoise_context *context, long long stop_total_us)
+{

tools/tracing/rtla/src/osnoise.c line 634

Quoted 10 lines
tools/tracing/rtla/src/osnoise.c new file +10 -0
+	long long curr_stop_total_us = osnoise_get_stop_total_us(context);
+	char buffer[BUFF_U64_STR_SIZE];
+	int retval;
+
+	if (curr_stop_total_us == OSNOISE_OPTION_INIT_VAL)
+		return -1;
+
+	snprintf(buffer, BUFF_U64_STR_SIZE, "%lld\n", stop_total_us);
+
+	retval = write(context->stop_total_us_fd, buffer, strlen(buffer) + 1);

And here.

Hmm, we should add a helper:

Quoted 7 lines

tracefs_instance_file_printf(instance, fmt, ...)

Where the above could be:

	tracefs_instance_file_printf(NULL, "%lld\n", stop_total_us);

Of course, for now, you could just add a helper function that does that.

sounds good to me.

[...]

Quoted 16 lines
Quoted 16 lines
tools/tracing/rtla/src/osnoise.c new file +16 -0
+/*
+ * osnoise_context_alloc - alloc an osnoise_context
+ *
+ * The osnoise context contains the information of the "osnoise/" configs.
+ * It is used to set and restore the config.
+ */
+struct osnoise_context *osnoise_context_alloc(void)
+{
+	struct osnoise_context *context;
+
+	context = calloc(1, sizeof(*context));
+	if (!context)
+		goto out_err;
+
+	context->cpus_fd 		= CLOSED_FD;
+	context->runtime_fd		= CLOSED_FD;
Quoted 8 lines

tools/tracing/rtla/src/osnoise.c line 846

Quoted 5 lines
tools/tracing/rtla/src/osnoise.c new file +5 -0
+	context->period_fd		= CLOSED_FD;
+	context->stop_us_fd		= CLOSED_FD;
+	context->stop_total_us_fd	= CLOSED_FD;
+	context->timerlat_period_us_fd	= CLOSED_FD;
+	context->print_stack_fd		= CLOSED_FD;

You could save a lot of code by using the tracefs_instance_file_*() helpers. And do you really need to keep around the file descriptors?

I will use the helpers in v9, I am convincede... :-)

Thanks Steve
-- Daniel

#11 lore raw

PATCH v8 4/14 rtla/osnoise: Add osnoise top mode

In reply to Daniel Bristot de Oliveira

The rtla osnoise tool is an interface for the osnoise tracer. The osnoise tracer dispatches a kernel thread per-cpu. These threads read the time in a loop while with preemption, softirqs and IRQs enabled, thus allowing all the sources of osnoise during its execution. The osnoise threads take note of the entry and exit point of any source of interferences, increasing a per-cpu interference counter. The osnoise tracer also saves an interference counter for each source of interference.

The rtla osnoise top mode displays information about the periodic summary from the osnoise tracer.

One example of rtla osnoise top output is:

[root@alien ~]# rtla osnoise top -c 0-3 -d 1m -q -r 900000 -P F:1
Operating System Noise
duration: 0 00:01:00 | time is in us
CPU Period Runtime Noise % CPU Aval Max Noise Max Single HW NMI IRQ Softirq Thread

  0 #58         52200000         1031    99.99802          91           60           0            0        52285            0          101
  1 #59         53100000            5    99.99999           5            5           0            9        53122            0           18
  2 #59         53100000            7    99.99998           7            7           0            8        53115            0           18
  3 #59         53100000         8274    99.98441         277           23           0            9        53778            0          660

"rtla osnoise top --help" works and provide information about the available options.

Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 tools/tracing/rtla/src/osnoise_top.c | 577 +++++++++++++++++++++++++++
 1 file changed, 577 insertions(+)
 create mode 100644 tools/tracing/rtla/src/osnoise_top.c
tools/tracing/rtla/src/osnoise_top.c new file +577 -0
new file mode 100644
index 000000000000..e23bae8d9696
@@ -0,0 +1,577 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
+ */
+
+#include <getopt.h>
+#include <stdlib.h>
+#include <string.h>
+#include <signal.h>
+#include <unistd.h>
+#include <stdio.h>
+#include <time.h>
+
+#include "osnoise.h"
+#include "utils.h"
+
+/*
+ * osnoise top parameters
+ */
+struct osnoise_top_params {
+	char			*cpus;
+	char			*monitored_cpus;
+	char			*trace_output;
+	unsigned long long	runtime;
+	unsigned long long	period;
+	long long		stop_us;
+	long long		stop_total_us;
+	int			sleep_time;
+	int			duration;
+	int			quiet;
+	int			set_sched;
+	struct sched_attr	sched_param;
+};
+
+struct osnoise_top_cpu {
+	unsigned long long	sum_runtime;
+	unsigned long long	sum_noise;
+	unsigned long long	max_noise;
+	unsigned long long	max_sample;
+
+	unsigned long long	hw_count;
+	unsigned long long	nmi_count;
+	unsigned long long	irq_count;
+	unsigned long long	softirq_count;
+	unsigned long long	thread_count;
+
+	int			sum_cycles;
+};
+
+struct osnoise_top_data {
+	struct osnoise_top_cpu	*cpu_data;
+	int			nr_cpus;
+};
+
+/*
+ * osnoise_free_top - free runtime data
+ */
+static void
+osnoise_free_top(struct osnoise_top_data *data)
+{
+	free(data->cpu_data);
+	free(data);
+}
+
+/*
+ * osnoise_alloc_histogram - alloc runtime data
+ */
+static struct osnoise_top_data *osnoise_alloc_top(int nr_cpus)
+{
+	struct osnoise_top_data *data;
+
+	data = calloc(1, sizeof(*data));
+	if (!data)
+		return NULL;
+
+	data->nr_cpus = nr_cpus;
+
+	/* one set of histograms per CPU */
+	data->cpu_data = calloc(1, sizeof(*data->cpu_data) * nr_cpus);
+	if (!data->cpu_data)
+		goto cleanup;
+
+	return data;
+
+cleanup:
+	osnoise_free_top(data);
+	return NULL;
+}
+
+/*
+ * osnoise_top_handler - this is the handler for osnoise tracer events
+ */
+static int
+osnoise_top_handler(struct trace_seq *s, struct tep_record *record,
+		    struct tep_event *event, void *context)
+{
+	struct trace_instance *trace = context;
+	struct osnoise_tool *tool;
+	unsigned long long val;
+	struct osnoise_top_cpu *cpu_data;
+	struct osnoise_top_data *data;
+	int cpu = record->cpu;
+
+	tool = container_of(trace, struct osnoise_tool, trace);
+
+	data = tool->data;
+	cpu_data = &data->cpu_data[cpu];
+
+	cpu_data->sum_cycles++;
+
+	tep_get_field_val(s, event, "runtime", record, &val, 1);
+	update_sum(&cpu_data->sum_runtime, &val);
+
+	tep_get_field_val(s, event, "noise", record, &val, 1);
+	update_max(&cpu_data->max_noise, &val);
+	update_sum(&cpu_data->sum_noise, &val);
+
+	tep_get_field_val(s, event, "max_sample", record, &val, 1);
+	update_max(&cpu_data->max_sample, &val);
+
+	tep_get_field_val(s, event, "hw_count", record, &val, 1);
+	update_sum(&cpu_data->hw_count, &val);
+
+	tep_get_field_val(s, event, "nmi_count", record, &val, 1);
+	update_sum(&cpu_data->nmi_count, &val);
+
+	tep_get_field_val(s, event, "irq_count", record, &val, 1);
+	update_sum(&cpu_data->irq_count, &val);
+
+	tep_get_field_val(s, event, "softirq_count", record, &val, 1);
+	update_sum(&cpu_data->softirq_count, &val);
+
+	tep_get_field_val(s, event, "thread_count", record, &val, 1);
+	update_sum(&cpu_data->thread_count, &val);
+
+	return 0;
+}
+
+/*
+ * osnoise_top_header - print the header of the tool output
+ */
+static void osnoise_top_header(struct osnoise_tool *top)
+{
+	struct trace_seq *s = top->trace.seq;
+	char duration[26];
+
+	get_duration(top->start_time, duration, sizeof(duration));
+
+	trace_seq_printf(s, "\033[2;37;40m");
+	trace_seq_printf(s, "                                          Operating System Noise");
+	trace_seq_printf(s, "                                     ");
+	trace_seq_printf(s, "                                     ");
+	trace_seq_printf(s, "\033[0;0;0m");
+	trace_seq_printf(s, "\n");
+
+	trace_seq_printf(s, "duration: %9s | time is in us\n", duration);
+
+	trace_seq_printf(s, "\033[2;30;47m");
+	trace_seq_printf(s, "CPU Period       Runtime ");
+	trace_seq_printf(s, "       Noise ");
+	trace_seq_printf(s, " %% CPU Aval ");
+	trace_seq_printf(s, "  Max Noise   Max Single ");
+	trace_seq_printf(s, "         HW          NMI          IRQ      Softirq       Thread");
+	trace_seq_printf(s, "\033[0;0;0m");
+	trace_seq_printf(s, "\n");
+}
+
+/*
+ * clear_terminal - clears the output terminal
+ */
+static void clear_terminal(struct trace_seq *seq)
+{
+	if (!config_debug)
+		trace_seq_printf(seq, "\033c");
+}
+
+/*
+ * osnoise_top_print - prints the output of a given CPU
+ */
+static void osnoise_top_print(struct osnoise_tool *tool, int cpu)
+{
+	struct trace_seq *s = tool->trace.seq;
+	struct osnoise_top_cpu *cpu_data;
+	struct osnoise_top_data *data;
+	int percentage;
+	int decimal;
+
+	data = tool->data;
+	cpu_data = &data->cpu_data[cpu];
+
+	if (!cpu_data->sum_runtime)
+		return;
+
+	percentage = ((cpu_data->sum_runtime - cpu_data->sum_noise) * 10000000)
+			/ cpu_data->sum_runtime;
+	decimal = percentage % 100000;
+	percentage = percentage / 100000;
+
+	trace_seq_printf(s, "%3d #%-6d %12llu ", cpu, cpu_data->sum_cycles, cpu_data->sum_runtime);
+	trace_seq_printf(s, "%12llu ", cpu_data->sum_noise);
+	trace_seq_printf(s, "  %3d.%05d", percentage, decimal);
+	trace_seq_printf(s, "%12llu %12llu", cpu_data->max_noise, cpu_data->max_sample);
+
+	trace_seq_printf(s, "%12llu ", cpu_data->hw_count);
+	trace_seq_printf(s, "%12llu ", cpu_data->nmi_count);
+	trace_seq_printf(s, "%12llu ", cpu_data->irq_count);
+	trace_seq_printf(s, "%12llu ", cpu_data->softirq_count);
+	trace_seq_printf(s, "%12llu\n", cpu_data->thread_count);
+}
+
+/*
+ * osnoise_print_stats - print data for all cpus
+ */
+static void
+osnoise_print_stats(struct osnoise_top_params *params, struct osnoise_tool *top)
+{
+	struct trace_instance *trace = &top->trace;
+	static int nr_cpus = -1;
+	int i;
+
+	if (nr_cpus == -1)
+		nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
+
+	if (!params->quiet)
+		clear_terminal(trace->seq);
+
+	osnoise_top_header(top);
+
+	for (i = 0; i < nr_cpus; i++) {
+		if (params->cpus && !params->monitored_cpus[i])
+			continue;
+		osnoise_top_print(top, i);
+	}
+
+	trace_seq_do_printf(trace->seq);
+	trace_seq_reset(trace->seq);
+}
+
+/*
+ * osnoise_top_usage - prints osnoise top usage message
+ */
+void osnoise_top_usage(char *usage)
+{
+	int i;
+
+	static const char * const msg[] = {
+		"  usage: rtla osnoise [top] [-h] [-q] [-p us] [-r us] [-s us] [-S us] [-T[=file]] \\",
+		"	  [-c cpu-list] -P priority",
+		"",
+		"	  -h/--help: print this menu",
+		"	  -p/--period us: osnoise period in us",
+		"	  -r/--runtime us: osnoise runtime in us",
+		"	  -s/--stop us: stop trace if a single sample is higher than the argument in us",
+		"	  -S/--stop-total us: stop trace if the total sample is higher than the argument in us",
+		"	  -c/--cpus cpu-list: list of cpus to run osnoise threads",
+		"	  -d/--duration time[s|m|h|d]: duration of the session",
+		"	  -t/--trace[=file]: save the stopped trace to [file|osnoise_trace.txt]",
+		"	  -q/--quiet print only a summary at the end",
+		"	  -P/--priority o:prio|r:prio|f:prio|d:runtime:period : set scheduling parameters",
+		"		o:prio - use SCHED_OTHER with prio",
+		"		r:prio - use SCHED_RR with prio",
+		"		f:prio - use SCHED_FIFO with prio",
+		"		d:runtime[us|ms|s]:period[us|ms|s] - use SCHED_DEADLINE with runtime and period",
+		"						       in nanoseconds",
+		NULL,
+	};
+
+	if (usage)
+		fprintf(stderr, "%s\n", usage);
+
+	fprintf(stderr, "rtla osnoise top: a per-cpu summary of the OS noise (version %s)\n",
+			VERSION);
+
+	for (i = 0; msg[i]; i++)
+		fprintf(stderr, "%s\n", msg[i]);
+	exit(1);
+}
+
+/*
+ * osnoise_top_parse_args - allocs, parse and fill the cmd line parameters
+ */
+struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv)
+{
+	struct osnoise_top_params *params;
+	int retval;
+	int c;
+
+	params = calloc(1, sizeof(*params));
+	if (!params)
+		exit(1);
+
+	while (1) {
+		static struct option long_options[] = {
+			{"cpus",		required_argument,	0, 'c'},
+			{"debug",		no_argument,		0, 'D'},
+			{"duration",		required_argument,	0, 'd'},
+			{"help",		no_argument,		0, 'h'},
+			{"period",		required_argument,	0, 'p'},
+			{"priority",		required_argument,	0, 'P'},
+			{"quiet",		no_argument,		0, 'q'},
+			{"runtime",		required_argument,	0, 'r'},
+			{"stop",		required_argument,	0, 's'},
+			{"stop-total",		required_argument,	0, 'S'},
+			{"trace",		optional_argument,	0, 't'},
+			{0, 0, 0, 0}
+		};
+
+		/* getopt_long stores the option index here. */
+		int option_index = 0;
+
+		c = getopt_long(argc, argv, "c:d:Dhp:P:qr:s:S:t::",
+				 long_options, &option_index);
+
+		/* Detect the end of the options. */
+		if (c == -1)
+			break;
+
+		switch (c) {
+		case 'c':
+			retval = parse_cpu_list(optarg, &params->monitored_cpus);
+			if (retval)
+				osnoise_top_usage("\nInvalid -c cpu list\n");
+			params->cpus = optarg;
+			debug_msg("-c for %s\n", params->cpus);
+			break;
+		case 'D':
+			config_debug = 1;
+			break;
+		case 'd':
+			params->duration = parse_seconds_duration(optarg);
+			if (!params->duration)
+				osnoise_top_usage("Invalid -D duration\n");
+			break;
+		case 'h':
+		case '?':
+			osnoise_top_usage(NULL);
+			break;
+		case 'p':
+			params->period = get_llong_from_str(optarg);
+			if (params->period > 10000000)
+				osnoise_top_usage("Period longer than 10 s\n");
+			break;
+		case 'P':
+			retval = parse_prio(optarg, &params->sched_param);
+			if (retval == -1)
+				osnoise_top_usage("Invalid -P priority");
+			params->set_sched = 1;
+			break;
+		case 'q':
+			params->quiet = 1;
+			break;
+		case 'r':
+			params->runtime = get_llong_from_str(optarg);
+			if (params->runtime < 100)
+				osnoise_top_usage("Runtime shorter than 100 us\n");
+			break;
+		case 's':
+			params->stop_us = get_llong_from_str(optarg);
+			break;
+		case 'S':
+			params->stop_total_us = get_llong_from_str(optarg);
+			break;
+		case 't':
+			if (optarg)
+				/* skip = */
+				params->trace_output = &optarg[1];
+			else
+				params->trace_output = "osnoise_trace.txt";
+			break;
+		default:
+			osnoise_top_usage("Invalid option");
+		}
+	}
+
+	if (geteuid()) {
+		err_msg("osnoise needs root permission\n");
+		exit(EXIT_FAILURE);
+	}
+
+	return params;
+}
+
+/*
+ * osnoise_top_apply_config - apply the top configs to the initialized tool
+ */
+static int
+osnoise_top_apply_config(struct osnoise_tool *tool, struct osnoise_top_params *params)
+{
+	int retval;
+
+	if (!params->sleep_time)
+		params->sleep_time = 1;
+
+	if (params->cpus) {
+		retval = osnoise_set_cpus(tool->context, params->cpus);
+		if (retval) {
+			err_msg("Failed to apply CPUs config\n");
+			goto out_err;
+		}
+	}
+
+	if (params->runtime || params->period) {
+		retval = osnoise_set_runtime_period(tool->context,
+						    params->runtime,
+						    params->period);
+		if (retval) {
+			err_msg("Failed to set runtime and/or period\n");
+			goto out_err;
+		}
+	}
+
+	if (params->stop_us) {
+		retval = osnoise_set_stop_us(tool->context, params->stop_us);
+		if (retval) {
+			err_msg("Failed to set stop us\n");
+			goto out_err;
+		}
+	}
+
+	if (params->stop_total_us) {
+		retval = osnoise_set_stop_total_us(tool->context, params->stop_total_us);
+		if (retval) {
+			err_msg("Failed to set stop total us\n");
+			goto out_err;
+		}
+	}
+
+	return 0;
+
+out_err:
+	return -1;
+}
+
+/*
+ * osnoise_init_top - initialize a osnoise top tool with parameters
+ */
+struct osnoise_tool *osnoise_init_top(struct osnoise_top_params *params)
+{
+	struct osnoise_tool *tool;
+	int nr_cpus;
+
+	nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
+
+	tool = osnoise_init_tool("osnoise_top");
+	if (!tool)
+		return NULL;
+
+	tool->data = osnoise_alloc_top(nr_cpus);
+	if (!tool->data)
+		goto out_err;
+
+	tool->params = params;
+
+	tep_register_event_handler(tool->trace.tep, -1, "ftrace", "osnoise",
+				   osnoise_top_handler, NULL);
+
+	return tool;
+
+out_err:
+	osnoise_free_top(tool->data);
+	osnoise_destroy_tool(tool);
+	return NULL;
+}
+
+static int stop_tracing;
+static void stop_top(int sig)
+{
+	stop_tracing = 1;
+}
+
+/*
+ * osnoise_top_set_signals - handles the signal to stop the tool
+ */
+static void osnoise_top_set_signals(struct osnoise_top_params *params)
+{
+	signal(SIGINT, stop_top);
+	if (params->duration) {
+		signal(SIGALRM, stop_top);
+		alarm(params->duration);
+	}
+}
+
+int osnoise_top_main(int argc, char **argv)
+{
+	struct osnoise_top_params *params;
+	struct trace_instance *trace;
+	struct osnoise_tool *record;
+	struct osnoise_tool *tool;
+	int return_value = 1;
+	int retval;
+
+	params = osnoise_top_parse_args(argc, argv);
+	if (!params)
+		exit(1);
+
+	tool = osnoise_init_top(params);
+	if (!tool) {
+		err_msg("Could not init osnoise top\n");
+		goto out_exit;
+	}
+
+	retval = osnoise_top_apply_config(tool, params);
+	if (retval) {
+		err_msg("Could not apply config\n");
+		goto out_top;
+	}
+
+	trace = &tool->trace;
+
+	retval = enable_osnoise(trace);
+	if (retval) {
+		err_msg("Failed to enable osnoise tracer\n");
+		goto out_top;
+	}
+
+	if (params->set_sched) {
+		retval = set_comm_sched_attr("osnoise/", &params->sched_param);
+		if (retval)
+			osnoise_top_usage("Failed to set sched parameters\n");
+	}
+
+	trace_instance_start(trace);
+
+	if (params->trace_output) {
+		record = osnoise_init_trace_tool("osnoise");
+		if (!record) {
+			err_msg("Failed to enable the trace instance\n");
+			goto out_top;
+		}
+		trace_instance_start(&record->trace);
+	}
+
+	tool->start_time = time(NULL);
+	osnoise_top_set_signals(params);
+
+	do {
+		sleep(params->sleep_time);
+
+		retval = tracefs_iterate_raw_events(trace->tep,
+						    trace->inst,
+						    NULL,
+						    0,
+						    collect_registered_events,
+						    trace);
+		if (retval < 0) {
+			err_msg("Error iterating on events\n");
+			goto out_top;
+		}
+
+		if (!params->quiet)
+			osnoise_print_stats(params, tool);
+
+		if (!tracefs_trace_is_on(trace->inst))
+			break;
+
+	} while (!stop_tracing);
+
+	osnoise_print_stats(params, tool);
+
+	return_value = 0;
+
+	if (!tracefs_trace_is_on(trace->inst)) {
+		printf("osnoise hit stop tracing\n");
+		if (params->trace_output) {
+			printf("  Saving trace to %s\n", params->trace_output);
+			save_trace_to_file(record->trace.inst, params->trace_output);
+		}
+	}
+
+out_top:
+	osnoise_free_top(tool->data);
+	osnoise_destroy_tool(tool);
+	if (params->trace_output)
+		osnoise_destroy_tool(record);
+out_exit:
+	exit(return_value);
+}
Signature · 1 line
2.31.1
#12 lore raw

Re: [PATCH V8 04/14] rtla/osnoise: Add osnoise top mode

Tao

Quoted 26 lines · Daniel Bristot de Oliveira
tools/tracing/rtla/src/osnoise_top.c new file +26 -0
+int osnoise_top_main(int argc, char **argv)
+{
+	struct osnoise_top_params *params;
+	struct trace_instance *trace;
+	struct osnoise_tool *record;
+	struct osnoise_tool *tool;
+	int return_value = 1;
+	int retval;
+
+	params = osnoise_top_parse_args(argc, argv);
+	if (!params)
+		exit(1);
+
+	tool = osnoise_init_top(params);
+	if (!tool) {
+		err_msg("Could not init osnoise top\n");
+		goto out_exit;
+	}
+
+	retval = osnoise_top_apply_config(tool, params);
+	if (retval) {
+		err_msg("Could not apply config\n");
+		goto out_top;
+	}
+
+	trace = &tool->trace;

tools/tracing/rtla/src/osnoise_top.c line 518

Quoted 10 lines
tools/tracing/rtla/src/osnoise_top.c new file +10 -0
+
+	retval = enable_osnoise(trace);
+	if (retval) {
+		err_msg("Failed to enable osnoise tracer\n");
+		goto out_top;
+	}
+
+	if (params->set_sched) {
+		retval = set_comm_sched_attr("osnoise/", &params->sched_param);
+		if (retval) 

You hit this case. It is correct to get this error if one sets SCHED_DEADLINE without disabling the admission control.

However, there is also a problem here. The tracer was not disabled, and the instance was not removed - so osnoise threads kept running after the error message and exit().

I will fix this on all tools for v9.

Thanks for testing!
--- Daniel

Quoted 3 lines
tools/tracing/rtla/src/osnoise_top.c new file +3 -0
+			osnoise_top_usage("Failed to set sched parameters\n");
+	}
+
#13 lore raw

PATCH v8 5/14 rtla/osnoise: Add the hist mode

In reply to Daniel Bristot de Oliveira

The rtla osnoise hist tool collects all osnoise:sample_threshold occurrence in a histogram, displaying the results in a user-friendly way. The tool also allows many configurations of the osnoise tracer and the collection of the tracer output.

Here is one example of the rtla osnoise hist tool output:
---------- %< ----------
[root@f34 ~]# rtla osnoise hist --bucket-size 10 --entries 100 -c 0-8 -d 1M -r 9000 -P F:1 # RTLA osnoise histogram
# Time unit is microseconds (us)
# Duration: 0 00:01:00
Index CPU-000 CPU-001 CPU-002 CPU-003 CPU-004 CPU-005 CPU-006 CPU-007 CPU-008

 0           430       434       352       455       440       463       467       436       484
 10           88        88        92       141       120       100       126       166       100
 20           19         7        12        22         8         8        13        13        16
 30            6         0         2         0         1         2         2         1         0
 50            0         0         0         0         0         0         1         0         0
 over:         0         0         0         0         0         0         0         0         0
 count:      543       529       458       618       569       573       609       616       600
 min:          0         0         0         0         0         0         0         0         0
 avg:          0         0         0         0         0         0         0         0         0

max: 30 20 30 20 30 30 50 30 20
---------- >% ----------

Running
- rtla osnoise hist --help

provides information about the available options.

Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 tools/tracing/rtla/src/osnoise.c      |   6 +-
 tools/tracing/rtla/src/osnoise.h      |   1 +
 tools/tracing/rtla/src/osnoise_hist.c | 799 ++++++++++++++++++++++++++
 3 files changed, 805 insertions(+), 1 deletion(-)
 create mode 100644 tools/tracing/rtla/src/osnoise_hist.c
tools/tracing/rtla/src/osnoise.c +5 -1
index 7ef686dddc09..d8bde8e45710 100644
@@ -970,7 +970,8 @@ static void osnoise_usage(void)
 		"  usage: [rtla] osnoise [MODE] ...",
 		"",
 		"  modes:",
-		"     top  - prints the summary from osnoise tracer",
+		"     top   - prints the summary from osnoise tracer",
+		"     hist  - prints a histogram of osnoise samples",
 		"",
 		"if no MODE is given, the top mode is called, passing the arguments",
 		NULL,
@@ -1005,6 +1006,9 @@ int osnoise_main(int argc, char *argv[])
 	} else if (strcmp(argv[1], "top") == 0) {
 		osnoise_top_main(argc-1, &argv[1]);
 		exit(0);
+	} else if (strcmp(argv[1], "hist") == 0) {
+		osnoise_hist_main(argc-1, &argv[1]);
+		exit(0);
 	}
 
 usage:
tools/tracing/rtla/src/osnoise.h +1 -0
index e3e8f8359d99..f6c9fe05c744 100644
@@ -91,5 +91,6 @@ void osnoise_destroy_tool(struct osnoise_tool *top);
 struct osnoise_tool *osnoise_init_tool(char *tool_name);
 struct osnoise_tool *osnoise_init_trace_tool(char *tracer);
 
+int osnoise_hist_main(int argc, char *argv[]);
 int osnoise_top_main(int argc, char **argv);
 int osnoise_main(int argc, char **argv);
tools/tracing/rtla/src/osnoise_hist.c new file +799 -0

799 lines not on this page · raw message

Signature · 1 line
2.31.1
#14 lore raw

PATCH v8 6/14 rtla: Add timerlat tool and timelart top mode

In reply to Daniel Bristot de Oliveira

The rtla timerlat tool is an interface for the timerlat tracer. The timerlat tracer dispatches a kernel thread per-cpu. These threads set a periodic timer to wake themselves up and go back to sleep. After the wakeup, they collect and generate useful information for the debugging of operating system timer latency.

The timerlat tracer outputs information in two ways. It periodically prints the timer latency at the timer IRQ handler and the Thread handler. It also provides information for each noise via the osnoise tracepoints.

The rtla timerlat top mode displays a summary of the periodic output from the timerlat tracer.

Here is one example of the rtla timerlat tool output:
---------- %< ----------
[root@alien ~]# rtla timerlat top -c 0-3 -d 1m
Timer Latency

  0 00:01:00   |          IRQ Timer Latency (us)        |         Thread Timer Latency (us)
CPU COUNT      |      cur       min       avg       max |      cur       min       avg       max
  0 #60001     |        0         0         0         3 |        1         1         1         6
  1 #60001     |        0         0         0         3 |        2         1         1         5
  2 #60001     |        0         0         1         6 |        1         1         2         7
  3 #60001     |        0         0         0         7 |        1         1         1        11

---------- >% ----------

Running:

  # rtla timerlat --help
  # rtla timerlat top --help
provides information about the available options.
Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 tools/tracing/rtla/Makefile           |   2 +
 tools/tracing/rtla/src/rtla.c         |   5 +
 tools/tracing/rtla/src/timerlat.c     |  68 +++
 tools/tracing/rtla/src/timerlat.h     |   4 +
 tools/tracing/rtla/src/timerlat_top.c | 615 ++++++++++++++++++++++++++
 5 files changed, 694 insertions(+)
 create mode 100644 tools/tracing/rtla/src/timerlat.c
 create mode 100644 tools/tracing/rtla/src/timerlat.h
 create mode 100644 tools/tracing/rtla/src/timerlat_top.c
tools/tracing/rtla/Makefile +2 -0
index ba6f327e815a..f254cfbe4fa3 100644
@@ -62,6 +62,8 @@ install:
 	$(STRIP) $(DESTDIR)$(BINDIR)/rtla
 	@test ! -f $(DESTDIR)$(BINDIR)/osnoise || rm $(DESTDIR)$(BINDIR)/osnoise
 	ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/osnoise
+	@test ! -f $(DESTDIR)$(BINDIR)/timerlat || rm $(DESTDIR)$(BINDIR)/timerlat
+	ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/timerlat
 
 .PHONY: clean tarball
 clean:
tools/tracing/rtla/src/rtla.c +5 -0
index 669b9750b3b3..09bd21b8af81 100644
@@ -9,6 +9,7 @@
 #include <stdio.h>
 
 #include "osnoise.h"
+#include "timerlat.h"
 
 /*
  * rtla_usage - print rtla usage
@@ -25,6 +26,7 @@ static void rtla_usage(void)
 		"",
 		"  commands:",
 		"     osnoise  - gives information about the operating system noise (osnoise)",
+		"     timerlat - measures the timer irq and thread latency",
 		"",
 		NULL,
 	};
@@ -45,6 +47,9 @@ int run_command(int argc, char **argv, int start_position)
 	if (strcmp(argv[start_position], "osnoise") == 0) {
 		osnoise_main(argc-start_position, &argv[start_position]);
 		goto ran;
+	} else if (strcmp(argv[start_position], "timerlat") == 0) {
+		timerlat_main(argc-start_position, &argv[start_position]);
+		goto ran;
 	}
 
 	return 0;
tools/tracing/rtla/src/timerlat.c new file +68 -0

68 lines not on this page · raw message

tools/tracing/rtla/src/timerlat.h new file +4 -0
new file mode 100644
index 000000000000..6f9d3925acd0
@@ -0,0 +1,4 @@
+// SPDX-License-Identifier: GPL-2.0
+
+int timerlat_top_main(int argc, char *argv[]);
+int timerlat_main(int argc, char *argv[]);
tools/tracing/rtla/src/timerlat_top.c new file +615 -0

615 lines not on this page · raw message

Signature · 1 line
2.31.1
#15 lore raw

Re: [PATCH V8 06/14] rtla: Add timerlat tool and timelart top mode

Quoted 467 lines · Daniel Bristot de Oliveira

The rtla timerlat tool is an interface for the timerlat tracer. The timerlat tracer dispatches a kernel thread per-cpu. These threads set a periodic timer to wake themselves up and go back to sleep. After the wakeup, they collect and generate useful information for the debugging of operating system timer latency.

The timerlat tracer outputs information in two ways. It periodically prints the timer latency at the timer IRQ handler and the Thread handler. It also provides information for each noise via the osnoise tracepoints.

The rtla timerlat top mode displays a summary of the periodic output from the timerlat tracer.

Here is one example of the rtla timerlat tool output:
---------- %< ----------
[root@alien ~]# rtla timerlat top -c 0-3 -d 1m
Timer Latency

  0 00:01:00   |          IRQ Timer Latency (us)        |         Thread Timer Latency (us)
CPU COUNT      |      cur       min       avg       max |      cur       min       avg       max
  0 #60001     |        0         0         0         3 |        1         1         1         6
  1 #60001     |        0         0         0         3 |        2         1         1         5
  2 #60001     |        0         0         1         6 |        1         1         2         7
  3 #60001     |        0         0         0         7 |        1         1         1        11

---------- >% ----------

Running:

  # rtla timerlat --help
  # rtla timerlat top --help
provides information about the available options.
Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 tools/tracing/rtla/Makefile           |   2 +
 tools/tracing/rtla/src/rtla.c         |   5 +
 tools/tracing/rtla/src/timerlat.c     |  68 +++
 tools/tracing/rtla/src/timerlat.h     |   4 +
 tools/tracing/rtla/src/timerlat_top.c | 615 ++++++++++++++++++++++++++
 5 files changed, 694 insertions(+)
 create mode 100644 tools/tracing/rtla/src/timerlat.c
 create mode 100644 tools/tracing/rtla/src/timerlat.h
 create mode 100644 tools/tracing/rtla/src/timerlat_top.c
tools/tracing/rtla/Makefile +2 -0
index ba6f327e815a..f254cfbe4fa3 100644
@@ -62,6 +62,8 @@ install:
 	$(STRIP) $(DESTDIR)$(BINDIR)/rtla
 	@test ! -f $(DESTDIR)$(BINDIR)/osnoise || rm $(DESTDIR)$(BINDIR)/osnoise
 	ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/osnoise
+	@test ! -f $(DESTDIR)$(BINDIR)/timerlat || rm $(DESTDIR)$(BINDIR)/timerlat
+	ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/timerlat
 
 .PHONY: clean tarball
 clean:
tools/tracing/rtla/src/rtla.c +5 -0
index 669b9750b3b3..09bd21b8af81 100644
@@ -9,6 +9,7 @@
 #include <stdio.h>
 
 #include "osnoise.h"
+#include "timerlat.h"
 
 /*
  * rtla_usage - print rtla usage
@@ -25,6 +26,7 @@ static void rtla_usage(void)
 		"",
 		"  commands:",
 		"     osnoise  - gives information about the operating system noise (osnoise)",
+		"     timerlat - measures the timer irq and thread latency",
 		"",
 		NULL,
 	};
@@ -45,6 +47,9 @@ int run_command(int argc, char **argv, int start_position)
 	if (strcmp(argv[start_position], "osnoise") == 0) {
 		osnoise_main(argc-start_position, &argv[start_position]);
 		goto ran;
+	} else if (strcmp(argv[start_position], "timerlat") == 0) {
+		timerlat_main(argc-start_position, &argv[start_position]);
+		goto ran;
 	}
 
 	return 0;
tools/tracing/rtla/src/timerlat.c new file +68 -0
new file mode 100644
index 000000000000..29a51c01b084
@@ -0,0 +1,68 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
+ */
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <pthread.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <stdio.h>
+
+#include "timerlat.h"
+
+static void timerlat_usage(void)
+{
+	int i;
+
+	static const char * const msg[] = {
+		"",
+		"timerlat version " VERSION,
+		"",
+		"  usage: [rtla] timerlat [MODE] ...",
+		"",
+		"  modes:",
+		"     top  - prints the summary from timerlat tracer",
+		"",
+		"if no MODE is given, the top mode is called, passing the arguments",
+		NULL,
+	};
+
+	for (i = 0; msg[i]; i++)
+		fprintf(stderr, "%s\n", msg[i]);
+	exit(1);
+}
+
+int timerlat_main(int argc, char *argv[])
+{
+	if (argc == 0)
+		goto usage;
+
+	/*
+	 * if timerlat was called without any argument, run the
+	 * default cmdline.
+	 */
+	if (argc == 1) {
+		timerlat_top_main(argc, argv);
+		exit(0);
+	}
+
+	if ((strcmp(argv[1], "-h") == 0) || (strcmp(argv[1], "--help") == 0)) {
+		timerlat_usage();
+		exit(0);
+	} else if (strncmp(argv[1], "-", 1) == 0) {
+		/* the user skipped the tool, call the default one */
+		timerlat_top_main(argc, argv);
+		exit(0);
+	} else if (strcmp(argv[1], "top") == 0) {
+		timerlat_top_main(argc-1, &argv[1]);
+		exit(0);
+	}
+
+usage:
+	timerlat_usage();
+	exit(1);
+}
tools/tracing/rtla/src/timerlat.h new file +4 -0
new file mode 100644
index 000000000000..6f9d3925acd0
@@ -0,0 +1,4 @@
+// SPDX-License-Identifier: GPL-2.0
+
+int timerlat_top_main(int argc, char *argv[]);
+int timerlat_main(int argc, char *argv[]);
tools/tracing/rtla/src/timerlat_top.c new file +278 -0
new file mode 100644
index 000000000000..9764681ad8b1
@@ -0,0 +1,615 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
+ */
+
+#include <getopt.h>
+#include <stdlib.h>
+#include <string.h>
+#include <signal.h>
+#include <unistd.h>
+#include <stdio.h>
+#include <time.h>
+
+#include "utils.h"
+#include "osnoise.h"
+#include "timerlat.h"
+
+struct timerlat_top_params {
+	char			*cpus;
+	char			*monitored_cpus;
+	char			*trace_output;
+	unsigned long long	runtime;
+	long long		stop_us;
+	long long		stop_total_us;
+	long long		timerlat_period_us;
+	long long		print_stack;
+	int			sleep_time;
+	int			output_divisor;
+	int			duration;
+	int			quiet;
+	int			set_sched;
+	struct sched_attr	sched_param;
+};
+
+struct timerlat_top_cpu {
+	int			irq_count;
+	int			thread_count;
+
+	unsigned long long	cur_irq;
+	unsigned long long	min_irq;
+	unsigned long long	sum_irq;
+	unsigned long long	max_irq;
+
+	unsigned long long	cur_thread;
+	unsigned long long	min_thread;
+	unsigned long long	sum_thread;
+	unsigned long long	max_thread;
+};
+
+struct timerlat_top_data {
+	struct timerlat_top_cpu	*cpu_data;
+	int			nr_cpus;
+};
+
+/*
+ * timerlat_free_top - free runtime data
+ */
+static void
+timerlat_free_top(struct timerlat_top_data *data)
+{
+	free(data->cpu_data);
+	free(data);
+}
+
+/*
+ * timerlat_alloc_histogram - alloc runtime data
+ */
+static struct timerlat_top_data *timerlat_alloc_top(int nr_cpus)
+{
+	struct timerlat_top_data *data;
+	int cpu;
+
+	data = calloc(1, sizeof(*data));
+	if (!data)
+		return NULL;
+
+	data->nr_cpus = nr_cpus;
+
+	/* one set of histograms per CPU */
+	data->cpu_data = calloc(1, sizeof(*data->cpu_data) * nr_cpus);
+	if (!data->cpu_data)
+		goto cleanup;
+
+	/* set the min to max */
+	for (cpu = 0; cpu < nr_cpus; cpu++) {
+		data->cpu_data[cpu].min_irq = ~0;
+		data->cpu_data[cpu].min_thread = ~0;
+	}
+
+	return data;
+
+cleanup:
+	timerlat_free_top(data);
+	return NULL;
+}
+
+/*
+ * timerlat_hist_update - record a new timerlat occurent on cpu, updating data
+ */
+static void
+timerlat_top_update(struct osnoise_tool *tool, int cpu,
+		    unsigned long long thread,
+		    unsigned long long latency)
+{
+	struct timerlat_top_data *data = tool->data;
+	struct timerlat_top_cpu *cpu_data = &data->cpu_data[cpu];
+
+	if (!thread) {
+		cpu_data->irq_count++;
+		cpu_data->cur_irq = latency;
+		update_min(&cpu_data->min_irq, &latency);
+		update_sum(&cpu_data->sum_irq, &latency);
+		update_max(&cpu_data->max_irq, &latency);
+	} else {
+		cpu_data->thread_count++;
+		cpu_data->cur_thread = latency;
+		update_min(&cpu_data->min_thread, &latency);
+		update_sum(&cpu_data->sum_thread, &latency);
+		update_max(&cpu_data->max_thread, &latency);
+	}
+}
+
+/*
+ * timerlat_top_handler - this is the handler for timerlat tracer events
+ */
+static int
+timerlat_top_handler(struct trace_seq *s, struct tep_record *record,
+		     struct tep_event *event, void *context)
+{
+	struct trace_instance *trace = context;
+	unsigned long long latency, thread;
+	struct osnoise_tool *top;
+	int cpu = record->cpu;
+
+	top = container_of(trace, struct osnoise_tool, trace);
+
+	tep_get_field_val(s, event, "context", record, &thread, 1);
+	tep_get_field_val(s, event, "timer_latency", record, &latency, 1);
+
+	timerlat_top_update(top, cpu, thread, latency);
+
+	return 0;
+}
+
+/*
+ * timerlat_top_header - print the header of the tool output
+ */
+static void timerlat_top_header(struct osnoise_tool *top)
+{
+	struct timerlat_top_params *params = top->params;
+	struct trace_seq *s = top->trace.seq;
+	char duration[26];
+
+	get_duration(top->start_time, duration, sizeof(duration));
+
+	trace_seq_printf(s, "\033[2;37;40m");
+	trace_seq_printf(s, "                                     Timer Latency                                              ");
+	trace_seq_printf(s, "\033[0;0;0m");
+	trace_seq_printf(s, "\n");
+
+	trace_seq_printf(s, "%-6s   |          IRQ Timer Latency (%s)        |         Thread Timer Latency (%s)\n", duration,
+			params->output_divisor == 1 ? "ns" : "us",
+			params->output_divisor == 1 ? "ns" : "us");
+
+	trace_seq_printf(s, "\033[2;30;47m");
+	trace_seq_printf(s, "CPU COUNT      |      cur       min       avg       max |      cur       min       avg       max");
+	trace_seq_printf(s, "\033[0;0;0m");
+	trace_seq_printf(s, "\n");
+}
+
+/*
+ * timerlat_top_print - prints the output of a given CPU
+ */
+static void timerlat_top_print(struct osnoise_tool *top, int cpu)
+{
+
+	struct timerlat_top_params *params = top->params;
+	struct timerlat_top_data *data = top->data;
+	struct timerlat_top_cpu *cpu_data = &data->cpu_data[cpu];
+	int divisor = params->output_divisor;
+	struct trace_seq *s = top->trace.seq;
+
+	if (divisor == 0)
+		return;
+
+	/*
+	 * Skip if no data is available: is this cpu offline?
+	 */
+	if (!cpu_data->irq_count && !cpu_data->thread_count)
+		return;
+
+	/*
+	 * Unless trace is being lost, IRQ counter is always the max.
+	 */
+	trace_seq_printf(s, "%3d #%-9d |", cpu, cpu_data->irq_count);
+
+	if (!cpu_data->irq_count) {
+		trace_seq_printf(s, "        - ");
+		trace_seq_printf(s, "        - ");
+		trace_seq_printf(s, "        - ");
+		trace_seq_printf(s, "        - |");
+	} else {
+		trace_seq_printf(s, "%9llu ", cpu_data->cur_irq / params->output_divisor);
+		trace_seq_printf(s, "%9llu ", cpu_data->min_irq / params->output_divisor);
+		trace_seq_printf(s, "%9llu ", (cpu_data->sum_irq / cpu_data->irq_count) / divisor);
+		trace_seq_printf(s, "%9llu |", cpu_data->max_irq / divisor);
+	}
+
+	if (!cpu_data->thread_count) {
+		trace_seq_printf(s, "        - ");
+		trace_seq_printf(s, "        - ");
+		trace_seq_printf(s, "        - ");
+		trace_seq_printf(s, "        -\n");
+	} else {
+		trace_seq_printf(s, "%9llu ", cpu_data->cur_thread / divisor);
+		trace_seq_printf(s, "%9llu ", cpu_data->min_thread / divisor);
+		trace_seq_printf(s, "%9llu ",
+				(cpu_data->sum_thread / cpu_data->thread_count) / divisor);
+		trace_seq_printf(s, "%9llu\n", cpu_data->max_thread / divisor);
+	}
+}
+
+/*
+ * clear_terminal - clears the output terminal
+ */
+static void clear_terminal(struct trace_seq *seq)
+{
+	if (!config_debug)
+		trace_seq_printf(seq, "\033c");
+}
+
+/*
+ * timerlat_print_stats - print data for all cpus
+ */
+static void
+timerlat_print_stats(struct timerlat_top_params *params, struct osnoise_tool *top)
+{
+	struct trace_instance *trace = &top->trace;
+	static int nr_cpus = -1;
+	int i;
+
+	if (nr_cpus == -1)
+		nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
+
+	if (!params->quiet)
+		clear_terminal(trace->seq);
+
+	timerlat_top_header(top);
+
+	for (i = 0; i < nr_cpus; i++) {
+		if (params->cpus && !params->monitored_cpus[i])
+			continue;
+		timerlat_top_print(top, i);
+	}
+
+	trace_seq_do_printf(trace->seq);
+	trace_seq_reset(trace->seq);
+}
+
+/*
+ * timerlat_top_usage - prints timerlat top usage message
+ */
+static void timerlat_top_usage(char *usage)
+{
+	int i;
+
+	static const char *const msg[] = {
+		"",
+		"  usage: rtla timerlat [top] [-h] [-q] [-p us] [-i us] [-t us] [-s us] [-T[=file]] \\",
+		"	  [-c cpu-list] [-P priority]",
+		"",
+		"	  -h/--help: print this menu",
+		"	  -p/--period us: timerlat period in us",
+		"	  -i/--irq us: stop trace if the irq latency is higher than the argument in us",
+		"	  -T/--thread us: stop trace if the thread latency is higher than the argument in us",
+		"	  -s/--stack us: save the stack trace at the IRQ if a thread latency is higher than the argument in us",
+		"	  -c/--cpus cpus: run the tracer only on the given cpus	  -d/--duration time[m|h|d]: duration of the session in seconds",
+		"	  -t/--trace[=file]: save the stopped trace to [file|timerlat_trace.txt]",

tools/tracing/rtla/src/timerlat_top.c line 288

Quoted 10 lines
tools/tracing/rtla/src/timerlat_top.c new file +10 -0
+		"	  -n/--nano: display data in nanoseconds",
+		"	  -q/--quiet print only a summary at the end",
+		"	  -P/--priority o:prio|r:prio|f:prio|d:runtime:period : set scheduling parameters",
+		"		o:prio - use SCHED_OTHER with prio",
+		"		r:prio - use SCHED_RR with prio",
+		"		f:prio - use SCHED_FIFO with prio",
+		"		d:runtime[us|ms|s]:period[us|ms|s] - use SCHED_DEADLINE with runtime and period",
+		"						       in nanoseconds",
+		NULL,
+	};

About help usage msg.

[-t us] --> [-T us]
[-T=[file]] --> [-t[=file]]

[-d] [-D] [-n] lack in the head usage info 'usage: ...'

Quoted 12 lines
tools/tracing/rtla/src/timerlat_top.c new file +12 -0
+
+	if (usage)
+		fprintf(stderr, "%s\n", usage);
+
+	fprintf(stderr, "rtla timerlat top: a per-cpu summary of the timer latency (version %s)\n",
+			VERSION);
+
+	for (i = 0; msg[i]; i++)
+		fprintf(stderr, "%s\n", msg[i]);
+	exit(1);
+}
+
#16 lore raw

Re: [PATCH V8 06/14] rtla: Add timerlat tool and timelart top mode

Quoted 31 lines · Tao Zhou
Quoted 19 lines
tools/tracing/rtla/src/timerlat_top.c new file +19 -0
+/*
+ * timerlat_top_usage - prints timerlat top usage message
+ */
+static void timerlat_top_usage(char *usage)
+{
+	int i;
+
+	static const char *const msg[] = {
+		"",
+		"  usage: rtla timerlat [top] [-h] [-q] [-p us] [-i us] [-t us] [-s us] [-T[=file]] \\",
+		"	  [-c cpu-list] [-P priority]",
+		"",
+		"	  -h/--help: print this menu",
+		"	  -p/--period us: timerlat period in us",
+		"	  -i/--irq us: stop trace if the irq latency is higher than the argument in us",
+		"	  -T/--thread us: stop trace if the thread latency is higher than the argument in us",
+		"	  -s/--stack us: save the stack trace at the IRQ if a thread latency is higher than the argument in us",
+		"	  -c/--cpus cpus: run the tracer only on the given cpus	  -d/--duration time[m|h|d]: duration of the session in seconds",
+		"	  -t/--trace[=file]: save the stopped trace to [file|timerlat_trace.txt]",

tools/tracing/rtla/src/timerlat_top.c line 288

Quoted 10 lines
tools/tracing/rtla/src/timerlat_top.c new file +10 -0
+		"	  -n/--nano: display data in nanoseconds",
+		"	  -q/--quiet print only a summary at the end",
+		"	  -P/--priority o:prio|r:prio|f:prio|d:runtime:period : set scheduling parameters",
+		"		o:prio - use SCHED_OTHER with prio",
+		"		r:prio - use SCHED_RR with prio",
+		"		f:prio - use SCHED_FIFO with prio",
+		"		d:runtime[us|ms|s]:period[us|ms|s] - use SCHED_DEADLINE with runtime and period",
+		"						       in nanoseconds",
+		NULL,
+	};

About help usage msg.

Quoted 5 lines

[-t us] --> [-T us]
[-T=[file]] --> [-t[=file]]

[-d] [-D] [-n] lack in the head usage info 'usage: ...'

Fixed in V9

-- Daniel

#17 lore raw

PATCH v8 7/14 rtla/timerlat: Add timerlat hist mode

In reply to Daniel Bristot de Oliveira

The rtla hist hist mode displays a histogram of each tracer event occurrence, both for IRQ and timer latencies. The tool also allows many configurations of the timerlat tracer and the collection of the tracer output.

Here is one example of the rtla timerlat hist mode output:
---------- %< ----------
[root@alien ~]# rtla timerlat hist -c 0-3 -d 1M # RTLA timerlat histogram
# Time unit is microseconds (us)
# Duration: 0 00:01:00
Index IRQ-000 Thr-000 IRQ-001 Thr-001 IRQ-002 Thr-002 IRQ-003 Thr-003
0 58572 0 59373 0 58691 0 58895 0
1 1422 57021 628 57241 1310 56160 1102 56805

 2             6      2931         0      2695         0      3567         4      3031
 3             1        40         0        53         0       260         0       142
 4             0         7         0         5         0         6         0        17
 5             0         2         0         5         0         7         0         4
 6             0         0         0         2         0         1         0         1
 8             0         0         0         0         0         0         0         1
 over:         0         0         0         0         0         0         0         0
 count:    60001     60001     60001     60001     60001     60001     60001     60001
 min:          0         1         0         1         0         1         0         1
 avg:          0         1         0         1         0         1         0         1
 max:          3         5         1         6         1         6         2         8

---------- >% ----------

Running
- rtla timerlat hist --help
provides information about the available options.

Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 tools/tracing/rtla/src/timerlat.c      |   6 +-
 tools/tracing/rtla/src/timerlat.h      |   2 +-
 tools/tracing/rtla/src/timerlat_hist.c | 820 +++++++++++++++++++++++++
 3 files changed, 826 insertions(+), 2 deletions(-)
 create mode 100644 tools/tracing/rtla/src/timerlat_hist.c
tools/tracing/rtla/src/timerlat.c +5 -1
index 29a51c01b084..97abbf494fee 100644
@@ -25,7 +25,8 @@ static void timerlat_usage(void)
 		"  usage: [rtla] timerlat [MODE] ...",
 		"",
 		"  modes:",
-		"     top  - prints the summary from timerlat tracer",
+		"     top   - prints the summary from timerlat tracer",
+		"     hist  - prints a histogram of timer latencies",
 		"",
 		"if no MODE is given, the top mode is called, passing the arguments",
 		NULL,
@@ -60,6 +61,9 @@ int timerlat_main(int argc, char *argv[])
 	} else if (strcmp(argv[1], "top") == 0) {
 		timerlat_top_main(argc-1, &argv[1]);
 		exit(0);
+	} else if (strcmp(argv[1], "hist") == 0) {
+		timerlat_hist_main(argc-1, &argv[1]);
+		exit(0);
 	}
 
 usage:
tools/tracing/rtla/src/timerlat.h +1 -1
index 6f9d3925acd0..88561bfd14f3 100644
@@ -1,4 +1,4 @@
 // SPDX-License-Identifier: GPL-2.0
-
+int timerlat_hist_main(int argc, char *argv[]);
 int timerlat_top_main(int argc, char *argv[]);
 int timerlat_main(int argc, char *argv[]);
tools/tracing/rtla/src/timerlat_hist.c new file +820 -0

820 lines not on this page · raw message

Signature · 1 line
2.31.1
#18 lore raw

Re: [PATCH V8 07/14] rtla/timerlat: Add timerlat hist mode

Quoted 25 lines · Daniel Bristot de Oliveira
tools/tracing/rtla/src/timerlat_hist.c new file +25 -0
+static void timerlat_hist_usage(char *usage)
+{
+	int i;
+
+	char *msg[] = {
+		"",
+		"  usage: [rtla] timerlat hist [-h] [-q] [-p us] [-i us] [-t us] [-s us] [-T[=file]] \\",
+		"         [-c cpu-list] [-P priority] [--no-irq] [--no-thread] [--no-header] [--no-summary] \\",
+		"         [--no-index] [--with-zeros]",
+		"",
+		"	  -h/--help: print this menu",
+		"	  -p/--period us: timerlat period in us",
+		"	  -i/--irq us: stop trace if the irq latency is higher than the argument in us",
+		"	  -T/--thread us: stop trace if the thread latency is higher than the argument in us",
+		"	  -s/--stack us: save the stack trace at the IRQ if a thread latency is higher than the argument in us",
+		"	  -c/--cpus cpus: run the tracer only on the given cpus",
+		"	  -d/--duration time[m|h|d]: duration of the session in seconds",
+		"	  -T/--trace[=file]: save the stopped trace to [file|timerlat_trace.txt]",
+		"	  -n/--nano: display data in nanoseconds",
+		"	  -b/--bucket-size N: set the histogram bucket size (default 1)",
+		"	  -e/--entries N: set the number of entries of the histogram (default 256)",
+		"	     --no-irq: ignore IRQ latencies",
+		"	     --no-thread: ignore thread latencies",
+		"	     --no-header: do not print header",
+		"	     --no-summary: do not print summary",

tools/tracing/rtla/src/timerlat_hist.c line 459

Quoted 10 lines
tools/tracing/rtla/src/timerlat_hist.c new file +10 -0
+		"	     --no-index: do not print index",
+		"	     --with-zeros: print zero only entries",
+		"	  -P/--priority o:prio|r:prio|f:prio|d:runtime:period : set scheduling parameters",
+		"		o:prio - use SCHED_OTHER with prio",
+		"		r:prio - use SCHED_RR with prio",
+		"		f:prio - use SCHED_FIFO with prio",
+		"		d:runtime[us|ms|s]:period[us|ms|s] - use SCHED_DEADLINE with runtime and period",
+		"						       in nanoseconds",
+		NULL,
+	};

[-t us] --> [-T us]
[-T[=file]] --> [-t[=file]]
[-d] [-b] [-e] [-n] lack in usage info. this is not important tho.

#19 lore raw

Re: [PATCH V8 07/14] rtla/timerlat: Add timerlat hist mode

Quoted 34 lines · Tao Zhou
Quoted 31 lines · Daniel Bristot de Oliveira
tools/tracing/rtla/src/timerlat_hist.c new file +31 -0
+static void timerlat_hist_usage(char *usage)
+{
+	int i;
+
+	char *msg[] = {
+		"",
+		"  usage: [rtla] timerlat hist [-h] [-q] [-p us] [-i us] [-t us] [-s us] [-T[=file]] \\",
+		"         [-c cpu-list] [-P priority] [--no-irq] [--no-thread] [--no-header] [--no-summary] \\",
+		"         [--no-index] [--with-zeros]",
+		"",
+		"	  -h/--help: print this menu",
+		"	  -p/--period us: timerlat period in us",
+		"	  -i/--irq us: stop trace if the irq latency is higher than the argument in us",
+		"	  -T/--thread us: stop trace if the thread latency is higher than the argument in us",
+		"	  -s/--stack us: save the stack trace at the IRQ if a thread latency is higher than the argument in us",
+		"	  -c/--cpus cpus: run the tracer only on the given cpus",
+		"	  -d/--duration time[m|h|d]: duration of the session in seconds",
+		"	  -T/--trace[=file]: save the stopped trace to [file|timerlat_trace.txt]",
+		"	  -n/--nano: display data in nanoseconds",
+		"	  -b/--bucket-size N: set the histogram bucket size (default 1)",
+		"	  -e/--entries N: set the number of entries of the histogram (default 256)",
+		"	     --no-irq: ignore IRQ latencies",
+		"	     --no-thread: ignore thread latencies",
+		"	     --no-header: do not print header",
+		"	     --no-summary: do not print summary",
+		"	     --no-index: do not print index",
+		"	     --with-zeros: print zero only entries",
+		"	  -P/--priority o:prio|r:prio|f:prio|d:runtime:period : set scheduling parameters",
+		"		o:prio - use SCHED_OTHER with prio",
+		"		r:prio - use SCHED_RR with prio",
+		"		f:prio - use SCHED_FIFO with prio",
Quoted 9 lines

tools/tracing/rtla/src/timerlat_hist.c line 459

Quoted 4 lines
tools/tracing/rtla/src/timerlat_hist.c new file +4 -0
+		"		d:runtime[us|ms|s]:period[us|ms|s] - use SCHED_DEADLINE with runtime and period",
+		"						       in nanoseconds",
+		NULL,
+	};

[-t us] --> [-T us]
[-T[=file]] --> [-t[=file]]
[-d] [-b] [-e] [-n] lack in usage info. this is not important tho.

Fixed in V9

-- Daniel

#20 lore raw

PATCH v8 8/14 rtla: Add Documentation

In reply to Daniel Bristot de Oliveira

Adds the basis for rtla documentation. This patch also includes the rtla(1) man page.

As suggested by Jonathan Corbet, we are placing these man pages at Documentation/tools/rtla, using rst format. It is not linked to the official documentation, though.

The Makefile is based on bpftool's Documentation one.

Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 Documentation/tools/rtla/Makefile            | 41 +++++++++++++++++
 Documentation/tools/rtla/common_appendix.rst | 12 +++++
 Documentation/tools/rtla/rtla.rst            | 48 ++++++++++++++++++++
 tools/tracing/rtla/Makefile                  | 30 ++++++++++--
 4 files changed, 127 insertions(+), 4 deletions(-)
 create mode 100644 Documentation/tools/rtla/Makefile
 create mode 100644 Documentation/tools/rtla/common_appendix.rst
 create mode 100644 Documentation/tools/rtla/rtla.rst
Documentation/tools/rtla/Makefile new file +41 -0
new file mode 100644
index 000000000000..9f2b84af1a6c
@@ -0,0 +1,41 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# Based on bpftool's Documentation Makefile
+
+INSTALL		?= install
+RM		?= rm -f
+RMDIR		?= rmdir --ignore-fail-on-non-empty
+
+PREFIX		?= /usr/share
+MANDIR		?= $(PREFIX)/man
+MAN1DIR		= $(MANDIR)/man1
+
+MAN1_RST	= $(wildcard rtla*.rst)
+
+_DOC_MAN1	= $(patsubst %.rst,%.1,$(MAN1_RST))
+DOC_MAN1	= $(addprefix $(OUTPUT),$(_DOC_MAN1))
+
+RST2MAN_DEP	:= $(shell command -v rst2man 2>/dev/null)
+RST2MAN_OPTS	+= --verbose
+
+$(OUTPUT)%.1: %.rst
+ifndef RST2MAN_DEP
+	$(error "rst2man not found, but required to generate man pages")
+endif
+	rst2man $(RST2MAN_OPTS) $< > $@
+
+man1: $(DOC_MAN1)
+man: man1
+
+clean:
+	$(RM) $(DOC_MAN1)
+
+install: man
+	$(INSTALL) -d -m 755 $(DESTDIR)$(MAN1DIR)
+	$(INSTALL) -m 644 $(DOC_MAN1) $(DESTDIR)$(MAN1DIR)
+
+uninstall:
+	$(RM) $(addprefix $(DESTDIR)$(MAN1DIR)/,$(_DOC_MAN1))
+	$(RMDIR) $(DESTDIR)$(MAN1DIR)
+
+.PHONY: man man1 clean install uninstall
+.DEFAULT_GOAL := man
Documentation/tools/rtla/common_appendix.rst new file +12 -0
new file mode 100644
index 000000000000..b494084acccd
@@ -0,0 +1,12 @@
+REPORTING BUGS
+==============
+Report bugs to <[email protected]>
+
+LICENSE
+=======
+**rtla** is Free Software licensed under the GNU GPLv2
+
+COPYING
+=======
+Copyright \(C) 2021 Red Hat, Inc. Free use of this software is granted under
+the terms of the GNU Public License (GPL).
Documentation/tools/rtla/rtla.rst new file +48 -0

48 lines not on this page · raw message

tools/tracing/rtla/Makefile +26 -4
index f254cfbe4fa3..8b9bee124eb3 100644
@@ -45,18 +45,28 @@ DATADIR	:=	/usr/share
 DOCDIR	:=	$(DATADIR)/doc
 MANDIR	:=	$(DATADIR)/man
 LICDIR	:=	$(DATADIR)/licenses
+SRCTREE	:=	$(if $(BUILD_SRC),$(BUILD_SRC),$(CURDIR))
+
+# If running from the tarball, man pages are stored in the Documentation
+# dir. If running from the kernel source, man pages are stored in
+# Documentation/tools/rtla/.
+ifneq ($(wildcard Documentation/.*),)
+DOCSRC	=	Documentation/
+else
+DOCSRC	=	$(SRCTREE)/../../../Documentation/tools/rtla/
+endif
 
 .PHONY:	all
 all:	rtla
 
-rtla: $(OBJ)
+rtla: $(OBJ) doc
 	$(CC) -o rtla $(LDFLAGS) $(OBJ) $(LIBS)
 
 static: $(OBJ)
 	$(CC) -o rtla-static $(LDFLAGS) --static $(OBJ) $(LIBS) -lpthread -ldl
 
 .PHONY: install
-install:
+install: doc_install
 	$(INSTALL) -d -m 755 $(DESTDIR)$(BINDIR)
 	$(INSTALL) rtla -m 755 $(DESTDIR)$(BINDIR)
 	$(STRIP) $(DESTDIR)$(BINDIR)/rtla
@@ -66,15 +76,27 @@ install:
 	ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/timerlat
 
 .PHONY: clean tarball
-clean:
+clean: doc_clean
 	@test ! -f rtla || rm rtla
 	@test ! -f rtla-static || rm rtla-static
 	@test ! -f src/rtla.o || rm src/rtla.o
 	@test ! -f $(TARBALL) || rm -f $(TARBALL)
 	@rm -rf *~ $(OBJ) *.tar.$(CEXT)
 
-tarball:  clean
+tarball: clean
 	rm -rf $(NAME)-$(VERSION) && mkdir $(NAME)-$(VERSION)
 	cp -r $(DIRS) $(FILES) $(NAME)-$(VERSION)
+	mkdir $(NAME)-$(VERSION)/Documentation/
+	cp -rp $(SRCTREE)/../../../Documentation/tools/rtla/* $(NAME)-$(VERSION)/Documentation/
 	tar $(TAROPTS) --exclude='*~' $(NAME)-$(VERSION)
 	rm -rf $(NAME)-$(VERSION)
+
+.PHONY: doc doc_clean doc_install
+doc:
+	$(MAKE) -C $(DOCSRC)
+
+doc_clean:
+	$(MAKE) -C $(DOCSRC) clean
+
+doc_install:
+	$(MAKE) -C $(DOCSRC) install
Signature · 1 line
2.31.1
#21 lore raw

PATCH v8 9/14 rtla: Add rtla osnoise man page

In reply to Daniel Bristot de Oliveira

Man page for rtla osnoise command.

Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 .../tools/rtla/common_osnoise_description.rst |  8 +++
 Documentation/tools/rtla/rtla-osnoise.rst     | 59 +++++++++++++++++++
 2 files changed, 67 insertions(+)
 create mode 100644 Documentation/tools/rtla/common_osnoise_description.rst
 create mode 100644 Documentation/tools/rtla/rtla-osnoise.rst
Documentation/tools/rtla/common_osnoise_description.rst new file +8 -0
new file mode 100644
index 000000000000..8973c5df888f
@@ -0,0 +1,8 @@
+The **rtla osnoise** tool is an interface for the *osnoise* tracer. The
+*osnoise* tracer dispatches a kernel thread per-cpu. These threads read the
+time in a loop while with preemption, softirq and IRQs enabled, thus
+allowing all the sources of operating systme noise during its execution.
+The *osnoise*'s tracer threads take note of the delta between each time
+read, along with an interference counter of all sources of interference.
+At the end of each period, the *osnoise* tracer displays a summary of
+the results.
Documentation/tools/rtla/rtla-osnoise.rst new file +59 -0

59 lines not on this page · raw message

Signature · 1 line
2.31.1
#22 lore raw

PATCH v8 10/14 rtla: Add rtla osnoise top documentation

In reply to Daniel Bristot de Oliveira

Man page for rtla osnoise top mode.

Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 Documentation/tools/rtla/common_options.rst   | 24 ++++++++
 .../tools/rtla/common_osnoise_options.rst     | 17 ++++++
 .../tools/rtla/common_top_options.rst         |  3 +
 Documentation/tools/rtla/rtla-osnoise-top.rst | 61 +++++++++++++++++++
 4 files changed, 105 insertions(+)
 create mode 100644 Documentation/tools/rtla/common_options.rst
 create mode 100644 Documentation/tools/rtla/common_osnoise_options.rst
 create mode 100644 Documentation/tools/rtla/common_top_options.rst
 create mode 100644 Documentation/tools/rtla/rtla-osnoise-top.rst
Documentation/tools/rtla/common_options.rst new file +24 -0
new file mode 100644
index 000000000000..ea6529667e7c
@@ -0,0 +1,24 @@
+**-c**, **--cpus** *cpu-list*
+
+        Set the osnoise tracer to run the sample threads in the cpu-list.
+
+**-d**, **--duration** *time[s|m|h|d]*
+
+        Set the duration of the session.
+
+**-t**, **--trace**\[*=file*]
+
+        Save the stopped trace to [*file|osnoise_trace.txt*].
+
+**-P**, **--priority** *o:prio|r:prio|f:prio|d:runtime:period*
+
+        Set scheduling parameters to the osnoise tracer threads, the format to set the priority are:
+
+        - *o:prio* - use SCHED_OTHER with *prio*;
+        - *r:prio* - use SCHED_RR with *prio*;
+        - *f:prio* - use SCHED_FIFO with *prio*;
+        - *d:runtime[us|ms|s]:period[us|ms|s]* - use SCHED_DEADLINE with *runtime* and *period* in nanoseconds.
+
+**-h**, **--help**
+
+        Print help menu.
Documentation/tools/rtla/common_osnoise_options.rst new file +17 -0
new file mode 100644
index 000000000000..d556883e4e26
@@ -0,0 +1,17 @@
+**-p**, **--period** *us*
+
+        Set the *osnoise* tracer period in microseconds.
+
+**-r**, **--runtime** *us*
+
+        Set the *osnoise* tracer runtime in microseconds.
+
+**-s**, **--stop** *us*
+
+        Stop the trace if a single sample is higher than the argument in microseconds.
+        If **-T** is set, it will also save the trace to the output.
+
+**-S**, **--stop-total** *us*
+
+        Stop the trace if the total sample is higher than the argument in microseconds.
+        If **-T** is set, it will also save the trace to the output.
Documentation/tools/rtla/common_top_options.rst new file +3 -0
new file mode 100644
index 000000000000..f48878938f84
@@ -0,0 +1,3 @@
+**-q**, **--quiet**
+
+        Print only a summary at the end of the session.
Documentation/tools/rtla/rtla-osnoise-top.rst new file +61 -0

61 lines not on this page · raw message

Signature · 1 line
2.31.1
#23 lore raw

PATCH v8 11/14 rtla: Add rtla osnoise hist documentation

In reply to Daniel Bristot de Oliveira

Man page for rtla osnoise hist mode.

Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 .../tools/rtla/common_hist_options.rst        | 23 +++++++
 .../tools/rtla/rtla-osnoise-hist.rst          | 66 +++++++++++++++++++
 2 files changed, 89 insertions(+)
 create mode 100644 Documentation/tools/rtla/common_hist_options.rst
 create mode 100644 Documentation/tools/rtla/rtla-osnoise-hist.rst
Documentation/tools/rtla/common_hist_options.rst new file +23 -0
new file mode 100644
index 000000000000..0266cd08a6c9
@@ -0,0 +1,23 @@
+**-b**, **--bucket-size** *N*
+
+        Set the histogram bucket size (default *1*).
+
+**-e**, **--entries** *N*
+
+        Set the number of entries of the histogram (default 256).
+
+**--no-header**
+
+        Do not print header.
+
+**--no-summary**
+
+        Do not print summary.
+
+**--no-index**
+
+        Do not print index.
+
+**--with-zeros**
+
+        Print zero only entries.
Documentation/tools/rtla/rtla-osnoise-hist.rst new file +66 -0

66 lines not on this page · raw message

Signature · 1 line
2.31.1
#24 lore raw

PATCH v8 12/14 rtla: Add rtla timerlat documentation

In reply to Daniel Bristot de Oliveira

Man page for rtla timerlat tool.

Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 .../rtla/common_timerlat_description.rst      | 10 ++++
 Documentation/tools/rtla/rtla-timerlat.rst    | 57 +++++++++++++++++++
 2 files changed, 67 insertions(+)
 create mode 100644 Documentation/tools/rtla/common_timerlat_description.rst
 create mode 100644 Documentation/tools/rtla/rtla-timerlat.rst
Documentation/tools/rtla/common_timerlat_description.rst new file +10 -0
new file mode 100644
index 000000000000..321201cb8597
@@ -0,0 +1,10 @@
+The **rtla timerlat** tool is an interface for the *timerlat* tracer. The
+*timerlat* tracer dispatches a kernel thread per-cpu. These threads
+set a periodic timer to wake themselves up and go back to sleep. After
+the wakeup, they collect and generate useful information for the
+debugging of operating system timer latency.
+
+The *timerlat* tracer outputs information in two ways. It periodically
+prints the timer latency at the timer *IRQ* handler and the *Thread*
+handler. It also enable the trace of the most relevant information via
+**osnoise:** tracepoints.
Documentation/tools/rtla/rtla-timerlat.rst new file +57 -0

57 lines not on this page · raw message

Signature · 1 line
2.31.1
#25 lore raw

PATCH v8 13/14 rtla: Add rtla timerlat top documentation

In reply to Daniel Bristot de Oliveira

Man page for rtla timerlat top mode.

Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 .../tools/rtla/common_timerlat_options.rst    |  16 ++
 .../tools/rtla/rtla-timerlat-top.rst          | 145 ++++++++++++++++++
 2 files changed, 161 insertions(+)
 create mode 100644 Documentation/tools/rtla/common_timerlat_options.rst
 create mode 100644 Documentation/tools/rtla/rtla-timerlat-top.rst
Documentation/tools/rtla/common_timerlat_options.rst new file +16 -0
new file mode 100644
index 000000000000..e9c1bfd55d48
@@ -0,0 +1,16 @@
+**-p**, **--period** *us*
+
+        Set the *timerlat* tracer period in microseconds.
+
+**-i**, **--irq** *us*
+
+        Stop trace if the *IRQ* latency is higher than the argument in us.
+
+**-T**, **--thread** *us*
+
+        Stop trace if the *Thread* latency is higher than the argument in us.
+
+**-s**, **--stack** *us*
+
+        Save the stack trace at the *IRQ* if a *Thread* latency is higher than the
+        argument in us.
Documentation/tools/rtla/rtla-timerlat-top.rst new file +145 -0

145 lines not on this page · raw message

Signature · 1 line
2.31.1
#26 lore raw

PATCH v8 14/14 rtla: Add rtla timerlat hist documentation

In reply to Daniel Bristot de Oliveira

Man page for rtla timerlat hist mode.

Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tom Zanussi <[email protected]>
Cc Masami Hiramatsu <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Clark Williams <[email protected]>
Cc John Kacur <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
---
 .../tools/rtla/rtla-timerlat-hist.rst         | 106 ++++++++++++++++++
 1 file changed, 106 insertions(+)
 create mode 100644 Documentation/tools/rtla/rtla-timerlat-hist.rst
Documentation/tools/rtla/rtla-timerlat-hist.rst new file +106 -0
new file mode 100644
index 000000000000..e12eae1f3301
@@ -0,0 +1,106 @@
+=====================
+rtla-timerlat-hist
+=====================
+------------------------------------------------
+Histograms of the operating system timer latency
+------------------------------------------------
+
+:Manual section: 1
+
+SYNOPSIS
+========
+**rtla timerlat hist** [*OPTIONS*] ...
+
+DESCRIPTION
+===========
+
+.. include:: common_timerlat_description.rst
+
+The **rtla timerlat hist** displays a histogram of each tracer event
+occurrence. This tool uses the periodic information, and the
+**osnoise:** tracepoints are enabled when using the **-T** option.
+
+OPTIONS
+=======
+
+.. include:: common_timerlat_options.rst
+
+.. include:: common_hist_options.rst
+
+.. include:: common_options.rst
+
+EXAMPLE
+=======
+In the example below, **rtla timerlat hist** is set to run for *10* minutes,
+in the cpus *0-4*, *skipping zero* only lines. Moreover, **rtla timerlat
+hist** will change the priority of the *timelat* threads to run under
+*SCHED_DEADLINE* priority, with a *10us* runtime every *1ms* period. The
+*1ms* period is also passed to the *timerlat* tracer::
+
+  [root@alien ~]# timerlat hist -d 10m -c 0-4 -P d:100us:1ms -p 1ms
+  # RTLA timerlat histogram
+  # Time unit is microseconds (us)
+  # Duration:   0 00:10:00
+  Index   IRQ-000   Thr-000   IRQ-001   Thr-001   IRQ-002   Thr-002   IRQ-003   Thr-003   IRQ-004   Thr-004
+  0        276489         0    206089         0    466018         0    481102         0    205546         0
+  1        318327     35487    388149     30024     94531     48382     83082     71078    388026     55730
+  2          3282    122584      4019    126527     28231    109012     23311     89309      4568     98739
+  3           940     11815       837      9863      6209     16227      6895     17196       910      9780
+  4           444     17287       424     11574      2097     38443      2169     36736       462     13476
+  5           206     43291       255     25581      1223    101908      1304    101137       236     28913
+  6           132    101501        96     64584       635    213774       757    215471        99     73453
+  7            74    169347        65    124758       350     57466       441     53639        69    148573
+  8            53     85183        31    156751       229      9052       306      9026        39    139907
+  9            22     10387        12     42762       161      2554       225      2689        19     26192
+  10           13      1898         8      5770       114      1247       128      1405        13      3772
+  11            9       560         9       924        71       686        76       765         8       713
+  12            4       256         2       360        50       411        64       474         3       278
+  13            2       167         2       172        43       256        53       350         4       180
+  14            1        88         1       116        15       198        42       223         0       115
+  15            2        63         3        94        11       139        20       150         0        58
+  16            2        37         0        56         5        78        10       102         0        39
+  17            0        18         0        28         4        57         8        80         0        15
+  18            0         8         0        17         2        50         6        56         0        12
+  19            0         9         0         5         0        19         0        48         0        18
+  20            0         4         0         8         0        11         2        27         0         4
+  21            0         2         0         3         1         9         1        18         0         6
+  22            0         1         0         3         1         7         0         3         0         5
+  23            0         2         0         4         0         2         0         7         0         2
+  24            0         2         0         2         1         3         0         3         0         5
+  25            0         0         0         1         0         1         0         1         0         3
+  26            0         1         0         0         0         2         0         2         0         0
+  27            0         0         0         3         0         1         0         0         0         1
+  28            0         0         0         3         0         0         0         1         0         0
+  29            0         0         0         2         0         2         0         1         0         3
+  30            0         1         0         0         0         0         0         0         0         0
+  31            0         1         0         0         0         0         0         2         0         2
+  32            0         0         0         1         0         2         0         0         0         0
+  33            0         0         0         2         0         0         0         0         0         1
+  34            0         0         0         0         0         0         0         0         0         2
+  35            0         1         0         1         0         0         0         0         0         1
+  36            0         1         0         0         0         1         0         1         0         0
+  37            0         0         0         1         0         0         0         0         0         0
+  40            0         0         0         0         0         1         0         1         0         0
+  41            0         0         0         0         0         0         0         0         0         1
+  42            0         0         0         0         0         0         0         0         0         1
+  44            0         0         0         0         0         1         0         0         0         0
+  46            0         0         0         0         0         0         0         1         0         0
+  47            0         0         0         0         0         0         0         0         0         1
+  50            0         0         0         0         0         0         0         0         0         1
+  54            0         0         0         1         0         0         0         0         0         0
+  58            0         0         0         1         0         0         0         0         0         0
+  over:         0         0         0         0         0         0         0         0         0         0
+  count:   600002    600002    600002    600002    600002    600002    600002    600002    600002    600002
+  min:          0         1         0         1         0         1         0         1         0         1
+  avg:          0         5         0         5         0         4         0         4         0         5
+  max:         16        36        15        58        24        44        21        46        13        50
+
+SEE ALSO
+========
+**rtla-timerlat**\(1), **rtla-timerlat-top**\(1)
+
+*timerlat* tracer documentation: <https://www.kernel.org/doc/html/latest/trace/timerlat-tracer.html>
+
+AUTHOR
+======
+Written by Daniel Bristot de Oliveira <[email protected]>
Signature · 1 line
2.31.1
#27 lore raw
In reply to Daniel Bristot de Oliveira
Quoted 14 lines · Daniel Bristot de Oliveira

The rtla(1) is a meta-tool that includes a set of commands that aims to analyze the real-time properties of Linux. But instead of testing Linux as a black box, rtla leverages kernel tracing capabilities to provide precise information about the properties and root causes of unexpected results.

To start, it presents an interface to the osnoise and timerlat tracers. In the future, it will also serve as home to the rtsl [1] and other latency/noise tracers.

If you just want to run it, you can download the tarball here:
- https://bristot.me/files/rtla/tarball/rtla-0.4.tar.bz2

To compile rtla on fedora you need:

Quoted 14 lines
  $ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.git
  $ cd libtraceevent/
  $ make
  $ sudo make install
  $ cd ..
  $ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtracefs.git
  $ cd libtracefs/
  $ make
  $ sudo make install
  $ cd ..
  $ sudo dnf install python3-docutils procps-devel
  $ cd $rtla_src
  $ make
  $ sudo make install

Set osnoise/x to DEADLINE, the return is not success. see:

tao@geo ~/opensource/rtla-0.4 $ sudo rtla osnoise top -P d:100us:1ms -c 0-3 -r 900000 -d 1M -q boost_with_deadline failed to boost pid 4766: Operation not permitted
Failed to set sched parameters

top shows:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND         
 4773 root      20   0       0      0      0 R  89.4   0.0   2:06.88 osnoise/0       
 4776 root      20   0       0      0      0 R  89.4   0.0   2:06.88 osnoise/3       
 4774 root      20   0       0      0      0 R  89.0   0.0   2:06.88 osnoise/1       
 4775 root      20   0       0      0      0 R  88.7   0.0   2:06.69 osnoise/2       

t

#28 lore raw
Quoted 31 lines · Tao Zhou
Quoted 14 lines · Daniel Bristot de Oliveira

The rtla(1) is a meta-tool that includes a set of commands that aims to analyze the real-time properties of Linux. But instead of testing Linux as a black box, rtla leverages kernel tracing capabilities to provide precise information about the properties and root causes of unexpected results.

To start, it presents an interface to the osnoise and timerlat tracers. In the future, it will also serve as home to the rtsl [1] and other latency/noise tracers.

If you just want to run it, you can download the tarball here:
- https://bristot.me/files/rtla/tarball/rtla-0.4.tar.bz2

To compile rtla on fedora you need:

Quoted 14 lines
  $ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.git
  $ cd libtraceevent/
  $ make
  $ sudo make install
  $ cd ..
  $ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtracefs.git
  $ cd libtracefs/
  $ make
  $ sudo make install
  $ cd ..
  $ sudo dnf install python3-docutils procps-devel
  $ cd $rtla_src
  $ make
  $ sudo make install
Quoted 5 lines

Set osnoise/x to DEADLINE, the return is not success. see:

tao@geo ~/opensource/rtla-0.4 $ sudo rtla osnoise top -P d:100us:1ms -c 0-3 -r 900000 -d 1M -q boost_with_deadline failed to boost pid 4766: Operation not permitted
Failed to set sched parameters

Did you disable the admission control?

e.g.,

  $ sudo sysctl -w kernel.sched_rt_runtime_us=-1

-- Daniel