RTLA: An interface for osnoise/timerlat tracers
Subsystems (2)
v7 cover RTLA: An interface for osnoise/timerlat tracers
The rtla(1) is a meta-tool that includes a set of commands that aims to analyze the real-time properties of Linux. But instead of testing Linux as a black box, rtla leverages kernel tracing capabilities to provide precise information about the properties and root causes of unexpected results.
To start, it presents an interface to the osnoise and timerlat tracers. In the future, it will also serve as home to the rtsl [1] and other latency/noise tracers.
If you just want to run it, you can download the tarball here:
- https://bristot.me/files/rtla/tarball/rtla-0.3.tar.bz2
To compile rtla on fedora you need:
$ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.git $ cd libtraceevent/ $ make $ sudo make install $ cd .. $ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtracefs.git $ cd libtracefs/ $ make $ sudo make install $ cd .. $ sudo dnf install python3-docutils procps-devel $ cd $rtla_src $ make $ sudo make install
The tracing option (-t) depends some kernel patches that are available at [2].
This tool was be discussed at the RT-MC during LPC2021 [3]
[1] rtsl: https://github.com/bristot/rtsl/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://youtu.be/cZUzc0U1jJ4
Changes from v6:
- Revisited osnoise option config functions
- Properly handles offline CPUs
- Some cleanups
- Fixed an histogram allocation problem (Tao Zhou)
- Revisited open()/read()/write() (Tao Zhou)
Changes from v5:
- Fix retval check in save_trace_to_file() (Tao Zhou)
- Fix goto logic in save_trace_to_file() (Tao Zhou)
- Removed unused save_trace_pipe_to_file() function
- Correctly handle an error on osnoise_set_* functions during "apply config" for all tools (Tao Zhou)
Changes from v3:
- Add cross-compile support (Ahmed S. Darwish)
- Move documentation to Documentation/tools/rtla (Jonathan Corbet)
- Use .rst format for documentation (Jonathan Corbet)
- Use include option from .rst to group common parts of the documentation
- Makefile (main and doc) cleanups
Changes from v2:
- Fix the miss conception of the "size" for kernel histograms (Steven/Tom)
- Change the --skip-zeros to --with-zeros option as the former is better for humans (and the latter for computers to plot charts).
- A lot of checkpatch fixes for the user-space part.
Changes from v1:
- Fixes -t options on osnoise tracers (-t means --trace for all tools now)
- Fixes --bucket-size references (not --bucket_size)
Daniel Bristot de Oliveira (14):
rtla: Real-Time Linux Analysis tool
rtla: Helper functions for rtla
rtla: Add osnoise tool
rtla/osnoise: Add osnoise top mode
rtla/osnoise: Add the hist mode
rtla: Add timerlat tool and timelart top mode
rtla/timerlat: Add timerlat hist mode
rtla: Add Documentation
rtla: Add rtla osnoise man page
rtla: Add rtla osnoise top documentation
rtla: Add rtla osnoise hist documentation
rtla: Add rtla timerlat documentation
rtla: Add rtla timerlat top documentation
rtla: Add rtla timerlat hist documentation
Documentation/tools/rtla/Makefile | 41 + Documentation/tools/rtla/common_appendix.rst | 12 + .../tools/rtla/common_hist_options.rst | 23 + Documentation/tools/rtla/common_options.rst | 24 + .../tools/rtla/common_osnoise_description.rst | 8 + .../tools/rtla/common_osnoise_options.rst | 17 + .../rtla/common_timerlat_description.rst | 10 + .../tools/rtla/common_timerlat_options.rst | 16 + .../tools/rtla/common_top_options.rst | 3 + .../tools/rtla/rtla-osnoise-hist.rst | 66 ++ Documentation/tools/rtla/rtla-osnoise-top.rst | 61 + Documentation/tools/rtla/rtla-osnoise.rst | 59 + .../tools/rtla/rtla-timerlat-hist.rst | 106 ++ .../tools/rtla/rtla-timerlat-top.rst | 145 +++ Documentation/tools/rtla/rtla-timerlat.rst | 57 + Documentation/tools/rtla/rtla.rst | 48 + tools/tracing/rtla/Makefile | 102 ++ tools/tracing/rtla/src/osnoise.c | 1017 +++++++++++++++++ tools/tracing/rtla/src/osnoise.h | 96 ++ tools/tracing/rtla/src/osnoise_hist.c | 799 +++++++++++++ tools/tracing/rtla/src/osnoise_top.c | 577 ++++++++++ tools/tracing/rtla/src/rtla.c | 87 ++ tools/tracing/rtla/src/timerlat.c | 72 ++ tools/tracing/rtla/src/timerlat.h | 4 + tools/tracing/rtla/src/timerlat_hist.c | 820 +++++++++++++ tools/tracing/rtla/src/timerlat_top.c | 615 ++++++++++ tools/tracing/rtla/src/trace.c | 192 ++++ tools/tracing/rtla/src/trace.h | 27 + tools/tracing/rtla/src/utils.c | 433 +++++++ tools/tracing/rtla/src/utils.h | 56 + 30 files changed, 5593 insertions(+) create mode 100644 Documentation/tools/rtla/Makefile create mode 100644 Documentation/tools/rtla/common_appendix.rst create mode 100644 Documentation/tools/rtla/common_hist_options.rst create mode 100644 Documentation/tools/rtla/common_options.rst create mode 100644 Documentation/tools/rtla/common_osnoise_description.rst create mode 100644 Documentation/tools/rtla/common_osnoise_options.rst create mode 100644 Documentation/tools/rtla/common_timerlat_description.rst create mode 100644 Documentation/tools/rtla/common_timerlat_options.rst create mode 100644 Documentation/tools/rtla/common_top_options.rst create mode 100644 Documentation/tools/rtla/rtla-osnoise-hist.rst create mode 100644 Documentation/tools/rtla/rtla-osnoise-top.rst create mode 100644 Documentation/tools/rtla/rtla-osnoise.rst create mode 100644 Documentation/tools/rtla/rtla-timerlat-hist.rst create mode 100644 Documentation/tools/rtla/rtla-timerlat-top.rst create mode 100644 Documentation/tools/rtla/rtla-timerlat.rst create mode 100644 Documentation/tools/rtla/rtla.rst create mode 100644 tools/tracing/rtla/Makefile create mode 100644 tools/tracing/rtla/src/osnoise.c create mode 100644 tools/tracing/rtla/src/osnoise.h create mode 100644 tools/tracing/rtla/src/osnoise_hist.c create mode 100644 tools/tracing/rtla/src/osnoise_top.c create mode 100644 tools/tracing/rtla/src/rtla.c create mode 100644 tools/tracing/rtla/src/timerlat.c create mode 100644 tools/tracing/rtla/src/timerlat.h create mode 100644 tools/tracing/rtla/src/timerlat_hist.c create mode 100644 tools/tracing/rtla/src/timerlat_top.c create mode 100644 tools/tracing/rtla/src/trace.c create mode 100644 tools/tracing/rtla/src/trace.h create mode 100644 tools/tracing/rtla/src/utils.c create mode 100644 tools/tracing/rtla/src/utils.h
Signature · 1 line
2.31.1
PATCH v7 1/14 rtla: Real-Time Linux Analysis tool
The rtla is a meta-tool that includes a set of commands that aims to analyze the real-time properties of Linux. But instead of testing Linux as a black box, rtla leverages kernel tracing capabilities to provide precise information about the properties and root causes of unexpected results.
rtla --help works and provide information about the available options.
This is just the "main" and the Makefile, no function yet.
--- tools/tracing/rtla/Makefile | 76 +++++++++++++++++++++++++++++++++++ tools/tracing/rtla/src/rtla.c | 72 +++++++++++++++++++++++++++++++++ 2 files changed, 148 insertions(+) create mode 100644 tools/tracing/rtla/Makefile create mode 100644 tools/tracing/rtla/src/rtla.c
tools/tracing/rtla/Makefile new file +76 -0
@@ -0,0 +1,76 @@ +NAME := rtla +VERSION := 0.3 + +# From libtracefs: +# Makefiles suck: This macro sets a default value of $(2) for the +# variable named by $(1), unless the variable has been set by +# environment or command line. This is necessary for CC and AR +# because make sets default values, so the simpler ?= approach +# won't work as expected. +define allow-override + $(if $(or $(findstring environment,$(origin $(1))),\ + $(findstring command line,$(origin $(1)))),,\ + $(eval $(1) = $(2))) +endef + +# Allow setting CC and AR, or setting CROSS_COMPILE as a prefix. +$(call allow-override,CC,$(CROSS_COMPILE)gcc) +$(call allow-override,AR,$(CROSS_COMPILE)ar) +$(call allow-override,STRIP,$(CROSS_COMPILE)strip) +$(call allow-override,PKG_CONFIG,pkg-config) +$(call allow-override,LD_SO_CONF_PATH,/etc/ld.so.conf.d/) +$(call allow-override,LDCONFIG,ldconfig) + +INSTALL = install +FOPTS := -flto=auto -ffat-lto-objects -fexceptions -fstack-protector-strong \ + -fasynchronous-unwind-tables -fstack-clash-protection +WOPTS := -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -Wno-maybe-uninitialized + +TRACEFS_HEADERS := $$($(PKG_CONFIG) --cflags libtracefs) + +CFLAGS := -O -g -DVERSION=\"$(VERSION)\" $(FOPTS) $(MOPTS) $(WOPTS) $(TRACEFS_HEADERS) +LDFLAGS := -ggdb +LIBS := -ltracefs -ltraceevent -lprocps + +SRC := $(wildcard src/*.c) +HDR := $(wildcard src/*.h) +OBJ := $(SRC:.c=.o) +DIRS := src +FILES := Makefile +CEXT := bz2 +TARBALL := $(NAME)-$(VERSION).tar.$(CEXT) +TAROPTS := -cvjf $(TARBALL) +BINDIR := /usr/bin +DATADIR := /usr/share +DOCDIR := $(DATADIR)/doc +MANDIR := $(DATADIR)/man +LICDIR := $(DATADIR)/licenses + +.PHONY: all +all: rtla + +rtla: $(OBJ) + $(CC) -o rtla $(LDFLAGS) $(OBJ) $(LIBS) + +static: $(OBJ) + $(CC) -o rtla-static $(LDFLAGS) --static $(OBJ) $(LIBS) -lpthread -ldl + +.PHONY: install +install: + $(INSTALL) -d -m 755 $(DESTDIR)$(BINDIR) + $(INSTALL) rtla -m 755 $(DESTDIR)$(BINDIR) + $(STRIP) $(DESTDIR)$(BINDIR)/rtla + +.PHONY: clean tarball +clean: + @test ! -f rtla || rm rtla + @test ! -f rtla-static || rm rtla-static + @test ! -f src/rtla.o || rm src/rtla.o + @test ! -f $(TARBALL) || rm -f $(TARBALL) + @rm -rf *~ $(OBJ) *.tar.$(CEXT) + +tarball: clean + rm -rf $(NAME)-$(VERSION) && mkdir $(NAME)-$(VERSION) + cp -r $(DIRS) $(FILES) $(NAME)-$(VERSION) + tar $(TAROPTS) --exclude='*~' $(NAME)-$(VERSION) + rm -rf $(NAME)-$(VERSION)
tools/tracing/rtla/src/rtla.c new file +72 -0
72 lines not on this page · raw message
Signature · 1 line
2.31.1
Steven Rostedt
maintainerRe: [PATCH V7 01/14] rtla: Real-Time Linux Analysis tool
Quoted 61 lines · On Fri, 29 Oct 2021 21:26:04 +0200 Daniel Bristot de Oliveira…
The rtla is a meta-tool that includes a set of commands that aims to analyze the real-time properties of Linux. But instead of testing Linux as a black box, rtla leverages kernel tracing capabilities to provide precise information about the properties and root causes of unexpected results.
rtla --help works and provide information about the available options.
This is just the "main" and the
Makefile, no function yet.Cc Steven Rostedt <[email protected]>Cc Ingo Molnar <[email protected]>Cc Tom Zanussi <[email protected]>Cc Masami Hiramatsu <[email protected]>Cc Juri Lelli <[email protected]>Cc Clark Williams <[email protected]>Cc John Kacur <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Sebastian Andrzej Siewior <[email protected]>Cc Daniel Bristot de Oliveira <[email protected]>Signed-off-by Daniel Bristot de Oliveira <[email protected]>--- tools/tracing/rtla/Makefile | 76 +++++++++++++++++++++++++++++++++++ tools/tracing/rtla/src/rtla.c | 72 +++++++++++++++++++++++++++++++++ 2 files changed, 148 insertions(+) create mode 100644 tools/tracing/rtla/Makefile create mode 100644 tools/tracing/rtla/src/rtla.ctools/tracing/rtla/Makefile new file +23
-0@@ -0,0 +1,76 @@ +NAME := rtla +VERSION := 0.3 + +# From libtracefs: +# Makefiles suck: This macro sets a default value of $(2) for the +# variable named by $(1), unless the variable has been set by +# environment or command line. This is necessary for CC and AR +# because make sets default values, so the simpler ?= approach +# won't work as expected. +define allow-override + $(if $(or $(findstring environment,$(origin $(1))),\ + $(findstring command line,$(origin $(1)))),,\ + $(eval $(1) = $(2))) +endef + +# Allow setting CC and AR, or setting CROSS_COMPILE as a prefix. +$(call allow-override,CC,$(CROSS_COMPILE)gcc) +$(call allow-override,AR,$(CROSS_COMPILE)ar) +$(call allow-override,STRIP,$(CROSS_COMPILE)strip) +$(call allow-override,PKG_CONFIG,pkg-config) +$(call allow-override,LD_SO_CONF_PATH,/etc/ld.so.conf.d/) +$(call allow-override,LDCONFIG,ldconfig) +
tools/tracing/rtla/Makefile line 33
Quoted 10 lines
tools/tracing/rtla/Makefile new file +10
-0+INSTALL = install +FOPTS := -flto=auto -ffat-lto-objects -fexceptions -fstack-protector-strong \ + -fasynchronous-unwind-tables -fstack-clash-protection +WOPTS := -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -Wno-maybe-uninitialized + +TRACEFS_HEADERS := $$($(PKG_CONFIG) --cflags libtracefs) + +CFLAGS := -O -g -DVERSION=\"$(VERSION)\" $(FOPTS) $(MOPTS) $(WOPTS) $(TRACEFS_HEADERS) +LDFLAGS := -ggdb +LIBS := -ltracefs -ltraceevent -lprocps
For the -ltracefs and -ltracevent, you could use:
$$($PKG_CONFIG) --libs libtracefs)
which would be more robust.
-- Steve
Quoted 15 lines
tools/tracing/rtla/Makefile new file +15
-0+ +SRC := $(wildcard src/*.c) +HDR := $(wildcard src/*.h) +OBJ := $(SRC:.c=.o) +DIRS := src +FILES := Makefile +CEXT := bz2 +TARBALL := $(NAME)-$(VERSION).tar.$(CEXT) +TAROPTS := -cvjf $(TARBALL) +BINDIR := /usr/bin +DATADIR := /usr/share +DOCDIR := $(DATADIR)/doc +MANDIR := $(DATADIR)/man +LICDIR := $(DATADIR)/licenses +
Re: [PATCH V7 01/14] rtla: Real-Time Linux Analysis tool
Quoted 77 lines · Steven Rostedt
Quoted 61 lines · On Fri, 29 Oct 2021 21:26:04 +0200 Daniel Bristot de Oliveira…
The rtla is a meta-tool that includes a set of commands that aims to analyze the real-time properties of Linux. But instead of testing Linux as a black box, rtla leverages kernel tracing capabilities to provide precise information about the properties and root causes of unexpected results.
rtla --help works and provide information about the available options.
This is just the "main" and the
Makefile, no function yet.Cc Steven Rostedt <[email protected]>Cc Ingo Molnar <[email protected]>Cc Tom Zanussi <[email protected]>Cc Masami Hiramatsu <[email protected]>Cc Juri Lelli <[email protected]>Cc Clark Williams <[email protected]>Cc John Kacur <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Sebastian Andrzej Siewior <[email protected]>Cc Daniel Bristot de Oliveira <[email protected]>Signed-off-by Daniel Bristot de Oliveira <[email protected]>--- tools/tracing/rtla/Makefile | 76 +++++++++++++++++++++++++++++++++++ tools/tracing/rtla/src/rtla.c | 72 +++++++++++++++++++++++++++++++++ 2 files changed, 148 insertions(+) create mode 100644 tools/tracing/rtla/Makefile create mode 100644 tools/tracing/rtla/src/rtla.ctools/tracing/rtla/Makefile new file +23
-0@@ -0,0 +1,76 @@ +NAME := rtla +VERSION := 0.3 + +# From libtracefs: +# Makefiles suck: This macro sets a default value of $(2) for the +# variable named by $(1), unless the variable has been set by +# environment or command line. This is necessary for CC and AR +# because make sets default values, so the simpler ?= approach +# won't work as expected. +define allow-override + $(if $(or $(findstring environment,$(origin $(1))),\ + $(findstring command line,$(origin $(1)))),,\ + $(eval $(1) = $(2))) +endef + +# Allow setting CC and AR, or setting CROSS_COMPILE as a prefix. +$(call allow-override,CC,$(CROSS_COMPILE)gcc) +$(call allow-override,AR,$(CROSS_COMPILE)ar) +$(call allow-override,STRIP,$(CROSS_COMPILE)strip) +$(call allow-override,PKG_CONFIG,pkg-config) +$(call allow-override,LD_SO_CONF_PATH,/etc/ld.so.conf.d/) +$(call allow-override,LDCONFIG,ldconfig) +tools/tracing/rtla/Makefile line 33
Quoted 10 lines
tools/tracing/rtla/Makefile new file +10
-0+INSTALL = install +FOPTS := -flto=auto -ffat-lto-objects -fexceptions -fstack-protector-strong \ + -fasynchronous-unwind-tables -fstack-clash-protection +WOPTS := -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -Wno-maybe-uninitialized + +TRACEFS_HEADERS := $$($(PKG_CONFIG) --cflags libtracefs) + +CFLAGS := -O -g -DVERSION=\"$(VERSION)\" $(FOPTS) $(MOPTS) $(WOPTS) $(TRACEFS_HEADERS) +LDFLAGS := -ggdb +LIBS := -ltracefs -ltraceevent -lprocpsFor the -ltracefs and -ltracevent, you could use:
Quoted 3 lines
$$($PKG_CONFIG) --libs libtracefs)which would be more robust.
Will use!
-- Daniel
Quoted 2 lines
-- Steve
PATCH v7 2/14 rtla: Helper functions for rtla
In reply to Daniel Bristot de OliveiraThis is a set of utils and tracer helper functions. They are used by rtla mostly to parse config, display data and some trace operations that are not part of libtracefs (because they are only useful it for this case).
--- tools/tracing/rtla/src/trace.c | 192 +++++++++++++++ tools/tracing/rtla/src/trace.h | 27 ++ tools/tracing/rtla/src/utils.c | 433 +++++++++++++++++++++++++++++++++ tools/tracing/rtla/src/utils.h | 56 +++++ 4 files changed, 708 insertions(+) create mode 100644 tools/tracing/rtla/src/trace.c create mode 100644 tools/tracing/rtla/src/trace.h create mode 100644 tools/tracing/rtla/src/utils.c create mode 100644 tools/tracing/rtla/src/utils.h
tools/tracing/rtla/src/trace.c new file +192 -0
@@ -0,0 +1,192 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE +#include <sys/sendfile.h> +#include <tracefs.h> +#include <signal.h> +#include <stdlib.h> +#include <unistd.h> +#include <errno.h> + +#include "trace.h" +#include "utils.h" + +/* +* enable_tracer_by_name - enable a tracer on the given instance +*/ +int enable_tracer_by_name (struct tracefs_instance *inst,const char *tracer) +{ +enum tracefs_tracers t; +int retval; + + t =TRACEFS_TRACER_CUSTOM ; + +debug_msg ("enabling %s tracer\n" , tracer); + + retval =tracefs_tracer_set (inst, t, tracer); +if (retval <0 ) { +if (errno ==ENODEV ) +err_msg ("tracer %s not found!\n" , tracer); + +err_msg ("failed to enable the tracer %s\n" , tracer); +return -1 ; + } + +return 0 ; +} + +/* +* disable_tracer - set nop tracer to the insta +*/ +void disable_tracer (struct tracefs_instance *inst) +{ +enum tracefs_tracers t =TRACEFS_TRACER_NOP ; +int retval; + + retval =tracefs_tracer_set (inst, t); +if (retval <0 ) +err_msg ("oops, error disabling tracer\n" ); +} + +/* +* create_instance - create a trace instance with *instance_name +*/ +struct tracefs_instance *create_instance (char *instance_name) +{ +return tracefs_instance_create (instance_name); +} + +/* +* destroy_instance - remove a trace instance and free the data +*/ +void destroy_instance (struct tracefs_instance *inst) +{ +tracefs_instance_destroy (inst); +tracefs_instance_free (inst); +} + +/* +* save_trace_to_file - save the trace output of the instance to the file +*/ +int save_trace_to_file (struct tracefs_instance *inst,const char *filename) +{ +const char *file ="trace" ; +mode_t mode =0644 ; +char *buffer[4096 ]; +int out_fd, in_fd; +int retval =-1 ; + + in_fd =tracefs_instance_file_open (inst, file,O_RDONLY ); +if (in_fd <0 ) { +err_msg ("Failed to open trace file\n" ); +return -1 ; + } + + out_fd =creat (filename, mode); +if (out_fd <0 ) { +err_msg ("Failed to create output file %s\n" , filename); + goto out_close_in; + } + +do { + retval =read (in_fd, buffer,sizeof (buffer)); +if (retval <=0 ) + goto out_close; + + retval =write (out_fd, buffer, retval); +if (retval <0 ) + goto out_close; + }while (retval >0 ); + + retval =0 ; +out_close: +close (out_fd); +out_close_in: +close (in_fd); +return retval; +} + +/* +* collect_registered_events - call the existing callback function for the event +* +* If an event has a registered callback function, call it. +* Otherwise, ignore the event. +*/ +int +collect_registered_events (struct tep_event *event,struct tep_record *record, +int cpu,void *context) +{ +struct trace_instance *trace = context; +struct trace_seq *s = trace->seq; + +if (!event->handler) +return 0 ; + + event->handler (s, record, event, context); + +return 0 ; +} + +/* +* trace_instance_destroy - destroy and free a rtla trace instance +*/ +void trace_instance_destroy (struct trace_instance *trace) +{ +if (trace->inst) { +disable_tracer (trace->inst); +destroy_instance (trace->inst); + } + +if (trace->seq) +free (trace->seq); + +if (trace->tep) +tep_free (trace->tep); +} + +/* +* trace_instance_init - create an rtla trace instance +* +* It is more than the tracefs instance, as it contains other +* things required for the tracing, such as the local events and +* a seq file. +* +* Note that the trace instance is returned disabled. This allows +* the tool to apply some other configs, like setting priority +* to the kernel threads, before starting generating trace entries. +*/ +int trace_instance_init (struct trace_instance *trace,char *tool_name) +{ + trace->seq =calloc (1 ,sizeof (*trace->seq)); +if (!trace->seq) + goto out_err; + +trace_seq_init (trace->seq); + + trace->inst =create_instance (tool_name); +if (!trace->inst) + goto out_err; + + trace->tep =tracefs_local_events (NULL ); +if (!trace->tep) + goto out_err; + +/* +* Let the main enable the record after setting some other +* things such as the priority of the tracer's threads. +*/ +tracefs_trace_off (trace->inst); + +return 0 ; + +out_err: +trace_instance_destroy (trace); +return 1 ; +} + +/* +* trace_instance_start - start tracing a given rtla instance +*/ +int trace_instance_start (struct trace_instance *trace) +{ +return tracefs_trace_on (trace->inst); +}
tools/tracing/rtla/src/trace.h new file +27 -0
@@ -0,0 +1,27 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <tracefs.h> +#include <stddef.h> + +struct trace_instance { +struct tracefs_instance *inst; +struct tep_handle *tep; +struct trace_seq *seq; +}; + +int trace_instance_init (struct trace_instance *trace,char *tool_name); +int trace_instance_start (struct trace_instance *trace); +void trace_instance_destroy (struct trace_instance *trace); + +struct trace_seq *get_trace_seq (void ); +int enable_tracer_by_name (struct tracefs_instance *inst,const char *tracer); +void disable_tracer (struct tracefs_instance *inst); + +int enable_osnoise (struct trace_instance *trace); +int enable_timerlat (struct trace_instance *trace); + +struct tracefs_instance *create_instance (char *instance_name); +void destroy_instance (struct tracefs_instance *inst); + +int save_trace_to_file (struct tracefs_instance *inst,const char *filename); +int collect_registered_events (struct tep_event *tep,struct tep_record *record, +int cpu,void *context);
tools/tracing/rtla/src/utils.c new file +433 -0
433 lines not on this page · raw message
tools/tracing/rtla/src/utils.h new file +56 -0
56 lines not on this page · raw message
Signature · 1 line
2.31.1
Steven Rostedt
maintainerRe: [PATCH V7 02/14] rtla: Helper functions for rtla
tools/tracing/rtla/src/trace.c line 25
Quoted 13 lines · On Fri, 29 Oct 2021 21:26:05 +0200 Daniel Bristot de Oliveira…
tools/tracing/rtla/src/trace.c new file +13
-0+/* +* enable_tracer_by_name - enable a tracer on the given instance +*/ +int enable_tracer_by_name (struct tracefs_instance *inst,const char *tracer) +{ +enum tracefs_tracers t; +int retval; + + t =TRACEFS_TRACER_CUSTOM ; + +debug_msg ("enabling %s tracer\n" , tracer); + + retval =tracefs_tracer_set (inst, t, tracer);
Interesting. We had discussions about having the custom option (which I fought for, for this very reason).
Quoted 39 lines
tools/tracing/rtla/src/trace.c new file +39
-0+if (retval <0 ) { +if (errno ==ENODEV ) +err_msg ("tracer %s not found!\n" , tracer); + +err_msg ("failed to enable the tracer %s\n" , tracer); +return -1 ; + } + +return 0 ; +} + +/* +* disable_tracer - set nop tracer to the insta +*/ +void disable_tracer (struct tracefs_instance *inst) +{ +enum tracefs_tracers t =TRACEFS_TRACER_NOP ; +int retval; + + retval =tracefs_tracer_set (inst, t); +if (retval <0 ) +err_msg ("oops, error disabling tracer\n" ); +} + +/* +* create_instance - create a trace instance with *instance_name +*/ +struct tracefs_instance *create_instance (char *instance_name) +{ +return tracefs_instance_create (instance_name); +} + +/* +* destroy_instance - remove a trace instance and free the data +*/ +void destroy_instance (struct tracefs_instance *inst) +{ +tracefs_instance_destroy (inst); +tracefs_instance_free (inst);
tools/tracing/rtla/src/trace.c line 74
Quoted 10 lines
tools/tracing/rtla/src/trace.c new file +10
-0+} + +/* +* save_trace_to_file - save the trace output of the instance to the file +*/ +int save_trace_to_file (struct tracefs_instance *inst,const char *filename) +{ +const char *file ="trace" ; +mode_t mode =0644 ; +char *buffer[4096 ];
Did you really mean to have buffer be 4096 strings?
Or did you mean:
char buffer[4096];
(i.e. a single string of 4096 size)?
-- Steve
Quoted 34 lines
tools/tracing/rtla/src/trace.c new file +34
-0+int out_fd, in_fd; +int retval =-1 ; + + in_fd =tracefs_instance_file_open (inst, file,O_RDONLY ); +if (in_fd <0 ) { +err_msg ("Failed to open trace file\n" ); +return -1 ; + } + + out_fd =creat (filename, mode); +if (out_fd <0 ) { +err_msg ("Failed to create output file %s\n" , filename); + goto out_close_in; + } + +do { + retval =read (in_fd, buffer,sizeof (buffer)); +if (retval <=0 ) + goto out_close; + + retval =write (out_fd, buffer, retval); +if (retval <0 ) + goto out_close; + }while (retval >0 ); + + retval =0 ; +out_close: +close (out_fd); +out_close_in: +close (in_fd); +return retval; +} + +/*
Re: [PATCH V7 02/14] rtla: Helper functions for rtla
Quoted 72 lines · Steven Rostedt
tools/tracing/rtla/src/trace.c line 25
Quoted 13 lines · On Fri, 29 Oct 2021 21:26:05 +0200 Daniel Bristot de Oliveira…
tools/tracing/rtla/src/trace.c new file +13
-0+/* +* enable_tracer_by_name - enable a tracer on the given instance +*/ +int enable_tracer_by_name (struct tracefs_instance *inst,const char *tracer) +{ +enum tracefs_tracers t; +int retval; + + t =TRACEFS_TRACER_CUSTOM ; + +debug_msg ("enabling %s tracer\n" , tracer); + + retval =tracefs_tracer_set (inst, t, tracer);Interesting. We had discussions about having the custom option (which I fought for, for this very reason).
Quoted 39 lines
tools/tracing/rtla/src/trace.c new file +39
-0+if (retval <0 ) { +if (errno ==ENODEV ) +err_msg ("tracer %s not found!\n" , tracer); + +err_msg ("failed to enable the tracer %s\n" , tracer); +return -1 ; + } + +return 0 ; +} + +/* +* disable_tracer - set nop tracer to the insta +*/ +void disable_tracer (struct tracefs_instance *inst) +{ +enum tracefs_tracers t =TRACEFS_TRACER_NOP ; +int retval; + + retval =tracefs_tracer_set (inst, t); +if (retval <0 ) +err_msg ("oops, error disabling tracer\n" ); +} + +/* +* create_instance - create a trace instance with *instance_name +*/ +struct tracefs_instance *create_instance (char *instance_name) +{ +return tracefs_instance_create (instance_name); +} + +/* +* destroy_instance - remove a trace instance and free the data +*/ +void destroy_instance (struct tracefs_instance *inst) +{ +tracefs_instance_destroy (inst); +tracefs_instance_free (inst);tools/tracing/rtla/src/trace.c line 74
Quoted 10 lines
tools/tracing/rtla/src/trace.c new file +10
-0+} + +/* +* save_trace_to_file - save the trace output of the instance to the file +*/ +int save_trace_to_file (struct tracefs_instance *inst,const char *filename) +{ +const char *file ="trace" ; +mode_t mode =0644 ; +char *buffer[4096 ];Did you really mean to have buffer be 4096 strings?
Quoted 3 lines
Or did you mean:
char buffer[4096];
I should be "char buffer[4096];" I will fix that.
(The * is probably a left over from the idea of using a malloc(4096)...)
-- Daniel
Quoted 4 lines
(i.e. a single string of 4096 size)?
-- Steve
PATCH v7 3/14 rtla: Add osnoise tool
In reply to Daniel Bristot de OliveiraThe osnoise tool is the interface for the osnoise tracer. The osnoise tool will have multiple "modes" with different outputs. At this point, no mode is included.
The osnoise.c includes the osnoise_context abstraction. It serves to read-save-change-restore the default values from tracing/osnoise/ directory. When the context is deleted, the default values are restored.
It also includes some other helper functions for managing osnoise tracer sessions.
With these bits and pieces in place, we can start adding some functionality to rtla.
--- tools/tracing/rtla/Makefile | 2 + tools/tracing/rtla/src/osnoise.c | 1013 ++++++++++++++++++++++++++++++ tools/tracing/rtla/src/osnoise.h | 95 +++ tools/tracing/rtla/src/rtla.c | 10 + 4 files changed, 1120 insertions(+) create mode 100644 tools/tracing/rtla/src/osnoise.c create mode 100644 tools/tracing/rtla/src/osnoise.h
tools/tracing/rtla/Makefile +2 -0
@@ -60,6 +60,8 @@ install: $(INSTALL) -d -m 755 $(DESTDIR)$(BINDIR) $(INSTALL) rtla -m 755 $(DESTDIR)$(BINDIR) $(STRIP) $(DESTDIR)$(BINDIR)/rtla + @test ! -f $(DESTDIR)$(BINDIR)/osnoise || rm $(DESTDIR)$(BINDIR)/osnoise + ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/osnoise .PHONY: clean tarball clean:
tools/tracing/rtla/src/osnoise.c new file +1013 -0
1,013 lines not on this page · raw message
tools/tracing/rtla/src/osnoise.h new file +95 -0
95 lines not on this page · raw message
tools/tracing/rtla/src/rtla.c +10 -0
@@ -8,6 +8,8 @@#include <string.h> #include <stdio.h> +#include "osnoise.h" +/* * rtla_usage - print rtla usage */
@@ -22,6 +24,7 @@ static void rtla_usage(void)" usage: rtla COMMAND ..." ,"" ," commands:" , +" osnoise - gives information about the operating system noise (osnoise)" ,"" , NULL, };
@@ -39,7 +42,14 @@ static void rtla_usage(void)*/ int run_command (int argc,char **argv,int start_position) { +if (strcmp (argv[start_position],"osnoise" ) ==0 ) { +osnoise_main (argc-start_position, &argv[start_position]); + goto ran; + } +return 0 ; +ran: +return 1 ; }int main (int argc,char *argv[])
Signature · 1 line
2.31.1
PATCH v7 4/14 rtla/osnoise: Add osnoise top mode
In reply to Daniel Bristot de OliveiraThe rtla osnoise tool is an interface for the osnoise tracer. The osnoise tracer dispatches a kernel thread per-cpu. These threads read the time in a loop while with preemption, softirqs and IRQs enabled, thus allowing all the sources of osnoise during its execution. The osnoise threads take note of the entry and exit point of any source of interferences, increasing a per-cpu interference counter. The osnoise tracer also saves an interference counter for each source of interference.
The rtla osnoise top mode displays information about the periodic summary from the osnoise tracer.
One example of rtla osnoise top output is:
[root@alien ~]# rtla osnoise top -c 0-3 -d 1m -q -r 900000 -P F:1
Operating System Noise
duration: 0 00:01:00 | time is in us
CPU Period Runtime Noise % CPU Aval Max Noise Max Single HW NMI IRQ Softirq Thread
0 #58 52200000 1031 99.99802 91 60 0 0 52285 0 101 1 #59 53100000 5 99.99999 5 5 0 9 53122 0 18 2 #59 53100000 7 99.99998 7 7 0 8 53115 0 18 3 #59 53100000 8274 99.98441 277 23 0 9 53778 0 660
"rtla osnoise top --help" works and provide information about the available options.
--- tools/tracing/rtla/src/osnoise_top.c | 577 +++++++++++++++++++++++++++ 1 file changed, 577 insertions(+) create mode 100644 tools/tracing/rtla/src/osnoise_top.c
tools/tracing/rtla/src/osnoise_top.c new file +577 -0
@@ -0,0 +1,577 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]> +*/ + +#include <getopt.h> +#include <stdlib.h> +#include <string.h> +#include <signal.h> +#include <unistd.h> +#include <stdio.h> +#include <time.h> + +#include "osnoise.h" +#include "utils.h" + +/* +* osnoise top parameters +*/ +struct osnoise_top_params { +char *cpus; +char *monitored_cpus; +char *trace_output; +unsigned long long runtime; +unsigned long long period; +long long stop_us; +long long stop_total_us; +int sleep_time; +int duration; +int quiet; +int set_sched; +struct sched_attr sched_param; +}; + +struct osnoise_top_cpu { +unsigned long long sum_runtime; +unsigned long long sum_noise; +unsigned long long max_noise; +unsigned long long max_sample; + +unsigned long long hw_count; +unsigned long long nmi_count; +unsigned long long irq_count; +unsigned long long softirq_count; +unsigned long long thread_count; + +int sum_cycles; +}; + +struct osnoise_top_data { +struct osnoise_top_cpu *cpu_data; +int nr_cpus; +}; + +/* +* osnoise_free_top - free runtime data +*/ +static void +osnoise_free_top (struct osnoise_top_data *data) +{ +free (data->cpu_data); +free (data); +} + +/* +* osnoise_alloc_histogram - alloc runtime data +*/ +static struct osnoise_top_data *osnoise_alloc_top (int nr_cpus) +{ +struct osnoise_top_data *data; + + data =calloc (1 ,sizeof (*data)); +if (!data) +return NULL ; + + data->nr_cpus = nr_cpus; + +/* one set of histograms per CPU */ + data->cpu_data =calloc (1 ,sizeof (*data->cpu_data) * nr_cpus); +if (!data->cpu_data) + goto cleanup; + +return data; + +cleanup: +osnoise_free_top (data); +return NULL ; +} + +/* +* osnoise_top_handler - this is the handler for osnoise tracer events +*/ +static int +osnoise_top_handler (struct trace_seq *s,struct tep_record *record, +struct tep_event *event,void *context) +{ +struct trace_instance *trace = context; +struct osnoise_tool *tool; +unsigned long long val; +struct osnoise_top_cpu *cpu_data; +struct osnoise_top_data *data; +int cpu = record->cpu; + + tool = container_of(trace, struct osnoise_tool, trace); + + data = tool->data; + cpu_data = &data->cpu_data[cpu]; + + cpu_data->sum_cycles++; + +tep_get_field_val (s, event,"runtime" , record, &val,1 ); +update_sum (&cpu_data->sum_runtime, &val); + +tep_get_field_val (s, event,"noise" , record, &val,1 ); +update_max (&cpu_data->max_noise, &val); +update_sum (&cpu_data->sum_noise, &val); + +tep_get_field_val (s, event,"max_sample" , record, &val,1 ); +update_max (&cpu_data->max_sample, &val); + +tep_get_field_val (s, event,"hw_count" , record, &val,1 ); +update_sum (&cpu_data->hw_count, &val); + +tep_get_field_val (s, event,"nmi_count" , record, &val,1 ); +update_sum (&cpu_data->nmi_count, &val); + +tep_get_field_val (s, event,"irq_count" , record, &val,1 ); +update_sum (&cpu_data->irq_count, &val); + +tep_get_field_val (s, event,"softirq_count" , record, &val,1 ); +update_sum (&cpu_data->softirq_count, &val); + +tep_get_field_val (s, event,"thread_count" , record, &val,1 ); +update_sum (&cpu_data->thread_count, &val); + +return 0 ; +} + +/* +* osnoise_top_header - print the header of the tool output +*/ +static void osnoise_top_header (struct osnoise_tool *top) +{ +struct trace_seq *s = top->trace.seq; +char duration[26 ]; + +get_duration (top->start_time, duration,sizeof (duration)); + +trace_seq_printf (s,"\033[2;37;40m" ); +trace_seq_printf (s," Operating System Noise" ); +trace_seq_printf (s," " ); +trace_seq_printf (s," " ); +trace_seq_printf (s,"\033[0;0;0m" ); +trace_seq_printf (s,"\n" ); + +trace_seq_printf (s,"duration: %9s | time is in us\n" , duration); + +trace_seq_printf (s,"\033[2;30;47m" ); +trace_seq_printf (s,"CPU Period Runtime " ); +trace_seq_printf (s," Noise " ); +trace_seq_printf (s," %% CPU Aval " ); +trace_seq_printf (s," Max Noise Max Single " ); +trace_seq_printf (s," HW NMI IRQ Softirq Thread" ); +trace_seq_printf (s,"\033[0;0;0m" ); +trace_seq_printf (s,"\n" ); +} + +/* +* clear_terminal - clears the output terminal +*/ +static void clear_terminal (struct trace_seq *seq) +{ +if (!config_debug) +trace_seq_printf (seq,"\033c" ); +} + +/* +* osnoise_top_print - prints the output of a given CPU +*/ +static void osnoise_top_print (struct osnoise_tool *tool,int cpu) +{ +struct trace_seq *s = tool->trace.seq; +struct osnoise_top_cpu *cpu_data; +struct osnoise_top_data *data; +int percentage; +int decimal; + + data = tool->data; + cpu_data = &data->cpu_data[cpu]; + +if (!cpu_data->sum_runtime) +return ; + + percentage = ((cpu_data->sum_runtime - cpu_data->sum_noise) *10000000 ) + / cpu_data->sum_runtime; + decimal = percentage %100000 ; + percentage = percentage /100000 ; + +trace_seq_printf (s,"%3d #%-6d %12llu " , cpu, cpu_data->sum_cycles, cpu_data->sum_runtime); +trace_seq_printf (s,"%12llu " , cpu_data->sum_noise); +trace_seq_printf (s," %3d.%05d" , percentage, decimal); +trace_seq_printf (s,"%12llu %12llu" , cpu_data->max_noise, cpu_data->max_sample); + +trace_seq_printf (s,"%12llu " , cpu_data->hw_count); +trace_seq_printf (s,"%12llu " , cpu_data->nmi_count); +trace_seq_printf (s,"%12llu " , cpu_data->irq_count); +trace_seq_printf (s,"%12llu " , cpu_data->softirq_count); +trace_seq_printf (s,"%12llu\n" , cpu_data->thread_count); +} + +/* +* osnoise_print_stats - print data for all cpus +*/ +static void +osnoise_print_stats (struct osnoise_top_params *params,struct osnoise_tool *top) +{ +struct trace_instance *trace = &top->trace; +static int nr_cpus =-1 ; +int i; + +if (nr_cpus ==-1 ) + nr_cpus =sysconf (_SC_NPROCESSORS_CONF); + +if (!params->quiet) +clear_terminal (trace->seq); + +osnoise_top_header (top); + +for (i =0 ; i < nr_cpus; i++) { +if (params->cpus && !params->monitored_cpus[i]) +continue ; +osnoise_top_print (top, i); + } + +trace_seq_do_printf (trace->seq); +trace_seq_reset (trace->seq); +} + +/* +* osnoise_top_usage - prints osnoise top usage message +*/ +void osnoise_top_usage (char *usage) +{ +int i; + +static const char *const msg[] = { +" usage: rtla osnoise [top] [-h] [-q] [-p us] [-r us] [-s us] [-S us] [-T[=file]] \\" , +" [-c cpu-list] -P priority" , +"" , +" -h/--help: print this menu" , +" -p/--period us: osnoise period in us" , +" -r/--runtime us: osnoise runtime in us" , +" -s/--stop us: stop trace if a single sample is higher than the argument in us" , +" -S/--stop-total us: stop trace if the total sample is higher than the argument in us" , +" -c/--cpus cpu-list: list of cpus to run osnoise threads" , +" -d/--duration time[s|m|h|d]: duration of the session" , +" -t/--trace[=file]: save the stopped trace to [file|osnoise_trace.txt]" , +" -q/--quiet print only a summary at the end" , +" -P/--priority o:prio|r:prio|f:prio|d:runtime:period : set scheduling parameters" , +" o:prio - use SCHED_OTHER with prio" , +" r:prio - use SCHED_RR with prio" , +" f:prio - use SCHED_FIFO with prio" , +" d:runtime[us|ms|s]:period[us|ms|s] - use SCHED_DEADLINE with runtime and period" , +" in nanoseconds" , +NULL , + }; + +if (usage) +fprintf (stderr,"%s\n" , usage); + +fprintf (stderr,"rtla osnoise top: a per-cpu summary of the OS noise (version %s)\n" , +VERSION ); + +for (i =0 ; msg[i]; i++) +fprintf (stderr,"%s\n" , msg[i]); +exit (1 ); +} + +/* +* osnoise_top_parse_args - allocs, parse and fill the cmd line parameters +*/ +struct osnoise_top_params *osnoise_top_parse_args (int argc,char **argv) +{ +struct osnoise_top_params *params; +int retval; +int c; + + params =calloc (1 ,sizeof (*params)); +if (!params) +exit (1 ); + +while (1 ) { +static struct option long_options[] = { + {"cpus" , required_argument,0 ,'c' }, + {"debug" , no_argument,0 ,'D' }, + {"duration" , required_argument,0 ,'d' }, + {"help" , no_argument,0 ,'h' }, + {"period" , required_argument,0 ,'p' }, + {"priority" , required_argument,0 ,'P' }, + {"quiet" , no_argument,0 ,'q' }, + {"runtime" , required_argument,0 ,'r' }, + {"stop" , required_argument,0 ,'s' }, + {"stop-total" , required_argument,0 ,'S' }, + {"trace" , optional_argument,0 ,'t' }, + {0 ,0 ,0 ,0 } + }; + +/* getopt_long stores the option index here. */ +int option_index =0 ; + + c =getopt_long (argc, argv,"c:d:Dhp:P:qr:s:S:t::" , + long_options, &option_index); + +/* Detect the end of the options. */ +if (c ==-1 ) +break ; + +switch (c) { +case 'c' : + retval =parse_cpu_list (optarg, ¶ms->monitored_cpus); +if (retval) +osnoise_top_usage ("\nInvalid -c cpu list\n" ); + params->cpus = optarg; +debug_msg ("-c for %s\n" , params->cpus); +break ; +case 'D' : + config_debug =1 ; +break ; +case 'd' : + params->duration =parse_seconds_duration (optarg); +if (!params->duration) +osnoise_top_usage ("Invalid -D duration\n" ); +break ; +case 'h' : +case '?' : +osnoise_top_usage (NULL ); +break ; +case 'p' : + params->period =get_llong_from_str (optarg); +if (params->period >10000000 ) +osnoise_top_usage ("Period longer than 10 s\n" ); +break ; +case 'P' : + retval =parse_prio (optarg, ¶ms->sched_param); +if (retval ==-1 ) +osnoise_top_usage ("Invalid -P priority" ); + params->set_sched =1 ; +break ; +case 'q' : + params->quiet =1 ; +break ; +case 'r' : + params->runtime =get_llong_from_str (optarg); +if (params->runtime <100 ) +osnoise_top_usage ("Runtime shorter than 100 us\n" ); +break ; +case 's' : + params->stop_us =get_llong_from_str (optarg); +break ; +case 'S' : + params->stop_total_us =get_llong_from_str (optarg); +break ; +case 't' : +if (optarg) +/* skip = */ + params->trace_output = &optarg[1 ]; +else + params->trace_output ="osnoise_trace.txt" ; +break ; +default : +osnoise_top_usage ("Invalid option" ); + } + } + +if (geteuid ()) { +err_msg ("osnoise needs root permission\n" ); +exit (EXIT_FAILURE ); + } + +return params; +} + +/* +* osnoise_top_apply_config - apply the top configs to the initialized tool +*/ +static int +osnoise_top_apply_config (struct osnoise_tool *tool,struct osnoise_top_params *params) +{ +int retval; + +if (!params->sleep_time) + params->sleep_time =1 ; + +if (params->cpus) { + retval =osnoise_set_cpus (tool->context, params->cpus); +if (retval) { +err_msg ("Failed to apply CPUs config\n" ); + goto out_err; + } + } + +if (params->runtime || params->period) { + retval =osnoise_set_runtime_period (tool->context, + params->runtime, + params->period); +if (retval) { +err_msg ("Failed to set runtime and/or period\n" ); + goto out_err; + } + } + +if (params->stop_us) { + retval =osnoise_set_stop_us (tool->context, params->stop_us); +if (retval) { +err_msg ("Failed to set stop us\n" ); + goto out_err; + } + } + +if (params->stop_total_us) { + retval =osnoise_set_stop_total_us (tool->context, params->stop_total_us); +if (retval) { +err_msg ("Failed to set stop total us\n" ); + goto out_err; + } + } + +return 0 ; + +out_err: +return -1 ; +} + +/* +* osnoise_init_top - initialize a osnoise top tool with parameters +*/ +struct osnoise_tool *osnoise_init_top (struct osnoise_top_params *params) +{ +struct osnoise_tool *tool; +int nr_cpus; + + nr_cpus =sysconf (_SC_NPROCESSORS_CONF); + + tool =osnoise_init_tool ("osnoise_top" ); +if (!tool) +return NULL ; + + tool->data =osnoise_alloc_top (nr_cpus); +if (!tool->data) + goto out_err; + + tool->params = params; + +tep_register_event_handler (tool->trace.tep,-1 ,"ftrace" ,"osnoise" , + osnoise_top_handler,NULL ); + +return tool; + +out_err: +osnoise_free_top (tool->data); +osnoise_destroy_tool (tool); +return NULL ; +} + +static int stop_tracing; +static void stop_top (int sig) +{ + stop_tracing =1 ; +} + +/* +* osnoise_top_set_signals - handles the signal to stop the tool +*/ +static void osnoise_top_set_signals (struct osnoise_top_params *params) +{ +signal (SIGINT , stop_top); +if (params->duration) { +signal (SIGALRM , stop_top); +alarm (params->duration); + } +} + +int osnoise_top_main (int argc,char **argv) +{ +struct osnoise_top_params *params; +struct trace_instance *trace; +struct osnoise_tool *record; +struct osnoise_tool *tool; +int return_value =1 ; +int retval; + + params =osnoise_top_parse_args (argc, argv); +if (!params) +exit (1 ); + + tool =osnoise_init_top (params); +if (!tool) { +err_msg ("Could not init osnoise top\n" ); + goto out_exit; + } + + retval =osnoise_top_apply_config (tool, params); +if (retval) { +err_msg ("Could not apply config\n" ); + goto out_top; + } + + trace = &tool->trace; + + retval =enable_osnoise (trace); +if (retval) { +err_msg ("Failed to enable osnoise tracer\n" ); + goto out_top; + } + +if (params->set_sched) { + retval =set_comm_sched_attr ("osnoise/" , ¶ms->sched_param); +if (retval) +osnoise_top_usage ("Failed to set sched parameters\n" ); + } + +trace_instance_start (trace); + +if (params->trace_output) { + record =osnoise_init_trace_tool ("osnoise" ); +if (!record) { +err_msg ("Failed to enable the trace instance\n" ); + goto out_top; + } +trace_instance_start (&record->trace); + } + + tool->start_time =time (NULL ); +osnoise_top_set_signals (params); + +do { +sleep (params->sleep_time); + + retval =tracefs_iterate_raw_events (trace->tep, + trace->inst, +NULL , +0 , + collect_registered_events, + trace); +if (retval <0 ) { +err_msg ("Error iterating on events\n" ); + goto out_top; + } + +if (!params->quiet) +osnoise_print_stats (params, tool); + +if (!tracefs_trace_is_on (trace->inst)) +break ; + + }while (!stop_tracing); + +osnoise_print_stats (params, tool); + + return_value =0 ; + +if (!tracefs_trace_is_on (trace->inst)) { +printf ("osnoise hit stop tracing\n" ); +if (params->trace_output) { +printf (" Saving trace to %s\n" , params->trace_output); +save_trace_to_file (record->trace.inst, params->trace_output); + } + } + +out_top: +osnoise_free_top (tool->data); +osnoise_destroy_tool (tool); +if (params->trace_output) +osnoise_destroy_tool (record); +out_exit: +exit (return_value); +}
Signature · 1 line
2.31.1
PATCH v7 5/14 rtla/osnoise: Add the hist mode
In reply to Daniel Bristot de OliveiraThe rtla osnoise hist tool collects all osnoise:sample_threshold occurrence in a histogram, displaying the results in a user-friendly way. The tool also allows many configurations of the osnoise tracer and the collection of the tracer output.
Here is one example of the rtla osnoise hist tool output:
---------- %< ----------
[root@f34 ~]# rtla osnoise hist --bucket-size 10 --entries 100 -c 0-8 -d 1M -r 9000 -P F:1 # RTLA osnoise histogram
# Time unit is microseconds (us)
# Duration: 0 00:01:00
Index CPU-000 CPU-001 CPU-002 CPU-003 CPU-004 CPU-005 CPU-006 CPU-007 CPU-008
0 430 434 352 455 440 463 467 436 484 10 88 88 92 141 120 100 126 166 100 20 19 7 12 22 8 8 13 13 16 30 6 0 2 0 1 2 2 1 0 50 0 0 0 0 0 0 1 0 0 over: 0 0 0 0 0 0 0 0 0 count: 543 529 458 618 569 573 609 616 600 min: 0 0 0 0 0 0 0 0 0 avg: 0 0 0 0 0 0 0 0 0
max: 30 20 30 20 30 30 50 30 20
---------- >% ----------
Running
- rtla osnoise hist --help
provides information about the available options.
--- tools/tracing/rtla/src/osnoise.c | 6 +- tools/tracing/rtla/src/osnoise.h | 1 + tools/tracing/rtla/src/osnoise_hist.c | 799 ++++++++++++++++++++++++++ 3 files changed, 805 insertions(+), 1 deletion(-) create mode 100644 tools/tracing/rtla/src/osnoise_hist.c
tools/tracing/rtla/src/osnoise.c +5 -1
@@ -970,7 +970,8 @@ static void osnoise_usage(void)" usage: [rtla] osnoise [MODE] ..." ,"" ," modes:" , -" top - prints the summary from osnoise tracer" , +" top - prints the summary from osnoise tracer" , +" hist - prints a histogram of osnoise samples" ,"" ,"if no MODE is given, the top mode is called, passing the arguments" , NULL,
@@ -1005,6 +1006,9 @@ int osnoise_main(int argc, char *argv[]) }else if (strcmp (argv[1 ],"top" ) ==0 ) {osnoise_top_main (argc-1 , &argv[1 ]);exit (0 ); + }else if (strcmp (argv[1 ],"hist" ) ==0 ) { +osnoise_hist_main (argc-1 , &argv[1 ]); +exit (0 ); } usage:
tools/tracing/rtla/src/osnoise.h +1 -0
@@ -91,5 +91,6 @@ void osnoise_destroy_tool(struct osnoise_tool *top);struct osnoise_tool *osnoise_init_tool (char *tool_name);struct osnoise_tool *osnoise_init_trace_tool (char *tracer); +int osnoise_hist_main (int argc,char *argv[]);int osnoise_top_main (int argc,char **argv);int osnoise_main (int argc,char **argv);
tools/tracing/rtla/src/osnoise_hist.c new file +799 -0
799 lines not on this page · raw message
Signature · 1 line
2.31.1
Steven Rostedt
maintainerRe: [PATCH V7 05/14] rtla/osnoise: Add the hist mode
Quoted 9 lines · On Fri, 29 Oct 2021 21:26:08 +0200 Daniel Bristot de Oliveira…
tools/tracing/rtla/src/osnoise_hist.c new file +9
-0+/* +* osnoise_init_trace_hist - enable events used to collect histogram +*/ +static int osnoise_init_trace_hist (struct osnoise_tool *tool) +{ +struct osnoise_hist_params *params = tool->params; +struct osnoise_hist_data *data = tool->data; +int bucket_size; +char buff[128 ];
tools/tracing/rtla/src/osnoise_hist.c line 173
Quoted 10 lines
tools/tracing/rtla/src/osnoise_hist.c new file +10
-0+int retval =0 ; + +/* +* Set the size of the bucket. +*/ + bucket_size = params->output_divisor * params->bucket_size; +snprintf (buff,sizeof (buff),"duration.buckets=%d" , bucket_size); + + data->trace_hist =tracefs_hist_alloc (tool->trace.tep,"osnoise" ,"sample_threshold" , + buff,TRACEFS_HIST_KEY_NORMAL );
FYI, we changed the API (haven't tagged it yet, so we can do that :-) and the above needs to be:
data->trace_hist = tracefs_hist1d_alloc(tool->trace.tep, "osnoise", "sample_threshold",
-- Steve
Quoted 18 lines
tools/tracing/rtla/src/osnoise_hist.c new file +18
-0+if (!data->trace_hist) +return 1 ; + + retval =tracefs_hist_add_key (data->trace_hist,"cpu" ,0 ); +if (retval) + goto out_err; + + retval =tracefs_hist_start (tool->trace.inst, data->trace_hist); +if (retval) + goto out_err; + +return 0 ; + +out_err: +osnoise_destroy_trace_hist (tool); +return 1 ; +} +
Steven Rostedt
maintainerRe: [PATCH V7 05/14] rtla/osnoise: Add the hist mode
Quoted 12 lines · On Wed, 24 Nov 2021 17:12:12 -0500 Steven Rostedt <[email protected]> wrote:
tools/tracing/rtla/src/osnoise_hist.c line 173
Quoted 8 lines
tools/tracing/rtla/src/osnoise_hist.c new file +8
-0+/* +* Set the size of the bucket. +*/ + bucket_size = params->output_divisor * params->bucket_size; +snprintf (buff,sizeof (buff),"duration.buckets=%d" , bucket_size); + + data->trace_hist =tracefs_hist_alloc (tool->trace.tep,"osnoise" ,"sample_threshold" , + buff,TRACEFS_HIST_KEY_NORMAL );FYI, we changed the API (haven't tagged it yet, so we can do that :-) and the above needs to be:
Quoted 1 line
data->trace_hist = tracefs_hist1d_alloc(tool->trace.tep, "osnoise", "sample_threshold",
Or do you think that 1d should be the default, and we have:
tracefs_hist_alloc() -- 1d histogram
tracefs_hist_alloc_2d() -- 2d histogram
tracefs_hist_alloc_nd() -- Nd histogram?
??
We haven't tagged it yet, so we could change it again (and then your code will work as is).
-- Steve
Re: [PATCH V7 05/14] rtla/osnoise: Add the hist mode
Quoted 11 lines · Steven Rostedt
Quoted 11 lines
tools/tracing/rtla/src/osnoise_hist.c line 173
Quoted 8 lines
tools/tracing/rtla/src/osnoise_hist.c new file +8
-0+/* +* Set the size of the bucket. +*/ + bucket_size = params->output_divisor * params->bucket_size; +snprintf (buff,sizeof (buff),"duration.buckets=%d" , bucket_size); + + data->trace_hist =tracefs_hist_alloc (tool->trace.tep,"osnoise" ,"sample_threshold" , + buff,TRACEFS_HIST_KEY_NORMAL );FYI, we changed the API (haven't tagged it yet, so we can do that :-) and the above needs to be:
Quoted 8 lines
Quoted 1 line
data->trace_hist = tracefs_hist1d_alloc(tool->trace.tep, "osnoise", "sample_threshold",Or do you think that 1d should be the default, and we have:
tracefs_hist_alloc()-- 1d histogram
tracefs_hist_alloc_2d()-- 2d histogram
tracefs_hist_alloc_nd()-- Nd histogram???
IMHO, the function names in your second email sound more intuitive, i.e., tracefs_hist_alloc()/tracefs_hist_alloc_2d()/tracefs_hist_alloc_nd().
Quoted 2 lines
We haven't tagged it yet, so we could change it again (and then your code will work as is).
two birds with a single stone :-)
-- Daniel
Quoted 1 line
-- Steve
Steven Rostedt
maintainerRe: [PATCH V7 05/14] rtla/osnoise: Add the hist mode
Quoted 8 lines · On Thu, 25 Nov 2021 14:45:30 +0100 Daniel Bristot de Oliveira…
Quoted 7 lines
Or do you think that 1d should be the default, and we have:
tracefs_hist_alloc()-- 1d histogram
tracefs_hist_alloc_2d()-- 2d histogram
tracefs_hist_alloc_nd()-- Nd histogram???
Quoted 7 lines
IMHO, the function names in your second email sound more intuitive, i.e., tracefs_hist_alloc()/tracefs_hist_alloc_2d()/tracefs_hist_alloc_nd().
Quoted 2 lines
We haven't tagged it yet, so we could change it again (and then your code will work as is).
two birds with a single stone :-)
Just to confirm (confused by "second email" above), you prefer that we update the API to:
tracefs_hist_alloc() tracefs_hist_alloc_2d() tracefs_hist_alloc_nd()
Right?
Thanks,
-- Steve
Re: [PATCH V7 05/14] rtla/osnoise: Add the hist mode
Quoted 19 lines · Steven Rostedt
Quoted 8 lines · On Thu, 25 Nov 2021 14:45:30 +0100 Daniel Bristot de Oliveira…
Quoted 7 lines
Or do you think that 1d should be the default, and we have:
tracefs_hist_alloc()-- 1d histogram
tracefs_hist_alloc_2d()-- 2d histogram
tracefs_hist_alloc_nd()-- Nd histogram???
Quoted 7 lines
IMHO, the function names in your second email sound more intuitive, i.e., tracefs_hist_alloc()/tracefs_hist_alloc_2d()/tracefs_hist_alloc_nd().
Quoted 2 lines
We haven't tagged it yet, so we could change it again (and then your code will work as is).
two birds with a single stone :-)
Quoted 8 lines
Just to confirm (confused by "second email" above), you prefer that we update the API to:
tracefs_hist_alloc() tracefs_hist_alloc_2d() tracefs_hist_alloc_nd()Right?
Right!
-- Daniel
Quoted 4 lines
Thanks,
-- Steve
PATCH v7 6/14 rtla: Add timerlat tool and timelart top mode
In reply to Daniel Bristot de OliveiraThe rtla timerlat tool is an interface for the timerlat tracer. The timerlat tracer dispatches a kernel thread per-cpu. These threads set a periodic timer to wake themselves up and go back to sleep. After the wakeup, they collect and generate useful information for the debugging of operating system timer latency.
The timerlat tracer outputs information in two ways. It periodically prints the timer latency at the timer IRQ handler and the Thread handler. It also provides information for each noise via the osnoise tracepoints.
The rtla timerlat top mode displays a summary of the periodic output from the timerlat tracer.
Here is one example of the rtla timerlat tool output:
---------- %< ----------
[root@alien ~]# rtla timerlat top -c 0-3 -d 1m
Timer Latency
0 00:01:00 | IRQ Timer Latency (us) | Thread Timer Latency (us) CPU COUNT | cur min avg max | cur min avg max 0 #60001 | 0 0 0 3 | 1 1 1 6 1 #60001 | 0 0 0 3 | 2 1 1 5 2 #60001 | 0 0 1 6 | 1 1 2 7 3 #60001 | 0 0 0 7 | 1 1 1 11
---------- >% ----------
Running:
# rtla timerlat --help # rtla timerlat top --help provides information about the available options.
--- tools/tracing/rtla/Makefile | 2 + tools/tracing/rtla/src/rtla.c | 5 + tools/tracing/rtla/src/timerlat.c | 68 +++ tools/tracing/rtla/src/timerlat.h | 4 + tools/tracing/rtla/src/timerlat_top.c | 615 ++++++++++++++++++++++++++ 5 files changed, 694 insertions(+) create mode 100644 tools/tracing/rtla/src/timerlat.c create mode 100644 tools/tracing/rtla/src/timerlat.h create mode 100644 tools/tracing/rtla/src/timerlat_top.c
tools/tracing/rtla/Makefile +2 -0
@@ -62,6 +62,8 @@ install: $(STRIP) $(DESTDIR)$(BINDIR)/rtla @test ! -f $(DESTDIR)$(BINDIR)/osnoise || rm $(DESTDIR)$(BINDIR)/osnoise ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/osnoise + @test ! -f $(DESTDIR)$(BINDIR)/timerlat || rm $(DESTDIR)$(BINDIR)/timerlat + ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/timerlat .PHONY: clean tarball clean:
tools/tracing/rtla/src/rtla.c +5 -0
@@ -9,6 +9,7 @@#include <stdio.h> #include "osnoise.h" +#include "timerlat.h" /* * rtla_usage - print rtla usage
@@ -25,6 +26,7 @@ static void rtla_usage(void)"" ," commands:" ," osnoise - gives information about the operating system noise (osnoise)" , +" timerlat - measures the timer irq and thread latency" ,"" , NULL, };
@@ -45,6 +47,9 @@ int run_command(int argc, char **argv, int start_position)if (strcmp (argv[start_position],"osnoise" ) ==0 ) {osnoise_main (argc-start_position, &argv[start_position]); goto ran; + }else if (strcmp (argv[start_position],"timerlat" ) ==0 ) { +timerlat_main (argc-start_position, &argv[start_position]); + goto ran; }return 0 ;
tools/tracing/rtla/src/timerlat.c new file +68 -0
68 lines not on this page · raw message
tools/tracing/rtla/src/timerlat.h new file +4 -0
@@ -0,0 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 + +int timerlat_top_main (int argc,char *argv[]); +int timerlat_main (int argc,char *argv[]);
tools/tracing/rtla/src/timerlat_top.c new file +615 -0
615 lines not on this page · raw message
Signature · 1 line
2.31.1
PATCH v7 7/14 rtla/timerlat: Add timerlat hist mode
In reply to Daniel Bristot de OliveiraThe rtla hist hist mode displays a histogram of each tracer event occurrence, both for IRQ and timer latencies. The tool also allows many configurations of the timerlat tracer and the collection of the tracer output.
Here is one example of the rtla timerlat hist mode output:
---------- %< ----------
[root@alien ~]# rtla timerlat hist -c 0-3 -d 1M # RTLA timerlat histogram
# Time unit is microseconds (us)
# Duration: 0 00:01:00
Index IRQ-000 Thr-000 IRQ-001 Thr-001 IRQ-002 Thr-002 IRQ-003 Thr-003
0 58572 0 59373 0 58691 0 58895 0
1 1422 57021 628 57241 1310 56160 1102 56805
2 6 2931 0 2695 0 3567 4 3031 3 1 40 0 53 0 260 0 142 4 0 7 0 5 0 6 0 17 5 0 2 0 5 0 7 0 4 6 0 0 0 2 0 1 0 1 8 0 0 0 0 0 0 0 1 over: 0 0 0 0 0 0 0 0 count: 60001 60001 60001 60001 60001 60001 60001 60001 min: 0 1 0 1 0 1 0 1 avg: 0 1 0 1 0 1 0 1 max: 3 5 1 6 1 6 2 8
---------- >% ----------
Running
- rtla timerlat hist --help
provides information about the available options.
--- tools/tracing/rtla/src/timerlat.c | 6 +- tools/tracing/rtla/src/timerlat.h | 2 +- tools/tracing/rtla/src/timerlat_hist.c | 820 +++++++++++++++++++++++++ 3 files changed, 826 insertions(+), 2 deletions(-) create mode 100644 tools/tracing/rtla/src/timerlat_hist.c
tools/tracing/rtla/src/timerlat.c +5 -1
@@ -25,7 +25,8 @@ static void timerlat_usage(void)" usage: [rtla] timerlat [MODE] ..." ,"" ," modes:" , -" top - prints the summary from timerlat tracer" , +" top - prints the summary from timerlat tracer" , +" hist - prints a histogram of timer latencies" ,"" ,"if no MODE is given, the top mode is called, passing the arguments" , NULL,
@@ -60,6 +61,9 @@ int timerlat_main(int argc, char *argv[]) }else if (strcmp (argv[1 ],"top" ) ==0 ) {timerlat_top_main (argc-1 , &argv[1 ]);exit (0 ); + }else if (strcmp (argv[1 ],"hist" ) ==0 ) { +timerlat_hist_main (argc-1 , &argv[1 ]); +exit (0 ); } usage:
tools/tracing/rtla/src/timerlat.h +1 -1
@@ -1,4 +1,4 @@// SPDX-License-Identifier: GPL-2.0 - +int timerlat_hist_main (int argc,char *argv[]);int timerlat_top_main (int argc,char *argv[]);int timerlat_main (int argc,char *argv[]);
tools/tracing/rtla/src/timerlat_hist.c new file +820 -0
820 lines not on this page · raw message
Signature · 1 line
2.31.1
PATCH v7 8/14 rtla: Add Documentation
In reply to Daniel Bristot de OliveiraAdds the basis for rtla documentation. This patch also includes the rtla(1) man page.
As suggested by Jonathan Corbet, we are placing these man pages at Documentation/tools/rtla, using rst format. It is not linked to the official documentation, though.
The Makefile is based on bpftool's Documentation one.
--- Documentation/tools/rtla/Makefile | 41 +++++++++++++++++ Documentation/tools/rtla/common_appendix.rst | 12 +++++ Documentation/tools/rtla/rtla.rst | 48 ++++++++++++++++++++ tools/tracing/rtla/Makefile | 30 ++++++++++-- 4 files changed, 127 insertions(+), 4 deletions(-) create mode 100644 Documentation/tools/rtla/Makefile create mode 100644 Documentation/tools/rtla/common_appendix.rst create mode 100644 Documentation/tools/rtla/rtla.rst
Documentation/tools/rtla/Makefile new file +41 -0
@@ -0,0 +1,41 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Based on bpftool's Documentation Makefile + +INSTALL ?= install +RM ?= rm -f +RMDIR ?= rmdir --ignore-fail-on-non-empty + +PREFIX ?= /usr/share +MANDIR ?= $(PREFIX)/man +MAN1DIR = $(MANDIR)/man1 + +MAN1_RST = $(wildcard rtla*.rst) + +_DOC_MAN1 = $(patsubst %.rst,%.1,$(MAN1_RST)) +DOC_MAN1 = $(addprefix $(OUTPUT),$(_DOC_MAN1)) + +RST2MAN_DEP := $(shell command -v rst2man 2>/dev/null) +RST2MAN_OPTS += --verbose + +$(OUTPUT)%.1: %.rst +ifndef RST2MAN_DEP + $(error "rst2man not found, but required to generate man pages") +endif + rst2man $(RST2MAN_OPTS) $< > $@ + +man1: $(DOC_MAN1) +man: man1 + +clean: + $(RM) $(DOC_MAN1) + +install: man + $(INSTALL) -d -m 755 $(DESTDIR)$(MAN1DIR) + $(INSTALL) -m 644 $(DOC_MAN1) $(DESTDIR)$(MAN1DIR) + +uninstall: + $(RM) $(addprefix $(DESTDIR)$(MAN1DIR)/,$(_DOC_MAN1)) + $(RMDIR) $(DESTDIR)$(MAN1DIR) + +.PHONY: man man1 clean install uninstall +.DEFAULT_GOAL := man
Documentation/tools/rtla/common_appendix.rst new file +12 -0
@@ -0,0 +1,12 @@ +REPORTING BUGS +============== +Report bugs to <[email protected]> + +LICENSE +======= +**rtla** is Free Software licensed under the GNU GPLv2 + +COPYING +======= +Copyright \(C) 2021 Red Hat, Inc. Free use of this software is granted under +the terms of the GNU Public License (GPL).
Documentation/tools/rtla/rtla.rst new file +48 -0
48 lines not on this page · raw message
tools/tracing/rtla/Makefile +26 -4
@@ -45,18 +45,28 @@ DATADIR := /usr/share DOCDIR := $(DATADIR)/doc MANDIR := $(DATADIR)/man LICDIR := $(DATADIR)/licenses +SRCTREE := $(if $(BUILD_SRC),$(BUILD_SRC),$(CURDIR)) + +# If running from the tarball, man pages are stored in the Documentation +# dir. If running from the kernel source, man pages are stored in +# Documentation/tools/rtla/. +ifneq ($(wildcard Documentation/.*),) +DOCSRC = Documentation/ +else +DOCSRC = $(SRCTREE)/../../../Documentation/tools/rtla/ +endif .PHONY: all all: rtla -rtla: $(OBJ) +rtla: $(OBJ) doc $(CC) -o rtla $(LDFLAGS) $(OBJ) $(LIBS) static: $(OBJ) $(CC) -o rtla-static $(LDFLAGS) --static $(OBJ) $(LIBS) -lpthread -ldl .PHONY: install -install: +install: doc_install $(INSTALL) -d -m 755 $(DESTDIR)$(BINDIR) $(INSTALL) rtla -m 755 $(DESTDIR)$(BINDIR) $(STRIP) $(DESTDIR)$(BINDIR)/rtla
@@ -66,15 +76,27 @@ install: ln -s $(DESTDIR)$(BINDIR)/rtla $(DESTDIR)$(BINDIR)/timerlat .PHONY: clean tarball -clean: +clean: doc_clean @test ! -f rtla || rm rtla @test ! -f rtla-static || rm rtla-static @test ! -f src/rtla.o || rm src/rtla.o @test ! -f $(TARBALL) || rm -f $(TARBALL) @rm -rf *~ $(OBJ) *.tar.$(CEXT) -tarball: clean +tarball: clean rm -rf $(NAME)-$(VERSION) && mkdir $(NAME)-$(VERSION) cp -r $(DIRS) $(FILES) $(NAME)-$(VERSION) + mkdir $(NAME)-$(VERSION)/Documentation/ + cp -rp $(SRCTREE)/../../../Documentation/tools/rtla/* $(NAME)-$(VERSION)/Documentation/ tar $(TAROPTS) --exclude='*~' $(NAME)-$(VERSION) rm -rf $(NAME)-$(VERSION) + +.PHONY: doc doc_clean doc_install +doc: + $(MAKE) -C $(DOCSRC) + +doc_clean: + $(MAKE) -C $(DOCSRC) clean + +doc_install: + $(MAKE) -C $(DOCSRC) install
Signature · 1 line
2.31.1
PATCH v7 9/14 rtla: Add rtla osnoise man page
In reply to Daniel Bristot de OliveiraMan page for rtla osnoise command.
--- .../tools/rtla/common_osnoise_description.rst | 8 +++ Documentation/tools/rtla/rtla-osnoise.rst | 59 +++++++++++++++++++ 2 files changed, 67 insertions(+) create mode 100644 Documentation/tools/rtla/common_osnoise_description.rst create mode 100644 Documentation/tools/rtla/rtla-osnoise.rst
Documentation/tools/rtla/common_osnoise_description.rst new file +8 -0
@@ -0,0 +1,8 @@ +The **rtla osnoise** tool is an interface for the *osnoise* tracer. The +*osnoise* tracer dispatches a kernel thread per-cpu. These threads read the +time in a loop while with preemption, softirq and IRQs enabled, thus +allowing all the sources of operating systme noise during its execution. +The *osnoise*'s tracer threads take note of the delta between each time +read, along with an interference counter of all sources of interference. +At the end of each period, the *osnoise* tracer displays a summary of +the results.
Documentation/tools/rtla/rtla-osnoise.rst new file +59 -0
59 lines not on this page · raw message
Signature · 1 line
2.31.1
PATCH v7 10/14 rtla: Add rtla osnoise top documentation
In reply to Daniel Bristot de OliveiraMan page for rtla osnoise top mode.
--- Documentation/tools/rtla/common_options.rst | 24 ++++++++ .../tools/rtla/common_osnoise_options.rst | 17 ++++++ .../tools/rtla/common_top_options.rst | 3 + Documentation/tools/rtla/rtla-osnoise-top.rst | 61 +++++++++++++++++++ 4 files changed, 105 insertions(+) create mode 100644 Documentation/tools/rtla/common_options.rst create mode 100644 Documentation/tools/rtla/common_osnoise_options.rst create mode 100644 Documentation/tools/rtla/common_top_options.rst create mode 100644 Documentation/tools/rtla/rtla-osnoise-top.rst
Documentation/tools/rtla/common_options.rst new file +24 -0
@@ -0,0 +1,24 @@ +**-c**, **--cpus** *cpu-list* + + Set the osnoise tracer to run the sample threads in the cpu-list. + +**-d**, **--duration** *time[s|m|h|d]* + + Set the duration of the session. + +**-t**, **--trace**\[*=file*] + + Save the stopped trace to [*file|osnoise_trace.txt*]. + +**-P**, **--priority** *o:prio|r:prio|f:prio|d:runtime:period* + + Set scheduling parameters to the osnoise tracer threads, the format to set the priority are: + + - *o:prio* - use SCHED_OTHER with *prio*; + - *r:prio* - use SCHED_RR with *prio*; + - *f:prio* - use SCHED_FIFO with *prio*; + - *d:runtime[us|ms|s]:period[us|ms|s]* - use SCHED_DEADLINE with *runtime* and *period* in nanoseconds. + +**-h**, **--help** + + Print help menu.
Documentation/tools/rtla/common_osnoise_options.rst new file +17 -0
@@ -0,0 +1,17 @@ +**-p**, **--period** *us* + + Set the *osnoise* tracer period in microseconds. + +**-r**, **--runtime** *us* + + Set the *osnoise* tracer runtime in microseconds. + +**-s**, **--stop** *us* + + Stop the trace if a single sample is higher than the argument in microseconds. + If **-T** is set, it will also save the trace to the output. + +**-S**, **--stop-total** *us* + + Stop the trace if the total sample is higher than the argument in microseconds. + If **-T** is set, it will also save the trace to the output.
Documentation/tools/rtla/common_top_options.rst new file +3 -0
@@ -0,0 +1,3 @@ +**-q**, **--quiet** + + Print only a summary at the end of the session.
Documentation/tools/rtla/rtla-osnoise-top.rst new file +61 -0
61 lines not on this page · raw message
Signature · 1 line
2.31.1
PATCH v7 11/14 rtla: Add rtla osnoise hist documentation
In reply to Daniel Bristot de OliveiraMan page for rtla osnoise hist mode.
--- .../tools/rtla/common_hist_options.rst | 23 +++++++ .../tools/rtla/rtla-osnoise-hist.rst | 66 +++++++++++++++++++ 2 files changed, 89 insertions(+) create mode 100644 Documentation/tools/rtla/common_hist_options.rst create mode 100644 Documentation/tools/rtla/rtla-osnoise-hist.rst
Documentation/tools/rtla/common_hist_options.rst new file +23 -0
@@ -0,0 +1,23 @@ +**-b**, **--bucket-size** *N* + + Set the histogram bucket size (default *1*). + +**-e**, **--entries** *N* + + Set the number of entries of the histogram (default 256). + +**--no-header** + + Do not print header. + +**--no-summary** + + Do not print summary. + +**--no-index** + + Do not print index. + +**--with-zeros** + + Print zero only entries.
Documentation/tools/rtla/rtla-osnoise-hist.rst new file +66 -0
66 lines not on this page · raw message
Signature · 1 line
2.31.1
PATCH v7 12/14 rtla: Add rtla timerlat documentation
In reply to Daniel Bristot de OliveiraMan page for rtla timerlat tool.
--- .../rtla/common_timerlat_description.rst | 10 ++++ Documentation/tools/rtla/rtla-timerlat.rst | 57 +++++++++++++++++++ 2 files changed, 67 insertions(+) create mode 100644 Documentation/tools/rtla/common_timerlat_description.rst create mode 100644 Documentation/tools/rtla/rtla-timerlat.rst
Documentation/tools/rtla/common_timerlat_description.rst new file +10 -0
@@ -0,0 +1,10 @@ +The **rtla timerlat** tool is an interface for the *timerlat* tracer. The +*timerlat* tracer dispatches a kernel thread per-cpu. These threads +set a periodic timer to wake themselves up and go back to sleep. After +the wakeup, they collect and generate useful information for the +debugging of operating system timer latency. + +The *timerlat* tracer outputs information in two ways. It periodically +prints the timer latency at the timer *IRQ* handler and the *Thread* +handler. It also enable the trace of the most relevant information via +**osnoise:** tracepoints.
Documentation/tools/rtla/rtla-timerlat.rst new file +57 -0
57 lines not on this page · raw message
Signature · 1 line
2.31.1
PATCH v7 13/14 rtla: Add rtla timerlat top documentation
In reply to Daniel Bristot de OliveiraMan page for rtla timerlat top mode.
--- .../tools/rtla/common_timerlat_options.rst | 16 ++ .../tools/rtla/rtla-timerlat-top.rst | 145 ++++++++++++++++++ 2 files changed, 161 insertions(+) create mode 100644 Documentation/tools/rtla/common_timerlat_options.rst create mode 100644 Documentation/tools/rtla/rtla-timerlat-top.rst
Documentation/tools/rtla/common_timerlat_options.rst new file +16 -0
@@ -0,0 +1,16 @@ +**-p**, **--period** *us* + + Set the *timerlat* tracer period in microseconds. + +**-i**, **--irq** *us* + + Stop trace if the *IRQ* latency is higher than the argument in us. + +**-T**, **--thread** *us* + + Stop trace if the *Thread* latency is higher than the argument in us. + +**-s**, **--stack** *us* + + Save the stack trace at the *IRQ* if a *Thread* latency is higher than the + argument in us.
Documentation/tools/rtla/rtla-timerlat-top.rst new file +145 -0
145 lines not on this page · raw message
Signature · 1 line
2.31.1
PATCH v7 14/14 rtla: Add rtla timerlat hist documentation
In reply to Daniel Bristot de OliveiraMan page for rtla timerlat hist mode.
--- .../tools/rtla/rtla-timerlat-hist.rst | 106 ++++++++++++++++++ 1 file changed, 106 insertions(+) create mode 100644 Documentation/tools/rtla/rtla-timerlat-hist.rst
Documentation/tools/rtla/rtla-timerlat-hist.rst new file +106 -0
@@ -0,0 +1,106 @@ +===================== +rtla-timerlat-hist +===================== +------------------------------------------------ +Histograms of the operating system timer latency +------------------------------------------------ + +:Manual section: 1 + +SYNOPSIS +======== +**rtla timerlat hist** [*OPTIONS*] ... + +DESCRIPTION +=========== + +.. include:: common_timerlat_description.rst + +The **rtla timerlat hist** displays a histogram of each tracer event +occurrence. This tool uses the periodic information, and the +**osnoise:** tracepoints are enabled when using the **-T** option. + +OPTIONS +======= + +.. include:: common_timerlat_options.rst + +.. include:: common_hist_options.rst + +.. include:: common_options.rst + +EXAMPLE +======= +In the example below, **rtla timerlat hist** is set to run for *10* minutes, +in the cpus *0-4*, *skipping zero* only lines. Moreover, **rtla timerlat +hist** will change the priority of the *timelat* threads to run under +*SCHED_DEADLINE* priority, with a *10us* runtime every *1ms* period. The +*1ms* period is also passed to the *timerlat* tracer:: + + [root@alien ~]# timerlat hist -d 10m -c 0-4 -P d:100us:1ms -p 1ms + # RTLA timerlat histogram + # Time unit is microseconds (us) + # Duration: 0 00:10:00 + Index IRQ-000 Thr-000 IRQ-001 Thr-001 IRQ-002 Thr-002 IRQ-003 Thr-003 IRQ-004 Thr-004 + 0 276489 0 206089 0 466018 0 481102 0 205546 0 + 1 318327 35487 388149 30024 94531 48382 83082 71078 388026 55730 + 2 3282 122584 4019 126527 28231 109012 23311 89309 4568 98739 + 3 940 11815 837 9863 6209 16227 6895 17196 910 9780 + 4 444 17287 424 11574 2097 38443 2169 36736 462 13476 + 5 206 43291 255 25581 1223 101908 1304 101137 236 28913 + 6 132 101501 96 64584 635 213774 757 215471 99 73453 + 7 74 169347 65 124758 350 57466 441 53639 69 148573 + 8 53 85183 31 156751 229 9052 306 9026 39 139907 + 9 22 10387 12 42762 161 2554 225 2689 19 26192 + 10 13 1898 8 5770 114 1247 128 1405 13 3772 + 11 9 560 9 924 71 686 76 765 8 713 + 12 4 256 2 360 50 411 64 474 3 278 + 13 2 167 2 172 43 256 53 350 4 180 + 14 1 88 1 116 15 198 42 223 0 115 + 15 2 63 3 94 11 139 20 150 0 58 + 16 2 37 0 56 5 78 10 102 0 39 + 17 0 18 0 28 4 57 8 80 0 15 + 18 0 8 0 17 2 50 6 56 0 12 + 19 0 9 0 5 0 19 0 48 0 18 + 20 0 4 0 8 0 11 2 27 0 4 + 21 0 2 0 3 1 9 1 18 0 6 + 22 0 1 0 3 1 7 0 3 0 5 + 23 0 2 0 4 0 2 0 7 0 2 + 24 0 2 0 2 1 3 0 3 0 5 + 25 0 0 0 1 0 1 0 1 0 3 + 26 0 1 0 0 0 2 0 2 0 0 + 27 0 0 0 3 0 1 0 0 0 1 + 28 0 0 0 3 0 0 0 1 0 0 + 29 0 0 0 2 0 2 0 1 0 3 + 30 0 1 0 0 0 0 0 0 0 0 + 31 0 1 0 0 0 0 0 2 0 2 + 32 0 0 0 1 0 2 0 0 0 0 + 33 0 0 0 2 0 0 0 0 0 1 + 34 0 0 0 0 0 0 0 0 0 2 + 35 0 1 0 1 0 0 0 0 0 1 + 36 0 1 0 0 0 1 0 1 0 0 + 37 0 0 0 1 0 0 0 0 0 0 + 40 0 0 0 0 0 1 0 1 0 0 + 41 0 0 0 0 0 0 0 0 0 1 + 42 0 0 0 0 0 0 0 0 0 1 + 44 0 0 0 0 0 1 0 0 0 0 + 46 0 0 0 0 0 0 0 1 0 0 + 47 0 0 0 0 0 0 0 0 0 1 + 50 0 0 0 0 0 0 0 0 0 1 + 54 0 0 0 1 0 0 0 0 0 0 + 58 0 0 0 1 0 0 0 0 0 0 + over: 0 0 0 0 0 0 0 0 0 0 + count: 600002 600002 600002 600002 600002 600002 600002 600002 600002 600002 + min: 0 1 0 1 0 1 0 1 0 1 + avg: 0 5 0 5 0 4 0 4 0 5 + max: 16 36 15 58 24 44 21 46 13 50 + +SEE ALSO +======== +**rtla-timerlat**\(1), **rtla-timerlat-top**\(1) + +*timerlat* tracer documentation: <https://www.kernel.org/doc/html/latest/trace/timerlat-tracer.html> + +AUTHOR +====== +Written by Daniel Bristot de Oliveira <[email protected]>
Signature · 1 line
2.31.1
Steven Rostedt
maintainerQuoted 14 lines · On Fri, 29 Oct 2021 21:26:03 +0200 Daniel Bristot de Oliveira…
The rtla(1) is a meta-tool that includes a set of commands that aims to analyze the real-time properties of Linux. But instead of testing Linux as a black box, rtla leverages kernel tracing capabilities to provide precise information about the properties and root causes of unexpected results.
To start, it presents an interface to the osnoise and timerlat tracers. In the future, it will also serve as home to the rtsl [1] and other latency/noise tracers.
If you just want to run it, you can download the tarball here:
- https://bristot.me/files/rtla/tarball/rtla-0.3.tar.bz2To compile rtla on fedora you need:
Quoted 14 lines
$ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.git $ cd libtraceevent/ $ make $ sudo make install $ cd .. $ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtracefs.git $ cd libtracefs/ $ make $ sudo make install $ cd .. $ sudo dnf install python3-docutils procps-devel $ cd $rtla_src $ make $ sudo make install
Add a README to the above, so that people will know what dependencies there are.
Thanks!
-- Steve
Quoted 115 lines
The tracing option (-t) depends some kernel patches that are available at [2].
This tool was be discussed at the RT-MC during LPC2021 [3]
[1] rtsl: https://github.com/bristot/rtsl/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://youtu.be/cZUzc0U1jJ4Changes from v6:
- Revisited osnoise option config functions
- Properly handles offline CPUs
- Some cleanups
- Fixed an histogram allocation problem (Tao Zhou)
- Revisited open()/read()/write() (Tao Zhou)
Changes from v5:
- Fix retval check insave_trace_to_file()(Tao Zhou)
- Fix goto logic insave_trace_to_file()(Tao Zhou)
- Removed unusedsave_trace_pipe_to_file()function
- Correctly handle an error on osnoise_set_* functions during "apply config" for all tools (Tao Zhou)
Changes from v3:
- Add cross-compile support (Ahmed S. Darwish)
- Move documentation toDocumentation/tools/rtla(Jonathan Corbet)
- Use .rst format for documentation (Jonathan Corbet)
- Use include option from .rst to group common parts of the documentation
-Makefile(main and doc) cleanups
Changes from v2:
- Fix the miss conception of the "size" for kernel histograms (Steven/Tom)
- Change the --skip-zeros to --with-zeros option as the former is better for humans (and the latter for computers to plot charts).
- A lot of checkpatch fixes for the user-space part.
Changes from v1:
- Fixes -t options on osnoise tracers (-t means --trace for all tools now)
- Fixes --bucket-size references (not --bucket_size)Daniel Bristot de Oliveira (14):
rtla: Real-Time Linux Analysis tool
rtla: Helper functions for rtla
rtla: Add osnoise tool
rtla/osnoise: Add osnoise top mode
rtla/osnoise: Add the hist mode
rtla: Add timerlat tool and timelart top mode
rtla/timerlat: Add timerlat hist mode
rtla: Add Documentation
rtla: Add rtla osnoise man page
rtla: Add rtla osnoise top documentation
rtla: Add rtla osnoise hist documentation
rtla: Add rtla timerlat documentation
rtla: Add rtla timerlat top documentation
rtla: Add rtla timerlat hist documentationDocumentation/tools/rtla/Makefile | 41 + Documentation/tools/rtla/common_appendix.rst | 12 + .../tools/rtla/common_hist_options.rst | 23 + Documentation/tools/rtla/common_options.rst | 24 + .../tools/rtla/common_osnoise_description.rst | 8 + .../tools/rtla/common_osnoise_options.rst | 17 + .../rtla/common_timerlat_description.rst | 10 + .../tools/rtla/common_timerlat_options.rst | 16 + .../tools/rtla/common_top_options.rst | 3 + .../tools/rtla/rtla-osnoise-hist.rst | 66 ++ Documentation/tools/rtla/rtla-osnoise-top.rst | 61 + Documentation/tools/rtla/rtla-osnoise.rst | 59 + .../tools/rtla/rtla-timerlat-hist.rst | 106 ++ .../tools/rtla/rtla-timerlat-top.rst | 145 +++ Documentation/tools/rtla/rtla-timerlat.rst | 57 + Documentation/tools/rtla/rtla.rst | 48 + tools/tracing/rtla/Makefile | 102 ++ tools/tracing/rtla/src/osnoise.c | 1017 +++++++++++++++++ tools/tracing/rtla/src/osnoise.h | 96 ++ tools/tracing/rtla/src/osnoise_hist.c | 799 +++++++++++++ tools/tracing/rtla/src/osnoise_top.c | 577 ++++++++++ tools/tracing/rtla/src/rtla.c | 87 ++ tools/tracing/rtla/src/timerlat.c | 72 ++ tools/tracing/rtla/src/timerlat.h | 4 + tools/tracing/rtla/src/timerlat_hist.c | 820 +++++++++++++ tools/tracing/rtla/src/timerlat_top.c | 615 ++++++++++ tools/tracing/rtla/src/trace.c | 192 ++++ tools/tracing/rtla/src/trace.h | 27 + tools/tracing/rtla/src/utils.c | 433 +++++++ tools/tracing/rtla/src/utils.h | 56 + 30 files changed, 5593 insertions(+) create mode 100644 Documentation/tools/rtla/Makefile create mode 100644 Documentation/tools/rtla/common_appendix.rst create mode 100644 Documentation/tools/rtla/common_hist_options.rst create mode 100644 Documentation/tools/rtla/common_options.rst create mode 100644 Documentation/tools/rtla/common_osnoise_description.rst create mode 100644 Documentation/tools/rtla/common_osnoise_options.rst create mode 100644 Documentation/tools/rtla/common_timerlat_description.rst create mode 100644 Documentation/tools/rtla/common_timerlat_options.rst create mode 100644 Documentation/tools/rtla/common_top_options.rst create mode 100644 Documentation/tools/rtla/rtla-osnoise-hist.rst create mode 100644 Documentation/tools/rtla/rtla-osnoise-top.rst create mode 100644 Documentation/tools/rtla/rtla-osnoise.rst create mode 100644 Documentation/tools/rtla/rtla-timerlat-hist.rst create mode 100644 Documentation/tools/rtla/rtla-timerlat-top.rst create mode 100644 Documentation/tools/rtla/rtla-timerlat.rst create mode 100644 Documentation/tools/rtla/rtla.rst create mode 100644 tools/tracing/rtla/Makefile create mode 100644 tools/tracing/rtla/src/osnoise.c create mode 100644 tools/tracing/rtla/src/osnoise.h create mode 100644 tools/tracing/rtla/src/osnoise_hist.c create mode 100644 tools/tracing/rtla/src/osnoise_top.c create mode 100644 tools/tracing/rtla/src/rtla.c create mode 100644 tools/tracing/rtla/src/timerlat.c create mode 100644 tools/tracing/rtla/src/timerlat.h create mode 100644 tools/tracing/rtla/src/timerlat_hist.c create mode 100644 tools/tracing/rtla/src/timerlat_top.c create mode 100644 tools/tracing/rtla/src/trace.c create mode 100644 tools/tracing/rtla/src/trace.h create mode 100644 tools/tracing/rtla/src/utils.c create mode 100644 tools/tracing/rtla/src/utils.h
Quoted 31 lines · Steven Rostedt
Quoted 14 lines · On Fri, 29 Oct 2021 21:26:03 +0200 Daniel Bristot de Oliveira…
The rtla(1) is a meta-tool that includes a set of commands that aims to analyze the real-time properties of Linux. But instead of testing Linux as a black box, rtla leverages kernel tracing capabilities to provide precise information about the properties and root causes of unexpected results.
To start, it presents an interface to the osnoise and timerlat tracers. In the future, it will also serve as home to the rtsl [1] and other latency/noise tracers.
If you just want to run it, you can download the tarball here:
- https://bristot.me/files/rtla/tarball/rtla-0.3.tar.bz2To compile rtla on fedora you need:
Quoted 14 lines
$ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.git $ cd libtraceevent/ $ make $ sudo make install $ cd .. $ git clone git://git.kernel.org/pub/scm/libs/libtrace/libtracefs.git $ cd libtracefs/ $ make $ sudo make install $ cd .. $ sudo dnf install python3-docutils procps-devel $ cd $rtla_src $ make $ sudo make install
Quoted 2 lines
Add a README to the above, so that people will know what dependencies there are.
I agree! Will add in the next version.
Thanks Steve!
-- Daniel
Quoted 5 lines
Thanks!
-- Steve