Improvements to the tracing documentation
Theodore Ts'o
maintainercover Improvements to the tracing documentation
At the kernel panel at the Linux Foundation Collaboration Summit, I mentioned that the documentation for the tracing subsystem could be improved; in particular, that very often the needed documentation for how the various tracers worked could only be found in the git commit logs, Kconfig files, and of course, reading the source.
Since criticisms should be followed up with some attempts to improve the situation, here are some patches to improve the tracing documentation.
I'm not sure the documentation is 100% correct, but by submitting it, hopefully that the tracer authors will get flushed out of the woodwork and be inspired to improve what has been submitted. :-)
- Ted
Theodore Ts'o
maintainerRFC PATCH 1/3 tracing: Update documentation references in kernel/trace/Kconfig
Commit 66bb7488 consolidated documentation files into Documentation/trace, but it didn't update references found in the kernel/trace/Kconfig file. Fix this.
--- kernel/trace/Kconfig | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
kernel/trace/Kconfig +2 -2
@@ -312,7 +312,7 @@ config KMEMTRACE and profile kernel code. This requires an userspace application to use. See - Documentation/vm/kmemtrace.txt for more information. + Documentation/trace/kmemtrace.txt for more information. Saying Y will make the kernel somewhat larger and slower. However, if you disable kmemtrace at run-time or boot-time, the performance
@@ -403,7 +403,7 @@ config MMIOTRACE implementation and works via page faults. Tracing is disabled by default and can be enabled at run-time. - See Documentation/tracers/mmiotrace.txt. + See Documentation/trace/mmiotrace.txt. If you are not helping to develop drivers, say N. config MMIOTRACE_TEST
Signature · 1 line
1.5.6.3
Theodore Ts'o
maintainerRFC PATCH 2/3 tracing: Document the event tracing system
--- Documentation/trace/events.txt | 132 ++++++++++++++++++++++++++++++++++++++++ 1 files changed, 132 insertions(+), 0 deletions(-) create mode 100644 Documentation/trace/events.txt
Documentation/trace/events.txt new file +132 -0
@@ -0,0 +1,132 @@ + Event Tracing + + Documentation written by Theodore Ts'o + +Introduction +============ + +Tracepoints (see Documentation/tracing/tracepoints.txt) can be used +without creating custom kernel modules to register probe functions using +the event tracing infrastructure. + +Not all tracepoints can be traced using the event tracing system; the +kernel developer must provide code snippets which define how the tracing +information is saved into the tracing buffer, and how the the tracing +information should be printed. + +Using Event Tracing +=================== + +The events which are available for tracing can be found in the file +/sys/kernel/debug/tracing/available_events. + +To enable a particular event, such as 'sched_wakeup', simply echo it to +/sys/debug/tracing/set_event. For example: + + # echo sched_wakeup > /sys/kernel/debug/tracing/set_event + +To disable an event, echo the event name to the set_event file prefixed +with an exclamation point: + + # echo '!sched_wakeup' >> /sys/kernel//debug/tracing/set_event + +To disable events, echo an empty line to the set_event file: + + # echo > /sys/kernel/debug/tracing/set_event + +The events are organized into subsystems, such as ext4, irq, sched, +etc., and a full event name looks like this: <subsystem>:<event>. The +subsystem name is optional, but it is displayed in the available_events +file. All of the events in a subsystem can be specified via the syntax +"<subsystem>:*"; for example, to enable all irq events, you can use the +command: + + # echo 'irq:*' > /sys/kernel/debug/tracing/set_event + +Defining an event-enabled tracepoint +------------------------------------ + +A kernel developer which wishes to define an event-enabled tracepoint +must declare the tracepoint using TRACE_EVENT instead of DECLARE_TRACE. +This is done via two header files in include/trace. For example, to +event-enable the jbd2 subsystem, we must create two files, +include/trace/jbd2.h and include/trace/jbd2_event_types.h. The +include/trace/jbd2.h file should be included by kernel source files that +will have a tracepoint inserted, and might look like this: + +#ifndef _TRACE_JBD2_H +#define _TRACE_JBD2_H + +#include <linux/jbd2.h> +#include <linux/tracepoint.h> + +#include <trace/jbd2_event_types.h> + +#endif + +In a file that utilizes a jbd2 tracepoint, this header file would be +included. Note that you still have to use DEFINE_TRACE(). So for +example, if fs/jbd2/commit.c planned to use the jbd2_start_commit +tracepoint, it would have the following near the beginning of the file: + +#include <trace/jbd2.h> + +DEFINE_TRACE(jbd2_start_commit); + +Then in the function that would call the tracepoint, it would call the +tracepoint function. (For more information, please see the tracepoint +documentation in Documentation/trace/tracepoints.txt): + + trace_jbd2_start_commit(journal, commit_transaction); + +The code snippets which allow jbd2_start_commit to be an event-enabled +tracepoint are placed in the file include/trace/jbd2_event_types.h: + +/* use <trace/jbd2.h> instead */ +#ifndef TRACE_EVENT +# error Do not include this file directly. +# error Unless you know what you are doing. +#endif + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM jbd2 + +#include <linux/jbd2.h> + +TRACE_EVENT(jbd2_start_commit, + TP_PROTO(journal_t *journal, transaction_t *commit_transaction), + TP_ARGS(journal, commit_transaction), + TP_STRUCT__entry( + __array( char, devname, BDEVNAME_SIZE+24 ) + __field( int, transaction ) + ), + TP_fast_assign( + memcpy(__entry->devname, journal->j_devname, BDEVNAME_SIZE+24); + __entry->transaction = commit_transaction->t_tid; + ), + TP_printk("dev %s transaction %d", + __entry->devname, __entry->transaction) +); + +The TP_PROTO and TP_ARGS are unchanged from DECLARE_TRACE. The new +arguments to TRACE_EVENT are TP_STRUCT__entry, TP_fast_assign, and +TP_printk. + +TP_STRUCT__entry defines the data structure which will be stored in the +trace buffer. Normally, fields in __entry will be arrays or simple +types. It is possible to place data structures in __entry --- however, +pointers in the data structure can not be trusted, since they will be +accessed sometime later by TP_printk, and if the data structure contains +fields that will not or cannot be used by TP_printk, this will waste +space in the trace buffer. In general, data structures should be +avoided, unless they do only contain non-pointer types and all of the +fields will be used by TP_printk. + +TP_fast_assign defines the code snippet which saves information into the +__entry data structure, using the passed-in arguments defined in +TP_PROTO and TP_ARGS. + +Finally, TP_printk will print the __entry data structure. At the time +when code snippet defined by TP_printk is executed, it will not have +access to the TP_ARGS arguments; it can only use the information saved +in the __entry data structure.
Signature · 1 line
1.5.6.3
Theodore Ts'o
maintainerRFC PATCH 3/3 tracing: Add documentation for the power tracer
--- Documentation/power.txt | 19 +++++++++++++++++++ 1 files changed, 19 insertions(+), 0 deletions(-) create mode 100644 Documentation/power.txt
Documentation/power.txt new file +19 -0
@@ -0,0 +1,19 @@ +The power trace collects detailed information about C-state and P-state +transitions, instead of just looking at the high-level "average" +information. + +There is a helper script found in scrips/tracing/power.pl in the kernel +sources which can be used to parse this information and create a +Scalable Vector Graphics (SVG) picture from the trace data. + +To use this tracer: + + echo 0 > /sys/kernel/debug/tracing/tracing_enabled + echo power > /sys/kernel/debug/tracing/current_tracer + echo 1 > /sys/kernel/debug/tracing/tracing_enabled + sleep 1 + echo 0 > /sys/kernel/debug/tracing/tracing_enabled + cat /sys/kernel/debug/tracing/trace | \ + perl scripts/tracing/power.pl > out.sv + +
Signature · 1 line
1.5.6.3
Re: [PATCH, RFC 3/3] tracing: Add documentation for the power tracer
Documentation/power.txt
Quoted 5 lines · Theodore Ts'o
Documentation/power.txt new file +0
-0
Documentation/trace/power.txt?Documentation/power/trace.txt?
Documentation/power.txt line 1
Quoted 2 lines
Documentation/power.txt new file +1
-0@@ -0,0 +1,19 @@ +There is a helper script found in scrips/tracing/power.pl in the kernel
scripts/tracing/power.pl
Re: [PATCH, RFC 3/3] tracing: Add documentation for the power tracer
In reply to Theodore Ts'oQuoted 3 lines · On Sat, 11 Apr 2009 15:51:19 -0400 Theodore Ts'o <[email protected]> wrote:
Signed-off-by "Theodore Ts'o" <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Steven Rostedt <[email protected]>
Signature · 3 lines
Arjan van de Ven Intel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org
Theodore Ts'o
maintainer[tip:tracing/core] tracing: Add documentation for the power tracer
In reply to Theodore Ts'oCommit-ID: f7a3b3845cdfa4d6383b74139392ff5c259531fc Gitweb: http://git.kernel.org/tip/f7a3b3845cdfa4d6383b74139392ff5c259531fc Author: Theodore Ts'o <[email protected]> AuthorDate: Sat, 11 Apr 2009 15:51:19 -0400 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 12 Apr 2009 11:23:49 +0200
tracing: Add documentation for the power tracer
LKML-Reference: <[email protected]>
--- Documentation/trace/power.txt | 17 +++++++++++++++++ 1 files changed, 17 insertions(+), 0 deletions(-)
Documentation/trace/power.txt new file +17 -0
@@ -0,0 +1,17 @@ +The power trace collects detailed information about C-state and P-state +transitions, instead of just looking at the high-level "average" +information. + +There is a helper script found in scrips/tracing/power.pl in the kernel +sources which can be used to parse this information and create a +Scalable Vector Graphics (SVG) picture from the trace data. + +To use this tracer: + + echo 0 > /sys/kernel/debug/tracing/tracing_enabled + echo power > /sys/kernel/debug/tracing/current_tracer + echo 1 > /sys/kernel/debug/tracing/tracing_enabled + sleep 1 + echo 0 > /sys/kernel/debug/tracing/tracing_enabled + cat /sys/kernel/debug/tracing/trace | \ + perl scripts/tracing/power.pl > out.sv
Theodore Ts'o
maintainer[tip:tracing/core] tracing: Add documentation for the power tracer
In reply to Theodore Ts'oCommit-ID: 56c49951747f250d8398582509e02ae5ce1d36d1 Gitweb: http://git.kernel.org/tip/56c49951747f250d8398582509e02ae5ce1d36d1 Author: Theodore Ts'o <[email protected]> AuthorDate: Sat, 11 Apr 2009 15:51:19 -0400 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 12 Apr 2009 14:51:42 +0200
tracing: Add documentation for the power tracer
LKML-Reference: <[email protected]>
--- Documentation/trace/power.txt | 17 +++++++++++++++++ 1 files changed, 17 insertions(+), 0 deletions(-)
Documentation/trace/power.txt new file +17 -0
@@ -0,0 +1,17 @@ +The power tracer collects detailed information about C-state and P-state +transitions, instead of just looking at the high-level "average" +information. + +There is a helper script found in scrips/tracing/power.pl in the kernel +sources which can be used to parse this information and create a +Scalable Vector Graphics (SVG) picture from the trace data. + +To use this tracer: + + echo 0 > /sys/kernel/debug/tracing/tracing_enabled + echo power > /sys/kernel/debug/tracing/current_tracer + echo 1 > /sys/kernel/debug/tracing/tracing_enabled + sleep 1 + echo 0 > /sys/kernel/debug/tracing/tracing_enabled + cat /sys/kernel/debug/tracing/trace | \ + perl scripts/tracing/power.pl > out.sv
Theodore Ts'o
maintainer[tip:tracing/core] tracing: Document the event tracing system
In reply to Theodore Ts'oCommit-ID: 1542263998a2d143596972b2776c59c9247578a7 Gitweb: http://git.kernel.org/tip/1542263998a2d143596972b2776c59c9247578a7 Author: Theodore Ts'o <[email protected]> AuthorDate: Sat, 11 Apr 2009 15:51:18 -0400 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 12 Apr 2009 11:23:48 +0200
tracing: Document the event tracing system
LKML-Reference: <[email protected]>
--- Documentation/trace/events.txt | 132 ++++++++++++++++++++++++++++++++++++++++ 1 files changed, 132 insertions(+), 0 deletions(-)
Documentation/trace/events.txt new file +132 -0
@@ -0,0 +1,132 @@ + Event Tracing + + Documentation written by Theodore Ts'o + +Introduction +============ + +Tracepoints (see Documentation/tracing/tracepoints.txt) can be used +without creating custom kernel modules to register probe functions using +the event tracing infrastructure. + +Not all tracepoints can be traced using the event tracing system; the +kernel developer must provide code snippets which define how the tracing +information is saved into the tracing buffer, and how the the tracing +information should be printed. + +Using Event Tracing +=================== + +The events which are available for tracing can be found in the file +/sys/kernel/debug/tracing/available_events. + +To enable a particular event, such as 'sched_wakeup', simply echo it to +/sys/debug/tracing/set_event. For example: + + # echo sched_wakeup > /sys/kernel/debug/tracing/set_event + +To disable an event, echo the event name to the set_event file prefixed +with an exclamation point: + + # echo '!sched_wakeup' >> /sys/kernel//debug/tracing/set_event + +To disable events, echo an empty line to the set_event file: + + # echo > /sys/kernel/debug/tracing/set_event + +The events are organized into subsystems, such as ext4, irq, sched, +etc., and a full event name looks like this: <subsystem>:<event>. The +subsystem name is optional, but it is displayed in the available_events +file. All of the events in a subsystem can be specified via the syntax +"<subsystem>:*"; for example, to enable all irq events, you can use the +command: + + # echo 'irq:*' > /sys/kernel/debug/tracing/set_event + +Defining an event-enabled tracepoint +------------------------------------ + +A kernel developer which wishes to define an event-enabled tracepoint +must declare the tracepoint using TRACE_EVENT instead of DECLARE_TRACE. +This is done via two header files in include/trace. For example, to +event-enable the jbd2 subsystem, we must create two files, +include/trace/jbd2.h and include/trace/jbd2_event_types.h. The +include/trace/jbd2.h file should be included by kernel source files that +will have a tracepoint inserted, and might look like this: + +#ifndef _TRACE_JBD2_H +#define _TRACE_JBD2_H + +#include <linux/jbd2.h> +#include <linux/tracepoint.h> + +#include <trace/jbd2_event_types.h> + +#endif + +In a file that utilizes a jbd2 tracepoint, this header file would be +included. Note that you still have to use DEFINE_TRACE(). So for +example, if fs/jbd2/commit.c planned to use the jbd2_start_commit +tracepoint, it would have the following near the beginning of the file: + +#include <trace/jbd2.h> + +DEFINE_TRACE(jbd2_start_commit); + +Then in the function that would call the tracepoint, it would call the +tracepoint function. (For more information, please see the tracepoint +documentation in Documentation/trace/tracepoints.txt): + + trace_jbd2_start_commit(journal, commit_transaction); + +The code snippets which allow jbd2_start_commit to be an event-enabled +tracepoint are placed in the file include/trace/jbd2_event_types.h: + +/* use <trace/jbd2.h> instead */ +#ifndef TRACE_EVENT +# error Do not include this file directly. +# error Unless you know what you are doing. +#endif + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM jbd2 + +#include <linux/jbd2.h> + +TRACE_EVENT(jbd2_start_commit, + TP_PROTO(journal_t *journal, transaction_t *commit_transaction), + TP_ARGS(journal, commit_transaction), + TP_STRUCT__entry( + __array( char, devname, BDEVNAME_SIZE+24 ) + __field( int, transaction ) + ), + TP_fast_assign( + memcpy(__entry->devname, journal->j_devname, BDEVNAME_SIZE+24); + __entry->transaction = commit_transaction->t_tid; + ), + TP_printk("dev %s transaction %d", + __entry->devname, __entry->transaction) +); + +The TP_PROTO and TP_ARGS are unchanged from DECLARE_TRACE. The new +arguments to TRACE_EVENT are TP_STRUCT__entry, TP_fast_assign, and +TP_printk. + +TP_STRUCT__entry defines the data structure which will be stored in the +trace buffer. Normally, fields in __entry will be arrays or simple +types. It is possible to place data structures in __entry --- however, +pointers in the data structure can not be trusted, since they will be +accessed sometime later by TP_printk, and if the data structure contains +fields that will not or cannot be used by TP_printk, this will waste +space in the trace buffer. In general, data structures should be +avoided, unless they do only contain non-pointer types and all of the +fields will be used by TP_printk. + +TP_fast_assign defines the code snippet which saves information into the +__entry data structure, using the passed-in arguments defined in +TP_PROTO and TP_ARGS. + +Finally, TP_printk will print the __entry data structure. At the time +when code snippet defined by TP_printk is executed, it will not have +access to the TP_ARGS arguments; it can only use the information saved +in the __entry data structure.
Re: [tip:tracing/core] tracing: Document the event tracing system
[Theodore Ts'o - Sun, Apr 12, 2009 at 09:27:58AM +0000]
| Commit-ID: 1542263998a2d143596972b2776c59c9247578a7 | Gitweb: http://git.kernel.org/tip/1542263998a2d143596972b2776c59c9247578a7 | Author: Theodore Ts'o <[email protected]> | AuthorDate: Sat, 11 Apr 2009 15:51:18 -0400 | Committer: Ingo Molnar <[email protected]> | CommitDate: Sun, 12 Apr 2009 11:23:48 +0200 | | tracing: Document the event tracing system | | Signed-off-by: "Theodore Ts'o" <[email protected]> | Cc: Theodore Ts'o <[email protected]> | Cc: Steven Rostedt <[email protected]> | LKML-Reference: <[email protected]> | Signed-off-by: Ingo Molnar <[email protected]> | | | --- | Documentation/trace/events.txt | 132 ++++++++++++++++++++++++++++++++++++++++ | 1 files changed, 132 insertions(+), 0 deletions(-) | ... | +The events which are available for tracing can be found in the file | +/sys/kernel/debug/tracing/available_events. | + | +To enable a particular event, such as 'sched_wakeup', simply echo it to | +/sys/debug/tracing/set_event. For example: | + | + # echo sched_wakeup > /sys/kernel/debug/tracing/set_event | + | +To disable an event, echo the event name to the set_event file prefixed | +with an exclamation point: | + | + # echo '!sched_wakeup' >> /sys/kernel//debug/tracing/set_event ^^^
Since it's just being added and easy to fix (later maybe via trivial tree), I suppose here is quite a small misprint :)
| + | +To disable events, echo an empty line to the set_event file: | + | + # echo > /sys/kernel/debug/tracing/set_event | +
...
Cyrill
Theodore Ts'o
maintainer[tip:tracing/core] tracing: Document the event tracing system
In reply to Theodore Ts'oCommit-ID: abd41443ac76d3e9c29a8c1d9e9a3312306cc55e Gitweb: http://git.kernel.org/tip/abd41443ac76d3e9c29a8c1d9e9a3312306cc55e Author: Theodore Ts'o <[email protected]> AuthorDate: Sat, 11 Apr 2009 15:51:18 -0400 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 12 Apr 2009 14:57:12 +0200
tracing: Document the event tracing system
LKML-Reference: <[email protected]>
--- Documentation/trace/events.txt | 135 ++++++++++++++++++++++++++++++++++++++++ 1 files changed, 135 insertions(+), 0 deletions(-)
Documentation/trace/events.txt new file +135 -0
@@ -0,0 +1,135 @@ + Event Tracing + + Documentation written by Theodore Ts'o + +Introduction +============ + +Tracepoints (see Documentation/trace/tracepoints.txt) can be used +without creating custom kernel modules to register probe functions +using the event tracing infrastructure. + +Not all tracepoints can be traced using the event tracing system; +the kernel developer must provide code snippets which define how the +tracing information is saved into the tracing buffer, and how the +the tracing information should be printed. + +Using Event Tracing +=================== + +The events which are available for tracing can be found in the file +/sys/kernel/debug/tracing/available_events. + +To enable a particular event, such as 'sched_wakeup', simply echo it +to /sys/debug/tracing/set_event. For example: + + # echo sched_wakeup > /sys/kernel/debug/tracing/set_event + +[ Note: events can also be enabled/disabled via the 'enabled' toggle + found in the /sys/kernel/tracing/events/ hierarchy of directories. ] + +To disable an event, echo the event name to the set_event file prefixed +with an exclamation point: + + # echo '!sched_wakeup' >> /sys/kernel/debug/tracing/set_event + +To disable events, echo an empty line to the set_event file: + + # echo > /sys/kernel/debug/tracing/set_event + +The events are organized into subsystems, such as ext4, irq, sched, +etc., and a full event name looks like this: <subsystem>:<event>. The +subsystem name is optional, but it is displayed in the available_events +file. All of the events in a subsystem can be specified via the syntax +"<subsystem>:*"; for example, to enable all irq events, you can use the +command: + + # echo 'irq:*' > /sys/kernel/debug/tracing/set_event + +Defining an event-enabled tracepoint +------------------------------------ + +A kernel developer which wishes to define an event-enabled tracepoint +must declare the tracepoint using TRACE_EVENT instead of DECLARE_TRACE. +This is done via two header files in include/trace. For example, to +event-enable the jbd2 subsystem, we must create two files, +include/trace/jbd2.h and include/trace/jbd2_event_types.h. The +include/trace/jbd2.h file should be included by kernel source files that +will have a tracepoint inserted, and might look like this: + +#ifndef _TRACE_JBD2_H +#define _TRACE_JBD2_H + +#include <linux/jbd2.h> +#include <linux/tracepoint.h> + +#include <trace/jbd2_event_types.h> + +#endif + +In a file that utilizes a jbd2 tracepoint, this header file would be +included. Note that you still have to use DEFINE_TRACE(). So for +example, if fs/jbd2/commit.c planned to use the jbd2_start_commit +tracepoint, it would have the following near the beginning of the file: + +#include <trace/jbd2.h> + +DEFINE_TRACE(jbd2_start_commit); + +Then in the function that would call the tracepoint, it would call the +tracepoint function. (For more information, please see the tracepoint +documentation in Documentation/trace/tracepoints.txt): + + trace_jbd2_start_commit(journal, commit_transaction); + +The code snippets which allow jbd2_start_commit to be an event-enabled +tracepoint are placed in the file include/trace/jbd2_event_types.h: + +/* use <trace/jbd2.h> instead */ +#ifndef TRACE_EVENT +# error Do not include this file directly. +# error Unless you know what you are doing. +#endif + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM jbd2 + +#include <linux/jbd2.h> + +TRACE_EVENT(jbd2_start_commit, + TP_PROTO(journal_t *journal, transaction_t *commit_transaction), + TP_ARGS(journal, commit_transaction), + TP_STRUCT__entry( + __array( char, devname, BDEVNAME_SIZE+24 ) + __field( int, transaction ) + ), + TP_fast_assign( + memcpy(__entry->devname, journal->j_devname, BDEVNAME_SIZE+24); + __entry->transaction = commit_transaction->t_tid; + ), + TP_printk("dev %s transaction %d", + __entry->devname, __entry->transaction) +); + +The TP_PROTO and TP_ARGS are unchanged from DECLARE_TRACE. The new +arguments to TRACE_EVENT are TP_STRUCT__entry, TP_fast_assign, and +TP_printk. + +TP_STRUCT__entry defines the data structure which will be stored in the +trace buffer. Normally, fields in __entry will be arrays or simple +types. It is possible to place data structures in __entry --- however, +pointers in the data structure can not be trusted, since they will be +accessed sometime later by TP_printk, and if the data structure contains +fields that will not or cannot be used by TP_printk, this will waste +space in the trace buffer. In general, data structures should be +avoided, unless they do only contain non-pointer types and all of the +fields will be used by TP_printk. + +TP_fast_assign defines the code snippet which saves information into the +__entry data structure, using the passed-in arguments defined in +TP_PROTO and TP_ARGS. + +Finally, TP_printk will print the __entry data structure. At the time +when the code snippet defined by TP_printk is executed, it will not have +access to the TP_ARGS arguments; it can only use the information saved +in the __entry data structure.
Ingo Molnar
AppliedQuoted 7 lines · Theodore Ts'o
At the kernel panel at the Linux Foundation Collaboration Summit, I mentioned that the documentation for the tracing subsystem could be improved; in particular, that very often the needed documentation for how the various tracers worked could only be found in the git commit logs,
Kconfigfiles, and of course, reading the source.
Quoted 8 lines
Since criticisms should be followed up with some attempts to improve the situation, here are some patches to improve the tracing documentation.
I'm not sure the documentation is 100% correct, but by submitting it, hopefully that the tracer authors will get flushed out of the woodwork and be inspired to improve what has been submitted. :-)
Applied, thanks Ted!
Patch #1 was already present in the tracing tree - i've added both the event and the power tracer docs to Documentation/trace/.
Ingo
Theodore Tso
maintainerQuoted 10 lines · Ingo Molnar
Quoted 4 lines
I'm not sure the documentation is 100% correct, but by submitting it, hopefully that the tracer authors will get flushed out of the woodwork and be inspired to improve what has been submitted. :-)
Applied, thanks Ted!
Patch #1 was already present in the tracing tree - i've added both the event and the power tracer docs to
Documentation/trace/.
Thanks for applying it so fast, but did you fix up the spelling and pathname (Documentation/power.txt was supposed to be Documentation/trace/power.txt) fixups that folks pointed out? I was going to send an v2 update of the patches. So if you haven't pushed them out yet, you might want to pull them until I get them fixed up. Or I can send patches to fix up those issues, whichever is more convenient for you.
Thanks, regards,
- Ted
Quoted 6 lines · Theodore Tso
Quoted 5 lines · Ingo Molnar
Quoted 4 lines
I'm not sure the documentation is 100% correct, but by submitting it, hopefully that the tracer authors will get flushed out of the woodwork and be inspired to improve what has been submitted. :-)
Quoted 7 lines
Quoted 5 lines
Applied, thanks Ted!
Patch #1 was already present in the tracing tree - i've added both the event and the power tracer docs to
Documentation/trace/.Thanks for applying it so fast, [...]
Heh, i'm used to getting perfect patches from you :) Also, documentation is always lagging behind so every little piece helps, and it's not like it can actually break anything in the kernel.
Quoted 3 lines
[...] but did you fix up the spelling and pathname (
Documentation/power.txtwas supposed to beDocumentation/trace/power.txt) fixups that folks pointed out?
Yeah, i did that. I also fixed up a few other small details no-one pointed out.
Quoted 4 lines
[...] I was going to send an v2 update of the patches. So if you haven't pushed them out yet, you might want to pull them until I get them fixed up. Or I can send patches to fix up those issues, whichever is more convenient for you.
No need to resend.
Btw., you mention ext4 and jbd2 new-style tracepoints in the text. Does this mean you already have them coded up (i havent seen any patch posting from you), just that you cannot push it upstream yet because ext4 can be a module? We'll have modular new-style tracepoints soon.
Ingo
Theodore Tso
maintainerQuoted 5 lines · Ingo Molnar
Btw., you mention ext4 and jbd2 new-style tracepoints in the text. Does this mean you already have them coded up (i havent seen any patch posting from you), just that you cannot push it upstream yet because ext4 can be a module? We'll have modular new-style tracepoints soon.
Yes, I coded them up recently. I wanted to do some performance measurements, and being able to interleave the ext4 tracepoint logs with the block I/o tracer was definitely handy. Also, not having to fight with balky userspace tools (whether it is out-of-tree kernel support code, or random graphical userspace libraries when I could really care less about a GUI interface) was definitely a big win.
I only had two real problems. One is that the block I/O tracer only traces "real" devices, and not device mapper devices --- I could user the blktrace userspace tool, but then the results wouldn't be properly interleaved with the ext4 tracepoint logs. The second is that ext4 has its localized header files in fs/ext4/*.h, and not in include/linux/*.h, and that was problematic given that the trace code snippets in include/trace/ext4_event_types.h needed access to some internal ext4 data structures. I ultimately solved the latter by creating a include/linux/ext4_tracing_types.h, but I suspect this problem will go away when you have modular new-style tracepoints --- if not, I'd appreciately greatly if folks could consider whether or not this support could be added.
I'll attach the patches as replies to this mail thread so you can see what I've done. Any comments of anything I might have done "wrong" would be greatly appreciated.
- Ted
Theodore Ts'o
maintainerPATCH 1/2 jbd2: Convert instrumentation from markers to tracepoints
--- fs/jbd2/checkpoint.c | 7 ++- fs/jbd2/commit.c | 14 +++-- include/trace/jbd2.h | 9 ++++ include/trace/jbd2_event_types.h | 93 +++++++++++++++++++++++++++++++++++++ include/trace/trace_event_types.h | 1 + include/trace/trace_events.h | 1 + 6 files changed, 116 insertions(+), 9 deletions(-) create mode 100644 include/trace/jbd2.h create mode 100644 include/trace/jbd2_event_types.h
fs/jbd2/checkpoint.c +4 -3
@@ -20,9 +20,11 @@#include <linux/time.h> #include <linux/fs.h> #include <linux/jbd2.h> -#include <linux/marker.h> #include <linux/errno.h> #include <linux/slab.h> +#include <trace/jbd2.h> + +DEFINE_TRACE (jbd2_checkpoint);/* * Unlink a buffer from a transaction checkpoint list.
@@ -358,8 +360,7 @@ int jbd2_log_do_checkpoint(journal_t *journal)* journal straight away. */ result =jbd2_cleanup_journal_tail (journal); -trace_mark (jbd2_checkpoint,"dev %s need_checkpoint %d" , - journal->j_devname, result); +trace_jbd2_checkpoint (journal, result);jbd_debug (1 ,"cleanup_journal_tail returned %d\n" , result);if (result <=0 )return result;
fs/jbd2/commit.c +8 -6
@@ -16,7 +16,6 @@#include <linux/time.h> #include <linux/fs.h> #include <linux/jbd2.h> -#include <linux/marker.h> #include <linux/errno.h> #include <linux/slab.h> #include <linux/mm.h>
@@ -26,6 +25,11 @@#include <linux/writeback.h> #include <linux/backing-dev.h> #include <linux/bio.h> +#include <trace/jbd2.h> + +DEFINE_TRACE (jbd2_start_commit); +DEFINE_TRACE (jbd2_end_commit); +DEFINE_TRACE (jbd2_submit_inode_data);/* * Default IO end handler for temporary BJ_IO buffer_heads.
@@ -253,6 +257,7 @@ static int journal_submit_data_buffers(journal_t *journal,* block allocation with delalloc. We need to write * only allocated blocks here. */ +trace_jbd2_submit_inode_data (jinode->i_vfs_inode); err =journal_submit_inode_data_buffers (mapping);if (!ret) ret = err;
@@ -394,8 +399,7 @@ void jbd2_journal_commit_transaction(journal_t *journal) commit_transaction = journal->j_running_transaction;J_ASSERT (commit_transaction->t_state ==T_RUNNING ); -trace_mark (jbd2_start_commit,"dev %s transaction %d" , - journal->j_devname, commit_transaction->t_tid); +trace_jbd2_start_commit (journal, commit_transaction);jbd_debug (1 ,"JBD: starting commit of transaction %d\n" , commit_transaction->t_tid);
@@ -1053,9 +1057,7 @@ restart_loop:if (journal->j_commit_callback) journal->j_commit_callback (journal, commit_transaction); -trace_mark (jbd2_end_commit,"dev %s transaction %d head %d" , - journal->j_devname, commit_transaction->t_tid, - journal->j_tail_sequence); +trace_jbd2_end_commit (journal, commit_transaction);jbd_debug (1 ,"JBD: commit %d complete, head %d\n" , journal->j_commit_sequence, journal->j_tail_sequence);if (to_free)
include/trace/jbd2.h new file +9 -0
@@ -0,0 +1,9 @@ +#ifndef _TRACE_JBD2_H +#define _TRACE_JBD2_H + +#include <linux/jbd2.h> +#include <linux/tracepoint.h> + +#include <trace/jbd2_event_types.h> + +#endif
include/trace/jbd2_event_types.h new file +93 -0
93 lines not on this page · raw message
include/trace/trace_event_types.h +1 -0
@@ -3,3 +3,4 @@#include <trace/sched_event_types.h> #include <trace/irq_event_types.h> #include <trace/lockdep_event_types.h> +#include <trace/jbd2_event_types.h>
include/trace/trace_events.h +1 -0
@@ -3,3 +3,4 @@#include <trace/sched.h> #include <trace/irq.h> #include <trace/lockdep.h> +#include <trace/jbd2.h>
Signature · 1 line
1.5.6.3
Theodore Ts'o
maintainerPATCH 2/2 ext4: Convert instrumentation from markers to tracepoints
--- fs/ext4/ext4.h | 22 +-- fs/ext4/ext4_i.h | 12 - fs/ext4/fsync.c | 8 +- fs/ext4/ialloc.c | 17 +- fs/ext4/inode.c | 79 ++--- fs/ext4/mballoc.c | 84 ++--- fs/ext4/mballoc.h | 68 +---- fs/ext4/super.c | 6 +- include/linux/ext4_trace_types.h | 108 ++++++ include/trace/ext4.h | 8 + include/trace/ext4_event_types.h | 690 +++++++++++++++++++++++++++++++++++++ include/trace/trace_event_types.h | 1 + include/trace/trace_events.h | 1 + 13 files changed, 886 insertions(+), 218 deletions(-) create mode 100644 include/linux/ext4_trace_types.h create mode 100644 include/trace/ext4.h create mode 100644 include/trace/ext4_event_types.h
fs/ext4/ext4.h +1 -21
@@ -21,6 +21,7 @@#include <linux/magic.h> #include <linux/jbd2.h> #include <linux/quota.h> +#include <linux/ext4_trace_types.h> #include "ext4_i.h" /*
@@ -70,27 +71,6 @@#define EXT4_MB_DELALLOC_RESERVED 1024 -struct ext4_allocation_request { -/* target inode for block we're allocating */ -struct inode *inode; -/* logical block in target inode */ -ext4_lblk_t logical; -/* phys. target (a hint) */ -ext4_fsblk_t goal; -/* the closest logical allocated block to the left */ -ext4_lblk_t lleft; -/* phys. block for ^^^ */ -ext4_fsblk_t pleft; -/* the closest logical allocated block to the right */ -ext4_lblk_t lright; -/* phys. block for ^^^ */ -ext4_fsblk_t pright; -/* how many blocks we want to allocate */ -unsigned int len; -/* flags. see above EXT4_MB_HINT_* */ -unsigned int flags; -}; -/* * Special inodes numbers */
fs/ext4/ext4_i.h +0 -12
@@ -21,18 +21,6 @@#include <linux/seqlock.h> #include <linux/mutex.h> -/* data type for block offset of block group */ -typedef int ext4_grpblk_t ; - -/* data type for filesystem-wide blocks number */ -typedef unsigned long long ext4_fsblk_t ; - -/* data type for file logical block number */ -typedef __u32 ext4_lblk_t ; - -/* data type for block group number */ -typedef unsigned int ext4_group_t ; -/* * storage for cached extent */
fs/ext4/fsync.c +4 -4
@@ -28,10 +28,12 @@#include <linux/writeback.h> #include <linux/jbd2.h> #include <linux/blkdev.h> -#include <linux/marker.h> +#include <trace/ext4.h> #include "ext4.h" #include "ext4_jbd2.h" +DEFINE_TRACE (ext4_sync_file); +/* * akpm: A new design for ext4_sync_file(). *
@@ -52,9 +54,7 @@ int ext4_sync_file(struct file *file, struct dentry *dentry, int datasync)J_ASSERT (ext4_journal_current_handle () ==NULL ); -trace_mark (ext4_sync_file,"dev %s datasync %d ino %ld parent %ld" , - inode->i_sb->s_id, datasync, inode->i_ino, - dentry->d_parent->d_inode->i_ino); +trace_ext4_sync_file (file, dentry, datasync);/* * data=writeback:
fs/ext4/ialloc.c +8 -9
@@ -22,6 +22,7 @@#include <linux/random.h> #include <linux/bitops.h> #include <linux/blkdev.h> +#include <trace/ext4.h> #include <asm/byteorder.h> #include "ext4.h" #include "ext4_jbd2.h"
@@ -29,6 +30,10 @@#include "acl.h" #include "group.h" +DEFINE_TRACE (ext4_free_inode); +DEFINE_TRACE (ext4_request_inode); +DEFINE_TRACE (ext4_allocate_inode); +/* * ialloc.c contains the inodes allocation and deallocation routines */
@@ -209,11 +214,7 @@ void ext4_free_inode(handle_t *handle, struct inode *inode) ino = inode->i_ino;ext4_debug ("freeing inode %lu\n" , ino); -trace_mark (ext4_free_inode, -"dev %s ino %lu mode %d uid %lu gid %lu bocks %llu" , - sb->s_id, inode->i_ino, inode->i_mode, - (unsigned long ) inode->i_uid, (unsigned long ) inode->i_gid, - (unsigned long long ) inode->i_blocks); +trace_ext4_free_inode (inode);/* * Note: we must free any quota before locking the superblock,
@@ -817,8 +818,7 @@ struct inode *ext4_new_inode(handle_t *handle, struct inode *dir, int mode)return ERR_PTR (-EPERM ); sb = dir->i_sb; -trace_mark (ext4_request_inode,"dev %s dir %lu mode %d" , sb->s_id, - dir->i_ino, mode); +trace_ext4_request_inode (dir, mode); inode =new_inode (sb);if (!inode)return ERR_PTR (-ENOMEM );
@@ -1050,8 +1050,7 @@ got: }ext4_debug ("allocating inode %lu\n" , inode->i_ino); -trace_mark (ext4_allocate_inode,"dev %s ino %lu dir %lu mode %d" , - sb->s_id, inode->i_ino, dir->i_ino, mode); +trace_ext4_allocate_inode (inode, dir, mode); goto really_out; fail:ext4_std_error (sb, err);
fs/ext4/inode.c +24 -55
156 lines not on this page · raw message
fs/ext4/mballoc.c +36 -48
172 lines not on this page · raw message
fs/ext4/mballoc.h +1 -67
92 lines not on this page · raw message
fs/ext4/super.c +4 -2
@@ -36,9 +36,9 @@#include <linux/seq_file.h> #include <linux/proc_fs.h> #include <linux/ctype.h> -#include <linux/marker.h> #include <linux/log2.h> #include <linux/crc16.h> +#include <trace/ext4.h> #include <asm/uaccess.h> #include "ext4.h"
@@ -48,6 +48,8 @@#include "namei.h" #include "group.h" +DEFINE_TRACE (ext4_sync_fs); +struct proc_dir_entry *ext4_proc_root;static struct kset *ext4_kset;
@@ -3287,7 +3289,7 @@ static int ext4_sync_fs(struct super_block *sb, int wait)int ret =0 ;tid_t target; -trace_mark (ext4_sync_fs,"dev %s wait %d" , sb->s_id, wait); +trace_ext4_sync_fs (sb, wait); sb->s_dirt =0 ;if (EXT4_SB (sb)->s_journal) { if (jbd2_journal_start_commit(EXT4_SB(sb)->s_journal,
include/linux/ext4_trace_types.h new file +108 -0
108 lines not on this page · raw message
include/trace/ext4.h new file +8 -0
@@ -0,0 +1,8 @@ +#ifndef _TRACE_EXT4_H +#define _TRACE_EXT4_H + +#include <linux/tracepoint.h> + +#include <trace/ext4_event_types.h> + +#endif
include/trace/ext4_event_types.h new file +690 -0
690 lines not on this page · raw message
include/trace/trace_event_types.h +1 -0
@@ -4,3 +4,4 @@#include <trace/irq_event_types.h> #include <trace/lockdep_event_types.h> #include <trace/jbd2_event_types.h> +#include <trace/ext4_event_types.h>
include/trace/trace_events.h +1 -0
@@ -4,3 +4,4 @@#include <trace/irq.h> #include <trace/lockdep.h> #include <trace/jbd2.h> +#include <trace/ext4.h>
Signature · 1 line
1.5.6.3
Re: [PATCH 2/2] ext4: Convert instrumentation from markers to tracepoints
Quoted 11 lines · Theodore Ts'o
Signed-off-by "Theodore Ts'o" <[email protected]>--- fs/ext4/ext4.h | 22 +-- fs/ext4/ext4_i.h | 12 - fs/ext4/fsync.c | 8 +- fs/ext4/ialloc.c | 17 +- fs/ext4/inode.c | 79 ++--- fs/ext4/mballoc.c | 84 ++--- fs/ext4/mballoc.h | 68 +---- fs/ext4/super.c | 6 +- include/linux/ext4_trace_types.h | 108 ++++++
Quoted 8 lines
include/trace/ext4.h | 8 + include/trace/ext4_event_types.h | 690 +++++++++++++++++++++++++++++++++++++ include/trace/trace_event_types.h | 1 + include/trace/trace_events.h | 1 + 13 files changed, 886 insertions(+), 218 deletions(-) create mode 100644 include/linux/ext4_trace_types.h create mode 100644 include/trace/ext4.h create mode 100644 include/trace/ext4_event_types.h
Really nice stuff!
A general, high-level observation: it is visible that the conversion to TRACE_EVENTS() does increase the total linecount, but it also removes code from the most often used source code areas (fs/ext4/*.c) - and using an intuitive function-call-alike single-line tracepoint there.
I'd call that a positive effect.
It would still be nice to compact the 690 lines of (out of line) ext4_event_types.h definitions some more - and i think we'll be able to do it without affecting the usage sites.
OTOH, even those bits look fairly readable, and there's a fair amount of bang-for-bucks ratio - these tracepoints give us:
- integrated tracepoints
- zero-copy and per-cpu splice() based tracing
- binary tracing without printf overhead
- structured logging records exposed under /debug/tracing/events
- trace events embedded in function tracer output and other plugins
- user-defined, per tracepoint filter expressions
... so even if we were to keep the 690 lines of overhead with no further compacting, it still looks like a good deal to me, all things considered.
Ingo
Quoted 7 lines · Theodore Tso
Quoted 5 lines · Ingo Molnar
Btw., you mention ext4 and jbd2 new-style tracepoints in the text. Does this mean you already have them coded up (i havent seen any patch posting from you), just that you cannot push it upstream yet because ext4 can be a module? We'll have modular new-style tracepoints soon.
Quoted 7 lines
Yes, I coded them up recently. I wanted to do some performance measurements, and being able to interleave the ext4 tracepoint logs with the block I/o tracer was definitely handy. Also, not having to fight with balky userspace tools (whether it is out-of-tree kernel support code, or random graphical userspace libraries when I could really care less about a GUI interface) was definitely a big win.
Cool. [ And i guess you'll like the per tracepoint filter expressions too :-) ]
Quoted 4 lines
I only had two real problems. One is that the block I/O tracer only traces "real" devices, and not device mapper devices --- I could user the blktrace userspace tool, but then the results wouldn't be properly interleaved with the ext4 tracepoint logs.
Ok - i forwarded your mail to Li Zefan and he already posted a draft patch to attempt to solve this. "Cannot trace the primary block device my filesystem is using" is IMO a showstopper in your tracing work-flow critical path, so it needs some sort of quick solution.
Quoted 9 lines
The second is that ext4 has its localized header files in
fs/ext4/*.h, and not ininclude/linux/*.h, and that was problematic given that the trace code snippets ininclude/trace/ext4_event_types.hneeded access to some internal ext4 data structures. I ultimately solved the latter by creating ainclude/linux/ext4_tracing_types.h, but I suspect this problem will go away when you have modular new-style tracepoints --- if not, I'd appreciately greatly if folks could consider whether or not this support could be added.
I'd expect these problems to go away with Steve's module support and TRACE_EVENT()-simplification/cleanup series, yes.
Quoted 3 lines
I'll attach the patches as replies to this mail thread so you can see what I've done. Any comments of anything I might have done "wrong" would be greatly appreciated.
I dont think you can do anything "wrong" at this stage - the event tracer is still fresh out of the oven. It was shaped based on the needs of a few select core kernel non-modular subsystems (scheduler, irqs, etc.). Feedback from people like you with non-trivial independent tracing bits (you have a handful of useful-looking markers in ext4) is very much needed to shape and complete it.
So we'll sure try to address all your feedback and it would be extremely useful if you labeled all the steps that you found unintuitive, unnecessary, over-complicated or redundant or painful in any way. Please dont hold any of your punches ;-)
Ingo
Theodore Tso
maintainerQuoted 2 lines · Ingo Molnar
Cool. [ And i guess you'll like the per tracepoint filter expressions too :-) ]
I haven't played with them yet, but I was looking over the source code at them (since they aren't documented yet :-). It looks like at the moment only integer matches are allowed, right? That's a bit of an issue for me, since one of the things I'd really like to be able to do is filter based on devname (i.e., sda2). (Most of the time we only want to collect information for a particular block device or filesystem.)
Actually, the fact that I'm having to drop some 32 bytes for each jbd2 and ext4 trace log for the bdevname in the ring buffer is really for the birds. What I really want to do is just to drop in the dev_t, and then for the tracing infrastructure to have an efficient (cached) way of taking the dev_t and turning that back into struct block_device at TP_printk time so we can print the bdevname when it's needed. We deifnitely don't want to be calling bdget() in fs/block_dev.c each time we print a line in the tracing buffer! I'm guessing that's something the blktrace tracer would find handy as well.
Of course having more kernel code play with dev_t's directly isn't considered politically correct in some circles, but tough. :-) We can't exactly drop a pointer to a struct block_device in the trace buffer, since there's no guarantee it will still be valid when we read it out. Dropping in a dev_t is exactly what we want. It would be nice though if there was a way to specify a major/minor number as the filter predicate for the dev_t, and not to have the user generate the MAJOR/MINOR encoding. So some way of parsing "MKDEV(8, 4)" as the input to the filter predicate would probably be a really good thing to do.
- Ted
Quoted 11 lines · Theodore Tso
Quoted 2 lines · Ingo Molnar
Cool. [ And i guess you'll like the per tracepoint filter expressions too :-) ]
I haven't played with them yet, but I was looking over the source code at them (since they aren't documented yet :-). It looks like at the moment only integer matches are allowed, right? That's a bit of an issue for me, since one of the things I'd really like to be able to do is filter based on devname (i.e., sda2). (Most of the time we only want to collect information for a particular block device or filesystem.)
You can already do:
aldebaran:/debug/tracing/events/sched/sched_process_wait> echo "comm == Xorg" > filter aldebaran:/debug/tracing/events/sched/sched_process_wait> cat filter comm == Xorg
But string values depends on the type of the format field - so you cannot do string matches on integer fields.
For kdev_t matches i think we'll need native support for that type - in addition to the integer/string types. It will come up in other places as well and user-space knows about devices as well.
Quoted 10 lines
Actually, the fact that I'm having to drop some 32 bytes for each jbd2 and ext4 trace log for the bdevname in the ring buffer is really for the birds. What I really want to do is just to drop in the
dev_t, and then for the tracing infrastructure to have an efficient (cached) way of taking thedev_tand turning that back into structblock_deviceatTP_printktime so we can print the bdevname when it's needed. We deifnitely don't want to be callingbdget()infs/block_dev.ceach time we print a line in the tracing buffer! I'm guessing that's something the blktrace tracer would find handy as well.
Yeah.
It could be worked around right now by converting it to an integer but i think what we want is native support for kdev_t, together with all the usual convenience forms of specifying it: sda1 should work the same way as 8:1 or 0801. Even /dev/sda1 should be recognized in a filter expression.
Quoted 10 lines
Of course having more kernel code play with dev_t's directly isn't considered politically correct in some circles, but tough. :-) We can't exactly drop a pointer to a struct
block_devicein the trace buffer, since there's no guarantee it will still be valid when we read it out. Dropping in adev_tis exactly what we want. It would be nice though if there was a way to specify a major/minor number as the filter predicate for thedev_t, and not to have the user generate the MAJOR/MINOR encoding. So some way of parsing "MKDEV(8, 4)" as the input to the filter predicate would probably be a really good thing to do.
Yeah, exactly. We already have smarts in init/* to recognize certain device string patterns (for rootdev specification) - that could be factored out (it already is to a large degree) and reused. We dont need full udev enumeration really - we just need the most common variants.
Regardless of whether it's considered politically correct or not ;-) It's clearly useful.
Ingo
Theodore Tso
maintainerQuoted 6 lines · Ingo Molnar
It could be worked around right now by converting it to an integer but i think what we want is native support for
kdev_t, together with all the usual convenience forms of specifying it: sda1 should work the same way as 8:1 or 0801. Even/dev/sda1should be recognized in a filter expression.
Yeah, I could just drop in the integer now, and and just have TP_printk() display "(8, 2)" instead of "sda2". It's really a question of how far we want to take pretty-printing and parsing for ftrace, I suppose.
But if we are going to have end-users use it, having real pretty-printed names would be a good thing, IMHO. Especially if major/minor numbers start becoming completely random beasts, as some have proposed. (I think it's a terrible idea, but I'm clearly not politically correct. :-)
- Ted
Quoted 8 lines · Theodore Tso
Quoted 6 lines · Ingo Molnar
It could be worked around right now by converting it to an integer but i think what we want is native support for
kdev_t, together with all the usual convenience forms of specifying it:
sda1 should work the same way as 8:1 or 0801. Even/dev/sda1should be recognized in a filter expression.
Quoted 4 lines
Yeah, I could just drop in the integer now, and and just have
TP_printk()display "(8, 2)" instead of "sda2". It's really a question of how far we want to take pretty-printing and parsing for ftrace, I suppose.
We try to do it as far as daily use in /debug/tracing/ dictates. Interacting with the kernel on such a direct channel is really intuitive and useful in debugging and development sessions IMHO.
Raw binary records would encode it in an efficient and well-specified manner, so information density is not hurt by pretty-printing.
Quoted 5 lines
But if we are going to have end-users use it, having real pretty-printed names would be a good thing, IMHO. Especially if major/minor numbers start becoming completely random beasts, as some have proposed. (I think it's a terrible idea, but I'm clearly not politically correct. :-)
Sounds like a terrible idea to me too. If more space is needed then perhaps dynamically allocate the _new_ bits needed - but leave the well-established spaces alone. Making everything random looking is just asking for all sorts of trouble IMO. Making the system harder to understand at a glance, on such a fundamental level, seems silly.
Ingo
Quoted 38 lines · Ingo Molnar
Quoted 11 lines · Theodore Tso
Quoted 2 lines · Ingo Molnar
Cool. [ And i guess you'll like the per tracepoint filter expressions too :-) ]
I haven't played with them yet, but I was looking over the source code at them (since they aren't documented yet :-). It looks like at the moment only integer matches are allowed, right? That's a bit of an issue for me, since one of the things I'd really like to be able to do is filter based on devname (i.e., sda2). (Most of the time we only want to collect information for a particular block device or filesystem.)
You can already do:
aldebaran:/debug/tracing/events/sched/sched_process_wait> echo "comm == Xorg" > filter aldebaran:/debug/tracing/events/sched/sched_process_wait> cat filter comm == Xorg
But string values depends on the type of the format field - so you cannot do string matches on integer fields.
For
kdev_tmatches i think we'll need native support for that type - in addition to the integer/string types. It will come up in other places as well and user-space knows about devices as well.Quoted 10 lines
Actually, the fact that I'm having to drop some 32 bytes for each jbd2 and ext4 trace log for the bdevname in the ring buffer is really for the birds. What I really want to do is just to drop in the
dev_t, and then for the tracing infrastructure to have an efficient (cached) way of taking thedev_tand turning that back into structblock_deviceatTP_printktime so we can print the bdevname when it's needed. We deifnitely don't want to be callingbdget()infs/block_dev.ceach time we print a line in the tracing buffer! I'm guessing that's something the blktrace tracer would find handy as well.
Quoted 8 lines
Yeah.
It could be worked around right now by converting it to an integer but i think what we want is native support for
kdev_t, together with all the usual convenience forms of specifying it: sda1 should work the same way as 8:1 or 0801. Even/dev/sda1should be recognized in a filter expression.
Yeah, I'll make sure the new filter parser can handle all these forms, but hopefully in a general way that would allow different forms to be used as predicate values for any special type such as kdev_t. Maybe a simple expression matcher that when setting up the filter would map the matched value expression to some code that would generate the final filter value to be used in the run-time filter.
Tom
Quoted 21 lines
Quoted 10 lines
Of course having more kernel code play with dev_t's directly isn't considered politically correct in some circles, but tough. :-) We can't exactly drop a pointer to a struct
block_devicein the trace buffer, since there's no guarantee it will still be valid when we read it out. Dropping in adev_tis exactly what we want. It would be nice though if there was a way to specify a major/minor number as the filter predicate for thedev_t, and not to have the user generate the MAJOR/MINOR encoding. So some way of parsing "MKDEV(8, 4)" as the input to the filter predicate would probably be a really good thing to do.Yeah, exactly. We already have smarts in
init/*to recognize certain device string patterns (for rootdev specification) - that could be factored out (it already is to a large degree) and reused. We dont need full udev enumeration really - we just need the most common variants.Regardless of whether it's considered politically correct or not ;-) It's clearly useful.
Ingo