tracing: Add dynamic strings for synthetic events
Subsystems (2)
v2 cover tracing: Add dynamic strings for synthetic events
Hi,
This is v2 of the dynamic string support for synthetic events.
v1 was missing some very important code in the in-kernel API functions that should have reserved room for dynamic strings. This adds it along also some additional test code in the synth_event_gen_test module that tests the dynamic strings for those APIs.
I also added Axel's changelog text to the 'tracing: Fix parse_synth_field() error handling' patch and used ssize_t as he also suggested. Thanks, Axel! I also noticed that the original patch didn't use <= as it should have, so changed that as well.
Neither my command-line tests or the synth_event_gen_test module are showing any problems at this point for me...
Tom
Original v1 text:
This patchset adds support for dynamic strings for synthetic events, as requested by Axel Rasmussen.
Actually, the first two patches should be applied in any case - the first just changes the current max string length and the second fixes a bug I found while testing.
It works fine for my simple test cases, but I still need to do a lot more testing, especially of the in-kernel API parts, which don't affect Axel's use case.
Anyway, Axel, please try it out and send me your Tested-by: if it works for you, and your broken testcase(s) if it doesn't. ;-)
Thanks,
Tom
The following changes since commit 106c076d23cca67c959a6fd1ccadb5b3ef01ddc9:
mmap_lock: add tracepoints around lock acquisition (2020-09-23 08:48:08 -0500)
are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/zanussi/linux-trace.git ftrace/synth-dynstring-v2
Tom Zanussi (3):
tracing: Change STR_VAR_MAX_LEN
tracing: Fix parse_synth_field() error handling
tracing: Add support for dynamic strings to synthetic events
Documentation/trace/events.rst | 15 +- Documentation/trace/histogram.rst | 18 +++ kernel/trace/synth_event_gen_test.c | 18 ++- kernel/trace/trace_events_hist.c | 9 ++ kernel/trace/trace_events_synth.c | 243 ++++++++++++++++++++++++---- kernel/trace/trace_synth.h | 6 +- 6 files changed, 269 insertions(+), 40 deletions(-)
Signature · 1 line
2.17.1
PATCH v2 1/3 tracing: Change STR_VAR_MAX_LEN
32 is too small for this value, and anyway it makes more sense to use MAX_FILTER_STR_VAL, as this is also the value used for variable-length __strings.
--- kernel/trace/trace_synth.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/trace/trace_synth.h +1 -1
@@ -7,7 +7,7 @@#define SYNTH_SYSTEM "synthetic"#define SYNTH_FIELDS_MAX 32 -#define STR_VAR_LEN_MAX 32/* must be multiple of sizeof(u64) */ +#define STR_VAR_LEN_MAX MAX_FILTER_STR_VAL/* must be multiple of sizeof(u64) */ struct synth_field {char *type;
Signature · 1 line
2.17.1
Re: [PATCH v2 1/3] tracing: Change STR_VAR_MAX_LEN
I still get the same WARN_ON behavior when trying to use bpftrace. But, I spent some time today reading through that call path, and at this point I'm convinced that it's the version of bpftrace I'm using which is slightly broken, not the kernel in this case. (To be fair, I'm trying to use some unreleased tip-of-tree version of it, with some not-yet-merged patches, and some extra hacks on top of all that, so it's not particularly surprising...)
In my experiments with just the synthetic event + histogram triggers, this patchset works as expected for my use case.
So (for the whole series, not just this one patch):
Quoted 26 lines · Tom Zanussi
32 is too small for this value, and anyway it makes more sense to use
MAX_FILTER_STR_VAL, as this is also the value used for variable-length__strings.Signed-off-by Tom Zanussi <[email protected]>--- kernel/trace/trace_synth.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/trace/trace_synth.h +1
-2@@ -7,7 +7,7 @@#define SYNTH_SYSTEM "synthetic"#define SYNTH_FIELDS_MAX 32 -#define STR_VAR_LEN_MAX 32/* must be multiple of sizeof(u64) */ +#define STR_VAR_LEN_MAX MAX_FILTER_STR_VAL/* must be multiple of sizeof(u64) */ struct synth_field {char *type; --2.17.1
Re: [PATCH v2 1/3] tracing: Change STR_VAR_MAX_LEN
Hi Axel,
Quoted 9 lines · Axel Rasmussen
I still get the same
WARN_ONbehavior when trying to use bpftrace. But, I spent some time today reading through that call path, and at this point I'm convinced that it's the version of bpftrace I'm using which is slightly broken, not the kernel in this case. (To be fair, I'm trying to use some unreleased tip-of-tree version of it, with some
not-yet-merged patches, and some extra hacks on top of all that, so it's not particularly surprising...)
Quoted 6 lines
In my experiments with just the synthetic event + histogram triggers, this patchset works as expected for my use case.
So (for the whole series, not just this one patch):
Tested-by Axel Rasmussen <[email protected]>
Thanks!
Tom
Quoted 34 lines
Quoted 31 lines · Tom Zanussi
32 is too small for this value, and anyway it makes more sense to use
MAX_FILTER_STR_VAL, as this is also the value used for variable- length__strings.Signed-off-by Tom Zanussi <[email protected]>--- kernel/trace/trace_synth.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/trace/trace_synth.h +0
-1@@ -7,7 +7,7 @@#define SYNTH_SYSTEM "synthetic"#define SYNTH_FIELDS_MAX 32 -#define STR_VAR_LEN_MAX 32/* must be multiple of sizeof(u64) */
+#defineSTR_VAR_LEN_MAXMAX_FILTER_STR_VAL/* must
be multiple of sizeof(u64) */struct synth_field { char *type;--
2.17.1
PATCH v2 2/3 tracing: Fix parse_synth_field() error handling
In reply to Tom Zanussisynth_field_size() returns either a positive size or an error (zero or a negative value). However, the existing code assumes the only error value is 0. It doesn't handle negative error codes, as it assigns directly to field->size (a size_t; unsigned), thereby interpreting the error code as a valid size instead.
Do the test before assignment to field->size.
[ [email protected]: changelog addition, first paragraph above ]
--- kernel/trace/trace_events_synth.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/trace/trace_events_synth.c +4 -2
@@ -465,6 +465,7 @@ static struct synth_field *parse_synth_field(int argc, const char **argv,struct synth_field *field;const char *prefix =NULL , *field_type = argv[0 ], *field_name, *array;int len, ret =0 ; +ssize_t size;if (field_type[0 ] ==';' ) field_type++;
@@ -520,11 +521,12 @@ static struct synth_field *parse_synth_field(int argc, const char **argv, field->type[len -1 ] ='\0' ; } - field->size =synth_field_size (field->type); -if (!field->size) { + size =synth_field_size (field->type); +if (size <=0 ) { ret = -EINVAL ; goto free; } + field->size = size;if (synth_field_is_string (field->type)) field->is_string = true;
Signature · 1 line
2.17.1
Re: [PATCH v2 2/3] tracing: Fix parse_synth_field() error handling
Hi Tom,
Quoted 7 lines · On Wed, 30 Sep 2020 13:40:51 -0500 Tom Zanussi <[email protected]> wrote:
synth_field_size()returns either a positive size or an error (zero or a negative value). However, the existing code assumes the only error value is 0. It doesn't handle negative error codes, as it assigns directly tofield->size(asize_t; unsigned), thereby interpreting the error code as a valid size instead.Do the test before assignment to
field->size.
This looks good to me. And I guess it is a kind of bugfix which should be backported? (doesn't this need a Fixes tag?)
Thank you,
Quoted 38 lines
[ [email protected]: changelog addition, first paragraph above ]
Signed-off-by Tom Zanussi <[email protected]>--- kernel/trace/trace_events_synth.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)kernel/trace/trace_events_synth.c +4
-2@@ -465,6 +465,7 @@ static struct synth_field *parse_synth_field(int argc, const char **argv,struct synth_field *field;const char *prefix =NULL , *field_type = argv[0 ], *field_name, *array;int len, ret =0 ; +ssize_t size;if (field_type[0 ] ==';' ) field_type++;@@ -520,11 +521,12 @@ static struct synth_field *parse_synth_field(int argc, const char **argv, field->type[len -1 ] ='\0' ; } - field->size =synth_field_size (field->type); -if (!field->size) { + size =synth_field_size (field->type); +if (size <=0 ) { ret = -EINVAL ; goto free; } + field->size = size;if (synth_field_is_string (field->type)) field->is_string = true;Signature · 1 line
2.17.1
Signature · 1 line
Masami Hiramatsu <[email protected]>
Re: [PATCH v2 2/3] tracing: Fix parse_synth_field() error handling
Hi Masami,
Quoted 14 lines · Masami Hiramatsu
Hi Tom,
Quoted 9 lines · On Wed, 30 Sep 2020 13:40:51 -0500 Tom Zanussi <[email protected]> wrote:
synth_field_size()returns either a positive size or an error (zero or
a negative value). However, the existing code assumes the only error
value is 0. It doesn't handle negative error codes, as it assigns directly tofield->size(asize_t; unsigned), thereby interpreting the
error code as a valid size instead.
Quoted 5 lines
Quoted 1 line
Do the test before assignment to
field->size.This looks good to me. And I guess it is a kind of bugfix which should be backported?
(doesn't this need a Fixes tag?)
Yep, thanks for pointing that out.
Quoted 3 lines
Reviewed-by Masami Hiramatsu <[email protected]>
Thanks!
Tom
Quoted 47 lines
Thank you,
Quoted 43 lines
[ [email protected]: changelog addition, first paragraph above ]
Signed-off-by Tom Zanussi <[email protected]>--- kernel/trace/trace_events_synth.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)kernel/trace/trace_events_synth.c +1
-0@@ -465,6 +465,7 @@ static struct synth_field *parse_synth_field(int argc, const char **argv, struct synth_field *field;const char *prefix =NULL , *field_type = argv[0 ], *field_name, *array;int len, ret =0 ; +ssize_t size;kernel/trace/trace_events_synth.c +3
-2if (field_type[0 ] ==';' ) field_type++;@@ -520,11 +521,12 @@ static struct synth_field *parse_synth_field(int argc, const char **argv, field->type[len - 1] = '\0'; } - field->size =synth_field_size (field->type); -if (!field->size) { + size =synth_field_size (field->type); +if (size <=0 ) { ret = -EINVAL ; goto free; } + field->size = size;if (synth_field_is_string (field->type))
field->is_string= true;Signature · 1 line
2.17.1
PATCH v2 3/3 tracing: Add support for dynamic strings to synthetic events
In reply to Tom ZanussiCurrently, sythetic events only support static string fields such as:
# echo 'test_latency u64 lat; char somename[32]' > /sys/kernel/debug/tracing/synthetic_events
Which is fine, but wastes a lot of space in the event.
It also prevents the most commonly-defined strings in the existing trace events e.g. those defined using __string(), from being passed to synthetic events via the trace() action.
With this change, synthetic events with dynamic fields can be defined:
# echo 'test_latency u64 lat; char somename[]' > /sys/kernel/debug/tracing/synthetic_events
And the trace() action can be used to generate events using either dynamic or static strings:
# echo 'hist:keys=name:lat=common_timestamp.usecs-$ts0:onmatch(sys.event).test_latency($lat,name)' > /sys/kernel/debug/tracing/events
The synthetic event dynamic strings are implemented in the same way as the existing __data_loc strings and appear as such in the format file.
--- Documentation/trace/events.rst | 15 +- Documentation/trace/histogram.rst | 18 +++ kernel/trace/synth_event_gen_test.c | 18 ++- kernel/trace/trace_events_hist.c | 9 ++ kernel/trace/trace_events_synth.c | 239 ++++++++++++++++++++++++---- kernel/trace/trace_synth.h | 4 + 6 files changed, 265 insertions(+), 38 deletions(-)
Documentation/trace/events.rst +13 -2
@@ -589,8 +589,19 @@ name:: { .type = "int", .name = "my_int_field" }, }; -See synth_field_size() for available types. If field_name contains [n] -the field is considered to be an array. +See synth_field_size() for available types. + +If field_name contains [n], the field is considered to be a static array. + +If field_names contains[] (no subscript), the field is considered to +be a dynamic array, which will only take as much space in the event as +is required to hold the array. + +Because space for an event is reserved before assigning field values +to the event, using dynamic arrays implies that the piecewise +in-kernel API described below can't be used with dynamic arrays. The +other non-piecewise in-kernel APIs can, however, be used with dynamic +arrays. If the event is created from within a module, a pointer to the module must be passed to synth_event_create(). This will ensure that the
Documentation/trace/histogram.rst +18 -0
@@ -1776,6 +1776,24 @@ consisting of the name of the new event along with one or more variables and their types, which can be any valid field type, separated by semicolons, to the tracing/synthetic_events file. +See synth_field_size() for available types. + +If field_name contains [n], the field is considered to be a static array. + +If field_names contains[] (no subscript), the field is considered to +be a dynamic array, which will only take as much space in the event as +is required to hold the array. + +A string field can be specified using either the static notation: + + char name[32]; + +Or the dynamic: + + char name[]; + +The size limit for either is 256. + For instance, the following creates a new event named 'wakeup_latency' with 3 fields: lat, pid, and prio. Each of those fields is simply a variable reference to a variable on another event::
kernel/trace/synth_event_gen_test.c +12 -6
@@ -242,9 +242,11 @@ static struct synth_field_desc create_synth_test_fields[] = { { .type = "pid_t", .name = "next_pid_field" }, { .type = "char[16]", .name = "next_comm_field" }, { .type = "u64", .name = "ts_ns" }, + { .type = "char[]", .name = "dynstring_field_1" }, { .type = "u64", .name = "ts_ms" }, { .type = "unsigned int", .name = "cpu" }, { .type = "char[64]", .name = "my_string_field" }, + { .type = "char[]", .name = "dynstring_field_2" }, { .type = "int", .name = "my_int_field" }, };
@@ -254,7 +256,7 @@ static struct synth_field_desc create_synth_test_fields[] = {*/ static int __init test_create_synth_event (void ) { -u64 vals[7 ]; +u64 vals[9 ];int ret;/* Create the create_synth_test event with the fields above */
@@ -292,10 +294,12 @@ static int __init test_create_synth_event(void) vals[0 ] =777 ;/* next_pid_field */ vals[1 ] = (u64 )(long )"tiddlywinks" ;/* next_comm_field */ vals[2 ] =1000000 ;/* ts_ns */ - vals[3 ] =1000 ;/* ts_ms */ - vals[4 ] =raw_smp_processor_id ();/* cpu */ - vals[5 ] = (u64 )(long )"thneed" ;/* my_string_field */ - vals[6 ] =398 ;/* my_int_field */ + vals[3 ] = (u64 )(long )"xrayspecs" ;/* dynstring_field_1 */ + vals[4 ] =1000 ;/* ts_ms */ + vals[5 ] =raw_smp_processor_id ();/* cpu */ + vals[6 ] = (u64 )(long )"thneed" ;/* my_string_field */ + vals[7 ] = (u64 )(long )"kerplunk" ;/* dynstring_field_2 */ + vals[8 ] =398 ;/* my_int_field */ /* Now generate a create_synth_test event */ ret =synth_event_trace_array (create_synth_test, vals,ARRAY_SIZE (vals));
@@ -422,13 +426,15 @@ static int __init test_trace_synth_event(void)int ret;/* Trace some bogus values just for testing */ - ret =synth_event_trace (create_synth_test,7 ,/* number of values */ + ret =synth_event_trace (create_synth_test,9 ,/* number of values */ (u64 )444 ,/* next_pid_field */ (u64 )(long )"clackers" ,/* next_comm_field */ (u64 )1000000 ,/* ts_ns */ + (u64 )(long )"viewmaster" ,/* dynstring_field_1 */ (u64 )1000 ,/* ts_ms */ (u64 )raw_smp_processor_id (),/* cpu */ (u64 )(long )"Thneed" ,/* my_string_field */ + (u64 )(long )"yoyos" ,/* dynstring_field_2 */ (u64 )999 );/* my_int_field */ return ret; }
kernel/trace/trace_events_hist.c +9 -0
@@ -3279,6 +3279,15 @@ static int check_synth_field(struct synth_event *event, field = event->fields[field_pos]; +/* +* A dynamic string synth field can accept static or +* dynamic. A static string synth field can only accept a +* same-sized static string, which is checked for later. +*/ +if (strstr (hist_field->type,"char[" ) && field->is_string + && field->is_dynamic) +return 0 ; +if (strcmp (field->type, hist_field->type) !=0 ) {if (field->size != hist_field->size || field->is_signed != hist_field->is_signed)
kernel/trace/trace_events_synth.c +209 -30
413 lines not on this page · raw message
kernel/trace/trace_synth.h +4 -0
@@ -16,6 +16,8 @@ struct synth_field {unsigned int offset;bool is_signed;bool is_string; +bool is_dynamic; +bool field_pos; };struct synth_event {
@@ -24,6 +26,8 @@ struct synth_event {char *name;struct synth_field **fields;unsigned int n_fields; +struct synth_field **dynamic_fields; +unsigned int n_dynamic_fields;unsigned int n_u64;struct trace_event_class class;struct trace_event_call call;
Signature · 1 line
2.17.1
Masami Hiramatsu
maintainerRe: [PATCH v2 3/3] tracing: Add support for dynamic strings to synthetic events
Hi Tom,
Quoted 6 lines · On Wed, 30 Sep 2020 13:40:52 -0500 Tom Zanussi <[email protected]> wrote:
Currently, sythetic events only support static string fields such as:
# echo 'test_latency u64 lat; char somename[32]' > /sys/kernel/debug/tracing/synthetic_eventsWhich is fine, but wastes a lot of space in the event.
Quoted 7 lines
It also prevents the most commonly-defined strings in the existing trace events e.g. those defined using
__string(), from being passed to synthetic events via thetrace()action.With this change, synthetic events with dynamic fields can be defined:
# echo 'test_latency u64 lat; char somename[]' > /sys/kernel/debug/tracing/synthetic_events
Could you add a testcase (and update existing one) of ftracetest for this new feature too?
Quoted 9 lines
And the
trace()action can be used to generate events using either dynamic or static strings:# echo 'hist:keys=name:lat=common_timestamp.usecs-$ts0:onmatch(sys.event).test_latency($lat,name)' > /sys/kernel/debug/tracing/eventsThe synthetic event dynamic strings are implemented in the same way as the existing
__data_locstrings and appear as such in the format file.
Quoted 8 lines
Signed-off-by Tom Zanussi <[email protected]>--- Documentation/trace/events.rst | 15 +- Documentation/trace/histogram.rst | 18 +++ kernel/trace/synth_event_gen_test.c | 18 ++- kernel/trace/trace_events_hist.c | 9 ++ kernel/trace/trace_events_synth.c | 239 ++++++++++++++++++++++++---- kernel/trace/trace_synth.h | 4 +
And you might also need to update tracefs/README so that user can check whether the kernel supports dynamic string or not.
Thank you,
Signature · 1 line
Masami Hiramatsu <[email protected]>
Re: [PATCH v2 3/3] tracing: Add support for dynamic strings to synthetic events
Hi Masami,
Quoted 18 lines · Masami Hiramatsu
Hi Tom,
Quoted 13 lines · On Wed, 30 Sep 2020 13:40:52 -0500 Tom Zanussi <[email protected]> wrote:
Currently, sythetic events only support static string fields such as:
# echo 'test_latency u64 lat; char somename[32]' > /sys/kernel/debug/tracing/synthetic_eventsWhich is fine, but wastes a lot of space in the event.
It also prevents the most commonly-defined strings in the existing trace events e.g. those defined using
__string(), from being passed to
synthetic events via thetrace()action.
Quoted 9 lines
Quoted 5 lines
With this change, synthetic events with dynamic fields can be defined:
# echo 'test_latency u64 lat; char somename[]' > /sys/kernel/debug/tracing/synthetic_eventsCould you add a testcase (and update existing one) of ftracetest for this new feature too?
Yes, I'll add it in the next version.
Quoted 23 lines
Quoted 14 lines
And the
trace()action can be used to generate events using either dynamic or static strings:# echo 'hist:keys=name:lat=common_timestamp.usecs- $ts0:onmatch(sys.event).test_latency($lat,name)' > /sys/kernel/debug/tracing/eventsThe synthetic event dynamic strings are implemented in the same way as
the existing__data_locstrings and appear as such in the format file.Signed-off-by Tom Zanussi <[email protected]>Quoted 8 lines
--- Documentation/trace/events.rst | 15 +- Documentation/trace/histogram.rst | 18 +++ kernel/trace/synth_event_gen_test.c | 18 ++- kernel/trace/trace_events_hist.c | 9 ++ kernel/trace/trace_events_synth.c | 239 ++++++++++++++++++++++++---- kernel/trace/trace_synth.h | 4 +
Quoted 3 lines
And you might also need to update tracefs/README so that user can check whether the kernel supports dynamic string or not.
Yeah, good to add regardless, will do.
Thanks,
Tom