perf record: Add support for limit perf output file size
PATCH v4 1/1 perf record: Add support for limit perf output file size
The patch adds a new option to limit the output file size, then based on it, we can create a wrapper of the perf command that uses the option to avoid exhausting the disk space by the unconscious user.
In order to make the perf.data parsable, we just limit the sample data size, since the perf.data consists of many headers and sample data and other data, the actual size of the recorded file will bigger than the setting value.
Testing it:
# ./perf record -a -g --max-size=10M Couldn't synthesize bpf events. WARNING: The perf data has already reached the limit, stop recording! [ perf record: Woken up 30 times to write data ] [ perf record: Captured and wrote 10.233 MB perf.data (175650 samples) ] Terminated
# ls -lh perf.data -rw------- 1 root root 11M Jul 17 14:01 perf.data
# ./perf record -a -g --max-size=10K WARNING: The perf data has already reached the limit, stop recording! Couldn't synthesize bpf events. [ perf record: Woken up 0 times to write data ] [ perf record: Captured and wrote 1.824 MB perf.data (67 samples) ] Terminated
# ls -lh perf.data -rw------- 1 root root 1.9M Jul 17 14:05 perf.data
---
v4 changes:
- Just show one WARNING message after reached the limit.
v3 changes:
- add a test result
- add the new option to tools/perf/Documentation/perf-record.txt
v2 changes:
- make patch based on latest Arnaldo's perf/core,
- display warning message when reached the limit.
--- tools/perf/Documentation/perf-record.txt | 4 +++ tools/perf/builtin-record.c | 42 ++++++++++++++++++++++++ 2 files changed, 46 insertions(+)
tools/perf/Documentation/perf-record.txt +4 -0
@@ -571,6 +571,10 @@ config terms. For example: 'cycles/overwrite/' and 'instructions/no-overwrite/'. Implies --tail-synthesize. +--max-size=<size>:: +Limit the sample data max size, <size> is expected to be a number with +appended unit character - B/K/M/G + SEE ALSO -------- linkperf:perf-stat[1], linkperf:perf-list[1]
tools/perf/builtin-record.c +42 -0
72 lines not on this page · raw message
Signature · 1 line
2.20.1
Hello Arnaldo & Jirka,
Do you have any other suggestions regarding the patch? Any suggestions are welcome. Thank you very much.
Regards,
Jiwei
Quoted 145 lines · Jiwei Sun
The patch adds a new option to limit the output file size, then based on it, we can create a wrapper of the perf command that uses the option to avoid exhausting the disk space by the unconscious user.
In order to make the perf.data parsable, we just limit the sample data size, since the perf.data consists of many headers and sample data and other data, the actual size of the recorded file will bigger than the setting value.
Testing it:
# ./perf record -a -g --max-size=10M Couldn't synthesize bpf events. WARNING: The perf data has already reached the limit, stop recording! [ perf record: Woken up 30 times to write data ] [ perf record: Captured and wrote 10.233 MB perf.data (175650 samples) ] Terminated# ls -lh perf.data -rw------- 1 root root 11M Jul 17 14:01 perf.data# ./perf record -a -g --max-size=10K WARNING: The perf data has already reached the limit, stop recording! Couldn't synthesize bpf events. [ perf record: Woken up 0 times to write data ] [ perf record: Captured and wrote 1.824 MB perf.data (67 samples) ] Terminated# ls -lh perf.data -rw------- 1 root root 1.9M Jul 17 14:05 perf.dataSigned-off-by Jiwei Sun <[email protected]>---
v4 changes:
- Just show one WARNING message after reached the limit.v3 changes:
- add a test result
- add the new option totools/perf/Documentation/perf-record.txtv2 changes:
- make patch based on latest Arnaldo's perf/core,
- display warning message when reached the limit.--- tools/perf/Documentation/perf-record.txt | 4 +++ tools/perf/builtin-record.c | 42 ++++++++++++++++++++++++ 2 files changed, 46 insertions(+)tools/perf/Documentation/perf-record.txt +4
-0@@ -571,6 +571,10 @@ config terms. For example: 'cycles/overwrite/' and 'instructions/no-overwrite/'. Implies --tail-synthesize. +--max-size=<size>:: +Limit the sample data max size, <size> is expected to be a number with +appended unit character - B/K/M/G + SEE ALSO -------- linkperf:perf-stat[1], linkperf:perf-list[1]tools/perf/builtin-record.c +42
-0@@ -91,6 +91,7 @@ struct record {struct switch_output switch_output;unsigned long long samples;cpu_set_t affinity_mask; +unsigned long output_max_size;/* = 0: unlimited */ };static volatile int auxtrace_record__snapshot_started;@@ -120,6 +121,12 @@ static bool switch_output_time(struct record *rec)trigger_is_ready (&switch_output_trigger); } +static bool record__output_max_size_exceeded (struct record *rec) +{ +return rec->output_max_size && + (rec->bytes_written >= rec->output_max_size); +} +static int record__write (struct record *rec,struct mmap *map__maybe_unused ,void *bf,size_t size) {@@ -132,6 +139,12 @@ static int record__write(struct record *rec, struct mmap *map __maybe_unused, rec->bytes_written += size; +if (record__output_max_size_exceeded (rec)) { +WARN_ONCE (1 ,"WARNING: The perf data has already reached " +"the limit, stop recording!\n" ); +raise (SIGTERM ); + } +if (switch_output_size (rec))trigger_hit (&switch_output_trigger);@@ -1936,6 +1949,33 @@ static int record__parse_affinity(const struct option *opt, const char *str, intreturn 0 ; } +static int parse_output_max_size (const struct option *opt, +const char *str,int unset) +{ +unsigned long *s = (unsigned long *)opt->value; +static struct parse_tag tags_size[] = { + { .tag ='B' , .mult =1 }, + { .tag ='K' , .mult =1 <<10 }, + { .tag ='M' , .mult =1 <<20 }, + { .tag ='G' , .mult =1 <<30 }, + { .tag =0 }, + }; +unsigned long val; + +if (unset) { + *s =0 ; +return 0 ; + } + + val =parse_tag_value (str, tags_size); +if (val != (unsigned long )-1 ) { + *s = val; +return 0 ; + } + +return -1 ; +} +static int record__parse_mmap_pages (const struct option *opt,const char *str,int unset__maybe_unused )@@ -2262,6 +2302,8 @@ static struct option __record_options[] = { "n", "Compressed records using specified level (default: 1 - fastest compression, 22 - greatest compression)", record__parse_comp_level),#endif +OPT_CALLBACK (0 ,"max-size" , &record.output_max_size, +"size" ,"Limit the maximum size of the output file" , parse_output_max_size), OPT_END() };
Quoted 4 lines · Jiwei Sun
Hello Arnaldo & Jirka,
Do you have any other suggestions regarding the patch? Any suggestions are welcome. Thank you very much.
oops, overlooked this one, sry.. I'll check on it today
jirka
Quoted 150 lines
Regards,
JiweiQuoted 145 lines · Jiwei Sun
The patch adds a new option to limit the output file size, then based on it, we can create a wrapper of the perf command that uses the option to avoid exhausting the disk space by the unconscious user.
In order to make the perf.data parsable, we just limit the sample data size, since the perf.data consists of many headers and sample data and other data, the actual size of the recorded file will bigger than the setting value.
Testing it:
# ./perf record -a -g --max-size=10M Couldn't synthesize bpf events. WARNING: The perf data has already reached the limit, stop recording! [ perf record: Woken up 30 times to write data ] [ perf record: Captured and wrote 10.233 MB perf.data (175650 samples) ] Terminated# ls -lh perf.data -rw------- 1 root root 11M Jul 17 14:01 perf.data# ./perf record -a -g --max-size=10K WARNING: The perf data has already reached the limit, stop recording! Couldn't synthesize bpf events. [ perf record: Woken up 0 times to write data ] [ perf record: Captured and wrote 1.824 MB perf.data (67 samples) ] Terminated# ls -lh perf.data -rw------- 1 root root 1.9M Jul 17 14:05 perf.dataSigned-off-by Jiwei Sun <[email protected]>---
v4 changes:
- Just show one WARNING message after reached the limit.v3 changes:
- add a test result
- add the new option totools/perf/Documentation/perf-record.txtv2 changes:
- make patch based on latest Arnaldo's perf/core,
- display warning message when reached the limit.--- tools/perf/Documentation/perf-record.txt | 4 +++ tools/perf/builtin-record.c | 42 ++++++++++++++++++++++++ 2 files changed, 46 insertions(+)tools/perf/Documentation/perf-record.txt +4
-0@@ -571,6 +571,10 @@ config terms. For example: 'cycles/overwrite/' and 'instructions/no-overwrite/'. Implies --tail-synthesize. +--max-size=<size>:: +Limit the sample data max size, <size> is expected to be a number with +appended unit character - B/K/M/G + SEE ALSO -------- linkperf:perf-stat[1], linkperf:perf-list[1]tools/perf/builtin-record.c +42
-0@@ -91,6 +91,7 @@ struct record {struct switch_output switch_output;unsigned long long samples;cpu_set_t affinity_mask; +unsigned long output_max_size;/* = 0: unlimited */ };static volatile int auxtrace_record__snapshot_started;@@ -120,6 +121,12 @@ static bool switch_output_time(struct record *rec)trigger_is_ready (&switch_output_trigger); } +static bool record__output_max_size_exceeded (struct record *rec) +{ +return rec->output_max_size && + (rec->bytes_written >= rec->output_max_size); +} +static int record__write (struct record *rec,struct mmap *map__maybe_unused ,void *bf,size_t size) {@@ -132,6 +139,12 @@ static int record__write(struct record *rec, struct mmap *map __maybe_unused, rec->bytes_written += size; +if (record__output_max_size_exceeded (rec)) { +WARN_ONCE (1 ,"WARNING: The perf data has already reached " +"the limit, stop recording!\n" ); +raise (SIGTERM ); + } +if (switch_output_size (rec))trigger_hit (&switch_output_trigger);@@ -1936,6 +1949,33 @@ static int record__parse_affinity(const struct option *opt, const char *str, intreturn 0 ; } +static int parse_output_max_size (const struct option *opt, +const char *str,int unset) +{ +unsigned long *s = (unsigned long *)opt->value; +static struct parse_tag tags_size[] = { + { .tag ='B' , .mult =1 }, + { .tag ='K' , .mult =1 <<10 }, + { .tag ='M' , .mult =1 <<20 }, + { .tag ='G' , .mult =1 <<30 }, + { .tag =0 }, + }; +unsigned long val; + +if (unset) { + *s =0 ; +return 0 ; + } + + val =parse_tag_value (str, tags_size); +if (val != (unsigned long )-1 ) { + *s = val; +return 0 ; + } + +return -1 ; +} +static int record__parse_mmap_pages (const struct option *opt,const char *str,int unset__maybe_unused )@@ -2262,6 +2302,8 @@ static struct option __record_options[] = { "n", "Compressed records using specified level (default: 1 - fastest compression, 22 - greatest compression)", record__parse_comp_level),#endif +OPT_CALLBACK (0 ,"max-size" , &record.output_max_size, +"size" ,"Limit the maximum size of the output file" , parse_output_max_size), OPT_END() };
On Wed, Sep 25, 2019 at 03:06:37PM +0800, Jiwei Sun wrote:
SNIP
Quoted 29 lines
SEE ALSO
--------
linkperf:perf-stat[1], linkperf:perf-list[1]tools/perf/builtin-record.c +7
-0@@ -91,6 +91,7 @@ struct record {struct switch_output switch_output;unsigned long long samples;cpu_set_t affinity_mask; +unsigned long output_max_size;/* = 0: unlimited */ };static volatile int auxtrace_record__snapshot_started;@@ -120,6 +121,12 @@ static bool switch_output_time(struct record *rec)trigger_is_ready (&switch_output_trigger); } +static bool record__output_max_size_exceeded (struct record *rec) +{ +return rec->output_max_size && + (rec->bytes_written >= rec->output_max_size); +} +static int record__write (struct record *rec,struct mmap *map__maybe_unused ,void *bf,size_t size) {
tools/perf/builtin-record.c static int record__write(struct record *rec, struct mmap *map __maybe_unused, · line 144
Quoted 6 lines
tools/perf/builtin-record.c +3
-0@@ -132,6 +139,12 @@ static int record__write(struct record *rec, struct mmap *map __maybe_unused, rec->bytes_written += size; +if (record__output_max_size_exceeded (rec)) { +WARN_ONCE (1 ,"WARNING: The perf data has already reached " +"the limit, stop recording!\n" );
I think the message whouldn't be a warning, the user asked for that, maybe something more like:
[ perf record: perf size limit reached (XXMB), stopping session ]
tools/perf/builtin-record.c static int record__write(struct record *rec, struct mmap *map __maybe_unused, · line 145
Quoted 1 line
tools/perf/builtin-record.c +1
-0+raise (SIGTERM );
could we just set 'done = 1' what's the benefit in killing perf?
thanks,
jirka
Hi Jirka,
Quoted 43 lines · Jiri Olsa
On Wed, Sep 25, 2019 at 03:06:37PM +0800, Jiwei Sun wrote:
SNIP
Quoted 29 lines
SEE ALSO
--------
linkperf:perf-stat[1], linkperf:perf-list[1]tools/perf/builtin-record.c +7
-0@@ -91,6 +91,7 @@ struct record {struct switch_output switch_output;unsigned long long samples;cpu_set_t affinity_mask; +unsigned long output_max_size;/* = 0: unlimited */ };static volatile int auxtrace_record__snapshot_started;@@ -120,6 +121,12 @@ static bool switch_output_time(struct record *rec)trigger_is_ready (&switch_output_trigger); } +static bool record__output_max_size_exceeded (struct record *rec) +{ +return rec->output_max_size && + (rec->bytes_written >= rec->output_max_size); +} +static int record__write (struct record *rec,struct mmap *map__maybe_unused ,void *bf,size_t size) {tools/perf/builtin-record.c static int record__write(struct record *rec, struct mmap *map __maybe_unused, · line 144
Quoted 6 lines
tools/perf/builtin-record.c +3
-0@@ -132,6 +139,12 @@ static int record__write(struct record *rec, struct mmap *map __maybe_unused, rec->bytes_written += size; +if (record__output_max_size_exceeded (rec)) { +WARN_ONCE (1 ,"WARNING: The perf data has already reached " +"the limit, stop recording!\n" );I think the message whouldn't be a warning, the user asked for that, maybe something more like:
Quoted 5 lines
[ perf record: perf size limit reached (XXMB), stopping session ]
tools/perf/builtin-record.c static int record__write(struct record *rec, struct mmap *map __maybe_unused, · line 145
Quoted 1 line
tools/perf/builtin-record.c +1
-0+raise (SIGTERM );could we just set 'done = 1' what's the benefit in killing perf?
Thanks for your suggestions. Yes, if just set "done == 1" is more efficient and more concise. And I will modify it and the output format, and then send a v5 patch.
Thanks again.
Regards,
Jiwei
Quoted 5 lines
thanks,
jirka