perf record: Delete file if a failure occurs writing the perf data file
PATCH 1/1 perf record: Delete file if a failure occurs writing the perf data file
If perf fails to write data to the data file (e.g., ENOSPC error) it fails with the message:
failed to write perf data, error: No space left on device
and stops — killing the workload too. The file is an unknown state. Trying to read it (e.g., perf report) fails with a SIGBUS error.
Fix by deleting the file on a failure.
--- tools/perf/builtin-record.c | 1 + tools/perf/util/data.c | 5 +++++ tools/perf/util/data.h | 1 + 3 files changed, 7 insertions(+)
tools/perf/builtin-record.c +1 -0
@@ -562,6 +562,7 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv) out_delete_session:perf_session__delete (session); +perf_data_file__unlink (session->file);return err; }
tools/perf/util/data.c +5 -0
@@ -118,3 +118,8 @@ void perf_data_file__close(struct perf_data_file *file) {close (file->fd); } + +void perf_data_file__unlink (struct perf_data_file *file) +{ +unlink (file->path); +}
tools/perf/util/data.h +1 -0
@@ -44,5 +44,6 @@ static inline unsigned long perf_data_file__size(struct perf_data_file *file)int perf_data_file__open (struct perf_data_file *file);void perf_data_file__close (struct perf_data_file *file); +void perf_data_file__unlink (struct perf_data_file *file);#endif /* __PERF_DATA_H */
Signature · 1 line
1.8.3.4 (Apple Git-47)
Quoted 8 lines · David Ahern
If perf fails to write data to the data file (e.g., ENOSPC error) it fails with the message:
failed to write perf data, error: No space left on deviceand stops — killing the workload too. The file is an unknown state. Trying to read it (e.g., perf report) fails with a SIGBUS error.
Fix by deleting the file on a failure.
I think that's right thing to do
jirka
Quoted 6 lines · David Ahern
If perf fails to write data to the data file (e.g., ENOSPC error) it fails with the message:
failed to write perf data, error: No space left on deviceand stops — killing the workload too. The file is an unknown state. Trying to read it (e.g., perf report) fails with a SIGBUS error.
Ouch - guys please first investiage that SIGBUS, we should not behave unexpectedly on _any_ (read: random) perf.data file contents. The SIGBUS likely suggests that the parsing isn't robust enough.
Quoted 1 line
Fix by deleting the file on a failure.
That only works around the issue - if the same data file is produced by some other method (or maliciously) then perf report will still SIGBUS ...
Thanks,
Ingo
Quoted 7 lines · Ingo Molnar
Quoted 4 lines · David Ahern
If perf fails to write data to the data file (e.g., ENOSPC error) it fails with the message:
failed to write perf data, error: No space left on device
Quoted 6 lines
Quoted 2 lines
and stops — killing the workload too. The file is an unknown state. Trying to read it (e.g., perf report) fails with a SIGBUS error.
Ouch - guys please first investiage that SIGBUS, we should not behave unexpectedly on _any_ (read: random) perf.data file contents. The SIGBUS likely suggests that the parsing isn't robust enough.
I think we know why the SIGBUS is happening. From 'man mmap':
From man mmap:
SIGBUS Attempted access to a portion of the buffer that does not correspond to the file (for example, beyond the end of the file, ...
With regards to perf-record, on a write() failure the header is not updated. From a recent change we try to proceed even though the data size is 0 - parsing the events we can. We finally hit upon an event that is only partially in the file (eg., header, but no data for event). Trying to read the event data leads to the SIGBUS:
Running perf-report in gdb:
WARNING: The /tmp/mnt/perf.data file's data size field is 0 which is unexpected. Was the 'perf record' command properly terminated?
Program received signal SIGBUS, Bus error.perf_evsel__parse_sample (evsel=0x94eec0, event=0x7ffff7ed9d80, data=0x7fffffffd260)
at util/evsel.c:1242
1242 u16 max_size = event->header.size;
(gdb) bt
#0 perf_evsel__parse_sample (evsel=0x94eec0, event=0x7ffff7ed9d80, data=0x7fffffffd260)
at util/evsel.c:1242
#1 0x000000000047c9ce in flush_sample_queue (s=0x94e2b0, tool=0x7fffffffde80)
at util/session.c:542
#2 0x000000000047e2d4 in __perf_session__process_events (session=0x94e2b0,
data_offset=<optimized out>, data_size=<optimized out>,
file_size=1048576, tool=0x7fffffffde80)
at util/session.c:1388
#3 0x000000000042993c in __cmd_report (rep=0x7fffffffde80) at builtin-report.c:509
#4 cmd_report (argc=0, argv=0x7fffffffe370, prefix=<optimized out>) at builtin-report.c:967
#5 0x000000000041b063 in run_builtin (p=0x7cdf28, argc=4, argv=0x7fffffffe370) at perf.c:319
#6 0x000000000041a8e3 in handle_internal_command (argv=0x7fffffffe370, argc=4) at perf.c:376
#7 run_argv (argv=0x7fffffffe180, argcp=0x7fffffffe18c) at perf.c:420
#8 main (argc=4, argv=0x7fffffffe370) at perf.c:521Quoted 5 lines
Quoted 2 lines
Fix by deleting the file on a failure.
That only works around the issue - if the same data file is produced by some other method (or maliciously) then perf report will still SIGBUS ...
We could handle SIGBUS in the analysis commands too. See the suggestion I had for handling the output failure using the mmap output option which uses lngjmp.
David
Ingo:
Quoted 9 lines · David Ahern
Quoted 8 lines · Ingo Molnar
Quoted 5 lines · David Ahern
If perf fails to write data to the data file (e.g., ENOSPC error) it fails
with the message:
failed to write perf data, error: No space left on device
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
and stops — killing the workload too. The file is an unknown state. Trying to read it (e.g., perf report) fails with a SIGBUS error.
Ouch - guys please first investiage that SIGBUS, we should not behave unexpectedly on _any_ (read: random) perf.data file contents. The SIGBUS likely suggests that the parsing isn't robust enough.
If you agree with the below summary then any further objections to deleting the file on write failure?
David
Quoted 61 lines
I think we know why the SIGBUS is happening. From 'man mmap':
From man mmap:
SIGBUS Attempted access to a portion of the buffer that does not correspond to the file (for example, beyond the end of the file, ...With regards to perf-record, on a
write()failure the header is not updated. From a recent change we try to proceed even though the data size is 0 - parsing the events we can. We finally hit upon an event that is only partially in the file (eg., header, but no data for event). Trying to read the event data leads to the SIGBUS:Running perf-report in gdb:
WARNING: The
/tmp/mnt/perf.datafile's data size field is 0 which is unexpected.
Was the 'perf record' command properly terminated?Program received signal SIGBUS, Bus error.
perf_evsel__parse_sample(evsel=0x94eec0, event=0x7ffff7ed9d80, data=0x7fffffffd260)at util/evsel.c:12421242 u16
max_size=event->header.size;
(gdb) bt
#0perf_evsel__parse_sample(evsel=0x94eec0, event=0x7ffff7ed9d80, data=0x7fffffffd260)at util/evsel.c:1242 #1 0x000000000047c9ce in flush_sample_queue (s=0x94e2b0, tool=0x7fffffffde80) at util/session.c:542 #2 0x000000000047e2d4 in __perf_session__process_events (session=0x94e2b0, data_offset=<optimized out>, data_size=<optimized out>, file_size=1048576, tool=0x7fffffffde80) at util/session.c:1388#3
0x000000000042993cin__cmd_report(rep=0x7fffffffde80) at builtin-report.c:509
#4cmd_report(argc=0, argv=0x7fffffffe370, prefix=<optimized out>) at builtin-report.c:967
#50x000000000041b063inrun_builtin(p=0x7cdf28, argc=4, argv=0x7fffffffe370) at perf.c:319
#60x000000000041a8e3inhandle_internal_command(argv=0x7fffffffe370, argc=4) at perf.c:376
#7run_argv(argv=0x7fffffffe180, argcp=0x7fffffffe18c) at perf.c:420 #8 main (argc=4, argv=0x7fffffffe370) at perf.c:521Quoted 5 lines
Quoted 2 lines
Fix by deleting the file on a failure.
That only works around the issue - if the same data file is produced by some other method (or maliciously) then perf report will still SIGBUS ...
We could handle SIGBUS in the analysis commands too. See the suggestion I had for handling the output failure using the mmap output option which uses lngjmp.
David
Peter Zijlstra
maintainerQuoted 4 lines · David Ahern
Quoted 4 lines
From man mmap:
SIGBUS Attempted access to a portion of the buffer that does not correspond to the file (for example, beyond the end of the file, ...
SIGBUS is basically the std fail for any fault; there's a ton more reasons than listed in that manpage.
Failing to dirty a page due to -ENOSPACE is one reason we'll trigger SIGBUS -- as you already found in that memcpy to mmap() instead of write() patch.
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · David Ahern
Quoted 4 lines
From man mmap:
SIGBUS Attempted access to a portion of the buffer that does not correspond to the file (for example, beyond the end of the file, ...
Quoted 6 lines
SIGBUS is basically the std fail for any fault; there's a ton more reasons than listed in that manpage.
Failing to dirty a page due to
-ENOSPACEis one reason we'll trigger SIGBUS -- as you already found in that memcpy tommap()instead ofwrite()patch.
Sure. Lots of mmaps flying around, so to make sure we are on the same page in this case we are talking about perf-report reading a file via mmap.
If the file creation (be it memcpy to mmap() or write()) has a failure due to lack of space, then only a partial event is in the file -- e.g., perf-header only. Trying to read the rest of the event leads to perf-report terminating due to SIGBUS. How should that condition be handled?
The patch in this thread deletes the file. Another option is to rewind the file to the last known good write (ie., length after last successful call to write_output).
David
Peter Zijlstra
maintainerQuoted 17 lines · David Ahern
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · David Ahern
Quoted 4 lines
From man mmap:
SIGBUS Attempted access to a portion of the buffer that does not correspond to the file (for example, beyond the end of the file, ...Quoted 6 lines
SIGBUS is basically the std fail for any fault; there's a ton more reasons than listed in that manpage.
Failing to dirty a page due to
-ENOSPACEis one reason we'll trigger SIGBUS -- as you already found in that memcpy tommap()instead ofwrite()patch.Sure. Lots of mmaps flying around, so to make sure we are on the same page in this case we are talking about perf-report reading a file via mmap.
Quoted 8 lines
If the file creation (be it memcpy to
mmap()orwrite()) has a failure due to lack of space, then only a partial event is in the file -- e.g., perf-header only. Trying to read the rest of the event leads to perf-report terminating due to SIGBUS. How should that condition be handled?The patch in this thread deletes the file. Another option is to rewind the file to the last known good write (ie., length after last successful call to
write_output).
I'd report a warning and continue with all events that you could read upto that point.
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · David Ahern
The patch in this thread deletes the file. Another option is to rewind the file to the last known good write (ie., length after last successful call to
write_output).I'd report a warning and continue with all events that you could read upto that point.
Coming back to this: the answer to that statement is known. The attached patch attempt to rewind to the last good write. Subsequent reads to the file fail with:
WARNING: The /tmp/mnt/perf.data file's data size field is 0 which is unexpected. Was the 'perf record' command properly terminated?
reading input file (size expected=3 received=-1)broken or missing trace data incompatible file format (rerun with -v to learn more)
Given that it would seem deleting the file is the best option.
David
tools/perf/builtin-record.c +16 -1
@@ -99,7 +99,22 @@ static int do_write_output(struct perf_record *rec, void *buf, size_t size)static int write_output (struct perf_record *rec,void *buf,size_t size) { -return do_write_output (rec, buf, size); +off_t len; +int rc; + +/* save current length */ + len = rec->session->header.data_offset + rec->bytes_written; + + rc =do_write_output (rec, buf, size); + +/* on failure reset file to last known good length */ +if (rc <0 ) { +pr_debug ("truncating file to last known good write: len %ld\n" , len); +if (ftruncate (rec->file.fd, len) !=0 ) +pr_err ("Double failure -- write failed and then ftruncate\n" ); + } + +return rc; } static int process_synthesized_event(struct perf_tool *tool,