perf: detect loops processing events
PATCH 1/1 perf: detect loops processing events
Recovery algorithm in __perf_session__process_events attempts to remap a perf.data file with a different file_offset and try again at a new head position. Both of these adjustment rely on page_offset. If page_offset is 0 then file_offset and head never change which means the remap attempt is the same and the fetch_mmaped_event is the same and the processing just loops forever.
Detect this condition and warn the user.
--- tools/perf/util/session.c | 6 ++++++ 1 file changed, 6 insertions(+)
tools/perf/util/session.c +6 -0
@@ -1235,6 +1235,12 @@ more: } page_offset = page_size * (head / page_size); +/* catch looping where we never make forward progress. */ +if (page_offset ==0 ) { +pr_err ("Loop detection processing events. Is file corrupted?\n" ); +return -1 ; + } + file_offset += page_offset; head -= page_offset; goto remap;
Signature · 1 line
1.7.10.1
Quoted 28 lines · David Ahern
Recovery algorithm in
__perf_session__process_eventsattempts to remap a perf.data file with a differentfile_offsetand try again at a new head position. Both of these adjustment rely onpage_offset. Ifpage_offsetis 0 thenfile_offsetand head never change which means the remap attempt is the same and thefetch_mmaped_eventis the same and the processing just loops forever.Detect this condition and warn the user.
Signed-off-by David Ahern <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Cc Ingo Molnar <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Jiri Olsa <[email protected]>Cc Namhyung Kim <[email protected]>Cc Stephane Eranian <[email protected]>--- tools/perf/util/session.c | 6 ++++++ 1 file changed, 6 insertions(+)tools/perf/util/session.c +0
-0@@ -1235,6 +1235,12 @@ more: }
tools/perf/util/session.c more: · line 1246
Quoted 10 lines
tools/perf/util/session.c +6
-0page_offset = page_size * (head / page_size); +/* catch looping where we never make forward progress. */ +if (page_offset ==0 ) { +pr_err ("Loop detection processing events. Is file corrupted?\n" ); +return -1 ; + } + file_offset += page_offset; head -= page_offset; goto remap;
Ah, nice!
Btw., would it make sense to emit a (once-only) warning and optimistically fix page_offset up to 1 (or 4096) and let things continue with the next set of data - can we recover most of the data in that case?
Thanks,
Ingo
Quoted 44 lines · Ingo Molnar
Quoted 28 lines · David Ahern
Recovery algorithm in
__perf_session__process_eventsattempts to remap a perf.data file with a differentfile_offsetand try again at a new head position. Both of these adjustment rely onpage_offset. Ifpage_offsetis 0 thenfile_offsetand head never change which means the remap attempt is the same and thefetch_mmaped_eventis the same and the processing just loops forever.Detect this condition and warn the user.
Signed-off-by David Ahern <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Cc Ingo Molnar <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Jiri Olsa <[email protected]>Cc Namhyung Kim <[email protected]>Cc Stephane Eranian <[email protected]>--- tools/perf/util/session.c | 6 ++++++ 1 file changed, 6 insertions(+)tools/perf/util/session.c +0
-0@@ -1235,6 +1235,12 @@ more: }tools/perf/util/session.c more: · line 1246
Quoted 10 lines
tools/perf/util/session.c +6
-0page_offset = page_size * (head / page_size); +/* catch looping where we never make forward progress. */ +if (page_offset ==0 ) { +pr_err ("Loop detection processing events. Is file corrupted?\n" ); +return -1 ; + } + file_offset += page_offset; head -= page_offset; goto remap;Ah, nice!
Quoted 3 lines
Btw., would it make sense to emit a (once-only) warning and optimistically fix
page_offsetup to 1 (or 4096) and let things continue with the next set of data - can we recover most of the data in that case?
Basically, what'd like to do with binary data is similar to what we do with text data if some trace output is corrupted:
sshd-15478 [002] 1100.859353: 15478:120:S ==> [002] 0:140:R <idle>
<idle>-0 [005] 1100.859378: 0:140:R ==> [005] 20169:120:R cat
cat-20169 [005] 1100.860718: 20169:120:R + [005] 15521:120:S bash
^@¨<81><92>^@^Bª^P^P^D<9c>8@<88>^A^M ;¤0 ^D<90>"ª<81>^B^T)Ò^C$^@^N^@^A
cat-20169 [005] 1100.860720: 20169:120:R + [005] 305:115:S kblockd/5
cat-20169 [005] 1100.860722: 20169:120:? ==> [005] 305:115:R kblockd/5
kblockd/5-305 [005] 1100.860755: 305:115:S ==> [005] 15521:120:R bash
bash-15521 [005] 1100.860792: 15521:120:S + [002] 15478:120:S sshd
<idle>-0 [002] 1100.860853: 0:140:R ==> [002] 15478:120:R sshd
sshd-15478 [002] 1100.860895: 15478:120:S ==> [002] 0:140:R <idle>
bash-15521 [005] 1100.860925: 15521:120:S + [002] 15478:120:S sshd
bash-15521 [005] 1100.860999: 15521:120:S ==> [005] 0:140:R <idle>See that junk in the middle, sign of some sort of file corruption? Instead of detecting it and aborting we just try to skip that line and try to find the next useful looking line, ignoring the junk and bits around it.
Is there a perf.data equivalent of intelligently trying to skip to the next plausible looking event record?
Thanks,
Ingo
Namhyung Kim
maintainerHi Ingo,
Quoted 21 lines · Ingo Molnar
Quoted 3 lines · Ingo Molnar
Btw., would it make sense to emit a (once-only) warning and optimistically fix
page_offsetup to 1 (or 4096) and let things continue with the next set of data - can we recover most of the data in that case?Basically, what'd like to do with binary data is similar to what we do with text data if some trace output is corrupted:
sshd-15478 [002] 1100.859353: 15478:120:S ==> [002] 0:140:R <idle> <idle>-0 [005] 1100.859378: 0:140:R ==> [005] 20169:120:R cat cat-20169 [005] 1100.860718: 20169:120:R + [005] 15521:120:S bash ^@¨<81><92>^@^Bª^P^P^D<9c>8@<88>^A^M ;¤0 ^D<90>"ª<81>^B^T)Ò^C$^@^N^@^A cat-20169 [005] 1100.860720: 20169:120:R + [005] 305:115:S kblockd/5 cat-20169 [005] 1100.860722: 20169:120:? ==> [005] 305:115:R kblockd/5 kblockd/5-305 [005] 1100.860755: 305:115:S ==> [005] 15521:120:R bash bash-15521 [005] 1100.860792: 15521:120:S + [002] 15478:120:S sshd <idle>-0 [002] 1100.860853: 0:140:R ==> [002] 15478:120:R sshd sshd-15478 [002] 1100.860895: 15478:120:S ==> [002] 0:140:R <idle> bash-15521 [005] 1100.860925: 15521:120:S + [002] 15478:120:S sshd bash-15521 [005] 1100.860999: 15521:120:S ==> [005] 0:140:R <idle>
Quoted 6 lines
See that junk in the middle, sign of some sort of file corruption? Instead of detecting it and aborting we just try to skip that line and try to find the next useful looking line, ignoring the junk and bits around it.
Is there a perf.data equivalent of intelligently trying to skip to the next plausible looking event record?
I think we should not truncate file_size for this case. It was decreased to data_offset + data_size in order not to read unrelated metadata (additional header feature info). But in this case, since data_size is 0 it'd have same value as data_offset, and in turn mmap_size truncated to data_offset too. So fetch_mmaped_event() always return NULL as head + sizeof(event->header) exceeds mmap_size.
If we keep original file_size, perf can report existing samples but no metadata. So does the patch below make sense?
tools/perf/util/session.c +1 -1
@@ -1196,7 +1196,7 @@ int __perf_session__process_events(struct perf_session *session, file_offset = page_offset; head = data_offset - page_offset; -if (data_offset + data_size < file_size) +if (data_size && (data_offset + data_size < file_size)) file_size = data_offset + data_size; progress_next = file_size /16 ;
Signature · 2 lines
Thanks, Namhyung
Quoted 17 lines · Namhyung Kim
I think we should not truncate
file_sizefor this case. It was decreased todata_offset+data_sizein order not to read unrelated metadata (additional header feature info). But in this case, sincedata_sizeis 0 it'd have same value asdata_offset, and in turnmmap_sizetruncated todata_offsettoo. Sofetch_mmaped_event()always return NULL as head +sizeof(event->header)exceedsmmap_size.If we keep original
file_size, perf can report existing samples but no metadata. So does the patch below make sense?tools/perf/util/session.c +0
-0
tools/perf/util/session.c int __perf_session__process_events(struct perf_session *session, · line 1202
Quoted 8 lines
tools/perf/util/session.c +1
-1@@ -1196,7 +1196,7 @@ int __perf_session__process_events(struct perf_session *session, file_offset = page_offset; head = data_offset - page_offset; -if (data_offset + data_size < file_size) +if (data_size && (data_offset + data_size < file_size)) file_size = data_offset + data_size; progress_next = file_size /16 ;
Nice. That does handle the case of the perf.data file not getting closed properly. With this, my patch should not return -1 just print the error message to the user which would explain why the feature data is not printed.
David