Skip to contentSkip to conversation

perf session: Fix compression processing

RFC In review Jiri Olsa 2 participants 8 messages Nov 3, 2019 – Nov 11, 2019 perf events

lore
Share & tools
lore

Jiri Olsa

author
#0 lore raw

RFC 1/1 perf session: Fix compression processing

hi,
I'm not sure I follow everything on compression,
so I might have missed something, but patch below
fixes the issue for me.

jirka

---
The compressed data processing occasionally fails with:

  $ perf report --stdio -vv
  decomp (B): 44519 to 163000
  decomp (B): 48119 to 174800
  decomp (B): 65527 to 131072
  fetch_mmaped_event: head=0x1ffe0 event->header_size=0x28, mmap_size=0x20000: fuzzed perf.data?
  Error:
  failed to process sample
  ...

It's caused by recent fuzzer fix that does not take into account that compressed data do not need to by fully present in the buffer, so it's ok to just return NULL and not to fail.

Fixes 57fc032ad643 ("perf session: Avoid infinite loop when seeing invalid header.size")
---
 tools/perf/util/session.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
tools/perf/util/session.c +5 -3
index f07b8ecb91bc..3589ed14a629 100644
@@ -1959,7 +1959,7 @@ static int __perf_session__process_pipe_events(struct perf_session *session)
 
 static union perf_event *
 fetch_mmaped_event(struct perf_session *session,
-		   u64 head, size_t mmap_size, char *buf)
+		   u64 head, size_t mmap_size, char *buf, bool decomp)
 {
 	union perf_event *event;
 
@@ -1979,6 +1979,8 @@ fetch_mmaped_event(struct perf_session *session,
 		/* We're not fetching the event so swap back again */
 		if (session->header.needs_swap)
 			perf_event_header__bswap(&event->header);
+		if (decomp)
+			return NULL;
 		pr_debug("%s: head=%#" PRIx64 " event->header_size=%#x, mmap_size=%#zx: fuzzed perf.data?\n",
 			 __func__, head, event->header.size, mmap_size);
 		return ERR_PTR(-EINVAL);
@@ -1997,7 +1999,7 @@ static int __perf_session__process_decomp_events(struct perf_session *session)
 		return 0;
 
 	while (decomp->head < decomp->size && !session_done()) {
-		union perf_event *event = fetch_mmaped_event(session, decomp->head, decomp->size, decomp->data);
+		union perf_event *event = fetch_mmaped_event(session, decomp->head, decomp->size, decomp->data, true);
 
 		if (IS_ERR(event))
 			return PTR_ERR(event);
@@ -2100,7 +2102,7 @@ reader__process_events(struct reader *rd, struct perf_session *session,
 	}
 
 more:
-	event = fetch_mmaped_event(session, head, mmap_size, buf);
+	event = fetch_mmaped_event(session, head, mmap_size, buf, false);
 	if (IS_ERR(event))
 		return PTR_ERR(event);
 
Signature · 1 line
2.21.0
#1 lore raw
Quoted 23 lines · Jiri Olsa

hi,
I'm not sure I follow everything on compression,
so I might have missed something, but patch below
fixes the issue for me.

jirka

---
The compressed data processing occasionally fails with:

  $ perf report --stdio -vv
  decomp (B): 44519 to 163000
  decomp (B): 48119 to 174800
  decomp (B): 65527 to 131072
  fetch_mmaped_event: head=0x1ffe0 event->header_size=0x28, mmap_size=0x20000: fuzzed perf.data?
  Error:
  failed to process sample
  ...

It's caused by recent fuzzer fix that does not take into account that compressed data do not need to by fully present in the buffer, so it's ok to just return NULL and not to fail.

Quoted 6 lines
Fixes 57fc032ad643 ("perf session: Avoid infinite loop when seeing invalid header.size")
---
 tools/perf/util/session.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

I am on vacations currently, getting back on Monday (11/11). Please expect delay in response.

~Alexey

#2 lore raw
In reply to Jiri Olsa
Quoted 1 line · Jiri Olsa

hi,

<SNIP>

Quoted 27 lines

---
The compressed data processing occasionally fails with:

  $ perf report --stdio -vv
  decomp (B): 44519 to 163000
  decomp (B): 48119 to 174800
  decomp (B): 65527 to 131072
  fetch_mmaped_event: head=0x1ffe0 event->header_size=0x28, mmap_size=0x20000: fuzzed perf.data?
  Error:
  failed to process sample
  ...

It's caused by recent fuzzer fix that does not take into account that compressed data do not need to by fully present in the buffer, so it's ok to just return NULL and not to fail.

Fixes 57fc032ad643 ("perf session: Avoid infinite loop when seeing invalid header.size")
---
 tools/perf/util/session.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
tools/perf/util/session.c +0 -0
index f07b8ecb91bc..3589ed14a629 100644

tools/perf/util/session.c static int __perf_session__process_pipe_events(struct perf_session *session) · line 1962

Quoted 5 lines
tools/perf/util/session.c +1 -1
@@ -1959,7 +1959,7 @@ static int __perf_session__process_pipe_events(struct perf_session *session)
 
 static union perf_event *
 fetch_mmaped_event(struct perf_session *session,
-		   u64 head, size_t mmap_size, char *buf)
+		   u64 head, size_t mmap_size, char *buf, bool decomp)

bools in interface make code less transparent.

Quoted 13 lines
tools/perf/util/session.c +2 -0
 {
 	union perf_event *event;
 
@@ -1979,6 +1979,8 @@ fetch_mmaped_event(struct perf_session *session,
 		/* We're not fetching the event so swap back again */
 		if (session->header.needs_swap)
 			perf_event_header__bswap(&event->header);
+		if (decomp)
+			return NULL;
 		pr_debug("%s: head=%#" PRIx64 " event->header_size=%#x, mmap_size=%#zx: fuzzed perf.data?\n",
 			 __func__, head, event->header.size, mmap_size);
 		return ERR_PTR(-EINVAL);

tools/perf/util/session.c static int __perf_session__process_decomp_events(struct perf_session *session) · line 2002

Quoted 5 lines
tools/perf/util/session.c +1 -1
@@ -1997,7 +1999,7 @@ static int __perf_session__process_decomp_events(struct perf_session *session)
 		return 0;
 
 	while (decomp->head < decomp->size && !session_done()) {
-		union perf_event *event = fetch_mmaped_event(session, decomp->head, decomp->size, decomp->data);
+		union perf_event *event = fetch_mmaped_event(session, decomp->head, decomp->size, decomp->data, true);

It looks like this call can be skipped, at all, in this case.

tools/perf/util/session.c reader__process_events(struct reader *rd, struct perf_session *session, · line 2108

Quoted 13 lines
tools/perf/util/session.c +1 -1
 		if (IS_ERR(event))
 			return PTR_ERR(event);
@@ -2100,7 +2102,7 @@ reader__process_events(struct reader *rd, struct perf_session *session,
 	}
 
 more:
-	event = fetch_mmaped_event(session, head, mmap_size, buf);
+	event = fetch_mmaped_event(session, head, mmap_size, buf, false);
 	if (IS_ERR(event))
 		return PTR_ERR(event);
 

~Alexey

author
#3 lore raw
Quoted 54 lines · Alexey Budankov
Quoted 1 line · Jiri Olsa

hi,

<SNIP>

Quoted 27 lines

---
The compressed data processing occasionally fails with:

  $ perf report --stdio -vv
  decomp (B): 44519 to 163000
  decomp (B): 48119 to 174800
  decomp (B): 65527 to 131072
  fetch_mmaped_event: head=0x1ffe0 event->header_size=0x28, mmap_size=0x20000: fuzzed perf.data?
  Error:
  failed to process sample
  ...

It's caused by recent fuzzer fix that does not take into account that compressed data do not need to by fully present in the buffer, so it's ok to just return NULL and not to fail.

Fixes 57fc032ad643 ("perf session: Avoid infinite loop when seeing invalid header.size")
---
 tools/perf/util/session.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
tools/perf/util/session.c +0 -0
index f07b8ecb91bc..3589ed14a629 100644

tools/perf/util/session.c static int __perf_session__process_pipe_events(struct perf_session *session) · line 1962

Quoted 5 lines
tools/perf/util/session.c +1 -1
@@ -1959,7 +1959,7 @@ static int __perf_session__process_pipe_events(struct perf_session *session)
 
 static union perf_event *
 fetch_mmaped_event(struct perf_session *session,
-		   u64 head, size_t mmap_size, char *buf)
+		   u64 head, size_t mmap_size, char *buf, bool decomp)

bools in interface make code less transparent.

Quoted 15 lines
tools/perf/util/session.c +2 -0
 {
 	union perf_event *event;
 
@@ -1979,6 +1979,8 @@ fetch_mmaped_event(struct perf_session *session,
 		/* We're not fetching the event so swap back again */
 		if (session->header.needs_swap)
 			perf_event_header__bswap(&event->header);
+		if (decomp)
+			return NULL;
 		pr_debug("%s: head=%#" PRIx64 " event->header_size=%#x, mmap_size=%#zx: fuzzed perf.data?\n",
 			 __func__, head, event->header.size, mmap_size);
 		return ERR_PTR(-EINVAL);
@@ -1997,7 +1999,7 @@ static int __perf_session__process_decomp_events(struct perf_session *session)
 		return 0;
 
Quoted 5 lines

tools/perf/util/session.c static int __perf_session__process_decomp_events(struct perf_session *session) · line 2002

Quoted 3 lines
tools/perf/util/session.c +1 -1
 	while (decomp->head < decomp->size && !session_done()) {
-		union perf_event *event = fetch_mmaped_event(session, decomp->head, decomp->size, decomp->data);
+		union perf_event *event = fetch_mmaped_event(session, decomp->head, decomp->size, decomp->data, true);

It looks like this call can be skipped, at all, in this case.

not sure what you mean, we are in decomp code no?

jirka

#4 lore raw
Quoted 55 lines · Jiri Olsa
Quoted 54 lines · Alexey Budankov
Quoted 1 line · Jiri Olsa

hi,

<SNIP>

Quoted 27 lines

---
The compressed data processing occasionally fails with:

  $ perf report --stdio -vv
  decomp (B): 44519 to 163000
  decomp (B): 48119 to 174800
  decomp (B): 65527 to 131072
  fetch_mmaped_event: head=0x1ffe0 event->header_size=0x28, mmap_size=0x20000: fuzzed perf.data?
  Error:
  failed to process sample
  ...

It's caused by recent fuzzer fix that does not take into account that compressed data do not need to by fully present in the buffer, so it's ok to just return NULL and not to fail.

Fixes 57fc032ad643 ("perf session: Avoid infinite loop when seeing invalid header.size")
---
 tools/perf/util/session.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
tools/perf/util/session.c +0 -0
index f07b8ecb91bc..3589ed14a629 100644

tools/perf/util/session.c static int __perf_session__process_pipe_events(struct perf_session *session) · line 1962

Quoted 5 lines
tools/perf/util/session.c +1 -1
@@ -1959,7 +1959,7 @@ static int __perf_session__process_pipe_events(struct perf_session *session)
 
 static union perf_event *
 fetch_mmaped_event(struct perf_session *session,
-		   u64 head, size_t mmap_size, char *buf)
+		   u64 head, size_t mmap_size, char *buf, bool decomp)

bools in interface make code less transparent.

Quoted 15 lines
tools/perf/util/session.c +2 -0
 {
 	union perf_event *event;
 
@@ -1979,6 +1979,8 @@ fetch_mmaped_event(struct perf_session *session,
 		/* We're not fetching the event so swap back again */
 		if (session->header.needs_swap)
 			perf_event_header__bswap(&event->header);
+		if (decomp)
+			return NULL;
 		pr_debug("%s: head=%#" PRIx64 " event->header_size=%#x, mmap_size=%#zx: fuzzed perf.data?\n",
 			 __func__, head, event->header.size, mmap_size);
 		return ERR_PTR(-EINVAL);
@@ -1997,7 +1999,7 @@ static int __perf_session__process_decomp_events(struct perf_session *session)
 		return 0;
 
Quoted 7 lines
Quoted 5 lines

tools/perf/util/session.c static int __perf_session__process_decomp_events(struct perf_session *session) · line 2002

Quoted 3 lines
tools/perf/util/session.c +1 -1
 	while (decomp->head < decomp->size && !session_done()) {
-		union perf_event *event = fetch_mmaped_event(session, decomp->head, decomp->size, decomp->data);
+		union perf_event *event = fetch_mmaped_event(session, decomp->head, decomp->size, decomp->data, true);

It looks like this call can be skipped, at all, in this case.

not sure what you mean, we are in decomp code no?

Ok, it is inside "not fetching" branch.
NULL return value means to proceed getting further over the trace. Checking record type == COMPRESSED at the higher level could probably be cleaner fix and also work faster.

~Alexey

Quoted 4 lines

jirka

author
#5 lore raw
Quoted 62 lines · Alexey Budankov
Quoted 61 lines · Jiri Olsa
Quoted 59 lines · Alexey Budankov
Quoted 1 line · Jiri Olsa

hi,

<SNIP>

Quoted 27 lines

---
The compressed data processing occasionally fails with:

  $ perf report --stdio -vv
  decomp (B): 44519 to 163000
  decomp (B): 48119 to 174800
  decomp (B): 65527 to 131072
  fetch_mmaped_event: head=0x1ffe0 event->header_size=0x28, mmap_size=0x20000: fuzzed perf.data?
  Error:
  failed to process sample
  ...

It's caused by recent fuzzer fix that does not take into account that compressed data do not need to by fully present in the buffer, so it's ok to just return NULL and not to fail.

Fixes 57fc032ad643 ("perf session: Avoid infinite loop when seeing invalid header.size")
---
 tools/perf/util/session.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
tools/perf/util/session.c +0 -0
index f07b8ecb91bc..3589ed14a629 100644

tools/perf/util/session.c static int __perf_session__process_pipe_events(struct perf_session *session) · line 1962

Quoted 5 lines
tools/perf/util/session.c +1 -1
@@ -1959,7 +1959,7 @@ static int __perf_session__process_pipe_events(struct perf_session *session)
 
 static union perf_event *
 fetch_mmaped_event(struct perf_session *session,
-		   u64 head, size_t mmap_size, char *buf)
+		   u64 head, size_t mmap_size, char *buf, bool decomp)

bools in interface make code less transparent.

Quoted 13 lines
tools/perf/util/session.c +2 -0
 {
 	union perf_event *event;
 
@@ -1979,6 +1979,8 @@ fetch_mmaped_event(struct perf_session *session,
 		/* We're not fetching the event so swap back again */
 		if (session->header.needs_swap)
 			perf_event_header__bswap(&event->header);
+		if (decomp)
+			return NULL;
 		pr_debug("%s: head=%#" PRIx64 " event->header_size=%#x, mmap_size=%#zx: fuzzed perf.data?\n",
 			 __func__, head, event->header.size, mmap_size);
 		return ERR_PTR(-EINVAL);

tools/perf/util/session.c static int __perf_session__process_decomp_events(struct perf_session *session) · line 2002

Quoted 5 lines
tools/perf/util/session.c +1 -1
@@ -1997,7 +1999,7 @@ static int __perf_session__process_decomp_events(struct perf_session *session)
 		return 0;
 
 	while (decomp->head < decomp->size && !session_done()) {
-		union perf_event *event = fetch_mmaped_event(session, decomp->head, decomp->size, decomp->data);
+		union perf_event *event = fetch_mmaped_event(session, decomp->head, decomp->size, decomp->data, true);

It looks like this call can be skipped, at all, in this case.

Quoted 6 lines
Quoted 1 line

not sure what you mean, we are in decomp code no?

Ok, it is inside "not fetching" branch.
NULL return value means to proceed getting further over the trace. Checking record type == COMPRESSED at the higher level could probably be cleaner fix and also work faster.

any chance you could post the fix? the patch I did was a quick fix to get the feature working for presentation ;-) you're probably thinking of the proper approach

thanks,
jirka

#6 lore raw
Quoted 65 lines · Jiri Olsa
Quoted 64 lines · Alexey Budankov
Quoted 62 lines · Jiri Olsa
Quoted 54 lines · Alexey Budankov
Quoted 1 line · Jiri Olsa

hi,

<SNIP>

Quoted 27 lines

---
The compressed data processing occasionally fails with:

  $ perf report --stdio -vv
  decomp (B): 44519 to 163000
  decomp (B): 48119 to 174800
  decomp (B): 65527 to 131072
  fetch_mmaped_event: head=0x1ffe0 event->header_size=0x28, mmap_size=0x20000: fuzzed perf.data?
  Error:
  failed to process sample
  ...

It's caused by recent fuzzer fix that does not take into account that compressed data do not need to by fully present in the buffer, so it's ok to just return NULL and not to fail.

Fixes 57fc032ad643 ("perf session: Avoid infinite loop when seeing invalid header.size")
---
 tools/perf/util/session.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
tools/perf/util/session.c +0 -0
index f07b8ecb91bc..3589ed14a629 100644

tools/perf/util/session.c static int __perf_session__process_pipe_events(struct perf_session *session) · line 1962

Quoted 5 lines
tools/perf/util/session.c +1 -1
@@ -1959,7 +1959,7 @@ static int __perf_session__process_pipe_events(struct perf_session *session)
 
 static union perf_event *
 fetch_mmaped_event(struct perf_session *session,
-		   u64 head, size_t mmap_size, char *buf)
+		   u64 head, size_t mmap_size, char *buf, bool decomp)

bools in interface make code less transparent.

Quoted 15 lines
tools/perf/util/session.c +2 -0
 {
 	union perf_event *event;
 
@@ -1979,6 +1979,8 @@ fetch_mmaped_event(struct perf_session *session,
 		/* We're not fetching the event so swap back again */
 		if (session->header.needs_swap)
 			perf_event_header__bswap(&event->header);
+		if (decomp)
+			return NULL;
 		pr_debug("%s: head=%#" PRIx64 " event->header_size=%#x, mmap_size=%#zx: fuzzed perf.data?\n",
 			 __func__, head, event->header.size, mmap_size);
 		return ERR_PTR(-EINVAL);
@@ -1997,7 +1999,7 @@ static int __perf_session__process_decomp_events(struct perf_session *session)
 		return 0;
 
Quoted 5 lines

tools/perf/util/session.c static int __perf_session__process_decomp_events(struct perf_session *session) · line 2002

Quoted 3 lines
tools/perf/util/session.c +1 -1
 	while (decomp->head < decomp->size && !session_done()) {
-		union perf_event *event = fetch_mmaped_event(session, decomp->head, decomp->size, decomp->data);
+		union perf_event *event = fetch_mmaped_event(session, decomp->head, decomp->size, decomp->data, true);

It looks like this call can be skipped, at all, in this case.

not sure what you mean, we are in decomp code no?

Quoted 8 lines
Quoted 4 lines

Ok, it is inside "not fetching" branch.
NULL return value means to proceed getting further over the trace. Checking record type == COMPRESSED at the higher level could probably be cleaner fix and also work faster.

any chance you could post the fix? the patch I did was a quick fix to get the feature working for presentation ;-) you're probably thinking of the proper approach

Please share the exact reproducing steps
so I could come up with something.

~Alexey

Quoted 5 lines

thanks,
jirka

author
#7 lore raw
Quoted 71 lines · Alexey Budankov
Quoted 70 lines · Jiri Olsa
Quoted 68 lines · Alexey Budankov
Quoted 55 lines · Jiri Olsa
Quoted 54 lines · Alexey Budankov
Quoted 1 line · Jiri Olsa

hi,

<SNIP>

Quoted 27 lines

---
The compressed data processing occasionally fails with:

  $ perf report --stdio -vv
  decomp (B): 44519 to 163000
  decomp (B): 48119 to 174800
  decomp (B): 65527 to 131072
  fetch_mmaped_event: head=0x1ffe0 event->header_size=0x28, mmap_size=0x20000: fuzzed perf.data?
  Error:
  failed to process sample
  ...

It's caused by recent fuzzer fix that does not take into account that compressed data do not need to by fully present in the buffer, so it's ok to just return NULL and not to fail.

Fixes 57fc032ad643 ("perf session: Avoid infinite loop when seeing invalid header.size")
---
 tools/perf/util/session.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
tools/perf/util/session.c +0 -0
index f07b8ecb91bc..3589ed14a629 100644

tools/perf/util/session.c static int __perf_session__process_pipe_events(struct perf_session *session) · line 1962

Quoted 5 lines
tools/perf/util/session.c +1 -1
@@ -1959,7 +1959,7 @@ static int __perf_session__process_pipe_events(struct perf_session *session)
 
 static union perf_event *
 fetch_mmaped_event(struct perf_session *session,
-		   u64 head, size_t mmap_size, char *buf)
+		   u64 head, size_t mmap_size, char *buf, bool decomp)

bools in interface make code less transparent.

Quoted 15 lines
tools/perf/util/session.c +2 -0
 {
 	union perf_event *event;
 
@@ -1979,6 +1979,8 @@ fetch_mmaped_event(struct perf_session *session,
 		/* We're not fetching the event so swap back again */
 		if (session->header.needs_swap)
 			perf_event_header__bswap(&event->header);
+		if (decomp)
+			return NULL;
 		pr_debug("%s: head=%#" PRIx64 " event->header_size=%#x, mmap_size=%#zx: fuzzed perf.data?\n",
 			 __func__, head, event->header.size, mmap_size);
 		return ERR_PTR(-EINVAL);
@@ -1997,7 +1999,7 @@ static int __perf_session__process_decomp_events(struct perf_session *session)
 		return 0;
 
Quoted 7 lines
Quoted 5 lines

tools/perf/util/session.c static int __perf_session__process_decomp_events(struct perf_session *session) · line 2002

Quoted 3 lines
tools/perf/util/session.c +1 -1
 	while (decomp->head < decomp->size && !session_done()) {
-		union perf_event *event = fetch_mmaped_event(session, decomp->head, decomp->size, decomp->data);
+		union perf_event *event = fetch_mmaped_event(session, decomp->head, decomp->size, decomp->data, true);

It looks like this call can be skipped, at all, in this case.

not sure what you mean, we are in decomp code no?

Ok, it is inside "not fetching" branch.
NULL return value means to proceed getting further over the trace. Checking record type == COMPRESSED at the higher level could probably be cleaner fix and also work faster.

Quoted 6 lines
Quoted 3 lines

any chance you could post the fix? the patch I did was a quick fix to get the feature working for presentation ;-) you're probably thinking of the proper approach

Please share the exact reproducing steps
so I could come up with something.

'perf record -z' for longer workloads

jirka